Capture OpenAI multimodal image content (image_url and Responses API input_image) as BlobPart, UriPart, or FilePart message parts by rads-1996 · Pull Request #540 · open-telemetry/opentelemetry-python-genai

rads-1996 · 2026-09-02T19:11:05Z

Description

Fixes # (#348)

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How has this been tested?

Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. List any relevant details for your test
configuration.

uv run tox -e py312-test-instrumentation-genai-langchain -- -q
uv run tox -e py312-test-instrumentation-genai-langchain-conformance -- -q
uv run --python 3.12 tox -e lint-instrumentation-genai-langchain

Checklist

See CONTRIBUTING.md
for the style guide, changelog guidance, and more.

Followed the style guidelines of this project
Changelog updated if the change requires an entry
Unit tests added
Documentation updated

opentelemetry-pr-dashboard · 2026-09-02T19:19:32Z

Pull request dashboard status

Waiting on the author · refreshed 2026-09-04 22:20 UTC

Resolve merge conflicts.

Respond to 3 review items (e.g. link a commit, explain why not, ask a follow-up):

Inline threads: 1, 2
Top-level threads: 3

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Should this be with reviewers? Comment /dashboard route:reviewers to route it to them.
Anything wrong — including the routing? Report it with what you expected; it helps us improve the dashboard.

Copilot

🟡 Changes recommended

_content_to_parts() currently drops image_url values when they are objects with a .url attribute (only dict-mappings are handled), which can silently miss image parts for SDK-model style inputs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds multimodal (image) input capture to the OpenAI GenAI instrumentation by converting OpenAI image_url (Chat Completions) and input_image (Responses API) content blocks into structured MessagePart instances (BlobPart, UriPart, FilePart) so they can be emitted via the existing GenAI message content model.

Changes:

Introduces _content_to_parts() to parse mixed text/image content blocks into MessagePart lists and wires it into input-message extraction paths.
Expands unit and conformance coverage for multimodal image inputs, including new conformance scenarios and VCR cassettes.
Adds a towncrier changelog fragment documenting the new capability.

File summaries

File	Description
instrumentation/opentelemetry-instrumentation-genai-openai/src/opentelemetry/instrumentation/genai/openai/utils.py	Adds `_content_to_parts()` and uses it when preparing input messages to capture text + image parts.
instrumentation/opentelemetry-instrumentation-genai-openai/src/opentelemetry/instrumentation/genai/openai/response_extractors.py	Reuses `_content_to_parts()` for Responses API input-message extraction (supports images + file IDs).
instrumentation/opentelemetry-instrumentation-genai-openai/tests/test_utils.py	Adds focused unit tests for `_content_to_parts()` across text, URL images, and data-URL images.
instrumentation/opentelemetry-instrumentation-genai-openai/tests/test_response_extractors.py	Updates/extends extraction tests to assert `UriPart`/`BlobPart`/`FilePart` results for multimodal inputs.
instrumentation/opentelemetry-instrumentation-genai-openai/tests/test_conformance.py	Registers new multimodal scenarios in the conformance suite.
instrumentation/opentelemetry-instrumentation-genai-openai/tests/conformance/multimodal.py	Adds new conformance scenarios validating that an image blob part appears on input messages.
instrumentation/opentelemetry-instrumentation-genai-openai/tests/cassettes/chat_completions_multimodal_conformance.yaml	VCR cassette for Chat Completions multimodal conformance scenario.
instrumentation/opentelemetry-instrumentation-genai-openai/tests/cassettes/responses_multimodal_conformance.yaml	VCR cassette for Responses API multimodal conformance scenario.
instrumentation/opentelemetry-instrumentation-genai-openai/.changelog/540.added	Changelog fragment for the new multimodal capture behavior.

Review details

Files reviewed: 9/9 changed files
Comments generated: 2
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

lmolkova

Would this work for streaming? Do we have any test coverage for streaming / async multi-modal content?

lmolkova · 2026-09-03T19:44:09Z

@@ -205,8 +252,7 @@ def _prepare_input_messages(messages) -> list[InputMessage]:
            tool_calls = get_property_value(message, "tool_calls")
            if tool_calls:
                chat_message.parts += extract_tool_calls_new(tool_calls)


In response_extractors.py, messages with empty parts are dropped (if parts: messages.append(...)). Should _prepare_input_messages also avoid appending messages when parts is empty?

Yes, that makes sense. I will add changes for it.

…input_image) as BlobPart, UriPart, or FilePart message parts

rads-1996 · 2026-09-04T19:19:41Z

Would this work for streaming? Do we have any test coverage for streaming / async multi-modal content?

Yes, this would work for streaming. I had missed adding the test coverage for it earlier. Added it now. Thanks for pointing it out.

lmolkova

Thanks for the update! It seems there is a regression my AI bot found, can you please check ?

lmolkova · 2026-09-04T20:21:10Z

+
+        part_type = get_property_value(item, "type")
+        text = get_property_value(item, "text")
+        if part_type in ("text", "input_text") or (


Regression: the type allowlist drops Responses API output_text parts, so a prior assistant turn fed back into input disappears from gen_ai.input.messages. The old code took any part with a text field.

Fails on this branch, passes on main:

def test_extract_input_messages_keeps_assistant_output_text(loaded_module): messages = loaded_module.get_input_messages( [ {"role": "user", "content": [{"type": "input_text", "text": "Hi"}]}, { "role": "assistant", "content": [ { "type": "output_text", "text": "Hello!", "annotations": [], } ], }, ] ) assert [(msg.role, msg.parts) for msg in messages] == [ ("user", [TextPart(content="Hi")]), ("assistant", [TextPart(content="Hello!")]), ]

ResponsesConversationScenario uses previous_response_id, so nothing else covers multi-turn Responses input.

lmolkova · 2026-09-04T20:21:11Z

        )


+def test_chat_completion_streaming_captures_multimodal_input(


Sync non-streaming multimodal is uncovered - async has test_async_chat_completion_captures_multimodal_input, sync only gets the streaming variant.

rads-1996 force-pushed the multimodal-openai-changes branch from 2ec1848 to 568ca8b Compare September 2, 2026 19:12

opentelemetry-pr-dashboard Bot mentioned this pull request Sep 2, 2026

Pull Request Dashboard #171

Open

rads-1996 force-pushed the multimodal-openai-changes branch 2 times, most recently from 5698089 to 8564fb8 Compare September 2, 2026 20:21

rads-1996 marked this pull request as ready for review September 2, 2026 22:57

rads-1996 requested a review from a team as a code owner September 2, 2026 22:57

Copilot AI lite review requested due to automatic review settings September 2, 2026 22:57

Copilot started reviewing on behalf of rads-1996 September 2, 2026 22:57 View session

Copilot AI reviewed Sep 2, 2026

View reviewed changes

Comment thread ...lemetry-instrumentation-genai-openai/src/opentelemetry/instrumentation/genai/openai/utils.py

Comment thread instrumentation/opentelemetry-instrumentation-genai-openai/tests/test_utils.py

lmolkova approved these changes Sep 3, 2026

View reviewed changes

aabmass approved these changes Sep 3, 2026

View reviewed changes

rads-1996 added 5 commits September 4, 2026 12:17

Capture OpenAI multimodal image content (image_url and Responses API …

68aae3a

…input_image) as BlobPart, UriPart, or FilePart message parts

Add CHANGELOG

47b84ea

Fix formatting

68d88eb

Address feedback

bd1aecd

Add the parts check and added tests for async and streaming scenarios

35db213

rads-1996 force-pushed the multimodal-openai-changes branch from e9af0ca to 35db213 Compare September 4, 2026 19:18

lmolkova reviewed Sep 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

Capture OpenAI multimodal image content (image_url and Responses API input_image) as BlobPart, UriPart, or FilePart message parts - #540

Capture OpenAI multimodal image content (image_url and Responses API input_image) as BlobPart, UriPart, or FilePart message parts#540
rads-1996 wants to merge 5 commits into
open-telemetry:mainfrom
rads-1996:multimodal-openai-changes

rads-1996 commented Sep 2, 2026

Uh oh!

opentelemetry-pr-dashboard Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

lmolkova left a comment

Uh oh!

lmolkova Sep 3, 2026

Uh oh!

rads-1996 Sep 3, 2026

Uh oh!

rads-1996 Sep 4, 2026

Uh oh!

Uh oh!

rads-1996 commented Sep 4, 2026

Uh oh!

lmolkova left a comment

Uh oh!

lmolkova Sep 4, 2026

Uh oh!

lmolkova Sep 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		)


		def test_chat_completion_streaming_captures_multimodal_input(

Uh oh!

Conversation

rads-1996 commented Sep 2, 2026

Description

Type of change

How has this been tested?

Checklist

Uh oh!

opentelemetry-pr-dashboard Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

lmolkova left a comment

Choose a reason for hiding this comment

Uh oh!

lmolkova Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

rads-1996 Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

rads-1996 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rads-1996 commented Sep 4, 2026

Uh oh!

lmolkova left a comment

Choose a reason for hiding this comment

Uh oh!

lmolkova Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

opentelemetry-pr-dashboard Bot commented Sep 2, 2026 •

edited

Loading