Skip to content

fix(py/plugins/openai): emit content, reasoning, and tool-call deltas that share a chunk - #6282

Merged
hilariie merged 5 commits into
fix/py-openai-response-metadatafrom
fix/py-openai-shared-chunk-deltas
Sep 11, 2026
Merged

hilariie merged 5 commits into
fix/py-openai-response-metadatafrom
fix/py-openai-shared-chunk-deltas

Conversation

@hilariie

@hilariie hilariie commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Closes #6233

Summary

The streaming loop now reads content, reasoning and tool-call deltas independently and appends them to a single outgoing chunk, in reasoning, text, tool-call order. The chunk is emitted only when it carries something. A tool-call fragment that shares a chunk with text reaches the accumulator, so the aggregate response is complete as well.

MessageConverter.to_genkit gets the same treatment on the non-streaming path: reasoning, text, then one part per tool call, rather than tool calls to the exclusion of both.

A text delta converts straight to a text part instead of going through to_genkit, which would otherwise return the reasoning and tool-call parts riding on the same delta a second time.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the OpenAI model implementation to support emitting and converting reasoning content, text content, and tool calls together in a single stream chunk or message, rather than treating them as mutually exclusive. Previously, tool calls took precedence over reasoning and text content; the changes ensure all three parts are preserved and ordered correctly (reasoning, then text, then tool calls). Unit tests have been added and updated to verify these streaming and conversion behaviors. I have no feedback to provide as there are no review comments.

@cabljac

cabljac commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

One thing worth being explicit about before I approve. From what I can see this brings Python in line with Go (go/plugins/compat_oai/generate.go:827-845 non-stream, :552-570 stream), which already emits reasoning, text and tool calls together, and #6233 says as much. JS fromOpenAIChoice / fromOpenAIChunkChoice (js/plugins/compat-oai/src/model.ts:396-415, :523-539) still make tool calls exclusive of text and reasoning, so it's now the odd one out and I couldn't find an issue tracking that. Happy to file one for JS if you agree it should follow.

Small knock-on: on the non-stream path _clean_json_response in DeepSeek json mode can now see text parts alongside tool parts, which it couldn't before. It filters on TextPart so I think it's fine, but can you run a quick DeepSeek json-mode + tool call case and drop in what you see.

Comment thread py/packages/genkit-openai/src/genkit_openai/models/model.py Outdated
@hilariie
hilariie force-pushed the fix/py-openai-shared-chunk-deltas branch from b83bd33 to 2546985 Compare September 7, 2026 15:31
@hilariie
hilariie force-pushed the fix/py-openai-shared-chunk-deltas branch from 2546985 to 1beae60 Compare September 8, 2026 09:18
@hilariie

hilariie commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

Yeah, a JS issue can follow.

On the DeepSeek json-mode + tool call part comment, I ran it: ai.generate with an output_schema and a tool against deepseek-chat in json mode, transport mocked so round 1 returns fenced JSON content and a tool call in one message.

round 1 model message: [TextPart('{"city": "NYC", "temp_c": 0}'), ToolRequestPart(get_weather)] get_weather runs

round 2: .output = Weather(city='NYC', temp_c=7), finish_reason=stop Fences stripped, the tool part carried through untouched, raw and custom kept. Pinned in TestCleanJsonResponse.test_keeps_tool_parts_beside_cleaned_text.

I found out that to_openai splits a mixed model message into two assistant messages, one with content and one with tool_calls, where Go builds a single assistant message carrying both (generate.go:87-103). It was unreachable before this PR because to_genkit returned either tool calls or text, never both. I have not seen a provider reject it, so I would rather file it than grow this PR.

@hilariie
hilariie force-pushed the fix/py-openai-shared-chunk-deltas branch from 1beae60 to 535f759 Compare September 8, 2026 10:22
@cabljac
cabljac force-pushed the fix/py-openai-shared-chunk-deltas branch from 535f759 to 556f708 Compare September 10, 2026 12:24

@cabljac cabljac left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, this is the right shape: three independent ifs, one callback per chunk, and a single order that to_genkit and the stream loop now share. The if parts: guard is the detail I like most. Refusal-only and heartbeat deltas emitted nothing under the old if/elif chain by accident, and a naive rewrite would have started sending empty chunks; you kept it deliberately.

I ran the suite on the branch (257 pass) and broke each behaviour the PR claims to check the tests bite. They all do, including the ordering and the null-fragment case.

One thing worth digging into on the null-arguments thread, then two inline.

Comment thread py/packages/genkit-openai/src/genkit_openai/models/utils.py
Comment thread py/packages/genkit-openai/src/genkit_openai/models/model.py Outdated
@cabljac
cabljac force-pushed the fix/py-openai-shared-chunk-deltas branch from 556f708 to d9642de Compare September 10, 2026 16:41
@hilariie
hilariie force-pushed the fix/py-openai-shared-chunk-deltas branch from d9642de to 7275c15 Compare September 11, 2026 10:29
@hilariie
hilariie force-pushed the fix/py-openai-shared-chunk-deltas branch from 7275c15 to c87b4ea Compare September 11, 2026 12:36
@hilariie
hilariie merged commit c1d820e into main Sep 11, 2026
24 checks passed
@hilariie
hilariie deleted the fix/py-openai-shared-chunk-deltas branch September 11, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(py/plugins/openai): streaming loop drops content, reasoning, or tool-call deltas that share a chunk

2 participants