Skip to content

test: harden Agents async timeouts and synchronization - #994

Merged
apcha-oai merged 3 commits into
mainfrom
apcha/double-agent-stream-test-timeout
Sep 10, 2026
Merged

test: harden Agents async timeouts and synchronization#994
apcha-oai merged 3 commits into
mainfrom
apcha/double-agent-stream-test-timeout

Conversation

@apcha-oai

@apcha-oai apcha-oai commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • standardize handwritten Agents async test waits on a 30-second harness deadline
  • replace busy-spin polling with event-driven response and handler synchronization
  • make cancellation registration deterministic and surface async completion errors on the test thread
  • close test clients and await executor shutdown to avoid leaking resources between tests

Testing

  • ./gradlew :openai-java-core:test --tests "com.openai.services.beta.agents.AgentSessionStreamTest" --tests "com.openai.services.blocking.beta.agents.environments.EnvironmentTokenPaginationTest" :openai-java-core:lintKotlin
  • 10 uncached repetitions of both handwritten test classes

The original release attempt and its retry both timed out at AgentSessionStreamTest.consume under the GraalVM native-image agent:
https://github.com/openai/openai-java/actions/runs/34522715468

@apcha-oai
apcha-oai requested a review from a team as a code owner September 10, 2026 20:32
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-10T21:34:29.780212Z 1e3f18f New commits
🔒 Security Review Completed 2026-09-10T21:35:02.281760Z 1e3f18f New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@openai-sdks

openai-sdks Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

OkTest Summary

236/236 SDK tests passed in 16.627s for Java SDK PR #994.

Test results — 42 files
Test Result Time
tests/chat-completions-complex-body.test.ts ✅ Passed 245ms
tests/chat-completions-create.test.ts ✅ Passed 610ms
tests/chat-completions-stream.test.ts ✅ Passed 766ms
tests/files-content-binary.test.ts ✅ Passed 316ms
tests/files-create-multipart.test.ts ✅ Passed 528ms
tests/files-list-pagination.test.ts ✅ Passed 266ms
tests/initialize-config.test.ts ✅ Passed 279ms
tests/instance-isolation.test.ts ✅ Passed 184ms
tests/models-list.test.ts ✅ Passed 392ms
tests/responses-background-lifecycle.test.ts ✅ Passed 392ms
tests/responses-body-method-errors.test.ts ✅ Passed 512ms
tests/responses-cancel-timeout.test.ts ✅ Passed 272ms
tests/responses-cancel.test.ts ✅ Passed 280ms
tests/responses-compact-retries.test.ts ✅ Passed 370ms
tests/responses-compact.test.ts ✅ Passed 249ms
tests/responses-create-advanced-stream.test.ts ✅ Passed 323ms
tests/responses-create-advanced.test.ts ✅ Passed 1.326s
tests/responses-create-disconnect.test.ts ✅ Passed 1.139s
tests/responses-create-errors.test.ts ✅ Passed 352ms
tests/responses-create-malformed-api-responses.test.ts ✅ Passed 281ms
tests/responses-create-retries.test.ts ✅ Passed 447ms
tests/responses-create-stream-failures.test.ts ✅ Passed 279ms
tests/responses-create-stream-timeout.test.ts ✅ Passed 302ms
tests/responses-create-stream-wire.test.ts ✅ Passed 5.91s
tests/responses-create-stream.test.ts ✅ Passed 538ms
tests/responses-create-terminal-states.test.ts ✅ Passed 319ms
tests/responses-create-timeout.test.ts ✅ Passed 238ms
tests/responses-create.test.ts ✅ Passed 1.014s
tests/responses-delete.test.ts ✅ Passed 310ms
tests/responses-input-items-errors.test.ts ✅ Passed 283ms
tests/responses-input-items-list.test.ts ✅ Passed 322ms
tests/responses-input-items-options.test.ts ✅ Passed 171ms
tests/responses-input-tokens-count-timeout.test.ts ✅ Passed 336ms
tests/responses-input-tokens-count.test.ts ✅ Passed 269ms
tests/responses-malformed-inputs.test.ts ✅ Passed 4.725s
tests/responses-not-found-errors.test.ts ✅ Passed 380ms
tests/responses-parse.test.ts ✅ Passed 665ms
tests/responses-retrieve-retries.test.ts ✅ Passed 447ms
tests/responses-retrieve.test.ts ✅ Passed 288ms
tests/responses-stored-method-errors.test.ts ✅ Passed 966ms
tests/retry-behavior.test.ts ✅ Passed 3.482s
tests/sdk-error-shape.test.ts ✅ Passed 477ms

View OkTest run #34532786694

SDK merge (0171cea4a4bd) · head (1e3f18fc6351) · base (e2ac1691d0b3) · OkTest (08ec4299e769)

@github-actions

github-actions Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Castiron custom code

✅ No new custom-code files detected.

70 mixed files remain; 0 existing customizations changed.

Compared e2ac1691d0b31e3f18fc6351. Generated baselines verified.

70 existing customizations unchanged
  • openai-java-core/src/main/kotlin/com/openai/models/audio/AudioResponseFormat.kt
  • openai-java-core/src/main/kotlin/com/openai/models/chat/completions/ChatCompletionCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/chat/completions/ChatCompletionMessageFunctionToolCall.kt
  • openai-java-core/src/main/kotlin/com/openai/models/chat/completions/ChatCompletionToolMessageParam.kt
  • openai-java-core/src/main/kotlin/com/openai/models/embeddings/Embedding.kt
  • openai-java-core/src/main/kotlin/com/openai/models/embeddings/EmbeddingCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseFunctionToolCall.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseFunctionWebSearch.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseInputItem.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseTextConfig.kt
  • openai-java-core/src/main/kotlin/com/openai/models/videos/Video.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/BetaServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/BetaServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/ResponseServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/ResponseServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/WebhookServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/WebhookServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/audio/TranscriptionServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/beta/agents/SessionServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/beta/agents/SessionServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/chat/ChatCompletionServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/finetuning/checkpoints/PermissionServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/BetaService.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/BetaServiceImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/ResponseService.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/ResponseServiceImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/WebhookService.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/WebhookServiceImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/audio/TranscriptionServiceImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/beta/agents/SessionService.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/beta/agents/SessionServiceImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/chat/ChatCompletionService.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/finetuning/checkpoints/PermissionServiceImpl.kt
  • openai-java-core/src/test/kotlin/com/openai/models/beta/responses/BetaResponseFunctionWebSearchTest.kt
  • openai-java-core/src/test/kotlin/com/openai/models/beta/responses/BetaResponsesServerEventTest.kt
  • openai-java-core/src/test/kotlin/com/openai/models/live/ClientEventTest.kt
  • openai-java-core/src/test/kotlin/com/openai/models/live/ServerEventTest.kt
  • openai-java-core/src/test/kotlin/com/openai/models/live/SessionClosedEventTest.kt
  • openai-java-core/src/test/kotlin/com/openai/models/live/SessionStartEventTest.kt

30 more in the full report.

A changed generated baseline means this report cannot reliably identify which handwritten lines changed.

Inspect the custom-code diff

Download the exact patch produced by this run (requires repository access):

gh run download 34532821986 --repo openai/openai-java \
  --name castiron-custom-code-34532821986-1 --dir /tmp/castiron-custom-code-34532821986-1
git apply --stat /tmp/castiron-custom-code-34532821986-1/custom-code.patch
cat /tmp/castiron-custom-code-34532821986-1/custom-code.patch

Or reproduce it from an SDK checkout containing the vendored reporter:

git fetch --no-tags origin e2ac1691d0b313582de19ca688822ef0e1ee31b0 1e3f18fc63518839079cb093939c883d7cd0f4a7
python3 scripts/castiron/custom_code_report.py report \
  --base e2ac1691d0b313582de19ca688822ef0e1ee31b0 \
  --head 1e3f18fc63518839079cb093939c883d7cd0f4a7 --fetch --require-head-hash --public \
  --out /tmp/castiron-custom-code-1e3f18fc6351
cat /tmp/castiron-custom-code-1e3f18fc6351/custom-code.patch

This is the current full custom patch for mixed files, not an attribution of only the handwritten lines changed by this PR.

Full report and patch

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 837cd16f03

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One in-scope blocking issue remains on this exact head: the blanket timeout expansion changes the deliberately distinct stream.close() nonblocking assertion from 1 second to 30 seconds. That allows a severe responsiveness regression to pass despite the test contract (closeDoesNotWaitForClaimedToolPostOrStartAnother). Please keep a short bound for that assertion while retaining the larger scheduling/completion budgets elsewhere. The existing inline P2 comment points to the exact line. I also verified the cited native-image failure was the consume(...).get(10s) timeout and confirmed the originally flaky bounded-cache case passes with the new completion budget.

@apcha-oai apcha-oai changed the title test: increase agent stream completion timeout test: harden Agents async timeouts and synchronization Sep 10, 2026

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the complete test-only diff at exact head 59f40cdd70f4025f667129c7f99bc23ccb8e356b. The prior blocking issue is fixed: the nonblocking stream.close() assertion again has its dedicated 1-second bound, while the larger 30-second budget is confined to load-sensitive scheduling and completion waits. The new latch/future synchronization removes polling races, response/client/executor cleanup is explicit, git diff --check passes, and CI lint/build/compatibility jobs are green (the full test job is still running). No unresolved in-scope blocking findings.

@apcha-oai
apcha-oai force-pushed the apcha/double-agent-stream-test-timeout branch from 59f40cd to 1e3f18f Compare September 10, 2026 21:32
@apcha-oai
apcha-oai enabled auto-merge September 10, 2026 21:35
@apcha-oai
apcha-oai added this pull request to the merge queue Sep 10, 2026
Merged via the queue into main with commit 41bdd89 Sep 10, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants