Skip to content

Adopt custom Copilot SDK with session-scoped OTel - #339640

Closed
roblourens wants to merge 3 commits into
mainfrom
roblou/agents/private-copilot-sdk-runtime-build
Closed

roblourens wants to merge 3 commits into
mainfrom
roblou/agents/private-copilot-sdk-runtime-build

Conversation

@roblourens

@roblourens roblourens commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Summary

Pin the signed custom Copilot SDK 1.0.15-unstable.37250855457.gbf21394 in the root and remote dependency sets. It bundles runtime 1.0.92-0.unstable.r37250855457.gbf21394, based directly on the runtime 1.0.92-0 already bundled by SDK 1.0.17-preview.0.

While this private build was publishing, #339423 landed SDK 1.0.17-preview.4 / runtime 1.0.92-4 on main, including the no-restart fix. This branch merges that current main and intentionally retains the specifically requested private build based on 1.0.92-0; it is an isolation/adoption draft, not a claim that the regular SDK now lacks the OTel fix.

The runtime backport includes github/copilot-agent-runtime#24069, allowing new sessions to honor changed managed telemetry policy without restarting the Agent Host. Required OTel portions of github/copilot-agent-runtime#23807, github/copilot-agent-runtime#24493, github/copilot-agent-runtime#24655, and github/copilot-agent-runtime#24112 are included. This is a prerequisite-stack backport, not the target PR alone. Unrelated benchmark, MCP, sandbox, mode-switch, and release-machinery changes are excluded.

The actual published SDK completed both turns and exported correlated inference spans under service A and then service B without a host restart. The no-restart regression is a normal passing test with complete replay consumption; its promotion and known-issue removal also landed in the intervening SDK roll and are retained from main.

Regenerate all 19 model prompt snapshots from strict, tokenless replay to reflect this private runtime's tool descriptors and model-enum ordering. The write_agent descriptor retains the baseline's queued follow-up behavior rather than the newer runtime's steering description. Adapt three test helpers to the pinned SDK's sandbox RPC type, which does not contain the newer proxyCa member. No production Agent Host changes or model fixture edits are required.

Build provenance

The SDK version prefix comes from the publishing workflow's release-history baseline; it does not indicate different SDK source. Public schema/projection checks passed unchanged.

Validation

  • npm run transpile-client — passed.
  • npm run typecheck-client — passed against the published SDK.
  • ./scripts/test-integration.sh --run src/vs/platform/agentHost/test/node/e2e/providers/copilotOtelAgentHostE2E.integrationTest.ts — passed, all four OTel scenarios (5 runner tests including the clean-state check).
  • ./scripts/test-integration.sh --run src/vs/platform/agentHost/test/node/e2e/providers/copilotOtelAgentHostE2E.integrationTest.ts --grep 'new sessions honor changed managed telemetry without restarting' — passed again, 1 test; the timing-sensitive no-restart scenario passed twice after removing the marker.
  • COPILOT_OTEL_ENABLED=false COPILOT_OTEL_DB_SPAN_EXPORTER_ENABLED=false OTEL_METRICS_EXPORTER=none ./scripts/test-integration.sh --run src/vs/platform/agentHost/test/node/e2e/providers/copilotAgentHostE2E.integrationTest.ts — passed, 234 tests with 25 existing pending tests; validates normal provider operation with OTel disabled.
  • ./scripts/test-integration.sh --run src/vs/platform/agentHost/test/node/e2e/conformance/agentHostConformance.integrationTest.ts — passed, 350 tests with 15 existing pending tests.
  • AGENT_HOST_UPDATE_AHP_SNAPSHOTS=1 ./scripts/test-integration.sh --run src/vs/platform/agentHost/test/node/e2e/providers/copilotPromptsE2E.integrationTest.ts — passed, 24 tests; generated the private-runtime prompt baselines without model traffic or credentials.
  • ./scripts/test-integration.sh --run src/vs/platform/agentHost/test/node/e2e/providers/copilotPromptsE2E.integrationTest.ts — passed, 24 tests against the regenerated baselines in strict replay.
  • env -u COPILOT_HOME ./scripts/test.sh --run src/vs/platform/agentHost/test/node/copilotAgent.test.ts --run src/vs/platform/agentHost/test/node/copilotAgentSession.test.ts --run src/vs/platform/agentHost/test/node/copilotSessionLauncher.test.ts --run src/vs/platform/agentHost/test/node/toolInstructions.test.ts --run src/vs/platform/agentHost/test/node/agentHostPromptOverride.test.ts --run src/vs/platform/agentHost/test/node/agentHostPromptRegistry.test.ts — passed, 1,495 tests and 1 existing pending test after merging current main. The first pre-merge run inherited the agent shell's real COPILOT_HOME, causing path-based fixture failures; removing that environment override restored the test's intended isolated paths without production code changes.
  • ./scripts/test-integration.sh --run src/vs/platform/agentHost/test/node/otel/agentHostOTelService.integrationTest.ts — passed, 33 tests.
  • node --test build/lib/test/copilot.test.ts build/lib/test/agentHostDependencies.test.ts — passed, 25 packaging tests.
  • node build/hygiene.ts package.json package-lock.json remote/package.json remote/package-lock.json src/vs/platform/agentHost/test/node/e2e/providers/copilotOtelAgentHostE2E.integrationTest.ts src/vs/platform/agentHost/test/node/e2e/README.md src/vs/platform/agentHost/test/node/e2e/KNOWN_ISSUES.md — passed.
  • git diff --check — passed.
  • npm run gulp -- --tasks-simple — passed; the actual CI gulp task graph loaded with the matching declared runtime version.
  • node --test build/lib/test/hygiene.test.ts — passed, 1 test.

Current cross-platform validation targets signed commit c0da89dad64e3afdda9c435166bd4c59b4964817:

  • Azure build 480542 — succeeded, including packaged Electron integration, unit, smoke, browser/remote, and CLI validation on Windows x64, Linux x64, and macOS ARM64.
  • Code OSS pull-request workflow 37264083689 — passed all 21 jobs, including Windows/Linux/macOS Electron integration, unit, smoke, browser, remote, Copilot, compile, and hygiene checks.

Windows and Linux x64 plus macOS are enabled; the current CI graph executes macOS tests on ARM64 despite the macOS product selector. Publishing, release, Web, alternate product architectures, Alpine, and Snap are disabled. All selected Azure stages passed, including Electron integration tests on every supported platform. Code OSS job logs explicitly confirm new sessions honor changed managed telemetry without restarting ran and passed on Windows, Linux, and macOS. The PR remains a draft as requested.

The initial Azure builds 480535 and 480540 were cancelled as obsolete. Code OSS run 37262643627 caught stale root/remote copilotRuntimeVersion fields after merging main: the declared regular-runtime version differed from the private SDK's runtime, causing a shared gulp initialization guard to reject compile/transpile before tests. Commit c0da89dad64e3afdda9c435166bd4c59b4964817 fixes both metadata fields; the guard passes locally and the complete replacement Code OSS workflow passed. Both CI systems use the corrected commit. No runtime or SDK package change was required.

Queue command (passed; build queued):

node .github/skills/azure-pipelines/azure-pipeline.ts queue --parameter 'VSCODE_BUILD_TYPE=CI' --parameter 'VSCODE_BUILD_WIN32=true' --parameter 'VSCODE_BUILD_LINUX=true' --parameter 'VSCODE_BUILD_MACOS=true' --parameter 'VSCODE_BUILD_WIN32_ARM64=false' --parameter 'VSCODE_BUILD_LINUX_SNAP=false' --parameter 'VSCODE_BUILD_LINUX_ARM64=false' --parameter 'VSCODE_BUILD_LINUX_ARMHF=false' --parameter 'VSCODE_BUILD_ALPINE=false' --parameter 'VSCODE_BUILD_ALPINE_ARM64=false' --parameter 'VSCODE_BUILD_MACOS_ARM64=false' --parameter 'VSCODE_BUILD_MACOS_UNIVERSAL=false' --parameter 'VSCODE_BUILD_WEB=false' --parameter 'VSCODE_PUBLISH=false' --parameter 'VSCODE_RELEASE=false' --parameter 'VSCODE_STEP_ON_IT=false'

The runtime backport also passed focused Rust OTel/core/managed-settings/admission suites, TypeScript host and SDK checks, CLI/SDK replay E2Es, native-process policy cutover/resume tests, lint/types/format, Clippy, and Rust documentation checks before publication.

Known behavior and release bookkeeping

The requested upstream behavior is retained: invalid optional OTel exporter configuration can reject session create/resume, where the baseline continued with a warning. An offline baseline-versus-candidate public-SDK matrix confirmed this for a file exporter without a path and missing TLS certificate/key; telemetry-disabled create and persisted cold resume passed both versions. The maintainer explicitly accepted this behavior for this private build. No runtime behavior fix is included in this adoption PR.

An Opus review found no backport-specific missing prerequisite, new deadlock, crash, or persisted-schema defect. It did identify broader upstream skill/tool telemetry and policy lifecycle changes, so this is not characterized as an exporter-only change.

The successful release workflow skipped its final SDK/language source-tag step. The public npm package, signed runtime assets, immutable source snapshot, retained release identity, and source SHA are verified; missing final release tags are an outstanding bookkeeping issue, not a missing Node package. The maintainer authorized VS Code validation/adoption without repairing release machinery or tags.

(Written by Copilot)

Pin the signed runtime 1.0.92-0 backport of session-scoped OTel and its required telemetry prerequisites. Restore strict no-restart regression coverage.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 5, 2026 04:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The root and remote copilotRuntimeVersion fields remain stale, causing the product build’s exact SDK runtime-version check to fail.

Review effort: Balanced
Findings: 2 High severity

Open (2)
What changed in this PR

Adopts a custom Copilot SDK backport enabling session-scoped OTel policy changes without restarting Agent Host.

Changes:

  • Pins the custom SDK across root and remote dependencies.
  • Enables strict replay coverage for no-restart telemetry changes.
  • Removes the resolved known-issue documentation.
File Description
package.json Updates the root SDK dependency.
package-lock.json Locks SDK and platform packages.
remote/​package.json Updates the remote SDK dependency.
remote/​package-lock.json Locks remote SDK packages.
copilotOtelAgentHostE2E.integrationTest.ts Enables the fixed no-restart scenario.
README.md Updates OTel test documentation.
KNOWN_ISSUES.md Removes the resolved limitation.
Files not reviewed (1)
  • remote/package-lock.json: Generated file

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread package.json
"@anthropic-ai/sdk": "^0.82.0",
"@devcontainers/cli": "0.88.0",
"@github/copilot-sdk": "1.0.17-preview.0",
"@github/copilot-sdk": "1.0.15-unstable.37250855457.gbf21394",
Comment thread remote/package.json
"dependencies": {
"@devcontainers/cli": "0.88.0",
"@github/copilot-sdk": "1.0.17-preview.0",
"@github/copilot-sdk": "1.0.15-unstable.37250855457.gbf21394",
Preserve the requested runtime 1.0.92-0 private build while integrating the intervening regular SDK rollout. Align test-only sandbox mocks and generated prompt snapshots with this package.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Screenshot Changes

Base: 40bb6fae Current: 92cb275a

Changed (2)

sessions/accountMenu/WeeklyAndFiveHourLimits/Light
Before After
before after
chat/aiCustomizations/aiCustomizationManagementEditor/DiscoverPluginsLoadingMore/Light
Before After
before after

Keep root and remote copilotRuntimeVersion metadata aligned with the exact published SDK runtime so the CI gulp task graph passes its version guard.

(Written by Copilot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants