Skip to content

describe_image fails with 403 for openai-codex provider: calls /chat/completions instead of /codex/responses #23

Open
Open
describe_image fails with 403 for openai-codex provider: calls /chat/completions instead of /codex/responses#23

Description

@solyarisoftware
solyarisoftware
opened on Aug 9, 2026

Summary

When vision.json is configured with the openai-codex provider (e.g. openai-codex / gpt-5.6-terra), describe_image always fails with HTTP 403:

Vision tool error: Vision model returned 403: <html>...<meta name="viewport" .../></html>

Root cause

callVisionModel in lib/delegate.ts unconditionally POSTs to ${baseUrl}/chat/completions. For the openai-codex provider, baseUrl is https://chatgpt.com/backend-api, and the Codex backend:

  • does not expose /chat/completions (hence the 403),
  • instead requires the Responses API over SSE at https://chatgpt.com/backend-api/codex/responses (same URL resolution pi's own openai-codex-responses provider uses in resolveCodexUrl).

The extension already receives the resolved Model (which carries api: "openai-codex-responses") and the OAuth access token via getApiKeyAndHeaders, but the request shape is hard-coded for OpenAI-compatible chat/completions.

Backend constraints (verified empirically against the live backend)

Iterating on the raw endpoint, the Codex backend rejects requests that don't match:

Requirement Detail
Endpoint {baseUrl}/codex/responses (not /chat/completions, not /responses)
store must be false → 400 {"detail":"Store must be set to false"} otherwise
stream must be true → 400 {"detail":"Stream must be set to true"} otherwise
temperature unsupported → 400 {"detail":"Unsupported parameter: temperature"}
max_output_tokens / max_tokens unsupported → 400 {"detail":"Unsupported parameter: max_output_tokens"}
Image part {"type":"input_image","image_url":"data:image/jpeg;base64,..."}
Text part {"type":"input_text","text":"..."}
Headers Authorization: Bearer <oauth access token>, chatgpt-account-id: <from JWT claim "https://api.openai.com/auth".chatgpt_account_id>, originator, OpenAI-Beta: responses=experimental, Accept: text/event-stream
Output SSE events; text is assembled from response.output_text.delta deltas (response.completed ends the stream)

Proposed fix

Branch inside callVisionModel on visionModel.api === "openai-codex-responses" and use the Responses-API-over-SSE path for that provider, keeping the existing chat/completions path for all other providers. The OAuth access token from getApiKeyAndHeaders(model).apiKey is a JWT; chatgpt-account-id is extracted from its payload (mirroring extractAccountId in pi's openai-codex-responses.js).

Full patch (against main, 0.5.2):

@@ -68,6 +68,150 @@
   return { reasoning_effort: level };
 }
 
+// ── OpenAI Codex backend (openai-codex-responses) support ─────────────
+// The Codex backend (chatgpt.com/backend-api) does NOT expose /chat/completions:
+// it requires the Responses API over SSE at /codex/responses, with specific
+// auth headers (chatgpt-account-id from the OAuth JWT, originator) and a body
+// that forbids temperature / max_output_tokens. This mirrors how pi's own
+// openai-codex-responses provider builds its requests.
+function isCodexApi(visionModel: Model<Api>): boolean {
+  return visionModel.api === "openai-codex-responses";
+}
+
+/** Extract the `chatgpt-account-id` claim from the Codex OAuth access token. */
+function extractCodexAccountId(apiKey: string | undefined): string | undefined {
+  if (!apiKey) return undefined;
+  try {
+    const parts = apiKey.split(".");
+    const payloadPart = parts[1];
+    if (parts.length !== 3 || !payloadPart) return undefined;
+    const payload = JSON.parse(atob(payloadPart)) as Record<string, unknown>;
+    const auth = payload?.["https://api.openai.com/auth"] as
+      | { chatgpt_account_id?: unknown }
+      | undefined;
+    const accountId = auth?.chatgpt_account_id;
+    return typeof accountId === "string" ? accountId : undefined;
+  } catch {
+    return undefined;
+  }
+}
+
+/** Resolve the Codex Responses endpoint from the provider baseUrl. */
+function resolveCodexUrl(baseUrl: string): string {
+  const normalized = baseUrl.replace(/\/+$/, "");
+  if (normalized.endsWith("/codex/responses")) return normalized;
+  if (normalized.endsWith("/codex")) return `${normalized}/responses`;
+  return `${normalized}/codex/responses`;
+}
+
+/**
+ * Call the OpenAI Codex backend (Responses API over SSE). The backend
+ * requires `store: false`, `stream: true`, no temperature / max tokens,
+ * plus the chatgpt-account-id + originator headers. The final text is
+ * assembled from `response.output_text.delta` SSE events.
+ */
+async function callCodexVisionModel(
+  visionModel: Model<Api>,
+  apiKey: string | undefined,
+  providerHeaders: Record<string, string> | undefined,
+  image: LoadedImage,
+  prompt: string,
+  signal: AbortSignal | undefined,
+  systemPrompt?: string,
+): Promise<string> {
+  const accountId = extractCodexAccountId(apiKey);
+  const headers: Record<string, string> = {
+    "Content-Type": "application/json",
+    Accept: "text/event-stream",
+    "OpenAI-Beta": "responses=experimental",
+    originator: "pi",
+  };
+  if (apiKey) headers.Authorization = `Bearer ${apiKey}`;
+  if (accountId) headers["chatgpt-account-id"] = accountId;
+  if (providerHeaders) Object.assign(headers, providerHeaders);
+
+  const body: Record<string, unknown> = {
+    model: visionModel.id,
+    store: false,
+    stream: true,
+    instructions:
+      systemPrompt && systemPrompt.length > 0
+        ? systemPrompt
+        : "You are a helpful assistant.",
+    input: [
+      {
+        role: "user",
+        content: [
+          {
+            type: "input_image",
+            image_url: `data:${image.mimeType};base64,${image.data}`,
+          },
+          { type: "input_text", text: prompt },
+        ],
+      },
+    ],
+    text: { verbosity: "low" },
+  };
+
+  const response = await fetch(resolveCodexUrl(visionModel.baseUrl), {
+    method: "POST",
+    headers,
+    body: JSON.stringify(body),
+    signal,
+  });
+
+  if (!response.ok) {
+    const errBody = await response.text().catch(() => "");
+    throw new Error(
+      `Vision model returned ${response.status}: ${errBody.slice(0, 500)}`,
+    );
+  }
+
+  if (!response.body) {
+    throw new Error("Vision model returned no body");
+  }
+  const reader = response.body.getReader();
+  const decoder = new TextDecoder();
+  let buffer = "";
+  const deltas: string[] = [];
+  try {
+    while (true) {
+      const { done, value } = await reader.read();
+      if (done) break;
+      buffer += decoder.decode(value, { stream: true });
+      let boundary: number;
+      while ((boundary = buffer.indexOf("\n\n")) !== -1) {
+        const event = buffer.slice(0, boundary);
+        buffer = buffer.slice(boundary + 2);
+        for (const line of event.split("\n")) {
+          if (!line.startsWith("data: ")) continue;
+          let data: unknown;
+          try {
+            data = JSON.parse(line.slice(6));
+          } catch {
+            continue;
+          }
+          if (
+            data &&
+            typeof data === "object" &&
+            (data as { type?: unknown }).type === "response.output_text.delta"
+          ) {
+            const delta = (data as { delta?: unknown }).delta;
+            if (typeof delta === "string") deltas.push(delta);
+          }
+        }
+      }
+    }
+  } finally {
+    reader.releaseLock();
+  }
+  const text = deltas.join("");
+  if (!text) {
+    throw new Error("Vision model returned no content in the response");
+  }
+  return text;
+}
+
 /**
  * Call the vision model's OpenAI-compat chat/completions endpoint with the
  * image as a data URL + the user's prompt (and an optional system prompt).
@@ -132,6 +276,10 @@
   reasoning: ReasoningLevel,
   systemPrompt?: string,
 ): Promise<string> {
+  // Codex backend: Responses API over SSE, not chat/completions.
+  if (isCodexApi(visionModel)) {
+    return callCodexVisionModel(visionModel, apiKey, providerHeaders, image, prompt, signal, systemPrompt);
+  }
   const baseUrl = visionModel.baseUrl.replace(/\/+$/, "");

Verification

  • pnpm typecheck (tsc --noEmit): passes
  • pnpm test:run (353 tests): passes with the patch (T44/T48 concurrency tests are timing-flaky on a slow CPU-only machine — they flake equally on clean main)
  • End-to-end against the live Codex backend with a real image: describe_image on an openai-codex / gpt-5.6-terra config returns a correct description (HTTP 200, SSE parsed, ~5s)

Notes / open questions

  • The patch relies on apiKey being the OAuth access token (JWT), which is what getApiKeyAndHeaders returns for the openai-codex provider — same contract pi's own codex provider uses (options.apiKey).
  • reasoning effort is currently ignored on the Codex path (the backend also rejects temperature); if reasoning should be forwarded, the mapping would be reasoning: { effort } per pi's buildRequestBody.
  • If the maintainers prefer, I can open a PR with this change (plus a unit test that asserts the codex branch builds the /codex/responses URL and body shape).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions