Skip to content

Let consumers pass their own tools to the OpenAI-compatible facade - #176

Merged
imaustink merged 8 commits into
mainfrom
feat/caller-supplied-tools
Aug 1, 2026
Merged

Let consumers pass their own tools to the OpenAI-compatible facade#176
imaustink merged 8 commits into
mainfrom
feat/caller-supplied-tools

Conversation

@imaustink

@imaustink imaustink commented Jul 31, 2026

Copy link
Copy Markdown
Owner

What

Adds a third level of tool calling: a consumer can pass its own tools to
POST /v1/chat/completions (or /invoke) and get standard tool_calls back to
execute in its own client. Recorded as ADR 0035.

Before this, the field was silently ignored — a client that offered tools got
prose back with no way to tell "the agent chose not to call them" from "the agent
never saw them". Open WebUI's native tool calling, LibreChat and anything on the
OpenAI SDK all sit on this contract.

How

The caller executes; the turn suspends. runTool gains a fourth dispatch kind
(callerTool, alongside jobTemplate/localExec/agentRunTemplate) and it is
the only one that executes nothing — it hands the call back with
finish_reason: "tool_calls" and ends the turn. Resumption is read off the wire,
not from a session, because there is no server-side conversation store:
buildAgentRequest now lifts assistant.tool_calls + role: "tool" pairs into
structured history.

That closes two silent losses. A role: "tool" message used to be dropped
outright (the history fold keeps only user/assistant), and an assistant message
carrying only tool_calls has content: null, which the fold skips — so without
this the planner would re-issue the same call forever. Seeding actionHistory is
also what keeps MAX_TOOL_STEPS bounding a resumed loop rather than resetting it
each round trip.

Not degrading the two existing levels is most of the work:

Concern Approach
Catalog contamination Own Qdrant collection. A caller's definitions never enter another caller's candidate set, selectFallbackTool's catalog query, or a sub-agent's toolRefs resolution.
Context pollution Pruned to top-K before the planner sees anything — the same discipline ADR 0008 applies to the catalog.
Latency Content-hash keys make the collection an embedding cache; since a client resends the same array every turn, steady-state cost is zero. Below top-K the index is skipped entirely — nothing to prune, so the common case costs what it did before.
Trust Caller text is untrusted (one level below a Tool CR description, two below Skill markdown) and rendered in a distinctly-labelled block.
Predictability Skill.spec.allowCallerTools (*bool, nil ⇒ allowed) lets a sensitive authored skill refuse them.

No RBAC applies, and that's argued rather than assumed. Authorization is
vacuous for a function the caller both supplies and runs itself, in its own
process, under its own credentials — the orchestrator gains no capability. Isolation
comes from has_id instead: every search is restricted to hashes taken from the
request being served, so it cannot surface anything the caller didn't just supply.
This is the design's least conventional decision and §2 of the ADR defends it.

Open WebUI's housekeeping completions still short-circuit before caller tools are
parsed, so a title-generation request carrying a tool array can never make the
client execute a real function as a side effect of rendering a chat title.

Four e2e bugs found and fixed along the way

All four are pre-existing and test-side — the feature code is unchanged since
this PR opened. They're here because the suite could not report the truth without
them, and each had to be fixed to see the next.

1. The harness could hang forever. waitFor promised "poll until timeoutMs"
but awaited probe() unbounded, making its own timeout unreachable if a probe
never settled. With un-timed fetch over a kubectl port-forward, one dropped
forward wedged an entire run: observed at 0% CPU for eight minutes, no kubectl
children, no sockets, no output, inside a test whose assertion was already
satisfied — then it burned a retry doing it again. Probes are now bounded, hung
attempts counted separately from failures, and fetchThrough reports whether the
forward died mid-request.

2. A test asserted behaviour ADR 0033 removed. The rollout spec waited 5
minutes for the run to reach Succeeded. But an agent now holds its reply until
acked, giving up only after REPLY_ACK_TIMEOUT_MS = 10 minutes — and that spec
rolls the orchestrator and never re-triggers, so nothing acks. It could only ever
fail; its own comment admitted it predated the change. It now asserts the run
survived the roll, which is what made the old fabricated error a lie rather than
a report. The next spec already owns "it completes once acked".

3. agentRunsSince hid runs whose timestamp truncated below the trigger. This
was the real cause of the intermittent, different-test-each-run failures.
Kubernetes stamps creationTimestamp to the second and truncates; since is a
host-side new Date() in milliseconds. A run created at …05.800 stamps to
…05Z → parses as …05.000 → a since of …05.200 excludes it permanently.

The proof: the spec reported timed out after 420000ms waiting for an AgentRun to be created (204 attempts) — 204 clean polls, zero hung, no probe errors — while
fake-github had recorded the gateway posting a full stub-agent-reply for that
very issue, mid-timeout
. The run was created, ran, replied, and its answer
reached the issue while the test insisted nothing had launched. Note fixing (1) is
what made this findable: 204 attempts, 0 hung is what said the harness is fine,
the filter is lying
.

The cutoff is now widened by the granularity plus a 2s clock-skew allowance
(minikube's VM clock drifts against the host). 2s is deliberately far below the
≥20s spacing between triggers, so it cannot do the worse thing — match a previous
turn's run and pass an assertion against the wrong one. A regression test pins that
boundary.

4. The rollout-recovery spec had no headroom. It passed alone and failed under
full-suite load. Recovery must land inside the gateway's 90s poll budget, and a
re-attaching turn only sees the held reply on the agent's next re-offer — 10s in
production. If that lands late the gateway posts a failure and the answer never
arrives, so waiting longer in the test cannot help. The fixture turns the re-offer
down to 2s via AGENT_REPLY_ACK_RETRY_MS (a real agent-runtime knob), on that one
spec only.

Verification

Everything below was run against a real deployed minikube stack.

  • 569/569 orchestrator unit tests (78 new). Whole-workspace typecheck clean.
  • Go controller: make test green; CRD manifests + deepcopy regenerated with
    controller-gen, not hand-edited.
  • Full e2e suite green: 6 files, 55 passed, 1 skipped by design, exit 0, in
    745s (down from 3013s once the phantom timeouts were gone).
    • caller-tools.e2e.ts 18/18 — the three Qdrant filter shapes the unit tests
      can only mock, the tool_calls round trip through the real planner, and proof
      the catalogs are untouched.
    • chat-harness.e2e.ts 9/9, no cluster — one test feeds the orchestrator's
      own toolCallDeltaChunk output through the harness so wire-shape drift fails
      fast.
    • waitfor-guard.e2e.ts 7/7, no cluster.
    • resilience.e2e.ts 4 passed, 1 skipped.

Reviewer notes

  • caller_tools grows ~6 points per e2e run, reclaimed by the TTL sweep.
    Bounded, but real residue — unlike cleanupAgentRunsSince there's no active
    teardown.
  • Two e2e assertions depend on the real model (that the planner calls the tool;
    that the refused skill emits its no-tool branch). Made as deterministic as
    possible via trusted skill markdown and narrow skill descriptions, but
    model-dependent in a way the rest of the suite isn't.
  • Documented limits, in the app README's known gaps: tool_choice: "required"
    is a planner directive not a guarantee; /invoke can offer tools but not resume
    (no message array to carry role: "tool"); caller tools are unreachable from a
    sub-agent's loop (no channel back to the HTTP caller's client). Per-turn planner
    loops are capped as always, but a client can always resend, so per-conversation
    depth is unbounded — unchanged from before.

Review follow-up

  • Fixed the one correctness gap the review flagged (graph.ts, verbatim-repeat
    finish guard). It returned a bare { plannedAction: "finish" } while its two
    sibling finish branches carry ...lastHistoryResult(state). On a resumed
    caller-tool turn no runTool runs, so state.result is unset and the answer
    lives only in seeded actionHistory; when tool_choice: "required" (re-applied
    on the resend) pushed the planner to re-issue the byte-identical already-run
    call, the guard finished with result === undefined and the blocking facade
    rendered ```json\nundefined``` instead of the last tool's result. The
    guard now matches its siblings, with a resume-with-duplicate-call regression test
    that fails on the old code (expected undefined to be '58F and raining') and
    passes with the fix. This is the one product-code change since the PR opened; the
    four e2e fixes remain test-side.
  • Synced with main (merged, matching this repo's history).

🤖 Generated with Claude Code

https://claude.ai/code/session_01SjyBwazMbdQMzPbupZiiXc

imaustink and others added 4 commits July 31, 2026 06:19
Two levels of tool calling existed, both sourced from the in-cluster CRD
catalog: the orchestrator's own Skill-scoped planner loop (ADR 0008/0021) and a
sub-agent's internal loop (ADR 0028). The third level the OpenAI wire format
assumes -- a consumer passing its own `tools` and getting `tool_calls` back to
execute itself -- was silently ignored, so a client that offered tools got prose
with no signal its tools were never seen.

The caller executes; the turn suspends. `runTool` gains a fourth dispatch kind
(`callerTool`, alongside jobTemplate/localExec/agentRunTemplate) and it is the
only one that executes nothing: it hands the call back as `tool_calls` with
`finish_reason: "tool_calls"` and ends the turn. Resumption is read off the wire
rather than from a session, since there is no server-side conversation store:
`buildAgentRequest` now lifts `assistant.tool_calls` + `role: "tool"` pairs into
structured history. That closes two silent losses -- a `role: "tool"` message
used to be dropped outright, and an assistant message carrying only tool_calls
has `content: null`, which the history fold skips -- and seeding `actionHistory`
is also what keeps MAX_TOOL_STEPS bounding a resumed loop.

Keeping this from degrading the two existing levels is most of the work:

- Its own Qdrant collection, keyed by a content hash, so a caller's ephemeral
  definitions never enter another caller's candidate set, selectFallbackTool's
  catalog query, or a sub-agent's toolRefs resolution. The hash keying makes the
  collection an embedding cache: since a client resends the same array every
  turn, steady-state embedding cost is zero.
- Pruned to top-K before the planner sees anything, and the index is skipped
  ENTIRELY below that threshold -- with nothing to prune, a round trip would be
  pure added latency, so the common case costs exactly what it did before.
- Refusable per-skill via `Skill.spec.allowCallerTools` (*bool, nil => allowed,
  so Go's zero value cannot silently mean "refuse").

No RBAC applies, and that is argued rather than assumed: authorization is
vacuous for a function the caller both supplies and runs itself, under its own
credentials, in its own process. Isolation instead comes from `has_id` -- every
search is restricted to hashes taken from the request being served, so it cannot
surface a definition the caller did not just supply. Caller-provided text is
rendered to the planner in a distinctly-labelled untrusted block, one level below
a Tool CR description and two below Skill markdown.

Open WebUI's housekeeping completions still short-circuit before caller tools are
parsed, so a title-generation request carrying a tool array can never make the
client execute a real function as a side effect of rendering a chat title.

See docs/adr/0035 for the alternatives weighed (notably orchestrator-side
execution, rejected: no off-the-shelf client would populate it, and it would put
caller-controlled egress in the pod holding the k8s identity).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SjyBwazMbdQMzPbupZiiXc
`waitFor` promised "poll until timeoutMs" but awaited `probe()` unbounded, which
made its own timeout UNREACHABLE whenever a probe failed to settle: the loop
stopped, permanently, without ever re-checking the deadline.

That is reachable, not theoretical. Several probes `fetch` through a `kubectl
port-forward`, Node's fetch has no default timeout, and a forward dropped by a
busy apiserver leaves a socket nobody will ever answer. `fakeGithubRequests` --
polled by every comment-wait in the suite -- was one of them.

Watched live rather than inferred: a resilience run sat at 0% CPU for eight
minutes with no kubectl children, no open sockets and no output, wedged inside a
test whose assertion had already been satisfied. Only vitest's per-test timeout
would eventually kill it, reporting "the test timed out" -- which says nothing
about which hop stalled, and is exactly the diagnosis waitFor exists to provide.
It then burned a retry doing it again.

So:

- Every probe attempt is bounded, by whichever is sooner -- the per-probe ceiling
  or what remains of the overall budget. Hung attempts are counted and reported
  SEPARATELY from failures, because "146 attempts, 0 hung" and "146 attempts, all
  hung" call for completely different fixes. (That distinction immediately
  diagnosed the next failure underneath this one.)
- `fetchThrough` replaces bare `fetch` against cluster services: bounded, and it
  reports whether the port-forward died mid-request, so a dropped forward reads
  as a dropped forward rather than as a service that went quiet.
- `openPortForward` records its child's exit and fails loudly if it never becomes
  ready, instead of leaving a doomed forward behind.

The practical effect is that a dropped forward now fails one poll and the next
poll opens a fresh forward, instead of wedging the whole run.

`specs/waitfor-guard.e2e.ts` pins the guarantee -- including "a probe that never
settles must still time out" -- and needs no cluster, so a regression here fails
in milliseconds rather than after an hour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SjyBwazMbdQMzPbupZiiXc
…acade

apps/agent-orchestrator's unit tests cover the decisions. They cannot cover
either half of what actually breaks this feature in a cluster.

**Qdrant has to accept the queries.** Those tests mock the Qdrant client
outright, so they prove which method the code MEANT to call and nothing about
whether the filter DSL is valid. ADR 0035 added three hand-written filter shapes
-- `has_id` for the id-restricted search, a payload-only `setPayload` for cache
touches, a delete-by-filter `range` for the TTL sweep -- and a mock accepts all
three whether or not Qdrant would. `has_id` matters beyond correctness: it IS the
isolation boundary for a collection that carries no RBAC filter, so a mis-shaped
filter is both a 500 and a cross-caller leak. The `range` sweep is the only thing
bounding a collection Qdrant gives no native TTL. So the first block drives the
real store against the real Qdrant, in its own throwaway collection, with a
deterministic stand-in embedder -- the subject is the filter DSL, and real
embeddings would add cost and nondeterminism to assertions that never look at
similarity.

**The catalog has to stay clean.** "A caller's definitions go somewhere else and
never perturb tool/skill/agent retrieval" is a claim about which collection holds
which points, unobservable anywhere but a real Qdrant.

Plus the wire contract over the real facade and real planner: `tool_calls` with
schema-conforming arguments, resumption from a client-executed result, the
per-skill gate taking its refusal branch, `tool_choice: "none"`, and the
housekeeping short-circuit.

Two things worth knowing before editing it.

The planner decision is made deterministic on purpose. Whether the planner CALLS
a caller tool is a real OpenAI decision, and this suite's rule is to assert on
deterministic dispatch rather than model judgement. A skill's markdown is trusted
system-prompt content -- the strongest lever over that decision short of faking
the planner -- so the two fixture Skill CRs differ ONLY in `allowCallerTools`, one
instructed to call the caller's tool and the other given an explicit no-tool
branch. Asserting that branch, rather than merely the absence of a tool call,
is what separates the gate working from the model declining anyway.

And the tool definitions carry a per-run token, which cost a red build to learn.
The index is content-hash keyed and persists, so a fixed set of definitions is
indexed by the first run that ever sends them and is a pure cache hit forever
after -- an "indexing happened" assertion against fixed definitions passes exactly
once in the lifetime of a Qdrant volume, then fails, reporting the cache working
correctly as a product failure. Varying the description (part of the hash) keeps
the name stable for the skill markdown while making each run genuinely new, and
let the assertion tighten from "grew" to "grew by exactly the number sent" -- plus
a new one: an EDITED tool that keeps its name must add exactly one point, which a
name-keyed cache would fail.

`support/invoke.ts` applies the suite's per-entry-point rule to the programmatic
`/invoke`: both facades may OFFER tools, only chat can RESUME, and `/invoke`
renders a pending call differently. A documented asymmetry with nothing asserting
it is just a claim.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SjyBwazMbdQMzPbupZiiXc
"an orchestrator rollout mid-turn does not fabricate a timeout" waited 5 minutes
for the AgentRun to reach `Succeeded` after rolling the orchestrator. It could
never pass.

Since docs/adr/0033 an agent HOLDS its concluding message until someone acks it
(the protocol's `reply_ack`), re-offering every 10s and giving up only after
`REPLY_ACK_TIMEOUT_MS` -- 10 minutes (packages/agent-runtime/src/runtime.ts).
This test rolls the orchestrator and never re-triggers, so nothing acks: the new
pod has no reason to re-attach. The run therefore stays Running for ten minutes
by design, twice this test's budget and past vitest's per-test timeout. The
assertion predates that change, as its own comment admitted ("this held even
BEFORE the fix"), and ADR 0033 quietly made it unreachable.

It now asserts what is true and still meaningful: the run SURVIVED the roll --
same run, not Failed. That is precisely what made the old fabricated error a lie
rather than a report, since the run it claimed had timed out was alive and holding
an answer. The dropped half is not lost: the very next spec re-triggers the same
issue and asserts `Succeeded` BECAUSE the re-attach acked it, which is the
assertion this one deliberately withholds.

Also gives each trigger a unique issue number. `Date.now() % 100000` repeats every
100 seconds' worth of values; the issue number decides the session id, sessions
outlive a spec, and a turn landing on a session that still carries an
awaiting-reply anchor re-attaches to the old run instead of launching -- surfacing
as "timed out waiting for an AgentRun to be created", a real product behaviour
triggered by a fixture and indistinguishable from a regression. Not the cause of
anything observed here (exact congruence is unlikely); it removes the class of
doubt for free. Test 5's deliberate same-issue re-trigger is unaffected.

Found only after bounding waitFor's probes: "146 attempts, 0 hung" is what said
the harness was healthy and the run genuinely was not terminating.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SjyBwazMbdQMzPbupZiiXc
@imaustink

Copy link
Copy Markdown
Owner Author

Correcting the verification claim for resilience.e2e.ts

The PR body says the full-file resilience run was still in flight. It has now
finished, and it failed — so the claim there is optimistic and I'm correcting
it rather than leaving it to be discovered in review.

Tests  2 failed | 2 passed | 1 skipped (5)
× a narrating turn outlives the idle window many times over
× an orchestrator rollout mid-turn does not fabricate a timeout, and the run still succeeds
Error: e2e: timed out after 420000ms waiting for an AgentRun to be created (202 attempts)

Both fixes in this PR are still doing their job, and the output shows it:

  • 202 attempts with zero hung — the harness polled cleanly for the full 7
    minutes. Before commit e1a6095 this same condition produced an eight-minute
    silent hang with no diagnosis at all.
  • The rollout test failed at runCreated (before the roll), not at the
    terminal-phase assertion commit 64358ef corrected. That correction is
    independently verified: the test passes when run alone.

What's left is a third, separate problem, and it is not caused by this PR.
Webhook turns stop producing AgentRuns after the first couple in a file. Evidence:

  • Every failure is waiting for an AgentRun to be created, including the
    warm-up, which involves no disruption whatsoever.
  • The failing set changes between runs (first run: tests 1+2 passed, 4+5 failed;
    this run: 2+5 passed, 1+4 failed). Each failing test passes when run alone.
  • During a live failure I found: no authorization verdict logged for the stalled
    turn, no [claude-refresh] line (so the orchestrator never reached the
    credential gate), and nothing at all in the gateway log — while the webhook
    POST itself returned 2xx.
  • Ruled out with evidence: credentials (verdict: authorized on the turns that do
    run), gateway crash (0 restarts), node pressure (no pressure, requests at 51%),
    port collisions, and the Agent-catalog re-index race (Qdrant upserts are atomic
    per point).
  • happy-path.e2e.ts — the same webhook → relay → /invoke → AgentRun → comment
    chain — passes on this build, run immediately after these failures.

So the webhook chain works from cold and stops working after a few turns within a
file. That points at accumulated state between turns rather than at anything here.
I have not isolated it yet and am not going to guess in a commit message.

Reviewing the two e2e commits on their own merits is still worthwhile — they turn
this class of failure from an undiagnosable hang into the specific, quotable line
above. But resilience.e2e.ts is not green, and this PR should not be read as
claiming it is.

imaustink and others added 2 commits July 31, 2026 07:41
…trigger

This is the bug behind resilience.e2e.ts failing intermittently on a different
test each run, and it was never a product fault.

Kubernetes stamps `metadata.creationTimestamp` in RFC3339 with SECOND precision
and truncates. `since` is a host-side `new Date()` in milliseconds. Compared
directly, a run created at 14:07:05.800 is stamped `14:07:05Z`, parses back as
14:07:05.000, and a `since` of 14:07:05.200 excludes it -- permanently. The
trigger looks like it launched nothing, however long you poll.

The evidence, once the harness stopped hanging and started reporting: the test
failed with "timed out after 420000ms waiting for an AgentRun to be created (204
attempts)" -- 204 clean polls, zero hung, no probe errors -- while fake-github had
recorded the gateway posting a full `stub-agent-reply` for that very issue,
mid-timeout. The run was created, ran, replied, and its answer reached the issue
while the test insisted nothing had launched.

Which explains every symptom that misled the investigation for hours:

- Intermittent, and a different test each run: it is a race on where in the
  wall-clock second the trigger happens to land.
- Passes in isolation: different timing.
- `204 attempts, 0 hung`: the harness was working perfectly and kubectl was
  telling the truth. The FILTER was wrong.
- Unreproducible by hand: the probes written to chase it compared run COUNTS
  before and after, so they never touched the timestamp comparison at all.

The cutoff is now widened by the granularity plus a clock-skew allowance --
minikube's VM clock drifts against the host and jumps after the host sleeps, and
skew in the node-behind direction hides runs exactly the same way. Two seconds,
which is deliberately far below the >=20s spacing between triggers in any spec: a
wider window would risk the WORSE failure of matching a previous turn's run and
passing an assertion against the wrong one.

Three tests in specs/waitfor-guard.e2e.ts pin all of it -- the truncated-below
case, a lagging node clock, and the boundary that must NOT reach back to a prior
turn -- by stubbing kubectl on PATH, so they need no cluster.

resilience.e2e.ts now passes in full (4 passed, 1 skipped by design), in 451s
rather than 2317s, since the phantom 7-minute timeouts are gone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SjyBwazMbdQMzPbupZiiXc
…dget

"recovers the reply on a follow-up turn after a rollout" passed whenever this file
ran alone and failed inside the full suite -- the signature of a budget with no
slack rather than of a broken recovery.

The recovery has to complete inside integration-gateway's own poll budget (90s in
values-e2e.yaml). A re-attaching turn only sees the held reply on the agent's NEXT
re-offer, and that cadence is 10s in production. Under full-suite load the re-offer
landed after the gateway had given up, so the gateway posted a failure and the
answer never arrived at all -- which is why making the test wait longer cannot fix
it, and why the gateway's budget should stay short (it is deliberately tight so an
abandoned turn releases the relay quickly).

So the fixture turns the re-offer down to 2s, via `AGENT_REPLY_ACK_RETRY_MS` --
a real packages/agent-runtime knob, not a test-only affordance -- and only on this
one spec, so every other spec keeps the production cadence and this stays a local,
deliberate choice.

The whole e2e suite now passes: 6 files, 55 passed, 1 skipped by design, in 745s
(down from 3013s, since the phantom timeouts are gone).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SjyBwazMbdQMzPbupZiiXc
@imaustink

Copy link
Copy Markdown
Owner Author

Resolved — the whole e2e suite is green

Superseding my earlier correction. The root cause is found and fixed, and the PR
body's verification section now reflects the real result.

Test Files  6 passed (6)
     Tests  55 passed | 1 skipped (56)
  Duration  745.48s   (was 3013s)
EXIT=0

The intermittent resilience failures were never a product bug.
agentRunsSince compared a host-side new Date() (milliseconds) against
Kubernetes' creationTimestamp, which is RFC3339 second precision and
truncates. A run created at …05.800 stamps to …05Z, parses back as …05.000,
and a since of …05.200 excludes it permanently — the trigger looks like it
launched nothing, however long you poll.

The evidence is unambiguous: the spec reported timed out after 420000ms waiting for an AgentRun to be created (204 attempts) — 204 clean polls, zero hung, no
probe errors — while fake-github had recorded the gateway posting a full
stub-agent-reply for that very issue, mid-timeout. The run was created, ran,
replied, and its answer reached the issue while the test insisted nothing had
launched.

That explains every symptom that made this look like a regression: intermittent
(it's a race on where in the wall-clock second the trigger lands), a different test
each run, and passing in isolation.

Fixing the waitFor hang first is what made it findable — 204 attempts, 0 hung
is what said the harness is fine, the filter is lying, which is what sent me to
fake-github's record.

Two things I'd want a reviewer to be sceptical of, so I'll name them myself:

  • The 2s skew tolerance could mask a real "no run launched" bug if it were wide
    enough to match the previous turn's run — a false pass is worse than a false
    fail. Triggers in these specs are ≥20s apart, so 2s cannot reach one, and
    waitfor-guard.e2e.ts pins that boundary explicitly with a 10s case that must
    still return nothing.
  • Turning the stub's re-offer cadence down to 2s (commit 63a2e8f) is fixture
    tuning, not a product change, and is applied to one spec only so everything else
    keeps the production 10s. It buys headroom against the gateway's poll budget,
    which stays deliberately short on purpose.

Four separate e2e bugs, all test-side, each needed to see the next. The feature
code has not changed since this PR opened.

@k5s-bot

k5s-bot Bot commented Aug 1, 2026

Copy link
Copy Markdown

🤖 Starting work on this now. Watch live or take over the session here: https://claude.ai/code/session_01XATGT61M6b9EKe6NfCbmPy

@k5s-bot k5s-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: caller-supplied tools over the OpenAI facade

I checked out feat/caller-supplied-tools, read the ADR and the full diff, and verified the build/test claims locally. This is a strong, unusually well-documented PR. The design holds up: the two existing tool levels really are untouched (separate caller_tools collection, no writes to the catalog collections, no path into selectFallbackTool's query or sub-agent toolRefs), caller:-namespaced ids can't shadow catalog ids, and the isolation argument for skipping RBAC (has_id restricted to hashes taken from the request being served, resolved back to the request's own descriptor in qdrant-caller-tool-store.ts:131) is sound — a shared content-hash namespace leaks nothing because retrieval never ranges beyond what the caller just supplied.

Verified locally

  • Whole-workspace typecheck clean (tsc --noEmit, after building @controller-agent/messaging).
  • 568/568 orchestrator unit tests pass (46 files, 189s) — matches the PR's claim. I re-ran the changed areas in isolation too (caller-tools, chat-completions, action-planner, graph, server, skills — 282 tests green).
  • The Go zz_generated.deepcopy.go addition is the correct controller-gen shape for a *bool (nil-guard → new(bool) → deref copy); the CRD YAML matches skill_types.go. I did not finish go build/make test in this environment (module download exceeded the sandbox time budget), so the Go side is assessed by inspection only, not executed.
  • e2e specs were not run (no minikube cluster available here); the four test-harness fixes are assessed by reading. The agentRunsSince widening (granularity 1s + 2s skew, argued to stay well under the ≥20s trigger spacing so it can't match a previous run) and the waitFor/fetchThrough bounding are convincing, and the withPortForward signature change is backward-compatible (existing callers ignore the added 2nd callback arg).

One correctness gap (minor, but real and untested)

The duplicate-call guard in planAction was not updated to carry the seeded result, unlike the other two finish branches in the same node — see the inline comment on apps/agent-orchestrator/src/agent/graph.ts:1759. On a resumed caller-tool turn the result lives only in seeded actionHistory (no runTool ran this invocation to set state.result), so if the planner re-issues a byte-identical call the guard finishes with result undefined and the blocking facade renders renderResult(undefined) → a literal ```json\nundefined``` block instead of the last tool's result. This is exactly the path tool_choice: "required" nudges toward on a resend, since that directive is re-applied on the resumed turn (server.ts sets required whenever tools are present) and pushes the planner to call the one already-run tool again. Two-word fix; details inline.

Smaller notes (non-blocking)

  • QdrantCallerToolStore.prune returns 1/0 ("did it run"), not a delete count — Qdrant's delete-by-filter gives no count, so this is inherent; the doc comment says so and the index.ts caller ignores the value. Fine.
  • parse.ts accepts parameters when typeof === "object", which also admits a JSON array. A non-object schema is the caller's own problem (it only ends up in their planner prompt and their own client), so not worth rejecting, but worth being aware of.
  • Reviewer notes already flag the two honest residues: caller_tools grows a few points per run until the TTL sweep (no active teardown, unlike cleanupAgentRunsSince), and two e2e assertions are genuinely model-dependent. Both are called out in the PR; no objection.

Net: I'd fix the finish-without-result inconsistency before merge (or add a resume-with-duplicate-call test that would have caught it); everything else is solid.

Reviewed at commit 63a2e8f. Unit tests + typecheck run locally; Go build and e2e not executed in this environment (noted above).

Comment thread apps/agent-orchestrator/src/agent/graph.ts
@k5s-bot

k5s-bot Bot commented Aug 1, 2026

Copy link
Copy Markdown

Review posted successfully with both the summary and the inline comment.

Summary

I reviewed PR #176 ("Let consumers pass their own tools to the OpenAI-compatible facade") by checking out the branch, reading ADR 0035 and the full 48-file diff, and verifying the build/test claims locally.

What I verified in-environment:

  • Whole-workspace typecheck clean (tsc --noEmit, after building @controller-agent/messaging).
  • 568/568 orchestrator unit tests pass (46 files) — matches the PR's headline claim. Also re-ran the changed areas in isolation (282 tests green).
  • The Go zz_generated.deepcopy.go *bool deepcopy and the CRD YAML are correct by inspection (Go build/make test and the e2e suite were not run here — no minikube cluster, and Go module download exceeded the sandbox budget; I said so in the review rather than claiming otherwise).

Assessment: Strong, exceptionally well-documented PR. The core design holds up — separate caller_tools Qdrant collection, content-hash cache, has_id-restricted retrieval resolving back to the request's own descriptor (real isolation, so no RBAC needed), caller:-namespaced ids, skill gate, and the housekeeping short-circuit ordered before caller-tool parsing.

One concrete finding (minor, non-blocking) posted as an inline comment: the verbatim-repeat finish guard at graph.ts:1759 was not updated with ...lastHistoryResult(state) like its two sibling finish branc…


✅ Opened pull request: imaustink/agent-controller#176

@k5s-bot k5s-bot Bot removed the ai-review label Aug 1, 2026
@k5s-bot

k5s-bot Bot commented Aug 1, 2026

Copy link
Copy Markdown

🤖 Starting work on this now. Watch live or take over the session here: https://claude.ai/code/session_01MZkSbDnWnREdmCtDFXPPkU

imaustink and others added 2 commits August 1, 2026 01:51
On a resumed caller-tool turn (docs/adr/0035) no runTool runs this
invocation, so state.result is unset and the last tool's answer lives
only in the seeded actionHistory. The duplicate-call guard in planAction
finished with a bare `{ plannedAction: "finish" }`, unlike its two
sibling finish branches which carry `...lastHistoryResult(state)`. When
tool_choice: "required" (re-applied on the resend) pushes the planner to
re-issue the byte-identical already-run call, this guard fired and the
blocking facade rendered `renderResult(undefined)` -- a literal
```json\nundefined``` block instead of the last tool's result.

Make it match its siblings and add a resume-with-duplicate-call
regression test that pins final.result to the seeded result.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MZkSbDnWnREdmCtDFXPPkU
@k5s-bot

k5s-bot Bot commented Aug 1, 2026

Copy link
Copy Markdown

Picked back up from ai-triage — review feedback addressed

What I changed and why

Fixed the one correctness gap from @k5s-bot's review (inline at graph.ts:1746, targeting the sibling guard below it). The verbatim-repeat finish guard in planAction returned a bare { plannedAction: "finish" }, unlike its two sibling finish branches which carry ...lastHistoryResult(state).

  • On a resumed caller-tool turn (ADR 0035) no runTool runs this invocation, so state.result is unset and the last tool's answer lives only in the seeded actionHistory. With tool_choice: "required" re-applied on the resend, the planner is pushed to re-issue the byte-identical already-run call → this guard fired → the turn finished with result === undefined → the blocking facade rendered a literal ```json\nundefined``` block instead of the last tool's result.
  • The guard now returns { plannedAction: "finish", ...lastHistoryResult(state) } — exactly the suggested fix.
  • Added the requested resume-with-duplicate-call regression test (graph.test.ts: "finishes on the seeded result when a resumed planner re-issues the already-run call"). Planner mock returns the same call_tool already in seeded actionHistory, callerToolChoiceRequired: true to reproduce the required nudge, asserts final.result equals the seeded result. I verified it fails on the old code (expected undefined to be '58F and raining') and passes with the fix, and traced through it to confirm the guard is genuinely the branch that fires (state.result is undefined there on a resume).

Commit 139039f. The review thread is resolved.

Synced with main — merged (not rebased, matching this repo's history, e.g. 52ad80d), no conflicts. main did not touch graph.ts.

Updated the PR description: 569/569 unit tests (78 new), and a new Review follow-up section documenting the guard fix as the one product-code change since the PR opened (the four e2e fixes remain test-side).

Verification (re-run locally on the merged branch)

  • 569/569 orchestrator unit tests pass (46 files), whole-workspace typecheck clean (tsc --noEmit, after building @controller-agent/messaging).
  • Go untouched; the validate CRD check is running on the new commit as I post this — it's orthogonal to this TS-only change and passed on the prior commit.

Deliberately not done

  • prune returning 1/0 instead of a delete count — the reviewer called this inherent (Qdrant's delete-by-filter gives no count) and "Fine". No change.
  • parse.ts accepting a JSON array as parameters — the reviewer noted a non-object schema is the caller's own problem (it only reaches their planner prompt and their own client) and it's "not worth rejecting". Left as-is; agreed.
  • The two honest residues (caller_tools TTL-swept growth; two model-dependent e2e assertions) are already flagged in the PR body and drew no objection. Unchanged.
  • The e2e suite requires a minikube cluster not available in this environment, so it was not re-run here; the last full run recorded green in the PR body stands.

@k5s-bot

k5s-bot Bot commented Aug 1, 2026

Copy link
Copy Markdown

Everything is done and verified. Here's the summary.

Summary

I picked up PR #176 ("Let consumers pass their own tools to the OpenAI-compatible facade") from the ai-triage label and got it merge-ready.

Feedback found: One code review (from k5s-bot) with a single concrete, non-blocking correctness finding, plus three smaller notes the reviewer themselves marked "Fine"/"not worth rejecting". The validate CI check was already passing. The two author comments were a self-correction saga that already resolved to "e2e suite green".

What I changed:

  1. Fixed the correctness gap (graph.ts, commit 139039f) — the verbatim-repeat finish guard in planAction returned a bare { plannedAction: "finish" }, unlike its two sibling finish branches that carry ...lastHistoryResult(state). On a resumed caller-tool turn no runTool runs, so state.result is unset and the answer lives only in seeded actionHistory; when tool_choice: "required" nudges the planner to re-issue the already-run call, the guard used to finish with undefined and the facade rendered a literal ```json\nundefined``` block. Now it matches its siblings.
  2. Added the requested regression test — a resume-with-duplicate-call case. I confirmed via clean A/B that it fails on the old code (expected undefined to be '58F and raining') and passes with the fix, and traced execution to prove the guard is the branch that actually fires.
  3. **Replied t…

✅ Opened pull request: imaustink/agent-controller#176

@k5s-bot k5s-bot Bot removed the ai-triage label Aug 1, 2026
@imaustink
imaustink merged commit e5feb8e into main Aug 1, 2026
1 check passed
@imaustink
imaustink deleted the feat/caller-supplied-tools branch August 1, 2026 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant