Skip to content

feat: Improve OpenAPI compatibility for real-world specs - #25

Closed
kevin-krause wants to merge 1 commit into
mainfrom
kevin-krause/enhance-use-case-support
Closed

kevin-krause wants to merge 1 commit into
mainfrom
kevin-krause/enhance-use-case-support

Conversation

@kevin-krause

Copy link
Copy Markdown
Collaborator
sono-good-night-gif-by-digi-995

@kevin-krause

Copy link
Copy Markdown
Collaborator Author

still needs some clean up seems like some files were commited by mistake, I will push the fix bf we can move this from draft to pr

lucianfialho added a commit that referenced this pull request Aug 2, 2026
Hand-written specs sometimes park query defaults after a `#`. URL parses that as
a fragment, which never leaves the client, so the request went out without them
and the server fell back to its own defaults.

When the fragment reads like a query string, treat it as one. Parameters the
caller passed are applied afterwards and still win, and a fragment that is not
query-shaped is left alone.

Ported from #25 by @kevin-krause, whose URL-building extraction this repeats;
the rest of that PR landed separately in the operation and parameter name work.

Co-authored-by: Kevin Krause <kevin-krause@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T
@lucianfialho

Copy link
Copy Markdown
Owner

Revisei isso hoje, com atraso — desculpa por ter demorado.

O trabalho é bom e o comentário que você deixou dizendo que precisava de limpeza já estava desatualizado no mesmo dia: o branch é um commit único, sem arquivos soltos, build limpo e 129 testes passando. Não havia nada pendente da sua parte.

Reproduzi o bug principal e ele é pior do que o título sugere. Dois operationId que colapsam no mesmo nome não degradam um comando — o Commander lança durante a construção da árvore, então toda invocação morre, inclusive --help:

$ spec2cli --spec api.json items --help
Error: cannot add command 'search' as already have command 'search'

Rebaseei seu trabalho em main e abri como #31, com você como co-autor. Duas coisas mudaram:

  1. Deixei de fora a metade de refactor (url.ts, http.ts, dry-run.ts). Ela colide com Adaptive --agent-help, machine-readable failures, and the bench behind both #29, que faz a mesma extração. Seu fix do fragmento # na URL não estava coberto por nada, então foi portado junto do Adaptive --agent-help, machine-readable failures, and the bench behind both #29 — também com sua co-autoria.
  2. Enquanto o PR esperava, parte dele foi reimplementada em main de forma independente e mais fraca (resolveParameter sem escape de JSON Pointer). Survive specs whose operation and parameter names collide #31 melhora essa versão em vez de reintroduzir a sua.

Adicionei testes de regressão e verifiquei que falham em main antes de passarem — 5 pegam bugs reais, 4 são guardas do comportamento que já funcionava.

Fechando aqui em favor do #31. Obrigado pela contribuição, e desculpa de novo pelo tempo que levou.

lucianfialho added a commit that referenced this pull request Aug 3, 2026
A spec with two operationIds that simplify to the same command name did not
degrade — it killed the CLI. Commander rejects a duplicate while the command
tree is being built, before any command runs, so `searchItems` and `searchItem`
under one tag made every invocation fail, `--help` included:

  $ spec2cli --spec api.json items --help
  Error: cannot add command 'search' as already have command 'search'

Colliding command names are now disambiguated by HTTP method, then by a numeric
suffix for whatever still overlaps. Closes #23.

Two more shapes that real specs use and this one mishandled:

Parameters sharing a name across locations produced two flags of the same name,
which Commander rejects the same way. The existing guard only caught body fields
clashing with parameters, so `id` in the path plus `id` in the query still got
through. Collisions are now tracked by name as well as by location, first
declaration winning.

Parameter names are not constrained to what a flag can be called. `filter[name]`
became an unusable option; it is now offered as `--filter-name` while the
request still goes out under the spec's own name.

Also decodes JSON Pointer escapes when resolving refs. `~1` is the only way to
write `/` inside a pointer segment (RFC 6901), so any ref into `paths` — such as
`#/paths/~1pets/parameters/0` — silently missed and fell back to an unresolved
`$ref`. Ref resolution now also refuses to walk into a non-object rather than
yielding a partial.

Reworked from #25 by @kevin-krause, rebased onto main after the original went
stale. The refactor half of that PR is left out: main has since grown its own
parameter-ref resolution, and the request-building extraction is handled
separately. Its remaining URL fragment fix is ported alongside that work.



Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T

Co-authored-by: Kevin Krause <kevin-krause@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
lucianfialho added a commit that referenced this pull request Aug 3, 2026
…d both (#29)

* feat: serve --agent-help progressively

--agent-help dumped the entire operation catalog on every invocation. On a
1000-operation spec that is ~84k tokens injected into an agent's context before
any useful work — the same context-coupling problem MCP is criticised for, only
with a tighter serialization.

Serve it in levels instead. The root lists groups and their command counts; the
agent drills into a group, then into a single command, paying for detail just
where it decided to act. Walking root -> group -> command costs ~1.1k tokens on
that same spec, against ~84k for the full dump.

  --agent-help                  groups and counts
  --agent-help pets             command names in one group
  --agent-help pets create      full parameters for one command
  --agent-help --find "create"  search across every group
  --agent-help --all            previous behaviour, kept as an escape hatch

Because only one command is expanded at a time, the detail level can afford the
full description rather than its first line. That recovers parameter semantics
which live in prose and have no schema equivalent — mcp-nixos documents
`type: "package" or "option"` with no enum to back it, and the truncated form
left an agent guessing.

--find and --all cannot collide with an operation's own parameters: the
--agent-help path returns before command dispatch.

Docstrings arrive indented by their source's code block, which forced the YAML
writer into escaped double-quoted output — unreadable, and it inflated the very
token count being cut. normalizeBlock dedents against the body so it emits a
literal block.

agent-help.ts is split to stay under the file-size gate: spec-hints.ts holds the
spec-derived helpers (auth hint, name simplification, base URL) that command
building also needs, and the view builders move into their own modules.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T

* test: add context-cost bench against MCP

Internal validation of the positioning bet: does driving an API through a CLI
cost an agent fewer tokens than driving it through MCP, and by how much.

Both arms are fed the same tools. An MCP server's tools/list is converted
mechanically into an OpenAPI spec — each tool becomes POST /tools/{name} with
its inputSchema as the request body — and spec2cli reads that. Same tools, same
schemas, same descriptions; only the access layer differs. Servers come from
MCP-Bench so results stay comparable to arXiv:2602.15945, which used the same
harness. Tokens are counted with tiktoken, not estimated from bytes.

Measured across math-mcp (13 tools), metmuseum-mcp (3) and mcp-nixos (18):

  1.55x  format efficiency of --all over MCP tool definitions, lossless
  3.6x   end-to-end for a task touching one of three connected servers
  76.9x  progressive path against the flat dump, at 1000 operations

The README records two caveats rather than the headline alone. First, --all
truncates descriptions to their first line, so its apparent 2.38x includes
dropped information; 1.55x is the number that survives adjustment. Second, the
multiplier is a function of catalog size, not architecture — which is why the
widely repeated "35x" claim, quoted without an operation count, is
unfalsifiable.

Scope is tokens only. Turns and task success need the execution half of the
shim and real API spend, and are not measured here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T

* fix: mask credentials in --dry-run output

--dry-run printed the Authorization header in the clear, including inside the
copy-pasteable curl line. The executor did the opposite in verbose mode, masking
via maskToken. Same information, two paths, opposite policies — and --dry-run is
the one whose output people paste into bug reports and chat.

The divergence was structural: dry-run.ts had its own copy of the URL, header
and body construction, so the two could not help drifting. It had also silently
fallen behind in another way, dropping the spec's header params from the preview
entirely.

Extract one BuiltRequest that both read from. Credentials are masked by default
in every surface, including the curl; --reveal opts back in for when the literal
value is actually wanted, and says so when it has masked something.

Found while benchmarking: a token from ~/.config/spec2cli/apis.json was applied
to an unrelated ad-hoc --spec run and printed in full.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T

* feat: classify failures and honour --output json on error paths

--output json was only honoured on the happy path. A caller that asked for JSON
got JSON when the call worked and prose when it did not — precisely when it most
needed to parse the answer. Every failure also exited 1, so a missing flag, a
rate limit and an unreachable host were indistinguishable, and nothing could
decide whether retrying was worth anything.

Give each kind its own exit code and mark the three that are worth retrying
(429, 5xx, network). Under --output json the failure is emitted as data on
stdout: for a caller that asked for machine-readable output, a failure is still
a result.

  0 ok · 2 schema · 3 missing input · 4 auth · 5 not found
  6 client · 7 rate limited · 8 server · 9 network · 10 spec

Missing inputs now report every missing parameter at once, with the type, enum
and description needed to fill them in and call again. Commander fails on the
first one, so required params are declared optional to it and checked in the
action instead — a caller learns the whole story in one round trip rather than
discovering it a failed invocation at a time.

The spec-loading failure is covered too, which meant resolving the output mode
before the spec loads.

Program bootstrap moves to cli/program.ts to keep index.ts under the file-size
gate; the help text gains the exit-code table.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T

* fix: hand over the flat catalog unless the spec is large

Phase B measured what Phase A could not: what an agent actually spends getting
work done. Progressive disclosure lost.

Same agent, same model, same tasks, three arms — MCP, spec2cli with the whole
catalog upfront, spec2cli drilling down:

                MCP      --all    progressive
  math-chain    4t/134k  3t/110k  6t/218k
  nixos-lookup  3t/100k  2t/71k   5t/180k
  cross-server  5t/105k  7t/238k  8t/182k
  TOTAL        12t/341k 12t/419k 19t/580k

Discovery is not free. Each drill-down is a round trip, and every round trip
resends the whole conversation — ~31k tokens per turn here. So progressive
disclosure saves `flat_catalog - 1.1k` and spends ~73k per discovery step. On a
34-tool catalog it saves 0.8k to spend 73k.

The crossover lands near 880 operations in this harness. It is a property of the
agent, not of spec2cli: a leaner loop resends less and crosses over sooner. So
--agent-help now hands over the flat catalog below 400 operations and drills down
above it, with --all and --progressive to force either. The threshold sits below
the measured crossover deliberately, since most agents resend less than this one.

The 76.9x from Phase A is real but only reachable above the crossover. Below it,
progressive disclosure is a pessimisation — which is what shipping it as the
unconditional default would have been for almost every real spec.

Adds the execution half of the shim, so both arms reach identical tools through
identical implementations and only the access layer differs.

One caveat recorded in bench/README.md: this Claude Code build defers MCP tool
schemas, so the MCP arm gets lazy loading of its own and is not a clean stand-in
for flat injection. An earlier run that excluded ToolSearch from --allowedTools
left the agent able to see tool names but never load them; it fell back to the
open web and burned 14-17 turns failing. Those numbers looked like evidence and
were not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T

* fix: send query defaults written after a # in the spec path

Hand-written specs sometimes park query defaults after a `#`. URL parses that as
a fragment, which never leaves the client, so the request went out without them
and the server fell back to its own defaults.

When the fragment reads like a query string, treat it as one. Parameters the
caller passed are applied afterwards and still win, and a fragment that is not
query-shaped is left alone.

Ported from #25 by @kevin-krause, whose URL-building extraction this repeats;
the rest of that PR landed separately in the operation and parameter name work.

Co-authored-by: Kevin Krause <kevin-krause@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Kevin Krause <kevin-krause@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants