feat: Improve OpenAPI compatibility for real-world specs - #25
kevin-krause wants to merge 1 commit into
Conversation
kevin-krause
commented
May 21, 2026
|
still needs some clean up seems like some files were commited by mistake, I will push the fix bf we can move this from draft to pr |
Hand-written specs sometimes park query defaults after a `#`. URL parses that as a fragment, which never leaves the client, so the request went out without them and the server fell back to its own defaults. When the fragment reads like a query string, treat it as one. Parameters the caller passed are applied afterwards and still win, and a fragment that is not query-shaped is left alone. Ported from #25 by @kevin-krause, whose URL-building extraction this repeats; the rest of that PR landed separately in the operation and parameter name work. Co-authored-by: Kevin Krause <kevin-krause@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T
|
Revisei isso hoje, com atraso — desculpa por ter demorado. O trabalho é bom e o comentário que você deixou dizendo que precisava de limpeza já estava desatualizado no mesmo dia: o branch é um commit único, sem arquivos soltos, build limpo e 129 testes passando. Não havia nada pendente da sua parte. Reproduzi o bug principal e ele é pior do que o título sugere. Dois Rebaseei seu trabalho em
Adicionei testes de regressão e verifiquei que falham em Fechando aqui em favor do #31. Obrigado pela contribuição, e desculpa de novo pelo tempo que levou. |
A spec with two operationIds that simplify to the same command name did not degrade — it killed the CLI. Commander rejects a duplicate while the command tree is being built, before any command runs, so `searchItems` and `searchItem` under one tag made every invocation fail, `--help` included: $ spec2cli --spec api.json items --help Error: cannot add command 'search' as already have command 'search' Colliding command names are now disambiguated by HTTP method, then by a numeric suffix for whatever still overlaps. Closes #23. Two more shapes that real specs use and this one mishandled: Parameters sharing a name across locations produced two flags of the same name, which Commander rejects the same way. The existing guard only caught body fields clashing with parameters, so `id` in the path plus `id` in the query still got through. Collisions are now tracked by name as well as by location, first declaration winning. Parameter names are not constrained to what a flag can be called. `filter[name]` became an unusable option; it is now offered as `--filter-name` while the request still goes out under the spec's own name. Also decodes JSON Pointer escapes when resolving refs. `~1` is the only way to write `/` inside a pointer segment (RFC 6901), so any ref into `paths` — such as `#/paths/~1pets/parameters/0` — silently missed and fell back to an unresolved `$ref`. Ref resolution now also refuses to walk into a non-object rather than yielding a partial. Reworked from #25 by @kevin-krause, rebased onto main after the original went stale. The refactor half of that PR is left out: main has since grown its own parameter-ref resolution, and the request-building extraction is handled separately. Its remaining URL fragment fix is ported alongside that work. Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T Co-authored-by: Kevin Krause <kevin-krause@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…d both (#29) * feat: serve --agent-help progressively --agent-help dumped the entire operation catalog on every invocation. On a 1000-operation spec that is ~84k tokens injected into an agent's context before any useful work — the same context-coupling problem MCP is criticised for, only with a tighter serialization. Serve it in levels instead. The root lists groups and their command counts; the agent drills into a group, then into a single command, paying for detail just where it decided to act. Walking root -> group -> command costs ~1.1k tokens on that same spec, against ~84k for the full dump. --agent-help groups and counts --agent-help pets command names in one group --agent-help pets create full parameters for one command --agent-help --find "create" search across every group --agent-help --all previous behaviour, kept as an escape hatch Because only one command is expanded at a time, the detail level can afford the full description rather than its first line. That recovers parameter semantics which live in prose and have no schema equivalent — mcp-nixos documents `type: "package" or "option"` with no enum to back it, and the truncated form left an agent guessing. --find and --all cannot collide with an operation's own parameters: the --agent-help path returns before command dispatch. Docstrings arrive indented by their source's code block, which forced the YAML writer into escaped double-quoted output — unreadable, and it inflated the very token count being cut. normalizeBlock dedents against the body so it emits a literal block. agent-help.ts is split to stay under the file-size gate: spec-hints.ts holds the spec-derived helpers (auth hint, name simplification, base URL) that command building also needs, and the view builders move into their own modules. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T * test: add context-cost bench against MCP Internal validation of the positioning bet: does driving an API through a CLI cost an agent fewer tokens than driving it through MCP, and by how much. Both arms are fed the same tools. An MCP server's tools/list is converted mechanically into an OpenAPI spec — each tool becomes POST /tools/{name} with its inputSchema as the request body — and spec2cli reads that. Same tools, same schemas, same descriptions; only the access layer differs. Servers come from MCP-Bench so results stay comparable to arXiv:2602.15945, which used the same harness. Tokens are counted with tiktoken, not estimated from bytes. Measured across math-mcp (13 tools), metmuseum-mcp (3) and mcp-nixos (18): 1.55x format efficiency of --all over MCP tool definitions, lossless 3.6x end-to-end for a task touching one of three connected servers 76.9x progressive path against the flat dump, at 1000 operations The README records two caveats rather than the headline alone. First, --all truncates descriptions to their first line, so its apparent 2.38x includes dropped information; 1.55x is the number that survives adjustment. Second, the multiplier is a function of catalog size, not architecture — which is why the widely repeated "35x" claim, quoted without an operation count, is unfalsifiable. Scope is tokens only. Turns and task success need the execution half of the shim and real API spend, and are not measured here. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T * fix: mask credentials in --dry-run output --dry-run printed the Authorization header in the clear, including inside the copy-pasteable curl line. The executor did the opposite in verbose mode, masking via maskToken. Same information, two paths, opposite policies — and --dry-run is the one whose output people paste into bug reports and chat. The divergence was structural: dry-run.ts had its own copy of the URL, header and body construction, so the two could not help drifting. It had also silently fallen behind in another way, dropping the spec's header params from the preview entirely. Extract one BuiltRequest that both read from. Credentials are masked by default in every surface, including the curl; --reveal opts back in for when the literal value is actually wanted, and says so when it has masked something. Found while benchmarking: a token from ~/.config/spec2cli/apis.json was applied to an unrelated ad-hoc --spec run and printed in full. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T * feat: classify failures and honour --output json on error paths --output json was only honoured on the happy path. A caller that asked for JSON got JSON when the call worked and prose when it did not — precisely when it most needed to parse the answer. Every failure also exited 1, so a missing flag, a rate limit and an unreachable host were indistinguishable, and nothing could decide whether retrying was worth anything. Give each kind its own exit code and mark the three that are worth retrying (429, 5xx, network). Under --output json the failure is emitted as data on stdout: for a caller that asked for machine-readable output, a failure is still a result. 0 ok · 2 schema · 3 missing input · 4 auth · 5 not found 6 client · 7 rate limited · 8 server · 9 network · 10 spec Missing inputs now report every missing parameter at once, with the type, enum and description needed to fill them in and call again. Commander fails on the first one, so required params are declared optional to it and checked in the action instead — a caller learns the whole story in one round trip rather than discovering it a failed invocation at a time. The spec-loading failure is covered too, which meant resolving the output mode before the spec loads. Program bootstrap moves to cli/program.ts to keep index.ts under the file-size gate; the help text gains the exit-code table. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T * fix: hand over the flat catalog unless the spec is large Phase B measured what Phase A could not: what an agent actually spends getting work done. Progressive disclosure lost. Same agent, same model, same tasks, three arms — MCP, spec2cli with the whole catalog upfront, spec2cli drilling down: MCP --all progressive math-chain 4t/134k 3t/110k 6t/218k nixos-lookup 3t/100k 2t/71k 5t/180k cross-server 5t/105k 7t/238k 8t/182k TOTAL 12t/341k 12t/419k 19t/580k Discovery is not free. Each drill-down is a round trip, and every round trip resends the whole conversation — ~31k tokens per turn here. So progressive disclosure saves `flat_catalog - 1.1k` and spends ~73k per discovery step. On a 34-tool catalog it saves 0.8k to spend 73k. The crossover lands near 880 operations in this harness. It is a property of the agent, not of spec2cli: a leaner loop resends less and crosses over sooner. So --agent-help now hands over the flat catalog below 400 operations and drills down above it, with --all and --progressive to force either. The threshold sits below the measured crossover deliberately, since most agents resend less than this one. The 76.9x from Phase A is real but only reachable above the crossover. Below it, progressive disclosure is a pessimisation — which is what shipping it as the unconditional default would have been for almost every real spec. Adds the execution half of the shim, so both arms reach identical tools through identical implementations and only the access layer differs. One caveat recorded in bench/README.md: this Claude Code build defers MCP tool schemas, so the MCP arm gets lazy loading of its own and is not a clean stand-in for flat injection. An earlier run that excluded ToolSearch from --allowedTools left the agent able to see tool names but never load them; it fell back to the open web and burned 14-17 turns failing. Those numbers looked like evidence and were not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T * fix: send query defaults written after a # in the spec path Hand-written specs sometimes park query defaults after a `#`. URL parses that as a fragment, which never leaves the client, so the request went out without them and the server fell back to its own defaults. When the fragment reads like a query string, treat it as one. Parameters the caller passed are applied afterwards and still win, and a fragment that is not query-shaped is left alone. Ported from #25 by @kevin-krause, whose URL-building extraction this repeats; the rest of that PR landed separately in the operation and parameter name work. Co-authored-by: Kevin Krause <kevin-krause@users.noreply.github.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QpcUFZcRHNurdJ2diVg37T --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Kevin Krause <kevin-krause@users.noreply.github.com>