Skip to content

feat: --out for binary payloads — media-stream download on get, base64 decode on action - #31

Merged
igor-ctrl merged 5 commits into
mainfrom
feat/out-media-download
Aug 10, 2026
Merged

feat: --out for binary payloads — media-stream download on get, base64 decode on action#31
igor-ctrl merged 5 commits into
mainfrom
feat/out-media-download

Conversation

@igor-ctrl

Copy link
Copy Markdown
Owner

Summary

--out means "write the payload to this path instead of stdout". It lands on two commands, because BC hands back binary two different ways and until now neither reached a file.

  • bcli get <endpoint> <record-id> --out invoice.pdf — BC advertises a record's binary attachment as a <field>@odata.mediaReadLink annotation. The record is read, the link it advertises is streamed to disk.
  • bcli action <endpoint> <key> <name> --out doc.pdf — a bound action returns its payload as base64 in the response's value property. That gets decoded and written.

The use case is the ordinary one: a posted purchase invoice has a scanned PDF attached to it, someone needs the PDF, and the only way to get it was to read the record, find the media link by hand, and curl it with a token you had to mint yourself.

SDK side: AsyncBCClient.get_media / BCClient.get_media over a new BCTransport.download.

Closes the client half of the installer repo's issue #21.

Design decisions

The media GET honours disable_standard_api. get_media resolves the record through the same _resolve_url every other read uses, so the endpoint registry, an explicit --publisher/--group/--version, and the lockdown flag all apply to a media download exactly as they apply to bcli get. A profile that can't read an entity can't download its attachments either — a media stream is not a side door around the allowlist.

The media link is re-validated as a BC origin. Unlike every other URL bcli builds, this one comes out of a response body. A tampered or compromised endpoint returning content@odata.mediaReadLink: https://attacker.example/leak would otherwise receive the bearer token. download calls assert_bc_origin first, same guard as get_absolute on @odata.nextLink.

download is a sibling of _request, not a branch inside it. _request buffers the response and calls response.json() on success — wrong for every byte of a PDF. The new loop mirrors it exactly on retry policy, error mapping, and structured logging, and differs only in body handling. A GET is always retry-safe, so it needs no retry_safe gate.

Temp file + truncate-per-attempt. One .part file is opened before the retry loop and seek(0); truncate() runs at the top of every attempt, then os.replace moves it onto the destination only on success. Without the truncate, a retry following a half-streamed response appends the second body to the first half of the first — a corrupt file that nothing in the output ever mentions. This matters more since 0.7.0 stopped absorbing transient 5xx on writes: a visible retry has to be exactly right about bytes. No .part survives any failure path, and a failed download never leaves a truncated file where a complete one is expected.

Overwrite refusal is the default. An existing file is never replaced without --overwrite, and a missing parent directory is an error rather than an mkdir — matching bcli extract's _check_writeable precedent. Both checks run before any network round trip. On action that ordering is load-bearing: the POST can change BC, so an unwritable destination has to fail before the server is touched, not after.

--media auto-discovery, and when it refuses. With no --media, the record's @odata.mediaReadLink annotations are scanned; exactly one is downloaded. Zero or several raise an error naming what was found, because writing whichever came first would be a silent guess into the user's file. --media <field> names one explicitly and also composes the conventional <record-url>/<field> sub-resource for pages that serve a media property without annotating it. The field name is validated as a single URL path component, so it can't retarget the request.

--out vs --result-out on action. They write different things — decoded payload bytes vs the JSON envelope describing the invocation — and the help text says so, at length, because "write the output to a file" describes both and an agent picking the wrong one gets no signal that it did. They compose. A 204 No Content reports that there was no payload rather than writing a zero-byte file that looks identical to a successful download, and decoding happens before emit_success, so an undecodable response is recorded as failed.

get stays a read. No SafeContext, no confirm_write_or_exit, READ_PATHS unchanged. Nothing is written to BC.

Also promotes four helpers to the public API — bcli.odata.{extract_field_references, suggest_field, validate_filter_fields} and bcli.registry.import_from_metadata. Downstream saved-query catalog validation needs them and was reaching into _filter_fields / _importers directly, which we couldn't have changed without silently breaking it. Same reasoning as the bcli.queries extraction in 0.7.0.

Notes

  • The new flags auto-propagate into the MCP tools. bc_get and bc_action are generated from describe, so --out/--media/--overwrite appear there without any change to bcli_mcp — a describe test now pins that. Worth stating plainly: file writes land on the MCP host, i.e. the user's own machine, not on a server.
  • action --out has not been exercised live. There is no server-side action returning a base64 payload to point it at yet; its coverage is unit tests only. get --out was verified against a real tenant (read-only).
  • Downloaded files inherit the temp file's 0600 mode rather than the umask, on the grounds that a downloaded invoice is the account's data.
  • Version bumped to 0.8.0 with a CHANGELOG entry. src/bcli/_version.py untouched — it reads from package metadata.

Test plan

  • ruff check src/ tests/ clean
  • Full suite: 1142 passed, 5 skipped (up from 1092; +50 new)
  • Transport/SDK: happy path with request order + Accept: */* + auth header; zero/two media fields; explicit field with and without an annotation; ../evil field rejected before any HTTP; off-origin mediaReadLink rejected with the media URL never requested; retry after 503 and after a mid-stream ReadTimeout both yield the body exactly once with no .part left; 404 leaves no file and no litter; sync wrapper delegates
  • get CLI: forwarding of every argument; --media without --out; --out without a record id; all eight query flags and an explicit --format conflict with get_media never awaited; existing file refused then accepted with --overwrite; missing parent dir not created; --dry-run writes nothing and calls nothing
  • action CLI: base64 decoded with no base64 on stdout; 204 fails and marks the envelope failed; dict without value lists the keys; invalid base64 fails; existing file refused with post.await_count == 0; --out and --result-out both written
  • describe pins --out/--media on get
  • Live read-only smoke against a real tenant: downloaded a 36,933-byte PDF (%PDF magic bytes confirmed), re-run refused with exit 1, --overwrite succeeded, a record with no media property produced the clean "no media stream" error with no file created, --dry-run wrote nothing, conflicting flags rejected, bcli describe -f json shows the three new options
  • action --out against a live action returning base64 — deferred until such an action exists server-side

BC publishes a record's binary attachment as a
`<field>@odata.mediaReadLink` annotation. There was no way to fetch one:
every read path in the transport ends at `response.json()`, which is
wrong for every byte of a PDF.

`BCTransport.download` is a deliberate sibling of `_request` rather than
a branch inside it — same retry policy, same error mapping, same
structured log, but it streams the body instead of buffering and parsing
it. The URL comes out of a response body, so it goes through
`assert_bc_origin` first: a tampered mediaReadLink must not receive the
bearer token.

The retry semantics are the part worth reading twice. 0.7.0 stopped
retrying non-idempotent requests, which widened the window in which a
transient 5xx is visible rather than silently absorbed — so a download
that retries has to be exactly right about bytes. One temp file is opened
before the loop and truncated at the top of every attempt: a retry
following a half-streamed response would otherwise append the second body
to the first half of the first, producing a corrupt file that no error
ever mentioned. The temp file is moved onto the destination with
os.replace only after the stream completes, so a failure leaves neither a
truncated destination nor a stray .part.

`AsyncBCClient.get_media` reads the record through the ordinary
`_resolve_url`, which is what keeps registry routing and the
`disable_standard_api` lockdown applying to a media download exactly as
they apply to `get` — a media stream is not a side door around the
profile's allowlist. It reads without `$select`, since a projection drops
the annotations it is looking for, and refuses to guess when a record
advertises several media properties or none.
One flag, two transports. `bcli get <endpoint> <id> --out invoice.pdf`
streams the record's media property; `bcli action <endpoint> <key> <name>
--out doc.pdf` decodes the base64 payload a bound action returns. Both
answer the same question — "give me the bytes, not a printout" — so they
share the destination policy in `_out_path`: expanduser, refuse an
existing file without --overwrite, and treat a missing parent directory
as an error rather than an mkdir, since a typo'd path is likelier than a
wanted new tree.

On `get`, --out is single-record mode, so every flag that shapes a record
*list* is rejected rather than ignored — a caller who passed --filter
expected records, and quietly writing one record's bytes instead would be
the wrong kind of helpful. An explicitly-passed --format is a conflict
for the same reason; a format inherited from config is not, because it
was never about this invocation.

On `action`, the destination is vetted before the POST. An action can
change BC, and finding out afterwards that the payload has nowhere to go
would leave the mutation applied and the bytes lost. Decoding likewise
happens before the success envelope is emitted, so an undecodable
response is recorded as failed rather than succeeded-with-no-file. A 204
No Content says so instead of writing a zero-byte file that looks exactly
like a successful download. The help text spells out the difference from
--result-out, which writes the JSON envelope *about* the invocation:
both are "write output to a file", and an agent picking the wrong one
gets no signal that it did.

The describe test pins --out/--media on `get`, because bc_get is
generated from describe — an option missing there is an option an agent
cannot reach.
`extract_field_references`, `suggest_field` and `validate_filter_fields`
now come from `bcli.odata`; `import_from_metadata` from `bcli.registry`.

Downstream tooling that validates a saved-query catalog against an
endpoint's live fields needs all four, and was importing
`bcli.odata._filter_fields` and `bcli.registry._importers` to get them —
private modules whose signatures we could not have changed without
breaking those consumers with no warning. Making the surface explicit is
the same move as the `bcli.queries` extraction in 0.7.0.
The action verb had no entry in the command reference at all, so it gets
one rather than just a --out row — including the distinction from
--result-out, which is the mistake the flag most invites.

The querying guide documents the two-step shape a media download really
has (find the record, then fetch its stream), because --out needs a
record id and the error message alone can't teach that.
…o-replace commit)

Three hardenings from a pre-merge security review of the --out feature:

1. is_bc_origin (and the ETL inline copy) now require https. Accepting http
   for an allowlisted host would have attached the bearer token to a cleartext
   request if a tampered @odata.mediaReadLink/nextLink used http://. Tightens
   every caller of the origin guard, not just --out.

2. get_media percent-encodes an explicit --media field as a single path
   component. validate_record_key blocks raw separators but accepted percent
   escapes, so '..%2F..%2Fapi%2Fv2.0%2F...' could splice encoded traversal into
   the token-bearing fallback URL for a gateway that decodes %2F before routing.
   quote(safe="") turns %2F into %252F.

3. The --out no-overwrite promise is enforced at commit, not only at pre-flight.
   The existence check and the os.replace publish resolved the parent twice, so
   a parent-directory symlink swapped in between could clobber a same-named file
   without --overwrite. download() and atomic_write_bytes() gain an overwrite
   flag; when false they publish with os.link (no-replace), which fails if the
   destination appeared after the check. SDK callers keep overwrite=True; the
   CLI threads its real --overwrite flag through.

Regression tests cover each: http BC URL rejected before the token, %2F in
--media neutralized, and a parent-symlink swap between pre-flight and commit
refused for both the media download and the decoded action payload. Full suite
1151 passed.
@igor-ctrl
igor-ctrl merged commit 646de5e into main Aug 10, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant