feat: --out for binary payloads — media-stream download on get, base64 decode on action - #31
Merged
Merged
Conversation
BC publishes a record's binary attachment as a `<field>@odata.mediaReadLink` annotation. There was no way to fetch one: every read path in the transport ends at `response.json()`, which is wrong for every byte of a PDF. `BCTransport.download` is a deliberate sibling of `_request` rather than a branch inside it — same retry policy, same error mapping, same structured log, but it streams the body instead of buffering and parsing it. The URL comes out of a response body, so it goes through `assert_bc_origin` first: a tampered mediaReadLink must not receive the bearer token. The retry semantics are the part worth reading twice. 0.7.0 stopped retrying non-idempotent requests, which widened the window in which a transient 5xx is visible rather than silently absorbed — so a download that retries has to be exactly right about bytes. One temp file is opened before the loop and truncated at the top of every attempt: a retry following a half-streamed response would otherwise append the second body to the first half of the first, producing a corrupt file that no error ever mentioned. The temp file is moved onto the destination with os.replace only after the stream completes, so a failure leaves neither a truncated destination nor a stray .part. `AsyncBCClient.get_media` reads the record through the ordinary `_resolve_url`, which is what keeps registry routing and the `disable_standard_api` lockdown applying to a media download exactly as they apply to `get` — a media stream is not a side door around the profile's allowlist. It reads without `$select`, since a projection drops the annotations it is looking for, and refuses to guess when a record advertises several media properties or none.
One flag, two transports. `bcli get <endpoint> <id> --out invoice.pdf` streams the record's media property; `bcli action <endpoint> <key> <name> --out doc.pdf` decodes the base64 payload a bound action returns. Both answer the same question — "give me the bytes, not a printout" — so they share the destination policy in `_out_path`: expanduser, refuse an existing file without --overwrite, and treat a missing parent directory as an error rather than an mkdir, since a typo'd path is likelier than a wanted new tree. On `get`, --out is single-record mode, so every flag that shapes a record *list* is rejected rather than ignored — a caller who passed --filter expected records, and quietly writing one record's bytes instead would be the wrong kind of helpful. An explicitly-passed --format is a conflict for the same reason; a format inherited from config is not, because it was never about this invocation. On `action`, the destination is vetted before the POST. An action can change BC, and finding out afterwards that the payload has nowhere to go would leave the mutation applied and the bytes lost. Decoding likewise happens before the success envelope is emitted, so an undecodable response is recorded as failed rather than succeeded-with-no-file. A 204 No Content says so instead of writing a zero-byte file that looks exactly like a successful download. The help text spells out the difference from --result-out, which writes the JSON envelope *about* the invocation: both are "write output to a file", and an agent picking the wrong one gets no signal that it did. The describe test pins --out/--media on `get`, because bc_get is generated from describe — an option missing there is an option an agent cannot reach.
`extract_field_references`, `suggest_field` and `validate_filter_fields` now come from `bcli.odata`; `import_from_metadata` from `bcli.registry`. Downstream tooling that validates a saved-query catalog against an endpoint's live fields needs all four, and was importing `bcli.odata._filter_fields` and `bcli.registry._importers` to get them — private modules whose signatures we could not have changed without breaking those consumers with no warning. Making the surface explicit is the same move as the `bcli.queries` extraction in 0.7.0.
The action verb had no entry in the command reference at all, so it gets one rather than just a --out row — including the distinction from --result-out, which is the mistake the flag most invites. The querying guide documents the two-step shape a media download really has (find the record, then fetch its stream), because --out needs a record id and the error message alone can't teach that.
…o-replace commit) Three hardenings from a pre-merge security review of the --out feature: 1. is_bc_origin (and the ETL inline copy) now require https. Accepting http for an allowlisted host would have attached the bearer token to a cleartext request if a tampered @odata.mediaReadLink/nextLink used http://. Tightens every caller of the origin guard, not just --out. 2. get_media percent-encodes an explicit --media field as a single path component. validate_record_key blocks raw separators but accepted percent escapes, so '..%2F..%2Fapi%2Fv2.0%2F...' could splice encoded traversal into the token-bearing fallback URL for a gateway that decodes %2F before routing. quote(safe="") turns %2F into %252F. 3. The --out no-overwrite promise is enforced at commit, not only at pre-flight. The existence check and the os.replace publish resolved the parent twice, so a parent-directory symlink swapped in between could clobber a same-named file without --overwrite. download() and atomic_write_bytes() gain an overwrite flag; when false they publish with os.link (no-replace), which fails if the destination appeared after the check. SDK callers keep overwrite=True; the CLI threads its real --overwrite flag through. Regression tests cover each: http BC URL rejected before the token, %2F in --media neutralized, and a parent-symlink swap between pre-flight and commit refused for both the media download and the decoded action payload. Full suite 1151 passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--outmeans "write the payload to this path instead of stdout". It lands on two commands, because BC hands back binary two different ways and until now neither reached a file.bcli get <endpoint> <record-id> --out invoice.pdf— BC advertises a record's binary attachment as a<field>@odata.mediaReadLinkannotation. The record is read, the link it advertises is streamed to disk.bcli action <endpoint> <key> <name> --out doc.pdf— a bound action returns its payload as base64 in the response'svalueproperty. That gets decoded and written.The use case is the ordinary one: a posted purchase invoice has a scanned PDF attached to it, someone needs the PDF, and the only way to get it was to read the record, find the media link by hand, and curl it with a token you had to mint yourself.
SDK side:
AsyncBCClient.get_media/BCClient.get_mediaover a newBCTransport.download.Closes the client half of the installer repo's issue #21.
Design decisions
The media GET honours
disable_standard_api.get_mediaresolves the record through the same_resolve_urlevery other read uses, so the endpoint registry, an explicit--publisher/--group/--version, and the lockdown flag all apply to a media download exactly as they apply tobcli get. A profile that can't read an entity can't download its attachments either — a media stream is not a side door around the allowlist.The media link is re-validated as a BC origin. Unlike every other URL bcli builds, this one comes out of a response body. A tampered or compromised endpoint returning
content@odata.mediaReadLink: https://attacker.example/leakwould otherwise receive the bearer token.downloadcallsassert_bc_originfirst, same guard asget_absoluteon@odata.nextLink.downloadis a sibling of_request, not a branch inside it._requestbuffers the response and callsresponse.json()on success — wrong for every byte of a PDF. The new loop mirrors it exactly on retry policy, error mapping, and structured logging, and differs only in body handling. A GET is always retry-safe, so it needs noretry_safegate.Temp file + truncate-per-attempt. One
.partfile is opened before the retry loop andseek(0); truncate()runs at the top of every attempt, thenos.replacemoves it onto the destination only on success. Without the truncate, a retry following a half-streamed response appends the second body to the first half of the first — a corrupt file that nothing in the output ever mentions. This matters more since 0.7.0 stopped absorbing transient 5xx on writes: a visible retry has to be exactly right about bytes. No.partsurvives any failure path, and a failed download never leaves a truncated file where a complete one is expected.Overwrite refusal is the default. An existing file is never replaced without
--overwrite, and a missing parent directory is an error rather than anmkdir— matchingbcli extract's_check_writeableprecedent. Both checks run before any network round trip. Onactionthat ordering is load-bearing: the POST can change BC, so an unwritable destination has to fail before the server is touched, not after.--mediaauto-discovery, and when it refuses. With no--media, the record's@odata.mediaReadLinkannotations are scanned; exactly one is downloaded. Zero or several raise an error naming what was found, because writing whichever came first would be a silent guess into the user's file.--media <field>names one explicitly and also composes the conventional<record-url>/<field>sub-resource for pages that serve a media property without annotating it. The field name is validated as a single URL path component, so it can't retarget the request.--outvs--result-outonaction. They write different things — decoded payload bytes vs the JSON envelope describing the invocation — and the help text says so, at length, because "write the output to a file" describes both and an agent picking the wrong one gets no signal that it did. They compose. A 204 No Content reports that there was no payload rather than writing a zero-byte file that looks identical to a successful download, and decoding happens beforeemit_success, so an undecodable response is recorded as failed.getstays a read. NoSafeContext, noconfirm_write_or_exit,READ_PATHSunchanged. Nothing is written to BC.Also promotes four helpers to the public API —
bcli.odata.{extract_field_references, suggest_field, validate_filter_fields}andbcli.registry.import_from_metadata. Downstream saved-query catalog validation needs them and was reaching into_filter_fields/_importersdirectly, which we couldn't have changed without silently breaking it. Same reasoning as thebcli.queriesextraction in 0.7.0.Notes
bc_getandbc_actionare generated fromdescribe, so--out/--media/--overwriteappear there without any change tobcli_mcp— a describe test now pins that. Worth stating plainly: file writes land on the MCP host, i.e. the user's own machine, not on a server.action --outhas not been exercised live. There is no server-side action returning a base64 payload to point it at yet; its coverage is unit tests only.get --outwas verified against a real tenant (read-only).0600mode rather than the umask, on the grounds that a downloaded invoice is the account's data.src/bcli/_version.pyuntouched — it reads from package metadata.Test plan
ruff check src/ tests/cleanAccept: */*+ auth header; zero/two media fields; explicit field with and without an annotation;../evilfield rejected before any HTTP; off-origin mediaReadLink rejected with the media URL never requested; retry after 503 and after a mid-streamReadTimeoutboth yield the body exactly once with no.partleft; 404 leaves no file and no litter; sync wrapper delegatesgetCLI: forwarding of every argument;--mediawithout--out;--outwithout a record id; all eight query flags and an explicit--formatconflict withget_medianever awaited; existing file refused then accepted with--overwrite; missing parent dir not created;--dry-runwrites nothing and calls nothingactionCLI: base64 decoded with no base64 on stdout; 204 fails and marks the envelope failed; dict withoutvaluelists the keys; invalid base64 fails; existing file refused withpost.await_count == 0;--outand--result-outboth writtendescribepins--out/--mediaonget%PDFmagic bytes confirmed), re-run refused with exit 1,--overwritesucceeded, a record with no media property produced the clean "no media stream" error with no file created,--dry-runwrote nothing, conflicting flags rejected,bcli describe -f jsonshows the three new optionsaction --outagainst a live action returning base64 — deferred until such an action exists server-side