feat: upstream SDK codegen and runtime foundations - #4087
Conversation
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
⏭️ Performance benchmarks were skippedPerf benchmarks (CodSpeed) are opt-in on pull requests — they no longer run on every push. They always run automatically after merge to To run them on this PR, do any of the following, then push a commit (or re-run CI):
|
📝 WalkthroughWalkthroughThis change adds ChangesTyped bridge and codegen flow
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Host
participant Bridge
participant BexEngine
participant VM
Host->>Bridge: send value with sparse type metadata
Bridge->>BexEngine: decode typed value and selected union arm
BexEngine->>VM: coerce contextual payload and execute call
VM->>BexEngine: return typed union or host result
BexEngine->>Bridge: encode canonical selected option index
Bridge->>Host: deliver converted value
Possibly related PRs
Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Binary size checks passed✅ 7 passed
Generated by |
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
baml_language/sdks/rust/bridge_rust/src/baml_value.rs (1)
219-223: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftPopulate collection type descriptors instead of always sending
None.These implementations know
T,K, andV, but every Rust collection decodes with fallback metadata. For example,Vec<String>becomes an array typed with the scalar fallback, while non-string maps default to string keys.Expose a wire-type descriptor through the value/key traits and emit
Some(...)for all three implementations. Add unit coverage for empty collections.As per coding guidelines, "Prefer writing Rust unit tests over integration tests where possible."
Also applies to: 260-271, 296-307
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@baml_language/sdks/rust/bridge_rust/src/baml_value.rs` around lines 219 - 223, The Rust collection implementations of __BamlValuePrivate for Vec, HashMap, and BTreeMap currently send item_type/key_type/value_type as None. Extend the relevant value/key traits with wire-type descriptor support, use T/K/V to emit Some(...) descriptors in all three implementations, and add Rust unit tests covering empty collections and their descriptors.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@baml_language/crates/bex_engine/src/conversion.rs`:
- Around line 919-928: Update the BexExternalValue::Union branch in
convert_external_to_vm_value_with_ty to validate metadata.selected_option
against the declared expected_ty union and verify that value inhabits the
selected arm before materializing it. Reuse the existing membership and payload
validation logic from coerce_arg_to_declared_type, rejecting malformed arms or
callable signatures before the recursive conversion.
- Around line 2239-2304: The union coercion logic does not recognize function
arms or callable host values. Extend runtime_ty_structurally_equal to compare
RuntimeTy::Function signatures recursively, and update value_matches_type to
accept the HostValue/FunctionRef representations for function types. Add a
regression test covering a host-selected callback in a union and preserve
existing matching behavior for other arms.
- Around line 2033-2067: Update both shared and schema-aware host-return
validators to mirror the matching rules used by runtime_ty_compatible: compare
float literals with float_literal_matches, recursively validate list and map
element/key/value descriptors even when containers are empty, and require the
returned media kind to match the declared type. Preserve exact literal checks
for other scalar values so mismatched literals, empty string arrays, and audio
returned for image declarations are rejected.
In `@baml_language/crates/bridge_ctypes/src/ty_decode.rs`:
- Around line 256-261: Update the BigintValue branch in the literal decoding
logic to reject decimal strings exceeding the existing value-level bigint length
limit before calling parse_bytes or formatting the input. Return a length-only
CtypesError for oversized values, while preserving the current invalid-format
error for bounded inputs.
In `@baml_language/crates/bridge_ctypes/src/value_decode.rs`:
- Around line 95-129: Update convert_union_variant and the corresponding
list/map decoding paths to validate each decoded payload recursively against its
wire-declared RuntimeTy, not only union membership. Ensure selected_type,
item_type, key_type, and value_type reject mismatched values before constructing
BexExternalValue containers or unions, while preserving valid nested values. Add
Rust unit tests covering mismatched union, list, and map payloads.
---
Outside diff comments:
In `@baml_language/sdks/rust/bridge_rust/src/baml_value.rs`:
- Around line 219-223: The Rust collection implementations of __BamlValuePrivate
for Vec, HashMap, and BTreeMap currently send item_type/key_type/value_type as
None. Extend the relevant value/key traits with wire-type descriptor support,
use T/K/V to emit Some(...) descriptors in all three implementations, and add
Rust unit tests covering empty collections and their descriptors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: e7abb70f-b98f-4744-aab8-627f95646797
⛔ Files ignored due to path filters (2)
baml_language/sdks/typescript/bridge_typescript/dist/proto/baml_cffi.d.tsis excluded by!**/dist/**baml_language/sdks/typescript/bridge_typescript/dist/proto/baml_cffi.jsis excluded by!**/dist/**
📒 Files selected for processing (31)
baml_language/crates/baml_project/src/client_codegen.rsbaml_language/crates/bex_engine/src/conversion.rsbaml_language/crates/bex_engine/src/lib.rsbaml_language/crates/bex_engine/tests/host_value_callable.rsbaml_language/crates/bex_external_types/src/host_return.rsbaml_language/crates/bex_external_types/src/lib.rsbaml_language/crates/bridge_cffi/src/ffi/host_value.rsbaml_language/crates/bridge_ctypes/README.mdbaml_language/crates/bridge_ctypes/src/error.rsbaml_language/crates/bridge_ctypes/src/ty_decode.rsbaml_language/crates/bridge_ctypes/src/ty_encode.rsbaml_language/crates/bridge_ctypes/src/value_decode.rsbaml_language/crates/bridge_ctypes/src/value_encode.rsbaml_language/crates/bridge_ctypes/types/baml_bridge/cffi/v1/baml_inbound.protobaml_language/crates/bridge_ctypes/types/baml_bridge/cffi/v1/baml_outbound.protobaml_language/crates/bridge_ctypes/types/baml_bridge/cffi/v1/baml_type.protobaml_language/crates/sys_native/src/host_dispatch.rsbaml_language/sdks/cpp/bridge_cpp/pb/baml_bridge/cffi/v1/baml_inbound.pb.ccbaml_language/sdks/cpp/bridge_cpp/pb/baml_bridge/cffi/v1/baml_inbound.pb.hbaml_language/sdks/cpp/bridge_cpp/pb/baml_bridge/cffi/v1/baml_outbound.pb.ccbaml_language/sdks/cpp/bridge_cpp/pb/baml_bridge/cffi/v1/baml_outbound.pb.hbaml_language/sdks/python/src/baml_bridge/cffi/v1/baml_inbound_pb2.pybaml_language/sdks/python/src/baml_bridge/cffi/v1/baml_inbound_pb2.pyibaml_language/sdks/python/src/baml_bridge/cffi/v1/baml_outbound_pb2.pybaml_language/sdks/python/src/baml_bridge/cffi/v1/baml_outbound_pb2.pyibaml_language/sdks/rust/bridge_rust/src/baml_value.rsbaml_language/sdks/rust/bridge_rust/src/wire/baml_bridge.cffi.v1.rsbaml_language/sdks/typescript/bridge_typescript/typescript_src/proto/baml_cffi.d.tsbaml_language/sdks/typescript/bridge_typescript/typescript_src/proto/baml_cffi.jstypescript2/pkg-proto/src/encode.tstypescript2/pkg-proto/src/test/encode-decode.test.ts
edf8aad to
de6eadb
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@baml_language/crates/bridge_ctypes/src/value_encode.rs`:
- Around line 366-387: The selected option lookup in selected_union_option_index
currently uses strict RuntimeTy equality; replace the members.iter().position
comparison with the existing structural union-member equality helper used by
selected_arm_equal/runtime_ty_structurally_equal. Preserve the current
not-member error and u32 index conversion behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: cb915954-7a05-482f-a70c-affbdaf2b612
⛔ Files ignored due to path filters (2)
baml_language/sdks/typescript/bridge_typescript/dist/proto/baml_cffi.d.tsis excluded by!**/dist/**baml_language/sdks/typescript/bridge_typescript/dist/proto/baml_cffi.jsis excluded by!**/dist/**
📒 Files selected for processing (32)
baml_language/crates/baml_project/src/client_codegen.rsbaml_language/crates/bex_engine/src/conversion.rsbaml_language/crates/bex_engine/src/lib.rsbaml_language/crates/bex_engine/tests/host_value_callable.rsbaml_language/crates/bex_external_types/src/bex_external_value.rsbaml_language/crates/bex_external_types/src/host_return.rsbaml_language/crates/bex_external_types/src/lib.rsbaml_language/crates/bridge_cffi/src/ffi/host_value.rsbaml_language/crates/bridge_ctypes/README.mdbaml_language/crates/bridge_ctypes/src/error.rsbaml_language/crates/bridge_ctypes/src/ty_decode.rsbaml_language/crates/bridge_ctypes/src/ty_encode.rsbaml_language/crates/bridge_ctypes/src/value_decode.rsbaml_language/crates/bridge_ctypes/src/value_encode.rsbaml_language/crates/bridge_ctypes/types/baml_bridge/cffi/v1/baml_inbound.protobaml_language/crates/bridge_ctypes/types/baml_bridge/cffi/v1/baml_outbound.protobaml_language/crates/bridge_ctypes/types/baml_bridge/cffi/v1/baml_type.protobaml_language/crates/sys_native/src/host_dispatch.rsbaml_language/sdks/cpp/bridge_cpp/pb/baml_bridge/cffi/v1/baml_inbound.pb.ccbaml_language/sdks/cpp/bridge_cpp/pb/baml_bridge/cffi/v1/baml_inbound.pb.hbaml_language/sdks/cpp/bridge_cpp/pb/baml_bridge/cffi/v1/baml_outbound.pb.ccbaml_language/sdks/cpp/bridge_cpp/pb/baml_bridge/cffi/v1/baml_outbound.pb.hbaml_language/sdks/python/src/baml_bridge/cffi/v1/baml_inbound_pb2.pybaml_language/sdks/python/src/baml_bridge/cffi/v1/baml_inbound_pb2.pyibaml_language/sdks/python/src/baml_bridge/cffi/v1/baml_outbound_pb2.pybaml_language/sdks/python/src/baml_bridge/cffi/v1/baml_outbound_pb2.pyibaml_language/sdks/rust/bridge_rust/src/baml_value.rsbaml_language/sdks/rust/bridge_rust/src/wire/baml_bridge.cffi.v1.rsbaml_language/sdks/typescript/bridge_typescript/typescript_src/proto/baml_cffi.d.tsbaml_language/sdks/typescript/bridge_typescript/typescript_src/proto/baml_cffi.jstypescript2/pkg-proto/src/encode.tstypescript2/pkg-proto/src/test/encode-decode.test.ts
🚧 Files skipped from review as they are similar to previous changes (20)
- baml_language/crates/bridge_ctypes/src/error.rs
- baml_language/crates/bridge_ctypes/types/baml_bridge/cffi/v1/baml_type.proto
- baml_language/crates/bridge_ctypes/types/baml_bridge/cffi/v1/baml_outbound.proto
- baml_language/sdks/python/src/baml_bridge/cffi/v1/baml_outbound_pb2.pyi
- baml_language/crates/bex_external_types/src/lib.rs
- baml_language/sdks/python/src/baml_bridge/cffi/v1/baml_inbound_pb2.py
- baml_language/crates/bex_engine/tests/host_value_callable.rs
- typescript2/pkg-proto/src/encode.ts
- baml_language/crates/bridge_ctypes/src/ty_encode.rs
- baml_language/crates/bridge_cffi/src/ffi/host_value.rs
- baml_language/sdks/rust/bridge_rust/src/baml_value.rs
- baml_language/crates/bridge_ctypes/src/ty_decode.rs
- baml_language/crates/sys_native/src/host_dispatch.rs
- baml_language/crates/bex_external_types/src/host_return.rs
- baml_language/crates/baml_project/src/client_codegen.rs
- baml_language/sdks/python/src/baml_bridge/cffi/v1/baml_outbound_pb2.py
- baml_language/crates/bex_engine/src/lib.rs
- baml_language/sdks/cpp/bridge_cpp/pb/baml_bridge/cffi/v1/baml_outbound.pb.h
- baml_language/crates/bex_engine/src/conversion.rs
- baml_language/sdks/cpp/bridge_cpp/pb/baml_bridge/cffi/v1/baml_outbound.pb.cc
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@baml_language/crates/bex_external_types/src/runtime_ty_identity.rs`:
- Around line 72-79: Update the T::Union branch of runtime_ty_structurally_equal
to require every member in both unions to have a structurally equal counterpart
in the other union, while retaining the length check. Ensure duplicate-member
cases produce the same result regardless of operand order.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 9741d196-0a3c-4943-b752-a4bffa9d463e
📒 Files selected for processing (7)
baml_language/crates/bex_engine/src/conversion.rsbaml_language/crates/bex_external_types/src/lib.rsbaml_language/crates/bex_external_types/src/runtime_ty_identity.rsbaml_language/crates/bex_project/src/lib.rsbaml_language/crates/bridge_ctypes/src/error.rsbaml_language/crates/bridge_ctypes/src/ty_decode.rsbaml_language/crates/bridge_ctypes/src/value_encode.rs
🚧 Files skipped from review as they are similar to previous changes (3)
- baml_language/crates/bridge_ctypes/src/ty_decode.rs
- baml_language/crates/bridge_ctypes/src/value_encode.rs
- baml_language/crates/bex_engine/src/conversion.rs
|
@coderabbitai resume |
✅ Action performedReviews resumed. |
…codegen-foundations
…codegen-foundations # Conflicts: # baml_language/crates/bex_engine/src/conversion.rs # baml_language/crates/bex_external_types/src/host_return.rs
…on (#4087) #4087 migrated the Java bridge to the node-level InboundValue.value_type sparse annotation (removing InboundClassValue.class_ty) and added the outbound union selected_option_index, but left the Java bridge-ref docs describing the pre-migration wire shape. Bring the two docs back to truth: ref-java-inbound-encoding.md - add value_type = 1 to the InboundValue proto block; document reserved field 1 on InboundClassValue - new "The value_type annotation" section: how Java (a Reject-policy typed producer) threads the contextual declared union — BamlTypedValue for top-level args, fieldDescs for class fields, item/value types for containers — and emits value_type for the three canonical cases (empty container arm, overlapping arm, literal-vs-primitive) - class identity / generic type_args now on value_type.class_ty (not the class_value payload); media kind on value_type.media.kind - union-arm and empty-container rows/notes updated for arm fidelity; deviation flag records typed-producer vs Python's dynamic SelectDefault ref-java-outbound-decoding.md - union-variant decoding: honors canonical selected_option_index (field 8) first, structural armMatchesValue only as fallback; deviation flag updated Docs-only; verified against ProtoWriter/ProtoReader/TypeRegistry/emit.rs and the engine (bridge_ctypes value_decode, bex_engine conversion). Full battery green: sdkgen_java 92 + clippy, baml_bridge gradle test+jar, sdk_test_java nextest 10/10 (+1 skip).
## Outcome This makes the generated Go SDK usable across the broad static/runtime surface we have completed so far, while keeping unsupported BAML shapes omitted instead of emitting uncompilable Go. Before, Go generation covered the initial free-function and basic-type slice. After this PR, generated callers can use packages/namespaces, classes, enums, optionals/defaults, closed and dynamic unions, aliases, generic classes/functions, media, canonical JSON, reflected `type` values, opaque `$rust_type` handles, host callbacks, methods/static methods, parse/build-request companions, cancellation, and BAML time values through one Go-native API. ```go result, err := baml_sdk.UserExtract(ctx, input) ``` Generated classes, closed unions, and callbacks remain statically typed where the configured union threshold permits it. Larger unions intentionally project to `any`, retain candidate documentation, and rely on BAML's runtime validation. ## User-visible behavior - Generates one Go package per BAML package, with namespace-qualified declarations inside it so BAML namespace cycles do not become Go import cycles. - Applies one canonical typed naming projection while retaining exact wire names for FFI serialization. - Generates compile-safe free functions, instance/static methods, functional options for defaulted arguments, and callback option structs for optional callback parameters. - Supports recursive classes, enums, lists/maps, nullability, literals, aliases, generic type arguments, media, JSON, reflected types, opaque Rust handles, and canonical union arm selection. - Preserves exact selected-node type information across the final shared inbound wire contract from BoundaryML#4087, including ambiguous empty containers and nullable bigint containers. - Omits unsupported functions without breaking the rest of the generated package. - Wires Go generation into the normal CLI flow and mirrors the shared SDK fixture harness. ## Audit map Review in this order: 1. `baml_language/sdks/go/sdkgen_go/src/types.rs` — canonical BAML-to-Go semantic projection and support filtering. 2. `baml_language/sdks/go/sdkgen_go/src/names.rs` — collision-safe name allocation and exact wire identity. 3. `baml_language/sdks/go/sdkgen_go/src/lib.rs` — package layout, declarations, codecs, functions, methods, callbacks, and descriptors. 4. `baml_language/sdks/go/baml_go/` — runtime encoding/decoding, ownership, callbacks, media, JSON, reflected types, and handles. 5. `baml_language/sdk_tests/crates/go/` plus Go customizable fixtures — generated compile/runtime coverage and Python-parity tests. 6. `baml_language/crates/baml_cli/` and `sdk_tests/harness_setup/src/go.rs` — CLI and test-harness integration. ## Validation Final post-BoundaryML#4087 sync validation: - `cargo test -p sdkgen_go` — 84/84 passed - `cargo test -p bex_external_types` — 21/21 passed; 3 doctests ignored - `cargo test -p bex_engine --test host_value_callable` — 23 passed; 1 documented compiler-gap test ignored - `cargo nextest run -p sdk_test_go --no-fail-fast` — 9/9 passed - Go runtime unit suite — passed - Repository pre-commit formatting, conflict checks, Clippy, and cargo-hawk dead-public analysis — passed ## Deliberate remaining scope - Streaming implementation is deferred. - Large unions intentionally use `any` above the configured typed-union threshold. - Structured BAML error values are currently surfaced to Go primarily through error text/trace behavior. - Packaging, publication, and coordinated nightly/canary releases are follow-up release work rather than generator semantics. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added Go-only `max_typed_union_arity` setting (default **3**) with `baml.toml` template support, validation, and generation behavior for typed unions. * Expanded the Go SDK with typed-union tuning support plus stronger generics, JSON-algebra utilities, media/time/reflection helpers, and improved host-callable integration. * **Bug Fixes** * Improved generated Go output installation to safely replace stale files while preserving user-created ones. * **Tests** * Added/extended extensive Go generator and SDK test suites, including additional gofmt-clean generation and broader runtime edge coverage. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…on (BoundaryML#4087) (BoundaryML#4153) ## Summary Parity investigation of the Java bridge against the new inbound `value_type` ABI annotation introduced in ceae8ea (BoundaryML#4087). **Finding: BoundaryML#4087 already implemented the Java encoder/decoder support and the empty-list arm-fidelity test contract end-to-end** — the one gap it left was the Java `bridge-ref` docs, which still describe the pre-migration wire shape. This PR is that docs parity fix. No runtime/codegen change was warranted. ### What BoundaryML#4087 already did (verified, no change needed) - **Inbound annotation.** Java threads the declared union as a *contextual type* and emits `value_type` for the selected arm exactly in the ambiguity cases: `BamlTypedValue(value, descriptor)` for top-level args (`emit.rs`), parallel `fieldDescs[]` for class fields, item/value types for containers (`ProtoWriter.encodeInboundValue`). Covers empty-container arms, overlapping arms, and literal-vs-primitive. Java registers the `Reject` ambiguity policy (a typed producer), so it annotates rather than leaning on Python's dynamic `SelectDefault`. - **Parity with Python.** Python sets `value_type` only for class identity and media kind; Java matches both (now on `value_type.class_ty` / `value_type.media.kind`) **and** additionally annotates selected union arms — the extra information the sparse channel exists to carry for a typed producer. - **Outbound.** `ProtoReader` honors the new canonical `selected_option_index` (field 8), falling back to structural `armMatchesValue` only when absent. - **Empty-list contract.** `TestUnions.test_round_trip_str_or_int_list` already asserts full arm fidelity (empty via `Arm0` stays `Arm0`, via `Arm1` stays `Arm1`). ### Docs brought back to truth - `ref-java-inbound-encoding.md`: add `value_type = 1` to the `InboundValue` proto; new "The `value_type` annotation" section (how Java threads the contextual union); class identity / generic args now on `value_type.class_ty`, media kind on `value_type.media.kind`; union-arm + empty-container rows/notes; typed-producer-vs-dynamic deviation flag. - `ref-java-outbound-decoding.md`: union-variant decoding documents the `selected_option_index`-first resolution. ## Test plan (full battery, cold builds) - `bridge_ctypes` value_decode: 18 passed - `bex_engine` conversion (union arm selection): 71 passed - `sdkgen_java`: 92 passed; `cargo clippy -p sdkgen_java`: clean - `baml_bridge` gradle `test jar`: BUILD SUCCESSFUL - `cargo nextest run -p sdk_test_java`: **10 passed, 1 skipped** — all four fixture suites (docstrings_etc, function_calls, llm_functions, type_shapes), each javac + junit Docs-only change; no fixture/test counts changed. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated Java inbound encoding guidance to match the latest wire/proto behavior, including node-level `value_type`, class identity and reified generic type args, and improved union/empty-container annotation rules. * Revised Java outbound union decoding guidance to use canonical `selected_option_index` when available, with clear structural fallback, null handling, and error behavior. * Adjusted Rust inbound decode mapping descriptions and updated related examples/test references to reflect the new contracts. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
What this PR actually changes
This PR changes the shared compiler/runtime/bridge contract and the generated/static SDK consumers that depend on it. Its central change is a node-local inbound type hint that complements the declared BAML context without turning the wire format into a recursively self-describing type tree:
Relative to
canary, field 1 was reserved and class identity lived inInboundClassValue.class_ty. This PR:InboundValue.value_typeannotation;InboundClassValue.class_tyand moves class identity/type arguments into the same node-level annotation used by every other payload kind;BamlTydecoding;Selfin SDK-facing argument and return types;This is deliberately not a recursively self-describing inbound format. The protocol has two producer modes:
string[]versus emptyint[]case end to end; Go's checked-in protobuf/runtime surfaces are regenerated, while a full generated-Go union implementation remains separate;value_typetherefore never means “the enclosing union.” It is either the exact type of the selected current node or, for an erased generated class instance, a nominal class hint that can be refined from context.The implementation snippets below are copied from the final code shape; comments such as
/* decode the payload oneof */and/* error */elide only repetitive decoding/error construction, not a validation step.Important base-to-head clarification
Earlier iterations of this branch considered
InboundTypedValue, container-localitem_type/key_type/value_type, and an outboundselected_type. None of those fields exist on the currentcanarybase, so they are not removals in this PR's review diff. They are rejected design alternatives, not shipped base-to-head surface area.Why payload shape is insufficient
Consider these signatures:
The payloads below are not enough by themselves to recover a selected type:
The two calls are not equivalent from the bridge's perspective. A generated Go/Rust/C++ union wrapper already knows which arm it contains, so its recursive inbound encoder must preserve
int[]versusstring[]when it projects that wrapper to the selected value. A raw Python/TypeScript[]contains no such fact, so the registered dynamic-language policy chooses a deterministic default instead of pretending an element type was inferred from nonexistent entries.The old inbound protocol had only one special type channel,
InboundClassValue.class_ty, so it could preserve a class FQN and generic arguments but could not annotate an arbitrary list, scalar, media wrapper, or nested child. The old inbound matcher also chose the first matching union member using shallow predicates. For example, it treated every array as matching every list arm and treated a string payload as matching a string literal without checking the literal's value. That meant declaration order—not the contents of["hello"]—could determine whetherint[] | string[]selectedint[]orstring[].The new contract separates three inputs:
value_type, when a typed producer has exact selected-node information or a generated instance has nominal information that payload shape cannot carry;Inbound contract in detail
1.
value_typebelongs to the current nodevalue_typedoes not repeat an enclosing union. It describes the current node's selected type.For a statically typed/generated producer,
(int[] | string[])[]is encoded with the selected child type on both list nodes:The non-empty second child deliberately carries
value_type: string[]even though its element currently makes the arm discoverable. Typed producers do not ask Rust to reverse-engineer a container type from its entries: they serialize the selected node type that their generated host value already knows. This also makes the encoding stable if the list later becomes empty.That design is now implemented by the typed producers in this diff. Rust
Vec<T>/maps and C++std::vector<T>/maps expose their exactBamlTy; generated union encoders project to the selected payload and attach the selected arm type without overwriting a more specific nested annotation. Java threads generated descriptors through an internal call carrier and annotates selected erased union arms. Swift's generic union family does the same throughBamlTypeDescriptor.Generated union wrappers are not themselves inbound BAML values. Their per-type encoder recursively projects them to the selected arm:
This rule applies at the root, in class fields, in map values, and in list entries. No inbound union envelope is required. The declared/contextual union remains in the BAML function or parent type; the wire value carries the selected payload and, when needed, that payload's exact selected-node type.
A raw dynamic-language value may remain sparse:
Because that empty child has no host-language element type, the registered dynamic-language policy chooses a deterministic default rather than inferring a list type from nonexistent entries.
At protobuf decode time, an annotation is represented inside the shared external value tree by a transient carrier:
This reuses the existing external union carrier internally, but
is_inbound_type_annotationdistinguishes “this node has a sparse type annotation” from “this value is already materialized as a declared union.” The enclosing declared union still comes from context.bridge_ctypes::inbound_to_externaldeliberately performs only syntax-level annotation validation and decoding:Assignability and payload compatibility require the declared program type and class/alias definitions, so those checks happen later in the engine.
2. Root unions and optionals are rejected
A root union names a set of choices; it does not identify which choice the producer selected. Generated SDK union wrappers intentionally disappear during recursive inbound encoding, leaving the selected value:
Encoding the wrapper type would add no selected-arm information:
An optional is represented as a union with
null, so it is rejected for the same reason:Nested unions remain valid beneath an exact outer type:
The same projection rule applies recursively to union-valued class fields, map values, and list entries because each generated host type participates in the inbound encoding protocol. This is why the protocol does not need an
InboundUnionValue: inbound reconstructs the declared union from context, while outbound needs a selected index so a generated SDK can reconstruct its host-language union wrapper.Direct union/optional descriptors are rejected in
bridge_ctypes. A named alias can only be recognized as union-shaped after the engine loads program-local aliases, so the engine also follows aliases defensively:Coverage:
bridge_ctypes::value_decode::root_union_value_type_is_rejectedbridge_ctypes::value_decode::root_optional_value_type_is_rejectedbridge_ctypes::value_decode::nested_union_value_type_is_allowedbex_engine::conversion::host_root_union_annotation_is_rejected_before_arm_matchingbex_engine::conversion::host_alias_to_root_union_annotation_is_rejected3. An annotation is checked against both context and payload
For a declared union, the engine first identifies the contextual member named by the annotation, accepting exact structural identity or ordinary assignability. It then coerces the payload using the annotation as its recursive type, verifies that the payload inhabits the annotation, and finally coerces/validates against the selected contextual member.
The important checks in
coerce_arg_to_declared_type_with_aliasesare:That produces these outcomes:
Coverage:
host_type_annotation_rejects_type_outside_contextual_unionhost_type_annotation_rejects_wrong_non_union_typehost_typed_literal_is_assignable_to_non_union_primitivehost_typed_literal_selects_broader_contextual_union_armhost_typed_overlapping_arm_uses_contextual_union4. Unannotated selection is structural first, then language-policy-specific
The new matcher recursively collects the structurally matching union members instead of accepting the first member with the same outer payload tag. The common cases are deterministic:
A bare
nullreceives one additional canonicalization rule. If it matches both an explicitnullarm and an alias whose body merely containsnull(for exampletype OptionalState = ResponseState?used asOptionalState?), the explicitnullarm wins. The payload already identifies the exact null leaf, so this is not erased host intent like an empty container. Coverage:unannotated_null_prefers_exact_null_over_nested_optional_alias, plus the generated Gotype_shapesround trip fornil.When more than one structurally distinct member remains, the process-global policy determines whether the engine rejects the missing type information or chooses a dynamic default:
This preserves specificity for overlapping literal/primitive and enum-variant/enum arms regardless of declaration order. Otherwise, the dynamic default is the first declared structurally matching member. For example:
The policy is set during the bridge's existing one-language-per-process registration:
The native mapping and registration are explicit:
bex_engine::inbound_configowns a single-assignmentOnceLock<InboundUnionAmbiguityPolicy>.bridge_cffi::register_bridgemaps its registeredBridgeLanguageto that policy, and the browser TypeScript/WASM start hook registersSelectDefault. Repeating the same policy is idempotent; conflicting registration is rejected. Direct engine users that do not register a bridge getReject, the safe default. This is process state rather thanFunctionCallContextstate because the native registry already forbids two different host bridges in one process.Defaulting happens only after recursive payload matching and structural duplicate removal, so this does not restore the old shallow “first outer tag wins” behavior. A future explicit dynamic arm-selector API can override this convention without changing the current wire contract.
Coverage:
unannotated_literal_vs_primitive_requires_value_typeunannotated_ambiguous_empty_child_requires_value_typedynamic_default_selects_first_matching_empty_container_armdynamic_default_prefers_exact_literal_over_broad_primitiveunannotated_structurally_duplicate_members_select_first_canonical_armbridge_language_selects_one_process_wide_inbound_policy5. Recursive context replaces recursive type serialization
Once the effective type is known, the coercer propagates it into the payload:
Maps do the same for each value. During VM materialization, class field templates are substituted with the instance's concrete generic arguments and each field is recursively coerced/materialized with that field type.
Consequences:
[ ["hello"] ]against(int[] | string[])[]selectsstring[]without annotations;string[][]/selectedstring[]arm serializes the empty child's knownvalue_type: string[]instead of deriving it from entries;[ [] ]against the same type selects the first matching child arm underSelectDefault; strict/unregistered typed contexts still reject the same unannotated ambiguity;Coverage:
sparse_nested_hint_selects_only_the_ambiguous_empty_childunannotated_nested_container_uses_payload_shapeunannotated_ambiguous_empty_child_requires_value_typeunannotated_recursive_alias_uses_contextual_payload_shapeempty_array_uses_its_declared_element_type_to_select_union_armempty_map_uses_its_declared_value_type_to_select_union_arm6. Literal descriptors remain literals
Before this PR,
proto_ty_to_runtime_tywidened aBamlTyLiteralto its primitive ("draft"becamestring,42becameint). That made a literalvalue_typeuseless.The decoder now constructs
RuntimeTy::Literalfor string, int, bigint, float, and bool. Decimal bigint type literals are bounded byMAX_BIGINT_DECIMAL_DIGITS/MAX_BIGINT_BITSbefore parsing; over-limit errors report only the input length, avoiding log amplification by hostile descriptors.Inbound literal matching checks the actual value. Float literals are parsed and compared by
f64::to_bits, preserving-0.0versus0.0and the parsed floating-point identity of precise decimal spellings.Coverage:
ty_encode::roundtrip_preserves_every_literal_identityty_decode::oversized_bigint_literal_is_rejected_before_parse_without_echoing_inputvalue_decode::typed_literal_preserves_identity_beyond_payload_shapeliteral_matching_checks_value_and_prefers_exact_literalfloat_literal_matching_preserves_negative_zero_and_decimal_precision7. Class annotations preserve nominal identity; plain objects use structural matching
Class
value_typeis not merely an ambiguity escape hatch. Generated class instances carry nominal identity even when the current signature has only one possible class:Plain JavaScript/TypeScript object literals remain unannotated
map_valuepayloads. When such a map is considered for a class union member, the engine uses the loaded class schema instead of treating every map as every class:This checks unknown keys, field aliases, required versus skipped/optional fields, and recursively visible field payloads. If both class schemas accept the same object, a strict typed/unregistered context rejects the ambiguity, while a registered dynamic language chooses the first declared structurally matching class. A typed producer is still expected to preserve its nominal selection rather than depend on that default.
These are two separate base-to-head changes:
InboundClassValue.class_tytoInboundValue.value_type.class_ty;The end-to-end example for the second behavior is the generated TypeScript test
round_trip_complex_profile accepts plain object literals (no class constructors)inbaml_language/sdk_tests/crates/typescript/type_shapes/customizable/roundtrip_complex_models.test.ts. Its nested object contains aCardPayment | WirePayment | nullfield:Before schema-aware matching, both class arms appeared to be generic map matches and declaration order could win. After this change, the loaded field schemas make only
CardPaymentaccept the card keys. This pre-existing high-level test exercises the new matcher; it does not prove class annotations, because the value is intentionally a plain object with no generated constructor tag.8. Generic classes: exact arguments when available, nominal refinement when erased
The wire location changes base-to-head:
Python Pydantic and TypeScript may know that an object is a
Boxwhile not retaining a concrete runtimeT. RemovingInboundClassValue.class_tywithout nominal refinement would discard the only class discriminator.The producers therefore use these encodings:
The empty-argument form is the one deliberate exception to “
value_typeis exact”: it is a nominal class hint. The engine may refine it against one contextual instantiation:It cannot select between multiple concrete instantiations:
Known arguments must remain compatible with the contextual arguments and arity. A bare map or shape-only instance can still become
Box<int>when the declared slot already suppliesBox<int>; this avoids requiring recursive metadata where context is sufficient.Concrete behavior:
The sparse annotation also participates in generic inference.
synth_ty_from_valuetreats it as stronger evidence than payload shape, and formal-aware generic reconstruction peels the transient annotation carrier before inspecting an erased generic instance's fields.Coverage:
nominal_generic_class_annotation_refines_from_contextnominal_generic_class_annotation_cannot_choose_concrete_union_armgenerics_explicit::inbound_instance_arg_missing_type_args_uses_contextgenerics_explicit::inbound_bare_map_against_generic_class_uses_contextgenerics_explicit.rstest_generic_instance_carries_sparse_value_typetest_unbound_generic_instance_carries_nominal_sparse_value_typeunbound generic instance carries only nominal class identitynon-generic class instance carries nominal identity9. Media uses an exact primitive annotation, not the Python wrapper class
Python media objects are class-shaped implementation shells containing
_data, but their BAML type is the primitive media kind. The Python encoder now emits:The important base-to-head distinction is:
Previously the only special inbound type channel was
InboundClassValue.class_ty, which cannot correctly describe a primitivepdf. Treating the shell as an ordinary class/map also loses which media arm the wrapper represents. The node-level primitive annotation lets the engine unwrap_datawhile retaining the actual BAML media kind.The engine recognizes the class shell in a media context, requires exactly
_data, and recursively materializes that handle as the annotated/contextual media type.Inbound union matching is media-kind-specific:
imagematchesimageand genericmedia, but notaudio. This exact-kind claim applies to the engine inbound matcher; the existing schema-free host-return validator still matches media by carrier rather than by kind.The Rust
media_pdf_roundtrips_through_engineintegration test verifies the_dataclass-shell-to-canonical-media conversion under contextualpdf. The generated Python media round-trip suite exercises the actual annotated wrapper decode/re-encode path. The Rust integration test itself does not construct a sparseBexExternalValue::typedcarrier, so it proves contextual shell conversion rather than Python's annotation emission.10. Reflected
typevalues remain payloadsvalue_typeandty_valueare independent:value_typedescribes the current inbound node;ty_valueis the payload when a BAMLtypevalue is passed as data.This is regression scope, not a new reflected-type feature:
ty_valuepredates this PR. The failure being guarded against is the new transient annotation/union carrier accidentally consuming a reflected type payload as metadata while coercing a union arm.host_selected_metatype_arm_survives_argument_coercioncovers that distinction.Outbound unions: one emitted arm discriminator
The outbound union envelope is now:
The selected type is derived from the union plus the index:
The
optionalscalar is important: presence distinguishes “the first arm, index 0” from “no discriminator was encoded.”value_option_nameremains human-readable metadata and is not authoritative.The encoder validates that
UnionMetadata.selected_optionis a member ofunion_typeand emits the first structurally matching index:runtime_ty_structurally_equalignores source-only attributes, treats union member order as semantically irrelevant, and preserves duplicate multiplicity when comparing whole union types.selected_arm_equaladditionally tolerates the legacy root representation where a non-null arm is wrapped inT | null.Duplicate-arm trade-off
The index preserves the position of distinguishable arms, including ambiguous payload shapes such as
int[]versusstring[]. It does not preserve which occurrence was selected among structurally identical duplicate members. Because runtime union metadata stores the selected type rather than an occurrence index,int | int | stringcanonicalizes anintselection to the firstintindex.Current consumers
This PR emits
selected_option_indexand makes it authoritative in the static C++, Rust, Java, Swift, and C# union decoders. They first resolve the index through the rawself_type.optionsarray and then match that selected descriptor to the generated host arm. They do not apply the wire index directly to a compact host union:self_typecan contain duplicate options, optional shells, or null holes that are absent from the generated host wrapper. Legacy/bare envelopes with no index retain structural fallback. Go's dynamic bridge value now exposes the optional index alongside the display-only variant name; Python and legacy TypeScript continue to expose their dynamic value shapes.The outbound encoder verifies selected-type membership. It does not independently revalidate that an arbitrarily forged nested payload inhabits the selected arm; normal engine materialization is expected to establish that invariant before encoding.
Coverage:
outbound_union_encodes_selected_index_for_ambiguous_numeric_armsartifact_safe_union_encodes_selected_indexoutbound_optional_null_preserves_declared_member_indexoutbound_union_rejects_selected_type_absent_from_declared_unionoutbound_union_matches_structurally_equivalent_selected_typeunion_structural_equality_ignores_member_orderunion_structural_equality_preserves_duplicate_multiplicityunannotated_structurally_duplicate_members_select_first_canonical_armBoundary matching changes, by layer
This PR does not introduce one universal matcher. There are two relevant layers with different jobs:
bex_external_types, shared by native and WASM callback return paths, with an existing engine schema-aware second pass.The exact changes are:
f64bits in this PRbaml.json.jsonvoidcallback returnnullis the canonical top-level representationnullnow accepted for top-levelvoidCanonical JSON alias
baml.json.jsonnow uses one recursive predicate in both inbound matching and host-return validation:This admits the JSON algebra and rejects non-finite floats, bigint, bytes, classes, enums, media, handles, and forged union metadata.
Coverage:
host_return::canonical_json_alias_accepts_only_the_json_value_algebraconversion::canonical_json_alias_matches_values_and_selected_union_armsHost-return validation is strengthened, not introduced
validate_host_return, native/WASM use of it, class-name checks, recursive list/map checks, class-field schema validation, wrong-returnHostContractViolation, declared throw enforcement, BAML catch routing, and the in-flight dispatch guard all predate this base-to-head diff.This PR specifically adds or changes:
RuntimeTy::EnumVariantvalidation in both the schema-free and engine-side validators;Nullas the canonical completed value for a top-levelvoidcallback;voidcallback binding while nested/unresolvedvoidremains rejected;It is inaccurate to present the existing wrong-return, throw/catch, or cancellation-table test suites as newly implemented by this PR.
Major host-callable change: preserve callback argument types on the BAML-to-host path
There are real host-callable changes in this PR. The primary regression addressed is on the BAML-to-host argument path—not the already-existing host-return/catch system.
Before this change, ordinary sysop conversion was used for the callback's argument pack. That conversion is intentionally type-erased and could not preserve the selected arm of an empty
int[]versusstring[].The engine now reads the realized parameter contract from the
Object::HostClosure, validates the[required_args, optional_args]pack, and converts each supplied value with its declared parameter type:The resulting callback wire value contains the declared union plus the selected index even when the nested payload is empty:
Optional parameters preserve omission: only entries actually present in the optional-argument map are serialized. A call to
f()sends no value, whilef(value = "supplied")sends the selectedstringarm.New end-to-end tests in
bex_engine/tests/host_value_callable.rs:host_callable_arguments_preserve_closed_union_selected_arm_on_wirehost_callable_union_envelope_preserves_empty_container_arm_identityhost_callable_optional_union_is_omitted_or_sent_with_selected_armSupporting unit tests:
host_call_parameter_type_tests::resolves_required_and_exact_optional_wire_nameshost_call_parameter_type_tests::rejects_malformed_required_arity_and_optional_nameAdjacent lifecycle changes
This PR moves the already-cancelled check before host-call argument traversal, avoiding a potentially large recursive conversion for an operation that will never dispatch. The cancellation behavior itself and in-flight eviction guard predate this diff; there is no new dedicated test that measures the O(1) pre-check.
At the native C-FFI completion boundary, a new drop guard drains queued host releases on every return. This covers a late throw/panic arriving after cancellation removed its call ID: decoding creates an opaque error handle, the unknown-ID path drops it, and the boundary now synchronously releases the host key. An accepted throw retains the key until the delivered payload is dropped.
Coverage:
bridge_cffi::host_value::late_unknown_throw_releases_opaque_handle_but_accepted_throw_retains_it.SDK-facing
SelfloweringSDK generators should not need to reconstruct the class that owns a method's
Self.baml_project::build_symbol_poolnow creates the owning TIR class type once:It passes that type as
self_tywhile lowering class-method parameters and return types. The normal compiler type-expression lowering performs recursive substitution, so nested forms work without generator-specific string rewriting:SDK-facing results include:
This covers instance and static method parameters and return types. The receiver was already omitted from public instance-method arguments before this PR. This diff does not thread
self_tythroughresolve_throws, so it should not claim coverage forSelfin a method'sthrowstype.Coverage:
baml_project::client_codegen::test_class_method_self_lowers_to_owning_codegen_class.Static bridge completion: exact empty-container arms
The cross-SDK regression fixture is intentionally payload-ambiguous:
Both selected values below have the same payload bytes:
Without a node annotation, a strict bridge cannot tell whether the caller selected
string[]orint[], and an outbound decoder cannot reconstruct the correct generated union wrapper from the empty payload. The static encoders now preserve the selected concrete arm while still dropping the host union envelope itself.Rust generates this shape for each union arm:
Vec<T>,IndexMap<K, V>, andHashMap<K, V>also expose/attach their exact container type.annotate_selected_typenever writes a root union/optional and never overwrites an annotation already emitted by a nested selected union.C++ follows the same rule in the generic codec:
The C++ codecs now provide
baml_ty()for primitives, literals, lists, maps, optionals, boxes, variants, generated enums, generated classes, and aliases. This keeps recursive type construction cheap and local to the host type rather than running a separate value-tree inference pass.Java has erasure, so generated call sites pair ambiguous arguments with their already-pooled BAML descriptor:
BamlTypedValueis only an internal generated-call carrier; it adds no protobuf field and does not touchFunctionCallContext.ProtoWriteruses the descriptor to unwrapUnion2.Arm0/Arm1or a generated nominal union record, select the contextual arm by declaration index, and annotate the selected payload. Context is then threaded through list elements, map values, and class fields. Ordinary Java lists/maps are not blindly marked exact when the runtime descriptor has erased nullable/generic details; exact annotations are emitted for selected arms and nominal instances where the descriptor is trustworthy. Media identity is encoded as the exactBamlTy.mediaonvalue_type, while the class-shaped payload remains only the_datahandle transport.Swift's
BamlUnion2throughBamlUnion8use the same selected-node projection throughBamlTypeDescriptor.The outbound side deliberately resolves the index in two steps:
An optional shell around a selected non-null value is transparent during host matching. This matters because an engine union such as
null | string[] | int[]can report index2even though the generated host union has only two non-null arms. Applying2directly to the host union would be out of range; resolvingself_type.options[2] == int[]and then matchingint[]to the host arm is correct.Regression coverage now asserts both non-empty and empty arms in Rust, C++, Java, and Swift. The empty
int[]case specifically failed before this completion because it canonicalized/decoded as the firststring[]arm.Producer behavior by SDK
The intended generated-SDK rule is compositional: every generated host type converts itself to an inbound BAML value, and a generated union type delegates to its selected arm. That delegation occurs recursively in class fields, map values, and list entries. It is by design that inbound drops the host-language union wrapper; outbound retains a selected index because the receiving SDK must reconstruct one.
For typed hosts, projection must not discard the selected arm's concrete container/literal/class type when the remaining payload is ambiguous. For dynamic hosts, raw values stay sparse and the engine uses the registered language policy rather than requiring recursive host-side type inference.
valueType.classTy; exact generic args are included only when every$typesbinding is present, otherwise only nominal class identity is sent. Typed host throws and built-in host-callable error classes use the same field.value_type.class_ty, including Pydantic generic arguments when available. Unparameterized generics emit the nominal class with zero args. Media wrappers emit exact primitive media kinds and keep_datain the class-shaped payload shell.InboundValue.value_type.class_ty. Generated unions annotate the projected selected arm; container codecs provide exact recursiveBamlTy; outbound union decoding resolves the canonical selected descriptor throughself_typebefore choosing the generated enum variant. Current Rust sdkgen still skips generic classes, so generated class annotations have zero type args.InboundValue.value_type. Generic codecs expose exactbaml_ty()descriptors, selected union arms are annotated inbound, and indexed outbound envelopes select by descriptor rather than payload guessing.BamlTypedValuecarrier for erased descriptors; selected union arms, class identity/generic arguments, fields, and media encode through node-localvalue_type. Generated union records consume outbound selected descriptors before structural fallback.ValueType; generated unions project to their selected payload and reject union/optional shells as inbound annotations. Outbound union decoding resolvesSelectedOptionIndexthrough rawSelfType.Union.Optionsbefore choosing a compact generated arm. C# registers as a strict static producer.baml_gocopies are regenerated.baml_go.ClassusesInboundValue.value_type.class_ty, andDynamicUnionexposesSelectedOptionIndexas canonical identity while retaining the display-only variant name.BamlEncodabletypes expose a cheapBamlTypeDescriptor.BamlUnion2throughBamlUnion8keep the inbound payload bare but annotate it with the selected arm's exact type, generated classes move FQN/type arguments to node-localvalue_type, and generated union decoders resolve outboundselected_option_indexthrough the rawself_typebefore matching the compact host union arm.BamlSerializable.toBaml()can return anyInboundValue, including an exact literal annotation; generated class serialization moves nominal identity to the node-level field. Ordinary scalar/list/map values explicitly leavevalueTypeundefined.Swift's static union projection is implemented directly on the generic union family:
That makes the two empty payloads below different on the wire without serializing a recursive type tree for every value:
Generated Swift classes use the same node-local channel for nominal identity and concrete generic arguments:
On the return path, Swift now consumes the canonical index before legacy metadata or structural fallback:
The end-to-end
type_shapesSDK test calls the generatedround_trip_str_or_int_listclient with both empty arms and asserts that the returnedBamlUnion2preserves.t0versus.t1. This fails if inbound selected-node metadata is dropped or if outbound decoding guesses from the identical empty-list payload shape.C# integration added by the final
canarymergeC# is a static producer (
BridgeLanguage::CSharp = 5), so it uses the same strict policy as Go/Rust/C++/Java/Swift. Its encoder moves type information to the current node instead of the container payload:Typed empty lists/maps similarly set
ValueType.List/ValueType.Map; media sets the exactValueType.Mediakind while retaining the class-shaped_datatransport shell. A generated C# union still drops unionness recursively, but adds the selected concrete descriptor to the projected payload when that payload has no more-specific annotation. A selected root union or optional is rejected.Outbound C# treats the index as canonical and the option name as display-only:
This fixes
null | string[] | int[]selecting raw index2: the decoder resolvesint[]first, then maps it to compact generated C# arm1, rather than applying raw index2directly.The native phase-11 callback fixture exposed a shared-boundary regression for
CallbackBox<int>: the wire had an exact node annotation plus an intentionally anonymous class payload. The schema-free host-return guard used to reject that payload before the engine could apply its annotation. It now validates the annotated type against the declared return type; the engine then coerces the payload with that exact type before recursive schema validation. A focused test accepts matchingCallbackBox<int>and rejects an annotation for a different class.Go's checked-in protobuf bindings and affected runtime surfaces are included here: class construction uses node-local
value_type, and dynamic outbound unions retainselected_option_index. A future full generated-Go union implementation should follow the same existing protocol shape—each generated type converts recursively and each union projects to its selected value—without introducing an inbound union envelope.Raw legacy TypeScript strings and Python strings are not automatically inferred as literal annotations. The explicit literal producer example in this diff is TypeScript2's
toBaml()test:value_typeis a protocol capability; this PR does not add a new high-level “select this literal arm” wrapper to every legacy SDK.Generated artifacts and scope
Updated checked-in protocol artifacts:
.pb.h/.pb.cc;.py/.pyi;.pb.swiftsources and generation hashes;distfiles;Other shared code updates include the C++ generator codec, C++ host-value transcoder, Python/Rust/C# bridge producers, C# host-callable and media protocols, Python Rust host-callable builder, C-FFI host completion path, process-global inbound ambiguity policy registration, the WASM dynamic-policy start hook, and WASM test fixtures that construct
InboundValuedirectly.Canonical Go protobuf files and the duplicate
baml_go/internal/cffiinbound/outbound copies are regenerated from the final schema. The vendored CFFI header is synchronized as well. This removes the prior generated-proto sync failure and lets the focusedbaml_goruntime suite exercise the new class layout. The broader generated-Go union API remains outside this PR.Proto backward compatibility is not a design constraint for this work, but tag 7 in the outbound union message remains reserved in the resulting schema.
Review map
Recommended review order:
baml_language/crates/bridge_ctypes/types/baml_bridge/cffi/v1/baml_inbound.protobaml_language/crates/bridge_ctypes/types/baml_bridge/cffi/v1/baml_outbound.protobaml_language/crates/bridge_ctypes/types/baml_bridge/cffi/v1/baml_type.protobaml_language/crates/bridge_ctypes/src/ty_decode.rsbaml_language/crates/bridge_ctypes/src/ty_encode.rsbaml_language/crates/bridge_ctypes/src/value_decode.rsbaml_language/crates/bridge_ctypes/src/value_encode.rsbaml_language/crates/bridge_ctypes/src/error.rsbaml_language/crates/bex_external_types/src/bex_external_value.rsbaml_language/crates/bex_external_types/src/runtime_ty_identity.rsbaml_language/crates/bex_external_types/src/host_return.rsbaml_language/crates/bex_engine/src/conversion.rsbaml_language/crates/bex_engine/src/inbound_config.rsbaml_language/crates/bex_engine/src/lib.rsbaml_language/crates/bex_engine/tests/generics_explicit.rsbaml_language/crates/bex_engine/tests/host_value_callable.rsbaml_language/crates/bex_engine/tests/media_roundtrip.rsbaml_language/crates/baml_project/src/client_codegen.rsbaml_language/crates/bridge_cffi/src/ffi/runtime.rsbaml_language/crates/bridge_cffi/src/ffi/host_value.rsbaml_language/crates/sys_native/src/host_dispatch.rs(documentation clarification only)baml_language/crates/sys_wasm/src/host_value.rsandbaml_language/crates/bridge_wasm/tests/host_callable.rs(wire fixture updates)baml_language/sdks/typescript/bridge_typescript/typescript_src/proto.tsbaml_language/sdks/typescript/bridge_typescript/tests/test_typemap.test.tsbaml_language/sdks/typescript/bridge_typescript/tests/call_function.test.tsbaml_language/sdks/python/src/baml_bridge/proto.pybaml_language/sdks/python/tests/test_proto_generics.pybaml_language/sdks/python/tests/test_engine.pybaml_language/sdks/rust/bridge_rust/src/{encode.rs,baml_value.rs}baml_language/sdks/cpp/bridge_cpp/include/baml/detail/host_value.hbaml_language/sdks/cpp/sdkgen_cpp/src/lib.rsbaml_language/sdks/swift/Sources/BamlBridge/{Encode.swift,Decode.swift,BamlUnions.swift}baml_language/sdks/swift/rust/sdkgen_swift/src/emit.rsbaml_language/sdks/swift/Tests/BamlBridgeTests/FFISmokeTests.swiftbaml_language/sdk_tests/crates/swift/type_shapes/customizable/roundtrip_tests/TestUnions.swiftbaml_language/sdks/csharp/bridge_csharp/src/Proto/{PrimitiveProtocol.cs,MediaProtocol.cs,HostCallableProtocol.cs}baml_language/sdks/csharp/bridge_csharp/tests/{Baml.Bridge.Tests,Baml.Bridge.HostCallable.Tests}/Program.cstypescript2/pkg-proto/src/{encode.ts,test/encode-decode.test.ts}Validation performed
The merged head was validated locally across the new C# surface and the shared bridge/runtime code:
cargo test -p sdkgen_csharp --libcargo nextest run -p sdk_test_csharp --no-fail-fastcargo hawkStatic SDK coverage run during this PR also passed:
type_shapes: 5/5, including both empty selected-list arms;test jar, and generatedtype_shapes;type_shapes, sdkgen tests, and clippy;env -u GOROOT go test ./...insdks/go/baml_go.Shared engine and dynamic-language coverage included
bex_engine,bridge_cffi, explicit generics, host callables, WASM compilation/clippy, Python engine calls, and the TypeScript bridge suite. Python and TypeScript compile the same discriminator and call it through their public SDK boundary with an ordinary untyped empty list:These are end-to-end assertions: the untyped
[]crosses the public encoder, registered dynamic bridge, wire decoder, engine coercion, and BAML match, and selects the first declared structural arm. Before the dynamic policy, the Python case failed withvalue of type array matches multiple union members; add an inbound value_type annotation to select one.Focused coverage proves:
Selfsubstitution and process-global native/WASM policy registration.GitHub Actions for merged head
341a54237passed every triggered test, precommit, protobuf-sync, MSRV, size, and platform job, including the C# SDK matrix on Linux, macOS, and Windows.Policy boundary and remaining work
This PR distinguishes static and dynamic ambiguity policy and completes selected-node propagation for the Rust, C++, Java, Swift, and C# static union surfaces covered by the shared regression fixtures:
[]or distinguish every raw value's host-side union wrapper. Registered dynamic bridges therefore default after recursive matching: prefer an exact literal/enum-variant candidate, otherwise choose the first declared structural match.BridgeLanguageper process. Node.js/Python select dynamic defaulting; Go/Rust/C#/C++/Java/Swift select strict rejection; direct engine users with no registered bridge remain strict. WASM/TypeScript registers the dynamic policy at startup.None of this requires an inbound union envelope. Union projection is intentional and recursive: static wrappers preserve the selected payload node's exact type, while genuinely untyped dynamic values use the registered defaulting policy described above.
Deliberately out of scope
Selfsubstitution in methodthrowstypes.Summary by CodeRabbit
New Features
Selfin class method types, including nested and generic usages.Bug Fixes
Documentation