[RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol

[ethan-scitix]

Motivation

Model's access interface is overfit to the OpenAI /v1/completions protocol, and
that assumption is about to break in three directions at once: RL/logprobs
consumers, the "big three" latest APIs (OpenAI Responses, Anthropic Messages,
Google genai / AI Studio), and richer eval logic (reasoning, multi-turn, continuation).

Evidence the current abstraction leaks:

• Model.alogprobs(prompt, *, max_tokens, logprobs: int, echo, temperature) is
  completions vocabulary. ChatModel._alogprobs_impl accepts echo and silently
  ignores it — a phantom param on the universal interface.
• Every non-completions backend must translate: echo -> logprob_start_len,
  logprobs: int -> top_logprobs_num, max_tokens -> max_new_tokens.
• echo-based prompt scoring is itself an OpenAI workaround for the lack of a
  native scoring endpoint; sglang /generate has return_logprob and no echo
  concept. The base models a workaround as if it were the universal primitive.
• The output shape (top_logprobs: list[dict[str, float]]) cannot carry token ids
  or a reasoning channel — both required by real consumers (RL, reasoning eval).

The interface was only ever validated by one protocol. The incoming sglang native
backend, plus the big-three protocols, are the "second implementors" that expose the
boundary. Rather than keep bolting workarounds on, define a provider-agnostic
IR for model capabilities and make every provider an adapter onto it.

Proposed Change

Core idea — one Model IR + transports as frontends

• One Model IR, two message shapes: Request and Response (the existing
  ModelOutput is the Response side; rename or alias).
• Transport = provider frontend (strategy, composed into Model, not a base
  class): it lowers a Request into its native wire and lifts the native
  response back into a Response. OpenAI completions / OpenAI chat / OpenAI
  Responses / Anthropic Messages / Google genai / sglang native / vLLM are all just
  frontends.
• Capabilities = which IR features a frontend can honor, declared per transport
  as optional Protocols. The catalog is open: adding a capability later =
  add a Protocol + optional Request/Response fields (additive, not a redesign).
• Validation = a Request using a feature the transport lacks is rejected at
  setup, never silently ignored (kills the echo-on-chat class of bug).

Composition (Model)

Model = { resource pool / limiter (explicit collaborator) } + { Transport (frontend) }
  async def arun(req: Request) -> Response          # the one primitive (acquires limiters)
  # capability-gated sugar (thin wrappers over arun, present only when supported):
  #   agenerate(...)      [any]
  #   achat(...)          [Chat]
  #   ascore_input(...)   [InputScoring]

• Drop as_type (cross-kind base<->inst conversion): its only use case
  (mixing base and inst on one endpoint in a single eval) has no real consumer.
• with_args decomposes: per-task infer_args become plain Request fields
  (no derived model object); concurrency_limit sub-quota becomes an explicit
  resource-pool collaborator. No "conversion capability" axis survives.

Consumer -> capability matrix

consumer	capabilities
1. RL / agentic	Chat + Tools + SampledLogprobs(+token_ids) + (Reasoning)
2. reasoning eval	Chat + Reasoning + big-three adapters (Responses/Messages/genai)
3. multi-turn	Chat
4. base ppl / bpb	Completion + InputScoring(+byte/char accounting) + TopKLogprobs
5. special (budget/truncate/continuation)	Reasoning(controls) + Prefill / FIM

Carved OUT of the model interface (task layer): fallback-extract, LLM-judge
orchestration, thinking-truncation post-processing. (judge = a Chat call + task scoring.)

Provider -> capability matrix

transport	capabilities
OpenAI completions	Completion, InputScoring(echo), TopK
OpenAI chat	Chat, Tools, TopK, Structured
OpenAI Responses	Chat, Tools, Reasoning, Structured
Anthropic Messages	Chat, Tools, Reasoning(+budget), Prefill
Google genai / AI Studio	Chat, Tools, Reasoning
sglang /generate	Completion, InputScoring, SampledLogprobs, TopK, Prefill
vLLM (oai-compat)	~OpenAI + prompt_logprobs

IR shapes + capability catalog

Request:  input: str | list[Message]            # modality (rides with transport)
          sampling (temp/top_p/max_tokens/stop/seed)
          tools? ; reasoning?(budget/effort) ; response_format?
          prefix? / suffix?                      # constrained-context corner (Prefill/FIM)
          return_logprobs? / score_input? / top_k?

Response: texts
          reasoning?: { text, opaque_roundtrip } # opaque needed for AnthropIc/Google multi-turn signatures
          tool_calls?
          logprobs?(+token_ids) ; top_logprobs?
          input_scoring?(byte/char)              # for BPB/CLP
          usage

Capabilities: Completion | Chat | Tools | Reasoning | Prefill | FIM |
              InputScoring | SampledLogprobs | TopKLogprobs | StructuredOutput

• Core-now (has consumers today: feat(theoremqa): add dataset and k-shot base-model task #3/feat(session): tolerate throughput-only config diffs on --resume #4 + sglang track):
  Completion, Chat, InputScoring, TopKLogprobs.
• Staged (design the seam now, implement on demand):
  Reasoning, Tools, Prefill, FIM, SampledLogprobs, StructuredOutput.
• token_ids field name/shape stays deferred until the RL consumer pins it
  (aligned-with-logprobs vs standalone output_ids; prompt ids or not).

Feedback Period

~1-2 weeks (design-heavy).

Any Other Things

• Lock with RFC [RFC]: Resume version-compatibility gate + per-record version provenance #24. The Response IR is the persisted record schema
  (@sieval_record). So IR evolution is exactly what [RFC]: Resume version-compatibility gate + per-record version provenance #24's resume version gate
  protects: a breaking IR change ships as a >= minor bump and the gate refuses
  cross-series resume. The two RFCs are coupled by design.
• Non-goals.
  • Environment / sandbox execution (e2b-style RPC) is a sibling abstraction,
    not a model transport. The Model IR only needs to represent tool_call /
    tool_result; who executes them (e2b, local) is a separate Executor IR /
    future RFC. Folding sandbox RPC into the model transport repeats infra-layer
    role confusion.
  • Task-layer eval logic (fallback-extract, judge orchestration, truncation).
  • Structured-FIM wiring until a FIM benchmark exists (OSS FIM is usually
    Completion + a task-built template, not a transport feature).
  • as_type / cross-kind model conversion.
• Sequencing. The sglang native backend (feat(models): add SglangGenModel for echoed-input logprobs via sglang /generate #21 + follow-up: SglangGenModel
  onto Model[str] + native /generate) lands first under the current interface,
  translating params internally. This RFC is the eventual clean target and does not
  block that work.
• Dependencies. Big-three adapters mean multiple SDKs -> optional-dependency
  groups + lazy transport loading.
• Capability requirement declaration. Extend @sieval_task to declare required
  capabilities (today it only declares model_type); assert at setup. Replaces the
  current type: chat | gen with a richer capability requirement.

Before submitting a new issue

• Make sure you already searched for relevant issues and documentation.

[jack-scitix-ai]

Based on the PR #45 review, I drafted the following pre-implementation revision for RFC #25. This is a design-only proposal; implementation will not begin until we agree on Section 2 (class structure).

RFC #25 revision: provider-neutral Model IR

Status: draft for pre-implementation review
Related: RFC #25, PR #45, #27, #47, #49/#57, #58, #59, #60/#61
Implementation gate: agree on Section 2 before changing production behaviour

Motivation

PR #45 successfully moves the existing OpenAI chat, OpenAI completions, and
sglang /generate wire behaviour behind a common IR. The expensive parsing and
lifting work should be retained. The surrounding abstraction is not ready to
merge because it has three different concerns in one flat Capability set:

1. whether a frontend consumes a request field;
2. what a frontend guarantees in the response;
3. what an engine and a running instance can safely provide.

That flattening produces both false negatives and false positives:

• request fields can be silently ignored because capability checks are manual;
• OpenAI chat lowers and lifts reasoning but does not declare either reasoning
  capability;
• a Flag composite stored in a frozenset is reported missing even when all
  constituent flags are present;
• production tasks still use model_type and isinstance, while no production
  task declares requires;
• SglangGenModel binds an engine name to one frontend even though sglang can
  serve more than one wire protocol;
• removing as_type removed cross-frontend client and limiter sharing without
  providing a replacement.

The revision keeps the IR and the validated wire transposition, but rebuilds the
class structure and validation boundary from observed frontends rather than from
a vendor-documentation vocabulary.

Proposed Change

1. Scope and invariants

The revision MUST preserve these invariants:

• Model.arun(Request) -> Response remains the provider-neutral primitive.
• Every non-default request leaf is lowered, explicitly rejected, deliberately
  treated as a documented no-op, or forwarded unchanged. There is no silent
  fifth state.
• Limiter hierarchy remains global -> task -> stage -> model.
• A derived model that changes frontend shares the same instance and resource
  pool when that pool can supply the connection required by both frontends.
• Response remains one closed persisted record with optional record-valued
  channels. It does not become a runtime-dependent subclass hierarchy.
• Wire results already A/B-verified in PR feat(models): rebuild the provider-neutral model binding plane (RFC #25) #45 remain behaviourally identical,
  except for separately documented bug fixes.
• Engine and frontend are independent axes. Neither class names nor frontend
  names are used as an engine proxy.

The following are explicitly outside this revision:

• RFC [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 serving constraints that depend on engine/version/instance facts or
  live probes;
• replacing the public YAML type: chat | gen schema;
• adding Anthropic, Google, or OpenAI Responses frontends;
• modelling fields solely because they appear in a vendor API reference.

1.1 Related work, ownership, and sequencing

This revision integrates with the following work rather than creating parallel
mechanisms:

Work	Owns	Boundary used here
#27 and #59	recipe hardware/capabilities separation and the shared legacy model-kind resolver	#25 supplies the normalized task modality to that resolver; it does not add another recipe or kind inference path
#47	serving constraints, plan-time remedies, and setup-time verification requirements	#25 supplies the frontend/engine/instance axes, immutable observed facts, and the request gate
#58	projecting observed serving facts into dataset-packing fingerprints and strict setup comparison	#25 keeps facts provider-neutral and return-valued; it does not design ServedWindow or repack samples
#49 and #57	additive task identity in run meta.json through get_task_run_identity()	per-call provenance remains in Response/ModelOutput; the requirement registry golden is CI coverage, not a second persistence path
#60 (merged) and #61 (open)	the stage-output records and persistence of a grader's full ModelOutput in extra.grader_output	grader provenance rides on that output's ModelMeta; #25 does not add another grader-feedback field

At the time of writing, #60 is merged while #57 and #59 remain open PRs.
Implementation must rebase onto main containing #60 and onto #59's final merged
resolver, or explicitly stack behind #59 while it remains open, rather than copy
their helpers or persisted schemas. #57 can land independently, but combined
serialization and meta-drift tests must cover both additive changes. If either
open PR or the eventual #61 migration changes shape, this section and the
dependent acceptance criteria are updated before implementation.

2. Class structure and ownership

This section is the pre-implementation review gate.

2.1 Object graph and roles

Model
 |- model id and legacy-builder compatibility defaults
 |- Instance ---- Engine
 |- bound Frontend <- FrontendFactory.bind(...)
 |- ResourcePool -- owns connections and the shared limiter
 `- optional model-local limiter

The eventual effective service contract is a function, not a class constant:

service_contract = f(frontend, instance.engine, instance.facts)

Role	Owns	Must not own
Model	orchestration, defaults, and references to instance/frontend/pool	provider SDK construction, engine inference from a class name
ResourcePool	live clients/connections and the shared capacity limiter	request translation or engine semantics
FrontendFactory	binding a borrowed compatible connection to one wire frontend	limiter policy or engine identity
Frontend	pure prepare/lower, terminal request execution, request/output contracts	client lifetime or deployment planning
Engine	static engine semantics and instance-probe rules	process lifecycle, frontend selection, or connections
Instance	immutable deployment identity, frontend endpoints, engine provenance, and resolved facts	credentials or live clients

Model MUST stop constructing AsyncOpenAI unconditionally. A provider-neutral
Model cannot own an OpenAI-specific client, and an injected frontend must not
bring an unrelated connection while the model retains an unused one.

ResourcePool is the sole owner of the one live connection family used by this
revision. It is bound to exactly one instance connection identity. Sharing is
explicit by object identity; pools are never globally deduplicated by URL
because the same URL can represent different credentials, retry policies, or
quotas. Every retained frontend borrows the exact same compatible connection and
must not close it. A frontend requiring another connection family requires a new
pool.

The shared pool and a model-local limiter retain the existing hierarchical
semantics: effective concurrency is the minimum of all applicable levels. A
derived concurrency_limit is a local cap, not a hard reservation from its
parent. If hard reservation is ever required, it needs a separate allocator.

2.2 Interface sketch

The names are illustrative; ownership and call order are normative.

@dataclass(frozen=True)
class Instance:
    engine: Engine
    engine_source: Literal["deployment", "config", "unknown"]
    frontend_endpoints: Mapping[str, str]  # frontend id -> endpoint
    facts: InstanceFacts
    deployment_id: str | None = None


class Engine(Protocol):
    id: str

    def supports_frontend(self, frontend_id: str) -> bool: ...
    async def verify_instance(
        self, candidate: Instance, probe: ProbeConnection
    ) -> InstanceFacts:
        """RFC #47/#58 seam; RFC #25 does not probe once per call."""


class ResourcePool:
    connection_family: str

    @classmethod
    def bind(
        cls,
        *,
        instance: Instance,
        connection_family: str,
        connection: object,
        shared_limiter: CapacityLimiter | None,
    ) -> Self:
        """Transfer ownership of connection into a pool bound to instance."""

    @property
    def connection(self) -> object:
        """Return a borrowed reference to the pool-owned connection."""

    @asynccontextmanager
    async def acquire(
        self, local_limiter: CapacityLimiter | None
    ) -> AsyncIterator[None]: ...

    async def aclose(self) -> None: ...


class FrontendFactory(Protocol):
    id: str
    connection_family: str

    def bind(
        self,
        *,
        model_id: str,
        instance: Instance,
        resources: ResourcePool,
    ) -> Frontend: ...


class Frontend(Protocol):
    id: str
    legacy_input_adapter: LegacyInputAdapter | None
    request_contract: RequestContract
    output_contract: OutputContract

    def validate_request(self, req: Request, audit: RequestAudit) -> None: ...
    def prepare(self, req: Request, audit: RequestAudit) -> PreparedRequest: ...
    async def execute(self, prepared: PreparedRequest) -> Response:
        """Consume streams/tool loops and return one terminal Response."""


class Model:
    @classmethod
    def bind(
        cls,
        *,
        model_id: str,
        instance: Instance,
        resources: ResourcePool,
        frontend: FrontendFactory,
        defaults: RequestDefaults | None = None,
        local_limiter: CapacityLimiter | None = None,
    ) -> "Model": ...

    def with_args(...) -> Self: ...
    def with_frontend(self, frontend: FrontendFactory) -> "Model": ...
    async def arun(self, req: Request) -> Response: ...

InstanceFacts is a frozen, provider-neutral record of observed values. It is
not a boolean verdict and is not private to a model frontend. verify_instance()
returns facts even when the immediate caller only needs to validate a constraint,
so #47 can enforce them and #58 can later define stable projections such as a
served-window fingerprint without reopening this signature. This revision does
not predefine ServedWindow or dataset-packing policy.

Engine owns engine-specific probe rules and interpretation of the returned
facts. A compatible frontend/ProbeConnection supplies the protocol execution
channel but does not maintain a second verification policy. This is the single
seam referred to as Frontend.verify_instance() in the earlier #47/#58 prose;
those RFCs must consume this ownership split rather than add another method.

Model defaults are compatibility defaults used only by agenerate() and
alogprobs() while they construct a Request. arun(Request) consumes the
supplied request verbatim. with_args() therefore changes only those legacy
builder defaults, schema-external metadata, and the local limit. Reserved keys
including frontend, transport, engine,
api_base, credentials, and retry policy MUST fail immediately instead of
falling into extra_wire_params.

A legacy builder merges model defaults first and per-call arguments second,
normalizes that effective value into one tagged Request, and only then enters
the same request gate as arun(). A per-call argument therefore overrides a
model default. Defaults never modify an explicit arun(Request).

with_frontend() is the only frontend-changing API and MUST:

1. require frontend.connection_family == resources.connection_family; a
   mismatch raises FrontendBindingError before binding or I/O and never
   allocates a connection or pool implicitly;
2. validate that the instance exposes the endpoint and the engine permits the
   requested frontend;
3. bind the new frontend to the compatible connection borrowed from the same
   pool;
4. preserve the instance, model id, limiter hierarchy, compatibility defaults,
   and schema-external metadata;
5. return a plain Model, not Self, so a ChatModel changed to completions
   does not remain a lying ChatModel at runtime;
6. perform no persistence I/O during the swap; subsequent call metadata records
   the new frontend id.

The swap does not rerun derive_model_type(), select another recipe capability
profile, or mutate an already resolved DeploymentPlan. Those are composition
and launch-time decisions, not frontend-binding behaviour.

Changing endpoint, engine, credentials, or retry policy requires a new
Instance + ResourcePool; it cannot masquerade as a frontend swap. The old
copy.copy() plus __class__ mutation is invalid once a frontend is bound.

ResourcePool.aclose() MUST be idempotent, reject new admissions, drain entered
requests, and close each owned connection exactly once. It must not depend on
__del__. The session owns and closes each unique pool; standalone callers use
an explicit pool-scoped context manager and thereby invalidate every model
sharing that pool.

2.3 Compatibility classes and names

ChatModel and GenModel remain thin wrapper classes for one deprecation cycle.
They select openai_chat and openai_completions factories respectively but are
not used by production isinstance checks. Keeping classes temporarily preserves
external subclassing and avoids calling a function/partial an "alias" while
breaking isinstance immediately.

The canonical Model.bind(...) path requires an existing pool and never creates
a provider client. For one compatibility cycle, the legacy
ChatModel(model, api_base, api_key, ...) and GenModel(...) constructors may
create a private pool. The returned root wrapper owns the pool lifecycle and
exposes aclose() and an async context manager; derived/swapped models borrow
the pool and retain a strong reference to that lifecycle owner. Their aclose()
and async context manager delegate to the owner, so
ChatModel(...).with_args(...) cannot orphan the connection. Closing remains
pool-scoped, invalidates every sibling, and is performed by the pool rather than
by a wrapper directly.

Internal kind decisions move to typed requirements and frontend metadata in the
same change that removes Task._validate_model_type(). Model[TModelInput] and
the public dependency on OpenAI's ChatCompletionMessageParam are removed.

Transport and transports/*Transport become deprecated re-exports of
Frontend and frontends/*Frontend for the same compatibility period. The new
implementation and documentation use only Frontend for the protocol axis;
backend is not a core model-layer term. The legacy infer.backend spelling is
normalized once into DeploymentPlan.engine_id; deployment implementation names
never substitute for Engine.

with_frontend() always returns a plain Model. If as_type() is retained, it
accepts only compatibility wrapper classes that declare a frontend factory and
implement _from_binding(...); other classes raise TypeError. It uses the same
private rebinding primitive as with_frontend() but reconstructs the requested
wrapper so its historical runtime type remains truthful. It preserves the
instance, pool, lifecycle owner, limiter hierarchy, defaults, and metadata, and
never mutates __class__. If that behaviour is not preserved, as_type() is
removed rather than advertised as a compatibility adapter.

SglangGenModel is handled by the probe decision in Section 8:

• if sglang OpenAI completions has required parity, deprecate the class in favour
  of engine=sglang plus the OpenAI completions frontend;
• if native /generate remains necessary, retain it as an explicit frontend,
  not as an engine-bearing model subclass.

2.4 Engine identity source

Engine resolution has exactly one implementation. For managed deployments, the
resolved DeploymentPlan.engine_id is authoritative. The current legacy
infer.backend spelling is normalized to engine_id exactly once by the
resolver. An explicit models.<name>.engine is an assertion that must agree; it
never overrides the deployment plan. InferHandle.backend, response model names,
and persisted audit metadata are not engine-identity sources. The current
endpoint_map[str, str] handoff therefore becomes an instance binding carrying
the endpoint, engine, version, and resolved deployment facts.

For remote endpoints:

• an explicit models.<name>.engine supplies the engine identity;
• absent an explicit value, the engine is unknown rather than guessed from
  type, frontend, URL, class, response model, or persisted audit metadata;
• probes refine the version and individual instance facts, not engine identity;
• if the normalized deployment-plan engine_id and explicit model assertion
  disagree, validation fails before launch.

Verification returns facts, after which the composition root constructs a new
frozen Instance before binding a frontend. A bound model's instance never
mutates.

Engine describes server semantics only. Process submit/readiness/cleanup stays
in sieval.infer; sieval.core does not import upward into that layer.

engine no longer selects a frontend and is valid with both chat and completions
frontends. This revision retains #59's derive_model_type() as the single
compatibility resolver for a configured model but tightens #59's current
explicit-value precedence: YAML type becomes an assertion, not an override. It
must agree with the normalized task modality requirements defined in Section 6.
The resolver uses chat only when neither a task requirement nor an explicit
value supplies an answer. A disagreement fails before launch; #59's tests and
error guidance are updated accordingly, and no caller independently re-derives
the value from task classes, model wrappers, or endpoint shape.

At initial composition/root binding, the one resolved value is consumed in both
existing compatibility roles:

• chat -> OpenAI chat-completions frontend;
• gen -> OpenAI completions frontend.

PR #59's recipe layer maps the same result from chat/gen to
instruct/base. That is a compatibility coupling, not a declaration that
checkpoint kind and frontend are the same axis. An explicit frontend: is
validated against both the tasks' request modality and the resolved legacy
type's modality, including when no task points at the model. It does not select
or rewrite a recipe capability profile. Derived/swapped models inherit the
already resolved plan and never consume this value for recipe selection again.

If the native sglang frontend survives the probes, this revision also adds an
explicit frontend: selector. For one compatibility cycle only, an explicitly
configured models.<name>.engine: sglang, a resolved legacy type of gen
(whether explicit or inferred from task requirements), and no frontend map to
sglang_generate with a deprecation warning. An engine learned only from a
managed DeploymentPlan never triggers that shim. If this configuration
addition is rejected, native sglang cannot survive this revision while
preserving the engine/frontend split.

3. Request structure and field promotion

First-level concerns map one-to-one to request subrecords. Only concerns observed
in retained frontends are populated now. Empty groups and fields copied from
future vendor documentation are not added.

3.1 Normative target schema

input is a tagged provider-neutral union, not str | list[dict]:

@dataclass(frozen=True)
class CompletionInput:
    text: str


@dataclass(frozen=True)
class ChatInput:
    messages: tuple[dict[str, JSONValue], ...]


type ModelInput = CompletionInput | ChatInput

The message sequence is an opaque leaf for request accounting; the IR does not
import an OpenAI SDK message type. A direct arun() caller must choose
explicitly. Legacy sugar on every bound model delegates untagged input coercion
to the bound frontend's small legacy_input_adapter: OpenAI chat
constructs ChatInput, while OpenAI completions and native generation construct
CompletionInput. Already-tagged input is preserved. A frontend without an
adapter rejects legacy agenerate()/alogprobs() and requires explicit
arun(Request(...)). This also defines the behaviour of a plain Model returned
by Model.bind() or with_frontend(); wrapper runtime type is not used to infer
modality.

The complete first-class leaf schema for this revision is:

Leaf pattern	Type	Default	Accounting shape
input.completion.text	str	required	opaque, union alternative
input.chat.messages	tuple[dict[str, JSONValue], ...]	required	opaque, union alternative
sampling.max_tokens	int | None	None	scalar
sampling.temperature	float | None	None	scalar
sampling.top_p	float | None	None	scalar
sampling.stop	tuple[str, ...] | None	None	opaque sequence
sampling.seed	int | None	None	scalar
sampling.frequency_penalty	float | None	None	scalar
sampling.presence_penalty	float | None	None	scalar
sampling.n	int	1	scalar
scoring.sampled_logprobs	bool	False	scalar
scoring.input_logprobs	bool	False	scalar
scoring.top_logprobs	int	0	scalar
reasoning.effort	closed effort enum or None	None	scalar
tools.functions	tuple[dict[str, JSONValue], ...] | None	None	opaque sequence
structured_output.format	dict[str, JSONValue] | None	None	opaque mapping
scheduling.stream	bool | None	None	scalar/advisory
extra_wire_params[key]	JSONValue	key absent	dynamic opaque leaf

The subrecords themselves use immutable defaults such as SamplingParams();
only their non-default leaves become active. Provider-neutral invariant checks
run before frontend validation, including n >= 1, max_tokens >= 0,
top_logprobs >= 0, and finite/range checks for sampling values.

This schema deliberately removes the unused vocabulary introduced by the first
RFC #25 implementation:

Current field	Revision treatment	Reason
sampling.top_k_sampling	extra_wire_params["top_k"]	engine extension, no cross-frontend semantic translation
sampling.stop_token_ids	extra_wire_params["stop_token_ids"]	tokenizer/engine-specific extension
server_tools	remove until a retained frontend executes server tools	no current frontend consumes it
reasoning axes other than effort	remove until a retained frontend consumes them	documentation vocabulary, not implemented coupling
prefix	extra_wire_params["prefill"] for native sglang	one implemented prefill mapping, no cross-frontend semantic translation
suffix	remove until a retained frontend consumes it	no current consumer; FIM semantics must not be inferred from prefill
session_id and opaque reasoning roundtrip	remove until a stateful frontend exists	no retained frontend implements the contract

Dynamic extensions still participate in exhaustive accounting. An OpenAI
frontend can consume extra_wire_params["top_k"] by placing the unchanged value
under extra_body; a retained native sglang frontend can consume it at
sampling_params.top_k. If native sglang is retained, it also consumes
extra_wire_params["prefill"] at sampling_params.prefill; a migration test
pins exact wire parity with the former first-class Request.prefix path. A
frontend with no safe destination rejects either extension.

3.2 Promotion rule

A provider-neutral field is promoted when there is demonstrated translation
coupling: at least two implemented frontends express the same semantic value in
different wire shapes. An exception requires an explicit consumer and recorded
rationale. Call count alone is not a promotion criterion.

Every promoted field gets an evidence row in the RFC or implementation docs:

IR field	consumer	frontend mappings	evidence
sampling.max_tokens	generation/scoring tasks	OpenAI max_tokens; sglang max_new_tokens	existing implementations/tests
scoring.input_logprobs	PPL/BPB tasks	OpenAI echo/prompt extension; sglang logprob controls	probe + existing tasks

Single-frontend exceptions in the target table (reasoning.effort, function
tools, and structured output) are retained because released legacy builders have
real callers and the frontend already implements them; each remains explicit
rather than seeding a larger speculative vocabulary.

4. Mechanically complete request gate

4.1 Exhaustive leaf schema

The gate operates on leaf patterns, not only top-level fields. The complete leaf
schema is derived recursively from the request dataclasses, and every frontend
explicitly lists every pattern it handles:

schema_leaf_patterns(Request) == frontend.request_contract.declared_leaf_patterns

This equality is a CI invariant. Adding a request field therefore breaks every
frontend contract test until the field is deliberately handled. Wildcards are
allowed only for sequence indices and dynamic map keys; a broad pattern such as
reasoning.* is forbidden because it would absorb future fields silently.

Active leaves are computed from the concrete request as follows:

• required fields such as input are always active;
• nested dataclasses are compared recursively to their own field defaults, so
  SamplingParams() activates nothing but SamplingParams(seed=1) activates
  sampling.seed;
• provider-neutral opaque values such as message input, tools, and structured
  output remain one leaf rather than being interpreted recursively;
• each extra_wire_params[key] is a separate active dynamic leaf, even when its
  value is None.

Dataclasses cannot distinguish an omitted value from a value explicitly supplied
equal to its default. This revision defines those cases as semantically
equivalent. Any future field that needs presence semantics must use an UNSET
sentinel.

This catches cases a top-level check misses: sampling.seed on native sglang,
reasoning.effort or structured output on completions, input scoring on chat,
and any dynamic extension that a frontend currently drops.

4.2 Four decisions and one per-call audit

Every active leaf resolves to exactly one of four decisions:

Decision	Meaning
Consumed	the value participates in wire data, endpoint/control flow, or a declared lift strategy
Rejected	this concrete value is unsupported and the request fails before I/O
NoOp	the IR explicitly defines this value as semantically equivalent for this frontend, with a stable rationale
Passthrough	a dynamic escape-hatch value is copied unchanged to a declared wire destination

Renaming or mapping a value, such as max_tokens -> max_new_tokens, is
Consumed, not passthrough. NoOp cannot mean "not implemented yet" and cannot
be used for a result-affecting field. Passthrough promises exact forwarding,
not that the remote engine will accept the extension.

The mechanism is deliberately small:

active = nondefault_leaf_paths(req)
audit = RequestAudit(active)
frontend.validate_request(req, audit)
audit.raise_rejections()
prepared = frontend.prepare(req, audit)
audit.finish(prepared)

validate_request() uses ordinary Python branches to record every known
Rejected decision, and raise_rejections() is a hard barrier before lower or
prepare. prepare() records Consumed, NoOp, and Passthrough decisions for
the remaining active leaves. There is no condition AST, relational
accepted region, subset prover, or theorem-style Unknown result in RFC #25.
Simple frontend-local value checks may use small OneOf, IntRange, or
Unsupported descriptors where data is clearer than code, but they do not form
a general predicate language.

finish() verifies that every active leaf was decided exactly once, every
Consumed leaf was actually claimed by prepare/lower, and every passthrough
value appears unchanged at its declared destination. An omitted or
double-accounted leaf is a FrontendContractError; a deliberate Rejected
decision is a user-facing unsupported-request error. All known rejections for
one request are aggregated before I/O.

A decision and a claim are different events: Consumed is the leaf's one
decision, while the later claim is evidence that lowering used it. Recording
the claim never counts as a second decision.

Each consumed field has a mutation or golden test that changes the leaf and
observes its declared wire/control/lift effect. The audit therefore checks the
implementation path rather than trusting a second handwritten capability
catalog.

Dynamic extras use only the strategies needed by retained frontends:

• forward every non-reserved key unchanged to one declared mapping destination;
• map a finite set of keys to explicit wire destinations and reject the rest;
• reject all dynamic keys.

Each concrete dynamic key is audited separately. No key-condition algebra is
introduced.

4.3 Call sequence and output validation

Model.arun() has one fixed order:

validate_request_invariants(req)  # provider-neutral ranges/shapes
active = nondefault_leaf_paths(req)
audit = RequestAudit(active)
frontend.validate_request(req, audit)  # deterministic frontend-local checks
audit.raise_rejections()  # hard barrier: unsupported values never reach lower
prepared = frontend.prepare(req, audit)  # pure lower/control preparation
audit.finish(prepared)
async with resources.acquire(local_limiter):
    response = await frontend.execute(prepared)
frontend.output_contract.validate(req, response)
return replace(response, provenance=self.provenance_snapshot())

execute() includes wire send, complete stream consumption, every internal tool
loop, and lift to a terminal Response; private lower/send/lift helpers remain
test seams only. The limiter is therefore held until all response work is done.
No limiter acquisition or network call occurs for invalid, rejected, or
unaccounted requests. Output postconditions run after execute, so a malformed
successful wire response is never handed to a task as if its required channel
were valid.

4.4 Passthrough rules

• A passthrough value, including None, must appear unchanged in the declared
  SDK/native wire destination.
• OpenAI-compatible engine extensions rejected as top-level SDK keywords travel
  through extra_body; using the escape hatch does not promote them into the IR.
• Unknown keys are rejected, not dropped, by frontends that cannot forward
  arbitrary fields.
• A passthrough key colliding with a first-class or reserved wire field is an
  error. Neither setdefault nor "first-class wins" may silently discard one
  of the caller's two conflicting intents.

5. Request support, output guarantees, and the RFC #47 boundary

The flat Capability(Flag) catalog is removed. RFC #25 keeps two small,
separate surfaces: the executable request audit above and explicit output
guarantees. It does not replace the catalog with a general constraint solver.

5.1 Request support

Concrete request support is decided by the frontend-local gate on every call.
Provider-neutral invariants run first; the frontend then checks its own known
values and combinations. Examples include the accepted reasoning-effort enum,
chat rejecting input scoring, and top_logprobs > 0 requiring
sampled_logprobs=True. These checks are deterministic from Request plus the
bound frontend and stay in RFC #25 as ordinary validation code.

Task setup needs only the small projection in Section 6: input modality and the
few scoring/output features used by current tasks. The projection is tested
against the executable request gate and output contract; it is not a second
vendor vocabulary and does not attempt to describe every possible Request.
Every concrete request, including one from a direct caller or activated only by
legacy model defaults, still passes through the exhaustive leaf audit.

The request half of that projection is explicit and finite:

@dataclass(frozen=True)
class TaskRequestSupport:
    input_kinds: frozenset[InputKind]
    input_scoring: bool = False
    sampled_logprobs: bool = False
    top_logprobs: bool = False

For every True feature, a canonical minimal Request must pass the pure
frontend validation/audit and visibly affect lowering; every False feature
must reject its canonical Request before I/O. Output requirements are not copied
here: setup reads their OutputRule directly. This projection contains only the
features present in TaskRequirements and grows only when a real task introduces
new coupling.

5.2 Output guarantees

Output guarantees are separate because request-field consumption cannot prove
that a response supplies token ids, reasoning text, citations, or another
channel.

class Guarantee(Enum):
    PRESENT_OR_ERROR = auto()
    BEST_EFFORT = auto()
    NEVER = auto()


@dataclass(frozen=True)
class OutputRule:
    guarantee: Guarantee
    validator: OutputValidator


@dataclass(frozen=True)
class OutputContract:
    rules: Mapping[ResponsePath, OutputRule]

    def validate(self, req: Request, response: Response) -> None: ...

Every frontend must classify every semantic response channel:

semantic_response_paths(Response) == frozenset(frontend.output_contract.rules)

This is the output-side CI analogue of request-leaf completeness. Adding a
Response channel breaks every incomplete frontend contract until it declares
PRESENT_OR_ERROR, BEST_EFFORT, or NEVER.

semantic_response_paths() returns the core and channel paths mechanically
derived from field metadata on the closed record schema, not from optional
typing. Every field reachable through a @sieval_record must be classified as
core, channel, payload, provenance, or structural; an unclassified new
root or nested field fails schema CI. Structural records are traversed, sequence
indices normalize to [*], and payload leaves are validated with their nearest
channel rather than becoming independent guarantees.

texts is core. Reasoning, tool/server-tool output, sampled and input logprobs,
top-logprob alternatives, citations, grounding, session state, usage, and finish
reasons are channels. request_params, response_model,
system_fingerprint, and provenance are provenance and are checked by Section
7 rather than by output availability. Currently emitted optional nested leaves
such as logprobs[*].token_id, top_logprobs[*][*].token_id, and
input_scoring.token_logprobs[*].token_id remain explicit channel paths so the
contracts describe actual lift behaviour rather than silently treating an
entire nested record as one guarantee. No production task consumes these token
ids. They are not a TaskRequirements feature and do not influence whether a
frontend is retained; their rules describe and validate optional output shape
only. A real consumer may promote them later.

Request-triggered postconditions are named, frontend-local validation code, not
a shared condition language. When scoring.input_logprobs=True, for example,
Response.input_scoring is either complete or the call fails; an empty or
partial channel is not success. PRESENT_OR_ERROR is mechanically checked after
lift. BEST_EFFORT validates the value when present but permits absence.
NEVER requires absence, catching a stale contract if a frontend begins
emitting the channel.

OutputContract.validate() iterates the rule mapping itself and dispatches each
rule's validator; it cannot maintain a second handwritten path list. Setup reads
the same rule's guarantee. Contract tests inject absent, malformed, and valid
values for every path under every applicable guarantee. The core texts path is
PRESENT_OR_ERROR for every retained frontend and is an implicit requirement of
all four model-calling task presets.

A task requiring a channel is accepted at setup only when the bound frontend
declares PRESENT_OR_ERROR for that feature. BEST_EFFORT and NEVER cannot
satisfy a strict task requirement. Runtime validation still runs for every
concrete request.

5.3 Boundary with RFC #47

The boundary is not "single field versus cross-field." It is whether validation
is deterministic from the request and frontend or requires engine/instance
serving facts.

RFC #25 owns:

• provider-neutral Request invariants and tagged-input validation;
• frontend-local supported values and request combinations;
• frontend wire branching and exhaustive leaf accounting;
• response-channel guarantees and per-call postconditions.

RFC #47 owns:

• engine/version/launch-flag facts and live instance probes;
• verified top-logprob breadth and other instance limits;
• prefix-cache state and conflicts such as input scoring versus an enabled
  cache;
• serving-axis task requirements and composition of those facts.

RFC #25 keeps Engine, Instance, and verify_instance() as ownership seams
but does not intersect engine/instance domains or prove relational request
regions. Until RFC #47, names such as prefix_cache and topk_breadth are not
members of TaskRequirements; attempting to configure them fails rather than
being accepted and ignored.

6. Task requirements and legacy model_type

Task.requires describes only compatibility facts used by production tasks in
this revision:

@dataclass(frozen=True)
class TaskRequirements:
    input: InputKind | None = None
    input_scoring: bool = False
    sampled_logprobs: bool = False
    top_logprobs: bool = False

top_logprobs=True requires sampled_logprobs=True; an invalid requirement
object fails at construction. This is a direct invariant of the current
response channels, not a general relation solver. Likewise, input=None
requires all feature flags to be false.

Task.__init__ calls model.validate_requirements(self.requires) and reports
all known unmet facts together. Request features are compared with
TaskRequestSupport; required response features must have a
PRESENT_OR_ERROR OutputRule. It does not wrap the model in a request-envelope
proxy. Every concrete request is protected by Section 4 regardless of what the
task declared; task under-declaration is a task-contract/migration test failure,
not a reason to add a runtime theorem system.

input=None is valid only for a task that makes no model calls. It is not a
wildcard modality and does not pass the migration scan for a model-calling task.
As of writing, the 39 registered model-calling tasks use four reusable
requirement presets:

• chat generation: chat input only;
• completion generation: completion input only;
• PPL/BPB: completion input plus input scoring;
• CLP: completion input plus sampled and top-logprob output.

For the migration only, the decorator accepts the existing model_type as input
and mechanically normalizes it to TaskRequirements.input. If a task also
declares an explicit modality requirement, the two must agree or import fails.
After normalization, Task.requires.input is the task's sole modality authority;
only scoring or special-output tasks need handwritten additions. This migration
and removal of runtime
isinstance(model, ChatModel | GenModel | SglangGenModel) checks land in the
same atomic change.

PR #59's shared derive_model_type() is updated to aggregate those normalized
requirements for every task pointing at a configured model. At initial/root
binding, its single result feeds both default frontend selection and #59's recipe
chat -> instruct/gen -> base compatibility mapping. The function does not
read Task.model_type as a second source after migration. The existing task
metadata model_type field remains in schema version 1 for metadata
compatibility, but is a derived projection of Task.requires.input. Replacing
that field, the public YAML type: chat | gen, or sieval/meta/index.json is a
separate RFC.

BrowseComp, SimpleQA Verified, HLE, and AA-LCR own separate grader models. Each
grader binding declares and validates the chat-generation preset independently;
grader calls use the same request gate and carry frontend/engine/instance
provenance through the normal ModelOutput.model path. Persisted grader-output
shape is owned by #60/#61, not by this RFC. The invariant here is that whenever
that protocol persists a complete grader ModelOutput, its nested
model.provenance survives. Merged #60 currently exercises that integration for
SimpleQA Verified at
feedback_result.rollouts[*].extra.grader_output.model.provenance; this exact
path describes #60's current schema rather than defining a new #25 schema.

Until #61 migrates the remaining LLM-judge group, HLE, BrowseComp, and AA-LCR
retain their legacy text-only grader feedback. After this revision their live
grader ModelOutput values carry provenance, but those three tasks persist no
ModelMeta and therefore no durable grader provenance. This revision neither
backfills those records nor solves the separate profiler aggregation follow-up
tracked by #61.

Legacy agenerate()/alogprobs() merge model defaults and per-call arguments
using the same precedence as today, construct the concrete tagged Request, and
then enter the Section 4 gate. A default-activated unsupported field therefore
fails before I/O, and an explicit per-call value overrides that default. Direct
arun(Request) never applies model defaults.

The tagged ModelInput in Section 3 makes modality part of each concrete
request; TaskRequirements is not used to guess what a runtime str means.
Task construction performs no asynchronous probes. Any future serving-fact
discovery belongs to session preflight and RFC #47.

7. Response and provenance

Response stays a single @sieval_record with optional record-valued channels.
New channels are additive. Existing fields are not renamed or retyped in this
revision. In particular, embedding/media support does not require subclasses.

Response gains an optional additive provenance field. The existing
ModelMeta carried by ModelOutput.model and ModelCallMeta.model gains the
same optional nested record:

@sieval_record
@dataclass(frozen=True)
class ModelProvenance:
    frontend_id: str
    engine_id: str
    engine_source: Literal["deployment", "config", "unknown"]
    engine_version: str | None = None
    instance_fingerprint: str | None = None
    verification_version: str | None = None


# Additive fields on the existing records/types:
Response.provenance: ModelProvenance | None = None
ModelMeta.provenance: NotRequired[ModelProvenance]

After output validation, Model.arun() attaches the immutable binding snapshot
to the terminal Response. The legacy bridge copies that snapshot into
ModelOutput.model.provenance; build_model_call_meta() already copies the
whole ModelMeta into ModelCallMeta.model, so there is no second handwritten
provenance copy that can drift. The existing stage-metadata serializer writes
that call meta when record_meta=True. Actual lowered request parameters
continue through the existing request_params path. Direct arun() callers and
legacy callers therefore see the same identity: two calls to the same
endpoint/model through different frontends have different frontend_id values.

For every newly executed call, frontend_id, engine_id, and engine_source
are present. An unlabelled remote engine is represented by the explicit
"unknown" values defined in Section 2. Optional version/fingerprint fields are
absent only when not known or not verified. On a legacy payload,
Response.provenance is None and ModelMeta.get("provenance") is None;
reporting renders the three identity axes as unknown without fabricating a
verification result.

Compatibility is explicit rather than inferred from the word "additive":

• legacy records without provenance remain readable under the None semantics
  above;
• a mixed-version resume preserves old records unchanged and writes provenance
  on newly executed calls, with each stage's existing producer version retained;
• record_meta=False omits automatic TaskStageMeta and its model_calls
  list. It does not strip provenance from a Response or ModelOutput stage
  result that is itself persisted. When the feat(core): uniform stage-output protocol, piloted on 7 tasks #60/[Feature]: finish the stage-output protocol migration — 32 remaining tasks #61 stage-output protocol
  persists a grader's ModelOutput, the same nested provenance survives without
  an additional feedback field;
• adding call provenance does not change feat(core): persist task identity in run meta.json #57's optional task block in run
  meta.json or sieval/meta/index.json. get_task_run_identity() remains the
  sole writer-side projection for that block; combined serialization and
  meta-drift tests must prove the two additive paths do not interfere;
• this RFC adds no resume bypass. Old-record, mixed-resume, nested round-trip,
  and record_meta=False tests must pass under the RFC [RFC]: Resume version-compatibility gate + per-record version provenance #24 resume rules.

8. Probe gates

The following probes determine which code paths survive. Static source evidence
is useful but does not replace a request against the deployed endpoint, which may
be an older version or sit behind a filtering proxy.

8.1 Sglang OpenAI completions: echo plus logprobs

For each run, generate a nonce-bearing prompt and send it once, then repeat that
exact prompt through the same instance or sticky route. The non-streaming body
is equivalent to:

{
  "model": "<served-model>",
  "prompt": "sieval probe <nonce> alpha beta gamma",
  "temperature": 0,
  "max_tokens": 1,
  "echo": true,
  "logprobs": 1
}

The streaming variant sets "stream": true and
"stream_options": {"include_usage": true}. Its assertions run on the fully
aggregated SSE response; usage is expected in a separate terminal chunk and is
not compared per chunk.

Success requires more than HTTP 200:

• echoed prompt text is present;
• prompt token logprobs are present and complete;
• completion token logprobs remain distinguishable;
• usage counts agree with the observed prompt boundary;
• streaming and non-streaming agree;
• joined prompt token text can locate the original nonce-bearing prompt,
  including a Qwen fixture with a leading-space option and newline;
• cache configuration and routing are recorded rather than inferred from two
  identical requests.

The blanket "sglang rejects it" claim is contradicted by static evidence. The
repository's Qwen recipes support SGLang from v0.4.6.post1; that version sets
logprob_start_len=0 for echo + logprobs, merges input logprobs into the
OpenAI response, and tests the combination across text/token-id inputs and
streaming/non-streaming calls. The local SGLang checkout at commit
dce8b0606c06d3a191a24c7b8cbe8e238ab316c9 does the same. A nearby warning is
not a rejection. This settles the source-level claim but is not a deployed
instance result.

The target-instance probe sends the same nonce-bearing prompt twice for each
stream mode. In addition to the parity checks above, each aggregated response
must have:

• usage.prompt_tokens > 1;
• equal lengths for tokens, token_logprobs, and top_logprobs;
• a prompt slice exactly usage.prompt_tokens long and a completion suffix
  exactly usage.completion_tokens long;
• numeric prompt logprobs except for the first token, which may be null.

If caching is enabled, cached_tokens, a cache report, or server metrics on a
single/sticky instance must prove a hit and the repeated response must retain the
full prompt slice. If caching is disabled, the probe records that fact and marks
the cache-hit case N/A.

The deployed launch is also probed with max_tokens=0 and inspected for
--skip-tokenizer-init and prefix/radix-cache behaviour. SGLang's static tests
run with a tokenizer and do not establish skip-tokenizer behaviour.

Decision:

• remove the native /generate frontend only if separate A/B coverage proves
  its complete currently consumed request/output contract through OpenAI
  completions,
  including sampling extensions, stop ids, n, finish/usage, token text, token
  logprobs, max_tokens=0, and tokenizer/cache launch modes, and no supported
  public requirement is silently weakened;
• otherwise retain /generate as an explicit frontend and record the exact
  failing requirement.

The native /generate frontend currently populates token_id while OpenAI
completions does not. No production task consumes this difference. It remains
documented and tested as observed optional output, but it is not a deletion
gate. A future real token-id consumer may promote it into TaskRequirements
and reopen the frontend decision.

8.2 vLLM prompt logprobs

Send a raw OpenAI completions request with the engine extension in the body:

{
  "model": "<served-model>",
  "prompt": "sieval probe <nonce> alpha beta gamma",
  "temperature": 0,
  "max_tokens": 1,
  "prompt_logprobs": 0,
  "stream": false
}

When using the OpenAI SDK, the extension must be placed in extra_body.

The independent request/response field first appears in vLLM v0.5.5; it is
absent in v0.5.4. The repository has no global vLLM minimum, although its
Qwen2.5/Qwen3 recipes require >=0.9.0, so those recipe paths have static
coverage. The target instance and any proxy must still be probed.

The same nonce-bearing prompt is sent twice through a single/sticky instance.
Success requires an independent non-empty
choices[0].prompt_logprobs array without echo=true, with length equal to
usage.prompt_tokens. Except for a nullable first item, each position must have
exactly one chosen-token entry shaped as
{token_id: {logprob, rank, decoded_token}}; logprob is numeric,
decoded_token is usable text, and joining decoded tokens locates the original
prompt. If caching is enabled, server/cache evidence on a single/sticky instance
must prove a hit and the repeated request must remain complete. If caching is
disabled, that fact is recorded and the cache-hit case is N/A.

This independent channel is non-streaming only in the supported vLLM response
schema: streaming choices do not carry prompt_logprobs. For input scoring, a
requested scheduling.stream=True is therefore a documented scheduling no-op
and the frontend executes non-streaming. It must not silently fall back to the
old echo split.

Decision:

• if both requests are complete, lift the independent prompt channel and remove
  the non-streaming vLLM echo split;
• if any request explicitly rejects a prefix-cache conflict, expose it as a
  conditional instance constraint for RFC [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47;
• if the endpoint returns success with a missing or shortened channel, reject
  the feature as silently unsafe;
• if absent, retain echo only with strict boundary validation and fail when usage
  is absent or inconsistent.

8.3 Evidence record

Each executed probe records:

• engine and version;
• launch flags relevant to the result;
• raw request with credentials/endpoints redacted;
• response-shape assertions and pass/fail result;
• date and source commit of the probe;
• server image digest and cache status; when caching is enabled, sticky-routing
  and cache-hit evidence.

The missing untracked 2026-07-01-recipe-capability-layer-design.md must either be
recovered and merged into this revision or replaced by re-runnable evidence. Its
claims are not treated as repository evidence while the file remains unavailable.

9. Conditional bug fixes and compatibility notes

PR #45 already changes one public legacy detail that must be called out in its
summary and generated changelog: during input scoring, IR
Response.top_logprobs and bridged ModelOutput.top_logprobs contain
completion-side alternatives only. The former echo layout exposed prompt plus
completion top-k entries. No current PPL/CLP task consumes prompt-side top-k, but
direct callers may. The bridge also collapses a present-but-empty IR tuple to
legacy None. Tests pin both behaviours; they are not described as invisible
wire-preserving refactors.

These fixes apply only if their code paths survive the probes:

• echo splitting rejects absent usage and boundaries outside the returned token
  range instead of defaulting to zero;
• native sglang clamps max_new_tokens for every logprob request, including
  echo=False, max_tokens=0;
• OpenAI-compatible engine extensions such as top_k and stop_token_ids travel
  through extra_body, not unsupported top-level SDK keyword arguments;
• unknown native sglang passthrough fields raise instead of being dropped.

10. Migration sequence

1. Rebase onto the main branch containing fix(scripts): enforce relative-import scope in check_layer_imports #46 and merged feat(core): uniform stage-output protocol, piloted on 7 tasks #60, plus feat(recipes): split recipe params into hardware and capability layers #59's final
   resolver, or explicitly stack behind feat(recipes): split recipe params into hardware and capability layers #59 while it remains open. Coordinate
   feat(core): persist task identity in run meta.json #57's additive meta.json change without copying its identity projection.
2. Recover/replace the missing backend-behaviour spec and run both deployed
   probes.
3. Agree on Section 2 and the probe-driven frontend set.
4. Introduce Engine, Instance, ResourcePool, and frontend factories without
   changing wire behaviour; retain deprecated names as re-exports.
5. Normalize engine resolution and upgrade endpoint injection to carry the
   immutable instance binding before any frontend is bound.
6. Move client ownership and the shared limiter into ResourcePool; add
   identity, drain, and exactly-once close tests.
7. Convert the retained transports to bound frontends without exposing the
   cross-frontend derivation API yet.
8. Introduce the exhaustive per-call leaf audit and complete executable output
   contracts; do not add the deferred condition/region proof layer.
9. In one atomic batch, synthesize modality requirements, update feat(recipes): split recipe params into hardware and capability layers #59's shared
   derive_model_type() to consume them, and change explicit type from an
   override to a checked assertion with updated tests and error guidance. Add the
   PPL/CLP presets and four grader bindings, remove internal class-kind checks,
   and expose with_frontend() plus the deprecated as_type() adapter if
   retained. Do not change grader-feedback persistence owned by feat(core): uniform stage-output protocol, piloted on 7 tasks #60/[Feature]: finish the stage-output protocol migration — 32 remaining tasks #61.
10. Add the explicit model-call provenance path and old-record/mixed-resume
    compatibility tests.
11. If native sglang survives, add the explicit frontend selector and legacy
    configuration shim described in Section 2.
12. Delete probe-obsoleted frontends/workarounds and apply the conditional fixes.
13. Run the complete unit/integration, type, import, serialization, resume, and
    wire-parity suites.
14. Implement RFC [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 on the resulting frontend/engine/instance seams; [RFC]: Observed-serving facts must flow back into dataset packing — the RULER case #58 then
    consumes immutable fact projections without changing the verification
    signature.

11. Acceptance criteria

• schema_leaf_patterns(Request) exactly equals every frontend's declared leaf
  patterns; adding a leaf fails all incomplete frontend contracts in CI.
• A parameterized contract test activates every request leaf for every frontend
  and observes consume, explicit reject, documented no-op, or exact passthrough.
• Every frontend-local enum/range boundary and request combination used by the
  implementation is exercised without a generic condition algebra.
• Dynamic-map tests cover an allowed key, an unknown key, a reserved-key
  collision, and a None value for each retained extra strategy.
• Every consumed leaf is claimed by lowering and has an observer showing that
  changing it changes the declared wire/control/lift effect.
• Every NoOp has a non-empty reason and an equivalence test.
• Passthrough assertions inspect the actual mock SDK/native request body, not
  only the prepared intermediate mapping.
• Rejected requests do not enter prepare/lower; rejected, unaccounted, and
  double-accounted requests make zero client calls. Passthrough collisions with
  first-class or reserved wire keys fail before I/O.
• Provider-neutral invariant failures such as n < 1 occur before frontend
  validation and before I/O.
• Legacy builders apply model defaults before per-call overrides, then gate the
  resulting concrete Request. A default-only unsupported leaf fails with zero
  I/O; explicit arun(Request) is unaffected by defaults.
• Native sglang's extra_wire_params["prefill"] produces exactly the former
  Request(prefix=...) wire body; unsupported frontends reject it.
• A plain bound/swapped Model uses its frontend's legacy-input adapter. A
  frontend without an adapter rejects legacy sugar and still accepts explicit
  tagged arun(Request).
• A frontend swap returns a plain Model, shares the expected pool/connection
  and limiter identities, and fails before binding when the instance does not
  expose the frontend or the engine rejects it.
• A connection-family mismatch raises FrontendBindingError without allocating
  or closing a connection or changing pool ownership.
• Pool close is idempotent, drains entered calls, and closes each connection once.
• Legacy root constructors own exactly one private pool; every derived/swapped
  binding retains the lifecycle owner, and closing through any binding delegates
  to it and invalidates all siblings.
• If retained, as_type() accepts only declared compatibility wrappers,
  reconstructs a truthful wrapper without __class__ mutation, and preserves
  pool, instance, limiters, defaults, metadata, and lifecycle owner.
• No production task relies on model-class isinstance for compatibility.
• feat(recipes): split recipe params into hardware and capability layers #59's derive_model_type() is the only configured-model kind resolver after
  task migration. It reads normalized Task.requires.input, rejects an explicit
  YAML type that disagrees, and supplies one result to both default frontend
  selection and recipe capability selection at initial/root binding; no consumer
  rereads Task.model_type independently and a derived model does not re-resolve
  its recipe.
• An explicit frontend whose input modality disagrees with the resolved legacy
  type fails before launch even when no task points at the model. The same check
  also reports all task-modality conflicts together.
• If native sglang survives, an explicit model engine: sglang plus a resolved
  gen type and no frontend exercises the deprecated native-frontend shim even
  when YAML omits type. An engine learned only from a managed deployment does
  not exercise the shim.
• A registry golden requires the live task registry to equal
  expected_preset_by_task.keys(), so every newly registered task must be
  classified explicitly. As of writing, its diagnostic baseline is 39 tasks:
  24 chat-generation, 6 completion-generation, 3 PPL/input-scoring, and 6 CLP,
  with exact scoring flags. Only PPL/BPB, CLP, and future special-output tasks
  require manual additions beyond synthesized modality.
• Every TaskRequestSupport=True feature has a canonical Request that passes the
  executable gate and changes lowering; every False feature rejects its
  canonical Request with zero I/O.
• input=None remains valid for a task with no model calls but fails the
  migration scan for any model-calling task.
• BrowseComp, SimpleQA Verified, HLE, and AA-LCR validate their grader bindings
  separately. Disk rehydration preserves every complete grader ModelOutput
  persisted by the stage-output protocol and its nested model.provenance; a
  path that makes no grader call has no grader output. An integration fixture for
  merged feat(core): uniform stage-output protocol, piloted on 7 tasks #60 checks SimpleQA Verified's current
  feedback_result.rollouts[*].extra.grader_output.model.provenance path without
  making that feat(core): uniform stage-output protocol, piloted on 7 tasks #60-owned location a new [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25 schema. Until [Feature]: finish the stage-output protocol migration — 32 remaining tasks #61 lands, no acceptance
  test claims durable grader provenance for HLE, BrowseComp, or AA-LCR. This RFC
  adds no grader-feedback field.
• semantic_response_paths(Response) exactly equals the keys of every
  frontend's output rules; adding an unclassified reachable schema field or
  response channel fails all incomplete contracts. Checks for optional nested
  token-id paths validate observed output shape only; they do not create a task
  requirement or a frontend-retention criterion.
• Parameterized output-contract tests inject absent, malformed, and valid values
  for every guarantee path and exercise each PRESENT_OR_ERROR, BEST_EFFORT,
  and NEVER branch. Every retained frontend declares core texts as
  PRESENT_OR_ERROR.
• A task-required output fails setup when its guarantee is BEST_EFFORT or
  NEVER. A malformed successful response violates PRESENT_OR_ERROR after
  lift and never reaches the task.
• verify_instance() returns frozen InstanceFacts, never a boolean verdict.
  Engine-owned interpretation and frontend/ProbeConnection execution produce
  one fact set that [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 can constrain and [RFC]: Observed-serving facts must flow back into dataset packing — the RULER case #58 can project without a second probe
  policy.
• Serving-axis requirements cannot be written in Task.requires before RFC [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47.
• Every new direct Response.provenance and bridged
  ModelOutput.model.provenance contains frontend_id, engine_id, and
  engine_source; a frontend swap changes the first value even at the same
  endpoint/model. Conflicting explicit engine identity fails before launch.
• Legacy records satisfy Response.provenance is None or
  ModelMeta.get("provenance") is None; mixed-version resume retains per-stage
  producer versions. With record_meta=False, a
  persisted Response/ModelOutput result retains provenance, no automatic
  stage ModelCallMeta is written, and any full grader ModelOutput already
  persisted by the stage-output protocol retains nested provenance. Nested
  record round-trip and meta-drift tests pass.
• feat(core): persist task identity in run meta.json #57's run meta.json task block still comes only from
  get_task_run_identity(). The requirements registry golden is not used to
  serialize run identity, and combined old/new run-meta fixtures pass.
• Response, ModelOutput, ModelMeta, and their nested records serialize and
  deserialize with old optional fields absent.
• Input-scoring top_logprobs is explicitly tested as completion-only, and the
  legacy bridge's empty-to-None conversion is documented in PR feat(models): rebuild the provider-neutral model binding plane (RFC #25) #45's summary
  and changelog, plus the public echo=True API compatibility test.
• Existing four-path sglang/vLLM echo on/off parity tests remain green for all
  retained paths.
• Import-policy, type-check, unit, integration, resume, serialization, and
  meta-drift checks pass.

12. Decisions requested from the reviewer

1. Confirm ResourcePool + FrontendFactory + Frontend + Instance + Engine
   ownership, especially that Model no longer constructs or closes clients.
2. Confirm thin compatibility classes for one cycle rather than immediate
   function/partial aliases.
3. Confirm deployment-plan engine normalization/conflict rules and unknown for
   unlabelled remote engines.
4. Confirm exact per-frontend leaf declarations plus the minimal per-call audit,
   rather than either a top-level check or a condition/region proof system.
5. Confirm the small task requirement projection and complete output guarantees,
   with every concrete Request still protected independently by the leaf gate.
6. Decide the native sglang frontend only after the deployed probe matrix.
7. Confirm the explicit frontend/engine/instance provenance write path and its
   old-record, mixed-resume, and record_meta=False behaviour.
8. Confirm that a canonical frontend: key must land here if native sglang
   survives, together with the narrowly scoped legacy config shim.
9. Confirm derived concurrency is a local cap, not a hard quota reservation.
10. Confirm the normative Request field table, tagged modality, and demotions in
    Section 3.
11. Confirm the request/frontend-local versus engine/instance/probed RFC [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25/[RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47
    boundary, and defer a generalized condition algebra until demonstrated
    coupling requires it.
12. Decide whether as_type() preserves its wrapper runtime type for one cycle
    or is removed rather than returning a plain Model.
13. Confirm feat(recipes): split recipe params into hardware and capability layers #59's derive_model_type() as the sole compatibility resolver fed
    by normalized task modality requirements, with its result shared by default
    frontend and recipe-capability selection only at initial/root binding.
14. Confirm that merged feat(core): uniform stage-output protocol, piloted on 7 tasks #60 owns the current grader-output location, [Feature]: finish the stage-output protocol migration — 32 remaining tasks #61 owns
    the remaining task migrations, and this revision only requires persisted
    grader ModelOutput values to retain nested ModelMeta.provenance.
15. Confirm that Engine owns instance-probe rules and fact interpretation,
    frontend/ProbeConnection supplies execution, and the immutable fact result
    remains projectable by [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47/[RFC]: Observed-serving facts must flow back into dataset packing — the RULER case #58.

Non-blocking open questions

These scope and implementation-shape questions do not change the Section 2
agreement or the no-silent-drop gate:

1. Decide whether output completeness needs the recursive five-way
   core/channel/payload/provenance/structural classification in this
   revision, or whether root-level channel versus non-channel classification
   plus frontend guarantees is sufficient until a second concrete nested
   coupling appears.
2. Decide whether the repeated engine-origin literal should become one shared
   EngineSource = Literal["deployment", "config", "unknown"] type alias.
3. Confirm that numeric task counts are an as-of-writing diagnostic baseline,
   while the durable golden invariant is
   registry == expected_preset_by_task.keys() and therefore tracks explicit
   classification of future tasks without freezing the count at 39.

Feedback Period

At least one week, with explicit agreement on Section 2 before implementation.

CC List

PR #45 reviewer and #25/#27/#47/#49/#58/#59/#60/#61 authors.

Any Other Things

This revision intentionally chooses strict rejection over silent downgrade. It
does not introduce bypass flags or permissive unknown-engine behaviour.

[ethan-scitix]

TL;DR

• The target this RFC should be measured against, which my PR feat(models): rebuild the provider-neutral model binding plane (RFC #25) #45 review failed to state: seven wire protocols behind one IR, such that adding a model is declare its protocol, declare which capabilities to enable — nothing else. The current IR covers two and a half of the seven.
• Accept §2, §7, §8. The class-structure rebuild is real work and lands the right object graph; provenance is clean and additive; the source evidence in §8 is the most valuable thing in the document and may delete a whole code path.
• Reject §3.1 and the catalog removal in §5. §3.1 prunes on a consumer-driven criterion — is there code reading this today? — which is right for internal plumbing and wrong for a declaration surface, where the criterion is is this a name a user must be able to write? Together with §5 removing the capability vocabulary, the two turn the declaration surface into hand-written per-dialect code.
• The missing paradigm is declare + reconcile. SiEval's contract is that one YAML does everything, for the CLI and the programmatic interface alike. That requires a plan-time reconcile step comparing declared capabilities against every axis, emitting launch flags and remedies before anything is spun up. --dry-run, cli/validation.py and the word "reconcile" appear zero times in this revision; plan-time appears once, delegated wholesale to [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47.
• Your own Motivation already derives the fix. The three concerns it says the flat Capability set flattens are three columns of one binding table. Keep the symbol, split the columns — and add the fourth, the model axis, which is present today as a launch-flag source and absent as a capability authority.
• Naming: Frontend was my error in the PR feat(models): rebuild the provider-neutral model binding plane (RFC #25) #45 review. Dialect / Deployment / ServingFacts / ConnectionPool / reconcile(), and reasoning rather than thinking as the capability name. Details in section 4.
• §12: reject 10 and 11, accept 1/13/15, and 8 unconditionally — dialect: must not be contingent on whether native sglang survives the probes. The two probe gates still decide native sglang's fate before anything else is settled.

---

Thanks — this is a serious piece of work, and the parts of it I disagree with are worth arguing about rather than waving through. Verdict first, then the reasoning.

Accept: the class-structure rebuild (§2), provenance (§7, with one addition), and the source-evidence work (§8). §2 fixes the two structural defects your Motivation identifies — SglangGenModel binding an engine name to a single wire protocol, and the loss of cross-frontend client/limiter sharing when as_type went away — and the object graph it lands on is your contribution, not a restatement of my review. §8 is the most valuable section in the document.

Reject: the §3.1 demotion table and the removal of the capability catalog in §5. Both follow from a premise this RFC should not be built on, and together they convert the declaration surface into hand-written per-frontend code.

Missing: the paradigm the rest of SiEval is built on. That is the substance of this comment.

---

1. The goal statement this RFC should be measured against

My PR #45 review set the direction but not the target, and that is my omission, so let me state it plainly now. The target is:

Support seven wire protocols — openai_chat, openai_completions, openai_responses, anthropic_messages, google_genai, vllm_native, sglang_native — behind one IR, such that adding a model is: declare its protocol, declare which capabilities to enable. Nothing else.

That target is what makes §3.1 the wrong move. The demotion table removes reasoning axes other than effort, server_tools, session_id and the opaque round-trip, on the stated grounds that they are "documentation vocabulary, not implemented coupling." That criterion is consumer-driven: is there code reading this field today? It is the right criterion for internal plumbing and the wrong one for a declaration surface, where the criterion is: is this a name a user must be able to write? reasoning.budget_tokens has no consumer today because the anthropic_messages dialect does not exist yet. It will be user-written on the day it does.

Two of the seven protocols are covered by the current IR, plus half of a third. Pruning to what today's two-and-a-half consume optimizes for the state we are trying to leave.

2. The paradigm: declare + reconcile

SiEval's usability contract is that one YAML does everything, for both the CLI and the programmatic interface. That is inherently declarative, and it implies a reconcile step: the user states desired capabilities, the system compares them against what is actually available across every axis, and either converges (deriving request defaults and launch flags) or fails with a remedy.

This is not an aspiration. Two instances ship today, and the third is the one whose absence this revision is quietly working around:

1. recipes/*.yaml → resolve_profile() → launch params. Declared profile in, actual launch command out. Note what it currently declares, though: reasoning_parser: qwen3 appears nine times in qwen3.yaml, once per size profile, with no base/instruct distinction — so a base checkpoint of the same family and size resolves to a profile that hands it a reasoning parser it cannot use. feat(recipes): split recipe params into hardware and capability layers #59 fixes the schema. Nothing yet connects the fixed schema to a declared capability, which is the other half.
2. cli/validation.py surfaced by sieval eval --dry-run. Seven _validate_* functions and a real run_dry_run(). This is a working plan-time gate that fails before anything is spun up, and it is where a _validate_capabilities belongs.
3. The eval-to-serving requirements channel: designed, never built. Zero files under sieval/ implement it. Meanwhile those launch flags are named across nine task files as instructions to a human operator — twenty-five literal flag occurrences, twenty-eight counting the prefix-cache prose carrying the same instruction — with no code reading any of them. That is the actual state: the plan-time reconcile step for exactly this problem was specified and dropped, and §4 now proposes 136 lines of request-time machinery in the space where it was supposed to go.

This revision engages with none of the three as a paradigm. Searching the text: YAML appears five times, every one of them the legacy type: chat | gen key as a compatibility burden to be honored, never as the declaration surface. --dry-run, cli/validation.py, _validate_models and the word "reconcile" appear zero times. plan-time appears exactly once — in §1.1's ownership table, as something delegated wholesale to #47. So plan time is not absent from the document; it is present as somebody else's problem, which is why no checkpoint in this revision runs before a deployment exists.

The recipe capability layer is the near-miss. §2.4, §6 and decision 13 do wire derive_model_type() into recipe capability selection, so the connection exists in one direction: model type decides which profile is used to launch. It does not exist in the other: the profile's contents — reasoning_parser: qwen3 under instruct versus an empty base — are never consulted to decide whether a declared capability is permissible. The model axis is present as a launch-flag source and absent as a capability authority, and that asymmetry is exactly what lets "enable reasoning on a base checkpoint" through.

Why the catalog removal in §5 forecloses this structurally, rather than merely differing in style:

• Desired state must be serializable — it lives in YAML and must land in the run record as reproducibility evidence. An if branch inside a dialect cannot be serialized.
• Desired and actual must be comparable by name. You need declared.reasoning to line up against available.reasoning. Replacing the catalog with per-dialect validation code means there is no available.reasoning to compare against.
• The output of reconciliation is a diff plus a remedy, not an exception. --max-logprobs 100 and --disable-radix-cache are remedies that must be applied at launch, so they must be computed before launch.
• Reconciliation runs once, at plan time. §4's gate runs per call, after the deployment is already up. Item 3 of section 5 below argues that a request-time check is genuinely needed — but as a projection of the declaration, not as the only checkpoint and not as separately hand-written per-dialect code.

3. The binding table your own Motivation derives

§Motivation states that the flat Capability set flattens three concerns:

1. whether a frontend consumes a request field
2. what a frontend guarantees in the response
3. what an engine and a running instance can safely provide

That diagnosis is correct, and those three concerns are three columns of one table. The remedy that follows is to split one column into three while keeping the symbol, not to delete the symbol and scatter the columns into per-dialect code.

A fourth column is missing, and it is the one that makes the seven-protocol target tractable:

Column	Answers	Owner
request_leaves	which IR leaves the dialect must consume	#25 (§4's gate already computes this)
response_channels	which channels the dialect must guarantee	#25 (§5.2's OutputRule)
model_requirements	what this checkpoint/hosted model must support	recipes' capability layer (#27/#59)
serving_requirements	launch flags and probed facts	#47

Column 3 is where base: {vllm: {}, sglang: {}} versus instruct: {vllm: {reasoning_parser: qwen3, ...}} lives. Base and instruct are different checkpoints served differently, which is why "which capabilities" splits along model type — that is a first-class axis, not a detail of the recipe file. Enabling reasoning on a base checkpoint must fail at plan time by reading that layer, and per §2 above the wiring for that read is half-built already: the profile is selected but never interrogated. As it stands the capability catalog and the recipe capability layer are two systems answering the same user question, only one of which the user can see.

Concretely, the shape the user writes:

models:
  claude-opus-4:
    dialect: anthropic_messages
    capabilities:
      reasoning:
        budget_tokens: 8192
      web_search: true

  qwen3-32b:
    dialect: openai_chat
    engine: sglang
    capabilities:
      reasoning: {}

dict[str, CapabilityConfig | bool | None], deliberately a mapping rather than a list of heterogeneous items: duplicates are caught by the load_yaml_with_duplicate_check() that config loading already runs, reasoning: false can express an explicit disable (which is what ReasoningAlwaysOn was reaching for), and there is no - name versus - name: ambiguity where one spelling parses to a string and the other to a single-key mapping. Values carry request defaults and capability options only — never launch flags, which are derived rather than written.

And the error that reconciliation produces, in the user's vocabulary rather than in leaf paths:

model 'qwen3-32b': capability 'reasoning' cannot be enabled
  dialect  openai_chat      can express it (reasoning.effort)   ok
  model    qwen3 / base     no reasoning_parser                 fail
  remedy   use the instruct checkpoint, or remove 'reasoning'

Four axes resolve internally; one name is what the user wrote; the remedy is actionable before anything launches.

One consequence worth stating: a Flag enum cannot carry budget_tokens: 8192. Capability(Flag) was the wrong container from the start, so the conclusion is a keyed capability registry with per-capability config — not deletion.

4. Naming

I named Frontend in my PR #45 review and that was a mistake; it sent the design in a direction where two weeks later the object graph no longer reads as what it is. Proposed corrections:

Current	Proposed	Why
Transport / Frontend	Dialect	Lowering IR to wire is a compiler backend; "frontend" is directionally backwards and reads as UI. Dialect carries exactly the intended meaning — one IR, per-target spelling plus per-target capability declarations — and is already the word used in the config above.
FrontendFactory.bind()	bind_dialect()	A factory class with a single method is indirection without a second implementation.
Instance	Deployment	DeploymentPlan is already the name for this in the infer layer. Two words for one concept.
instance.facts	ServingFacts	Names the actual side of the loop, to be read against a ServingRequirements on the declared side: reconciliation is literally requirements against facts, and the pairing makes a missing half visible.
ResourcePool	ConnectionPool	It owns connections and the shared limiter. "Resource" says nothing.
service_contract = f(frontend, engine, instance)	reconcile(declared, dialect, model, engine, serving_facts)	Not a contract — the result of comparing declared against actual. Naming it reconcile is what keeps the plan-time checkpoint from being forgotten.
Engine	keep	Already the codebase term for the serving framework.
ModelProvenance.frontend_id	dialect_id	Follows the rename; the persisted record is the worst place to leave a name we have agreed is wrong, since it is the one field a rename cannot quietly fix later.

On the capability's own spelling: reasoning, not thinking, even though thinking is what I wrote in my own config sketch. Both engines' launch flag is --reasoning-parser; OpenAI and the self-hosted stack both spell it reasoning-ish; only Anthropic says "extended thinking". Absorbing that difference is precisely what a dialect is for, so it must not leak into the symbol. One spelling across the YAML key, the IR field group, and the error text.

5. Serving behaviour that belongs in this RFC's body

These are the constraints the seven-protocol target has to survive. Stating them with their mechanism, because the mechanism is what makes them re-testable:

• sglang radix cache (on by default): on a prefix-cache hit it truncates input_token_logprobs to prompt_tokens − cached_tokens and does not recompute cached positions — a silently partial, non-reproducible token set. Remedy --disable-radix-cache.
• vLLM V1: prompt_logprobs/echo is mutually exclusive with automatic prefix caching, and the request errors rather than truncating. Remedy --no-enable-prefix-caching.
• Top-k breadth: vLLM defaults to --max-logprobs 20, so A/B/C/D are not guaranteed to appear; sglang serves 100 out of the box. Remedy --max-logprobs 100.
• The trigger is "reads input logprobs", not the task's eval mode. Output-only logprob tasks are unaffected and actually benefit from the cache. So the constraint keys off a request leaf — scoring.input_logprobs — which is exactly the granularity §4's leaf gate operates at.

Three things follow that neither this revision nor #47 currently specifies:

1. Aggregation semantics. Many tasks share one deployment, so requirements must combine: boolean needs are OR'd, numeric floors take the max, across every task resolving to the same served base model. Without this, the fourth column has no defined value for a multi-task run.
2. Precedence against explicit user config. When a user's engine params already set a knob that a requirement would derive, one of them has to win, and emitting a contradictory --enable-…/--no-enable-… pair is not an option. "Trust the user and suppress the derived flag" is the natural answer and it directly contradicts [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47's second red line — no silent flip of a requested capability, hard-fail with a remedy instead. My call: user-explicit wins on performance knobs, hard-fails on correctness knobs, where prefix-cache-versus-input-logprobs is a correctness knob because its failure mode is silently wrong scores rather than a slow run. This has to be settled in one place; two subsystems resolving it differently is how the current prose-only state came about.
3. Reconciliation completeness depends on the entry point. sieval run owns the launch, so it can derive flags. sieval eval against a pre-launched endpoint and sieval infer start without task context cannot — there is no launch to influence. This is the honest justification for a request-time check, and it sharpens what §4 should be: the last line of defense for the entry points where plan-time has no leverage, derived from the same declaration, not a separately hand-written per-dialect code path. As written, §4 is 136 lines of machinery doing at request time what belongs at plan time, and my review asked for roughly ten lines of structural gate for the residual case.

6. §12

• 1 — agree on ownership, subject to the renames in §4 above, and with Model no longer constructing or closing clients.
• 4 — the per-dialect leaf declarations and the per-call audit are right in kind; scope them to the residual role described in item 3 of section 5 above, rather than making them the primary validation boundary.
• 5 — cannot confirm as written. The task projection is fine; the removal of the capability catalog it sits on is not.
• 8 — dialect: must land in this RFC unconditionally. Making it contingent on whether native sglang survives the probes couples a user-facing config key to an implementation-deletion decision. Seven protocols need the key regardless.
• 10 — reject the demotion table, for the reason in §1.
• 11 — reject the scope transfer. §5.3 leaves prefix_cache and topk_breadth outside the vocabulary until [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47, which means [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 must define the symbol space itself. [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 explicitly depends on not having to. The symbol space is user-facing, so it belongs where models are declared, not where constraints are declared. Keeping the vocabulary here and the constraints in [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 is the correct split — and §5.3's rule that configuring an unenforced name must fail rather than be accepted-and-ignored is right and should stay.
• 13 — agree.
• 15 — agree, and this is an improvement on [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 as filed: Engine owning probe rules and fact interpretation is more correct than the frontend owning them, because four of five known constraints are engine-level. I will update [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 accordingly.
• Non-blocking 1 — root-level channel classification is sufficient; defer the recursive five-way classification until a second concrete nested coupling exists.
• Non-blocking 3 — agree; the durable invariant is the registry equality, not the count.

7. Sequencing

1. Fold §5's serving behaviour and the three unspecified items into the RFC body, with each constraint carrying a re-runnable probe rather than prose.
2. Land §2 with the renames, plus §7 provenance.
3. Restore the capability symbol space as the four-column binding table, with the declarative config surface and a plan-time _validate_capabilities reconcile step beside the existing _validate_models / _validate_tasks.
4. Run the two probe gates in §8; only then decide the native sglang dialect's fate.
5. [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 supplies column 4's constraints and probes on top, without changing the vocabulary.

8. One addition to §7

Hosted-API identity is incomplete. engine_source is permanently unknown for anthropic_messages, google_genai and the OpenAI dialects, so provenance for hosted models degenerates to frontend_id plus a model name. The fact that actually identifies a hosted call is the snapshot the vendor served, and it currently lands in Response.response_model with per-dialect semantics that nothing declares — the sglang path records the requested model name while the OpenAI path records the served snapshot. One identity record, one meaning, or an explicit OutputRule stating the difference.

[jack-scitix-ai]

Thanks — I agree with the distinction between a consumer-driven internal schema and a user-facing declaration surface. I optimized the revision around today’s consumers, but that is the wrong boundary if the acceptance target is seven dialects behind one IR.

Before I draft revision 2, one scope decision directly determines its size and acceptance criteria:

Should this implementation land all seven dialect adapters — openai_chat, openai_completions, openai_responses, anthropic_messages, google_genai, vllm_native, and sglang_native — or should #25 establish the complete declaration, binding, and reconciliation model while the adapters are implemented in stages?

I will design the symbol space against all seven either way, but those two implementation scopes are materially different.

My reading of the rest of the review is:

• Keep the substance of §2: Model does not construct or close clients; dialect, engine, running deployment, and connection ownership remain separate axes; derived bindings share the same connection and limiter hierarchy.
• Adopt Dialect, ConnectionPool, ServingFacts, dialect_id, reasoning, and bind_dialect() rather than retaining a single-method factory abstraction.
• Keep §7, adding a consistent hosted-model identity contract instead of giving response_model different meanings across dialects.
• Keep §8 and run both probes before deciding whether the native SGLang dialect survives.
• Rewrite §3–§6 around a keyed, serializable capability registry and plan-time declare + reconcile.
• Add dialect: unconditionally.
• Capability configuration carries behaviour defaults and options, never raw launch flags; reconciliation derives launch parameters and remedies.
• Retain the per-call leaf audit as a projection of the reconciled declaration and the residual last line of defence, not the primary capability boundary.
• Keep the small TaskRequirements projection, aggregate requirements per root deployment, and define boolean OR and numeric-floor maximum semantics. Task requirements contribute automatically; users should not have to repeat them in models.<name>.capabilities.
• Define explicit-configuration precedence in one place, including the proposed split: performance knobs defer to explicit user configuration, while correctness conflicts hard-fail with an actionable remedy.
• Keep capability names and reconciliation seams in [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25. The known serving mechanisms and their re-runnable probe evidence will be documented in [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25, while [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 owns the executable constraints, remedies, probe integration, and enforcement.
• Continue to defer the generalized condition algebra and recursive five-way output classification; root-level channel classification remains sufficient for this revision.

One naming refinement I want to confirm rather than treat as already agreed: I see DeploymentPlan as desired pre-launch state and the runtime object carrying endpoints and observed ServingFacts as realized state. I therefore think they should remain distinct types, even if the latter is renamed from Instance to Deployment. If your intent is instead to merge them into one object across both phases, please correct me.

I also read reconcile() as a reusable domain function, not merely _validate_capabilities() inside the CLI. Normal sieval run must call it before launch, while run --dry-run, eval setup, and the Python API consume the same typed declaration and reconciliation result. A remote deployment may leave serving checks unresolved for setup probes or request-time enforcement rather than pretending plan-time convergence occurred.

Revision 2 will therefore specify:

1. the seven-dialect declaration model and implementation scope;
2. the four-column capability binding contract;
3. typed reconciliation inputs, outputs, derived defaults, launch remedies, unresolved checks, and persisted evidence;
4. the relationship between user declarations, aggregated task requirements, model support, and serving requirements;
5. task aggregation and explicit-parameter precedence;
6. behaviour across run, dry-run, remote eval, infer-only, and Python entry points;
7. the naming changes and hosted identity semantics;
8. the revised migration sequence and acceptance criteria.

I will not begin production implementation until that revision is reviewed. I read the requested change as preserving the accepted runtime skeleton while redesigning the declaration and validation plane.

[ethan-scitix]

Scope decision: six dialects land in #25; only vllm_native stages

Answering the blocking question directly. #25 lands the complete declaration,
binding, and reconciliation plane and six of the seven adapters:
openai_chat (chat completions), openai_completions, openai_responses,
anthropic_messages, google_genai, sglang_native. Only vllm_native
defers.

The symbol space is still designed against all seven, as you proposed.

Land them as a PR sequence, not one PR

Six adapters plus a rebuilt plane is not a reviewable single diff, and the
project's minimal-change-surface rule applies to us too. Revision 2 stays a
design document for the plane and the full symbol space; the adapters land as a
sequence of PRs under #25 in this order:

1. the plane — declaration, keyed capability registry, binding, reconcile() —
   plus openai_chat and openai_completions, i.e. PR feat(models): rebuild the provider-neutral model binding plane (RFC #25) #45's existing wire
   behaviour rebased onto it;
2. openai_responses;
3. anthropic_messages;
4. google_genai;
5. sglang_native.

The order is chosen to front-load falsification rather than to go easiest-first.
The two dialects in step 1 share one capability shape: logprobs exist in some
form, messages are a flat list, there is no first-class reasoning channel. A
plane validated only against those is validated against one protocol family
wearing two hats — the same mistake the current Model made with one. Steps 2–4
are where it breaks if it is going to:

• openai_responses — a different response envelope and a real reasoning
  channel, which is where output classification either holds or does not;
• anthropic_messages — no logprobs at all, forcing the four-column binding
  table to express capability absence as a first-class outcome rather than a
  translation; system prompt as a separate top-level field, forcing genuine
  message-shape lowering rather than key renaming; native thinking blocks;
• google_genai — a third message shape (contents / parts) and its own
  safety/config surface, which tests whether step 3 generalised or just special-
  cased Anthropic.

Discovering a plane defect at step 3 of 5 is cheap. Discovering it at step 5
means reworking four adapters.

The diff rule applies inside this sequence

For steps 2 through 5, the PR adding a dialect must have a diff confined to
(a) new files in that dialect's package and (b) one registry entry. If a step has
to touch the IR, the registry schema, reconcile(), or an existing dialect, then
the plane is wrong: we fix the plane in its own PR and continue, rather than
absorbing the edit into the adapter and declaring the seam validated.
vllm_native after #25 is then confirmation, not the test.

I recognise this is a blunt instrument and I would rather have a blunt checkable
rule than an aspirational one. Let us run this cycle with it and revisit if it
turns out to reject changes that are actually fine.

Two notes on scope

sglang_native is in scope per this decision, so the §8 probes are no longer
gating whether it ships — but still run them before step 5, because their
result should shape how it is implemented (a distinct dialect versus
collapsing into a shared native-generate shape). I would rather that be an
evidence-driven decision than a default.

Each adapter PR should show evidence it was exercised against a live endpoint,
not only mocks. That is reachable for all of these:
grep -rl 'anthropic\|genai\|responses\.create' sieval/ returns nothing today,
and our own gateway serves claude-* and gemini-* over their native
protocols — which is exactly why ChatModel 400s against them. Steps 3 and 4
turn a known dead end into working targets and give themselves a live validation
endpoint in the process.

Naming: your reading is correct, and my rationale was wrong

Keep DeploymentPlan and the realized object as distinct types. Do not merge
them. But my justification for the rename was wrong in a way that changes the
shape of the change, so let me correct it rather than have you design around it.

I wrote "DeploymentPlan is already the name for this in the infer layer."
Checking the code:

• DeploymentPlan (sieval/infer/topology/models.py:163) holds checkpoint,
  backend, assignments, attachments, deterministic, seed. It has no
  api_base and no endpoint of any kind.
• The realized side already exists under a different name:
  DeploymentCapabilities (same file, :208) — api_base, endpoints,
  roles, total_gpus, metrics_url, constructed at deployer.py:387 only
  after every handle is ready, with api_base read off handle.endpoint.

So plan/realized is already load-bearing in the infer layer, and §2.2's
Instance corresponds to DeploymentCapabilities, not to DeploymentPlan.
"Two words for one concept" was a real observation aimed at the wrong pair.

Two consequences:

1. Both types stay, and the justification is stronger than the divergence
   argument. It is not that the plan and the facts might disagree — it is that a
   plan has no endpoint to disagree about. And for sieval eval against a
   pre-launched endpoint there is no plan at all, which is the same entry-point
   asymmetry as my review's point 3. A merged type would be half-None on
   exactly the path where I asked §4 to keep a request-time gate, moving that
   asymmetry out of the type system and into runtime None checks.

2. The rename to resolve is DeploymentCapabilities, not Instance. Its
   "capabilities" means api_base / endpoints / GPU count, which collides head-on
   with this RFC's capability vocabulary (the keyed registry,
   models.<name>.capabilities, reasoning, top_logprobs). Introducing
   Deployment + ServingFacts while leaving it in place gives us three terms
   for two concepts. Revision 2 should fold it into the realized side:
   Instance → Deployment and instance.facts → ServingFacts both stand,
   and DeploymentCapabilities either becomes that Deployment or is
   absorbed into it. If that touches the infer→eval handoff, put it in the
   migration sequence — I would rather pay it here than ship a third word.

Also make the optionality explicit in the signature rather than incidental:
deployment_id: str | None should carry a documented meaning (None = a
deployment we did not launch, therefore no plan, therefore serving checks may be
unresolved). Same for whichever field references the plan.

reconcile()

Your reading is right and is what I intended. reconcile() is a reusable domain
function; _validate_capabilities in cli/validation.py is one caller of it,
sitting beside _validate_models / _validate_tasks. sieval run calls it
before launch and may derive launch parameters from the result; run --dry-run,
eval setup, infer start, and the Python API call the same function and
consume the same typed result.

And yes — a remote deployment legitimately returns unresolved serving checks
rather than faking plan-time convergence, and those unresolved checks are
precisely what the residual request-time gate consumes. That is the division of
labour I was asking for.

Everything else

The rest of your list reads as agreed; go ahead on revision 2. Three additions to
its structure:

• the implementation scope and PR sequence above, with the diff rule as an
  acceptance criterion for steps 2–5;
• the migration sequence must state what happens to DeploymentCapabilities;
• vllm_native is explicitly out of [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25's implementation scope but in its symbol
  space, so revision 2 should say what adding it later is expected to cost.

[jack-scitix-ai]

Based on the latest review and scope decision, here is Revision 2 for final pre-implementation review.

RFC #25 Revision 2: provider-neutral model dialect and capability plane

Status: draft for pre-implementation review
Related: RFC #25, PR #45, #24, #27, #46, #47, #49/#57, #58, #59,
#60/#61
Implementation gate: approve this revision before changing production code

Decision summary

Revision 2 keeps the provider-neutral Model.arun(Request) -> Response IR and
the wire behaviour already verified in PR #45, but replaces the flat
Capability(Flag) model with three deliberately separate mechanisms:

1. a keyed, serializable capability declaration and binding registry;
2. a pure reconcile() step that joins dialect, model, task, engine, and
   deployment facts before launch whenever those facts are available;
3. a small per-call request-leaf audit as the residual no-silent-drop guard.

The implementation target contains seven dialect identifiers:

• openai_chat
• openai_completions
• openai_responses
• anthropic_messages
• google_genai
• sglang_native
• vllm_native

#25 implements the first six in a five-PR sequence. vllm_native is designed
into the symbol space but is the only deferred adapter.

This remains a design revision, not authorization to implement all five PRs at
once. The plane and the first two dialects land first; every later adapter must
validate the plane without expanding its shared production surface.

1. Motivation and scope

PR #45's OpenAI/SGLang lower/lift work is worth retaining, but its flat
capability set conflates task need, dialect expression, model support, and
deployment facts. That permits silent request drops, false rejection, broken
Flag composites, and a task requirement surface disconnected from kind and
launch planning.

Revision 2 makes those axes explicit while preserving these invariants:

• Model.arun(Request) -> Response is the provider-neutral primitive.
• Every active request leaf is consumed, explicitly rejected, deliberately
  treated as a documented no-op, or forwarded unchanged. There is no silent
  fifth state.
• Response remains one closed additive persisted record.
• Pools own connections/shared limits; limiter hierarchy stays global -> task
  -> stage -> model.
• Dialect, model, engine, deployment, and connection family stay independent.
• Existing PR feat(models): rebuild the provider-neutral model binding plane (RFC #25) #45 wire results remain behaviourally identical except for
  separately named bug fixes.
• Evidence remains additive/strict under RFC [RFC]: Resume version-compatibility gate + per-record version provenance #24; ordinary typed handlers add
  no condition AST, accepted-region prover, or generic solver.

1.1 In and out of scope

#25 owns the seven-dialect/six-adapter Request/Response plane, keyed declaration
and dialect bindings, task aggregation, pure reconciliation, residual call
audit, deployment/connection seams, and additive call provenance. It does not
own #47 engine constraints/probes, #27/#59 model-profile inference, #58 dataset
packing, #49/#57 run task identity, #60/#61 grader schemas/profiling,
vllm_native implementation, or vendor-documentation-only fields.

Known serving mechanisms and rerunnable evidence are recorded in this RFC so
the design is grounded. Their executable constraints, remedies, and enforcement
belong to #47.

1.2 Related work and sequencing assumptions

#27/#59 own model profiles and the sole legacy kind resolver; #47 owns serving
constraints/remedies/probes; #58 consumes shared ServingFacts; #49/#57 remain
the only run-identity path; #60/#61 own grader records/profiling; #46 governs new
dialect imports. #25 consumes those seams without copying their mechanisms.

Implementation rebases onto merged #60 and the final #59 resolver, or stacks
explicitly behind #59 while it remains open. If an assumed schema changes before
implementation, this document is updated rather than copying another PR's
helper or persisted shape.

2. Object model and ownership

The terminology in Revision 2 is normative:

Old/provisional name	Revision 2 name
Frontend	Dialect
FrontendFactory.bind()	free bind_dialect() composition function
frontend_id	dialect_id
ResourcePool	ConnectionPool
Instance	Deployment
InstanceFacts	ServingFacts
infer-layer DeploymentCapabilities	folded into Deployment + ServingFacts

DeploymentPlan keeps its existing name. A desired plan and a realized
deployment are different, load-bearing concepts: a plan has no endpoint, while
an externally launched endpoint has no SiEval plan.

2.1 Object graph

DeploymentPlan (desired, optional)
        |
        v
Deployment (realized) ---- Engine
    | endpoints               | static semantics / verification rules
    | ServingFacts            ` concrete enforcement remains #47
    v
ConnectionPool ---- owns connection/client + shared limiter
    |
    +---- bind_dialect(dialect_id, ...) ------> bound Dialect
                                                   |
Model ---------------------------------------------+
  | requested model id, defaults, local limiter, RuntimeBindingPlan
  ` arun(Request) -> Response

The effective service contract is a join, not a class constant:

effective contract =
    dialect binding
  ∩ model capability profile
  ∩ engine serving rules
  ∩ observed deployment facts
  ∩ task requirements

Model orchestrates a bound dialect and local defaults/limit; ConnectionPool
alone owns clients and the shared limit; DialectSpec is serializable metadata;
free bind_dialect() resolves an executable binder; a bound Dialect validates,
lowers, executes, and lifts; Engine interprets server semantics; Deployment
is immutable realized state; DeploymentPlan remains desired infer state. None
may infer another axis from a class name or take over its lifecycle.

2.2 Interface sketch

Names below are illustrative; ownership, optionality, and call order are
normative.

EngineSource = Literal["deployment", "config", "unknown"]


@dataclass(frozen=True)
class ServingFacts:
    engine_version: str | None = None
    tokenizer_available: bool | None = None
    prefix_cache_enabled: bool | None = None
    max_top_logprobs: int | None = None


@dataclass(frozen=True)
class DeploymentPlanProjection:
    fingerprint: str
    engine_id: str


@dataclass(frozen=True)
class DeploymentTopology:
    is_disaggregated: bool
    roles: tuple[str, ...]
    total_gpus: int


@dataclass(frozen=True)
class RouteIntent:
    service_role: str | None = None


@dataclass(frozen=True)
class ResolvedRoute:
    service_role: str
    endpoint: str
    connection_family: str
    fingerprint: str


@dataclass(frozen=True)
class Deployment:
    deployment_id: str | None
    plan: DeploymentPlanProjection | None
    engine: Engine
    engine_source: EngineSource
    api_base: str
    endpoints: Mapping[str, str]
    topology: DeploymentTopology | None
    metrics_url: str | None
    facts: ServingFacts


@dataclass(frozen=True)
class DialectSpec:
    dialect_id: str
    connection_family: str
    implementation_status: Literal["active", "reserved"]
    capability_outcomes: Mapping[
        str, Literal["supported", "unsupported", "reserved"]
    ]

Engine.verify_deployment(deployment, probe) -> ServingFacts is the #47/#58
seam. bind_dialect(dialect_id, requested_model_id, deployment, pool, runtime_plan) -> Dialect is a free composition function, not a one-method
factory. Bound dialects expose request/output contracts plus
validate/prepare/execute.

Deployment.deployment_id is None and Deployment.plan is None explicitly
mean an external deployment not launched by SiEval. They are not accidental
missing values: launch remedies may be unavailable and serving checks may
remain unresolved. A managed deployment has both a stable id and the
provider-neutral projection of its desired DeploymentPlan.

ServingFacts contains observations, not a pass/fail verdict. The initial fields
above are defined by #25 because current dialect contracts consume them; #47 may
add typed optional facts when a concrete constraint requires one. It is shared
by the model plane, #47 constraints, and #58 dataset packing. Engine-specific
code may populate and interpret it, but consumers receive immutable
provider-neutral data.

The core layer does not import DeploymentPlan from sieval.infer. The CLI
composition layer supplies a small typed plan/launch projection to
reconcile(). DeploymentPlan remains the authoritative infer-layer desired
object and the serialized infer_plans.yaml source.

2.3 Connection lifetime and dialect rebinding

Model stops constructing AsyncOpenAI or any other provider client.
ConnectionPool owns one connection family and exactly one connection identity.
Pools are not globally deduplicated by URL because identical URLs may use
different credentials, retry policies, or quotas.

The plane PR provides the connection families required by the six-adapter
sequence: the existing OpenAI SDK family and a provider-neutral async HTTP/JSON
family whose credentials/headers are supplied by composition. Anthropic and
Google dialect packages can therefore speak their native protocols without
adding an official-SDK dependency or editing pool code. Any genuinely new
family/dependency discovered later is a separate infrastructure correction
before the adapter PR, preserving the Section 3 diff rule.

Dialect change is composition-mediated because a bound Model does not own the
model profile, task sources, or serving reconciler. Session/Python composition
reconciles the target dialect from that full context, then calls the low-level
model.with_dialect(dialect_id, runtime_plan). The method verifies the plan's
target/root/deployment fingerprint, borrows the same compatible pool/limiters,
and returns a plain Model; it never reconstructs or reuses a plan locally.

A connection-family mismatch fails before allocation or I/O and directs the
caller to construct a new Deployment + ConnectionPool. A dialect swap never
changes endpoint credentials, engine, or deployment plan.

ConnectionPool.aclose() is idempotent, rejects new admissions, drains entered
requests, and closes each owned connection exactly once. Sessions close every
unique pool; standalone callers use an explicit async context manager. No
correctness depends on __del__.

Limiter migration preserves current base/derived semantics: the root/base model
quota becomes the pool's shared limiter, while each derived model's additional
quota remains a model-local limiter. ConnectionPool.acquire(local_limiter)
enters both, so derivation cannot oversubscribe the base quota and a child limit
is a cap rather than a reserved slice.

2.4 Deployment migration

The current infer-layer DeploymentCapabilities is not left beside the new
types. Its fields migrate as follows:

Current field	Destination
api_base	Deployment.api_base
endpoints	Deployment.endpoints, preserving the existing role/service keys
is_disaggregated, roles, total_gpus	Deployment.topology
metrics_url	Deployment.metrics_url

Engine identity is added from the normalized desired plan or explicit external
configuration; it is not currently a DeploymentCapabilities field. Observed
engine version, tokenizer/cache state, and numeric limits go to ServingFacts.
The deployer constructs Deployment only after every handle is ready. sieval eval constructs an external deployment with no plan and possibly unknown
engine/facts. The former endpoint_map[str, str] handoff becomes a typed
deployment binding; it cannot erase topology, engine, or verification
provenance.

Deployment.endpoints keeps role/service keys; they are not relabelled as
dialect ids. The selected binder resolves its callable route from
api_base + endpoints + explicit service role and fails if that route is
ambiguous or absent. The existing DeploymentCapabilities type/build helper
and the production per-model endpoint_map handoff are replaced atomically so
no intermediate path drops the topology or selects "the first endpoint"
implicitly.

Route intent is normalized before launch. After readiness,
resolve_route(deployment, dialect, intent) chooses the sole endpoint by
default or requires an explicit role when several are viable. The resolved
route is fingerprinted into RuntimeBindingPlan; its endpoint/family is the
route component used to create or look up ConnectionPool. The runtime pool key
also includes secret-free credential-scope, retry-policy, and quota-scope ids,
so equal URLs never share the wrong client/limiter; raw secrets are neither
persisted nor hashed into evidence. bind_dialect() verifies the complete pool
identity. Missing/ambiguous routes fail before client I/O.

3. Dialect symbol space and delivery boundary

3.1 Seven dialects

The stable descriptor registry contains all seven identifiers from the start:

Dialect	Connection/protocol shape	#25 delivery status	Primary falsification surface
openai_chat	OpenAI chat completions	PR 1	messages, sampled logprobs, tools, structured output
openai_completions	OpenAI legacy completions	PR 1	completion text, input scoring, echo boundaries, FIM
openai_responses	OpenAI Responses API	PR 2	different envelope, reasoning channel, hosted tools, continuation state
anthropic_messages	Anthropic Messages API	PR 3	no logprobs, top-level system prompt, thinking blocks, provider-native tools
google_genai	Google Gen AI native API	PR 4	contents/parts, multimodal content, thinking, safety/config split, grounding
sglang_native	SGLang native generate endpoint	PR 5	native scoring, prefill, cache/tokenizer behaviour
vllm_native	vLLM native endpoint	deferred after #25	native prompt scoring, token ids, structured decoding, parser constraints

The row says what an adapter must confront; it does not claim every model behind
that protocol supports every feature. Dialect support is intersected with an
explicit model profile and serving facts.

Two registries have deliberately different jobs:

DIALECT_SPECS: Mapping[str, DialectSpec]       # all seven stable ids
DIALECT_BINDERS: Mapping[str, DialectBinder]  # executable package entrypoints

DIALECT_SPECS is serializable symbol metadata and marks not-yet-active rows as
reserved. DIALECT_BINDERS contains executable functions, not single-method
factory objects. The "one registry entry" rule below always means one binder
entry; it does not mean inventing a second descriptor.

vllm_native has a reserved descriptor and designed capability outcomes but no
executable binder in #25. Selecting it fails with a named
DialectNotImplemented error. Adding it later should require one new dialect
package, tests/live evidence, and one binder-registry entry. If it requires
shared IR, registry-schema, reconcile, or existing-dialect changes, the plane
has not met this RFC's extension criterion.

3.2 Five-PR implementation sequence

The implementation is not one six-adapter diff:

1. Plane + OpenAI chat/completions: revise PR feat(models): rebuild the provider-neutral model binding plane (RFC #25) #45 onto the declaration,
   registry, binding, reconciliation, Deployment, and ConnectionPool plane while
   preserving its verified OpenAI wire behaviour. The existing legacy
   SglangGenModel path remains operational until step 5. PR 1 pre-stages it as
   a binder-aware compatibility facade that dispatches to the unchanged legacy
   implementation while no sglang_native binder exists; the later binder
   entry switches the facade without another existing-file edit.
2. OpenAI Responses: add openai_responses.
3. Anthropic Messages: add anthropic_messages and prove unsupported
   logprobs are represented as first-class absence, not a translation gap.
4. Google Gen AI: add google_genai and prove a third message/content shape
   does not require special-casing the shared plane.
5. SGLang native: add sglang_native; probes determine its internal shape,
   not whether it ships.

PR 1 establishes the shared plane and preserves two OpenAI wire contracts, but
those dialects are one protocol family wearing two hats; they do not by
themselves validate the plane's generality. PRs 2–4 are the deliberate
falsification steps for the output envelope, capability absence, and genuinely
different message shapes.

The PR 1–4 fallback is an explicit temporary legacy bypass, not a canonical
sglang_native binding: old configs retain old guards, explicit
dialect: sglang_native and new capability declarations fail, and no
RuntimeBindingPlan is claimed. PR 5 activates the binder and removes the
bypass through the pre-staged facade. Tests pin both sides of that boundary;
#25 is complete only after PR 5.

For steps 2–5, adapter production changes are confined to:

• new files in that dialect's package; and
• one executable binder-registry entry.

Tests, fixtures, evidence, and documentation may be added normally. If an
adapter requires a dependency/lockfile, connection-family support, shared IR,
registry schema, reconcile(), or an existing dialect change, that change stops
the adapter PR and lands as a separately reviewed infrastructure/plane
correction first. It is not hidden in the adapter diff.

3.3 Native-endpoint evidence

Every adapter PR includes live-native-endpoint evidence: redacted wire shapes;
requested and raw reported model ids (including absence); observable
model/server version, date/commit/image; exercised support and absence cases;
stream modes; relevant proxy/cache state; and a rerunnable probe. Mocks still
cover deterministic/error branches but cannot establish protocol support.

4. Declarative capability surface

4.1 YAML semantics

A configured model declares its wire dialect independently of engine and may
declare semantic capabilities:

models:
  qwen3:
    dialect: openai_chat
    capabilities:
      reasoning:
        effort: high
        summary: auto
      top_logprobs:
        minimum: 100
      hosted_tools: false

The normalized value rules are:

• missing key: no user opinion; task requirements or an explicit mandatory
  model rule may require it, but dialect/model defaults never do;
• false: explicitly disabled;
• true or {}: enabled with typed defaults;
• mapping: enabled with typed options;
• null: invalid to avoid an ambiguous shorthand (the YAML loader can
  distinguish it from an absent key, but it communicates neither enable nor
  disable clearly);
• unknown names, unknown options, invalid domains, and duplicate keys: errors.

Canonical serialization expands true to {} and retains explicit false.
Capability options describe semantic behaviour and minimums, never raw launch
flags. Reconciliation derives launch remedies through #47.

Duplicate-key rejection requires one strict YAML loader across normal
run/eval/infer and every dry-run path. This intentionally replaces the current
split where yaml.safe_load silently keeps the last value while dry-run can
only warn. The loader change and compatibility test land in the plane PR; no
entry point may validate a different effective configuration.

4.2 Keyed registry, not Flag + frozenset

CAPABILITY_SPECS: Mapping[str, CapabilitySpec] maps each key to one typed
config dataclass. It stores no bit masks and implements no boolean algebra.

The Revision 2 symbol space is:

Key	Typed options	Semantic contract
input_scoring	none	return a complete score sequence for input tokens
sampled_logprobs	none	return logprobs for sampled output tokens
top_logprobs	minimum: int >= 1	return at least the requested alternative breadth where candidates exist
reasoning	optional effort, budget_tokens, summary	control reasoning and, when requested, expose a reasoning channel
function_tools	optional parallel	lower function definitions/choice and return function calls
hosted_tools	kinds: tuple[str, ...]	invoke dialect-hosted tools and preserve their observable outputs/citations
structured_output	formats: tuple[json_object, json_schema, ...]	constrain and return a declared structured format
stateful_session	none	continue from a provider/session response id and return updated state
opaque_continuation	none	round-trip provider-originated opaque reasoning/tool state unchanged
multimodal_input	modalities: tuple[str, ...]	accept declared non-text content parts
prefill	none	provide a dialect-native assistant prefill
fim	none	provide prefix/suffix fill-in-the-middle input

These are end-to-end semantic keys, not a mirror of every request field.
Reasoning effort/budget/summary are domains under one key; top-logprob breadth is
numeric rather than a boolean. Prefix-cache state, tokenizer availability,
parser flags, and maximum breadth are serving requirements/facts, not user-level
capabilities. Citations, usage, finish reasons, token ids, and provenance are
response data, not independent declaration keys unless a real task later
creates that coupling.

The declaration criterion is whether users must be able to name an end-to-end
semantic in the seven-dialect target, not how many tasks currently consume it.
Thus prefill remains a typed key even though its SGLang request projection is
the audited DialectOptions("sglang_native", {"prefill": ...}); Anthropic/native
bindings may project it differently. This declaration rule is distinct from the
internal Request-field promotion rule in Section 5. Vendor documentation alone
still cannot add a key without a target binding and user-visible semantic.

Specs needed by a later adapter may be present as reserved so the seven-
dialect symbol space is reviewable in PR 1. The parser rejects a reserved key.
The adapter's one binder entry activates only the specs for which that binder
contains an executable decision. Configuring an active key for which the
selected dialect has no implementation also fails explicitly.

4.3 Four independently owned columns

Every effective capability is joined by key across four columns:

Column	Question	Owner
dialect request leaves	how does this dialect express the request?	#25 dialect binding
dialect response channels	what must be returned or rejected?	#25 dialect output contract
model support/domain	can this checkpoint/hosted model provide it?	#27/#59 model profile or hosted metadata
serving requirements/facts	what launch setting or observation is required?	#47 handlers and probes

The registry owns the stable join key and config parser. It does not centralize
all four implementations.

DialectCapabilityBinding holds request leaves, response channels, and an
ordinary validate_config() method. ModelCapabilityProfile maps each key to
SUPPORTED | UNSUPPORTED | UNKNOWN plus source/reason and says whether the
profile is authoritative.

Every executable dialect declares one of Supported(binding) or
Unsupported(reason) for every active key. Anthropic's absence of logprobs is
therefore a tested registry result, not an omitted dictionary entry. Registry
completeness breaks CI when a key or dialect is added without a decision.

Absence of a recipe launch parameter such as reasoning_parser does not prove a
model capability is unsupported. A #59 recipe/profile must say whether it is
authoritative or provide an explicit entry; otherwise support is UNKNOWN and
must name a verifier. Unknown without an owner is an error, not optimistic
success.

ModelCapabilityProfile is a new typed schema requested from the #27/#59
model/recipe layer; the current launch-parameter overlay is not silently treated
as that schema. Hosted models use an explicit provider/model catalog or user
configuration at the composition layer. If neither is authoritative, their
status remains UNKNOWN; endpoint naming is never used as support evidence.

The four columns remain in their owning modules. sieval.core does not import
recipe or engine implementations; the composition layer supplies typed
projections to reconcile().

Each normalized declaration/task projection produces one small intent rather
than conflating support, defaults, and correctness minimums:

@dataclass(frozen=True)
class CapabilityIntent:
    key: str
    required: bool
    minimums: Mapping[str, JSONValue]
    request_defaults: RequestDefaults
    sources: tuple[str, ...]

For example, input_scoring: true requires the binding to make input scoring
available; it does not turn input scoring on for every generation call.
top_logprobs.minimum is a correctness floor, while a reasoning effort can be
a request default. An absent declaration expresses no user intent and cannot be
silently enabled by a model default; only task requirements or an explicit
mandatory model rule make it required. It may remain available on demand when
the runtime plan has proved support; a concrete Request then activates it
without retroactively creating a model default.

Each defaultable Request leaf has at most one owning capability key. Registry CI
rejects duplicate owners; reconciliation never resolves competing capability
defaults by precedence. Cross-capability implications may add requirements, but
only the leaf owner writes its default.

5. Request, response, and residual call gate

5.1 Provider-neutral Request

Input remains explicitly tagged and does not expose an OpenAI SDK type:

@dataclass(frozen=True)
class CompletionInput:
    text: str
    suffix: str | None = None


type ContentPart = TextPart | ImagePart | ToolCallPart | ToolResultPart


@dataclass(frozen=True)
class ChatMessage:
    role: Literal["system", "developer", "user", "assistant", "tool"]
    content: tuple[ContentPart, ...]


@dataclass(frozen=True)
class ChatInput:
    messages: tuple[ChatMessage, ...]


@dataclass(frozen=True)
class DialectOptions:
    dialect_id: str
    values: Mapping[str, JSONValue]


type ModelInput = CompletionInput | ChatInput

This is a deliberately small common content model, not a copy of an OpenAI SDK
type. Text/image/tool-result modality is visible to requirement projection and
the request audit; payloads inside one typed part remain opaque. Anthropic
extracts system content to its top-level field, Google lowers messages into
contents/parts, and OpenAI constructs its corresponding message objects.
Legacy OpenAI-shaped dictionaries pass through a compatibility normalizer before
the Request is built. Additional audio/video/document part types are activated
only with executable dialect evidence.

Request concerns are grouped once:

@dataclass(frozen=True)
class Request:
    input: ModelInput
    sampling: SamplingParams = SamplingParams()
    scoring: ScoringParams = ScoringParams()
    reasoning: ReasoningParams = ReasoningParams()
    tools: ToolParams = ToolParams()
    structured_output: StructuredOutputParams = StructuredOutputParams()
    session: SessionParams = SessionParams()
    scheduling: SchedulingParams = SchedulingParams()
    dialect_options: DialectOptions | None = None

The grouped records cover common sampling; input/output scoring and top-k;
reasoning effort/budget/summary; function/hosted tools; structured output;
session/opaque continuation; completion suffix/FIM; streaming; and tagged
dialect options.

Effort and token budget are mutually exclusive tagged choices. Opaque
continuation values retain their originating dialect_id and may only be
round-tripped through that dialect. Normal message-history replay is input, not
opaque session state.

Promotion into a shared first-class Request leaf follows implementation
coupling: two executable dialects translate the semantic into different wire
shapes, or one live dialect has an explicit IR consumer/rationale. This does not
govern which semantics users may declare in the capability registry. It
justifies reasoning/session/tool leaves across the target adapters and
sampling.top_k across Google/SGLang-native shapes. Engine-only values such as
stop_token_ids, Google-only safety knobs, provider beta headers, and raw parser
settings remain audited dialect extras until real coupling promotes them.
SGLang prefill is likewise DialectOptions("sglang_native", {"prefill": ...}) and has a golden migration test preserving its existing
wire body. The legacy extra_wire_params builder argument normalizes into
DialectOptions tagged with the currently bound dialect. with_dialect()
rejects defaults tagged for another dialect unless the caller explicitly
replaces them; it never carries or clears them silently.

5.2 Lightweight leaf audit

The registry/reconcile plane is the primary capability mechanism. The leaf audit
is the residual per-call defence for direct callers, per-call overrides,
external deployments, and implementation drift.

Active paths are derived recursively: input is always active; non-default nested
values activate their leaf; typed message modality/kind is visible while part,
tool, schema, and continuation payloads stay opaque; each dialect-option key is
an independent leaf even when its value is None.

Each active leaf receives exactly one decision:

Decision	Meaning
Consumed	the value affects wire data, endpoint/control flow, or a declared lift strategy
Rejected	the concrete value is unsupported and fails before I/O
NoOp	the dialect documents and tests semantic equivalence for this value
Passthrough	a dialect extra is copied unchanged to one declared wire destination

The implementation remains small:

active = nondefault_leaf_paths(req)
audit = RequestAudit(active)
dialect.validate_request(req, audit)
audit.raise_rejections()
prepared = dialect.prepare(req, audit)
audit.finish(prepared)

finish() rejects omitted, double-accounted, unclaimed-consumed, or altered
passthrough leaves. Broad wildcards such as reasoning.* cannot absorb future
fields; dynamic keys are audited individually. NoOp cannot mean "not yet
implemented." There is no relational accepted region or setup-time subset
proof.

A concrete request that activates a registered semantic capability must find it
available in the runtime binding plan. A dialect extra cannot bypass a
registered capability key or its minimums by spelling the same wire parameter
under dialect options. Per-call values may override request defaults but
cannot weaken a task/config correctness minimum.

Every dialect contract must cover the mechanically derived Request leaf schema.
Adding a leaf breaks incomplete dialect contract tests. Ordinary local branches
handle small relationships such as top_logprobs > 0 requiring a logprob
channel; this does not create a general condition language.

5.3 Closed Response and output guarantees

Response remains a closed @sieval_record. New semantic channels are
additive optional root fields, including reasoning, tool calls/tool results,
structured output, input/output scoring, citations/grounding, session state,
usage, and finish reasons. Existing field shapes are preserved unless a
separate migration explicitly versions them.

The current schema has plural texts but singular auxiliary channels. Until a
separately versioned per-choice response record exists, any Request combining
sampling.n > 1 with reasoning output, tools, structured output, or scoring is
rejected before I/O. A dialect may not silently keep choice zero and discard the
other choices' auxiliary data.

The additive structured-output channel uses a record wrapper around a parsed
JSONValue, so JSON null remains distinguishable from an absent channel. The
raw text stays in texts; parsing/schema validation is part of the channel's
postcondition.

Every dialect classifies each root semantic channel:

class Guarantee(Enum):
    PRESENT_OR_ERROR = auto()
    BEST_EFFORT = auto()
    NEVER = auto()


@dataclass(frozen=True)
class OutputRule:
    guarantee: Guarantee
    validator: OutputValidator

• PRESENT_OR_ERROR: a request/task requiring the channel either receives a
  valid value or the call fails after lift;
• BEST_EFFORT: validate when present, but it cannot satisfy a strict task
  requirement;
• NEVER: absence is part of the contract.

Whether a rule is required for a particular call is determined by the runtime
binding plan plus the concrete Request using an ordinary dialect-local branch.
For example, reasoning output is required only when the effective reasoning
intent/request asks for visible reasoning. No condition object or shared
predicate language is serialized.

The completeness invariant is intentionally root-level: every Response channel
has an output rule or is explicitly classified as core payload/provenance.
Nested payload leaves do not each become independent capabilities. This keeps
the output side symmetric enough to catch a new unclassified channel without
reintroducing a five-way recursive schema proof system.

Token ids already emitted by native SGLang remain faithfully represented and
validated as optional observed output. No production task consumes them. They
are not a TaskRequirements flag, do not gate removal or retention of another
wire path, and do not require nested PRESENT_OR_ERROR rules. A real consumer
may promote them later.

5.4 Fixed Model.arun() order

validate_request_invariants(req)
validate_runtime_binding_plan(capability_plan, req)
active = nondefault_leaf_paths(req)
audit = RequestAudit(active)
dialect.validate_request(req, audit)
audit.raise_rejections()
prepared = dialect.prepare(req, audit)
audit.finish(prepared)
async with pool.acquire(local_limiter):
    response = await dialect.execute(prepared)
dialect.output_contract.validate(capability_plan, req, response)
return attach_provenance(
    response,
    requested_model_id=binding.requested_model_id,
    provider_reported_model_id=response.response_model,
    binding_snapshot=binding.provenance,
)

The limiter is held through complete stream consumption, tool loops, and lift.
No connection or limiter is touched for an invalid, rejected, or unaccounted
request. A malformed successful wire response never reaches a task.

6. Typed reconciliation

6.1 Inputs and outputs

reconcile() is a pure reusable batch domain function. It performs no network
I/O, process launch, or probe. Internally it has an explicit binding phase and a
root-deployment phase so a derived model cannot produce a launch patch before
its siblings' requirements are known.

Typed artifact	Required contents
AggregatedTaskRequirements	input kinds/modalities, scoring booleans, numeric top-k minimum, source tasks
BindingReconcileInput	binding/root ids, route intent, declarations, task requirements, dialect descriptor, model profile
DeploymentReconcileInput	root/id/optional plan, engine/facts, recipe params, explicit user params
BindingCapabilityPlan	declared/available/required keys, request defaults, output channels, serving requirements; no launch patch
DeploymentCapabilityPlan	aggregated serving requirements, launch patch, named setup checks
RuntimeBindingPlan	binding plan, deployment/resolved-route fingerprints, safe request postconditions
ReconcileResult	maps of all three plans plus aggregated diagnostics

Each deferred check records capability, setup | request stage, verifier, and
reason.

runtime_plans is populated only when the supplied deployment facts are enough
to discharge setup checks or convert a check into an explicitly safe
request/response postcondition. All plan data is deterministically serializable.
Functions, clients, credentials, and dialect implementation objects are never
persisted.

6.2 Algorithm

1. For every binding, `reconcile_binding()`:
   a. normalizes declarations and TaskRequirements into CapabilityIntents;
   b. rejects explicit disablement that conflicts with a task;
   c. validates the dialect binding/config domain and model support;
   d. derives availability, request defaults, output requirements, and typed
      serving requirements;
   e. returns one BindingCapabilityPlan without a launch patch.
2. Group successful binding plans by `root_deployment_key`.
3. For every group, `reconcile_deployment()`:
   a. aggregates typed serving requirements exactly once;
   b. reconciles recipe defaults, explicit user parameters, plan, and facts;
   c. returns one DeploymentCapabilityPlan with the launch patch/setup checks.
4. When facts are present, finalize each binding/deployment pair into an
   immutable RuntimeBindingPlan; otherwise leave `runtime_plans` empty.
5. Return all plans and all diagnostics; any error prevents launch/binding.

This is a deterministic loop over keyed records with ordinary branches, not a
solver. UNKNOWN for a required key needs a named verifier or fails; an unused
optional key without one is marked unavailable rather than failing the whole
binding, and a later Request activating it is rejected.

The public reconcile(batch, serving_reconciler) function orchestrates the two
pure helpers. ServingReconciler is an injected Protocol implemented/composed
by #47; core never imports an upper-layer engine implementation. A single-model
Python caller passes a one-element batch and receives the same semantics.

Derived models retain their own dialect and request defaults but contribute
serving requirements once to their shared root deployment. Request defaults are
never aggregated across bindings.

6.3 Task aggregation and precedence

Task requirements aggregate as follows:

• booleans use OR;
• numeric minimums use max;
• input kinds are retained as a set and must all be expressible by the selected
  dialect/model;
• input modalities are unioned and must be supported by the dialect and model;
• contributing task names are preserved in diagnostics.

Conflict precedence is normative:

Conflict	Result
task requirement vs explicit capability false	hard error; report contributing tasks
explicit capability option vs dialect/model default	explicit option wins if supported
explicit option weaker than a task minimum	hard error; never silently strengthen it
recipe launch default vs correctness remedy	derived remedy may replace the recipe default
explicit user engine parameter satisfying a requirement	preserve it; emit no duplicate flag
explicit user performance parameter with no correctness conflict	user value wins; record the visible plan diff
explicit user engine parameter conflicting with correctness	hard error with one actionable remedy
per-call argument vs model/task request default	per-call value wins, then passes the residual audit

#47 owns the classification of each concrete serving knob as correctness or
performance and supplies remedy text. #25 owns the invariant that correctness
is never silently downgraded and engine translators never emit contradictory
positive/negative flags.

Supplying one semantic option through both a canonical capability and a legacy
models.args/tasks.infer_args surface is an ambiguity error during migration,
not last-wins behaviour.

6.4 Managed and external deployments

Managed runs call reconcile() before launch to derive a deployment patch, then
call the same pure function again with the realized Deployment/ServingFacts
before binding any Model. The second call receives the frozen prelaunch
deployment plan and is verification-only: it must reproduce the same launch
patch. Facts requiring a different patch hard-fail and require a relaunch; they
never mutate a running service. This explicit transition is:

BindingCapabilityPlan + DeploymentCapabilityPlan
    -- re-reconcile with ServingFacts --> RuntimeBindingPlan
    -- bind_dialect / Model.bind ------> executable Model

External deployments legitimately have Deployment.plan=None, incomplete
facts, and unresolved serving checks. Unresolved does not mean success. Each
check has one named owner:

• setup-only checks must be discharged by supplied/probed ServingFacts before
  a required capability can enter a runtime plan;
• checks that are intrinsically about the concrete request/response may become
  immutable RuntimeBindingPlan.request_checks and run on every relevant call;
• a check that is neither discharged nor a safe request postcondition blocks
  that capability and reports the required launch/probe evidence.

There is no hidden first-call verification cache in Model. If a probe yields
new reusable knowledge, #47 returns a new immutable ServingFacts value and the
composition root re-runs reconcile().

The request leaf audit still runs independently, even after full plan-time
reconciliation. That separation protects direct Python callers and catches code
drift without making the audit the primary capability catalog.

7. Task requirements and legacy model kind

TaskRequirements stays a small projection of real production consumers:

@dataclass(frozen=True)
class TaskRequirements:
    input: InputKind | None = None
    input_modalities: frozenset[InputModality] = frozenset({InputModality.TEXT})
    input_scoring: bool = False
    sampled_logprobs: bool = False
    min_top_logprobs: int | None = None


type NormalizedModelBinding = (
    NamedModelBinding | InlineModelBinding | ExternalModelBinding
)


@dataclass(frozen=True)
class RequirementContext:
    model_bindings: Mapping[str, NormalizedModelBinding]
    task_args: Mapping[str, JSONValue]
    dataset_config: Mapping[str, JSONValue]
    infer_args: Mapping[str, JSONValue]


@dataclass(frozen=True)
class TaskModelRequirement:
    role: str  # "candidate", "grader", ...
    binding: NormalizedModelBinding
    requires: TaskRequirements


class Task:
    @classmethod
    def model_requirements_for(
        cls, context: RequirementContext
    ) -> tuple[TaskModelRequirement, ...]: ...

A positive min_top_logprobs implies sampled logprobs. There is no token-id
requirement. The type grows only when a real task introduces new coupling.

Most tasks return one primary-model reusable preset. During migration, the decorator
mechanically converts existing model_type metadata into chat/completion input
modality; only PPL/BPB, CLP, and future special-output tasks add scoring fields.
model_requirements_for() is evaluated from normalized task args, dataset
configuration, and infer args before launch, so configurable logprob breadth,
scoring mode, and HLE's text-only versus image input are not frozen in a class
constant.
A registry golden requires every registered model-calling task to select a
preset/implementation, without freezing the task count in the RFC.

Before this hook, composition normalizes each named model, inline grader mapping,
or live Python Model into Named | Inline | External binding records. Inline
ids/root/connection identities are deterministically derived from task+role and
normalized config; an external record references the live model's existing
binding/runtime plan. Model-valued args are removed from the JSON task-arg view.
After reconciliation, composition injects models_by_role into task
construction, so tasks do not discover or create graders after launch.

Per model binding, the session aggregates requirements before launch. A task
using a derived model contributes serving requirements to the shared root
deployment. Per-task infer_args are normalized before aggregation: ordinary
sampling values remain request defaults, while a value that activates or raises
a capability minimum contributes the corresponding CapabilityIntent and
serving requirement. No raw infer arg becomes an engine launch flag directly.

#59's merged derive_model_type() remains the only legacy compatibility
resolver. Its task input becomes the normalized primary binding's
TaskModelRequirement.requires.input. The result is
consumed once at root composition for:

• default dialect selection when canonical dialect: is absent; and
• feat(recipes): split recipe params into hardware and capability layers #59's legacy chat/gen -> instruct/base recipe mapping.

Legacy YAML type: becomes a checked assertion against task modality, not an
independent override. A mismatch fails before launch. No caller re-derives kind
from Task.model_type, isinstance(ChatModel/GenModel), URL, response model, or
engine name.

The schema-version-1 metadata model_type field remains as a derived projection
for compatibility. Replacing it and sieval/meta/index.json is a separate RFC.

Grader models use the same requirements/reconciliation path. LLM-judge tasks
override model_requirements_for() to expose an explicit secondary grader
binding from their normalized task args before task construction; they no longer
hide a newly constructed ChatModel inside Task.__init__. A pre-built Python
grader's normalized external binding must already carry a compatible runtime
plan. #25 adds no grader_call field. When #60/#61 persist a full grader
ModelOutput, provenance rides at that protocol's existing grader-output
location through ModelOutput.model.provenance. Until #61 migrates HLE,
BrowseComp, and AA-LCR, their legacy text-only records contain no durable grader
ModelMeta; #25 does not claim otherwise. Grader spend aggregation in
profile.json remains #61's separate follow-up.

8. Entry points, identity, and persistence

8.1 One reconcile function at every entry point

Entry point	Required behaviour
sieval run [--dry-run]	batch-plan every binding/root; normal run launches and verifies facts, dry-run prints the identical prelaunch plan without probes/I/O
sieval eval [--dry-run]	no launch patch; use available endpoint facts/probes and bind only safe runtime postconditions; dry-run lists unresolved checks
sieval infer start [--dry-run]	use only requirements in its YAML (none in checkpoint-only mode); derive launch state, with dry-run omitting launch/probes
Python API	same typed batch with caller-supplied requirements/facts; direct calls retain residual auditing

sieval/cli/validation.py adds _validate_capabilities(...) beside
_validate_models(...) and _validate_tasks(...). It is a caller of the same
reconcile() domain function, not a CLI-specific capability implementation.

Normal sieval run becomes:

parse config
→ resolve task classes and model bindings
→ aggregate TaskRequirements
→ resolve model profiles and desired DeploymentPlans
→ reconcile the batch into binding plans + root-deployment launch patches
→ launch deployments
→ collect ServingFacts and re-run the same batch reconciliation
→ require one RuntimeBindingPlan per bound model
→ bind pools, dialects, models, and tasks
→ execute samples through the residual request gate

No service launches on any binding error; normal and dry-run are callers of the
same domain function.

8.2 Hosted and requested model identity

Configured identity and server-reported identity are not conflated:

@sieval_record
@dataclass(frozen=True)
class ModelIdentity:
    requested_model_id: str
    provider_reported_model_id: str | None = None

requested_model_id is always the configured/wire-request identity.
provider_reported_model_id is populated only when the native response actually
reports it. A gateway alias may therefore remain distinct from its reported
backend model.

Bind-time state contains only requested_model_id. After lift,
attach_provenance() constructs the per-call ModelIdentity from that binding
value plus the raw Response.response_model. It never mutates the bound model
and never substitutes the requested id when the response omitted one.

The current native SGLang path must stop copying the requested id into
Response.response_model when the server did not return one. In that case the
provider-reported value is None; provenance still retains the requested id.

@sieval_record
@dataclass(frozen=True)
class ModelProvenance:
    dialect_id: str
    engine_id: str
    engine_source: EngineSource
    deployment_id: str | None
    model_identity: ModelIdentity
    engine_version: str | None = None
    deployment_fingerprint: str | None = None
    capabilities: CapabilityEvidence | None = None


# Additive fields on existing persisted types:
Response.provenance: ModelProvenance | None = None
ModelMeta.provenance: NotRequired[ModelProvenance]

Every newly executed call carries the binding snapshot. A dialect swap changes
dialect_id even when endpoint and requested model remain the same.
An external engine with no declared identity uses the explicit canonical
engine_id="unknown" together with engine_source="unknown"; it is not guessed
from URL, model name, or dialect.

8.3 Capability and resume evidence

Desired, resolved, and observed state use existing persistence boundaries:

1. effective_config.yaml retains canonical dialect and capability
   declarations.
2. Managed serving requirements and derived launch parameters serialize through
   DeploymentPlan/infer_plans.yaml and its existing strict comparison.
3. Each model binding carries additive, secret-free evidence in provenance:

@sieval_record
@dataclass(frozen=True)
class CapabilityEvidence:
    declared: Mapping[str, JSONValue]
    effective: Mapping[str, JSONValue]
    plan_fingerprint: str
    verification_fingerprint: str | None = None

Fingerprints use canonical sorted serialization and never include credentials
or live client objects. External endpoints and Python-created models can still
record declaration/plan evidence even when no infer_plans.yaml exists.

For an external run that makes zero model calls, resolved per-binding evidence
has no persisted call record; effective_config.yaml and diagnostics are the
available evidence. This is an intentional #25 limitation rather than a hidden
third run-level artifact. If durable zero-call resolution is required, its
placement must be coordinated with #57/resume ownership before implementation.

Response.provenance and ModelMeta.provenance are additive. After every call,
the legacy bridge assigns ModelMeta.provenance = response.provenance; it does
not use a bind-time self.meta() value that cannot know the server-reported
model. build_model_call_meta() then continues to copy the whole ModelMeta;
no second handwritten path is added. Old records
deserialize with absent provenance. Mixed-version resume preserves old records
and writes evidence only for newly executed calls, under their existing producer
versions.

record_meta=False omits automatic stage model_calls; it does not strip
provenance from a persisted Response, ModelOutput, or full grader output.
#57 remains the sole owner of task identity in run meta.json. This RFC adds no
resume bypass and must pass old-record, mixed-resume, nested-round-trip, and
meta-drift tests.

9. Serving evidence and the #47 boundary

9.1 Ownership boundary

#25 defines capability vocabulary, dialect bindings, task aggregation,
ServingRequirement/ServingFacts seams, reconcile precedence, diagnostics,
and persistence of the outcome.

#47 defines concrete serving records/handlers, engine/version constraints and
remedies, requirement aggregation/launch translation, probes/fact
interpretation, and correctness-versus-performance classification. The injected
ServingReconciler exchanges typed requirements/results; it is not a predicate
DSL.

Until a required #47 handler or named residual verifier exists, the dependent
capability fails rather than being accepted without enforcement.

This rule does not permit #25 to break the existing PPL/CLP paths while #47 is
still landing. The plane PR keeps the current SGLang input-score completeness
guard and CLP/top-alternative postconditions as named request/response
verifiers. TaskRequirements become mandatory only when each required serving
condition has either that real verifier or the minimum stacked #47 handler.
Task launch instructions and runtime guards are removed only after their #47
replacement is active. ServingReconciler is injected by the composition layer;
core does not import #47 engine implementations.

9.2 Known serving mechanisms

The following evidence motivates the seams; #47 owns executable enforcement:

Mechanism	Correctness impact	#25 representation	#47 work
SGLang radix/prefix cache can return input logprobs only for the uncached suffix	PPL input scoring can silently omit cached tokens	input_scoring plus cache/token-count ServingFacts; existing runtime completeness guard remains until superseded	constraint, --disable-radix-cache remedy, setup probe
vLLM prompt/input-logprob support and prefix-cache interaction vary by version/V1 mode	an apparently successful response may omit or shorten input scores	deferred completeness check plus version/cache facts	versioned constraint and probe
vLLM --max-logprobs can default below CLP needs	required answer alternatives can be absent	numeric top_logprobs.minimum task requirement	derive/check max-logprobs, classify conflicts, remedy
reasoning/tool output on open-source engines can require parser flags	model support alone does not create a parseable response channel	dialect output guarantee plus model UNKNOWN/SUPPORTED and parser fact	parser requirement, launch translation, verification
tokenizer-disabled native modes change available token text/scoring	a dialect can return ids without usable text or complete scores	tokenizer fact and output postcondition	engine/version constraint and live probe

These facts may appear in #25 as probe-backed evidence. Their remedies and
enforcement do not migrate back into dialect-local ad hoc capability flags.

9.3 SGLang and vLLM probes

The SGLang OpenAI-completions probe sends a nonce-bearing prompt with
echo=true, logprobs, streaming/non-streaming variants, and repeated requests
on a sticky route. It verifies prompt/completion boundaries, complete prompt
scores, usage, token text, max_tokens=0, tokenizer mode, and cache-hit
behaviour. Source evidence at SGLang v0.4.6.post1 indicates the combination is
implemented, but source evidence is not a deployed-instance result.

The probe must run and its evidence must be recorded before PR 5 implementation
begins. It selects implementation shape, never delivery scope.

Because sglang_native is now in #25 scope, the probe no longer decides whether
the dialect ships. It decides implementation shape:

• if OpenAI completions has complete parity for every consumed SGLang-native
  contract, the sglang_native package may share validation/lift helpers and
  the compatibility path may recommend openai_completions where appropriate;
• otherwise it retains the distinct /generate lowering/lifting path and names
  the failing parity requirement.

The sglang_native dialect id always means the native /generate protocol; it
does not masquerade as OpenAI completions. The probe changes how much code can
be shared, not the wire meaning of the id or its inclusion in #25.

Optional token ids are not a parity gate because no production consumer
requires them.

The vLLM probe requests the independent prompt_logprobs response through its
native/OpenAI-compatible extension without echo, repeats it under recorded
cache conditions, and verifies length/text/rank shape against prompt usage. Its
result informs the deferred vllm_native binding and #47 constraints. If
independent prompt scores are absent, an echo fallback may survive only with a
strict usage boundary; missing/inconsistent usage fails instead of splitting at
zero.

Probe records follow Section 3's live-evidence fields. An unavailable untracked
design file is not evidence; recover it or replace its claims with a rerunnable
probe.

10. Compatibility and migration

10.1 Public compatibility

For one deprecation cycle:

• ChatModel and GenModel remain thin wrappers selecting openai_chat and
  openai_completions; production compatibility decisions never use
  isinstance.
• PR 1 pre-stages SglangGenModel as a compatibility facade: before PR 5 it
  delegates to the unchanged legacy implementation; after the binder entry
  appears it selects sglang_native. PR 5 therefore need not edit the facade.
• legacy constructors may create a private ConnectionPool, while canonical
  Model.bind() always receives an existing pool.
• Transport names may re-export the corresponding dialect names, but all new
  production code uses Dialect.
• as_type() is retained only for the compatibility wrappers and delegates to
  the same target-dialect re-reconciliation and binding path as
  with_dialect(). It reconstructs a truthful wrapper without __class__
  mutation and preserves pool/deployment/limiters, but receives a newly
  fingerprinted target-dialect runtime plan. Arbitrary classes raise
  TypeError. It is removed after the deprecation cycle.

Canonical dialect: is independent of engine:. When dialect is omitted,
the legacy resolver maps chat -> openai_chat and gen -> openai_completions. For one cycle, an explicitly configured legacy
engine: sglang plus resolved gen and no dialect selects the named legacy
bypass before PR 5, then sglang_native with a deprecation warning once its
binder is active. An engine learned only from a managed plan does not trigger
this shim.

with_args() changes legacy request-builder defaults and a local concurrency
cap only. Reserved binding keys such as dialect/transport, engine, endpoint,
credentials, retry policy, and connection fail immediately instead of becoming
request defaults or dialect options.

10.2 Wire-preserving migration and named fixes

The plane PR ports only PR #45's OpenAI chat/completions lower/lift behaviour
with golden/mutation tests. The already verified SGLang-native logic is
preserved for replay in PR 5; it does not land in PR 1. The following PR-1 fixes
are explicit, not invisible refactors:

• OpenAI completions input-scoring split rejects absent or inconsistent usage
  rather than treating the prompt boundary as zero.
• OpenAI-compatible engine extensions travel through the dialect's declared extension body, not
  unsupported SDK top-level keywords.
• OpenAI chat streaming reconstructs function tool calls when that capability
  is active; until the reconstruction test passes, stream=True plus function
  tools is rejected before I/O rather than dropped.
• Requests with n > 1 and a singular auxiliary channel are rejected instead
  of retaining choice zero only.
• provider-reported model identity is never fabricated from the requested id.
• PR feat(models): rebuild the provider-neutral model binding plane (RFC #25) #45's completion-only top_logprobs behaviour during input scoring and
  its present-empty-to-legacy-None bridge are called out in its summary and
  changelog.

PR 5 separately applies and tests the SGLang-native fixes:

• clamp max_new_tokens where /generate requires at least one token,
  including the legacy max_tokens=0 scoring path;
• reject unknown native extras instead of dropping them;
• preserve the old prefill wire body through the audited dialect-extra path;
• stop fabricating a provider-reported model id.

10.3 Migration order inside the plane PR

1. Rebase/stack on fix(scripts): enforce relative-import scope in check_layer_imports #46, feat(recipes): split recipe params into hardware and capability layers #59, and merged feat(core): uniform stage-output protocol, piloted on 7 tasks #60; coordinate feat(core): persist task identity in run meta.json #57 persistence tests.
2. Add registries/reconcile records and introduce Engine, Deployment,
   ServingFacts, ConnectionPool, binders, and the lossless
   DeploymentCapabilities migration.
3. Atomically replace endpoint injection and move client/shared-limit ownership
   into the pool with lifecycle tests.
4. Port OpenAI chat/completions without wire drift and
   pre-stage the binder-aware/legacy-delegating SGLang compatibility facade.
5. Add reconciliation and residual contracts; atomically migrate task modality,
   feat(recipes): split recipe params into hardware and capability layers #59 input/type assertions, and class-kind checks. Activate each PPL/CLP serving
   requirement only with its existing named residual verifier or the minimum
   [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47 handler; retain guards/instructions until replaced.
6. Add provenance/identity and expose rebinding shims only after pool sharing is
   proven.
7. Apply named fixes and run wire parity, import, type, unit,
   integration, serialization, resume, and meta-drift suites.

Adapter PRs then follow the sequence and diff rule in Section 3.

11. Acceptance criteria

• One strict loader rejects duplicate/unknown/malformed/null/inactive
  declarations identically in normal and dry-run paths.
• Registry completeness requires typed configs and explicit
  supported/unsupported outcomes for every active key/dialect cell.
• Every required/available key has dialect, model, and serving outcomes; tasks
  cannot disable/weaken their requirements, and OR/max aggregation preserves
  sources across derived bindings.
• Reconciliation is pure, deterministic, serializable, batch-aware, and free of
  condition algebra; correctness conflicts fail while safe explicit performance
  tuning remains visible.
• Normal run reconciles before launch and again with realized facts before
  binding; dry-run derives the same plan without probes, and external dry-run
  reports rather than fabricates facts.
• Request schema additions break incomplete dialect contracts. Every active leaf
  gets exactly one observed consume/reject/no-op/passthrough decision; bad or
  unaccounted requests make zero client calls.
• Requests cannot activate unavailable capabilities, weaken minimums, or bypass
  registered keys through dialect options. Output channels have complete
  guarantees and required channels fail on absence/malformed values.
• n > 1 with singular auxiliary output and unsupported streaming tool calls
  fail before I/O. Optional token ids remain output-only evidence.
• Models/dialects never own clients. Rebinding preserves a compatible pool and
  limiters but produces a new target plan; mismatched families fail before I/O.
  Pool close is draining, idempotent, and exactly-once.
• DeploymentPlan/Deployment stay distinct, external plan/id None is
  explicit, and all DeploymentCapabilities fields migrate without loss.
• Dynamic task/dataset/infer args and explicit grader bindings contribute
  requirements prelaunch; feat(recipes): split recipe params into hardware and capability layers #59 is the sole legacy kind resolver and production
  code has no model-class compatibility isinstance.
• Requested/reported model identities remain distinct. Provenance and nested
  evidence round-trip through old records, mixed resume, record_meta=False,
  grader outputs, feat(core): persist task identity in run meta.json #57 metadata, and strict infer-plan checks.
• PR 1 preserves OpenAI wire parity plus named fixes. PRs 2–5 obey the
  new-package-plus-one-binder-entry rule or stop for a plane correction; all six
  adapters include live native evidence. SGLang probes choose shape, not scope,
  and deferred vllm_native must meet the same extension cost.

12. Decisions requested in this review

The scope and six-adapter delivery decision are now resolved. No choice here
blocks writing Revision 2, but implementation should wait for explicit reviewer
agreement on the following concrete proposals:

1. Confirm the keyed catalog, explicit CapabilityIntent, and four-column join,
   with the leaf audit retained only as a residual per-call defence.
2. Confirm desired DeploymentPlan versus realized
   Deployment + ServingFacts, plus the five-PR/diff/live-evidence rules.
3. Confirm capability YAML semantics: null rejected,
   false is an explicit disable, true canonicalizes to {}, and all entry
   points use the strict loader; correctness conflicts fail while safe explicit
   performance tuning wins.
4. Confirm root-level output guarantees, typed message modality and prelaunch
   primary/grader projection, plus rejecting n > 1 with singular auxiliary
   channels pending a versioned per-choice Response.
5. Confirm one-cycle compatibility wrappers/rebinding and that declarations stay
   in effective_config.yaml, managed
   launch resolution remains in infer_plans.yaml, and per-binding evidence is
   additive provenance rather than a third run-level artifact, accepting that
   an external zero-call run has no persisted resolved call plan.
6. Confirm batch binding/deployment reconciliation and verification-only
   post-launch runtime plans; serving mechanisms remain probe-backed evidence
   here while executable constraints/remedies stay in [RFC]: Capability constraint layer — declare and enforce the silent cross-group constraints #47.

Feedback period

At least one week after Revision 2 is circulated. The plane/object ownership,
catalog/reconcile boundary, and PR sequence require explicit agreement before
production implementation begins.

CC list

PR #45 reviewer and the authors/reviewers of #25, #27, #47, #49/#57, #58, #59,
#60/#61.

Any other things

This revision deliberately prefers a visible unsupported/unresolved error over
silent downgrade. It restores a declaration vocabulary without restoring the
over-engineered condition algebra, and it treats each native adapter as an
attempt to falsify the shared plane rather than an excuse to expand it.

[ethan-scitix]

Confirmations

Revision 2 is the document I asked for. Taking §12 in order: 1, 2, 5 and 6
are confirmed as written. 3 and 4 are confirmed with one carve-out each. §7
carries a stale premise that changes what it has to do.

§12.4 — the n > 1 rejection is too broad; reasoning is already per-choice

§5.3 justifies the rule with "the current schema has plural texts but
singular auxiliary channels", and §10.2/§11 carry it through as a named fix
and an acceptance criterion. That premise does not hold for reasoning.

ModelOutput.reasoning_texts is list[str] | None
(sieval/core/models/model.py:68), allocated as [""] * num_choices and
filled by choice.index on both the streaming and non-streaming branches
(chat_model.py:38, :79, :106, and again at :165, :211, :249).
finish_reasons is per-choice by the same construction. Reasoning at n > 1
is not a gap we are tolerating — it works today, and the MathArena family
depends on it: those tasks run n=4 by default and Apex 2025 publishes at n=16,
against reasoning models. Rejecting it before I/O is a regression, not a
guard.

The rule is right for the channels that genuinely are singular:
logprobs_tokens, logprobs and top_logprobs are per-token lists for one
choice produced by the separate alogprobs path, and tools and structured
output have no Response representation at all yet.

Make the criterion mechanical instead of a list: a channel is subject to the
n > 1 rejection iff its Response field is not indexed by choice. That
covers scoring, tools and structured output, excludes reasoning and finish
reasons, and stays correct as channels are added. The dialect contract test
should assert each channel's classification rather than restate the list.

§12.3 — the strict YAML loader is out of #25's scope

Confirmed on the value semantics: null rejected, false is an explicit
disable, true canonicalises to {}, correctness conflicts fail, and safe
explicit performance tuning wins while recording a visible plan diff. The
clause I am not confirming is "all entry points use the strict loader".

§4.1 bundles two different checks under one sentence:

• schema layer — unknown capability names, unknown options, invalid
  domains, null, reserved keys. Scoped to the capabilities: subtree, which
  is [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25's own new surface. [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25 owns this; keep it.
• loader layer — duplicate mapping keys anywhere in the file. A
  pre-existing repo-wide gap with no coupling to the IR: a duplicate
  api_base is silently last-wins today and would be with or without [RFC]: A capability-based Model IR — decouple model access from the OpenAI protocol #25.

The second is the same shape as the rule I set for PRs 2–5 — an infrastructure
correction discovered while doing plane work, which lands as its own reviewed
change rather than inside the diff that found it. Revision 2 applies that
discipline to the adapters and exempts PR 1. It should apply to PR 1 too.

cli/run.py:64 will be edited by the plane PR anyway, since §8.1 rewrites the
run pipeline — but "the line is already being touched" is not the criterion.
By §10.2's own standard a behaviour change there is a named fix, not an
invisible refactor, and once it is a named breaking fix it may as well be its
own PR, where it is easier to review and to revert.

Concretely:

• add the global YAML loader to §1.1's "does not own" list;
• split §11's criterion: keep "normal and dry-run validate identically", which
  is what one reconcile() at every entry point buys (§8.1), and drop "one
  strict loader … duplicate";
• the loader change becomes its own PR: promote
  load_yaml_with_duplicate_check (cli/validation.py:90, today dry-run-only
  and documented at :93 as preserving last-wins) into the single loader for
  the eval config, with duplicates as errors. cli/run.py:64 and
  cli/leaderboard/session.py:753 are the two bare safe_load sites behind
  the run/eval-versus-dry-run divergence that motivates it. Whether it also
  covers recipe YAML (infer/recipes/registry.py:200,342,362) and the
  leaderboard scanner/catalog/card front-matter is that PR author's call —
  whatever they choose, state the file scope explicitly in the PR. It is
  user-visible breaking and needs its own CHANGELOG entry.

Order between that PR and the plane PR is not constrained; land them whichever
way is convenient. Record the residual in revision 2 instead of sequencing
around it: until the loader PR lands, a duplicate capabilities: key is
swallowed by PyYAML before #25's schema validator ever sees it. That is a known
one-cycle gap, not a reason to merge the two changes.

One pre-existing inconsistency for the loader PR to settle:
_validate_structure records an unrecognised top-level key as a warning
(cli/validation.py:110) while §4.1 makes unknown names errors. State whether
the tightening stops at capabilities: or reaches the top level.

§7 — #61 is merged; the grader caveat is stale

§7 says "Until #61 migrates HLE, BrowseComp, and AA-LCR, their legacy
text-only records contain no durable grader ModelMeta; #25 does not claim
otherwise."

#61 closed as completed on 2026-08-05 and #62 finished the migration 40/40.
HLE, BrowseComp, AA-LCR and SimpleQA-Verified all persist the grader's whole
ModelOutput as extra.grader_output, model meta included
(hle_0shot_gen.py:40, browsecomp_0shot_gen.py:20, aa_lcr_0shot_gen.py:17,
simpleqa_verified_0shot_gen.py:75). ModelOutput.model.provenance is
therefore available on every judge task from PR 1 — drop the caveat and the
"until #61" sequencing rather than designing a degraded path for it.

The adjacent claim in the same paragraph is still accurate and still worth
doing: HLE builds a ChatModel inside Task.__init__
(hle_0shot_gen.py:152), which is exactly the pattern
model_requirements_for() has to remove.

Bookkeeping: §7 leaves "grader spend aggregation in profile.json" to #61 as a
separate follow-up and §1.2 lists "#60/#61" as owning grader
records/profiling. Both are closed and no open issue covers that follow-up.
File one or drop the reference — revision 2 should not point at a closed owner.

Everything else

The naming table and the DeploymentCapabilities field migration in §2.4 are
what I asked for, and keeping DeploymentPlan distinct with documented None
semantics resolves the entry-point asymmetry in the type system rather than at
runtime. The keyed catalog, CapabilityIntent, the four-column join and
demoting the leaf audit to a residual per-call defence all read as correct.
Pre-staging the SglangGenModel facade in PR 1 so PR 5 needs no existing-file
edit is a better answer to the diff rule than I had in mind.

Do not wait out the feedback window. This RFC opened on 1 July and has been
through two full revisions; the design plane is settled enough that further
review is better spent on real code than on a third document. Fold §12.3 and
§12.4 in as above and start PR 1 now — remaining CC input lands as review
comments on the PR.