Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
exists (it never deletes it). The default `.gitignore` advice
(`.evalshift/*`) already covers the new file.

### Fixed

- **A malformed `captures.store` bucket or account is no longer quoted.** A
URI whose bucket (or Azure account) Python's `urlsplit` rejects itself, such
as a bracketed host that is not an IP address or a character that
normalises to `/`, raised `urlsplit`'s own error, and two of those quote
that part of the URI, which may hold a pasted secret. Loading
`evalshift.yaml` now fails with `store URI names an invalid bucket or
account; accepted forms: ...`, with no chained exception. Same fix as
evalshift-sdk.
- **An `az://` store can no longer read outside its prefix.** The Azure client
resolves `.` and `..` segments in the request URL, so
`az://acct/evals/../other` read from container `other`. An `az://` container
or prefix with an empty, `.` or `..` segment now fails to load (`Azure store
URI has an empty, '.' or '..' segment in its container or prefix, ...`), and
the Azure reader refuses such a container, prefix or blob key. Keys
`capture sync` downloads were already filtered to the SDK's layout. S3 and
GCS keep such keys literal and are unchanged. Same rule as evalshift-sdk.

## [1.6.2] - 2026-10-27

### Changed
Expand Down
2 changes: 1 addition & 1 deletion DOCS.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ captures:
max_cases: 200 # optional, per suite
```

and `capture sync` and `capture list` fetch what is new before they read the local directory — captures are immutable and toolsets content-addressed, so only objects missing locally are downloaded, captures already promoted are skipped (so `capture clean` never causes a re-download, unless their conversation has a turn in the bucket, inside the window, that is not promoted yet: the fetch then brings its promoted turns back for that turn's history; `capture sync --repair` alone re-fetches the promoted captures its rebuilds need, past `--since` too), and each file is written atomically. Bucket keys are untrusted: one not shaped like the SDK's layout — a suite directory containing `:`, `/`, `\`, NUL or `..`, or starting or ending with `.` or a space, included — is counted as malformed and never written (a key with a trailing newline included), and no key can place a file outside `.evalshift/`; folder-marker keys ending in `/` (the S3 console's "Create folder", ADLS directories) are skipped without being counted. One summary line reports it (`fetched 12 capture(s) and 1 toolset(s) from s3://…`; with `capture list --json` it goes to stderr, so stdout stays pure JSON). `--since 24h` limits a fetch to recently written captures, `--offline` skips it, and `capture fetch` does only that step. `promote`, `diff` and `clean` stay local. Install the matching client library — `pip install boto3`, `google-cloud-storage` or `azure-storage-blob azure-identity` — and authenticate with the provider's normal chain (an OIDC role in CI, `aws sso login` / `gcloud auth application-default login` / `az login` locally); credentials never go in the URI, and a value containing `@` or `?` fails to load; no rejected value is echoed back, at most its scheme. A missing client library exits 1 naming the `pip install <package>` to run; a list or download failure exits 1, so `sync` promotes nothing from a partial fetch, with a hint naming the provider's credential chain (`aws sso login` / `AWS_PROFILE` / the CI OIDC role, `gcloud auth application-default login`, `az login`); a local write failure names the file and asks you to check free disk space and permissions instead. If `evalshift.yaml` cannot be read or does not validate, `sync` and `list` skip the fetch and work on the local directory as before, warning only when the file has a top-level `captures:` key. Full field reference: [`captures`](docs/configuration.md#captures).
and `capture sync` and `capture list` fetch what is new before they read the local directory — captures are immutable and toolsets content-addressed, so only objects missing locally are downloaded, captures already promoted are skipped (so `capture clean` never causes a re-download, unless their conversation has a turn in the bucket, inside the window, that is not promoted yet: the fetch then brings its promoted turns back for that turn's history; `capture sync --repair` alone re-fetches the promoted captures its rebuilds need, past `--since` too), and each file is written atomically. Bucket keys are untrusted: one not shaped like the SDK's layout — a suite directory containing `:`, `/`, `\`, NUL or `..`, or starting or ending with `.` or a space, included — is counted as malformed and never written (a key with a trailing newline included), and no key can place a file outside `.evalshift/`; folder-marker keys ending in `/` (the S3 console's "Create folder", ADLS directories) are skipped without being counted. One summary line reports it (`fetched 12 capture(s) and 1 toolset(s) from s3://…`; with `capture list --json` it goes to stderr, so stdout stays pure JSON). `--since 24h` limits a fetch to recently written captures, `--offline` skips it, and `capture fetch` does only that step. `promote`, `diff` and `clean` stay local. Install the matching client library — `pip install boto3`, `google-cloud-storage` or `azure-storage-blob azure-identity` — and authenticate with the provider's normal chain (an OIDC role in CI, `aws sso login` / `gcloud auth application-default login` / `az login` locally); credentials never go in the URI, and a value containing `@` or `?` fails to load; no rejected value is echoed back, at most its scheme, a malformed bucket or account name included. An `az://` container or prefix with an empty, `.` or `..` segment fails to load too (`az://acmeprod/evals/../other`): the Azure client resolves such segments in the request URL, even into another container, so the Azure reader also refuses a blob key that has one. A missing client library exits 1 naming the `pip install <package>` to run; a list or download failure exits 1, so `sync` promotes nothing from a partial fetch, with a hint naming the provider's credential chain (`aws sso login` / `AWS_PROFILE` / the CI OIDC role, `gcloud auth application-default login`, `az login`); a local write failure names the file and asks you to check free disk space and permissions instead. If `evalshift.yaml` cannot be read or does not validate, `sync` and `list` skip the fetch and work on the local directory as before, warning only when the file has a top-level `captures:` key. Full field reference: [`captures`](docs/configuration.md#captures).

**Bounded suites.** Together, `captures.max_age` (a duration: `30m`, `24h`, `14d`, at most `36500d`; omit it for no age limit) and `captures.max_cases` (an int ≥ 1), both optional, keep each captured suite a capped, route-balanced sample of recent production on `defaults.source_model`; `evalshift init` scaffolds `max_age: 14d` and `max_cases: 200`. Setting either one turns the feature on: `max_age` filters what enters a suite and marks old cases stale, and `max_cases` caps and route-balances each suite. Without them `capture sync` promotes and keeps exactly what it did before, and `defaults.source_model` alone changes nothing. Case files record `captured_at` and `source_model` (the first `model_call`'s `model_id`) either way, and sync backfills both on existing cases whose capture is still local (`backfilled capture time and model on N case(s)`). With the limits set, sync promotes only captures recorded within `max_age` on `defaults.source_model`. Models match on bare names: lower-cased, provider prefix and one trailing snapshot date (`-20260301`, `-2026-08-07`) dropped, with no alias lookup, so set `defaults.source_model` to the id your app sends. If none of a suite's cases and captures was recorded on it, sync warns (`alpha: none of 212 case(s) and capture(s) were recorded on claude-sonnet-5-5 (seen: …); not filtering by model — check defaults.source_model`) and does not filter that suite by model, so a typo or a rollout whose first new-model capture has not arrived can neither empty nor freeze a suite. `max_cases` caps every suite under `.evalshift/suites/`, including ones promoted by hand with `capture promote` and ones whose entry is `managed: false` (that flag freezes the YAML entry, not the case files): new conversations enter whole, least-represented tool route first, in a fixed hash order, so the same captures give the same suite on any machine; once no whole conversation fits, the slots left take the next one's leading turns. A conversation turn that recorded no messages list of its own rebuilds its history from the turns before it, so under the limits it is promoted only when every earlier turn is local (`not eligible: 3 missing earlier turns`). A stale case (too old, or recorded on another model) is retired outright when a fresh capture of the same question is added in its place; if its fresh copy is not added, the stale case leaves only to make room, like any other stale case. Stale cases that leave to make room go other-model first, then oldest, so a suite never shrinks because it went stale. A suite still over the cap loses fresh cases from its most-represented route, oldest first. Cases marked `provenance: reviewed` are never retired; in a captured suite, mark them in the case file `.evalshift/suites/<suite>/<case>.json` (on the calls under `example.expected_tools`, and `example.expected_tool_rounds` when present), not in `golden.jsonl`, which sync regenerates from the case files on every run. The limits also bound the automatic fetch of `capture sync` and `capture list`: `max_age` skips older captures (`skipped 11 older than captures.max_age (14d)`), and `max_cases` downloads at most `3 × max_cases` captures per suite per sync inside that window, a conversation at a time: conversation folders and lone captures newest first by their newest capture (ties by name), each taken whole while it fits; once none fits, what is left takes the earliest turns of the next one whose turns that fit include one not yet promoted, passing over one whose would all be promoted and leaving the rest unused when none qualifies (`2400 beyond each suite's newest 600`). Captures already local or promoted count, so a re-sync with nothing new downloads nothing. A conversation folder with a new turn inside the window also brings its promoted turns, whatever their age while the bucket still holds them, so a fresh checkout rebuilds the new turn's history; its unpromoted turns outside the window stay behind. Turns written flat (an evalshift-sdk older than 0.7.0, `EVALSHIFT_CAPTURE_LAYOUT=flat`, or the `@capture.agent` decorator) are fetched one at a time, as before. `--since` replaces the age cutoff, not the budget, and `capture fetch` applies neither. A turn without its own messages list is also missing earlier turns when the window or the budget cut them off, when the bucket's lifecycle rule expired them, when `capture clean` removed them, when they were written flat (an evalshift-sdk older than 0.7.0, `EVALSHIFT_CAPTURE_LAYOUT=flat`, or the `@capture.agent` decorator) and promoted on another machine (a fresh checkout never fetches promoted flat captures) or dropped by a per-turn sample rate, when the SDK's `max_captures` / `capture_ttl` evicted them, or when `turn_index` starts at 1 instead of 0; sync warns naming these causes. Pass the full messages list to `capture.model_call` so each turn carries its own history. With `max_cases` set, build warnings print only for captures that become cases. `doctor` and `compare` report stale suites (below). Expire the bucket's `captures/` prefix with a lifecycle rule after a period ≥ `max_age`; the CLI never deletes from the bucket. Full rules, output and lifecycle examples: [Bounded suites](docs/configuration.md#bounded-suites).

Expand Down
9 changes: 6 additions & 3 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1150,9 +1150,12 @@ Accepted URI forms — the same grammar the SDK uses:
| `gs://<bucket>/<prefix>` | Google Cloud Storage | `pip install google-cloud-storage` | Workload Identity in CI, `gcloud auth application-default login` locally |
| `az://<account>/<container>/<prefix>` | Azure Blob Storage | `pip install azure-storage-blob azure-identity` | Managed Identity / federated credential in CI, `az login` locally |

`<prefix>` is optional. Credentials never go in the URI: a value containing `@` or `?` fails
to load, naming the accepted forms. A rejected value is never echoed back (it may be a pasted
key or connection string); the error names at most its scheme. Under the prefix the layout is exactly the local one —
`<prefix>` is optional. An `az://` container or prefix may not have an empty, `.` or `..`
segment (`az://acmeprod/evals/../other` fails to load): the Azure client resolves such segments
in the request URL, even into another container. Credentials never go in the URI: a value
containing `@` or `?` fails to load, naming the accepted forms. A rejected value is never echoed
back (it may be a pasted key or connection string), a malformed bucket or account name included
(`store URI names an invalid bucket or account`); the error names at most its scheme. Under the prefix the layout is exactly the local one —
`captures/<suite>/cap_<hex>.json`, `captures/<suite>/conv_<16 hex>/cap_<hex>.json` for a
`capture.agent_session` conversation's turns (evalshift-sdk 0.7.0 and later), and `toolsets/<hex>.json` — so the bucket
is a mirror of `.evalshift/`, not a different format.
Expand Down
8 changes: 6 additions & 2 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1007,8 +1007,12 @@ retention:
run_ttl_days: int>=1|null = null
captures:
store: s3://<bucket>/<prefix> | gs://<bucket>/<prefix> | az://<account>/<container>/<prefix>
-- optional block; store validated at load (bad scheme, missing bucket/container, '@' or '?'
-> load error naming the forms; no error echoes the value, at most its scheme); prefix optional,
-- optional block; store validated at load (bad scheme, missing or malformed bucket/account,
missing container, an az:// container or prefix with an empty, '.' or '..' segment, '@' or
'?' -> load error naming the forms; no error echoes the value, at most its scheme, nor chains
one that does; a malformed netloc -> "store URI names an invalid bucket or account"); the
Azure reader also refuses a blob key with such a segment (the Azure client resolves dot
segments in the URL, even across containers); prefix optional,
trailing slash stripped; default none = local disk only. Client libraries are plain packages:
pip install boto3 (s3://; AWS_ENDPOINT_URL covers MinIO/R2/B2), google-cloud-storage (gs://),
azure-storage-blob azure-identity (az://). Credentials: each provider's default chain only.
Expand Down
59 changes: 50 additions & 9 deletions src/evalshift_cli/captures/store_uri.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@
gs://<bucket>/<prefix>
az://<account>/<container>/<prefix>

``<prefix>`` is optional and may be empty; a trailing slash is stripped. Credentials never
belong in a URI: any ``@`` or ``?`` is rejected so nobody is tempted to embed a key or a
query parameter, and every adapter uses its provider's default credential chain instead.
``<prefix>`` is optional and may be empty; a trailing slash is stripped. An ``az://`` container
or prefix may not have an empty, ``.`` or ``..`` segment (see :func:`has_unsafe_segment`).
Credentials never belong in a URI: any ``@`` or ``?`` is rejected so nobody is tempted to embed
a key or a query parameter, and every adapter uses its provider's default credential chain
instead.

This module deliberately imports nothing from the project so :mod:`evalshift_cli.config.models`
can validate ``captures.store`` with it without an import cycle.
Expand Down Expand Up @@ -52,6 +54,20 @@ def package_for_module(module: str) -> str:
return _PACKAGE_FOR_MODULE[module]


def has_unsafe_segment(path: str) -> bool:
"""Whether ``path`` has an empty, ``.`` or ``..`` segment between its ``/`` separators.

Azure only. The Azure SDK puts the container and blob name into the request URL, and its HTTP
stack resolves dot segments there before sending: blob ``team/../../other/x.json`` in
container ``evals`` is read as blob ``x.json`` in container ``other``. So an Azure container,
prefix or key with such a segment is refused rather than allowed to reach outside the store's
prefix. Empty segments go too: a blob name is never meant to hold ``//``. S3 sends the key as
given and GCS percent-encodes ``/`` in object names, so both keep such keys literal and accept
them. Shared verbatim with the SDK.
"""
return any(segment in ("", ".", "..") for segment in path.split("/"))


@dataclass(frozen=True, slots=True)
class StoreURI:
"""A parsed ``captures.store`` URI.
Expand Down Expand Up @@ -83,10 +99,12 @@ def parse_store_uri(uri: str) -> StoreURI:
"""Parse ``uri`` against the grammar above.

Raises:
ValueError: for an unknown scheme, a missing bucket or container, or a URI carrying
credentials (``@``) or query parameters (``?``). The message names the accepted
forms so a config error is self-explanatory, and never echoes the value: a rejected
value may be a key, a signature or a pasted connection string.
ValueError: for an unknown scheme, a missing or malformed bucket, a missing container,
an Azure container or prefix with an empty, ``.`` or ``..`` segment, or a URI
carrying credentials (``@``) or query parameters (``?``). The message names the
accepted forms so a config error is self-explanatory, and never echoes the value: a
rejected value may be a key, a signature or a pasted connection string. Nor does it
chain another exception that quotes it.
"""
text = uri.strip()
if "@" in text:
Expand All @@ -100,7 +118,17 @@ def parse_store_uri(uri: str) -> StoreURI:
)
# No message below echoes the value either: a rejected value may be a pasted connection
# string or key. Only the parsed scheme, which cannot hold one, is quoted back.
parts = urlsplit(text)
# urlsplit rejects some netlocs itself (a bracketed host that is not an IP address, characters
# that NFKC-normalise to a delimiter), and two of those errors quote the netloc. Raise a fixed
# message instead, outside the ``except`` so urlsplit's error is not chained onto it either.
try:
parts = urlsplit(text)
except ValueError:
parts = None
if parts is None:
raise ValueError(
f"store URI names an invalid bucket or account; accepted forms: {STORE_URI_FORMS}"
)
if not parts.scheme:
raise ValueError(f"store URI has no scheme; accepted forms: {STORE_URI_FORMS}")
if parts.scheme not in _EXTRA_FOR_SCHEME:
Expand All @@ -117,8 +145,21 @@ def parse_store_uri(uri: str) -> StoreURI:
raise ValueError(
"Azure store URI names no container; expected az://<account>/<container>/<prefix>"
)
if has_unsafe_segment(path):
raise ValueError(
"Azure store URI has an empty, '.' or '..' segment in its container or prefix, "
"which the Azure client would resolve to another place; expected "
"az://<account>/<container>/<prefix>"
)
return StoreURI(scheme, parts.netloc, container, prefix.strip("/"))
return StoreURI(scheme, parts.netloc, None, path)


__all__ = ["STORE_URI_FORMS", "Scheme", "StoreURI", "package_for_module", "parse_store_uri"]
__all__ = [
"STORE_URI_FORMS",
"Scheme",
"StoreURI",
"has_unsafe_segment",
"package_for_module",
"parse_store_uri",
]
Loading
Loading