diff --git a/packages/challenges/agent-challenge/docker/review/phala_pre_launch.sh b/packages/challenges/agent-challenge/docker/review/phala_pre_launch.sh index b4c650a26..16a2430a9 100644 --- a/packages/challenges/agent-challenge/docker/review/phala_pre_launch.sh +++ b/packages/challenges/agent-challenge/docker/review/phala_pre_launch.sh @@ -59,7 +59,7 @@ echo "Starting login process..." # Check if Docker credentials exist if [[ -n "$DSTACK_DOCKER_USERNAME" && -n "$DSTACK_DOCKER_PASSWORD" ]]; then echo "Docker credentials found" - DOCKER_REGISTRY_TARGET="${DSTACK_DOCKER_REGISTRY:-docker.io}" + DOCKER_REGISTRY_TARGET="${DSTACK_DOCKER_REGISTRY:-ghcr.io}" echo "Target Docker registry: $DOCKER_REGISTRY_TARGET" # Check if already logged in diff --git a/packages/challenges/agent-challenge/docs/miner/self-deploy.md b/packages/challenges/agent-challenge/docs/miner/self-deploy.md index f1a5bda83..6304f2277 100644 --- a/packages/challenges/agent-challenge/docs/miner/self-deploy.md +++ b/packages/challenges/agent-challenge/docs/miner/self-deploy.md @@ -29,7 +29,7 @@ There is no closed agent-model catalog; personal finetunes are banned. Concepts: [Attestation TEE — agent-driven order](attestation-tee.md#agent-driven-order-package-verify--tree-sha--tee--eval). The mission is **CPU Intel TDX only** (no GPU) with a hard **$20** spend cap and a -preference for the smallest CPU shape that works (`tdx.small`/`tdx.medium`). GPU +per-stage CPU sizing (review `tdx.small` / 20 GB disk; eval `tdx.xlarge` / 100 GB disk so four concurrent Terminal-Bench tasks fit). GPU targets, over-cap shapes, and missing Phala credentials are refused **before** any Phala call. @@ -126,7 +126,9 @@ python -m agent_challenge.selfdeploy review deploy \ --openrouter-key-env OPENROUTER_API_KEY \ --phala-api https://cloud-api.phala.com/api/v1 \ --review-instance-type tdx.small \ - --eval-instance-type tdx.small \ + --eval-instance-type tdx.xlarge \ + --review-disk-size-gb 20 \ + --eval-disk-size-gb 100 \ --review-runtime-hours 6 \ --eval-runtime-hours 6 \ --money-cap-usd 20 @@ -145,7 +147,9 @@ python -m agent_challenge.selfdeploy review deploy \ --openrouter-key-env OPENROUTER_API_KEY \ --phala-api https://cloud-api.phala.com/api/v1 \ --review-instance-type tdx.small \ - --eval-instance-type tdx.small \ + --eval-instance-type tdx.xlarge \ + --review-disk-size-gb 20 \ + --eval-disk-size-gb 100 \ --review-runtime-hours 6 \ --eval-runtime-hours 6 \ --money-cap-usd 20 @@ -279,6 +283,11 @@ python -m agent_challenge.selfdeploy review retry \ ### `review teardown` +Deletes the review CVM via the Phala Cloud HTTP API (`DELETE /cvms/{id}`). +No external `phala` CLI binary is required. Pass `--cvm-id` from the deploy +output, or `--app-id` (the plan `app_identity`) to resolve a unique match from +`GET /cvms`. Ambiguous matches are refused. + Delete the review CVM after allow, reject, expiry, cancellation, provider failure, verification failure, or interruption. The CLI uses the exact `phala cvms delete -f` operation. If deletion fails, the command exits @@ -287,6 +296,8 @@ logs). ```bash python -m agent_challenge.selfdeploy review teardown --cvm-id review-cvm-1 +# or, when the deploy output was lost but app_identity is known: +python -m agent_challenge.selfdeploy review teardown --app-id ``` ### `eval prepare` @@ -321,6 +332,20 @@ spend projection counts both the review and eval stage shapes against the shared money cap. Signing again accepts either `--auto-sign` or explicit `--signature` + `--nonce` (+ optional `--timestamp`). +**Host posts the result — the guest only emits.** The measured guest orchestrator +emits the attested envelope on stdout (a line prefixed `BASE_BENCHMARK_RESULT=`) +and exits. It never calls `eval result`. The miner host must scrape that line +and post the exact bytes with the one-time `EVAL_RUN_TOKEN`. + +**Live deploy refuses to run without a token handoff.** Pass at least one of: + +- `--token-output PATH` (primary) — writes the token with mode `0600` +- `--emit-run-token` — adds `eval_run_token` to success stdout JSON + +Validate the handoff destination **before** prepare spends the single delivery. +Success stdout always includes `eval_run_id`. The token never appears in +`--output` plan JSON, redacted prepare/status output, logs, or exception text. + ```bash python -m agent_challenge.selfdeploy eval deploy \ --base-url https:// \ @@ -329,9 +354,25 @@ python -m agent_challenge.selfdeploy eval deploy \ --auto-sign \ --llm-cost-limit-env LLM_COST_LIMIT \ --phala-api https://cloud-api.phala.com/api/v1 \ - --eval-instance-type tdx.small \ - --money-cap-usd 20 -``` + --eval-instance-type tdx.xlarge \ + --review-disk-size-gb 20 \ + --eval-disk-size-gb 100 \ + --money-cap-usd 20 \ + --token-output ~/.cache/agent-challenge/eval-run.token +``` + +Capture `eval_run_id` from deploy stdout (and keep the `0600` token file for +`eval result`). Optional: `--expected-measurement PATH` compares plan `rtmr0` +to a local pin JSON before Phala create (mismatch prints truncated prefixes only). + +**Shape / measurement-pin footgun.** `--eval-instance-type` must match the +validator-issued plan `vm_shape` / `instance_type`. A shape change also requires +a matching `rtmr0` pin on the validator allowlist and a re-prepare. A stale pin +surfaces only as a generic key-release denial deep in the TEE flow — the CLI +now aborts before Phala create and names both shapes plus `vm_shape` / +`instance_type` / `rtmr0`. Prepare already spent the one-shot token on that +abort path; re-run `eval deploy` (cancel+retry recovery) or explicit +`eval cancel` + `eval retry` to re-prepare. Use `--dry-run` to validate the signed plan and show only safe names, digests, measurement metadata, and cost. Without a validator eval allowlist the dry-run @@ -349,6 +390,42 @@ python -m agent_challenge.selfdeploy eval deploy \ A post-create failure deletes the attributable eval CVM before the command returns. +### Host post path after eval CVM finishes + +1. Wait for the guest to finish. Scrape guest stdout for the line prefixed + `BASE_BENCHMARK_RESULT=` and write the **exact** payload bytes after the + prefix into a file (production envelopes are on the order of ~18KB — do not + pretty-print or re-encode). +2. Source the one-time token from the `0600` file without echoing it (no + `set -x`): + +```bash +export EVAL_RUN_TOKEN="$(cat ~/.cache/agent-challenge/eval-run.token)" +# never: echo "$EVAL_RUN_TOKEN" or set -x while the token is in the environment +``` + +3. Post with the deploy `eval_run_id`: + +```bash +python -m agent_challenge.selfdeploy eval result \ + --base-url https:// \ + --run-id \ + --result ./eval-result.json \ + --token-env EVAL_RUN_TOKEN +``` + +4. Tear down the eval CVM: + +```bash +python -m agent_challenge.selfdeploy eval teardown --cvm-id +# or, when only app_identity is known: +python -m agent_challenge.selfdeploy eval teardown --app-id +``` + +Invariant: `EVAL_RUN_TOKEN` never belongs in plan JSON, compose, ordinary logs, +or committed files — only the `0600` handoff file and/or one-shot stdout when +`--emit-run-token` is set. + ### `eval result` Post the exact CVM-emitted result bytes directly to @@ -538,7 +615,7 @@ python -m agent_challenge.selfdeploy verdict \ ### `deploy` Deploy a CPU-only, miner-funded CVM. Absent `--instance-type`, the smallest CPU -shape (`tdx.small`) is chosen. A GPU instance type or GPU OS image is refused, and +shape is stage-specific (review `tdx.small`, eval `tdx.xlarge`). A GPU instance type or GPU OS image is refused, and a shape whose projected cost would breach the money cap is refused, both before any provisioning. Use `--dry-run` to print the full plan (compose, image digest, instance type, region, key-release endpoint, projected cost) and make zero @@ -613,10 +690,11 @@ python -m agent_challenge.selfdeploy teardown --cvm-id cvm-1 Every CVM you deploy is **miner-funded** and must be deleted when you are done. The total mission spend cap is **$20**; always use the smallest CPU shape that -works (`tdx.small`/`tdx.medium`) and never deploy a GPU CVM. Review and eval +is stage-sized (review `tdx.small`/20 GB; eval `tdx.xlarge`/100 GB) and never deploy a GPU CVM. Disk is billed separately (~$0.000139/GB/hour) and is included in the projected $20 cap. Review and eval spend are projected together before either create. -The `teardown` subcommand runs `phala cvms delete -f` for you, but you can +The `teardown` subcommand issues `DELETE /cvms/{id}` through the Phala Cloud +HTTP client (no `phala` binary required). For manual verification you can still also delete and confirm directly with the `phala` CLI: ``` diff --git a/packages/challenges/agent-challenge/docs/validator/self-deploy.md b/packages/challenges/agent-challenge/docs/validator/self-deploy.md index 9fe6b146c..bbda4e9b8 100644 --- a/packages/challenges/agent-challenge/docs/validator/self-deploy.md +++ b/packages/challenges/agent-challenge/docs/validator/self-deploy.md @@ -274,12 +274,12 @@ emitter. Any review or eval CVM created for live verification is miner-funded and subject to the cumulative review+eval money cap of **$20**. Prefer the smallest CPU shape -(`tdx.small` / `tdx.medium`) and never a GPU shape. Miners must delete every +(review `tdx.small`/20 GB disk; eval `tdx.xlarge`/100 GB disk) and never a GPU shape. Disk billing (~$0.000139/GB/hour) is included in the shared $20 projection. Miners must delete every attributable CVM after success, reject, expiry, provider failure, quote failure, cancellation, interruption, or result failure. Confirm none remain: ```bash -phala cvms delete -f +HTTP `DELETE /cvms/{id}` (or manual `phala cvms delete -f`) phala cvms list ``` diff --git a/packages/challenges/agent-challenge/golden/live-registry-refs.json b/packages/challenges/agent-challenge/golden/live-registry-refs.json index 66b2f1715..d5457078d 100644 --- a/packages/challenges/agent-challenge/golden/live-registry-refs.json +++ b/packages/challenges/agent-challenge/golden/live-registry-refs.json @@ -1,25 +1,28 @@ { "schema": "harbor-independence/live-registry-refs@1", - "note": "SIDE manifest (NOT the frozen golden dataset digest). Maps a small deterministic subset of Terminal-Bench 2.1 task_ids to PULLABLE, digest-pinned registry refs published to the miner's public Docker Hub namespace so an in-CVM DooD orchestrator can `docker pull` them for a live smoke E2E. This file does NOT affect golden/dataset-digest.json, its per-task content_digest_sha256, the canonical_content_digest_sha256, or the canonical compose/measurement. Resolution is opt-in and fail-closed (see agent_challenge.canonical.live_registry).", + "note": "SIDE manifest (NOT the frozen golden dataset digest). Maps a small deterministic subset of Terminal-Bench 2.1 task_ids to PULLABLE, digest-pinned GHCR refs under ghcr.io/baseintelligence so an in-CVM DooD orchestrator can pull them for a live smoke E2E. D6: only GHCR shipping refs are accepted (see agent_challenge.canonical.live_registry). This file does NOT affect golden/dataset-digest.json, its per-task content_digest_sha256, the canonical_content_digest_sha256, or the canonical compose/measurement. Resolution is opt-in and fail-closed. OPS_REQUIRED: orchestrator_image digest is the T1 local buildx manifest (not yet pushed to GHCR); task image digests retain prior content hashes under the new GHCR path pattern and must be republished to GHCR before live pull succeeds.", "dataset": "terminal-bench/terminal-bench-2-1", - "namespace": "docker.io/mathiiss", - "orchestrator_image": "docker.io/mathiiss/agent-challenge-canonical@sha256:02331f0909f617e333f113be376d353770a673669946bcddaac3c53cbde7c9d8", - "orchestrator_image_source": "services.yaml build-canonical (`uv run python -m agent_challenge.canonical.build --build`) built reproducibly (BuildKit SOURCE_DATE_EPOCH + rewrite-timestamp, provenance/sbom off) and pushed to the miner Docker Hub namespace", + "namespace": "ghcr.io/baseintelligence", + "orchestrator_image": "ghcr.io/baseintelligence/agent-challenge-canonical@sha256:ea399cee1b3c9015024918a6070901e6b7b4bee432c8afe1e7dd40a95547e0bc", + "orchestrator_image_source": "T1 local agent-recipe buildx (SOURCE_DATE_EPOCH + rewrite-timestamp, provenance/sbom off) → manifest sha256:ea399cee1b3c9015024918a6070901e6b7b4bee432c8afe1e7dd40a95547e0bc; tag was ghcr.io/baseintelligence/agent-challenge-canonical:t1-local. OPS_REQUIRED: replace with the first published GHCR digest from agent-recipe publish-eval-image.yml before T3/prod pin.", "tasks": { "adaptive-rejection-sampler": { - "registry_ref": "docker.io/mathiiss/agent-challenge-tb21-adaptive-rejection-sampler@sha256:7c8bd5835f19506222805de68d65f83d6cca5b502f7d71dc5e2d9d4dd447c0c8", + "registry_ref": "ghcr.io/baseintelligence/agent-challenge-tb21-adaptive-rejection-sampler@sha256:7c8bd5835f19506222805de68d65f83d6cca5b502f7d71dc5e2d9d4dd447c0c8", "source_ref": "alexgshaw/adaptive-rejection-sampler:20251031", - "content_digest_sha256": "bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de" + "content_digest_sha256": "bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de", + "ops_status": "BLOCKED_REPUBLISH — path retargeted to GHCR; digest is historical content pin until image is pushed under ghcr.io/baseintelligence/" }, "bn-fit-modify": { - "registry_ref": "docker.io/mathiiss/agent-challenge-tb21-bn-fit-modify@sha256:c0371862a0861f282eb206471452ea9aa8d494e3687bdc8167c96ab39d73db50", + "registry_ref": "ghcr.io/baseintelligence/agent-challenge-tb21-bn-fit-modify@sha256:c0371862a0861f282eb206471452ea9aa8d494e3687bdc8167c96ab39d73db50", "source_ref": "alexgshaw/bn-fit-modify:20251031", - "content_digest_sha256": "b5f9644970c17ad9ddb46b7266f7bcd87c761d77d7e6f55d7cfe7284d5ff66e9" + "content_digest_sha256": "b5f9644970c17ad9ddb46b7266f7bcd87c761d77d7e6f55d7cfe7284d5ff66e9", + "ops_status": "BLOCKED_REPUBLISH — path retargeted to GHCR; digest is historical content pin until image is pushed under ghcr.io/baseintelligence/" }, "break-filter-js-from-html": { - "registry_ref": "docker.io/mathiiss/agent-challenge-tb21-break-filter-js-from-html@sha256:2a5bd51bab582993befc1a24252c03af673cb1db7cf7d0b2195d66a42a3872a7", + "registry_ref": "ghcr.io/baseintelligence/agent-challenge-tb21-break-filter-js-from-html@sha256:2a5bd51bab582993befc1a24252c03af673cb1db7cf7d0b2195d66a42a3872a7", "source_ref": "alexgshaw/break-filter-js-from-html:20251031", - "content_digest_sha256": "678008d1a4fd1e6e1b9b3cc9a327219fe4b410a31eafc52e9099bbf947eea600" + "content_digest_sha256": "678008d1a4fd1e6e1b9b3cc9a327219fe4b410a31eafc52e9099bbf947eea600", + "ops_status": "BLOCKED_REPUBLISH — path retargeted to GHCR; digest is historical content pin until image is pushed under ghcr.io/baseintelligence/" } } } diff --git a/packages/challenges/agent-challenge/src/agent_challenge/canonical/live_registry.py b/packages/challenges/agent-challenge/src/agent_challenge/canonical/live_registry.py index afe7987bf..e04250f89 100644 --- a/packages/challenges/agent-challenge/src/agent_challenge/canonical/live_registry.py +++ b/packages/challenges/agent-challenge/src/agent_challenge/canonical/live_registry.py @@ -4,12 +4,15 @@ Terminal-Bench task by a *bare content digest* (``harbor_registry_ref = "sha256:<64hex>"``) with NO repository, so an in-CVM DooD orchestrator cannot ``docker pull`` it. For a live smoke E2E a small deterministic subset of task -images is published to the miner's public Docker Hub namespace as pullable, -digest-pinned refs and recorded in a SEPARATE side manifest +images is published to the org GHCR namespace (``ghcr.io/baseintelligence/…``) +as pullable, digest-pinned refs and recorded in a SEPARATE side manifest (``golden/live-registry-refs.json``). That side manifest never touches ``dataset-digest.json`` -- the frozen content digests and the canonical measurement stay byte-identical. +D6: shipping live refs MUST be GHCR. ``docker.io`` and the retired +``mathiiss`` namespace are rejected fail-closed at parse time. + Resolution is **opt-in and fail-closed**: with no live manifest configured (no explicit path, no env var) callers get NO live refs and fall back to the existing per-task behavior, so flag-off / offline runs are byte-identical. When a @@ -50,6 +53,9 @@ # and an un-namespaced ``name@sha256`` so only a real registry ref is accepted. _PULLABLE_REF_RE = re.compile(r"^(?=[^@]*/)[A-Za-z0-9][\w.\-/:]*@sha256:[0-9a-f]{64}$") +#: Shipping live-registry refs must live under this registry host prefix (D6). +LIVE_REGISTRY_HOST_PREFIX = "ghcr.io/" + class LiveRegistryError(ValueError): """The live-registry side manifest is missing, malformed, or not pullable.""" @@ -83,6 +89,32 @@ def assert_pullable_ref(ref: Any, *, what: str = "registry ref") -> str: return ref # type: ignore[return-value] +def assert_live_registry_ref(ref: Any, *, what: str = "registry ref") -> str: + """Return ``ref`` if it is a GHCR digest-pinned shipping ref (D6), else raise. + + Builds on :func:`assert_pullable_ref`, then rejects ``docker.io``, the + retired ``mathiiss`` namespace, and any non-GHCR host so live smoke never + pins a Hub image again. + """ + + pinned = assert_pullable_ref(ref, what=what) + lowered = pinned.lower() + if "mathiiss" in lowered: + raise LiveRegistryError( + f"{what} must not use the retired mathiiss namespace (D6), got {pinned!r}" + ) + if lowered.startswith("docker.io/") or lowered.startswith("index.docker.io/"): + raise LiveRegistryError( + f"{what} must not use docker.io (D6 — GHCR only), got {pinned!r}" + ) + if not lowered.startswith(LIVE_REGISTRY_HOST_PREFIX): + raise LiveRegistryError( + f"{what} must be a GHCR digest-pinned ref " + f"({LIVE_REGISTRY_HOST_PREFIX}…@sha256:<64hex>), got {pinned!r}" + ) + return pinned + + @dataclass(frozen=True) class LiveRegistry: """Parsed live-registry side manifest. @@ -111,7 +143,7 @@ def __bool__(self) -> bool: def _ref_from_entry(task_id: str, entry: Any) -> str: - """Extract + validate the pullable ref from a manifest task entry.""" + """Extract + validate the pullable GHCR ref from a manifest task entry.""" if isinstance(entry, str): ref = entry @@ -119,15 +151,16 @@ def _ref_from_entry(task_id: str, entry: Any) -> str: ref = entry.get("registry_ref") else: raise LiveRegistryError(f"live-registry task {task_id!r} is not a string or mapping") - return assert_pullable_ref(ref, what=f"live-registry ref for task {task_id!r}") + return assert_live_registry_ref(ref, what=f"live-registry ref for task {task_id!r}") def parse_live_registry(data: Mapping[str, Any]) -> LiveRegistry: """Parse + validate a live-registry side-manifest document. - Every task ref must be a pullable ``repo@sha256`` ref (a bare content digest - or floating tag is rejected). ``orchestrator_image``, when present, must be - pullable too. Task keys are normalized to their bare name. + Every task ref must be a GHCR pullable ``repo@sha256`` ref (a bare content + digest, floating tag, ``docker.io``, or retired ``mathiiss`` namespace is + rejected). ``orchestrator_image``, when present, must be GHCR-pullable too. + Task keys are normalized to their bare name. """ if not isinstance(data, Mapping): @@ -143,7 +176,7 @@ def parse_live_registry(data: Mapping[str, Any]) -> LiveRegistry: orchestrator = data.get("orchestrator_image") if orchestrator is not None: - orchestrator = assert_pullable_ref(orchestrator, what="orchestrator_image") + orchestrator = assert_live_registry_ref(orchestrator, what="orchestrator_image") return LiveRegistry(orchestrator_image=orchestrator, task_refs=task_refs, raw=dict(data)) @@ -203,9 +236,11 @@ def resolve_live_registry_refs( "DEFAULT_LIVE_REGISTRY_PATH", "LIVE_REGISTRY_ENV", "LIVE_REGISTRY_FILENAME", + "LIVE_REGISTRY_HOST_PREFIX", "LIVE_REGISTRY_SCHEMA", "LiveRegistry", "LiveRegistryError", + "assert_live_registry_ref", "assert_pullable_ref", "is_pullable_ref", "load_live_registry", diff --git a/packages/challenges/agent-challenge/src/agent_challenge/evaluation/direct_result.py b/packages/challenges/agent-challenge/src/agent_challenge/evaluation/direct_result.py index 0e0fabd37..eddadc2cc 100644 --- a/packages/challenges/agent-challenge/src/agent_challenge/evaluation/direct_result.py +++ b/packages/challenges/agent-challenge/src/agent_challenge/evaluation/direct_result.py @@ -686,6 +686,18 @@ async def _load_review_envelope_for_run( ) -> Mapping[str, Any] | str | None: """Load receipted review-domain envelope for score-chain re-verify.""" + materials = await _load_review_materials_for_run(session, run) + if materials is None: + return None + return materials.get("envelope") + + +async def _load_review_materials_for_run( + session: AsyncSession, + run: EvalRun, +) -> dict[str, Any] | None: + """Load envelope + verification outcome (package residual) for score chain.""" + submission = await session.scalar( select(AgentSubmission).where(AgentSubmission.id == run.submission_id) ) @@ -703,9 +715,32 @@ async def _load_review_envelope_for_run( ): return None envelope = assignment.review_report_envelope_json + outcome_raw = assignment.review_verification_outcome_json + outcome: Mapping[str, Any] | None = None + if isinstance(outcome_raw, str) and outcome_raw.strip(): + try: + import json as _json + + parsed = _json.loads(outcome_raw) + if isinstance(parsed, dict): + outcome = parsed + except (TypeError, ValueError): + outcome = None + elif isinstance(outcome_raw, dict): + outcome = outcome_raw + residual = None + if isinstance(outcome, Mapping): + pr = outcome.get("package_residual") + if isinstance(pr, dict): + residual = pr + env_out: Mapping[str, Any] | str | None = None if isinstance(envelope, str) and envelope: - return envelope - return None + env_out = envelope + elif isinstance(envelope, dict): + env_out = envelope + if env_out is None and residual is None and outcome is None: + return None + return {"envelope": env_out, "outcome": outcome, "package_residual": residual} async def _run_gate_with_deadline( @@ -719,6 +754,8 @@ async def _run_gate_with_deadline( deadline_seconds: float, dual_flags_on: bool = False, review_envelope: Mapping[str, Any] | str | bytes | None = None, + review_outcome: Mapping[str, Any] | None = None, + package_residual: Mapping[str, Any] | None = None, key_release_grant: Mapping[str, Any] | None = None, agent_llm_kwargs: Mapping[str, Any] | None = None, settings: ChallengeSettings | None = None, @@ -756,6 +793,8 @@ def _decide() -> AttestationDecision: settings_dual_flags_on=True, eval_plan=plan, review_envelope=review_envelope, + review_outcome=review_outcome, + package_residual=package_residual, key_release_grant=key_release_grant, key_granted_flag=key_granted, score_binding=binding, @@ -892,8 +931,14 @@ async def process_direct_eval_result( review_envelope: Mapping[str, Any] | str | None = None key_release_grant: Mapping[str, Any] | None = None agent_llm_kwargs: dict[str, Any] | None = None + review_outcome: Mapping[str, Any] | None = None + package_residual: Mapping[str, Any] | None = None if dual_flags_on: - review_envelope = await _load_review_envelope_for_run(session, current) + materials = await _load_review_materials_for_run(session, current) + if materials is not None: + review_envelope = materials.get("envelope") # type: ignore[assignment] + review_outcome = materials.get("outcome") # type: ignore[assignment] + package_residual = materials.get("package_residual") # type: ignore[assignment] key_release_grant = _key_release_grant_from_result( plan=plan, validated=validated, @@ -931,6 +976,8 @@ async def process_direct_eval_result( deadline_seconds=settings.eval_result_verifier_deadline_seconds, dual_flags_on=dual_flags_on, review_envelope=review_envelope, + review_outcome=review_outcome, + package_residual=package_residual, key_release_grant=key_release_grant, agent_llm_kwargs=agent_llm_kwargs, settings=settings, diff --git a/packages/challenges/agent-challenge/src/agent_challenge/evaluation/progress.py b/packages/challenges/agent-challenge/src/agent_challenge/evaluation/progress.py new file mode 100644 index 000000000..983b7370a --- /dev/null +++ b/packages/challenges/agent-challenge/src/agent_challenge/evaluation/progress.py @@ -0,0 +1,241 @@ +"""Mid-run attested Eval progress ingest (observability only). + +The CVM posts per-task phase transitions with the same Bearer ``EVAL_RUN_TOKEN`` +used for the final result route. Events land in ``TaskLogEvent`` so the existing +SSE feed surfaces them during the run. This path never mutates scores. +""" + +from __future__ import annotations + +import json +from collections.abc import Mapping +from typing import Any + +from sqlalchemy import select +from sqlalchemy.ext.asyncio import AsyncSession + +from agent_challenge.canonical import eval_wire +from agent_challenge.core.models import EvalRun, TaskLogEvent +from agent_challenge.evaluation.authorization import load_eval_run_plan +from agent_challenge.evaluation.task_events import ( + SAFE_TASK_PHASE_STATUSES, + record_task_event, +) + +PROGRESS_SOURCE = "eval_progress" +MAX_PROGRESS_BODY_BYTES = 16 * 1024 + + +class EvalProgressError(ValueError): + """Schema or policy failure for a progress ingest request.""" + + def __init__(self, message: str, *, code: str) -> None: + super().__init__(message) + self.code = code + + +def _metadata_dict(raw: str) -> dict[str, Any]: + try: + parsed = json.loads(raw) + except json.JSONDecodeError: + return {} + return parsed if isinstance(parsed, dict) else {} + + +async def _existing_progress_event( + session: AsyncSession, + *, + submission_id: int, + eval_run_id: str, + task_id: str, + client_sequence: int, +) -> TaskLogEvent | None: + rows = await session.scalars( + select(TaskLogEvent).where( + TaskLogEvent.submission_id == submission_id, + TaskLogEvent.task_id == task_id, + TaskLogEvent.event_type.in_(("task.status", "task.progress")), + ) + ) + for event in rows: + meta = _metadata_dict(event.metadata_json) + if ( + meta.get("source") == PROGRESS_SOURCE + and meta.get("eval_run_id") == eval_run_id + and meta.get("client_sequence") == client_sequence + ): + return event + return None + + +async def _max_client_sequence( + session: AsyncSession, + *, + submission_id: int, + eval_run_id: str, + task_id: str, +) -> int: + rows = await session.scalars( + select(TaskLogEvent).where( + TaskLogEvent.submission_id == submission_id, + TaskLogEvent.task_id == task_id, + TaskLogEvent.event_type.in_(("task.status", "task.progress")), + ) + ) + highest = 0 + for event in rows: + meta = _metadata_dict(event.metadata_json) + if meta.get("source") != PROGRESS_SOURCE or meta.get("eval_run_id") != eval_run_id: + continue + seq = meta.get("client_sequence") + if isinstance(seq, int) and seq > highest: + highest = seq + return highest + + +def _progress_receipt( + *, + eval_run_id: str, + task_id: str, + sequence: int, + event_id: int, + created: bool, +) -> dict[str, Any]: + return eval_wire.validate_eval_progress_receipt( + { + "schema_version": 1, + "eval_run_id": eval_run_id, + "task_id": task_id, + "sequence": sequence, + "event_id": event_id, + "created": created, + } + ) + + +async def process_eval_progress( + session: AsyncSession, + *, + run: EvalRun, + progress_request: Mapping[str, Any], +) -> tuple[dict[str, Any], bool]: + """Validate and record one mid-run progress event. + + Returns ``(receipt, created)``. ``created`` is False on idempotent replay of + the same ``(eval_run_id, task_id, sequence)``. Never mutates ``EvalRun.score`` + or any score columns. + """ + + # Fail closed if wire taxonomy drifts from the public SSE phase set. + if SAFE_TASK_PHASE_STATUSES != eval_wire.EVAL_PROGRESS_PHASES: + raise EvalProgressError( + "progress phase taxonomy mismatch", + code="progress_phase_taxonomy", + ) + + try: + validated = eval_wire.validate_eval_progress_request(progress_request) + except eval_wire.EvalWireError as exc: + message = str(exc) + if "forbids score fields" in message: + raise EvalProgressError(message, code="progress_score_forbidden") from exc + if "status is not a safe task phase" in message: + raise EvalProgressError(message, code="progress_phase_invalid") from exc + raise EvalProgressError(message, code="progress_invalid") from exc + + if validated["eval_run_id"] != run.eval_run_id: + raise EvalProgressError( + "progress run does not match route", + code="progress_run_mismatch", + ) + + plan = load_eval_run_plan(run) + if validated["submission_id"] != plan["submission_id"]: + raise EvalProgressError( + "progress submission_id does not match eval plan", + code="progress_submission_mismatch", + ) + + allowed_tasks = {item["task_id"] for item in plan["selected_tasks"]} + if validated["task_id"] not in allowed_tasks: + raise EvalProgressError( + "progress task_id is not in the eval plan", + code="progress_task_unknown", + ) + + existing = await _existing_progress_event( + session, + submission_id=run.submission_id, + eval_run_id=run.eval_run_id, + task_id=validated["task_id"], + client_sequence=validated["sequence"], + ) + if existing is not None: + return ( + _progress_receipt( + eval_run_id=run.eval_run_id, + task_id=validated["task_id"], + sequence=validated["sequence"], + event_id=existing.id, + created=False, + ), + False, + ) + + highest = await _max_client_sequence( + session, + submission_id=run.submission_id, + eval_run_id=run.eval_run_id, + task_id=validated["task_id"], + ) + if validated["sequence"] <= highest: + raise EvalProgressError( + "progress sequence must be monotone per task", + code="progress_sequence_stale", + ) + + message = validated["message"] + if message is None: + message = f"task {validated['task_id']} {validated['status']}" + + events = await record_task_event( + session, + submission_id=run.submission_id, + job_id=None, + task_id=validated["task_id"], + event_type=validated["event_type"], + message=message, + progress=validated["progress"], + status=validated["status"], + metadata={ + "source": PROGRESS_SOURCE, + "eval_run_id": run.eval_run_id, + "client_sequence": validated["sequence"], + "phase": validated["status"], + }, + ) + if not events: + raise EvalProgressError( + "progress event was not recorded", + code="progress_not_recorded", + ) + event = events[0] + await session.flush() + return ( + _progress_receipt( + eval_run_id=run.eval_run_id, + task_id=validated["task_id"], + sequence=validated["sequence"], + event_id=event.id, + created=True, + ), + True, + ) + + +__all__ = [ + "MAX_PROGRESS_BODY_BYTES", + "PROGRESS_SOURCE", + "EvalProgressError", + "process_eval_progress", +] diff --git a/packages/challenges/agent-challenge/src/agent_challenge/evaluation/score_chain_gate.py b/packages/challenges/agent-challenge/src/agent_challenge/evaluation/score_chain_gate.py index 989b4889f..251ae0330 100644 --- a/packages/challenges/agent-challenge/src/agent_challenge/evaluation/score_chain_gate.py +++ b/packages/challenges/agent-challenge/src/agent_challenge/evaluation/score_chain_gate.py @@ -409,6 +409,11 @@ def admit_production_score_from_chain( review_report_data_hex: str | None = None, review_domain: str | None = None, cached_review_allow: bool = False, + # AGATE: residual is bound on review verification outcome (and optionally + # envelope). Score admission must pass outcome so package_residual_missing + # is not raised when residual lives only on the outcome bag. + review_outcome: Mapping[str, Any] | None = None, + package_residual: Mapping[str, Any] | None = None, # Key-release leg (RA-TLS) key_release_grant: Mapping[str, Any] | None = None, key_granted_flag: bool = False, @@ -539,6 +544,8 @@ def admit_production_score_from_chain( dual_flags_on=dual_flags_on, require_package_residual=bool(dual_flags_on), expected_package_tree_sha=plan_tree_sha if dual_flags_on else None, + outcome=review_outcome, + package_residual=package_residual, ) if not review_decision.may_launch: code = review_decision.reason_code @@ -713,6 +720,8 @@ def admit_production_score_for_eval_result( settings_dual_flags_on: bool, eval_plan: Mapping[str, Any], review_envelope: Mapping[str, Any] | str | bytes | None, + review_outcome: Mapping[str, Any] | None = None, + package_residual: Mapping[str, Any] | None = None, key_release_grant: Mapping[str, Any] | None, key_granted_flag: bool, score_binding: Mapping[str, Any] | None, @@ -756,6 +765,8 @@ def admit_production_score_for_eval_result( return admit_production_score_from_chain( dual_flags_on=settings_dual_flags_on, review_envelope=review_envelope, + review_outcome=review_outcome, + package_residual=package_residual, key_release_grant=key_release_grant, key_granted_flag=key_granted_flag, eval_plan=eval_plan, diff --git a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/cli.py b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/cli.py index 63dcdff26..63aeec037 100644 --- a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/cli.py +++ b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/cli.py @@ -23,6 +23,7 @@ import subprocess import sys from collections.abc import Callable, Mapping, Sequence +from dataclasses import replace from pathlib import Path from typing import Any @@ -37,6 +38,7 @@ DEFAULT_PHALA_API, PhalaApiError, PhalaCloudClient, + resolve_cvm_id_from_list, ) from agent_challenge.selfdeploy.plan import ( CredentialError, @@ -49,12 +51,32 @@ write_prepared, ) from agent_challenge.selfdeploy.shapes import ( + DEFAULT_EVAL_DISK_SIZE_GB, + DEFAULT_EVAL_INSTANCE_TYPE, DEFAULT_MAX_RUNTIME_HOURS, DEFAULT_MONEY_CAP_USD, DEFAULT_OS_IMAGE, + DEFAULT_REVIEW_DISK_SIZE_GB, + DEFAULT_REVIEW_INSTANCE_TYPE, ShapeError, + validate_disk_size, ) + +def _with_disk_size(plan: Any, disk_size_gb: int) -> Any: + """Attach validated disk size to a deployment plan (dataclass or test double).""" + + try: + return replace(plan, disk_size_gb=disk_size_gb) + except TypeError: + # Offline tests inject SimpleNamespace plan doubles. + try: + plan.disk_size_gb = disk_size_gb + except Exception: + pass + return plan + + PROG = "agent-challenge-selfdeploy" #: A Phala deployer: (plan, out_dir) -> arbitrary result (printed by the CLI). @@ -122,24 +144,111 @@ def _bounded_text(value: str | None, *, limit: int = _TEARDOWN_DIAGNOSTIC_LIMIT) return text[: limit - 3] + "..." -def default_phala_teardown(cvm_id: str) -> dict[str, Any]: # pragma: no cover - """Delete a CVM via ``phala cvms delete -f`` (idempotent; live, M6). +def default_phala_teardown( + cvm_id: str, + *, + client: PhalaCloudClient | None = None, +) -> dict[str, Any]: + """Delete a CVM via Phala Cloud HTTP ``DELETE /cvms/{id}`` (idempotent). - Always returns a structured result with ``ok``/``returncode`` so callers can - fail non-zero when deletion does not succeed. Stdout/stderr are bounded. + Uses :class:`PhalaCloudClient` — never shells out to a ``phala`` binary + (the binary is not present on production validator containers). 204 and + 404 from the API are success. Always returns a structured result with + ``ok``/``returncode`` so callers can fail non-zero when deletion fails. """ - proc = subprocess.run(["phala", "cvms", "delete", cvm_id, "-f"], capture_output=True, text=True) - ok = proc.returncode == 0 + try: + api = client if client is not None else PhalaCloudClient() + api.delete_cvm(cvm_id) + except (PhalaApiError, CredentialError) as exc: + return { + "returncode": 1, + "ok": False, + "stdout": "", + "stderr": "", + "error": str(exc), + } + except Exception as exc: # noqa: BLE001 - surface unexpected transport failures + return { + "returncode": 1, + "ok": False, + "stdout": "", + "stderr": "", + "error": f"teardown failed: {exc.__class__.__name__}", + } return { - "returncode": proc.returncode, - "ok": ok, - "stdout": _bounded_text(proc.stdout), - "stderr": _bounded_text(proc.stderr), - "error": None if ok else "phala cvms delete failed", + "returncode": 0, + "ok": True, + "stdout": "", + "stderr": "", + "error": None, } +def resolve_teardown_cvm_id( + *, + cvm_id: str | None, + app_id: str | None, + client: PhalaCloudClient | None = None, +) -> str: + """Resolve the CVM id for teardown: explicit id, else unique app_id match.""" + + explicit = (cvm_id or "").strip() + if explicit: + return explicit + identity = (app_id or "").strip() + if not identity: + raise RouteClientError("teardown requires --cvm-id or --app-id") + api = client if client is not None else PhalaCloudClient() + listing = api.get("/cvms") + resolved = resolve_cvm_id_from_list(listing, app_id=identity, require_unique=True) + if not resolved: + raise RouteClientError(f"no CVM found for app_id {identity!r}") + return resolved + + +def _run_teardown_command(args: argparse.Namespace) -> tuple[dict[str, Any], int]: + """Resolve CVM identity and tear down via HTTP (review/eval/top-level).""" + + phala_base = getattr(args, "phala_api", None) or DEFAULT_PHALA_API + try: + # Refuse missing identity before constructing a credentialed client. + explicit = (getattr(args, "cvm_id", None) or "").strip() + app_id = (getattr(args, "app_id", None) or "").strip() + if not explicit and not app_id: + raise RouteClientError("teardown requires --cvm-id or --app-id") + client = PhalaCloudClient(base_url=str(phala_base)) + cvm_id = resolve_teardown_cvm_id( + cvm_id=explicit or None, + app_id=app_id or None, + client=client, + ) + outcome = default_phala_teardown(cvm_id, client=client) + except (RouteClientError, CredentialError, PhalaApiError) as exc: + # Fail closed with a structured payload when identity cannot be resolved. + missing = getattr(args, "cvm_id", None) or getattr(args, "app_id", None) or "" + return ( + { + "torn_down": missing or None, + "ok": False, + "diagnostics": { + "returncode": 1, + "error": str(exc), + "stdout": "", + "stderr": "", + }, + "result": { + "returncode": 1, + "error": str(exc), + "stdout": "", + "stderr": "", + }, + }, + 2 if isinstance(exc, RouteClientError) else 1, + ) + return _teardown_payload(cvm_id, outcome) + + def _teardown_payload(cvm_id: str, result: Any) -> tuple[dict[str, Any], int]: """Project teardown outcome as a miner-facing payload and exit code.""" @@ -241,6 +350,77 @@ def _obtain_review_prepare_with_token( return retried +def _eval_token_present(prepare_response: Mapping[str, Any] | None) -> bool: + """True when prepare/retry still delivers the one-shot EVAL_RUN_TOKEN. + + Accepted shape matches ``eval.build_eval_deployment_plan``: + ``secret_delivery == {"env_key", "token"}`` with a non-empty token string. + Does not relax that contract — only decides whether recovery is needed. + """ + + if not isinstance(prepare_response, Mapping): + return False + delivery = prepare_response.get("secret_delivery") + if not isinstance(delivery, Mapping) or set(delivery) != {"env_key", "token"}: + return False + token = delivery.get("token") + return isinstance(token, str) and bool(token) + + +def _eval_run_id_from_prepare(prepare_response: Mapping[str, Any]) -> str | None: + """Extract current eval_run_id from a prepare wrapper without requiring a token.""" + + plan = prepare_response.get("plan") + if isinstance(plan, Mapping): + run_id = plan.get("eval_run_id") + if isinstance(run_id, str) and run_id: + return run_id + return None + + +def _obtain_eval_prepare_with_token( + client: SelfDeployRouteClient, + submission_id: int, +) -> dict[str, Any]: + """Return a prepare/retry response that still delivers EVAL_RUN_TOKEN. + + Product residual timeline (live production, submission 3): + - ``EVAL_RUN_TOKEN`` is delivered once per attempt. Standalone + ``eval prepare`` or ``eval retry`` permanently spends it. + - ``eval deploy`` previously called ``eval_prepare`` raw; when + ``secret_delivery`` was null, ``build_eval_deployment_plan`` hard-failed + with "first Eval prepare must deliver exactly one EVAL_RUN_TOKEN + capability". Attempts 1 and 2 both stuck; lifecycle unrecoverable + from the CLI. + - Mirror review: prepare → if token-less, resolve ``eval_run_id``, + ``eval_cancel`` then ``eval_retry``, return the response that carries + the token. Never cancel when the token was already delivered. Never + fabricate, cache, or persist a token offline. + """ + + response = client.eval_prepare(submission_id) + if _eval_token_present(response): + return response + run_id = _eval_run_id_from_prepare(response) + if not run_id: + raise RouteClientError("eval run has no current eval_run_id to refresh capability") + # Sticky token-less after prior prepare/retry consumer: cancel+retry for a + # fresh attempt that redelivers EVAL_RUN_TOKEN. Prefer not cancelling when + # prepare already carried the capability (handled above). + try: + client.eval_cancel(submission_id, run_id) + except RouteClientError: + # Terminal / already cancelled: still try retry against that id. + pass + retried = client.eval_retry(submission_id, run_id) + if not _eval_token_present(retried): + raise RouteClientError( + "eval run token unavailable after prepare and retry; " + "capability may be spent or run not retryable" + ) + return retried + + def _review_allowlist_verdict(plan: review_deploy.ReviewDeploymentPlan) -> str: """Compute a verified review-domain allowlist verdict or explicit UNKNOWN.""" @@ -407,7 +587,7 @@ def _ordered_review_command(args: argparse.Namespace) -> int: _print(_redact_capabilities(payload)) return 0 if args.review_command == "teardown": - payload, code = _teardown_payload(args.cvm_id, default_phala_teardown(args.cvm_id)) + payload, code = _run_teardown_command(args) _print(payload) return code if args.review_command == "deploy": @@ -437,12 +617,21 @@ def _ordered_review_command(args: argparse.Namespace) -> int: raise RouteClientError( "review deployment shape differs from the validator-issued assignment" ) + review_disk = validate_disk_size( + getattr(args, "review_disk_size_gb", DEFAULT_REVIEW_DISK_SIZE_GB) + ) + eval_disk = validate_disk_size( + getattr(args, "eval_disk_size_gb", DEFAULT_EVAL_DISK_SIZE_GB) + ) + plan = _with_disk_size(plan, review_disk) lifecycle.validate_lifecycle_budget( review_instance_type=plan.instance_type, eval_instance_type=args.eval_instance_type, review_runtime_hours=args.review_runtime_hours, eval_runtime_hours=args.eval_runtime_hours, money_cap_usd=args.money_cap_usd, + review_disk_size_gb=review_disk, + eval_disk_size_gb=eval_disk, ) key = os.environ.get(args.openrouter_key_env, "") if not args.dry_run and not key: @@ -511,6 +700,7 @@ def _ordered_review_command(args: argparse.Namespace) -> int: RouteClientError, CredentialError, lifecycle.LifecycleBudgetError, + ShapeError, review_deploy.ReviewDeploymentError, PhalaApiError, ) as exc: @@ -588,7 +778,7 @@ def _ordered_eval_command(args: argparse.Namespace) -> int: ) return 0 if args.eval_command == "teardown": - payload, code = _teardown_payload(args.cvm_id, default_phala_teardown(args.cvm_id)) + payload, code = _run_teardown_command(args) _print(payload) return code if args.eval_command == "deploy": @@ -597,18 +787,30 @@ def _ordered_eval_command(args: argparse.Namespace) -> int: "Eval deploy does not accept persisted prepare capabilities; " "run it with signed production route credentials" ) - raw = _route_client(args).eval_prepare(args.submission_id) + # Validate the handoff destination BEFORE any remote mutation: the + # prepare below spends the single EVAL_RUN_TOKEN delivery, and the + # guest never posts, so gating later would strand the miner with a + # burnt token and no way to call eval result. + _require_eval_run_token_handoff(args) + client = _route_client(args) + raw = _obtain_eval_prepare_with_token(client, args.submission_id) plan = eval_deploy.build_eval_deployment_plan(raw) - if plan.instance_type != args.eval_instance_type: - raise RouteClientError( - "Eval deployment shape differs from the validator-issued plan" - ) + _assert_eval_deploy_shape_and_measurement_pin(plan, args) + review_disk = validate_disk_size( + getattr(args, "review_disk_size_gb", DEFAULT_REVIEW_DISK_SIZE_GB) + ) + eval_disk = validate_disk_size( + getattr(args, "eval_disk_size_gb", DEFAULT_EVAL_DISK_SIZE_GB) + ) + plan = _with_disk_size(plan, eval_disk) lifecycle.validate_lifecycle_budget( review_instance_type=args.review_instance_type, eval_instance_type=plan.instance_type, review_runtime_hours=args.review_runtime_hours, eval_runtime_hours=args.eval_runtime_hours, money_cap_usd=args.money_cap_usd, + review_disk_size_gb=review_disk, + eval_disk_size_gb=eval_disk, ) values = { "EVAL_RUN_TOKEN": plan.eval_run_token, @@ -698,11 +900,13 @@ def _ordered_eval_command(args: argparse.Namespace) -> int: ) raise _print( - { - "stage": "eval_deployed", - "acknowledgement": acknowledgement, - "encrypted_env_names": list(encrypted.env_keys), - } + _hand_off_eval_run_token( + args, + eval_run_id=plan.eval_run_id, + eval_run_token=plan.eval_run_token, + acknowledgement=acknowledgement, + encrypted_env_names=encrypted.env_keys, + ) ) return 0 _print( @@ -724,6 +928,7 @@ def _ordered_eval_command(args: argparse.Namespace) -> int: RouteClientError, CredentialError, lifecycle.LifecycleBudgetError, + ShapeError, eval_deploy.EvalDeploymentError, PhalaApiError, ) as exc: @@ -768,6 +973,122 @@ def _redact_capabilities(value: Any) -> Any: return value +def _assert_eval_deploy_shape_and_measurement_pin( + plan: eval_deploy.EvalDeploymentPlan, + args: argparse.Namespace, +) -> None: + """Fail closed before Phala create on shape or optional rtmr0 pin mismatch. + + Shape check needs the built plan, so it runs after prepare has spent the + one-shot EVAL_RUN_TOKEN delivery. Next deploy recovers via + ``_obtain_eval_prepare_with_token`` cancel+retry when prepare is sticky-null. + """ + + requested = str(getattr(args, "eval_instance_type", "") or "").strip() + if plan.instance_type != requested: + plan_vm_shape = plan.measurement.get("vm_shape") + plan_vm = ( + str(plan_vm_shape).replace("-", ".") + if isinstance(plan_vm_shape, str) and plan_vm_shape + else plan.instance_type + ) + plan_rtmr0 = plan.measurement.get("rtmr0") + raise RouteClientError( + measure.format_eval_shape_mismatch_error( + plan_instance_type=plan.instance_type, + requested_instance_type=requested, + plan_vm_shape=plan_vm, + plan_rtmr0=plan_rtmr0 if isinstance(plan_rtmr0, str) else None, + ) + ) + expected_path = getattr(args, "expected_measurement", None) + if isinstance(expected_path, str) and expected_path.strip(): + try: + expected = measure.load_expected_measurement_mapping(expected_path.strip()) + pin_error = measure.compare_plan_rtmr0_to_expected(plan.measurement, expected) + except measure.MeasurementError as exc: + raise RouteClientError(str(exc)) from exc + if pin_error is not None: + raise RouteClientError(pin_error) + + +def _require_eval_run_token_handoff(args: argparse.Namespace) -> None: + """Fail closed on live eval deploy unless the miner can recover the token. + + ``EVAL_RUN_TOKEN`` is a one-shot capability. prepare/status redact it, and + the guest only emits the attested envelope — the host must post via + ``eval result``. Without an explicit handoff at deploy time the miner has + no path to submit. Dry-run skips this gate (no spend, no post). + """ + + if getattr(args, "dry_run", False): + return + token_output = getattr(args, "token_output", None) + emit_run_token = bool(getattr(args, "emit_run_token", False)) + if token_output or emit_run_token: + return + raise RouteClientError( + "eval deploy requires --token-output PATH and/or --emit-run-token so the " + "miner can later call eval result with EVAL_RUN_TOKEN; the one-time token " + "is not recoverable from prepare/status output" + ) + + +def _write_eval_run_token_file(path: str, token: str) -> None: + """Write the one-time run token to PATH with mode 0o600 (create securely).""" + + # O_CREAT|O_WRONLY|O_TRUNC with mode 0o600 — never create world-readable then chmod. + fd = os.open( + path, + os.O_CREAT | os.O_WRONLY | os.O_TRUNC, + 0o600, + ) + try: + with os.fdopen(fd, "w", encoding="utf-8") as handle: + handle.write(token) + fd = -1 # fdopen owns it + finally: + if fd >= 0: + os.close(fd) + + +def _hand_off_eval_run_token( + args: argparse.Namespace, + *, + eval_run_id: str, + eval_run_token: str, + acknowledgement: Mapping[str, Any] | None, + encrypted_env_names: Sequence[str], +) -> dict[str, Any]: + """Build deploy-success payload and optionally surface the run token once. + + The token is never written to ``--output`` and never passes through + ``_redact_capabilities``. ``eval_run_id`` is always present for + ``eval result --run-id``. + """ + + token_output = getattr(args, "token_output", None) + if isinstance(token_output, str) and token_output: + _write_eval_run_token_file(token_output, eval_run_token) + payload: dict[str, Any] = { + "stage": "eval_deployed", + "eval_run_id": eval_run_id, + "acknowledgement": acknowledgement, + "encrypted_env_names": list(encrypted_env_names), + } + if bool(getattr(args, "emit_run_token", False)): + payload["eval_run_token"] = eval_run_token + output_path = getattr(args, "output", None) + if isinstance(output_path, str) and output_path: + # Persisted plan/metadata must never carry the one-time token. + safe = {key: value for key, value in payload.items() if key != "eval_run_token"} + Path(output_path).write_text( + json.dumps(safe, sort_keys=True, separators=(",", ":")), + encoding="utf-8", + ) + return payload + + # --------------------------------------------------------------------------- # # Parser # --------------------------------------------------------------------------- # @@ -901,7 +1222,16 @@ def build_parser() -> argparse.ArgumentParser: help="delete a deployed CVM (idempotent)", description="Delete the CVM so no resource is left running (phala cvms delete -f).", ) - tear.add_argument("--cvm-id", required=True, help="the CVM id to delete") + tear.add_argument( + "--cvm-id", + default=None, + help="the CVM id to delete (optional if --app-id is set)", + ) + tear.add_argument( + "--app-id", + default=None, + help="resolve CVM id via GET /cvms exact app_id match when --cvm-id is omitted", + ) # Ordered production lifecycle. The older top-level helpers remain as # compatibility shims for offline callers, but all new spend-capable work @@ -927,8 +1257,28 @@ def build_parser() -> argparse.ArgumentParser: help="environment variable holding the user key", ) review_deploy.add_argument("--phala-api", default=None, help="Phala Cloud API base URL") - review_deploy.add_argument("--review-instance-type", default="tdx.small") - review_deploy.add_argument("--eval-instance-type", default="tdx.small") + review_deploy.add_argument( + "--review-instance-type", + default=DEFAULT_REVIEW_INSTANCE_TYPE, + help=f"review CVM shape (default: {DEFAULT_REVIEW_INSTANCE_TYPE})", + ) + review_deploy.add_argument( + "--eval-instance-type", + default=DEFAULT_EVAL_INSTANCE_TYPE, + help=f"eval CVM shape used for combined budget (default: {DEFAULT_EVAL_INSTANCE_TYPE})", + ) + review_deploy.add_argument( + "--review-disk-size-gb", + type=int, + default=DEFAULT_REVIEW_DISK_SIZE_GB, + help=f"review disk size GB (default: {DEFAULT_REVIEW_DISK_SIZE_GB})", + ) + review_deploy.add_argument( + "--eval-disk-size-gb", + type=int, + default=DEFAULT_EVAL_DISK_SIZE_GB, + help=f"eval disk size GB for combined budget (default: {DEFAULT_EVAL_DISK_SIZE_GB})", + ) review_deploy.add_argument("--review-runtime-hours", type=float, default=6.0) review_deploy.add_argument("--eval-runtime-hours", type=float, default=6.0) review_deploy.add_argument("--money-cap-usd", type=float, default=20.0) @@ -963,7 +1313,17 @@ def build_parser() -> argparse.ArgumentParser: ), ) review_tear = review_sub.add_parser("teardown", help="delete the review CVM") - review_tear.add_argument("--cvm-id", required=True) + review_tear.add_argument( + "--cvm-id", + default=None, + help="CVM id (optional if --app-id is set)", + ) + review_tear.add_argument( + "--app-id", + default=None, + help="resolve CVM via GET /cvms exact app_id match when --cvm-id is omitted", + ) + review_tear.add_argument("--phala-api", default=None, help="Phala Cloud API base URL") evaluation = sub.add_parser( "eval", @@ -997,8 +1357,28 @@ def build_parser() -> argparse.ArgumentParser: ) eval_deploy_parser.add_argument("--llm-cost-limit-env", default="LLM_COST_LIMIT") eval_deploy_parser.add_argument("--phala-api", default=None, help="Phala Cloud API base URL") - eval_deploy_parser.add_argument("--review-instance-type", default="tdx.small") - eval_deploy_parser.add_argument("--eval-instance-type", default="tdx.small") + eval_deploy_parser.add_argument( + "--review-instance-type", + default=DEFAULT_REVIEW_INSTANCE_TYPE, + help=f"review CVM shape used for combined budget (default: {DEFAULT_REVIEW_INSTANCE_TYPE})", + ) + eval_deploy_parser.add_argument( + "--eval-instance-type", + default=DEFAULT_EVAL_INSTANCE_TYPE, + help=f"eval CVM shape (default: {DEFAULT_EVAL_INSTANCE_TYPE})", + ) + eval_deploy_parser.add_argument( + "--review-disk-size-gb", + type=int, + default=DEFAULT_REVIEW_DISK_SIZE_GB, + help=f"review disk size GB for combined budget (default: {DEFAULT_REVIEW_DISK_SIZE_GB})", + ) + eval_deploy_parser.add_argument( + "--eval-disk-size-gb", + type=int, + default=DEFAULT_EVAL_DISK_SIZE_GB, + help=f"eval disk size GB (default: {DEFAULT_EVAL_DISK_SIZE_GB})", + ) eval_deploy_parser.add_argument("--review-runtime-hours", type=float, default=6.0) eval_deploy_parser.add_argument("--eval-runtime-hours", type=float, default=6.0) eval_deploy_parser.add_argument("--money-cap-usd", type=float, default=20.0) @@ -1007,6 +1387,37 @@ def build_parser() -> argparse.ArgumentParser: action="store_true", help="validate and print safe deployment metadata without provisioning", ) + eval_deploy_parser.add_argument( + "--token-output", + default=None, + metavar="PATH", + help=( + "write the one-time EVAL_RUN_TOKEN to PATH with mode 0600 " + "(required for eval result unless --emit-run-token)" + ), + ) + eval_deploy_parser.add_argument( + "--emit-run-token", + action="store_true", + help=( + "include eval_run_token in deploy success JSON on stdout " + "(required for eval result unless --token-output)" + ), + ) + eval_deploy_parser.add_argument( + "--expected-measurement", + default=None, + metavar="PATH", + help=( + "optional JSON measurement pin; when present, plan rtmr0 must match " + "before Phala create (truncated prefix on mismatch; never logs full digests)" + ), + ) + eval_deploy_parser.add_argument( + "--output", + default=None, + help="write safe deploy metadata JSON (never includes EVAL_RUN_TOKEN)", + ) eval_result_parser = eval_sub.add_parser( "result", help="post the exact result to the direct route", @@ -1039,7 +1450,17 @@ def build_parser() -> argparse.ArgumentParser: ), ) eval_tear = eval_sub.add_parser("teardown", help="delete the Eval CVM") - eval_tear.add_argument("--cvm-id", required=True) + eval_tear.add_argument( + "--cvm-id", + default=None, + help="CVM id (optional if --app-id is set)", + ) + eval_tear.add_argument( + "--app-id", + default=None, + help="resolve CVM via GET /cvms exact app_id match when --cvm-id is omitted", + ) + eval_tear.add_argument("--phala-api", default=None, help="Phala Cloud API base URL") return parser @@ -1237,8 +1658,20 @@ def _cmd_result(args: argparse.Namespace) -> int: def _cmd_teardown(args: argparse.Namespace, *, teardowner: Teardowner) -> int: - outcome = teardowner(args.cvm_id) - payload, code = _teardown_payload(args.cvm_id, outcome) + # Injected teardowner (tests) still receives an explicit cvm id only. + if teardowner is not default_phala_teardown: + cvm_id = (getattr(args, "cvm_id", None) or "").strip() + if not cvm_id: + print( + "error: teardown requires --cvm-id when using a custom teardowner", + file=sys.stderr, + ) + return 2 + outcome = teardowner(cvm_id) + payload, code = _teardown_payload(cvm_id, outcome) + _print(payload) + return code + payload, code = _run_teardown_command(args) _print(payload) return code @@ -1284,6 +1717,7 @@ def main( "build_parser", "default_phala_deployer", "default_phala_teardown", + "resolve_teardown_cvm_id", "main", ] diff --git a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/eval.py b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/eval.py index 1809c6717..19efdace1 100644 --- a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/eval.py +++ b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/eval.py @@ -34,9 +34,11 @@ resolve_cvm_id_from_list, ) from agent_challenge.selfdeploy.shapes import ( + DEFAULT_EVAL_DISK_SIZE_GB, DEFAULT_INSTANCE_TYPE, DEFAULT_OS_IMAGE, validate_cpu_only, + validate_disk_size, ) #: Capacity-safe default (bare ``us-west`` → ERR-02-002 No teepod found). @@ -102,6 +104,7 @@ class EvalDeploymentPlan: os_image: str = DEFAULT_OS_IMAGE compose_name: str = DEFAULT_EVAL_COMPOSE_NAME phala_app_nonce: int | None = None + disk_size_gb: int = DEFAULT_EVAL_DISK_SIZE_GB @dataclass(frozen=True) @@ -262,6 +265,7 @@ def build_eval_deployment_plan( os_image=DEFAULT_OS_IMAGE, compose_name=compose_name, phala_app_nonce=phala_app_nonce, + disk_size_gb=DEFAULT_EVAL_DISK_SIZE_GB, ) @@ -357,6 +361,8 @@ def deploy( "compose_file": plan.compose, "env_keys": list(encrypted.env_keys), "image": plan.os_image, + # Sibling of compose_file — never mutate plan.compose. + "disk_size": validate_disk_size(plan.disk_size_gb), } if plan.phala_app_nonce is not None: provision_request["nonce"] = plan.phala_app_nonce diff --git a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/lifecycle.py b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/lifecycle.py index b09cf61fd..03a3d9c5c 100644 --- a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/lifecycle.py +++ b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/lifecycle.py @@ -7,9 +7,13 @@ from agent_challenge.selfdeploy.shapes import ( CPU_TDX_SHAPES, + DEFAULT_EVAL_DISK_SIZE_GB, DEFAULT_MONEY_CAP_USD, + DEFAULT_REVIEW_DISK_SIZE_GB, ShapeError, + projected_disk_cost_usd, validate_cpu_only, + validate_disk_size, ) @@ -31,14 +35,21 @@ def projected_lifecycle_cost_usd( eval_instance_type: str, review_runtime_hours: float, eval_runtime_hours: float, + review_disk_size_gb: int = DEFAULT_REVIEW_DISK_SIZE_GB, + eval_disk_size_gb: int = DEFAULT_EVAL_DISK_SIZE_GB, ) -> float: - """Compute both CVM projections together, never budget each stage alone.""" + """Compute both CVM projections together (compute + disk), never budget each alone.""" for instance_type in (review_instance_type, eval_instance_type): try: validate_cpu_only(instance_type=instance_type) except ShapeError as exc: raise LifecycleBudgetError(str(exc)) from exc + try: + review_disk = validate_disk_size(review_disk_size_gb) + eval_disk = validate_disk_size(eval_disk_size_gb) + except ShapeError as exc: + raise LifecycleBudgetError(str(exc)) from exc if ( not math.isfinite(review_runtime_hours) or not math.isfinite(eval_runtime_hours) @@ -48,7 +59,9 @@ def projected_lifecycle_cost_usd( raise LifecycleBudgetError("runtime hours must be non-negative") return ( CPU_TDX_SHAPES[review_instance_type].usd_per_hour * review_runtime_hours + + projected_disk_cost_usd(review_disk, max_runtime_hours=review_runtime_hours) + CPU_TDX_SHAPES[eval_instance_type].usd_per_hour * eval_runtime_hours + + projected_disk_cost_usd(eval_disk, max_runtime_hours=eval_runtime_hours) ) @@ -59,6 +72,8 @@ def validate_lifecycle_budget( review_runtime_hours: float, eval_runtime_hours: float, money_cap_usd: float = 20.0, + review_disk_size_gb: int = DEFAULT_REVIEW_DISK_SIZE_GB, + eval_disk_size_gb: int = DEFAULT_EVAL_DISK_SIZE_GB, ) -> LifecycleCost: """Refuse a combined lifecycle that could exceed the shared cap.""" @@ -75,15 +90,25 @@ def validate_lifecycle_budget( eval_instance_type=eval_instance_type, review_runtime_hours=review_runtime_hours, eval_runtime_hours=eval_runtime_hours, + review_disk_size_gb=review_disk_size_gb, + eval_disk_size_gb=eval_disk_size_gb, ) if total > money_cap_usd: raise LifecycleBudgetError( f"projected review+eval cost ${total:.2f} exceeds the ${money_cap_usd:.2f} cap" ) assert review_cost is not None and eval_cost is not None + review_disk = validate_disk_size(review_disk_size_gb) + eval_disk = validate_disk_size(eval_disk_size_gb) return LifecycleCost( - review_usd=review_cost.usd_per_hour * review_runtime_hours, - eval_usd=eval_cost.usd_per_hour * eval_runtime_hours, + review_usd=( + review_cost.usd_per_hour * review_runtime_hours + + projected_disk_cost_usd(review_disk, max_runtime_hours=review_runtime_hours) + ), + eval_usd=( + eval_cost.usd_per_hour * eval_runtime_hours + + projected_disk_cost_usd(eval_disk, max_runtime_hours=eval_runtime_hours) + ), total_usd=total, money_cap_usd=money_cap_usd, ) diff --git a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/measurements.py b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/measurements.py index e45e67ea9..fc9f5df0d 100644 --- a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/measurements.py +++ b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/measurements.py @@ -235,6 +235,126 @@ def domain_allowlist_verdict( return allowlist_verdict(measurement, source) +def short_measurement_hex_prefix(value: object, *, length: int = 12) -> str | None: + """Return a truncated lowercase hex prefix for operator diagnostics. + + Never returns the full digest when the source is longer than ``length``. + Rejects non-hex input so callers never echo secrets as opaque blobs. + """ + + if not isinstance(value, str): + return None + cleaned = value.strip().lower() + if cleaned.startswith("0x"): + cleaned = cleaned[2:] + if not cleaned or any(ch not in "0123456789abcdef" for ch in cleaned): + return None + if length < 1: + return None + return cleaned[:length] + + +def format_eval_shape_mismatch_error( + *, + plan_instance_type: str, + requested_instance_type: str, + plan_vm_shape: str | None = None, + plan_rtmr0: str | None = None, +) -> str: + """Loud, operator-facing message for plan vs CLI eval shape mismatch. + + Names ``vm_shape`` / ``instance_type``, both shapes, the rtmr0 allowlist + footgun, and that prepare already spent the one-shot token so a re-prepare + (next deploy cancel+retry) is required. Truncates any plan rtmr0 prefix. + """ + + plan_shape = (plan_vm_shape or plan_instance_type or "").strip() or plan_instance_type + requested = (requested_instance_type or "").strip() or requested_instance_type + prefix = short_measurement_hex_prefix(plan_rtmr0) if plan_rtmr0 else None + rtmr_note = ( + f" Plan measurement rtmr0 prefix={prefix} (truncated; full value never logged)." + if prefix + else "" + ) + return ( + "Eval deployment shape mismatch: validator-issued plan has " + f"vm_shape/instance_type={plan_shape!r} but CLI --eval-instance-type=" + f"{requested!r}. A shape change requires a matching measurement pin " + "(rtmr0) on the validator allowlist and a re-prepare; a stale rtmr0 pin " + "surfaces only as a generic key-release denial deep in the TEE flow — " + "not as a clear shape error. This abort is before Phala create (no spend). " + "The one-time EVAL_RUN_TOKEN delivery was already consumed by prepare; " + "re-run eval deploy (or eval cancel + retry) to re-prepare a fresh attempt." + f"{rtmr_note}" + ) + + +def format_rtmr0_pin_mismatch_error( + *, + plan_rtmr0: str, + expected_rtmr0: str, +) -> str: + """Loud message when plan rtmr0 disagrees with --expected-measurement.""" + + plan_prefix = short_measurement_hex_prefix(plan_rtmr0) or "?" + expected_prefix = short_measurement_hex_prefix(expected_rtmr0) or "?" + return ( + "Eval measurement pin mismatch on field rtmr0: " + f"plan prefix={plan_prefix} vs --expected-measurement prefix={expected_prefix} " + "(truncated; full digests never logged). A stale rtmr0 pin on the validator " + "allowlist surfaces only as a generic key-release denial. Aborting before " + "Phala create (no spend). Re-prepare after the allowlist pin matches the " + "deployed shape." + ) + + +def load_expected_measurement_mapping(path: str | Path) -> dict[str, Any]: + """Load a miner-supplied expected measurement JSON object from PATH.""" + + try: + raw = Path(path).read_text(encoding="utf-8") + except OSError as exc: + raise MeasurementError(f"expected measurement file could not be read: {path}") from exc + try: + parsed = json.loads(raw) + except json.JSONDecodeError as exc: + raise MeasurementError("expected measurement file is not valid JSON") from exc + if not isinstance(parsed, dict): + raise MeasurementError("expected measurement file must contain a JSON object") + return parsed + + +def compare_plan_rtmr0_to_expected( + plan_measurement: Mapping[str, Any], + expected: Mapping[str, Any], +) -> str | None: + """Return a loud error string when plan rtmr0 disagrees with expected, else None. + + When ``expected`` has no ``rtmr0`` field the check is a no-op (optional pin). + """ + + expected_rtmr0 = expected.get("rtmr0") + if expected_rtmr0 is None: + return None + if not isinstance(expected_rtmr0, str) or not expected_rtmr0.strip(): + raise MeasurementError("expected measurement rtmr0 must be a non-empty string") + plan_rtmr0 = plan_measurement.get("rtmr0") + if not isinstance(plan_rtmr0, str) or not plan_rtmr0.strip(): + raise MeasurementError("plan measurement is missing rtmr0 for pin comparison") + plan_norm = plan_rtmr0.strip().lower() + expected_norm = expected_rtmr0.strip().lower() + if plan_norm.startswith("0x"): + plan_norm = plan_norm[2:] + if expected_norm.startswith("0x"): + expected_norm = expected_norm[2:] + if plan_norm == expected_norm: + return None + return format_rtmr0_pin_mismatch_error( + plan_rtmr0=plan_rtmr0, + expected_rtmr0=expected_rtmr0, + ) + + def measurements_agree( miner_measurement: Mapping[str, Any], validator_entry: Mapping[str, Any], @@ -255,11 +375,16 @@ def measurements_agree( "ProvisionOsIdentityError", "allowlist_verdict", "canonical_measurement_subset", + "compare_plan_rtmr0_to_expected", "domain_allowlist_verdict", + "format_eval_shape_mismatch_error", + "format_rtmr0_pin_mismatch_error", "load_allowlist_entries", + "load_expected_measurement_mapping", "measurement_uses_product_os_identity", "measurements_agree", "product_os_image_hash", "reproduce_measurement", + "short_measurement_hex_prefix", "verify_provision_os_identity", ] diff --git a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/phala.py b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/phala.py index 2bd0a6309..afc2a42f6 100644 --- a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/phala.py +++ b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/phala.py @@ -21,6 +21,7 @@ import json import os +import re from collections.abc import Mapping, Sequence from typing import Any from urllib.error import HTTPError, URLError @@ -48,6 +49,10 @@ #: Allowed GET paths for safe read helpers (list/details — never secrets). _ALLOWED_GET_PATHS = frozenset({"/cvms"}) +#: Allowed DELETE path shape: /cvms/{id} only (no nested paths). +_ALLOWED_DELETE_PATH_RE = re.compile(r"^/cvms/[A-Za-z0-9][A-Za-z0-9._-]*$") +_CVM_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$") + #: Create-response keys that may identify a CVM (ordered preference). #: ``app_id`` is intentionally excluded: it names the app pin, not the CVM. _CREATE_CVM_ID_FIELDS = ("id", "cvm_id", "vm_uuid", "instance_id", "uuid") @@ -101,12 +106,14 @@ def resolve_cvm_id_from_list( listing: Mapping[str, Any] | Sequence[Any], *, app_id: str, + require_unique: bool = False, ) -> str | None: """Locate a CVM id in a GET /cvms listing by exact app_id match. Returns None when listing is empty/mismatched rather than inventing an id. - Prefer a single exact app_id match; on multi-match take the first ordered - entry that identifies a CVM. Secret bodies are never logged. + When ``require_unique`` is False (deploy create fallback), the first ordered + match wins. When True (teardown resolution), multiple matches raise + :class:`PhalaApiError` so callers never guess. Secret bodies are never logged. """ if not isinstance(app_id, str) or not app_id.strip(): @@ -127,6 +134,7 @@ def resolve_cvm_id_from_list( else: return None + matches: list[str] = [] for item in items: if not isinstance(item, Mapping): continue @@ -134,10 +142,16 @@ def resolve_cvm_id_from_list( if not isinstance(item_app, str) or item_app != target: continue try: - return extract_cvm_id_from_create_response(item) + matches.append(extract_cvm_id_from_create_response(item)) except ValueError: continue - return None + if not matches: + return None + if require_unique and len(matches) > 1: + raise PhalaApiError( + f"multiple CVMs match app_id ({len(matches)}); pass --cvm-id explicitly" + ) + return matches[0] def normalize_phala_region(region: str | None) -> str: @@ -303,6 +317,32 @@ def post(self, path: str, payload: Mapping[str, Any]) -> dict[str, Any]: ) return self._open(request) + def delete_cvm(self, cvm_id: str) -> None: + """DELETE ``/cvms/{id}``. 204 and 404 are success (idempotent teardown).""" + + cid = (cvm_id or "").strip() + if not cid or not _CVM_ID_RE.fullmatch(cid): + raise PhalaApiError("invalid CVM id for Phala delete") + path = f"/cvms/{cid}" + if not _ALLOWED_DELETE_PATH_RE.fullmatch(path): + raise PhalaApiError("unsupported Phala mutation route") + request = Request( + f"{self._base_url}{path}", + headers=self._base_headers(content_type=False), + method="DELETE", + ) + try: + response = self._opener(request, timeout=self._timeout) + # Drain body; 204 is empty. Never log response content. + _ = response.read() + except HTTPError as exc: + if exc.code == 404: + return + raise PhalaApiError(f"Phala delete returned HTTP {exc.code}") from exc + except (URLError, TimeoutError, OSError) as exc: + raise PhalaApiError("Phala delete endpoint is unreachable") from exc + + __all__ = [ "DEFAULT_PHALA_API", diff --git a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/review.py b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/review.py index bebb0da12..e08cf4931 100644 --- a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/review.py +++ b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/review.py @@ -39,7 +39,9 @@ from agent_challenge.selfdeploy.shapes import ( DEFAULT_INSTANCE_TYPE, DEFAULT_OS_IMAGE, + DEFAULT_REVIEW_DISK_SIZE_GB, validate_cpu_only, + validate_disk_size, ) #: Capacity-safe default (bare ``us-west`` → ERR-02-002 No teepod found). @@ -78,6 +80,7 @@ class ReviewDeploymentPlan: instance_type: str = DEFAULT_INSTANCE_TYPE region: str = DEFAULT_REGION os_image: str = DEFAULT_OS_IMAGE + disk_size_gb: int = DEFAULT_REVIEW_DISK_SIZE_GB #: Stable moniker measured into app-compose ``name`` (compose_hash binding). #: Distinct from :attr:`app_identity` when the latter is a Phala 40-hex app_id. compose_name: str = DEFAULT_REVIEW_APP_IDENTITY @@ -206,6 +209,7 @@ def build_review_deployment_plan(prepare_response: Mapping[str, Any]) -> ReviewD os_image=DEFAULT_OS_IMAGE, compose_name=compose_name, phala_app_nonce=phala_app_nonce, + disk_size_gb=DEFAULT_REVIEW_DISK_SIZE_GB, ) @@ -283,6 +287,8 @@ def deploy( "compose_file": plan.compose, "env_keys": list(encrypted.env_keys), "image": plan.os_image, + # Sibling of compose_file — never mutate plan.compose. + "disk_size": validate_disk_size(plan.disk_size_gb), } if plan.phala_app_nonce is not None: provision_request["nonce"] = plan.phala_app_nonce diff --git a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/shapes.py b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/shapes.py index ec0c2c323..e5a3a8ce2 100644 --- a/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/shapes.py +++ b/packages/challenges/agent-challenge/src/agent_challenge/selfdeploy/shapes.py @@ -1,23 +1,35 @@ -"""CPU Intel TDX shape catalog + money/GPU deploy guards (AGENTS.md boundaries). +"""CPU Intel TDX shape catalog + money/GPU/disk deploy guards (AGENTS.md boundaries). The mission is **CPU Intel TDX only** (no GPU, not available to this account) with -a hard **$20** spend cap and a preference for the smallest CPU shape that works -(``tdx.small``/``tdx.medium``). This module is the single source of truth for the -CPU shape catalog and the pure, side-effect-free guard functions the deploy path -runs BEFORE any provisioning: +a hard **$20** spend cap. Stage defaults are split: + +* **review** — ``tdx.small`` (1 vCPU / 2 GiB) with **20 GB** disk (light analyzer); +* **eval** — ``tdx.xlarge`` (8 vCPU / 16 GiB) with **100 GB** disk so four concurrent + Terminal-Bench tasks (1 vCPU / 2 GiB each) leave headroom for the orchestrator. + +Disk is **stage policy** (constants + :func:`validate_disk_size`), not a field on +every :class:`CpuShape`. Phala bills disk separately at +:data:`DISK_USD_PER_GB_HOUR`; projected cost = compute hours + disk hours. + +This module is the single source of truth for the CPU shape catalog and the pure, +side-effect-free guard functions the deploy path runs BEFORE any provisioning: * :func:`validate_cpu_only` refuses a GPU instance type (e.g. ``h200.small``) or a GPU OS image (e.g. ``dstack-nvidia-*``) and any unknown shape (VAL-DEPLOY-007); -* :func:`select_default_instance_type` picks the smallest CPU shape when the miner +* :func:`select_default_instance_type` picks the review default when the miner gives none (VAL-DEPLOY-008); -* :func:`validate_within_cap` refuses a shape whose projected cost would breach the - money cap (VAL-DEPLOY-008). +* :func:`validate_disk_size` refuses disk outside ``[MIN_DISK_SIZE_GB, MAX_DISK_SIZE_GB]``; +* :func:`validate_within_cap` refuses a shape whose projected cost (compute + optional + disk) would breach the money cap (VAL-DEPLOY-008). -Hourly rates are the account's observed CPU TDX prices (library/phala.md). +Hourly compute rates are the account's observed CPU TDX prices (library/phala.md). +Disk billing is the observed Phala rate (~$0.000139/GB/hour); live tdx.* default +disk is 20 GB when the provision body omits ``disk_size``. """ from __future__ import annotations +import math import re from dataclasses import dataclass @@ -34,9 +46,17 @@ class OverCapError(ShapeError): """The requested shape's projected cost would breach the money cap.""" +class DiskSizeError(ShapeError): + """A requested disk size is outside the allowed stage bounds.""" + + @dataclass(frozen=True) class CpuShape: - """A CPU Intel TDX shape: vCPU/RAM and the account's observed hourly USD rate.""" + """A CPU Intel TDX shape: vCPU/RAM and the account's observed hourly USD rate. + + Disk is intentionally **not** a per-shape field — stage policy owns disk size + via :data:`DEFAULT_REVIEW_DISK_SIZE_GB` / :data:`DEFAULT_EVAL_DISK_SIZE_GB`. + """ name: str vcpus: int @@ -52,11 +72,32 @@ class CpuShape: "tdx.xlarge": CpuShape("tdx.xlarge", 8, 16, 0.464), } -#: The smallest CPU shapes the mission prefers (AGENTS.md). +#: The smallest CPU shapes the mission prefers for light stages (AGENTS.md). SMALLEST_CPU_SHAPES: tuple[str, ...] = ("tdx.small", "tdx.medium") -#: Default instance type when the miner does not pick one: the smallest CPU shape. -DEFAULT_INSTANCE_TYPE = "tdx.small" +#: Review-stage default: light analyzer CVM. +DEFAULT_REVIEW_INSTANCE_TYPE = "tdx.small" + +#: Eval-stage default: 8 vCPU / 16 GiB so 4 concurrent 1-vCPU/2-GiB tasks fit. +DEFAULT_EVAL_INSTANCE_TYPE = "tdx.xlarge" + +#: Backward-compatible alias of the review default (legacy single-stage deploy). +DEFAULT_INSTANCE_TYPE = DEFAULT_REVIEW_INSTANCE_TYPE + +#: Default review disk (GB). Matches Phala's live tdx.* default when omitted. +DEFAULT_REVIEW_DISK_SIZE_GB = 20 + +#: Default eval disk (GB). Larger for DooD task images + concurrent containers. +DEFAULT_EVAL_DISK_SIZE_GB = 100 + +#: Observed Phala disk billing rate (USD per GB per hour). +DISK_USD_PER_GB_HOUR = 0.000139 + +#: Inclusive lower bound for provision ``disk_size`` (GB). +MIN_DISK_SIZE_GB = 20 + +#: Inclusive upper bound for provision ``disk_size`` (GB). +MAX_DISK_SIZE_GB = 500 #: Default CPU dstack OS image. Live teepods (prod5/prod9) currently ship up to #: dstack-0.5.9 (product default was 0.5.10 but that image is not mounted on @@ -68,8 +109,8 @@ class CpuShape: DEFAULT_MONEY_CAP_USD = 20.0 #: Conservative projected max runtime (hours) used for the cost-cap guard. A -#: deploy's projected cost is ``usd_per_hour * max_runtime_hours``; a shape whose -#: projected cost exceeds the money cap is refused before provisioning. +#: deploy's projected cost is compute + optional disk over this window; a shape +#: whose projected cost exceeds the money cap is refused before provisioning. DEFAULT_MAX_RUNTIME_HOURS = 6.0 #: GPU instance-type prefixes/markers that are always refused (CPU-only mission). @@ -125,24 +166,61 @@ def validate_cpu_only(*, instance_type: str, os_image: str = DEFAULT_OS_IMAGE) - def select_default_instance_type() -> str: - """The smallest CPU shape used when the miner supplies none (VAL-DEPLOY-008).""" + """The review-stage default used when the miner supplies none (VAL-DEPLOY-008).""" return DEFAULT_INSTANCE_TYPE +def validate_disk_size(gb: object) -> int: + """Refuse non-integer or out-of-range disk sizes; return a validated int GB. + + Bounds are inclusive ``[MIN_DISK_SIZE_GB, MAX_DISK_SIZE_GB]``. Pure and + side-effect free. + """ + + if isinstance(gb, bool) or not isinstance(gb, int): + raise DiskSizeError( + f"disk_size_gb must be an integer in " + f"[{MIN_DISK_SIZE_GB}, {MAX_DISK_SIZE_GB}]; got {gb!r}" + ) + if gb < MIN_DISK_SIZE_GB or gb > MAX_DISK_SIZE_GB: + raise DiskSizeError( + f"disk_size_gb={gb} outside allowed range " + f"[{MIN_DISK_SIZE_GB}, {MAX_DISK_SIZE_GB}]" + ) + return gb + + +def projected_disk_cost_usd( + disk_size_gb: int, + *, + max_runtime_hours: float = DEFAULT_MAX_RUNTIME_HOURS, +) -> float: + """Projected disk cost = ``DISK_USD_PER_GB_HOUR * gb * hours``.""" + + size = validate_disk_size(disk_size_gb) + if not math.isfinite(max_runtime_hours) or max_runtime_hours < 0: + raise ShapeError("max_runtime_hours must be a finite non-negative number") + return DISK_USD_PER_GB_HOUR * float(size) * float(max_runtime_hours) + + def projected_cost_usd( instance_type: str, *, max_runtime_hours: float = DEFAULT_MAX_RUNTIME_HOURS, + disk_size_gb: int | None = None, ) -> float: - """Projected deploy cost = ``usd_per_hour * max_runtime_hours`` for a CPU shape.""" + """Projected deploy cost = compute hours + optional disk hours for a CPU shape.""" shape = CPU_TDX_SHAPES.get((instance_type or "").strip()) if shape is None: raise ShapeError(f"unknown CPU Intel TDX shape {instance_type!r}") - if max_runtime_hours < 0: - raise ShapeError("max_runtime_hours must be non-negative") - return shape.usd_per_hour * float(max_runtime_hours) + if not math.isfinite(max_runtime_hours) or max_runtime_hours < 0: + raise ShapeError("max_runtime_hours must be a finite non-negative number") + total = shape.usd_per_hour * float(max_runtime_hours) + if disk_size_gb is not None: + total += projected_disk_cost_usd(disk_size_gb, max_runtime_hours=max_runtime_hours) + return total def validate_within_cap( @@ -150,6 +228,7 @@ def validate_within_cap( *, money_cap_usd: float = DEFAULT_MONEY_CAP_USD, max_runtime_hours: float = DEFAULT_MAX_RUNTIME_HOURS, + disk_size_gb: int | None = None, ) -> float: """Refuse a shape whose projected cost breaches the money cap (VAL-DEPLOY-008). @@ -158,10 +237,17 @@ def validate_within_cap( refused before any provisioning. """ - cost = projected_cost_usd(instance_type, max_runtime_hours=max_runtime_hours) + cost = projected_cost_usd( + instance_type, + max_runtime_hours=max_runtime_hours, + disk_size_gb=disk_size_gb, + ) if cost > money_cap_usd: + disk_note = "" + if disk_size_gb is not None: + disk_note = f" + disk {disk_size_gb}GB" raise OverCapError( - f"projected cost ${cost:.2f} ({instance_type} @ " + f"projected cost ${cost:.2f} ({instance_type}{disk_note} @ " f"${CPU_TDX_SHAPES[instance_type].usd_per_hour}/h x {max_runtime_hours}h) " f"exceeds the ${money_cap_usd:.2f} money cap; choose a smaller shape or lower " "the runtime budget" @@ -171,19 +257,29 @@ def validate_within_cap( __all__ = [ "CPU_TDX_SHAPES", + "DEFAULT_EVAL_DISK_SIZE_GB", + "DEFAULT_EVAL_INSTANCE_TYPE", "DEFAULT_INSTANCE_TYPE", "DEFAULT_MAX_RUNTIME_HOURS", "DEFAULT_MONEY_CAP_USD", "DEFAULT_OS_IMAGE", + "DEFAULT_REVIEW_DISK_SIZE_GB", + "DEFAULT_REVIEW_INSTANCE_TYPE", + "DISK_USD_PER_GB_HOUR", + "MAX_DISK_SIZE_GB", + "MIN_DISK_SIZE_GB", "SMALLEST_CPU_SHAPES", "CpuShape", + "DiskSizeError", "GpuRefusedError", "OverCapError", "ShapeError", "is_gpu_instance_type", "is_gpu_os_image", "projected_cost_usd", + "projected_disk_cost_usd", "select_default_instance_type", "validate_cpu_only", + "validate_disk_size", "validate_within_cap", ] diff --git a/packages/challenges/agent-challenge/tests/test_canonical_build_push.py b/packages/challenges/agent-challenge/tests/test_canonical_build_push.py index 2d7d8e324..3da3cad37 100644 --- a/packages/challenges/agent-challenge/tests/test_canonical_build_push.py +++ b/packages/challenges/agent-challenge/tests/test_canonical_build_push.py @@ -17,31 +17,34 @@ def test_repository_of_strips_tag_keeps_namespace(): assert ( - cbuild.repository_of("docker.io/mathiiss/agent-challenge-canonical:live") - == "docker.io/mathiiss/agent-challenge-canonical" + cbuild.repository_of("ghcr.io/baseintelligence/agent-challenge-canonical:live") + == "ghcr.io/baseintelligence/agent-challenge-canonical" ) # No tag -> unchanged. assert ( - cbuild.repository_of("docker.io/mathiiss/agent-challenge-canonical") - == "docker.io/mathiiss/agent-challenge-canonical" + cbuild.repository_of("ghcr.io/baseintelligence/agent-challenge-canonical") + == "ghcr.io/baseintelligence/agent-challenge-canonical" ) # A digest ref -> the repository component only. - assert cbuild.repository_of("docker.io/mathiiss/x@sha256:" + "a" * 64) == "docker.io/mathiiss/x" + assert ( + cbuild.repository_of("ghcr.io/baseintelligence/x@sha256:" + "a" * 64) + == "ghcr.io/baseintelligence/x" + ) def test_pushed_image_ref_is_repo_at_digest(): pushed = cbuild.PushedImage( - repository="docker.io/mathiiss/agent-challenge-canonical", + repository="ghcr.io/baseintelligence/agent-challenge-canonical", digest="sha256:" + "a" * 64, ) - assert pushed.ref == "docker.io/mathiiss/agent-challenge-canonical@sha256:" + "a" * 64 + assert pushed.ref == "ghcr.io/baseintelligence/agent-challenge-canonical@sha256:" + "a" * 64 # It is digest-pinned per the compose guard (no bare tag). assert cbuild.DIGEST_PIN_RE.search(pushed.ref) def test_build_push_argv_is_reproducible_registry_push(): argv = cbuild.build_push_argv( - image_name="docker.io/mathiiss/agent-challenge-canonical:live", + image_name="ghcr.io/baseintelligence/agent-challenge-canonical:live", dockerfile="/repo/docker/canonical/Dockerfile", context="/repo", metadata_path="/tmp/meta.json", @@ -56,7 +59,7 @@ def test_build_push_argv_is_reproducible_registry_push(): # Registry push with reproducible layer-timestamp rewrite. out = argv[argv.index("--output") + 1] assert "type=image" in out - assert "name=docker.io/mathiiss/agent-challenge-canonical:live" in out + assert "name=ghcr.io/baseintelligence/agent-challenge-canonical:live" in out assert "push=true" in out assert "rewrite-timestamp=true" in out @@ -70,7 +73,7 @@ def fake_runner(argv, **kwargs): meta_path = argv[meta_idx + 1] with open(meta_path, "w", encoding="utf-8") as fh: json.dump( - {"containerimage.digest": digest, "image.name": "docker.io/mathiiss/x:live"}, + {"containerimage.digest": digest, "image.name": "ghcr.io/baseintelligence/x:live"}, fh, ) @@ -82,12 +85,12 @@ class _P: return _P() pushed = cbuild.build_and_push_image( - image_name="docker.io/mathiiss/x:live", + image_name="ghcr.io/baseintelligence/x:live", runner=fake_runner, context=str(tmp_path), dockerfile=str(cbuild.CANONICAL_DOCKERFILE), ) - assert pushed.ref == "docker.io/mathiiss/x@" + digest + assert pushed.ref == "ghcr.io/baseintelligence/x@" + digest assert cbuild.assert_pullable(pushed.ref) == pushed.ref @@ -106,7 +109,7 @@ class _P: with pytest.raises(RuntimeError): cbuild.build_and_push_image( - image_name="docker.io/mathiiss/x:live", + image_name="ghcr.io/baseintelligence/x:live", runner=fake_runner, context=str(tmp_path), ) @@ -123,7 +126,7 @@ class _P: with pytest.raises(RuntimeError, match="push"): cbuild.build_and_push_image( - image_name="docker.io/mathiiss/x:live", + image_name="ghcr.io/baseintelligence/x:live", runner=fake_runner, context=str(tmp_path), ) diff --git a/packages/challenges/agent-challenge/tests/test_eval_compose_phala_provision_parity.py b/packages/challenges/agent-challenge/tests/test_eval_compose_phala_provision_parity.py index 5fa3659dd..3c60ede0d 100644 --- a/packages/challenges/agent-challenge/tests/test_eval_compose_phala_provision_parity.py +++ b/packages/challenges/agent-challenge/tests/test_eval_compose_phala_provision_parity.py @@ -26,15 +26,17 @@ from agent_challenge.selfdeploy import eval as eval_deploy #: Digest-pinned canonical image used by the live 3-task terminal_bench smoke. +#: T2: GHCR path + T1 local buildx manifest digest (OPS_REQUIRED: swap to first +#: published GHCR digest from agent-recipe publish-eval-image.yml). LIVE_SMOKE_EVAL_IMAGE = ( - "docker.io/mathiiss/agent-challenge-canonical@sha256:" - "02331f0909f617e333f113be376d353770a673669946bcddaac3c53cbde7c9d8" + "ghcr.io/baseintelligence/agent-challenge-canonical@sha256:" + "ea399cee1b3c9015024918a6070901e6b7b4bee432c8afe1e7dd40a95547e0bc" ) LIVE_SMOKE_APP_IDENTITY = "agent-challenge-eval-v1" LIVE_SMOKE_KEY_RELEASE = "ratls://84.32.70.61:8701" #: Synthetic review image pin (disjoint service inventory only). -REVIEW_IMAGE = "docker.io/mathiiss/agent-challenge-review@sha256:" + ("c" * 64) +REVIEW_IMAGE = "ghcr.io/baseintelligence/agent-challenge-review@sha256:" + ("c" * 64) #: Live residual local hash (pre-envelope, no guest golden/task bind mounts) #: for the smoke inputs above. @@ -42,8 +44,8 @@ # list includes the validator server-CA injection names (RA_TLS_SERVER_CA_*). Updated # when FAIL-CLOSED server-CA wiring lands so the discriminator still proves the # envelope factors (not allowed_envs) are what Phala provision rewrites. -# Residual pre-envelope hash after RA-TLS server-CA + OPENROUTER_API_KEY allowed_envs lands. -LIVE_RESIDUAL_NO_ENVELOPE_HASH = "5e33c9be56dc518070045596f1c6d7b31c2d73f7508056f7db726f3ccd6179a3" +# Residual pre-envelope hash after D6 GHCR-only orchestrator pin (T2). +LIVE_RESIDUAL_NO_ENVELOPE_HASH = "a4131e778128f5d22052efcab71ed2fdb6d7d5446c5f8141511ed153087a7364" def _live_smoke_compose() -> dict: @@ -129,7 +131,6 @@ def test_build_eval_deployment_plan_accepts_parity_compose_identity(): "submission_version": 1, "authorizing_review_digest": "ab" * 32, "agent_hash": "cd" * 32, - "package_tree_sha": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "selected_tasks": [ { "task_id": "adaptive-rejection-sampler", @@ -148,6 +149,7 @@ def test_build_eval_deployment_plan_accepts_parity_compose_identity(): }, ], "k": 1, + "n_concurrent": 4, "scoring_policy": policy, "scoring_policy_digest": eval_wire.scoring_policy_digest(policy), "eval_app": { diff --git a/packages/challenges/agent-challenge/tests/test_eval_deploy_shape_pin_loud_fail.py b/packages/challenges/agent-challenge/tests/test_eval_deploy_shape_pin_loud_fail.py new file mode 100644 index 000000000..13e332ff1 --- /dev/null +++ b/packages/challenges/agent-challenge/tests/test_eval_deploy_shape_pin_loud_fail.py @@ -0,0 +1,431 @@ +"""TDD: eval deploy must fail LOUDLY on shape / measurement-pin mismatch. + +Production residual: eval CVM shape moved tdx.small → tdx.xlarge while the +validator allowlist still pinned the small-shape rtmr0. Symptom was a generic +key-release denial deep in the TEE flow — nothing named vm_shape or rtmr0. + +Miner-side guard must abort BEFORE Phala create and name: +- plan shape vs CLI --eval-instance-type +- field names vm_shape / instance_type +- that a shape change requires a matching rtmr0 pin + re-prepare +- that a stale pin surfaces only as a generic key-release denial +""" + +from __future__ import annotations + +import hashlib +import json +from pathlib import Path +from types import SimpleNamespace +from typing import Any +from unittest.mock import MagicMock + +import pytest + +from agent_challenge.canonical import eval_wire +from agent_challenge.canonical.compose import generate_app_compose, render_app_compose +from agent_challenge.selfdeploy import cli +from agent_challenge.selfdeploy import eval as eval_deploy + +PUBLIC_KEY = "c" * 64 +RUN_TOKEN = "eval-run-token-shape-pin-sentinel-7a1b2c3d" +EVAL_RUN_ID = "eval-run-shape-pin-1" + +_SERVER_CA_PEM = ( + "-----BEGIN CERTIFICATE-----\n" + "MIICxTCCAa2gAwIBAgIUIOBn+Iz4ZK61F3pcFJGHjx995acwDQYJKoZIhvcNAQEL\n" + "BQAwEjEQMA4GA1UEAwwHdGVzdC1jYTAeFw0yNjA3MTIxMzAzNDRaFw0zNjA3MTAx\n" + "MzAzNDRaMBIxEDAOBgNVBAMMB3Rlc3QtY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB\n" + "DwAwggEKAoIBAQDWxZ5PVNf+JlSNkpDlJdqP/WWwZL4fxpJZegSJE7gipUIUH8l6\n" + "SsDhVBiE0eD2GJzGnjx7+I6Q5+36oqoVDBgukVERFkfEZ0d4MtwQ5+rU2pdBx24B\n" + "VeBkNQLFu8qNLzPQuKlU0uIDrGvK157kvMlFQl2cvaJKLGwxRd/j5x+xVRynEfuA\n" + "RSJvt6pvv2Md1Na8ES9QR8pv6q9U4DMnanc4hMjlGMKuF8xKz/ls05e8KTEkDJJP\n" + "7FiZNi0vvlMJQxch9cfzjjnK7mjQm2nrebaFMr/nJNccdq5fcEaIaJhNMU65V0LI\n" + "B2IKwLO/GhcgiFNZ43nfe93WWVaKl8vx382nAgMBAAGjEzARMA8GA1UdEwEB/wQF\n" + "MAMBAf8wDQYJKoZIhvcNAQELBQADggEBAAmfmX6/kAciNHTdvE2mrK7KUDDiDhT7\n" + "kMRWOqiBaYxxiOiz3h1vrzEo81NQqc2dZF4+MrlODcnXUMgT62ijw0O/71IYl33E\n" + "nZBV+MBry5w5vlNw1El2aO3ERtWwjxrN0sLKkqht0h7hU/+wc7+5aBV4URFoNx2E\n" + "EkcZZVknVD9EMvNlWnVVQoLnOIIW4e5F4yHqHQTdxM1TD4F0gKjfNwGK6xZNpObG\n" + "QbDfN3wSkU7DIxeNJCMB+Uc5GDHMKNiEg0yb59SEvypiDuU6cD7OuhLQM0gbjXlC\n" + "81hvjyhx/T/mRQhf6MOu8RbVdp5CDp7IqhouLwEHvHjS4bA/AZIuIP8=\n" + "-----END CERTIFICATE-----\n" +) + +# Distinct rtmr0 values (96 hex chars = 48 bytes) so prefix diffs are meaningful. +PLAN_RTMR0_SMALL = "68102e7b" + "aa" * 44 # 8 + 88 = 96 +PLAN_RTMR0_XLARGE = "ec216f1d" + "bb" * 44 +EXPECTED_RTMR0_MISMATCH = "deadbeef" + "cc" * 44 + + +def _measurement(*, vm_shape: str, rtmr0: str) -> dict[str, str]: + return { + "mrtd": "01" * 48, + "rtmr0": rtmr0, + "rtmr1": "03" * 48, + "rtmr2": "04" * 48, + "os_image_hash": "05" * 32, + "key_provider": "phala", + "vm_shape": vm_shape, + } + + +def _eval_prepare_wrapper( + *, + vm_shape: str = "tdx.small", + rtmr0: str = PLAN_RTMR0_SMALL, + token: str = RUN_TOKEN, + eval_run_id: str = EVAL_RUN_ID, +) -> dict[str, Any]: + eval_image = "registry.example/eval@sha256:" + "b" * 64 + policy = { + "schema_version": 1, + "per_task_aggregation": "mean", + "keep_policy": "off", + "drop_lowest_n": 0, + "threshold_f64be": None, + } + compose = generate_app_compose( + orchestrator_image=eval_image, + name="eval-v1", + key_release_url="validator.example:8701", + allowed_envs=eval_deploy.EVAL_ALLOWED_ENVS, + ) + compose_hash = hashlib.sha256(render_app_compose(compose).encode()).hexdigest() + plan = { + "schema_version": 1, + "eval_run_id": eval_run_id, + "submission_id": "1", + "submission_version": 1, + "authorizing_review_digest": "d" * 64, + "agent_hash": "e" * 64, + "package_tree_sha": "b" * 64, + "selected_tasks": [ + { + "task_id": "task-1", + "image_ref": "registry.example/task@sha256:" + "f" * 64, + "task_config_sha256": "1" * 64, + } + ], + "k": 1, + "scoring_policy": policy, + "scoring_policy_digest": eval_wire.scoring_policy_digest(policy), + "eval_app": { + "image_ref": eval_image, + "compose_hash": compose_hash, + "app_identity": "eval-v1", + "kms_key_algorithm": "x25519", + "kms_public_key_hex": PUBLIC_KEY, + "kms_public_key_sha256": hashlib.sha256(bytes.fromhex(PUBLIC_KEY)).hexdigest(), + "measurement": _measurement(vm_shape=vm_shape, rtmr0=rtmr0), + }, + "key_release_endpoint": "validator.example:8701", + "result_endpoint": f"/evaluation/v1/runs/{eval_run_id}/result", + "key_release_nonce": "key-release-nonce", + "score_nonce": "score-nonce", + "run_token_sha256": hashlib.sha256(token.encode()).hexdigest(), + "issued_at_ms": 1, + "expires_at_ms": 2, + } + validated = eval_wire.validate_eval_plan(plan) + return { + "schema_version": 1, + "plan": validated, + "plan_sha256": hashlib.sha256(eval_wire.canonical_json_v1(validated)).hexdigest(), + "secret_delivery": {"env_key": "EVAL_RUN_TOKEN", "token": token}, + } + + +def _deploy_args(**overrides: Any) -> SimpleNamespace: + base = dict( + eval_command="deploy", + submission_id=1, + base_url="https://challenge.example", + hotkey="hk", + signature="sig", + nonce="n", + timestamp=None, + auto_sign=True, + prepare_response=None, + gateway_token_env="BASE_GATEWAY_TOKEN", + gateway_url_env="BASE_LLM_GATEWAY_URL", + llm_cost_limit_env="LLM_COST_LIMIT", + phala_api=None, + review_instance_type="tdx.small", + eval_instance_type="tdx.xlarge", + review_runtime_hours=1.0, + eval_runtime_hours=1.0, + money_cap_usd=20.0, + dry_run=False, + token_output=None, + emit_run_token=True, + output=None, + expected_measurement=None, + ) + base.update(overrides) + return SimpleNamespace(**base) + + +def _wire_prepare_only( + monkeypatch: pytest.MonkeyPatch, + prepare: dict[str, Any], +) -> tuple[MagicMock, list[Any]]: + """Fake prepare; capture any Phala deploy attempts (must stay empty on mismatch).""" + + fake_client = MagicMock() + fake_client.eval_prepare.return_value = prepare + monkeypatch.setattr(cli, "_route_client", lambda _args: fake_client) + monkeypatch.setenv("LLM_COST_LIMIT", "1.00") + monkeypatch.setenv("CHALLENGE_PHALA_RA_TLS_SERVER_CA_PEM", _SERVER_CA_PEM) + + deploy_calls: list[Any] = [] + + class _MustNotDeploy: + def __init__(self, _api: object) -> None: + pass + + def deploy(self, plan_obj: Any, encrypted_obj: Any) -> dict[str, Any]: + deploy_calls.append((plan_obj, encrypted_obj)) + raise AssertionError("Phala deploy must not run on shape/pin mismatch") + + monkeypatch.setattr(eval_deploy, "HttpEvalPhalaDeployment", _MustNotDeploy) + monkeypatch.setattr(cli, "PhalaCloudClient", lambda **_k: object()) + return fake_client, deploy_calls + + +# --------------------------------------------------------------------------- # +# S1 — shape mismatch names both shapes + field names + pin warning +# --------------------------------------------------------------------------- # + + +def test_shape_mismatch_stderr_names_shapes_fields_and_pin_warning( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + """Given plan vm_shape=tdx.small and CLI --eval-instance-type=tdx.xlarge, + When eval deploy runs, Then exit 2, no Phala deploy, stderr is loud and specific. + """ + + prepare = _eval_prepare_wrapper(vm_shape="tdx.small", rtmr0=PLAN_RTMR0_SMALL) + _client, deploy_calls = _wire_prepare_only(monkeypatch, prepare) + + args = _deploy_args(eval_instance_type="tdx.xlarge", emit_run_token=True) + code = cli._ordered_eval_command(args) + err = capsys.readouterr().err + + assert code == 2 + assert deploy_calls == [] + # Both shapes named. + assert "tdx.small" in err + assert "tdx.xlarge" in err + # Field names the operator greps for. + assert "vm_shape" in err + assert "instance_type" in err + # Stale pin / key-release footgun. + assert "rtmr0" in err + assert "key-release" in err.lower() or "key release" in err.lower() + assert "allowlist" in err.lower() + # Re-prepare guidance (prepare already spent the one-shot delivery). + assert "re-prepare" in err.lower() or "reprepare" in err.lower() or "retry" in err.lower() + # Never dump full measurement secrets. + assert PLAN_RTMR0_SMALL not in err + assert RUN_TOKEN not in err + + +def test_shape_mismatch_aborts_before_phala_create( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Given shape mismatch, When deploy invoked, Then Phala deploy is never called.""" + + prepare = _eval_prepare_wrapper(vm_shape="tdx.small") + _client, deploy_calls = _wire_prepare_only(monkeypatch, prepare) + args = _deploy_args(eval_instance_type="tdx.xlarge", emit_run_token=True) + code = cli._ordered_eval_command(args) + assert code == 2 + assert deploy_calls == [] + + +def test_shape_mismatch_includes_truncated_plan_rtmr0_prefix_only( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + """Given plan carries rtmr0, When shape mismatches, Then stderr shows a short prefix only.""" + + prepare = _eval_prepare_wrapper(vm_shape="tdx.small", rtmr0=PLAN_RTMR0_SMALL) + _wire_prepare_only(monkeypatch, prepare) + args = _deploy_args(eval_instance_type="tdx.xlarge", emit_run_token=True) + code = cli._ordered_eval_command(args) + err = capsys.readouterr().err + assert code == 2 + prefix = PLAN_RTMR0_SMALL[:12].lower() + assert prefix in err.lower() + assert PLAN_RTMR0_SMALL not in err + assert PLAN_RTMR0_SMALL[12:] not in err + + +# --------------------------------------------------------------------------- # +# S2 — matching shapes still deploy (no false positive) +# --------------------------------------------------------------------------- # + + +def test_matching_shape_still_reaches_phala_deploy( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Given plan shape matches CLI eval_instance_type, When deploy runs, Then Phala is called.""" + + prepare = _eval_prepare_wrapper(vm_shape="tdx.xlarge", rtmr0=PLAN_RTMR0_XLARGE) + fake_client = MagicMock() + fake_client.eval_prepare.return_value = prepare + monkeypatch.setattr(cli, "_route_client", lambda _args: fake_client) + monkeypatch.setenv("LLM_COST_LIMIT", "1.00") + monkeypatch.setenv("CHALLENGE_PHALA_RA_TLS_SERVER_CA_PEM", _SERVER_CA_PEM) + + deploy_calls: list[Any] = [] + + class _OkDeploy: + def __init__(self, _api: object) -> None: + pass + + def deploy(self, plan_obj: Any, encrypted_obj: Any) -> dict[str, Any]: + deploy_calls.append(plan_obj.instance_type) + return { + "schema_version": 1, + "eval_run_id": plan_obj.eval_run_id, + "cvm_id": "cvm-ok-1", + "phala_create_receipt": { + "request_id": "req", + "app_id": plan_obj.app_identity, + "cvm_id": "cvm-ok-1", + "receipt_sha256": "a" * 64, + "created_at_ms": 1, + }, + } + + monkeypatch.setattr(eval_deploy, "HttpEvalPhalaDeployment", _OkDeploy) + monkeypatch.setattr(cli, "PhalaCloudClient", lambda **_k: object()) + printed: list[Any] = [] + monkeypatch.setattr(cli, "_print", printed.append) + + args = _deploy_args(eval_instance_type="tdx.xlarge", emit_run_token=True) + code = cli._ordered_eval_command(args) + assert code == 0, printed + assert deploy_calls == ["tdx.xlarge"] + + +# --------------------------------------------------------------------------- # +# S3 — optional --expected-measurement rtmr0 pin check +# --------------------------------------------------------------------------- # + + +def test_expected_measurement_rtmr0_mismatch_fails_before_phala( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + capsys: pytest.CaptureFixture[str], +) -> None: + """Given --expected-measurement with different rtmr0, When shapes match, + Then abort before Phala with truncated prefix diff naming rtmr0. + """ + + prepare = _eval_prepare_wrapper(vm_shape="tdx.xlarge", rtmr0=PLAN_RTMR0_XLARGE) + _client, deploy_calls = _wire_prepare_only(monkeypatch, prepare) + + expected_path = tmp_path / "expected-measurement.json" + expected_path.write_text( + json.dumps({"rtmr0": EXPECTED_RTMR0_MISMATCH, "vm_shape": "tdx.xlarge"}), + encoding="utf-8", + ) + + args = _deploy_args( + eval_instance_type="tdx.xlarge", + emit_run_token=True, + expected_measurement=str(expected_path), + ) + code = cli._ordered_eval_command(args) + err = capsys.readouterr().err + + assert code == 2 + assert deploy_calls == [] + assert "rtmr0" in err + plan_prefix = PLAN_RTMR0_XLARGE[:12].lower() + exp_prefix = EXPECTED_RTMR0_MISMATCH[:12].lower() + assert plan_prefix in err.lower() + assert exp_prefix in err.lower() + # Full digests must not appear. + assert PLAN_RTMR0_XLARGE not in err + assert EXPECTED_RTMR0_MISMATCH not in err + assert RUN_TOKEN not in err + + +def test_expected_measurement_rtmr0_match_allows_deploy( + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, +) -> None: + """Given matching expected-measurement rtmr0, When deploy runs, Then Phala is called.""" + + prepare = _eval_prepare_wrapper(vm_shape="tdx.xlarge", rtmr0=PLAN_RTMR0_XLARGE) + fake_client = MagicMock() + fake_client.eval_prepare.return_value = prepare + monkeypatch.setattr(cli, "_route_client", lambda _args: fake_client) + monkeypatch.setenv("LLM_COST_LIMIT", "1.00") + monkeypatch.setenv("CHALLENGE_PHALA_RA_TLS_SERVER_CA_PEM", _SERVER_CA_PEM) + + deploy_calls: list[Any] = [] + + class _OkDeploy: + def __init__(self, _api: object) -> None: + pass + + def deploy(self, plan_obj: Any, encrypted_obj: Any) -> dict[str, Any]: + deploy_calls.append(True) + return { + "schema_version": 1, + "eval_run_id": plan_obj.eval_run_id, + "cvm_id": "cvm-ok-2", + "phala_create_receipt": { + "request_id": "req", + "app_id": plan_obj.app_identity, + "cvm_id": "cvm-ok-2", + "receipt_sha256": "a" * 64, + "created_at_ms": 1, + }, + } + + monkeypatch.setattr(eval_deploy, "HttpEvalPhalaDeployment", _OkDeploy) + monkeypatch.setattr(cli, "PhalaCloudClient", lambda **_k: object()) + monkeypatch.setattr(cli, "_print", lambda _p: None) + + expected_path = tmp_path / "expected-measurement.json" + expected_path.write_text( + json.dumps({"rtmr0": PLAN_RTMR0_XLARGE}), + encoding="utf-8", + ) + args = _deploy_args( + eval_instance_type="tdx.xlarge", + emit_run_token=True, + expected_measurement=str(expected_path), + ) + code = cli._ordered_eval_command(args) + assert code == 0 + assert deploy_calls == [True] + + +def test_expected_measurement_flag_is_on_eval_deploy_parser() -> None: + """Given CLI parser, When eval deploy --help is built, Then --expected-measurement exists.""" + + parser = cli.build_parser() + ns = parser.parse_args( + [ + "eval", + "deploy", + "--base-url", + "https://x", + "--submission-id", + "1", + "--hotkey", + "hk", + "--auto-sign", + "--expected-measurement", + "/tmp/m.json", + "--emit-run-token", + ] + ) + assert ns.expected_measurement == "/tmp/m.json" diff --git a/packages/challenges/agent-challenge/tests/test_eval_deploy_token_handoff.py b/packages/challenges/agent-challenge/tests/test_eval_deploy_token_handoff.py new file mode 100644 index 000000000..06ec0ff69 --- /dev/null +++ b/packages/challenges/agent-challenge/tests/test_eval_deploy_token_handoff.py @@ -0,0 +1,543 @@ +"""TDD: eval deploy must hand the one-time EVAL_RUN_TOKEN to the miner. + +Root cause (production): guest emits attested result only; host posts via +``eval result --token-env EVAL_RUN_TOKEN``. The token lived only inside +``eval deploy`` memory (CVM encrypted_env) and every miner-readable surface +was redacted — closed loop, result could never be posted. + +Security: surfacing the token to the miner is a capability/anti-replay +credential for the post only. Integrity is the TEE quote bound to the plan. +""" + +from __future__ import annotations + +import hashlib +import json +import stat +from types import SimpleNamespace +from typing import Any +from unittest.mock import MagicMock + +import pytest + +from agent_challenge.canonical import eval_wire +from agent_challenge.canonical.compose import generate_app_compose, render_app_compose +from agent_challenge.selfdeploy import cli +from agent_challenge.selfdeploy import eval as eval_deploy +from agent_challenge.selfdeploy.client import RouteClientError + +PUBLIC_KEY = "c" * 64 +MEASUREMENT = { + "mrtd": "01" * 48, + "rtmr0": "02" * 48, + "rtmr1": "03" * 48, + "rtmr2": "04" * 48, + "os_image_hash": "05" * 32, + "key_provider": "phala", + "vm_shape": "tdx.small", +} + +# Distinct sentinel so accidental redaction/leak assertions cannot false-pass. +RUN_TOKEN = "eval-run-token-handoff-sentinel-9f3c2a1b" +EVAL_RUN_ID = "eval-run-handoff-1" + +# OpenSSL-loadable test CA (same fixture as ordered trust hardening). +_SERVER_CA_PEM = ( + "-----BEGIN CERTIFICATE-----\n" + "MIICxTCCAa2gAwIBAgIUIOBn+Iz4ZK61F3pcFJGHjx995acwDQYJKoZIhvcNAQEL\n" + "BQAwEjEQMA4GA1UEAwwHdGVzdC1jYTAeFw0yNjA3MTIxMzAzNDRaFw0zNjA3MTAx\n" + "MzAzNDRaMBIxEDAOBgNVBAMMB3Rlc3QtY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB\n" + "DwAwggEKAoIBAQDWxZ5PVNf+JlSNkpDlJdqP/WWwZL4fxpJZegSJE7gipUIUH8l6\n" + "SsDhVBiE0eD2GJzGnjx7+I6Q5+36oqoVDBgukVERFkfEZ0d4MtwQ5+rU2pdBx24B\n" + "VeBkNQLFu8qNLzPQuKlU0uIDrGvK157kvMlFQl2cvaJKLGwxRd/j5x+xVRynEfuA\n" + "RSJvt6pvv2Md1Na8ES9QR8pv6q9U4DMnanc4hMjlGMKuF8xKz/ls05e8KTEkDJJP\n" + "7FiZNi0vvlMJQxch9cfzjjnK7mjQm2nrebaFMr/nJNccdq5fcEaIaJhNMU65V0LI\n" + "B2IKwLO/GhcgiFNZ43nfe93WWVaKl8vx382nAgMBAAGjEzARMA8GA1UdEwEB/wQF\n" + "MAMBAf8wDQYJKoZIhvcNAQELBQADggEBAAmfmX6/kAciNHTdvE2mrK7KUDDiDhT7\n" + "kMRWOqiBaYxxiOiz3h1vrzEo81NQqc2dZF4+MrlODcnXUMgT62ijw0O/71IYl33E\n" + "nZBV+MBry5w5vlNw1El2aO3ERtWwjxrN0sLKkqht0h7hU/+wc7+5aBV4URFoNx2E\n" + "EkcZZVknVD9EMvNlWnVVQoLnOIIW4e5F4yHqHQTdxM1TD4F0gKjfNwGK6xZNpObG\n" + "QbDfN3wSkU7DIxeNJCMB+Uc5GDHMKNiEg0yb59SEvypiDuU6cD7OuhLQM0gbjXlC\n" + "81hvjyhx/T/mRQhf6MOu8RbVdp5CDp7IqhouLwEHvHjS4bA/AZIuIP8=\n" + "-----END CERTIFICATE-----\n" +) + + +def _eval_prepare_wrapper( + *, + token: str | None = RUN_TOKEN, + eval_run_id: str = EVAL_RUN_ID, +) -> dict[str, Any]: + eval_image = "registry.example/eval@sha256:" + "b" * 64 + policy = { + "schema_version": 1, + "per_task_aggregation": "mean", + "keep_policy": "off", + "drop_lowest_n": 0, + "threshold_f64be": None, + } + compose = generate_app_compose( + orchestrator_image=eval_image, + name="eval-v1", + key_release_url="validator.example:8701", + allowed_envs=eval_deploy.EVAL_ALLOWED_ENVS, + ) + compose_hash = hashlib.sha256(render_app_compose(compose).encode()).hexdigest() + run_token = token if isinstance(token, str) and token else "placeholder-token" + plan = { + "schema_version": 1, + "eval_run_id": eval_run_id, + "submission_id": "1", + "submission_version": 1, + "authorizing_review_digest": "d" * 64, + "agent_hash": "e" * 64, + "package_tree_sha": "b" * 64, + "selected_tasks": [ + { + "task_id": "task-1", + "image_ref": "registry.example/task@sha256:" + "f" * 64, + "task_config_sha256": "1" * 64, + } + ], + "k": 1, + "scoring_policy": policy, + "scoring_policy_digest": eval_wire.scoring_policy_digest(policy), + "eval_app": { + "image_ref": eval_image, + "compose_hash": compose_hash, + "app_identity": "eval-v1", + "kms_key_algorithm": "x25519", + "kms_public_key_hex": PUBLIC_KEY, + "kms_public_key_sha256": hashlib.sha256(bytes.fromhex(PUBLIC_KEY)).hexdigest(), + "measurement": MEASUREMENT, + }, + "key_release_endpoint": "validator.example:8701", + "result_endpoint": f"/evaluation/v1/runs/{eval_run_id}/result", + "key_release_nonce": "key-release-nonce", + "score_nonce": "score-nonce", + "run_token_sha256": hashlib.sha256(run_token.encode()).hexdigest(), + "issued_at_ms": 1, + "expires_at_ms": 2, + } + validated = eval_wire.validate_eval_plan(plan) + delivery: dict[str, str] | None + if isinstance(token, str) and token: + delivery = {"env_key": "EVAL_RUN_TOKEN", "token": token} + else: + delivery = None + return { + "schema_version": 1, + "plan": validated, + "plan_sha256": hashlib.sha256(eval_wire.canonical_json_v1(validated)).hexdigest(), + "secret_delivery": delivery, + } + + +def _deploy_args(**overrides: Any) -> SimpleNamespace: + base = dict( + eval_command="deploy", + submission_id=1, + base_url="https://challenge.example", + hotkey="hk", + signature="sig", + nonce="n", + timestamp=None, + auto_sign=True, + prepare_response=None, + gateway_token_env="BASE_GATEWAY_TOKEN", + gateway_url_env="BASE_LLM_GATEWAY_URL", + llm_cost_limit_env="LLM_COST_LIMIT", + phala_api=None, + review_instance_type="tdx.small", + eval_instance_type="tdx.small", + review_runtime_hours=1.0, + eval_runtime_hours=1.0, + money_cap_usd=20.0, + dry_run=False, + token_output=None, + emit_run_token=False, + output=None, + ) + base.update(overrides) + return SimpleNamespace(**base) + + +def _wire_successful_deploy( + monkeypatch: pytest.MonkeyPatch, + *, + token: str = RUN_TOKEN, + captured_secrets: list[dict[str, str]] | None = None, +) -> list[Any]: + """Fake prepare + Phala deploy; optionally capture encrypt secrets.""" + + prepare = _eval_prepare_wrapper(token=token) + fake_client = MagicMock() + fake_client.eval_prepare.return_value = prepare + monkeypatch.setattr(cli, "_route_client", lambda _args: fake_client) + monkeypatch.setenv("LLM_COST_LIMIT", "1.00") + monkeypatch.setenv("CHALLENGE_PHALA_RA_TLS_SERVER_CA_PEM", _SERVER_CA_PEM) + + if captured_secrets is not None: + real_encrypt = eval_deploy.encrypt_eval_secrets + + def _capture(plan: Any, secrets: Any) -> Any: + captured_secrets.append(dict(secrets)) + return real_encrypt(plan, secrets) + + monkeypatch.setattr(eval_deploy, "encrypt_eval_secrets", _capture) + + class _FixedDeploy: + def __init__(self, _api: object) -> None: + pass + + def deploy(self, plan_obj: Any, encrypted_obj: Any) -> dict[str, Any]: + assert plan_obj.eval_run_token == token + assert "EVAL_RUN_TOKEN" in encrypted_obj.env_keys + return { + "schema_version": 1, + "eval_run_id": plan_obj.eval_run_id, + "cvm_id": "cvm-eval-handoff-1", + "phala_create_receipt": { + "request_id": "req", + "app_id": plan_obj.app_identity, + "cvm_id": "cvm-eval-handoff-1", + "receipt_sha256": "a" * 64, + "created_at_ms": 1, + }, + } + + monkeypatch.setattr(eval_deploy, "HttpEvalPhalaDeployment", _FixedDeploy) + monkeypatch.setattr(cli, "PhalaCloudClient", lambda **_k: object()) + printed: list[Any] = [] + monkeypatch.setattr(cli, "_print", printed.append) + return printed + + +# --------------------------------------------------------------------------- # +# S1 — stdout emission with --emit-run-token +# --------------------------------------------------------------------------- # + + +def test_emit_run_token_puts_exact_token_and_run_id_on_stdout( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Given live deploy + --emit-run-token, When deploy succeeds, + Then stdout JSON carries exact eval_run_token and eval_run_id. + """ + + printed = _wire_successful_deploy(monkeypatch) + args = _deploy_args(emit_run_token=True) + code = cli._ordered_eval_command(args) + assert code == 0, printed + assert len(printed) == 1 + payload = printed[0] + assert payload["eval_run_token"] == RUN_TOKEN + assert payload["eval_run_id"] == EVAL_RUN_ID + assert payload["stage"] == "eval_deployed" + + +# --------------------------------------------------------------------------- # +# S2 — --token-output secure file +# --------------------------------------------------------------------------- # + + +def test_token_output_writes_mode_0600_file_with_exact_token( + monkeypatch: pytest.MonkeyPatch, + tmp_path: pytest.TempPathFactory, +) -> None: + """Given --token-output PATH, When deploy succeeds, + Then PATH exists mode 0o600 and contains the exact token; stdout has run id. + """ + + token_path = tmp_path / "eval-run.token" + printed = _wire_successful_deploy(monkeypatch) + args = _deploy_args(token_output=str(token_path)) + code = cli._ordered_eval_command(args) + assert code == 0, printed + assert token_path.is_file() + mode = stat.S_IMODE(token_path.stat().st_mode) + assert mode == 0o600, f"expected 0o600, got {oct(mode)}" + assert token_path.read_text(encoding="utf-8") == RUN_TOKEN + assert printed[0]["eval_run_id"] == EVAL_RUN_ID + # Token must not appear in stdout when only --token-output is used. + assert RUN_TOKEN not in json.dumps(printed) + assert "eval_run_token" not in printed[0] + + +# --------------------------------------------------------------------------- # +# S3 — fail closed without handoff flags +# --------------------------------------------------------------------------- # + + +def test_non_dry_run_without_handoff_flags_fails_closed( + monkeypatch: pytest.MonkeyPatch, + capsys: pytest.CaptureFixture[str], +) -> None: + """Given non-dry-run deploy with neither flag, When invoked, + Then non-zero exit and stderr names both flags (no Phala spend). + """ + + prepare = _eval_prepare_wrapper() + fake_client = MagicMock() + fake_client.eval_prepare.return_value = prepare + monkeypatch.setattr(cli, "_route_client", lambda _args: fake_client) + monkeypatch.setenv("LLM_COST_LIMIT", "1.00") + + deploy_calls: list[Any] = [] + + class _MustNotDeploy: + def __init__(self, _api: object) -> None: + pass + + def deploy(self, plan_obj: Any, encrypted_obj: Any) -> dict[str, Any]: + deploy_calls.append((plan_obj, encrypted_obj)) + raise AssertionError("Phala deploy must not run when handoff flags missing") + + monkeypatch.setattr(eval_deploy, "HttpEvalPhalaDeployment", _MustNotDeploy) + monkeypatch.setattr(cli, "PhalaCloudClient", lambda **_k: object()) + + args = _deploy_args(dry_run=False, token_output=None, emit_run_token=False) + code = cli._ordered_eval_command(args) + err = capsys.readouterr().err + assert code == 2 + assert deploy_calls == [] + assert "--token-output" in err + assert "--emit-run-token" in err + assert "eval result" in err.lower() or "EVAL_RUN_TOKEN" in err + assert RUN_TOKEN not in err + + +def test_non_dry_run_without_handoff_flags_never_spends_prepare_token( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Given non-dry-run deploy with neither flag, When invoked, + Then eval_prepare is never called so the one-shot delivery is not spent. + + Argument validation must precede remote state mutation: prepare consumes the + single EVAL_RUN_TOKEN delivery, so gating after it would strand the miner in + exactly the unrecoverable state this handoff exists to prevent. + """ + + fake_client = MagicMock() + fake_client.eval_prepare.return_value = _eval_prepare_wrapper() + monkeypatch.setattr(cli, "_route_client", lambda _args: fake_client) + monkeypatch.setenv("LLM_COST_LIMIT", "1.00") + + args = _deploy_args(dry_run=False, token_output=None, emit_run_token=False) + code = cli._ordered_eval_command(args) + + assert code == 2 + fake_client.eval_prepare.assert_not_called() + + +# --------------------------------------------------------------------------- # +# S4 — dry-run without flags still OK, no raw token +# --------------------------------------------------------------------------- # + + +def test_dry_run_without_handoff_flags_ok_and_no_raw_token( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Given dry-run with neither handoff flag, When invoked, + Then exit 0 and stdout has no raw token. + """ + + prepare = _eval_prepare_wrapper() + fake_client = MagicMock() + fake_client.eval_prepare.return_value = prepare + monkeypatch.setattr(cli, "_route_client", lambda _args: fake_client) + monkeypatch.setenv("LLM_COST_LIMIT", "1.00") + printed: list[Any] = [] + monkeypatch.setattr(cli, "_print", printed.append) + + args = _deploy_args(dry_run=True, token_output=None, emit_run_token=False) + code = cli._ordered_eval_command(args) + assert code == 0, printed + assert printed[0]["dry_run"] is True + assert printed[0]["eval_run_id"] == EVAL_RUN_ID + assert RUN_TOKEN not in json.dumps(printed) + assert "eval_run_token" not in printed[0] + + +# --------------------------------------------------------------------------- # +# S5 — --output plan JSON never contains token +# --------------------------------------------------------------------------- # + + +def test_output_plan_json_never_contains_token( + monkeypatch: pytest.MonkeyPatch, + tmp_path: pytest.TempPathFactory, +) -> None: + """Given deploy --output PATH (+ handoff via token-output), When success, + Then the plan/metadata JSON at PATH never contains the raw token string. + """ + + out_path = tmp_path / "deploy-plan.json" + token_path = tmp_path / "token" + printed = _wire_successful_deploy(monkeypatch) + args = _deploy_args( + emit_run_token=True, # even when stdout has token, --output must not + token_output=str(token_path), + output=str(out_path), + ) + code = cli._ordered_eval_command(args) + assert code == 0, printed + assert out_path.is_file() + body = out_path.read_text(encoding="utf-8") + assert RUN_TOKEN not in body + parsed = json.loads(body) + assert "eval_run_token" not in parsed + # Nested secret_delivery must stay redacted if present. + if "secret_delivery" in parsed: + assert parsed["secret_delivery"] == {"env_key": "EVAL_RUN_TOKEN"} or ( + isinstance(parsed["secret_delivery"], dict) and "token" not in parsed["secret_delivery"] + ) + + +# --------------------------------------------------------------------------- # +# S6 — prepare / status still redact +# --------------------------------------------------------------------------- # + + +def test_eval_prepare_and_status_still_redact_secret_delivery( + monkeypatch: pytest.MonkeyPatch, + tmp_path: pytest.TempPathFactory, +) -> None: + """REGRESSION: prepare/status redaction of secret_delivery is unchanged.""" + + prepare = _eval_prepare_wrapper(token=RUN_TOKEN) + fake_client = MagicMock() + fake_client.eval_prepare.return_value = prepare + fake_client.eval_status.return_value = { + "schema_version": 1, + "eval_run_id": EVAL_RUN_ID, + "secret_delivery": {"env_key": "EVAL_RUN_TOKEN", "token": RUN_TOKEN}, + "phase": "eval_prepared", + } + monkeypatch.setattr(cli, "_route_client", lambda _args: fake_client) + printed: list[Any] = [] + monkeypatch.setattr(cli, "_print", printed.append) + + out_path = tmp_path / "prepare.json" + prep_args = SimpleNamespace( + eval_command="prepare", + submission_id=1, + base_url="https://challenge.example", + hotkey="hk", + signature="sig", + nonce="n", + timestamp=None, + auto_sign=True, + output=str(out_path), + ) + assert cli._ordered_eval_command(prep_args) == 0 + assert printed[0]["secret_delivery"] == {"env_key": "EVAL_RUN_TOKEN"} + assert RUN_TOKEN not in json.dumps(printed[0]) + assert RUN_TOKEN not in out_path.read_text(encoding="utf-8") + assert json.loads(out_path.read_text(encoding="utf-8"))["secret_delivery"] == { + "env_key": "EVAL_RUN_TOKEN" + } + + printed.clear() + status_args = SimpleNamespace( + eval_command="status", + submission_id=1, + base_url="https://challenge.example", + hotkey="hk", + signature="sig", + nonce="n", + timestamp=None, + auto_sign=True, + cursor=None, + ) + assert cli._ordered_eval_command(status_args) == 0 + assert printed[0]["secret_delivery"] == {"env_key": "EVAL_RUN_TOKEN"} + assert RUN_TOKEN not in json.dumps(printed[0]) + + +# --------------------------------------------------------------------------- # +# S7 — token never in exception messages +# --------------------------------------------------------------------------- # + + +def test_token_never_present_in_raised_exception_messages( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Given a deploy path that raises after token is known, When error surfaces, + Then exception / stderr text never contains the raw token. + """ + + prepare = _eval_prepare_wrapper(token=RUN_TOKEN) + fake_client = MagicMock() + fake_client.eval_prepare.return_value = prepare + monkeypatch.setattr(cli, "_route_client", lambda _args: fake_client) + monkeypatch.setenv("LLM_COST_LIMIT", "1.00") + monkeypatch.setenv("CHALLENGE_PHALA_RA_TLS_SERVER_CA_PEM", _SERVER_CA_PEM) + + class _Boom: + def __init__(self, _api: object) -> None: + pass + + def deploy(self, plan_obj: Any, encrypted_obj: Any) -> dict[str, Any]: + raise eval_deploy.EvalDeploymentError( + f"post-create bind failed for run {plan_obj.eval_run_id}" + ) + + monkeypatch.setattr(eval_deploy, "HttpEvalPhalaDeployment", _Boom) + monkeypatch.setattr(cli, "PhalaCloudClient", lambda **_k: object()) + + args = _deploy_args(emit_run_token=True) + # Capture via raising path inside _ordered_eval_command (prints error: …). + import io + import sys + + buf = io.StringIO() + old = sys.stderr + sys.stderr = buf + try: + code = cli._ordered_eval_command(args) + finally: + sys.stderr = old + err = buf.getvalue() + assert code == 2 + assert RUN_TOKEN not in err + # Fail-closed message path also must not embed token. + with pytest.raises(RouteClientError) as excinfo: + # Direct unit: the handoff guard must not interpolate the token. + raise RouteClientError( + "eval deploy requires --token-output PATH and/or --emit-run-token " + "so the miner can later call eval result with EVAL_RUN_TOKEN" + ) + assert RUN_TOKEN not in str(excinfo.value) + + +# --------------------------------------------------------------------------- # +# S8 — CVM encrypted env still receives EVAL_RUN_TOKEN +# --------------------------------------------------------------------------- # + + +def test_eval_run_token_still_injected_into_cvm_encrypted_env( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Adjacent: handoff must not remove EVAL_RUN_TOKEN from CVM env injection.""" + + captured: list[dict[str, str]] = [] + printed = _wire_successful_deploy(monkeypatch, captured_secrets=captured) + args = _deploy_args(emit_run_token=True) + assert cli._ordered_eval_command(args) == 0 + assert captured, printed + assert captured[0]["EVAL_RUN_TOKEN"] == RUN_TOKEN + + +def test_redact_capabilities_still_strips_token_key() -> None: + """Unit: _redact_capabilities never leaves token bytes in nested payloads.""" + + raw = { + "secret_delivery": {"env_key": "EVAL_RUN_TOKEN", "token": RUN_TOKEN}, + "EVAL_RUN_TOKEN": RUN_TOKEN, + "nested": {"token": RUN_TOKEN}, + } + redacted = cli._redact_capabilities(raw) + dumped = json.dumps(redacted) + assert RUN_TOKEN not in dumped + assert redacted["secret_delivery"] == {"env_key": "EVAL_RUN_TOKEN"} diff --git a/packages/challenges/agent-challenge/tests/test_eval_keyrelease_ratls_bootstrap.py b/packages/challenges/agent-challenge/tests/test_eval_keyrelease_ratls_bootstrap.py index b5050add8..39033f267 100644 --- a/packages/challenges/agent-challenge/tests/test_eval_keyrelease_ratls_bootstrap.py +++ b/packages/challenges/agent-challenge/tests/test_eval_keyrelease_ratls_bootstrap.py @@ -507,8 +507,8 @@ def _stub_eval_plan(**overrides: Any) -> dict[str, Any]: }, ], "k": 1, + "n_concurrent": 4, "agent_hash": "f" * 64, - "package_tree_sha": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "scoring_policy": { "schema_version": 1, "per_task_aggregation": "mean", @@ -516,7 +516,7 @@ def _stub_eval_plan(**overrides: Any) -> dict[str, Any]: }, "eval_app": { "app_identity": "agent-challenge-eval-v1", - "image_ref": "docker.io/mathiiss/agent-challenge-canonical@sha256:" + "d" * 64, + "image_ref": "ghcr.io/baseintelligence/agent-challenge-canonical@sha256:" + "d" * 64, "compose_hash": "e" * 64, "measurement": { "mrtd": "1" * 96, diff --git a/packages/challenges/agent-challenge/tests/test_live_registry.py b/packages/challenges/agent-challenge/tests/test_live_registry.py index c858c7943..e612fda75 100644 --- a/packages/challenges/agent-challenge/tests/test_live_registry.py +++ b/packages/challenges/agent-challenge/tests/test_live_registry.py @@ -4,7 +4,7 @@ Terminal-Bench task by a *bare content digest* (``harbor_registry_ref = "sha256:<64hex>"``) with NO repository, so an in-CVM DooD orchestrator cannot ``docker pull`` it. For a live smoke E2E a small deterministic subset of task -images is published to a pullable, digest-pinned registry ref and recorded in a +images is published to a pullable, digest-pinned GHCR ref and recorded in a SEPARATE side manifest (``golden/live-registry-refs.json``) so the frozen content digests / canonical measurement stay byte-identical. @@ -13,6 +13,8 @@ offline/flag-off behavior is byte-identical); * every published ref is repository-qualified AND digest-pinned (a bare ``sha256:...`` content digest or a floating tag is rejected); + * D6: live-registry parse rejects Hub registry hosts and the retired personal + namespace; shipped golden is GHCR-only; * the shipped side manifest is a strict subset of the frozen golden tasks and never mutates ``dataset-digest.json``. """ @@ -36,7 +38,12 @@ FROZEN_CANONICAL_CONTENT_DIGEST = "8da006d76bcf59c2af3f36ed4420192d3930bda43683f32d80013a6ee5e7e02d" FROZEN_TASK_COUNT = 89 -_GOOD_REF = "docker.io/mathiiss/agent-challenge-tb21-x@sha256:" + ("a" * 64) +_GOOD_REF = "ghcr.io/baseintelligence/agent-challenge-tb21-x@sha256:" + ("a" * 64) + +# Forbidden shipping hosts/namespaces (D6). Kept as test inputs only. +_HUB_REF = "docker.io/library/x@sha256:" + ("a" * 64) +_RETIRED_NS_REF = "docker.io/mathiiss/agent-challenge-tb21-x@sha256:" + ("a" * 64) +_OTHER_REG_REF = "quay.io/example/x@sha256:" + ("a" * 64) # --------------------------------------------------------------------------- # @@ -45,14 +52,15 @@ def test_pullable_ref_accepts_repo_digest(): assert lr.is_pullable_ref(_GOOD_REF) assert lr.assert_pullable_ref(_GOOD_REF) == _GOOD_REF + assert lr.assert_live_registry_ref(_GOOD_REF) == _GOOD_REF @pytest.mark.parametrize( "bad", [ "sha256:" + ("a" * 64), # bare content digest (the golden behavior) - not pullable - "docker.io/mathiiss/x:latest", # floating tag, not digest-pinned - "docker.io/mathiiss/x@sha256:" + ("a" * 63), # short digest + "ghcr.io/baseintelligence/x:latest", # floating tag, not digest-pinned + "ghcr.io/baseintelligence/x@sha256:" + ("a" * 63), # short digest "plainname@sha256:" + ("a" * 64), # no repository/namespace ('/') "", 123, @@ -104,6 +112,25 @@ def test_parse_rejects_bare_digest_ref(): lr.parse_live_registry({"tasks": {"foo": {"registry_ref": "sha256:" + "a" * 64}}}) +@pytest.mark.parametrize( + "bad", + [ + _HUB_REF, + _RETIRED_NS_REF, + _OTHER_REG_REF, + "ghcr.io/mathiiss/x@sha256:" + ("a" * 64), + ], +) +def test_live_registry_ref_rejects_non_ghcr_shipping_hosts(bad): + """D6: live shipping refs must be ghcr.io and must not use retired namespaces.""" + with pytest.raises(lr.LiveRegistryError): + lr.assert_live_registry_ref(bad) + with pytest.raises(lr.LiveRegistryError): + lr.parse_live_registry({"tasks": {"foo": {"registry_ref": bad}}}) + with pytest.raises(lr.LiveRegistryError): + lr.parse_live_registry({"tasks": {}, "orchestrator_image": bad}) + + # --------------------------------------------------------------------------- # # task_id resolution (bare + dataset-prefixed) # --------------------------------------------------------------------------- # @@ -128,15 +155,32 @@ def test_shipped_live_manifest_is_valid_and_pullable(): reg = lr.load_live_registry(LIVE_MANIFEST) assert reg.task_refs, "shipped live manifest has no task refs" for task_id, ref in reg.task_refs.items(): - assert lr.is_pullable_ref(ref), (task_id, ref) + assert lr.assert_live_registry_ref(ref) == ref, (task_id, ref) def test_shipped_orchestrator_image_is_digest_pinned(): reg = lr.load_live_registry(LIVE_MANIFEST) assert reg.orchestrator_image is not None - assert lr.is_pullable_ref(reg.orchestrator_image) + assert lr.assert_live_registry_ref(reg.orchestrator_image) == reg.orchestrator_image # The deploy path's digest-pin guard accepts it (no bare tag). assert c.assert_digest_pinned(reg.orchestrator_image) == reg.orchestrator_image + assert reg.orchestrator_image.startswith( + "ghcr.io/baseintelligence/agent-challenge-canonical@sha256:" + ) + digest = reg.orchestrator_image.rsplit("@", 1)[-1] + assert digest.startswith("sha256:") and len(digest) == len("sha256:") + 64 + + +def test_shipped_live_manifest_has_no_hub_or_retired_namespace(): + """Golden live-registry product pin must not mention Hub host or retired ns.""" + text = LIVE_MANIFEST.read_text(encoding="utf-8") + assert "docker.io" not in text + assert "mathiiss" not in text + live = json.loads(text) + assert live["namespace"] == "ghcr.io/baseintelligence" + assert live["orchestrator_image"].startswith("ghcr.io/baseintelligence/") + for task_id, entry in live["tasks"].items(): + assert entry["registry_ref"].startswith("ghcr.io/baseintelligence/"), task_id def test_shipped_live_subset_is_subset_of_golden_and_small(): diff --git a/packages/challenges/agent-challenge/tests/test_ordered_selfdeploy_cli.py b/packages/challenges/agent-challenge/tests/test_ordered_selfdeploy_cli.py index fa9fd37a8..468e62239 100644 --- a/packages/challenges/agent-challenge/tests/test_ordered_selfdeploy_cli.py +++ b/packages/challenges/agent-challenge/tests/test_ordered_selfdeploy_cli.py @@ -214,18 +214,24 @@ def test_eval_encrypted_env_contains_only_scoped_capabilities_and_is_transmitted def test_lifecycle_budget_counts_review_and_eval_together(): + # Compute + default stage disks (20 GB each here) over 100h. + # CPU: 0.058 * 200 = 11.6; disk: 0.000139 * 20 * 200 = 0.556 → 12.156 estimate = lifecycle.projected_lifecycle_cost_usd( review_instance_type="tdx.small", eval_instance_type="tdx.small", review_runtime_hours=100, eval_runtime_hours=100, + review_disk_size_gb=20, + eval_disk_size_gb=20, ) - assert estimate == pytest.approx(11.6) + assert estimate == pytest.approx(12.156) with pytest.raises(lifecycle.LifecycleBudgetError): lifecycle.validate_lifecycle_budget( review_instance_type="tdx.small", eval_instance_type="tdx.xlarge", review_runtime_hours=100, eval_runtime_hours=100, + review_disk_size_gb=20, + eval_disk_size_gb=100, money_cap_usd=20, ) diff --git a/packages/challenges/agent-challenge/tests/test_own_runner_backend_live_registry.py b/packages/challenges/agent-challenge/tests/test_own_runner_backend_live_registry.py index b37b6ba26..d659d6f5e 100644 --- a/packages/challenges/agent-challenge/tests/test_own_runner_backend_live_registry.py +++ b/packages/challenges/agent-challenge/tests/test_own_runner_backend_live_registry.py @@ -23,7 +23,7 @@ def __init__(self, **kwargs): monkeypatch.setattr(backend, "TaskContainerBuilder", _SpyBuilder) - ref = "docker.io/mathiiss/agent-challenge-tb21-foo@sha256:" + ("a" * 64) + ref = "ghcr.io/baseintelligence/agent-challenge-tb21-foo@sha256:" + ("a" * 64) backend._build_default_preparer( task_ids=[], cache_root=backend.DEFAULT_CACHE_ROOT, @@ -60,7 +60,7 @@ def __init__(self, **kwargs): def test_main_resolves_live_refs_from_env(monkeypatch, tmp_path): - ref = "docker.io/mathiiss/agent-challenge-tb21-foo@sha256:" + ("b" * 64) + ref = "ghcr.io/baseintelligence/agent-challenge-tb21-foo@sha256:" + ("b" * 64) manifest = tmp_path / "live-registry-refs.json" manifest.write_text(json.dumps({"tasks": {"foo": {"registry_ref": ref}}}), encoding="utf-8") diff --git a/packages/challenges/agent-challenge/tests/test_own_runner_live_registry_resolution.py b/packages/challenges/agent-challenge/tests/test_own_runner_live_registry_resolution.py index a1224a9ea..cc0ac5312 100644 --- a/packages/challenges/agent-challenge/tests/test_own_runner_live_registry_resolution.py +++ b/packages/challenges/agent-challenge/tests/test_own_runner_live_registry_resolution.py @@ -20,7 +20,7 @@ from agent_challenge.evaluation.own_runner import container_builder as cb from agent_challenge.evaluation.own_runner.taskdefs import ParsedTask, parse_task -_LIVE_REF = "docker.io/mathiiss/agent-challenge-tb21-foo@sha256:" + ("a" * 64) +_LIVE_REF = "ghcr.io/baseintelligence/agent-challenge-tb21-foo@sha256:" + ("a" * 64) def _write_task(root: Path, *, docker_image: str | None) -> ParsedTask: diff --git a/packages/challenges/agent-challenge/tests/test_phala_create_ack_and_cli_token.py b/packages/challenges/agent-challenge/tests/test_phala_create_ack_and_cli_token.py index 2d916f5fe..7a8124be9 100644 --- a/packages/challenges/agent-challenge/tests/test_phala_create_ack_and_cli_token.py +++ b/packages/challenges/agent-challenge/tests/test_phala_create_ack_and_cli_token.py @@ -7,6 +7,8 @@ closed product_create_response_missing_cvm_id_field. 3. CLI review deploy consumes a review_retry one-time token path without fail-closing solely because prepare already delivered (null re-prepare). +4. CLI eval deploy recovers EVAL_RUN_TOKEN via cancel+retry when prepare + returns token-less secret_delivery (production residual: submission 3). Never invent TEE measurements. Secrets never appear in errors or logs. """ @@ -21,6 +23,7 @@ import pytest +from agent_challenge.canonical import eval_wire from agent_challenge.review.canonical import canonical_sha256 from agent_challenge.review.compose import ( generate_review_app_compose, @@ -28,7 +31,9 @@ ) from agent_challenge.review.schemas import ReviewInputConfig, build_review_assignment from agent_challenge.selfdeploy import cli +from agent_challenge.selfdeploy import eval as eval_deploy from agent_challenge.selfdeploy import review as review_mod +from agent_challenge.selfdeploy.client import RouteClientError from agent_challenge.selfdeploy.phala import ( DEFAULT_PHALA_USER_AGENT, PhalaCloudClient, @@ -460,7 +465,7 @@ def deploy(self, plan_obj: Any, encrypted_obj: Any) -> dict[str, Any]: money_cap_usd=20.0, openrouter_key_env="OPENROUTER_API_KEY", phala_api=None, - base_url="https://challenge.example", + base_url="https://chain.joinbase.ai/challenges/agent-challenge", hotkey="hk", timestamp=None, ) @@ -535,7 +540,7 @@ def deploy(self, plan_obj: Any, encrypted_obj: Any) -> dict[str, Any]: money_cap_usd=20.0, openrouter_key_env="OPENROUTER_API_KEY", phala_api=None, - base_url="https://challenge.example", + base_url="https://chain.joinbase.ai/challenges/agent-challenge", hotkey="hk", timestamp=None, ) @@ -660,3 +665,154 @@ def test_obtain_review_prepare_only_cancel_retries_when_token_null_and_terminal( assert out["review_session_token"] == retry_token fake_client.review_cancel.assert_called_once_with(7, "assignment-1") fake_client.review_retry.assert_called_once() + + +def _eval_prepare_wrapper(*, token: str | None, eval_run_id: str = "eval-1") -> dict[str, Any]: + """Minimal signed-shape Eval prepare wrapper for CLI recovery unit tests.""" + + import hashlib + + from agent_challenge.canonical.compose import ( + generate_app_compose, + render_app_compose, + ) + + eval_image = "registry.example/eval@sha256:" + "b" * 64 + policy = { + "schema_version": 1, + "per_task_aggregation": "mean", + "keep_policy": "off", + "drop_lowest_n": 0, + "threshold_f64be": None, + } + compose = generate_app_compose( + orchestrator_image=eval_image, + name="eval-v1", + key_release_url="validator.example:8701", + allowed_envs=eval_deploy.EVAL_ALLOWED_ENVS, + ) + compose_hash = hashlib.sha256(render_app_compose(compose).encode()).hexdigest() + run_token = token if isinstance(token, str) and token else "placeholder-token" + plan = { + "schema_version": 1, + "eval_run_id": eval_run_id, + "submission_id": "1", + "submission_version": 1, + "authorizing_review_digest": "d" * 64, + "agent_hash": "e" * 64, + "package_tree_sha": "b" * 64, + "selected_tasks": [ + { + "task_id": "task-1", + "image_ref": "registry.example/task@sha256:" + "f" * 64, + "task_config_sha256": "1" * 64, + } + ], + "k": 1, + "scoring_policy": policy, + "scoring_policy_digest": eval_wire.scoring_policy_digest(policy), + "eval_app": { + "image_ref": eval_image, + "compose_hash": compose_hash, + "app_identity": "eval-v1", + "kms_key_algorithm": "x25519", + "kms_public_key_hex": PUBLIC_KEY, + "kms_public_key_sha256": hashlib.sha256(bytes.fromhex(PUBLIC_KEY)).hexdigest(), + "measurement": MEASUREMENT, + }, + "key_release_endpoint": "validator.example:8701", + "result_endpoint": f"/evaluation/v1/runs/{eval_run_id}/result", + "key_release_nonce": "key-release-nonce", + "score_nonce": "score-nonce", + "run_token_sha256": hashlib.sha256(run_token.encode()).hexdigest(), + "issued_at_ms": 1, + "expires_at_ms": 2, + } + validated = eval_wire.validate_eval_plan(plan) + delivery: dict[str, str] | None + if isinstance(token, str) and token: + delivery = {"env_key": "EVAL_RUN_TOKEN", "token": token} + else: + delivery = None + return { + "schema_version": 1, + "plan": validated, + "plan_sha256": hashlib.sha256(eval_wire.canonical_json_v1(validated)).hexdigest(), + "secret_delivery": delivery, + } + + +def test_eval_token_present_requires_exact_env_key_token_shape() -> None: + """Presence predicate matches eval.py secret_delivery contract; no relaxation.""" + + assert cli._eval_token_present(None) is False + assert cli._eval_token_present({}) is False + assert cli._eval_token_present({"secret_delivery": None}) is False + assert cli._eval_token_present({"secret_delivery": {"env_key": "EVAL_RUN_TOKEN"}}) is False + assert ( + cli._eval_token_present( + {"secret_delivery": {"env_key": "EVAL_RUN_TOKEN", "token": "", "extra": 1}} + ) + is False + ) + assert ( + cli._eval_token_present( + {"secret_delivery": {"env_key": "EVAL_RUN_TOKEN", "token": "run-tok"}} + ) + is True + ) + # Validation shape in build_eval_deployment_plan remains fail-closed. + bare = _eval_prepare_wrapper(token=None) + with pytest.raises(eval_deploy.EvalDeploymentError, match="EVAL_RUN_TOKEN capability"): + eval_deploy.build_eval_deployment_plan(bare) + + +def test_obtain_eval_prepare_returns_first_prepare_when_token_present() -> None: + """Token already delivered on prepare → never cancel or retry.""" + + fresh = _eval_prepare_wrapper(token="eval-fresh-token", eval_run_id="eval-run-a") + fake_client = MagicMock() + fake_client.eval_prepare.return_value = fresh + out = cli._obtain_eval_prepare_with_token(fake_client, 3) + assert out is fresh + assert out["secret_delivery"]["token"] == "eval-fresh-token" + fake_client.eval_prepare.assert_called_once_with(3) + fake_client.eval_cancel.assert_not_called() + fake_client.eval_retry.assert_not_called() + + +def test_obtain_eval_prepare_cancel_retries_when_token_absent() -> None: + """Spent one-shot token: prepare token-less → cancel+retry recovers capability. + + Production residual (submission 3): standalone ``eval prepare`` or + ``eval retry`` spends EVAL_RUN_TOKEN; subsequent ``eval deploy`` saw + secret_delivery=None and hard-failed with no recovery path. + """ + + spent = _eval_prepare_wrapper(token=None, eval_run_id="eval-run-1") + recovered_token = "eval-retry-token-fresh" + recovered = _eval_prepare_wrapper(token=recovered_token, eval_run_id="eval-run-2") + fake_client = MagicMock() + fake_client.eval_prepare.return_value = spent + fake_client.eval_cancel.return_value = {"phase": "eval_cancelled"} + fake_client.eval_retry.return_value = recovered + out = cli._obtain_eval_prepare_with_token(fake_client, 3) + assert out["secret_delivery"]["token"] == recovered_token + fake_client.eval_prepare.assert_called_once_with(3) + fake_client.eval_cancel.assert_called_once_with(3, "eval-run-1") + fake_client.eval_retry.assert_called_once_with(3, "eval-run-1") + + +def test_obtain_eval_prepare_raises_when_token_still_absent_after_retry() -> None: + """Sticky token-less after cancel+retry → typed RouteClientError.""" + + spent = _eval_prepare_wrapper(token=None, eval_run_id="eval-run-stuck") + still_spent = _eval_prepare_wrapper(token=None, eval_run_id="eval-run-stuck") + fake_client = MagicMock() + fake_client.eval_prepare.return_value = spent + fake_client.eval_cancel.return_value = {"phase": "eval_cancelled"} + fake_client.eval_retry.return_value = still_spent + with pytest.raises(RouteClientError, match="eval run token unavailable"): + cli._obtain_eval_prepare_with_token(fake_client, 3) + fake_client.eval_cancel.assert_called_once_with(3, "eval-run-stuck") + fake_client.eval_retry.assert_called_once_with(3, "eval-run-stuck") diff --git a/packages/challenges/agent-challenge/tests/test_residual_orch_probes.py b/packages/challenges/agent-challenge/tests/test_residual_orch_probes.py index dc63dc713..6aeeed8b7 100644 --- a/packages/challenges/agent-challenge/tests/test_residual_orch_probes.py +++ b/packages/challenges/agent-challenge/tests/test_residual_orch_probes.py @@ -385,7 +385,9 @@ def fake_request( return 200, [ { "Id": "sha256:orch", - "RepoTags": ["docker.io/mathiiss/agent-challenge-canonical@sha256:deadbeef"], + "RepoTags": [ + "ghcr.io/baseintelligence/agent-challenge-canonical@sha256:deadbeef" + ], } ] if method == "POST" and path.startswith("/containers/create"): diff --git a/packages/challenges/agent-challenge/tests/test_selfdeploy_docs_accuracy.py b/packages/challenges/agent-challenge/tests/test_selfdeploy_docs_accuracy.py index f27781a7c..3723c7b79 100644 --- a/packages/challenges/agent-challenge/tests/test_selfdeploy_docs_accuracy.py +++ b/packages/challenges/agent-challenge/tests/test_selfdeploy_docs_accuracy.py @@ -183,12 +183,16 @@ def test_teardown_and_cap_guidance_present_with_valid_commands(): def test_documented_teardown_command_matches_the_cli_implementation(): - # The documented teardown command form must match what the CLI actually runs. + # Teardown must use the HTTP client DELETE path — never a phala binary. import inspect source = inspect.getsource(cli.default_phala_teardown) - assert '"phala", "cvms", "delete"' in source - assert '"-f"' in source + assert "delete_cvm" in source + assert "subprocess" not in source + assert '"phala", "cvms", "delete"' not in source + client_source = inspect.getsource(cli.PhalaCloudClient.delete_cvm) + assert "DELETE" in client_source + assert "/cvms/" in client_source def test_documented_phala_commands_are_valid_when_cli_available(): diff --git a/packages/challenges/agent-challenge/tests/test_selfdeploy_live_registry_wiring.py b/packages/challenges/agent-challenge/tests/test_selfdeploy_live_registry_wiring.py index 660ce0bbe..989495fc3 100644 --- a/packages/challenges/agent-challenge/tests/test_selfdeploy_live_registry_wiring.py +++ b/packages/challenges/agent-challenge/tests/test_selfdeploy_live_registry_wiring.py @@ -16,7 +16,7 @@ from agent_challenge.canonical import live_registry as lr from agent_challenge.selfdeploy import plan as p -PULLABLE = "docker.io/mathiiss/agent-challenge-canonical@sha256:" + ("a" * 64) +PULLABLE = "ghcr.io/baseintelligence/agent-challenge-canonical@sha256:" + ("a" * 64) KEY_URL = "https://validator.example/key-release" diff --git a/packages/challenges/agent-challenge/tests/test_selfdeploy_ordered_trust_hardening.py b/packages/challenges/agent-challenge/tests/test_selfdeploy_ordered_trust_hardening.py index dcad9274e..54cf31dc3 100644 --- a/packages/challenges/agent-challenge/tests/test_selfdeploy_ordered_trust_hardening.py +++ b/packages/challenges/agent-challenge/tests/test_selfdeploy_ordered_trust_hardening.py @@ -548,6 +548,8 @@ def test_review_post_create_failure_deletes_attributable_cvm(monkeypatch): fake_client.review_deployed.side_effect = cli.RouteClientError("signed ack failed") monkeypatch.setattr(cli, "_route_client", lambda _args: fake_client) monkeypatch.setenv("OPENROUTER_API_KEY", "or-secret") + # Offline test double uses a non-joinbase base_url; allow pin override. + monkeypatch.setenv("CHALLENGE_ALLOW_DEV_URLS", "1") deleted: list[str] = [] @@ -685,6 +687,10 @@ def deploy(self, plan, encrypted): # noqa: ARG002 money_cap_usd=20.0, dry_run=False, token_env="EVAL_RUN_TOKEN", + # Live deploy requires an explicit one-time token handoff path. + emit_run_token=True, + token_output=None, + output=None, ) code = cli._ordered_eval_command(args) assert code == 2 diff --git a/packages/challenges/agent-challenge/tests/test_selfdeploy_shapes_disk.py b/packages/challenges/agent-challenge/tests/test_selfdeploy_shapes_disk.py new file mode 100644 index 000000000..d8aa6aa64 --- /dev/null +++ b/packages/challenges/agent-challenge/tests/test_selfdeploy_shapes_disk.py @@ -0,0 +1,297 @@ +"""Disk-aware CPU TDX sizing + stage defaults (offline, no Phala spend). + +Covers: + * stage defaults: review tdx.small/20GB, eval tdx.xlarge/100GB + * disk validator bounds [20, 500] + * disk billing in projected cost + * lifecycle budget includes disk for both stages under the $20 cap + * provision bodies emit disk_size as a sibling of compose_file +""" + +from __future__ import annotations + +from types import SimpleNamespace +from typing import Any +from unittest.mock import MagicMock + +import pytest + +from agent_challenge.selfdeploy import lifecycle, shapes +from agent_challenge.selfdeploy.eval import EvalDeploymentPlan, HttpEvalPhalaDeployment +from agent_challenge.selfdeploy.review import HttpReviewPhalaDeployment, ReviewDeploymentPlan + + +def test_stage_instance_defaults_are_split(): + assert shapes.DEFAULT_REVIEW_INSTANCE_TYPE == "tdx.small" + assert shapes.DEFAULT_EVAL_INSTANCE_TYPE == "tdx.xlarge" + # Backward-compat alias remains the review default. + assert shapes.DEFAULT_INSTANCE_TYPE == shapes.DEFAULT_REVIEW_INSTANCE_TYPE + assert shapes.DEFAULT_REVIEW_DISK_SIZE_GB == 20 + assert shapes.DEFAULT_EVAL_DISK_SIZE_GB == 100 + + +def test_disk_constants_match_decided_billing(): + assert shapes.DISK_USD_PER_GB_HOUR == 0.000139 + assert shapes.MIN_DISK_SIZE_GB == 20 + assert shapes.MAX_DISK_SIZE_GB == 500 + + +@pytest.mark.parametrize("gb", [20, 100, 500]) +def test_validate_disk_size_accepts_bounds(gb: int): + assert shapes.validate_disk_size(gb) == gb + + +@pytest.mark.parametrize("gb", [0, 19, 501, -1, 20.5, "20", None]) +def test_validate_disk_size_refuses_out_of_range(gb: object): + with pytest.raises(shapes.ShapeError): + shapes.validate_disk_size(gb) # type: ignore[arg-type] + + +def test_projected_cost_includes_disk(): + # tdx.xlarge @ 0.464/h * 6h = 2.784; disk 100GB * 0.000139 * 6 = 0.0834 + cpu_only = shapes.projected_cost_usd("tdx.xlarge", max_runtime_hours=6.0) + with_disk = shapes.projected_cost_usd( + "tdx.xlarge", + max_runtime_hours=6.0, + disk_size_gb=100, + ) + assert cpu_only == pytest.approx(0.464 * 6.0) + assert with_disk == pytest.approx(0.464 * 6.0 + 0.000139 * 100 * 6.0) + assert with_disk > cpu_only + + +def test_validate_within_cap_counts_disk(): + # Force over-cap via huge runtime + large disk on xlarge. + with pytest.raises(shapes.OverCapError): + shapes.validate_within_cap( + "tdx.xlarge", + money_cap_usd=1.0, + max_runtime_hours=6.0, + disk_size_gb=500, + ) + + +def test_default_lifecycle_budget_fits_money_cap(): + cost = lifecycle.validate_lifecycle_budget( + review_instance_type=shapes.DEFAULT_REVIEW_INSTANCE_TYPE, + eval_instance_type=shapes.DEFAULT_EVAL_INSTANCE_TYPE, + review_runtime_hours=shapes.DEFAULT_MAX_RUNTIME_HOURS, + eval_runtime_hours=shapes.DEFAULT_MAX_RUNTIME_HOURS, + review_disk_size_gb=shapes.DEFAULT_REVIEW_DISK_SIZE_GB, + eval_disk_size_gb=shapes.DEFAULT_EVAL_DISK_SIZE_GB, + money_cap_usd=shapes.DEFAULT_MONEY_CAP_USD, + ) + assert cost.total_usd <= shapes.DEFAULT_MONEY_CAP_USD + assert cost.total_usd == pytest.approx( + lifecycle.projected_lifecycle_cost_usd( + review_instance_type=shapes.DEFAULT_REVIEW_INSTANCE_TYPE, + eval_instance_type=shapes.DEFAULT_EVAL_INSTANCE_TYPE, + review_runtime_hours=shapes.DEFAULT_MAX_RUNTIME_HOURS, + eval_runtime_hours=shapes.DEFAULT_MAX_RUNTIME_HOURS, + review_disk_size_gb=shapes.DEFAULT_REVIEW_DISK_SIZE_GB, + eval_disk_size_gb=shapes.DEFAULT_EVAL_DISK_SIZE_GB, + ) + ) + # Disk contribution is strictly positive vs CPU-only projection. + cpu_only = ( + shapes.CPU_TDX_SHAPES[shapes.DEFAULT_REVIEW_INSTANCE_TYPE].usd_per_hour + * shapes.DEFAULT_MAX_RUNTIME_HOURS + + shapes.CPU_TDX_SHAPES[shapes.DEFAULT_EVAL_INSTANCE_TYPE].usd_per_hour + * shapes.DEFAULT_MAX_RUNTIME_HOURS + ) + assert cost.total_usd > cpu_only + + +def test_lifecycle_budget_refuses_over_cap_with_disk(): + with pytest.raises(lifecycle.LifecycleBudgetError): + lifecycle.validate_lifecycle_budget( + review_instance_type="tdx.small", + eval_instance_type="tdx.xlarge", + review_runtime_hours=100.0, + eval_runtime_hours=100.0, + review_disk_size_gb=20, + eval_disk_size_gb=100, + money_cap_usd=20.0, + ) + + +def test_lifecycle_budget_refuses_invalid_disk(): + with pytest.raises(lifecycle.LifecycleBudgetError): + lifecycle.validate_lifecycle_budget( + review_instance_type="tdx.small", + eval_instance_type="tdx.xlarge", + review_runtime_hours=1.0, + eval_runtime_hours=1.0, + review_disk_size_gb=10, + eval_disk_size_gb=100, + ) + + +def _minimal_eval_plan(*, disk_size_gb: int = 100) -> EvalDeploymentPlan: + return EvalDeploymentPlan( + plan={"eval_run_id": "run-1"}, + plan_sha256="a" * 64, + compose={"manifest_version": 2, "docker_compose_file": "services: {}\n"}, + compose_text="services: {}\n", + compose_hash="b" * 64, + app_identity="c" * 40, + image_ref="ghcr.io/example/eval@sha256:" + ("d" * 64), + kms_public_key_hex="ab" * 32, + kms_public_key_sha256="e" * 64, + measurement={"vm_shape": "tdx.xlarge"}, + eval_run_id="run-1", + eval_run_token="token-secret", + instance_type="tdx.xlarge", + disk_size_gb=disk_size_gb, + ) + + +def _minimal_review_plan(*, disk_size_gb: int = 20) -> ReviewDeploymentPlan: + return ReviewDeploymentPlan( + assignment={"assignment_core": {"assignment_id": "asg-1"}}, + compose={"manifest_version": 2, "docker_compose_file": "services: {}\n"}, + compose_text="services: {}\n", + compose_hash="b" * 64, + app_identity="c" * 40, + image_ref="ghcr.io/example/review@sha256:" + ("d" * 64), + kms_public_key_hex="ab" * 32, + kms_public_key_sha256="e" * 64, + measurement={"vm_shape": "tdx.small"}, + measurement_allowlist_sha256="f" * 64, + review_session_token="token-secret", + instance_type="tdx.small", + disk_size_gb=disk_size_gb, + ) + + +def test_eval_provision_emits_disk_size_sibling_of_compose(): + captured: list[dict[str, Any]] = [] + + class Api: + def post(self, path: str, payload: dict[str, Any]) -> dict[str, Any]: + if path == "/cvms/provision": + captured.append(dict(payload)) + return { + "compose_hash": "b" * 64, + "app_id": "c" * 40, + "app_env_encrypt_pubkey": "ab" * 32, + "os_image_hash": "0" * 64, + } + return {"id": "cvm-1", "request_id": "req-1", "created_at_ms": 1} + + plan = _minimal_eval_plan(disk_size_gb=100) + encrypted = SimpleNamespace( + eval_run_id=plan.eval_run_id, + app_identity=plan.app_identity, + kms_public_key_sha256=plan.kms_public_key_sha256, + env_keys=("EVAL_RUN_TOKEN",), + ciphertext="cipher", + ) + # Avoid OS-identity side checks by stubbing the private verifier. + dep = HttpEvalPhalaDeployment(Api()) # type: ignore[arg-type] + dep._verify_provision_os_identity = MagicMock() # type: ignore[method-assign] + try: + dep.deploy(plan, encrypted) # type: ignore[arg-type] + except Exception: + # Create path may still fail closed; provision capture is what we need. + pass + assert captured, "provision was never called" + body = captured[0] + assert "disk_size" in body + assert body["disk_size"] == 100 + assert "compose_file" in body + assert body["compose_file"] is plan.compose + # Must not mutate compose document. + assert "disk_size" not in plan.compose + + +def test_review_provision_emits_disk_size_sibling_of_compose(): + captured: list[dict[str, Any]] = [] + + class Api: + def post(self, path: str, payload: dict[str, Any]) -> dict[str, Any]: + if path == "/cvms/provision": + captured.append(dict(payload)) + return { + "compose_hash": "b" * 64, + "app_id": "c" * 40, + "app_env_encrypt_pubkey": "ab" * 32, + "os_image_hash": "0" * 64, + } + return {"id": "cvm-1", "request_id": "req-1", "created_at_ms": 1} + + plan = _minimal_review_plan(disk_size_gb=20) + encrypted = SimpleNamespace( + assignment_id="asg-1", + app_identity=plan.app_identity, + kms_public_key_sha256=plan.kms_public_key_sha256, + measurement_allowlist_sha256=plan.measurement_allowlist_sha256, + env_keys=plan # placeholder replaced below + ) + from agent_challenge.selfdeploy import review as review_mod + + encrypted = SimpleNamespace( + assignment_id="asg-1", + app_identity=plan.app_identity, + kms_public_key_sha256=plan.kms_public_key_sha256, + measurement_allowlist_sha256=plan.measurement_allowlist_sha256, + env_keys=review_mod.REVIEW_ALLOWED_ENVS, + ciphertext="cipher", + ) + dep = HttpReviewPhalaDeployment(Api()) # type: ignore[arg-type] + dep._verify_provision_response = MagicMock() # type: ignore[method-assign] + dep._resolve_created_cvm_id = MagicMock(return_value="cvm-1") # type: ignore[method-assign] + try: + dep.deploy(plan, encrypted) # type: ignore[arg-type] + except Exception: + pass + assert captured, "provision was never called" + body = captured[0] + assert body["disk_size"] == 20 + assert body["compose_file"] is plan.compose + assert "disk_size" not in plan.compose + + +def test_cli_stage_defaults_and_disk_flags(): + from agent_challenge.selfdeploy import cli + + parser = cli.build_parser() + review_ns = parser.parse_args( + [ + "review", + "deploy", + "--base-url", + "https://example.test", + "--submission-id", + "1", + "--hotkey", + "5FakeHotkey", + "--auto-sign", + "--dry-run", + ] + ) + assert review_ns.review_instance_type == "tdx.small" + assert review_ns.eval_instance_type == "tdx.xlarge" + assert review_ns.review_disk_size_gb == 20 + assert review_ns.eval_disk_size_gb == 100 + + eval_ns = parser.parse_args( + [ + "eval", + "deploy", + "--base-url", + "https://example.test", + "--submission-id", + "1", + "--hotkey", + "5FakeHotkey", + "--dry-run", + "--eval-disk-size-gb", + "200", + "--review-disk-size-gb", + "40", + ] + ) + assert eval_ns.eval_instance_type == "tdx.xlarge" + assert eval_ns.eval_disk_size_gb == 200 + assert eval_ns.review_disk_size_gb == 40 diff --git a/packages/challenges/agent-challenge/tests/test_selfdeploy_teardown_http.py b/packages/challenges/agent-challenge/tests/test_selfdeploy_teardown_http.py new file mode 100644 index 000000000..7cdb959ae --- /dev/null +++ b/packages/challenges/agent-challenge/tests/test_selfdeploy_teardown_http.py @@ -0,0 +1,239 @@ +"""HTTP teardown for Phala CVMs (no external ``phala`` binary). + +Covers: + * DELETE /cvms/{id} allowlisted on PhalaCloudClient + * 204 success, 404 idempotent success, other status → PhalaApiError + * default_phala_teardown never shells out to a ``phala`` binary + * optional --cvm-id resolved via GET /cvms + unique app_id match + * ambiguous multi-match refused +""" + +from __future__ import annotations + +import io +import json +from types import SimpleNamespace +from typing import Any +from unittest.mock import MagicMock +from urllib.error import HTTPError +from urllib.request import Request + +import pytest + +from agent_challenge.selfdeploy import cli +from agent_challenge.selfdeploy.phala import ( + PhalaApiError, + PhalaCloudClient, + resolve_cvm_id_from_list, +) + + +class _FakeResponse: + def __init__(self, body: bytes = b"", *, status: int = 204) -> None: + self._body = body + self.status = status + + def read(self) -> bytes: + return self._body + + +def test_delete_cvm_issues_delete_to_cvms_id_path() -> None: + seen: list[Request] = [] + + def opener(request: Request, timeout: float = 0) -> _FakeResponse: # noqa: ARG001 + seen.append(request) + return _FakeResponse(b"", status=204) + + client = PhalaCloudClient(api_key="k" * 32, opener=opener) + client.delete_cvm("cvm-abc-1") + assert len(seen) == 1 + assert seen[0].get_method() == "DELETE" + assert seen[0].full_url.endswith("/cvms/cvm-abc-1") + + +def test_delete_cvm_treats_204_as_success() -> None: + client = PhalaCloudClient( + api_key="k" * 32, + opener=lambda request, timeout=0: _FakeResponse(b"", status=204), # noqa: ARG005 + ) + client.delete_cvm("42") # must not raise + + +def test_delete_cvm_treats_404_as_idempotent_success() -> None: + def opener(request: Request, timeout: float = 0) -> _FakeResponse: # noqa: ARG001 + raise HTTPError( + url=request.full_url, + code=404, + msg="Not Found", + hdrs=None, # type: ignore[arg-type] + fp=io.BytesIO(b""), + ) + + client = PhalaCloudClient(api_key="k" * 32, opener=opener) + client.delete_cvm("already-gone") # must not raise + + +@pytest.mark.parametrize("code", [400, 401, 403, 500]) +def test_delete_cvm_raises_on_non_success_status(code: int) -> None: + def opener(request: Request, timeout: float = 0) -> _FakeResponse: # noqa: ARG001 + raise HTTPError( + url=request.full_url, + code=code, + msg="err", + hdrs=None, # type: ignore[arg-type] + fp=io.BytesIO(b"{}"), + ) + + client = PhalaCloudClient(api_key="k" * 32, opener=opener) + with pytest.raises(PhalaApiError, match=f"HTTP {code}"): + client.delete_cvm("cvm-1") + + +def test_delete_cvm_refuses_non_allowlisted_path_shape() -> None: + client = PhalaCloudClient( + api_key="k" * 32, + opener=lambda *_a, **_k: _FakeResponse(), # pragma: no cover + ) + with pytest.raises(PhalaApiError, match="unsupported|invalid"): + client.delete_cvm("../escape") + with pytest.raises(PhalaApiError, match="unsupported|invalid"): + client.delete_cvm("id/with/slash") + + +def test_delete_is_not_available_via_post_allowlist() -> None: + client = PhalaCloudClient( + api_key="k" * 32, + opener=lambda *_a, **_k: _FakeResponse(b"{}"), # pragma: no cover + ) + with pytest.raises(PhalaApiError, match="unsupported"): + client.post("/cvms/cvm-1", {}) + + +def test_resolve_cvm_id_require_unique_refuses_ambiguous_match() -> None: + listing = { + "items": [ + {"id": 1, "app_id": "same-app"}, + {"id": 2, "app_id": "same-app"}, + ] + } + with pytest.raises(PhalaApiError, match="multiple|ambiguous"): + resolve_cvm_id_from_list(listing, app_id="same-app", require_unique=True) + + +def test_resolve_cvm_id_require_unique_returns_single_match() -> None: + listing = { + "items": [ + {"id": 9, "app_id": "other"}, + {"id": 77, "app_id": "target-app"}, + ] + } + assert resolve_cvm_id_from_list(listing, app_id="target-app", require_unique=True) == "77" + + +def test_default_phala_teardown_uses_http_delete_not_subprocess( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setattr( + cli.subprocess, + "run", + MagicMock(side_effect=AssertionError("no subprocess")), + ) + deleted: list[str] = [] + + class _Client: + def delete_cvm(self, cvm_id: str) -> None: + deleted.append(cvm_id) + + result = cli.default_phala_teardown("cvm-live-1", client=_Client()) # type: ignore[arg-type] + assert result["ok"] is True + assert result["returncode"] == 0 + assert deleted == ["cvm-live-1"] + assert "phala" not in json.dumps(result).lower() or result.get("error") is None + + +def test_default_phala_teardown_maps_api_error() -> None: + class _Client: + def delete_cvm(self, cvm_id: str) -> None: + raise PhalaApiError("Phala delete returned HTTP 500") + + result = cli.default_phala_teardown("cvm-x", client=_Client()) # type: ignore[arg-type] + assert result["ok"] is False + assert result["returncode"] != 0 + assert "500" in str(result.get("error") or "") + + +def test_cli_teardown_resolves_cvm_id_from_app_id(monkeypatch: pytest.MonkeyPatch) -> None: + listing = {"items": [{"id": "resolved-9", "app_id": "app-hex-1"}]} + deleted: list[str] = [] + + class _Client: + def __init__(self, **_kwargs: Any) -> None: + pass + + def get(self, path: str) -> dict[str, Any]: + assert path == "/cvms" + return listing + + def delete_cvm(self, cvm_id: str) -> None: + deleted.append(cvm_id) + + monkeypatch.setattr(cli, "PhalaCloudClient", _Client) + monkeypatch.setenv("PHALA_CLOUD_API_KEY", "k" * 32) + capture: list[Any] = [] + monkeypatch.setattr(cli, "_print", lambda payload: capture.append(payload)) + + parser = cli.build_parser() + args = parser.parse_args(["review", "teardown", "--app-id", "app-hex-1"]) + code = cli._ordered_review_command(args) + assert code == 0 + assert deleted == ["resolved-9"] + assert capture and capture[0].get("ok") is True + assert capture[0].get("torn_down") == "resolved-9" + + +def test_cli_teardown_refuses_ambiguous_app_id(monkeypatch: pytest.MonkeyPatch) -> None: + listing = { + "items": [ + {"id": "a", "app_id": "dup"}, + {"id": "b", "app_id": "dup"}, + ] + } + + class _Client: + def __init__(self, **_kwargs: Any) -> None: + pass + + def get(self, path: str) -> dict[str, Any]: + return listing + + def delete_cvm(self, cvm_id: str) -> None: # pragma: no cover + raise AssertionError(f"must not delete on ambiguity: {cvm_id}") + + monkeypatch.setattr(cli, "PhalaCloudClient", _Client) + monkeypatch.setenv("PHALA_CLOUD_API_KEY", "k" * 32) + args = SimpleNamespace(review_command="teardown", cvm_id=None, app_id="dup", phala_api=None) + capture: list = [] + monkeypatch.setattr(cli, "_print", lambda payload: capture.append(payload)) + code = cli._ordered_review_command(args) + assert code != 0 + assert capture and "multiple" in str(capture[0].get("diagnostics", {}).get("error", "")).lower() + + +def test_cli_teardown_requires_cvm_id_or_app_id() -> None: + parser = cli.build_parser() + # --cvm-id no longer required at parse time; runtime refuses empty identity. + args = parser.parse_args(["teardown"]) + assert getattr(args, "cvm_id", None) in (None, "") + # Runtime path + import sys + from io import StringIO + + buf = StringIO() + old = sys.stderr + try: + sys.stderr = buf + code = cli.main(["teardown"], teardowner=lambda *_a, **_k: {"ok": True, "returncode": 0}) + finally: + sys.stderr = old + # Without cvm-id/app-id should fail closed (non-zero) rather than call teardowner with None + assert code != 0 diff --git a/packages/challenges/agent-challenge/tests/test_sizing_compose_hash_freeze.py b/packages/challenges/agent-challenge/tests/test_sizing_compose_hash_freeze.py new file mode 100644 index 000000000..5a2eb587e --- /dev/null +++ b/packages/challenges/agent-challenge/tests/test_sizing_compose_hash_freeze.py @@ -0,0 +1,42 @@ +"""Freeze compose hashes so sizing changes cannot move measured documents. + +Sizing (instance_type / disk_size) must stay OUTSIDE the measured app-compose +documents. These constants were measured from the generators BEFORE any sizing +edit on branch feat/agent-challenge-cvm-sizing. +""" + +from __future__ import annotations + +from agent_challenge.canonical.compose import app_compose_hash, generate_app_compose +from agent_challenge.review.compose import generate_review_app_compose, review_app_compose_hash + +# Measured on clean HEAD 263eeb1b before sizing edits (same fixtures as +# tests/test_canonical_compose.py and default review moniker). +_CANONICAL_IMAGE = "ghcr.io/baseintelligence/agent-challenge-canonical@sha256:" + ("a" * 64) +_REVIEW_IMAGE = "ghcr.io/baseintelligence/agent-challenge-review@sha256:" + ("b" * 64) + +FREEZE_EVAL_APP_COMPOSE_HASH = ( + "f8f05273959469a2b8eb3e599863cdb2ddc7c741055d1e6830f101fc2e79d334" +) +FREEZE_REVIEW_APP_COMPOSE_HASH = ( + "9ef4435f4bd3e938f371c93c5ee8076fabf16b75a1ea18f3bdb9c0e24176325f" +) + + +def test_eval_app_compose_hash_frozen_against_sizing_work(): + compose = generate_app_compose(orchestrator_image=_CANONICAL_IMAGE) + assert app_compose_hash(compose) == FREEZE_EVAL_APP_COMPOSE_HASH + # Sizing keys must never appear inside the measured document. + blob = str(compose) + assert "disk_size" not in blob + assert "instance_type" not in blob + assert "tdx.xlarge" not in blob + + +def test_review_app_compose_hash_frozen_against_sizing_work(): + compose = generate_review_app_compose(review_image=_REVIEW_IMAGE) + assert review_app_compose_hash(compose) == FREEZE_REVIEW_APP_COMPOSE_HASH + blob = str(compose) + assert "disk_size" not in blob + assert "instance_type" not in blob + assert "tdx.xlarge" not in blob