[Feature] Presets#4058
Merged
Merged
Conversation
Sequential experimental trials for endpoint preset creation: the agent searches serving configurations across hardware, frameworks, and variants in `dstack` tasks, records reproducible trial records (trials.jsonl), promotes the best to a verified service, and saves the preset. Verified across nine live e2e sessions (Qwen2.5-0.5B, Qwen3-32B on H100, Qwen3-32B under $1/hr — best result 533 tok/s on a single RTX 5090 via NVFP4 + EAGLE3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ycle list - Preset ID minted at session start and stable through run names, the session directory, and the final preset; sessions survive Ctrl+C, hard kills, and host restarts and continue with --resume <preset id> - Agent deaths without a submitted report retry in-process; the retry budget refills only when the failed attempt made progress - Durable workspace under the session directory with a /tmp symlink alias (Unix socket path limit); byte-offset persistence prevents duplicated output on resume; stale running manifests downgrade to interrupted - Flat ~/.dstack/presets/<id>/ store holding preset.yaml next to session internals; lazy migration from the models--* layout; delete archives to .archive/ instead of destroying - dstack endpoint preset list shows in-flight and interrupted sessions with colored statuses, trial progress after the status, best-trial benchmark and GPU, -w watch mode, and -v details - Top-level base:/repo: configuration shorthand; nested model: deprecated unless model.name is set; --base/--repo filters on list and delete Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- A watchdog process holds a pipe from the CLI and kills the agent's process tree the moment the pipe snaps — covering deaths the CLI cannot react to (SIGKILL included), which previously left an orphaned agent able to keep submitting GPU runs. Graceful stops disarm the watchdog. - POSIX kills the agent's process group; Windows kills the process tree - Presets and sessions sort newest-first within each base-model group, matching dstack ps Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sion A final report echoing a known secret (e.g. the bearer token in benchmark.command) was rejected wholesale, discarding an otherwise verified session at its very last step. Scrub known secret values from the report structure before validation; the bearer check still rejects unknown leaked tokens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ish list
- Configuration `type: preset`; flat `dstack preset {list,get,create,apply,delete}`
command tree; user-facing strings follow (internal names are a follow-up)
- `apply` selects among repeatable `--id` candidates (given order,
capacity-aware) replacing the machine-local `preset:` configuration field
- The bearer-token report guard only rejects credential-shaped values;
prose such as "auth via bearer header from env" failed two live sessions
- Preset rows show `success (n/m)` from the completed creation session;
trial count is the record count again (one long-lived task commonly hosts
several trials, so distinct task names undercount)
- List polish: dimmed base group rows and trial progress, green `success`
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- BENCHMARK renders identically with and without -v; the verbose-only detail dump is gone (full data remains in `preset get --json`) - -v adds exactly two things: a `ctx=` benchmark prefix (replacing the CONTEXT column) and a dimmed indented `repo=` on the preset's own row - `apply` shows the selected preset with the same `ctx= con= tok/s TTFT` string Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- `prompt:` in the configuration, inline or as a file `path:` relative to the configuration file; validated and capped - The agent contract gains the text via template directives in system_prompt.md (`<!--?prompt:...-->`); without a prompt the rendered contract stays byte-identical. The default no-patching limits gain an escape clause the prompt can invoke explicitly - The resolved prompt is pinned per session and kept across resumes, with a warning when the configuration changes meanwhile - The list shows `verifying` once a running session's trial budget is spent and the final service is being deployed and verified Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ation - The configuration `name` names the preset, claimed at session start: creating with a used name detaches it from the current holder (preset or session of any status) after confirmation, docker-tag style; the old preset survives nameless. `get`, `delete`, and `apply --id` accept names wherever they accept ids; the list gains a NAME column - Name-less creation is allowed; run names derive from a model slug - `create` shows an apply-style plan before starting the agent — Project, User, the effective fleets, and their offers (agent-free, best-effort) — and always asks for confirmation; `-y` skips - `print_offers` extracted from `print_run_plan` for reuse Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The agent's streams go to workspace files instead of a pipe into the CLI, so the agent survives CLI death and a later attach continues parsing from the persisted offsets. The session manifest tracks the agent's own pid, and list liveness keys on it, so a CLI-less agent shows as running (clauding/verifying) rather than interrupted. - Ctrl+C during create/attach asks to stop or detach; detaching (or any hard CLI death) leaves the agent working and visible - `dstack preset attach <id|name>` follows a detached session to completion and finalizes it; `stop <id|name>` terminates the agent and offers to stop its active runs - Stopping a session offers to stop its non-terminal runs; the cost warning only shows when the user keeps them for a later resume - Preset name conflicts prompt to reassign the name, leaving the old preset name-less - The watchdog is removed: CLI death now means detach, which is safe because detached sessions stay visible and attachable Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The stream readers and the record mirrors each open a separate _OffsetStore on the same file with disjoint keys; writing the whole in-memory view dropped the other store's offsets, causing duplicated mirror records on the next attach or resume. Update only the written key, re-reading the file first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Statuses use the same palette as `dstack ps` (verified=grey, clauding= sea_green3, verifying/interrupted/failed matching), and NAME becomes a trailing dimmed column so the data columns stay contiguous. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`model` is what makes the run a served, readiness-checked model endpoint — i.e. a preset. Tell the agent a service that never passes the readiness probe is a real failure, not something to work around by removing `model`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # src/dstack/_internal/cli/commands/endpoint.py # src/dstack/_internal/cli/utils/run.py
… read Replace `dstack preset attach` with `dstack preset logs [-f]`: dump a preset creation's log at any status; `-f` re-owns and finalizes a detached or running creation, and extra followers tail read-only instead of being rejected. Reconcile-on-read: `list`, `get`, and `apply` finalize a completed-but-orphaned creation from its on-disk report, so the preset is saved regardless of how the CLI went away (detached, killed, or crashed). Use advisory file locks (flock/msvcrt) for creation ownership instead of pid claims — no steal race, no stale locks, kernel auto-release on exit. Ctrl+C during create reports Detached/interrupted via the interrupt handler; stop auto-stops runs with a spinner and never re-saves a finished creation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the preset feature's internal identifiers, modules, and directories from endpoint/Endpoint to preset/Preset. CLI-only — no server, core, or API code is involved (that design was already removed). Notable moves: cli/services/endpoints -> cli/services/presets, cli/commands/endpoint.py -> preset.py, and the models split into configurations.py (PresetConfiguration) and presets.py (Preset). Also drops the redundant `endpoint` session-manifest key, which held the same value as `name`. Require max_trials for preset creation instead of silently defaulting to 3: create now errors unless it is set in the configuration or passed via --max-trials. The model field stays optional so apply and partial configs are unaffected. Trim the saved-preset line to `Preset <id> saved`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
peterschmidt85
marked this pull request as ready for review
July 22, 2026 18:33
Rename the endpoint docs to preset: concepts/presets.md, the dstack preset CLI reference, and the preset .dstack.yml reference, plus the mkdocs nav. Rewrite the presets concept page to the standard concept-page structure (Create a preset, Configuration options, Apply a preset, Manage presets), reflecting the current feature: type: preset, the flat dstack preset CLI, logs/detach/resume, list statuses, apply --id, delete --base/--repo, prompt, and required max_trials. Repoint the config-reference #SCHEMA# directives to the renamed models (cli.models.configurations) and add logs/stop to the CLI reference. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_RecordMirror binary-reads the source to track byte offsets, so a source written in text mode on Windows already carries \r\n. Appending it in text mode re-translated \n to \r\n, producing \r\r\n, which reads back as a blank extra line and failed the mirror/offset tests on windows-latest. Write the chunk with newline="" so it is appended verbatim. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_preset_agent is the only dstack subprocess that passes file objects as stdout/stderr. Inheriting those handles with close_fds=False leaked the CLI's other descriptors to the untrusted claude agent, and on Windows the broad inheritance flaked CreateProcess (WinError 87) under parallel test spawns. Pass close_fds=True so the agent inherits only its redirected std handles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The connection-resume tests spawned the fake agent with env={}, which fails
CreateProcess with WinError 87 on Windows Python 3.10: an empty environment
block is rejected (cpython gh-105436, fixed in 3.11+ but never backported to
3.10). Production is unaffected because build_preset_agent_env always
populates PATH, DSTACK_*, HOME, and the temp dirs.
Spawn with a minimal realistic env instead, mirroring what production
guarantees.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix the wheel force-include paths left pointing at the removed endpoints package, so installed CLIs find the bundled skills, and drop stale endpoint terminology from the agent system prompt. Robustness: a corrupt preset file no longer breaks list/find and stays deletable through the CLI (its warning goes to a new stderr console so --json stdout stays parseable); session manifests and offsets are written atomically (tmp + fsync + replace, with a Windows retry for concurrent readers); a read-only logs -f exits with a re-follow hint when the owner and agent are both gone instead of polling forever; follow always takes the finalize claim; a failed cleanup no longer retries without keep_final_service. Dead code: the write-only session timestamp field, the no-op stderr report merge, the unused PresetTrial model, the _LineStream protocol, and dead defensive branches. Deduplicate: structure redaction (now also redacting dict keys in report scrubbing), EnvSentinel resolution, run-name parsing, session close + workspace removal, and the no-fleets error. Tests: hoist repeated scaffolding (fake claude, patch blocks, plain console, session dirs), parametrize near-duplicates, merge overlapping tests, move model and output tests next to what they test, and cover the corrupt-preset CLI paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An interrupted session keeps its workspace for resume, which left the agent's dstack config, with a live token, on disk indefinitely. Remove the config on suspend; resume already re-mints it via build_preset_agent_env. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TestPresetNameClaims ran the CLI without the ssh-info mock that TestPresetLocalCommands had, so a slow runner's `ssh -V` timeout failed an unrelated test (windows-latest 3.10). Hoist the autouse fixture to module scope so every CLI invocation in the file stays off the real ssh binary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
agent.py had grown to 1591 lines mixing six concerns. Split it, moving every symbol verbatim: - session.py: on-disk session state, loaders, ownership claims, liveness - workspace.py: the agent working directory, aliasing, wrappers, skills - tail.py: offset-persistent tailers over stream and record files - redaction.py: secret redaction shared by the tailers and verify - agent.py: the Claude subprocess lifecycle (600 lines) Imports follow one direction: session <- workspace <- (tail, redaction) <- agent. No behavior change; all symbol bodies are byte-identical to before. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_create_preset selected fresh/resume/attach behavior via two booleans, with mode-dependent locals threaded through one 190-line body. Extract a _CreationSetup dataclass built by _fresh_setup, _resume_setup, and _attach_setup — each holding exactly its former branch — and have the shared execution path read from it. No behavior change. Verified by the full suite and a live end-to-end drill: fresh create, interrupt mid-deploy (token scrubbed), resume (pinned model and Claude session, token re-minted), and finalize to a saved preset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move business logic out of commands/preset.py per the repo convention that command modules are argparse wiring over cli/services: - Name-claim lookup and reassignment move to the create service; the command only asks for confirmation. - The hand-rolled watch loop becomes the standard rich Live table, sharing one data path with the non-watch listing. - Session and preset id-or-name resolution move to the session service and PresetStore.find_by_id_or_name. - The name-validation error no longer duplicates the validator's regex; it reuses the validator from core services. - PresetCommand registers in its alphabetical slot in main.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each session's readers and mirrors now share a single _OffsetStore (created by open_session_offsets and threaded through the run/attach paths), so set() writes its own in-memory state instead of re-reading and merging the file on every offset advance. The exclusive session claim already guarantees a single writing process; a lock serializes the in-process writers. Tailer and reader file IO runs via asyncio.to_thread so a hung filesystem cannot freeze the CLI; the shared store's lock is what makes those threaded flushes safe. Also: cancel the output collector when _run_claude_process unwinds instead of orphaning it, and derive both SIGTERM-to-SIGKILL ladders from one grace constant with cross-references. Tests: the two-store clobber test becomes a threaded shared-store test, and the two multi-behavior mega-tests are split into focused ones. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Matching presets were tried in ID order, so apply could deploy a slower preset while a faster one for the same model sat unused. Order candidates by the benchmark output rate (the same tok/s the list shows) before the capacity-aware selection; availability still gates the final choice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploying is now fully deterministic: `preset apply` takes a required, single `--id` (ID or name) and deploys exactly that preset. The automatic selection among matching presets — including the just-added fastest-first ordering and the capacity-based fallback — is removed: choosing implicitly among presets made the deployed artifact depend on point-in-time offer availability. The referenced preset is still validated against the configuration (served model, base or exact repo, context length), with a specific error for each mismatch. apply.py drops from 173 to 98 lines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Like a finished run, a completed creation ends silently: the log's last line already reports the verified service, and the preset shows in dstack preset. Only --keep-service still prints, since a run is left billing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
logs -f no longer repeats the resume hint on an interrupted preset: the hint already prints at interruption time, and logs' job is the log. Error messages raised as CLIError are printed escaped, so the rich [code] markup embedded in three of them rendered literally; make them plain text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stop now works from any terminal, like dstack stop. The interruption is recorded before the agent is terminated, and the owner's retry loop reads a recorded stop as a decision rather than an outage, ends its own terminal quietly, and never resurrects the agent. On a session whose agent already finished, stop finalizes it the way a read would (silent reconcile) instead of leaving a not-yet-saved intermediate state; sessions already in a terminal state are reported plainly. The stopped-elsewhere owner path is a dedicated CreationStopped signal, so an external stop is never misrecorded as a failed creation. apply --id now takes a preset ID only; names stay on the inspection commands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The end-of-create cleanup stopped runs and then silently polled for up to ten minutes, so the CLI looked hung after the last log line. Show the same Stopping runs... spinner the stop command shows; the silent reconcile path is unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
dstack preset— agent-driven search for the best model serving configuration. A headless agent (Claude Code) runs sequential trials on real fleets, benchmarks each, deploys the best as a verifieddstackservice, and saves the result as a reusable preset. Purely client-side: works against any existing dstack server.Key changes
type: presetconfiguration:base:orrepo:, requiredmax_trials, plusconcurrency,context_length,fleets,env, andprompt:for custom agent instructionsdstack presetCLI:create(--resume),list(-w),logs [-f],stop,apply --id,get,deleteapplydeploys exactly the preset named by--idas a standarddstackserviceHow to test
Install from this branch:
Prerequisites: any dstack server with a GPU-capable backend (the feature is client-side, so an existing deployment works as is), at least one fleet, and an authenticated
claudeCLI.Interrupt a
createwith Ctrl+C and continue it withdstack preset create -f preset.dstack.yml --resume <ID>.🤖 Generated with Claude Code