Skip to content

feat(flows): add the local and ssh environment drivers - #79

Merged
futrime merged 2 commits into
feat/new-flow-apifrom
flow-api/u4-envs
Sep 25, 2026
Merged

futrime merged 2 commits into
feat/new-flow-apifrom
flow-api/u4-envs

Conversation

@futrime

@futrime futrime commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Wave 1, unit 4: the environment drivers behind hmz.runtime.flowing.spi.EnvDriver. This PR is based on feat/new-flow-api.

Public API (hmz.runtime.flowing.environments)

  • open_env(spec: EnvSpec) -> EnvDriver: local@/abs or ssh@[user@]host[:port]/abs or ssh@host/~/rel. It never touches the network. It raises EnvUnavailable when a local workdir is missing or the ssh destination is invalid (for example, one starting with -).
  • local_env(workdir: Path) -> EnvDriver: fills a LocalEnv role.
  • probe(driver) -> None (new): connects and learns the machine's resources off the loop, then checks that the workdir exists. It raises EnvUnavailable or EnvConnectionError. The ways in should await it on every root env before checking resource requirements. Until it runs, an ssh env reports available=False and cpu 1, memory 0, gpus 0. It is not in the lazy-export table, because runtime/flowing/__init__.py belongs to unit 2.
  • Both drivers are MachineEnvDriver, and each serves every item of ENV_CAPABILITIES.

Semantics the engine relies on

  • exec
    • Every command runs in its own process group with stdin empty. stdout and stderr come back separately.
    • A timeout, a cancellation, or close() kills the whole group and ends the wait. This holds even when a setsid daemon keeps the output open.
    • A non-zero exit is returned rather than raised. A process killed by signal N reports 128 + N.
  • Holders of temp clones
    • Holders are compared with ==, and they are tracked process-wide.
    • On this machine, an flock also makes other processes' holds TempCloneBusy. Over ssh, exclusivity is per process only.
    • Closing the root drops its holds, so a run resumed in the same process can take them again.
  • destroy_* must be called before closing the root. Closing the root is final: every driver on that machine then raises EnvError("... is closed").
  • A temp clone is a reflinked copy where the filesystem supports it, .git included, and its repository stands alone:
    • Worktrees nested inside it are repointed to the copy.
    • A linked-worktree workdir is re-cloned with --shared and keeps its index.
    • Gitfiles that lead outside the copy are cut loose.

On disk

Everything lives under humanize's home on that machine: hmz.home() locally, ${HUMANIZE_HOME:-~/.humanize} over ssh.

  • envs/<name>-<digest(workdir)>/clones/<id>-<digest>/, plus a .lock file beside each clone.
  • envs/<name>-<digest(workdir)>/scratch/<id>-<digest>/.
  • envs/<name>-<digest(workdir)>/worktrees/<ref>-<random>/.

Names are deterministic, so a resumed run reattaches. A copy is built next to its final path and renamed into place; removal renames it away before deleting. The whole envs/ directory is safe to delete whenever no run is using it.

SSH

  • It uses coganchor's transport: transport.connect(ssh://…, ["/"]) plus RemoteClient, with the zipapp bootstrapped over ssh and ControlMaster giving one master connection per host.
  • One probe command reports home, humanize's home, CPUs, memory, and GPUs.
  • placement() returns Placement(SSH, provider, workdir, AnchoredConfig(AnchorConfig("ssh://provider", workspace=<abs workdir>))). For a ~ workdir it returns the absolute path once probed, and remote_path="~/…" before that.
  • There is one coganchor fix, in its own commit: ExecSession.signal now reaches the group after the leader has exited, so remote timeouts actually kill background children.

Tests

  • Unit (tests/unit/flows/test_env_drivers.py): the driver contract over an in-memory machine, plus holds, closing, paths, error mapping, GPU and probe parsing, and building an ssh env without touching the network.
  • Integration (tests/integration/flows/test_{local,ssh}_envs.py): check_env_driver(..., repo=True) against real processes and the version-control tool. The ssh path runs through a stand-in ssh with coganchor's real serving half.
  • System (tests/system/flows/test_ssh_envs.py): the contract over real ssh. It uses localhost if passwordless ssh works; otherwise it starts a throwaway loopback sshd with its own keys and leaves ~/.ssh untouched. Here it ran against the throwaway sshd: 2 passed.
  • uv run pre-commit run --all-files passes, and uv run pytest passes across all three tiers.

🤖 Generated with Claude Code

futrime and others added 2 commits September 25, 2026 10:51
`ExecSession.signal` returned early once the process it started had
exited, so a signal sent to end a session never reached what that
process left in its group -- a `sleep &` holding stdout open -- and the
session, and the command, went on running on the target.

Signal the group while the process is alive, and after it has exited
for as long as the session's output is still being pumped. Once the
process is reaped its number is free, so the group is only signalled
while no process holds that number again: a group is only ever made
with its leader's number.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Implement `open_env`, `local_env` and a new `probe` in
`hmz.runtime.flowing.environments`: one `MachineEnvDriver` written against
a `Machine`, with a machine for this host and one for a host reached over
ssh. Both serve every environment capability of the SPI.

- exec: argv or bash script, in a process group of its own with nothing
  on stdin; stdout and stderr apart; a timeout, a cancellation or closing
  the driver kills the whole group and ends the wait, even where a daemon
  that left the group holds the output open; a non-zero status is
  returned, a death by signal N is 128 + N.
- read/write: relative to the workdir, absolute or `~/...`; writes are
  atomic, make their parent directories and keep an existing file's mode.
- derive_subdir; derive_worktree (`git worktree add --detach`, fresh
  directories under humanize's home); temporary copies, reflinked where
  the filesystem can, made beside their place and renamed in, with the
  copied repository made to stand alone (worktrees kept inside it
  repointed, gitfiles leading out cut loose); holders kept process-wide
  and, on this machine, an flock other processes honour; scratch
  directories; removal renames out of place first; everything named
  deterministically so a resumed run finds what it left.
- Closing the root is final: it stops every command of its machine, the
  machine's own copying included and whatever was still starting, lets
  go of its holds and its connection.
- ssh rides coganchor's transport: the serving half bootstrapped over
  `ssh` with one master connection per host, `/` exported, one probe
  command learning home, humanize's home, CPUs, memory and GPUs. Nothing
  touches the network until something is asked, or `probe`. Only the
  link failing is taken for a broken connection, never an errno the host
  reports or one slow answer.

Tests: the driver contract over an in-memory machine and the pure logic
(unit), the contract and the rest against real processes and git and
against a stand-in `ssh` (integration), and the contract over a real
`ssh` to `localhost` or to a throwaway loopback `sshd` (system).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@futrime
futrime merged commit 0667c45 into feat/new-flow-api Sep 25, 2026
2 checks passed
@futrime
futrime deleted the flow-api/u4-envs branch September 25, 2026 11:10
futrime pushed a commit that referenced this pull request Sep 25, 2026
feat(flows): add the local and ssh environment drivers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant