Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,10 @@ BUILDER_TAG=latest
#PODMAN_MIN_MEMORY_MIB=6144
#PODMAN_MIN_CPUS=4

# On Apple Silicon the studio image is amd64-only (emulated). Leave as-is unless you have an arm64 image.
STUDIO_PLATFORM=linux/amd64
# Studio images publish both amd64 and arm64, so leave this unset and the right one is pulled natively.
# Set it only to force a specific arch — e.g. an amd64-only private mirror, which on Apple Silicon then
# requires Rosetta in the podman machine.
#STUDIO_PLATFORM=

# ── Host ports ──
# Offset from the platform's standard ports (60021 / 4200 / 8000 / 27017 / 6060) so this dev-kit can run
Expand Down
7 changes: 4 additions & 3 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,10 @@ services:

duplo-ai-studio:
image: quay.io/duplocloud/backend:${STUDIO_TAG}
# Feature-branch studio images are built amd64-only (arm64 is built on main). On Apple Silicon
# this runs under emulation. Override per-arch by setting STUDIO_PLATFORM in .env.
platform: ${STUDIO_PLATFORM:-linux/amd64}
# No platform pin: every studio release publishes amd64 and arm64, so this resolves natively from
# the manifest — the same as every other service here. Set STUDIO_PLATFORM in .env only to force a
# specific arch (an amd64-only private mirror, say), which on Apple Silicon then needs Rosetta.
platform: ${STUDIO_PLATFORM:-}
restart: unless-stopped
ports:
- "${STUDIO_PORT:-60021}:60021"
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ Pin the published images the stack runs.
| `AGENT_TAG` | `main-d27e511` | The `claude-code-agent` image that executes provisioning tickets. A `main` build — it carries no extension-specific changes. |
| `UI_TAG` | `2a949c536be71efd4501dd60ca6fdda4f7b4676a` | The Angular portal image. A `duplo-ui` git SHA, pinned to a build that renders the license state. |
| `XTERM_TAG` | `main-d323e0b` | The in-browser terminal image. |
| `STUDIO_PLATFORM` | `linux/amd64` | Docker platform for the studio image. On Apple Silicon the studio image is amd64-only and runs emulated. Leave as-is unless you have an arm64 image. |
| `STUDIO_PLATFORM` | *(unset)* | Platform override for the studio image. Leave unset: releases publish amd64 and arm64, so the native image resolves from the manifest. Set it only to force one arch against a single-arch registry — on Apple Silicon that then needs Rosetta. |

`STUDIO_TAG` and `UI_TAG` are required — `run.sh` exits if either is empty:

Expand Down
4 changes: 2 additions & 2 deletions docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ Yes. Set `DUPLO_TARGET=remote` with `DUPLO_HOST` and `DUPLO_TOKEN` (an Administr

### Can I run it on Windows, WSL, or an ARM Mac?

- **ARM Macs (Apple Silicon)** — yes. The studio image is amd64-only and runs under emulation;
`STUDIO_PLATFORM=linux/amd64` in `.env.example` handles this. Expect it to be slower than native.
- **ARM Macs (Apple Silicon)** — yes, natively. Studio images publish arm64 alongside amd64 and the
platform is unpinned, so the native image is pulled and nothing is emulated.
- **Windows and WSL** — not verified. Docker Compose v2 and `python3` are the only hard requirements and
the scripts are POSIX shell, so WSL2 is the likely path, but nobody has confirmed it end to end. If you
try it, please say how it went.
Expand Down
41 changes: 9 additions & 32 deletions docs/getting-started/prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,44 +55,21 @@ What matters is that the runtime is reachable and that its `compose` subcommand
> podman needs nothing else; the build scripts handle the user-namespace uid mapping for you, so
> extension bundles come out owned by you rather than by root.
>
> **podman on Apple Silicon** needs three more things before the studio will start, because the studio
> image is amd64 on some tags and runs emulated:
>
> 1. **Create the machine with `--provider applehv`.** Rosetta is an applehv feature. podman 6 defaults
> to **libkrun** on Apple Silicon, and libkrun has no Rosetta support at all — upstream's
> `LibKrunStubber.GetRosetta` returns false unconditionally, so a `rosetta = true` under libkrun is
> read and silently discarded, and `podman machine inspect` keeps reporting `Rosetta: false` however
> many times you edit the config. **Unlike the rosetta key, the provider is fixed at
> `podman machine init`** — it is the one setting an existing machine cannot be talked out of, so
> switching means destroying and recreating the VM.
> 2. **Enable Rosetta.** podman does **not** enable it by default, and without it amd64 binaries fall
> through to QEMU, which cannot run the studio's .NET runtime. Create
> `~/.config/containers/containers.conf` with `[machine] provider = "applehv"` and
> `[machine] rosetta = true` before starting the machine (on applehv the rosetta key is re-read on
> every `podman machine start`, so *that* key only needs a stop/start, not re-creating). **If that
> file already exists, edit its `[machine]` section — never append a second `[machine]` table, which
> is a TOML duplicate-key error that stops podman running at all.**
> 3. **Size the machine.** `podman machine init` defaults to 2048 MiB; `run.sh` hard-fails below
> 6144 MiB, because an undersized VM does not stop the stack coming up — it poisons later extension
> builds with an OOM kill that reports itself as `exit status 137`.
>
> Which makes the whole first-time sequence:
> **podman on Apple Silicon** needs one more thing: **size the machine.** `podman machine init` defaults
> to 2048 MiB; `run.sh` hard-fails below 6144 MiB, because an undersized VM does not stop the stack coming
> up — it poisons later extension builds with an OOM kill that reports itself as `exit status 137`.
>
> ```bash
> mkdir -p ~/.config/containers
> printf '[machine]\nprovider = "applehv"\nrosetta = true\n' > ~/.config/containers/containers.conf
> podman machine init --provider applehv --cpus 4 -m 8192 --disk-size 100
> podman machine init --cpus 4 -m 8192 --disk-size 100
> podman machine start
> ```
>
> Verify all three before your first run:
>
> ```bash
> podman machine list --format '{{.Name}} {{.VMType}}' # want applehv, NOT libkrun
> podman machine ssh 'ls /proc/sys/fs/binfmt_misc/' # want a `rosetta` entry, no `qemu-x86_64`
> podman machine list # want MEMORY >= 6144 MiB
> ```
>
> Rosetta is **not** required. Studio images publish arm64 alongside amd64, so the native image is pulled
> and nothing is emulated. You only need Rosetta if you pin `STUDIO_PLATFORM` to amd64 against a registry
> that carries no arm64 build — see
> [troubleshooting](../troubleshooting.md#runsh-hangs-on-waiting-for-studio-apple-silicon).
>
> If the studio hangs on startup anyway, see
> [troubleshooting](../troubleshooting.md#runsh-hangs-on-waiting-for-studio-apple-silicon).
>
Expand Down
187 changes: 26 additions & 161 deletions docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,10 +102,9 @@ maps it, and podman additionally provides `host.containers.internal` for the sam

### `./run.sh` hangs on "Waiting for studio" (Apple Silicon)

> `run.sh` now catches this before it happens — `runtime_rosetta_check` in `scripts/_runtime.sh` stops
> the run with the fix below rather than letting it hang. You should only reach this entry if the check
> could not reach the machine to judge it (it stays silent rather than guessing), or if you are on a
> version of the kit that predates it. The diagnosis and fix are the same either way.
> `run.sh` catches this before it happens — `runtime_rosetta_check` in `scripts/_runtime.sh` stops the
> run with the fix below. You should only reach this entry if the check could not reach the machine to
> judge it (it stays silent rather than guessing), or on a version of the kit that predates it.

**Symptom** — minutes of dots, then a failure that blames the wrong thing:

Expand All @@ -122,181 +121,47 @@ at Program.<Main>$(String[] args) in /src/Duplo.ai.studio/Program.cs:line 255
qemu: uncaught target signal 6 (Aborted) - core dumped
```

**Why** — feature-branch studio images are built amd64-only (`platform: ${STUDIO_PLATFORM:-linux/amd64}`
in `docker-compose.yml`), so on Apple Silicon the studio runs emulated. There are two emulators and only
one of them works:
**Why** — `STUDIO_PLATFORM` is pinning the studio image to amd64 on an arm64 host, so it runs under QEMU
user-mode emulation, which cannot run .NET's JIT/reflection. The process takes `SIGABRT` while
`MapControllers()` walks controller attributes at startup. The crash does not take PID 1 down promptly,
so the container stays `Up`, `/healthz` never answers, and the wait loop burns ~4.5 minutes before
failing on the login step — which has nothing to do with it.

- **Rosetta** — used when the podman VM has the Rosetta share attached. Runs .NET correctly.
- **QEMU user-mode** — the fallback. Cannot run .NET's JIT/reflection; the process takes `SIGABRT` while
`MapControllers()` walks controller attributes at startup.

**First check the provider — under libkrun none of the rest of this applies.** podman 6 defaults to the
`libkrun` provider on Apple Silicon, and libkrun has **no Rosetta support at all**: upstream's
`LibKrunStubber.GetRosetta` returns false unconditionally, so `rosetta = true` is read and discarded
without a warning anywhere. The symptom is identical to a missing key — no `rosetta` binfmt handler, QEMU,
a studio that aborts — but every fix aimed at the config or at restarting the machine is wasted, because
the config was never the problem:

```bash
podman machine list --format '{{.Name}} {{.VMType}}' # want applehv; libkrun can never have Rosetta
```

Unlike the rosetta key, **the provider is fixed at `podman machine init`**. Switching to applehv means
recreating the VM, which destroys its images and volumes:

```bash
podman machine stop && podman machine rm -f
podman machine init --provider applehv --cpus 4 -m 8192 --disk-size 100
podman machine start
```

Podman Desktop muddies this further: it has a known bug ([#16341](https://github.com/podman-desktop/podman-desktop/issues/16341))
where it reports Rosetta as enabled on libkrun machines that cannot have it.

**On applehv, Rosetta is opt-in, and podman does not enable it for you.** Unless
`~/.config/containers/containers.conf` sets `[machine] rosetta = true`, the VM is started with no Rosetta
share and every amd64 binary falls through to QEMU. The key is read on every `podman machine start`, not
only at `init`. Measured on an **applehv** machine, podman 6.1.2, reading `AppleHypervisor.Vfkit.Rosetta`
from the machine's JSON (these numbers do not describe libkrun, which is false in every row):

| `containers.conf` | resulting `Vfkit.Rosetta` |
| --- | --- |
| `[machine] rosetta = true` | `true` |
| *file absent* | `false` |
| `[machine] rosetta = false` | `false` |

An absent file behaves exactly like an explicit `false`. Do not assume the default is on.

Nothing tells you it is off, because it then fails **silently** twice over:

1. Inside the VM, `rosetta-activation.service` runs `mount -t virtiofs rosetta /var/mnt`, fails with
`wrong fs type, bad option, bad superblock on rosetta`, and **exits 0 anyway**. `systemctl status`
reports the unit as `status=0/SUCCESS`.
2. The studio container stays `Up` after the .NET process aborts — the crash does not take PID 1 down
promptly — so `podman ps` shows nothing wrong.

A second possible cause, **not observed here and not verified** — recorded because the strings are present
in the `vfkit` binary and it would produce identical symptoms even with the config set correctly: vfkit
degrades on its own with `Rosetta installation failed. Continuing without Rosetta.` and still exits 0,
which would happen if Rosetta were not registered on the host when the machine starts and vfkit probes
for it. The diagnosis below catches it either way; step 1 of the fix rules it out.

With the studio dead, `/healthz` never answers and the 90 × 3s poll in `run.sh` spins for ~4.5 minutes
before falling through to a login failure that has nothing to do with your password.

> **Do not trust the top-level `Rosetta` field** in `~/.config/containers/podman/machine/applehv/podman-machine-default.json`.
> It is not the field that decides, and it is routinely the *opposite* of the effective value. The one
> that matters is `AppleHypervisor.Vfkit.Rosetta`, which is what `podman machine inspect` reports.

**Diagnose** — the decisive check is the binfmt handler list inside the VM:
**Fix — remove the pin.** Studio releases publish arm64 alongside amd64, so an unset `STUDIO_PLATFORM`
pulls the native image and there is nothing to emulate:

```bash
podman machine ssh 'ls /proc/sys/fs/binfmt_misc/'
```

| You see | Meaning |
| --- | --- |
| `rosetta` present | Rosetta is live. This is not your problem — look elsewhere. |
| `qemu-x86_64` present, no `rosetta` | Broken. amd64 runs under QEMU and .NET will abort. |

Corroborate:

```bash
podman machine list --format '{{.VMType}}' # libkrun means Rosetta is impossible, not absent
podman machine inspect --format '{{.Rosetta}}' # false when broken
podman machine ssh 'mount | grep -i rosetta' # no output when broken
podman machine ssh 'systemctl status rosetta-activation.service --no-pager -l'
# "wrong fs type" despite status=0/SUCCESS
```

**Fix** — add the config, then restart the machine. **The machine does not need to be recreated.** podman
reads `[machine] rosetta` at every `podman machine start`, not only at `init`, so a stop/start picks it up
and nothing is destroyed.

1. Confirm Rosetta on the host first — the VM share cannot be created if the host has no Rosetta:

```bash
arch -x86_64 /usr/bin/true && echo "Rosetta OK" || softwareupdate --install-rosetta --agree-to-license
```

If you just installed it, re-run the `arch -x86_64` check and see it succeed before going on.

2. Opt in to Rosetta. **This is mandatory, not a nicety** — see the table above.

**Only if `~/.config/containers/containers.conf` does not exist yet:**

```bash
mkdir -p ~/.config/containers
printf '[machine]\nrosetta = true\n' > ~/.config/containers/containers.conf
grep -n '^STUDIO_PLATFORM=' .env # if this prints anything, that is the problem
sed -i '' '/^STUDIO_PLATFORM=/d' .env
./run.sh
```

> ⚠️ **If the file already exists, edit it — do not append.** Add `rosetta = true` under its
> existing `[machine]` section, or add a `[machine]` section if it has none. Appending a second
> `[machine]` table is a TOML duplicate-key error and podman then refuses to run **at all**:
>
> ```
> Failed to obtain podman configuration: parsing containers.conf:
> toml: line 11: Key 'machine' has already been defined.
> ```
>
> That is a worse state than the missing Rosetta you started with. `run.sh`'s preflight reads your
> actual config and prints the right instruction for whichever case you are in.

Keep the file — it is read on every start, so deleting it silently loses Rosetta at the next one.
`./run.sh` removes a stale `STUDIO_PLATFORM=linux/amd64` for you on its next run, but only when it can
see you never changed it yourself (`.env.defaults` has to agree). A value you set by hand is always left
alone — so if you pinned it deliberately, the line above is the deliberate way to unpin it.

3. Restart the machine. No `rm`, no `init`, no re-pull — images and volumes are untouched:
**If the tag really is amd64-only** — a private or mirrored registry that carries one arch — then
unsetting changes nothing and you do need Rosetta. Check what your tag actually publishes:

```bash
podman machine stop
podman machine start
podman manifest inspect "quay.io/duplocloud/backend:$(grep '^STUDIO_TAG=' .env | cut -d= -f2)" \
| grep architecture
```

> Verified by toggling one machine twice: with the file absent a stop/start yields `Vfkit.Rosetta: false`
> and a `qemu-x86_64` binfmt handler; with the file present the same machine yields `true` and a `rosetta`
> handler. The machine's `Created` timestamp is unchanged throughout, so no re-creation is involved.
>
> Only recreate the machine if you need to change its **sizing** — and then pass your sizing back
> explicitly, because the `init` default is 2048 MiB and `runtime_machine_check` rejects anything under
> 6144: `podman machine init --cpus 4 -m 8192 --disk-size 100`.
With no `arm64` in that list, either use Docker Desktop (it ships Rosetta itself, with nothing to
configure) or give the podman machine an `applehv` provider with Rosetta enabled — `applehv` is the only
provider that supports it, and the provider is fixed when the machine is created, so switching means
recreating the VM.

**Confirm** — run this gate *before* `./run.sh`, so a failure costs seconds rather than a confusing hang:

```bash
podman machine inspect --format '{{.Rosetta}}' # true
podman machine ssh 'ls /proc/sys/fs/binfmt_misc/' | grep rosetta # rosetta
podman machine ssh 'mount | grep -i rosetta' # rosetta on /var/mnt type virtiofs
podman run --rm --platform linux/amd64 docker.io/library/alpine:3 uname -m # x86_64
```

The first three must pass, and `qemu-x86_64` should now be **absent** from the binfmt list — Rosetta
replaces it. The fourth is a liveness check only: a trivial amd64 binary runs under QEMU too, so `x86_64`
there confirms amd64 executes but does **not** prove Rosetta is the thing executing it. The binfmt list is
the discriminating check.
Then start the stack and verify the studio is alive rather than merely `Up`:
**Confirm the fix:**

```bash
./run.sh
curl -fsS http://localhost:60031/healthz && echo " studio is answering"
podman logs devkit-duplo-ai-studio-1 2>&1 | grep -c 'uncaught target signal' # must be 0
```

**Avoiding it entirely** — if an arm64 studio image exists for your tag, use it and skip emulation.
Observed so far: `branch-*` tags are published amd64-only, while `dev-*` tags are multi-arch. Never assume
either way — check:

```bash
podman manifest inspect "quay.io/duplocloud/backend:$(grep '^STUDIO_TAG=' .env | cut -d= -f2)" \
| grep architecture
```

If an `arm64` variant is listed, set `STUDIO_PLATFORM=linux/arm64` in `.env` and re-run `./run.sh`.

**Or use Docker instead** — Docker Desktop on Apple Silicon provides Rosetta itself, with no
`containers.conf` and no machine configuration. The same amd64 studio image that aborts under podman's
QEMU starts normally under Docker: `uname -m` reports `x86_64` and the log carries zero
`uncaught target signal` entries. This whole entry is podman-specific.

> **If you have both installed, check which one you are actually using.** `runtime_detect` in
> **If you have both runtimes installed, check which one you are actually using.** `runtime_detect` in
> `scripts/_runtime.sh` walks `docker podman` and takes the first on `PATH` — presence only, with no
> daemon check — so installing Docker silently takes precedence over podman. A podman machine can be up
> and correctly configured while `./run.sh` uses Docker. Pin it deliberately:
Expand Down
6 changes: 4 additions & 2 deletions docs/upgrading.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,13 @@ Flags:

## How defaults are adopted

Framework-shipped defaults — the image tags and `STUDIO_PLATFORM` — live in
Framework-shipped defaults — the image tags — live in
`.env.example`. An upgrade changes that file, but it never touches your `.env`. `run.sh` bridges the two on
every run.

The tracked keys are `STUDIO_TAG`, `AGENT_TAG`, `UI_TAG`, `XTERM_TAG`, and `STUDIO_PLATFORM`. Nothing
The tracked keys are `STUDIO_TAG`, `AGENT_TAG`, `UI_TAG`, and `XTERM_TAG`. (`STUDIO_PLATFORM` is still
tracked in code, but `.env.example` no longer ships a value for it, and a key the example leaves blank is
skipped — so it is adopted only if a future release pins it again.) Nothing
licensing-related is adopted or overwritten this way — `Licensing__Token` is yours, not a framework
default, and the license server URL is a built-in rather than a shipped `.env` value.

Expand Down
Loading
Loading