Run local LLMs on the RTX 5880 Ada

.gitignore

@@ -3,3 +3,4 @@ __pycache__/
 /.direnv/
 /nix-flake-update.txt
 /result*
+/.claude/

dev/README.md

@@ -0,0 +1,120 @@
+# Development stack
+
+A second Home Assistant, running in the agent VM, so voice work does not need a
+rebuild of the real machine for every experiment.
+
+    dev/run-ha.sh                    # start it (builds .#hass-dev if needed)
+    python3 dev/ha-onboard.py        # once, on an empty config dir
+    python3 dev/ha-setup.py          # configure it; idempotent
+    python3 dev/ha-ask.py "is the bed light on?"
+    dev/run-ha.sh stop
+
+Voice and model experiments, all driven through `dev/halib.py`. See
+`dev/semantic-endpointing.md` for what they found:
+
+    dev/turn-test.py <clip>          # does it keep listening through a pause?
+    dev/silence-sweep.py             # latency against pause tolerance
+    dev/stage-breakdown.py           # where the milliseconds go
+    dev/e2e-compare.py               # old settings against new
+    dev/eval.py 3 <model>...         # scenario scores, state reset each run
+    dev/model-compare.py             # models, end to end
+    dev/toolcall-stress.py           # the tool-call path, repeatedly
+    dev/smart-turn-probe.py <clip>   # the turn model's opinion, cut by cut
+    dev/smart-turn-eval.py           # its accuracy on real labelled speech
+    dev/speculation-sweep.py         # what speculating is worth, against the wait
+    dev/speculation-safety.py        # a wrong guess must not act
+    dev/speculation-speech.py        # a wrong guess must not be spoken
+    dev/stt-compare.py <port>:<name> # transcribers, side by side
+    dev/local-intent-test.py         # what an alias is worth
+    dev/ollama-overhead.py <url> <model>  # time spent before inference starts
+    dev/ollama-tap.py                # what Home Assistant really sends ollama
+    dev/host-e2e.py                  # the real machine, audio in to audio out
+    dev/kokoro-stream-test.py <main.py>   # the synthesiser's protocol handling
+
+Run them with `dev/py`, which supplies websockets, numpy and ffmpeg; which
+packages a bare `python3` happens to have is not something to depend on.
+
+`dev/speculation-speech.py` needs the stand-in synthesiser, added to Home
+Assistant as a Wyoming entry on port 10211:
+
+    dev/run-fake-tts.sh
+
+Anything that streams audio needs a transcriber in this VM, because the host's
+is bound to loopback:
+
+    $(nix build --no-link --print-out-paths nixpkgs#wyoming-faster-whisper)/bin/wyoming-faster-whisper \
+      --model tiny-int8 --language en --uri tcp://127.0.0.1:10300 \
+      --data-dir ~/ha-dev/whisper --download-dir ~/ha-dev/whisper
+
+`stt.demo_stt` cannot stand in for it: it accepts only stereo and the pipeline
+sends mono.
+
+State lives in `~/ha-dev`, the token in `scratch/hadev/token.txt`. Delete the
+directory to start over; the three scripts rebuild everything.
+
+## What runs where
+
+Only two things need the GPU, and they stay on the NixOS host:
+
+| | where | why |
+|---|---|---|
+| ollama | host | CUDA, and the model is 28 GiB |
+| Kokoro TTS | host | CUDA |
+| Home Assistant | **here** | no GPU; this is what we iterate on |
+| faster-whisper, openWakeWord | either | CPU; ~90 ms to transcribe |
+
+The dev instance talks to the host's ollama at `192.168.122.1:11434`, which is
+open on `virbr0`. The host's speech services are bound to loopback and are not
+reachable from here; run local ones if a test needs them.
+
+## Things that cost an hour to find
+
+- The package is pinned to **nixpkgs-stable**, matching the host. The Silero
+  patch does not apply to Home Assistant 2026.8.2 in unstable. A dev instance on
+  a different version teaches you nothing transferable.
+- `extraComponents` does not change the derivation. The NixOS module passes the
+  component dependencies through `environment.PYTHONPATH = package.pythonPath`,
+  which is why `.#hass-dev` is a wrapper that exports it.
+- The module also always adds `defaultIntegrations`, including **frontend**.
+  Without it `hass_frontend` is missing, frontend setup fails, and Home
+  Assistant silently drops into **recovery mode** -- which ignores
+  `configuration.yaml`, so nothing loads and the failure looks like anything but
+  a missing frontend.
+- Do not use `default_config:`; it pulls dhcp, go2rtc, logbook, my, ssdp and
+  stream, and taking everything it would have set up down with it when they are
+  missing.
+- `pkill -f hass` matches the shell running it. Use the bracketed pattern in
+  `run-ha.sh`.
+
+## More things that cost an hour to find
+
+- **Websocket ids must increase.** Home Assistant rejects a lower id with
+  `id_reuse`, so `halib` hands them out centrally rather than letting callers
+  pick.
+- **Do not truncate `hass.log` while Home Assistant holds it open.** The write
+  offset stays where it was and the file fills with nul bytes, so `grep` finds
+  nothing and the log looks empty. Restart it instead.
+- **Subentry ids are not in the REST entry listing**, which reports only
+  `num_subentries`. They come from `config_entries/subentries/list` over the
+  websocket.
+- **Turn-detection verdicts log at debug level.** Without the `logger:` block in
+  `configuration.yaml` the decision is invisible and you can only infer it from
+  timing.
+- **`pkill -f` matches the shell that typed it.** A command containing
+  `fake-tts.py` is itself a match, so the pattern kills the session. Bracketing
+  only helps when the name does not appear elsewhere on the line; a pid file and
+  a script, as in `run-fake-tts.sh`, always works.
+- **Home Assistant caches synthesised speech.** A test that plays the same clip
+  repeatedly is served from the cache and never reaches the synthesiser, which
+  reads as "nothing was spoken". `tts.clear_cache` between cases.
+- **A Wyoming synthesiser must send an audio header even with nothing to say.**
+  Without an `AudioStart`/`AudioStop` pair Home Assistant waits for audio that
+  never arrives, and the whole pipeline appears to hang somewhere else entirely.
+- **Home Assistant sends the whole message again after the chunks**, as a plain
+  `Synthesize`, for servers that cannot stream. One that can must ignore it or
+  it says everything twice.
+- **Do not reconstruct a request you can capture.** Estimating Home Assistant's
+  prompt from its parts gave a wrong answer twice. `dev/ollama-tap.py` proxies
+  the real one and the numbers stopped moving.
+- **ollama's `load_duration` is not loading**, and is not zero for a resident
+  model: it brackets everything before the runner is handed the request.

dev/configuration.yaml

@@ -0,0 +1,30 @@
+# Development Home Assistant, run in the agent VM. Copied to the config dir by
+# dev/run-ha.sh.
+#
+# Deliberately NOT default_config: that pulls dhcp, go2rtc, logbook, my, ssdp and
+# stream, which are not in the package's extraComponents, and when it fails to
+# set up everything it would have pulled in fails with it -- including
+# conversation and assist_pipeline.
+homeassistant:
+  name: Dev
+  time_zone: America/New_York
+  unit_system: us_customary
+  country: US
+
+http:
+api:
+websocket_api:
+config:
+
+conversation:
+assist_pipeline:
+
+# Fake lights and sensors, so the assistant has something to control.
+demo:
+
+# Turn detection logs its verdict at debug level; without this the decision is
+# invisible and a test can only infer it from timing.
+logger:
+  default: info
+  logs:
+    homeassistant.components.assist_pipeline: debug

dev/e2e-compare.py

@@ -0,0 +1,79 @@
+"""End to end, old settings against new: audio in, answer out."""
+import asyncio, json, statistics, subprocess, sys, time, urllib.request
+sys.argv = sys.argv[:1]
+exec(open("dev/turn-test.py").read().split("async def main")[0])
+
+async def one(ws, ident, pid, audio, **settings):
+    stream = audio + b"\x00" * (16000 * 2 * 4)
+    await ws.send(json.dumps({"id": ident, "type": "assist_pipeline/run",
+                              "start_stage": "stt", "end_stage": "intent",
+                              "input": {"sample_rate": 16000, **settings},
+                              "pipeline": pid, "timeout": 60}))
+    hid = None; task = None; marks = {}; t_audio_end = None; reply = ""
+    async def pump():
+        nonlocal t_audio_end
+        for i in range(0, len(stream), 3200):
+            await ws.send(bytes([hid]) + stream[i:i + 3200])
+            if i <= len(audio) < i + 3200:
+                t_audio_end = time.monotonic()
+            await asyncio.sleep(0.1)
+    while True:
+        m = json.loads(await ws.recv())
+        if m.get("id") != ident: continue
+        if m.get("type") == "result" and not m.get("success"):
+            return None, m.get("error"), ""
+        if m.get("type") != "event": continue
+        e = m["event"]
+        if e["type"] == "run-start":
+            hid = e["data"]["runner_data"]["stt_binary_handler_id"]
+            task = asyncio.create_task(pump())
+        marks[e["type"]] = time.monotonic()
+        if e["type"] == "intent-end":
+            reply = e["data"]["intent_output"]["response"]["speech"]["plain"]["speech"]
+        if e["type"] in ("run-end", "error"): break
+    if task: task.cancel()
+    if "intent-end" not in marks or t_audio_end is None:
+        return None, "no intent-end", ""
+    return (marks["intent-end"] - t_audio_end) * 1000, None, reply
+
+async def main():
+    ensure_whisper()
+    ws = await websockets.connect(URL, max_size=None); await ws.recv()
+    await ws.send(json.dumps({"type": "auth", "access_token": TOKEN}))
+    assert json.loads(await ws.recv())["type"] == "auth_ok"
+    states = rest("/api/states")
+    stt = next(s["entity_id"] for s in states
+               if s["entity_id"].startswith("stt.") and "demo" not in s["entity_id"])
+    conv = next((s["entity_id"] for s in states
+                 if s["entity_id"].startswith("conversation.") and "ollama" in s["entity_id"]),
+                "conversation.home_assistant")
+    pls = (await call(ws, 1, type="assist_pipeline/pipeline/list"))["result"]["pipelines"]
+    p = next((x for x in pls if x.get("name") == "e2e"), None)
+    if not p:
+        res = await call(ws, 2, type="assist_pipeline/pipeline/create", name="e2e",
+                         language="en", conversation_engine=conv, conversation_language="en",
+                         stt_engine=stt, stt_language="en", tts_engine=None,
+                         tts_language=None, tts_voice=None, wake_word_entity=None,
+                         wake_word_id=None)
+        pid = res["result"]["id"]
+    else:
+        pid = p["id"]
+    print(f"conversation agent: {conv}\n")
+    audio = pcm(sys.argv[1] if len(sys.argv) > 1 else "scratch/hadev/cmd.wav")
+    ident = 400
+    for label, s in (("before: silence 0.7, no turn model",
+                      dict(silence_seconds=0.7, turn_detection=False)),
+                     ("after:  silence 0.1, turn model",
+                      dict(silence_seconds=0.1, turn_detection=True,
+                           turn_threshold=0.9, turn_max_seconds=2.0))):
+        runs = []
+        for _ in range(3):
+            ident += 1
+            ms, err, reply = await one(ws, ident, pid, audio, **s)
+            if err: print(f"  {label}: ERROR {err}"); break
+            runs.append(ms)
+            await asyncio.sleep(2)
+        if runs:
+            print(f"  {label:38} {statistics.median(runs):6.0f} ms   {reply[:40]!r}")
+
+asyncio.run(main())

dev/eval.py

@@ -0,0 +1,122 @@
+"""Score conversation models on scenarios, against the development instance.
+
+Latency scripts cannot tell you whether a model does the right thing. Each
+scenario sets entity state, says something, and checks either the resulting
+state or the words of the reply. State is reset before every single run, so a
+light left on by one scenario cannot make the next one look correct.
+
+    dev/eval.py [reps] [model ...]
+"""
+import asyncio, json, statistics, sys, time
+sys.path.insert(0, "dev")
+from halib import call, connect, engines, rest
+
+SCENARIOS = json.load(open("dev/scenarios.json"))
+
+
+def set_state(entity, state):
+    rest(f"/api/states/{entity}", {"state": state})
+
+
+def get_state(entity):
+    try:
+        return rest(f"/api/states/{entity}")["state"]
+    except Exception:
+        return None
+
+
+async def set_model(ws, entry_id, sub_id, model):
+    flow = rest("/api/config/config_entries/subentries/flow",
+                {"handler": [entry_id, "conversation"], "subentry_id": sub_id,
+                 "show_advanced_options": True})
+    cur = {f["name"]: f.get("description", {}).get("suggested_value")
+           for f in flow["data_schema"] if "name" in f}
+    body = {k: v for k, v in cur.items() if v is not None}
+    body["model"] = model
+    for k in ("num_ctx", "max_history", "keep_alive"):
+        if k in body:
+            body[k] = int(body[k])
+    return rest(f"/api/config/config_entries/subentries/flow/{flow['flow_id']}",
+                body).get("reason")
+
+
+async def say(ws, pipeline, text):
+    ident_start = time.monotonic()
+    await ws.send(json.dumps({"id": (ident := _next()), "type": "assist_pipeline/run",
+                              "start_stage": "intent", "end_stage": "intent",
+                              "input": {"text": text}, "pipeline": pipeline,
+                              "timeout": 120}))
+    reply = ""
+    while True:
+        m = json.loads(await ws.recv())
+        if m.get("id") != ident:
+            continue
+        if m.get("type") == "result" and not m.get("success"):
+            return "", 0.0
+        if m.get("type") != "event":
+            continue
+        e = m["event"]
+        if e["type"] == "intent-end":
+            reply = e["data"]["intent_output"]["response"]["speech"]["plain"]["speech"]
+        if e["type"] in ("run-end", "error"):
+            break
+    return reply, (time.monotonic() - ident_start) * 1000
+
+
+_id = 1000
+def _next():
+    global _id
+    _id += 1
+    return _id
+
+
+async def main():
+    reps = int(sys.argv[1]) if len(sys.argv) > 1 else 3
+    models = sys.argv[2:] or ["qwen3.8:27b-mtp-q8_0"]
+    ws = await connect()
+    entry = next(e for e in rest("/api/config/config_entries/entry")
+                 if e["domain"] == "ollama")
+    subs = (await call(ws, type="config_entries/subentries/list",
+                       entry_id=entry["entry_id"]))["result"]
+    entry_id, sub_id = entry["entry_id"], subs[0]["subentry_id"]
+    _stt, conv = engines()
+    pls = (await call(ws, type="assist_pipeline/pipeline/list"))["result"]["pipelines"]
+    pipe = next((p["id"] for p in pls if p.get("conversation_engine") == conv), None)
+    if pipe is None:
+        res = await call(ws, type="assist_pipeline/pipeline/create", name="eval",
+                         language="en", conversation_engine=conv,
+                         conversation_language="en", stt_engine=None, stt_language=None,
+                         tts_engine=None, tts_language=None, tts_voice=None,
+                         wake_word_entity=None, wake_word_id=None)
+        pipe = res["result"]["id"]
+
+    for model in models:
+        if await set_model(ws, entry_id, sub_id, model) != "reconfigure_successful":
+            print(f"{model}: could not select"); continue
+        await asyncio.sleep(2)
+        await say(ws, pipe, "hello")           # load the model
+        print(f"\n### {model}")
+        total = passed = 0
+        latencies = []
+        for sc in SCENARIOS:
+            ok = 0
+            for _ in range(reps):
+                for entity, state in (sc.get("setup") or {}).items():
+                    set_state(entity, state)
+                reply, ms = await say(ws, pipe, sc["say"])
+                latencies.append(ms)
+                good = True
+                for entity, want in (sc.get("expect_state") or {}).items():
+                    if get_state(entity) != want:
+                        good = False
+                if sc.get("expect_any"):
+                    good = good and any(w.lower() in reply.lower()
+                                        for w in sc["expect_any"])
+                ok += good
+                await asyncio.sleep(0.3)
+            total += reps; passed += ok
+            flag = "" if ok == reps else f"   <- {ok}/{reps}"
+            print(f"  {sc['id']:16} {ok}/{reps}{flag}")
+        print(f"  {'TOTAL':16} {passed}/{total}   median {statistics.median(latencies):.0f} ms")
+
+asyncio.run(main())