PLAN-v3 Phase 1: pin the web stack and fix the first-wave bugs

README.md

@@ -122,7 +122,7 @@ These optional feeds are **off** in the shipped `config.yaml`: bits, channel poi
 
 Pixel's spoken audio can go to a virtual cable (VB-Audio, VoiceMeeter). The device list labels those outputs; pick one in the booth. The default output is not changed until you pick it.
 
-Music ducking is off. Preferred setup: music on its own OBS track, sidechained to the cohost cable. `audio.duck_enabled: true` lowers other apps' per-session volume (pycaw) while Pixel talks, and only when `speaker_device` is a different device than the default output. With `speaker_device: null` the booth shows a note and does not change the volume. The pre-duck levels are stored in `data/duck.json` and restored on the next launch, and again if the process exits while ducked.
+Music ducking is off. Preferred setup: music on its own OBS track, sidechained to the cohost cable. `audio.duck_enabled: true` is per-app session ducking: on Windows, pycaw lowers other apps' session volume while Pixel talks. `speaker_device` may stay null. Pixel's own process is never ducked. `audio.duck_skip` (default `obs64.exe` and `Discord.exe`) is also left alone. An empty `audio.duck_apps` list ducks every other app; a non-empty list ducks only those process names. The pre-duck levels are stored in `data/duck.json` and restored on the next launch, and again if the process exits while ducked. Without pycaw the booth stays on the OBS sidechain and does not change the volume.
 
 Edit `persona.yaml` anytime, then click **Reload persona**. No restart needed.
 
@@ -183,7 +183,7 @@ Logs go to `logs/cohost.log` (rotating, about 2 MB, three backups). That directo
 
 Optional extras are not installed by either file. Add them yourself when you turn the feature on:
 
-- `pycaw` — Windows endpoint ducking when `audio.duck_enabled: true`
+- `pycaw` — per-app session ducking when `audio.duck_enabled: true`
 - `openwakeword` — local wake word when `audio.wake_word: true`
 
 To bump a pin: upgrade that package in the project venv (`python -m pip install --upgrade some-package`), run `python -m pip freeze`, and copy the new `name==version` into `requirements.txt` if the app imports it, or into `requirements-dev.txt` if it is only `pytest` or `ruff`. Leave `pytest` and `ruff` out of `requirements.txt`. Run the tests before you commit. Do not commit `.env`.
@@ -253,7 +253,7 @@ powershell -File scripts\backup.ps1
 
 `start.bat` prefers `py -3.11`, then `python`, and requires 3.11 or newer. It reads `server.host` and `server.port`, skips pip when `requirements.txt` has not changed, and opens the booth unless you pass `--no-browser`. If creating the venv, installing dependencies, or reading the config fails, the window stays open and the next launch tries the install again. The install stamp is written only after pip succeeds. Binding anything other than localhost requires `server.allow_remote: true`.
 
-The booth only accepts `Host` values of `127.0.0.1`, `localhost`, and `[::1]`. A page on another site cannot POST into it: writes need `Content-Type: application/json` and the `X-Cohost-Token` from `GET /api/session`, and a foreign `Origin` is rejected. The OBS overlay stays read-only. Each person in the memory panel has **Forget**, which deletes that viewer from `data/memory.json`. Viewer lines are kept for `memory.retention_days` (14). Piper and the three listed voices are checksummed; a partial download stays as a `.part` file until it finishes.
+The booth only accepts `Host` values of `127.0.0.1`, `localhost`, `::1`, and `[::1]`, including the configured port. A page on another site cannot POST into it: writes need `Content-Type: application/json` and the `X-Cohost-Token` from `GET /api/session`, and a foreign `Origin` is rejected. The OBS overlay stays read-only. Each person in the memory panel has **Forget**, which deletes that viewer from `data/memory.json`. Viewer lines are kept for `memory.retention_days` (14). Piper and the three listed voices are checksummed; a partial download stays as a `.part` file until it finishes.
 
 ## Out of scope
 

config.yaml

@@ -13,7 +13,7 @@ ollama:
   num_predict_chat: 80
   num_predict_voice: 160
   keep_alive: 30m
-  # Chat completions give up after this so a slow model can't stall the chat worker.
+  # Bounds chat replies, alert replies, and the recap vibe paragraph.
   chat_timeout_seconds: 20
 
 # snappy: Whisper on CUDA + qwen2.5:3b (fits an 8 GB GPU with Whisper).
@@ -54,9 +54,15 @@ audio:
   min_utterance_ms: 350
   energy_threshold: 0.012   # RMS; raise if it false-triggers
   max_utterance_seconds: 20
-  # Optional. Off by default. Needs pycaw to duck the Windows output; otherwise use an OBS sidechain.
+  # Optional per-app session ducking. Off by default. On Windows, pycaw lowers matching
+  # app sessions while Pixel talks. Pixel's own process is never ducked. An empty
+  # duck_apps list ducks every other app except duck_skip. Without pycaw, use an OBS sidechain.
   duck_enabled: false
   duck_gain: 0.25
+  duck_apps: []
+  duck_skip:
+    - obs64.exe
+    - Discord.exe
   # Talk-over stops Pixel. Echo gate ignores speaker bleed unless PTT is held or the level is much louder.
   barge_in: true
   echo_gate: true

requirements.txt

@@ -1,6 +1,22 @@
 # Pinned runtime install used by start.bat.
 # pytest and ruff live in requirements-dev.txt. See README "Dependencies" for how to bump pins.
 fastapi==0.141.1
+# Transitive web stack, pinned to the versions CI resolved for these fastapi and uvicorn pins.
+# starlette 1.6.0 splits Host on the first colon, so Host: [::1]:port is rejected.
+# uvloop stays unpinned: it is Unix-only and is not installed on the Windows CI runner.
+starlette==1.7.0
+anyio==4.15.1
+idna==3.20
+pydantic==2.14.0
+pydantic-core==2.50.0
+annotated-types==0.8.0
+annotated-doc==0.0.5
+typing-extensions==4.16.0
+typing-inspection==0.4.4
+click==8.5.0
+h11==0.16.0
+httptools==0.9.0
+watchfiles==1.3.0
 uvicorn[standard]==0.52.4
 httpx==0.28.1
 PyYAML==6.0.3

src/audio/duck.py

@@ -15,10 +15,6 @@
     "Ducking is off in-app. In OBS, put music on its own track and sidechain a compressor "
     "to the cohost output (a VB-Audio cable works). audio.duck_enabled stays false unless you opt in."
 )
-REFUSE_NOTE = (
-    "Ducking is off: speaker_device is null or is the default output, so Pixel's own voice would be ducked. "
-    "Set speaker_device to a different device, or sidechain music in OBS."
-)
 
 
 class _PycawSessions:
@@ -54,11 +50,12 @@ def set_volume(self, key: str, level: float) -> None:
 
 
 class Duck:
-    """Optional per-session volume duck. Off by default.
+    """Optional per-app session duck. Off by default.
 
-    Endpoint ducking is refused unless ``speaker_device`` is a different device
-    than the default output, so Pixel's own voice is not lowered. Saved levels
-    go to ``data/duck.json`` and are restored at startup and on exit.
+    Pixel's own process is skipped by the Windows snapshot. ``audio.duck_skip``
+    names more processes to leave alone, and ``audio.duck_apps`` limits ducking
+    to that allow-list when it is not empty. Saved levels go to ``data/duck.json``
+    and are restored at startup and on exit. ``speaker_device`` may stay null.
     """
 
     def __init__(self, settings: Settings, *, backend=None, data_dir: Path | None = None) -> None:
@@ -75,33 +72,20 @@ def __init__(self, settings: Settings, *, backend=None, data_dir: Path | None =
         if settings.duck_enabled:
             self._try_bind()
 
-    def _default_output(self):
-        try:
-            import sounddevice as sd
+    def _session_name(self, key: str) -> str:
+        return str(key).split(":", 1)[0]
 
-            _mic, speaker = sd.default.device
-            return speaker
-        except Exception:
-            return None
-
-    def _speaker_is_separate(self) -> bool:
-        speaker = self.settings.speaker_device
-        if speaker is None or speaker == "":
-            return False
-        default = self._default_output()
-        if default is None:
-            return False
-        try:
-            return int(speaker) != int(default)
-        except (TypeError, ValueError):
+    def _should_duck(self, key: str) -> bool:
+        name = self._session_name(key).lower()
+        skip = {item.lower() for item in (self.settings.duck_skip or [])}
+        if name in skip:
             return False
+        allow = [item for item in (self.settings.duck_apps or []) if str(item).strip()]
+        if allow:
+            return name in {item.lower() for item in allow}
+        return True
 
     def _try_bind(self) -> None:
-        if not self._speaker_is_separate():
-            self.mode = "obs-sidechain"
-            self.note = REFUSE_NOTE
-            self._warn_once()
-            return
         if self._backend is None:
             if sys.platform != "win32":
                 self.mode = "obs-sidechain"
@@ -195,7 +179,11 @@ def down(self) -> None:
         if self._saved:
             return
         try:
-            current = {str(key): float(value) for key, value in (self._backend.snapshot() or {}).items()}
+            current = {
+                str(key): float(value)
+                for key, value in (self._backend.snapshot() or {}).items()
+                if self._should_duck(str(key))
+            }
         except Exception:
             log.exception("duck snapshot failed")
             self.mode = "obs-sidechain"

src/audio/mic.py

@@ -10,7 +10,7 @@
 import sounddevice as sd
 
 from src.audio.vad import has_speech_silero, rms
-from src.audio.wake import WakeWord
+from src.audio.wake import WAKE_FRAME, WakeWord
 from src.settings import Settings
 
 log = logging.getLogger("cohost.mic")
@@ -46,8 +46,8 @@ def __init__(
         self.level = 0.0
         self.wake = WakeWord(settings)
         self.wake_score = 0.0
-        self._wake_buf: deque[np.ndarray] = deque()
-        self._wake_samples = 0
+        self._wake_carry = np.zeros(0, dtype=np.float32)
+        self._wake_backlog: list[np.ndarray] = []
         self._wake_latch = False
         self._spoke_at = 0.0
         self._barge_sent = False
@@ -139,6 +139,24 @@ def _request_barge(self) -> None:
         if self.on_barge is not None:
             self.on_barge()
 
+    def _take_wake_frames(self, chunk: np.ndarray) -> None:
+        piece = np.asarray(chunk, dtype=np.float32)
+        audio = np.concatenate((self._wake_carry, piece)) if self._wake_carry.size else piece
+        count = (int(audio.size) // WAKE_FRAME) * WAKE_FRAME
+        if count:
+            self._wake_backlog.append(np.ascontiguousarray(audio[:count]))
+        self._wake_carry = np.ascontiguousarray(audio[count:])
+
+    def _pump_wake(self, *, force: bool = False) -> None:
+        if self._wake_latch or not self._wake_backlog:
+            return
+        if not force and self._wake_task is not None and not self._wake_task.done():
+            return
+        batch = self._wake_backlog
+        self._wake_backlog = []
+        audio = np.concatenate(batch)
+        self._wake_task = asyncio.create_task(self._score_wake(audio), name="wake-score")
+
     async def _score_wake(self, audio: np.ndarray) -> None:
         try:
             score, heard = await asyncio.to_thread(self.wake.evaluate, audio)
@@ -148,6 +166,8 @@ async def _score_wake(self, audio: np.ndarray) -> None:
         self.wake_score = score
         if heard:
             self._wake_latch = True
+            return
+        self._pump_wake(force=True)
 
     def _schedule_utterance(self, audio: np.ndarray) -> None:
         previous = self._utterance_task
@@ -170,7 +190,6 @@ async def _run(self) -> None:
         captured = 0
         silent = 0
         ptt_was_down = False
-        wake_need = int(sr * 0.08)
 
         while True:
             chunk = await self._queue.get()
@@ -184,15 +203,8 @@ async def _run(self) -> None:
                 wake_hit = True
             armed = self.listen_enabled or self.ptt_down or self._wake_latch or wake_hit
             if not armed and self.wake.model is not None:
-                self._wake_buf.append(chunk)
-                self._wake_samples += len(chunk)
-                while self._wake_samples > wake_need * 4 and self._wake_buf:
-                    dropped = self._wake_buf.popleft()
-                    self._wake_samples -= len(dropped)
-                ready = self._wake_samples >= wake_need and (self._wake_task is None or self._wake_task.done())
-                if ready:
-                    audio = np.concatenate(list(self._wake_buf))
-                    self._wake_task = asyncio.create_task(self._score_wake(audio), name="wake-score")
+                self._take_wake_frames(chunk)
+                self._pump_wake()
 
             if not armed:
                 if capturing and captured >= min_len:

src/audio/wake.py

@@ -78,7 +78,13 @@ def evaluate(self, audio: np.ndarray) -> tuple[float, bool]:
                     continue
                 best = max(best, float(value))
         self.score = best
-        return best, best >= self.threshold
+        heard = best >= self.threshold
+        if heard and self.model is not None and hasattr(self.model, "reset"):
+            try:
+                self.model.reset()
+            except Exception:
+                log.exception("wake reset failed")
+        return best, heard
 
     def heard(self, audio: np.ndarray) -> bool:
         _score, heard = self.evaluate(audio)

src/health.py

@@ -8,7 +8,7 @@
 
 from src.audio.devices import list_devices
 from src.settings import Settings
-from src.twitch.auth import FEED_ORDER
+from src.twitch.auth import FEED_ORDER, action_status, remaining_ttl
 
 _gpu_cache: tuple[float, dict[str, Any]] | None = None
 _ollama_cache: tuple[float, dict[str, Any]] | None = None
@@ -102,6 +102,7 @@ async def snapshot(settings: Settings, orch) -> dict[str, Any]:
             "has_event_token": bool(settings.event_token()),
             "events_enabled": settings.events_enabled,
             "feeds": _twitch_feeds(orch),
+            "actions": action_status(getattr(getattr(orch, "eventsub", None), "scopes", None)),
         },
         "persona": orch.persona.get("name"),
         "streamer": {
@@ -158,9 +159,17 @@ def _twitch_feeds(orch) -> dict[str, str]:
 
 def _setup_hints(ollama: dict[str, Any], settings: Settings, gpu: dict[str, Any] | None = None, orch=None) -> list[str]:
     hints: list[str] = []
-    expires = getattr(getattr(orch, "eventsub", None), "expires_in", None)
+    client = getattr(orch, "eventsub", None)
+    expires = remaining_ttl(
+        getattr(client, "expires_in", None),
+        getattr(client, "validated_at", None),
+        time.time(),
+    )
     if isinstance(expires, int) and expires < 24 * 60 * 60:
         hints.append("Twitch token expires in under 24 hours. Generate a new one before the next stream.")
+    note = str(getattr(orch, "irc_auth_note", "") or "")
+    if note:
+        hints.append(note)
     hints.extend(settings.secret_hints())
     if not ollama.get("ok"):
         hints.append("Ollama is not reachable. Install it from https://ollama.com and keep it running.")

src/main.py

@@ -109,6 +109,24 @@ class RecapBody(BaseModel):
     vibe: bool = False
 
 
+def _booth_port(prepared: tuple[Settings, Hub, Orchestrator] | None) -> int:
+    """Port for the Host allow list.
+
+    A prepared app (tests) uses that settings object. The process app reads
+    ``server.port`` from config without loading ``.env``.
+    """
+    if prepared is not None:
+        return int(prepared[0].port)
+    try:
+        import yaml
+
+        with (ROOT / "config.yaml").open(encoding="utf-8") as handle:
+            data = yaml.safe_load(handle) or {}
+        return int((data.get("server") or {}).get("port") or 8080)
+    except (OSError, TypeError, ValueError):
+        return 8080
+
+
 def create_app(prepared: tuple[Settings, Hub, Orchestrator] | None = None) -> FastAPI:
     @asynccontextmanager
     async def life(app: FastAPI):
@@ -138,7 +156,7 @@ async def life(app: FastAPI):
 
     app = FastAPI(title="Local Twitch Cohost", lifespan=life)
     app.add_middleware(BoothGuard)
-    app.add_middleware(TrustedHostMiddleware, allowed_hosts=trusted_hosts(), www_redirect=False)
+    app.add_middleware(TrustedHostMiddleware, allowed_hosts=trusted_hosts(_booth_port(prepared)), www_redirect=False)
     app.mount("/static", StaticFiles(directory=STATIC), name="static")
 
     @app.get("/")

src/memory.py

@@ -577,6 +577,10 @@ def llm_messages(self, thread_key: str, limit: int | None = None) -> list[dict[s
         for turn in list(self._thread(thread_key))[-n:]:
             if turn.role == "cohost":
                 msgs.append({"role": "assistant", "content": turn.text})
+            elif turn.role == "viewer":
+                msgs.append(
+                    {"role": "user", "content": f"{turn.who} ({turn.via}): {fence_viewer_text(turn.text)}"}
+                )
             else:
                 msgs.append({"role": "user", "content": f"{turn.who} ({turn.via}): {turn.text}"})
         return msgs