Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions HARDWARE_SETUP.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,38 @@ broadly to receive on whatever interface Tailscale creates. The actual
security boundary is "only devices on your VPN can reach this port," which
is the VPN's job, not the app's — see `MindMonitorOSCStream`'s doc comment.

### Outbound biofeedback OSC and Mind Monitor port separation

The Mind Monitor transport above is inbound: `MindMonitorOSCStream` listens
on UDP 5000 (or `--port`) for the phone's packets. The biofeedback egress
(`./Scripts/run-biofeedback-osc.sh`, ADR-011) is outbound: `OSCUDPSender`
sends derived features to `127.0.0.1:7000` by default, or to the
`--host`/`--port` you give it. They are different sockets in different
directions, so both can run at once:

```bash
./Scripts/run-biofeedback-osc.sh --board osc --port 7000 # in: Mind Monitor on UDP 5000; out: features on UDP 7000
```

Keep the two ports distinct. Sending features back to the port Mind Monitor
delivers on would feed NeuralCompose its own output. `7000` is the Muse Port
Max for Live default; many OSC tools use `9000`, so `--port 9000` is the
other common choice. Loopback stays on this Mac; any other `--host` is the
LAN tier and the privacy banner shows it in red.

**One BLE central at a time (Muse native-BLE profiles).** The direct BrainFlow
paths (`museTwoNativeBLE`, `museSNativeBLE`, `museSAthena`) make this Mac the
headband's Bluetooth central. On these supported Muse revisions the headband
maintains a single active BLE central connection, so Mind Monitor on the phone
must be quit (not merely backgrounded) before the Mac can discover the
band, and the band must be power-cycled if it was paired to the phone
moments ago. The reverse holds too: while Mind Monitor streams to you over
OSC, do not also launch a native BLE profile. Pick one path per session:

- Headband near the Mac: native BLE profile, biofeedback out on 7000.
- Headband on the phone: Mind Monitor in on 5000 over the VPN, biofeedback
out on 7000. The phone, not this Mac, is the BLE central.

## Installing BrainFlow

NeuralCompose links BrainFlow as a system library, not as a SwiftPM package.
Expand Down
41 changes: 41 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -487,9 +487,47 @@ Some MLX, Core ML, live network, hardware, and operator-attended tests are inten
| `NEURALCOMPOSE_RUNTIME` | Selects generation runtime, currently `claude` or `ollama` on the packaged runtime path |
| `NEURALCOMPOSE_MODEL` | Requested model identity |
| `NEURALCOMPOSE_INTERACTION_LOG` | Enables opt-in local interaction logging when supported by the selected path |
| `NEURALCOMPOSE_BIOFEEDBACK_OSC` | Biofeedback egress over OSC: `1` for loopback on UDP 7000, a bare port for loopback on that port, or `host:port` for a LAN destination. Absent, `0` or malformed: off |
| `NEURALCOMPOSE_BIOFEEDBACK_MIDI` | `1` creates the CoreMIDI virtual source "NeuralCompose Biofeedback" (in-machine, no network) |
| `NEURALCOMPOSE_BIOFEEDBACK_RAW_EEG` | `1` re-broadcasts raw `/muse/eeg` over OSC. Ignored without OSC; MIDI cannot carry it |
| `NEURALCOMPOSE_BIOFEEDBACK_PER_CHANNEL` | `1` sends four floats per band (one per channel) instead of the single healthy-channel average |
| `NEURALCOMPOSE_BIOFEEDBACK_MAPPINGS` | Path to a JSON array of `BiofeedbackMapping` routes. Optional; without it only the Mind Monitor mirror runs |

Environment variables configure a launch; they are not durable application preferences. Start a new process after changing them.

## Biofeedback out (OSC and CoreMIDI)

Off by default. When enabled (ADR-011, `decision_registry.md` entry 10), `BCIBiofeedback` turns each EEG window into a `BiofeedbackFrame` of derived features and sends them to music and visual tools: per-channel band powers, per-channel contact quality, the Track A blink and jaw clench events, and IMU readings when the transport has them. Raw EEG never leaves the process unless you pass the explicit raw toggle, and then only over OSC.

```bash
./Scripts/run-biofeedback-osc.sh # synthetic EEG -> OSC loopback 7000 (dry run, no hardware)
./Scripts/run-biofeedback-osc.sh --board muses # live Muse S over BLE
./Scripts/run-biofeedback-osc.sh --board osc --port 7000 # Mind Monitor in (UDP 5000) -> features out (7000)
./Scripts/run-biofeedback-osc.sh --host 192.168.1.20 # send to another machine on the LAN
./Scripts/run-biofeedback-osc.sh --midi # also create the "NeuralCompose Biofeedback" MIDI source
./Scripts/run-biofeedback-osc.sh --per-channel # 4 floats per band instead of 1 average
./Scripts/run-biofeedback-osc.sh --mappings ~/maps.json # load a [BiofeedbackMapping] JSON set
./Scripts/run-biofeedback-osc.sh --raw-eeg # explicit raw /muse/eeg re-broadcast
```

The script only exports the five `NEURALCOMPOSE_BIOFEEDBACK_*` variables above and runs the app; set them yourself for any other launch path.

**Mind Monitor-compatible addresses.** `MindMonitorMirror` speaks Mind Monitor's OSC scheme so a consumer written for it (the Muse Port Max for Live device, TouchDesigner, SuperCollider, Bitwig's OSC input) can be pointed at NeuralCompose unchanged:

| Address | Payload |
|---|---|
| `/muse/elements/delta_absolute`, `theta_absolute`, `alpha_absolute`, `beta_absolute`, `gamma_absolute` | 1 float (average over healthy channels) or, with `--per-channel`, 4 floats; Bels |
| `/muse/elements/horseshoe` | 4 floats, 1 good / 2 medium / 4 bad |
| `/muse/elements/blink`, `/muse/elements/jaw_clench` | 1 int, sent only on the event tick |
| `/muse/acc`, `/muse/gyro` | 3 floats (g, deg/s) when the frame has them |
| `/muse/eeg` | raw samples, only with `--raw-eeg` |

Not mirrored: `/muse/batt` and `/muse/elements/touching_forehead` (neither reaches the frame; the horseshoe carries contact quality). Mirror values are not health gated, exactly as Mind Monitor's are not: the consumer gets the horseshoe and decides.

**Custom mappings.** A `BiofeedbackMapping` routes one feature through shaping (linear or log10 normalisation into 0 to 1, smoothing) to an output of your choice: an OSC address of your own with one float, a MIDI control change (7 or 14 bit) or a MIDI note gate for blink and jaw clench. Every mapping carries a `HealthPolicy` for the electrode under it (dead, saturated, stale), which is the difference from feeding Mind Monitor straight into a DAW. The virtual MIDI source appears in Ableton Live, Logic, Reaper and friends as "NeuralCompose Biofeedback".

**Privacy.** Loopback (`127.0.0.1`, the default) and MIDI never leave the machine; the privacy banner shows them in orange. Any other host is a LAN destination over unauthenticated UDP, shown in red along with the raw EEG toggle; keep it on a private network. The badge is hidden while the feature is off. The OSC port here is an outbound port and is separate from the inbound Mind Monitor listener; see `HARDWARE_SETUP.md`, "Outbound biofeedback OSC and Mind Monitor port separation".

## Architecture

```text
Expand All @@ -500,6 +538,7 @@ NeuralComposeApp
├── BCILLM isolated MLX runtime and deterministic fallbacks
├── BCIVoice speech recognition and synthesis seams
├── BCICloudBridge explicit opt-in generation egress
├── BCIBiofeedback opt-in OSC and CoreMIDI egress of derived EEG features (ADR-011)
└── WorldModelDemo synthetic-task research demo, separate from live EEG claims
```

Expand All @@ -508,6 +547,7 @@ Load-bearing boundaries:
- `BCICore` owns portable interfaces and deterministic mechanisms.
- MLX imports are isolated to `BCILLM`.
- Cloud generation is isolated to `BCICloudBridge`.
- Biofeedback egress is isolated to `BCIBiofeedback`: derived features only, off by default, loopback by default.
- The interface consumes resolved runtime and pipeline state rather than inventing provider or locality claims.
- Synthetic research demos do not imply validation on physical EEG.

Expand All @@ -531,6 +571,7 @@ NeuralCompose/
│ ├── BCILLM/
│ ├── BCIVoice/
│ ├── BCICloudBridge/
│ ├── BCIBiofeedback/
│ ├── NeuralComposeApp/
│ ├── DialecticSession/
│ └── WorldModelDemo/
Expand Down
124 changes: 123 additions & 1 deletion Sources/BCICore/Composition/HypnagogicDialecticLoop.swift
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,23 @@ public actor HypnagogicDialecticLoop {
private var field: DialecticalField
/// Fast biological clock - EMA-smoothed `SpectralState` gloss.
private var gloss = SpectralGloss()
/// The last utterance this loop voiced, so a transport command ("say that
/// again", "slow down") can re-speak it without a generation call. Holds
/// the unadjusted competition prosody; the rate offset is applied at speak
/// time so repeated slow-downs compound from the same base.
private var lastVoicedTurn: VoicedTurn?
/// Cumulative rate offset from "slow down" requests, applied to every later
/// voiced turn and to replays. Steps by `slowDownStep`, floored at
/// `maxSlowDownOffset`; the resulting rate never drops below
/// `minimumSpeechRate`.
private var speechRateOffset: Float = 0

private struct VoicedTurn {
let text: String
let prosody: SpeechProsody
let nodeID: String
let embedding: Embedding
}

public init(
listener: any HypnagogicListening,
Expand Down Expand Up @@ -340,6 +357,14 @@ public actor HypnagogicDialecticLoop {
try await logSilentTurn(heard: heard, advanceSilenceCounter: false)
return
}
// Transport commands are spoken back to the voice, not to the topic:
// without this intercept "say that again" reached both generators as
// a philosophical prompt. Re-speak (and, for slow-down, re-rate) the
// last voiced turn; never generate, never advance the silence count.
if let transport = Self.classifyTransportCommand(heard) {
try await performTransportCommand(transport, heard: heard)
return
}
let discourseKind = Self.classifyDiscourse(heard)
if discourseKind == .ambientBackchannel {
try await logSilentTurn(heard: heard, advanceSilenceCounter: false)
Expand Down Expand Up @@ -542,8 +567,14 @@ public actor HypnagogicDialecticLoop {
// a close call carries the losing pole's colour (audible tension).
let probs = DialecticalDynamics.probabilities(
potentials: scored.map(\.potential), tau: result.selectionTemperature)
let turnProsody = SpeechProsody.blend(
let blendedProsody = SpeechProsody.blend(
zip(scored, probs).map { (roleProsody(for: $0.candidate.roleID), $1) })
// Any standing "slow down" request shapes this turn too; the base
// blend is what a later replay re-rates from.
let turnProsody = adjustedForSpeechRate(blendedProsody)
lastVoicedTurn = VoicedTurn(
text: candidate.text, prosody: blendedProsody,
nodeID: nodeID, embedding: candidate.embedding)
await logRequestedProsody(for: record, text: candidate.text, prosody: turnProsody)
try await speakChunks(candidate.text, prosody: turnProsody,
grounding: (nodeID: nodeID, embedding: candidate.embedding))
Expand Down Expand Up @@ -602,6 +633,97 @@ public actor HypnagogicDialecticLoop {
turnIndex += 1
}

// MARK: - Transport commands

/// What a transport utterance asks of the voice itself, as opposed to a
/// prompt about the topic.
public enum TransportCommand: String, Sendable, Equatable {
/// Voice the last spoken turn again, unchanged.
case repeatLast
/// Step the speech rate down, then voice the last spoken turn again at
/// the new rate.
case slowDown
}

/// Each "slow down" lowers the requested rate by this much (`SpeechRatePolicy.offsetStep`).
public static let slowDownStep: Float = SpeechRatePolicy.offsetStep
/// The offset never goes below this (`SpeechRatePolicy.maxSlowDownOffset`),
/// so a run of slow-downs cannot push the voice to a crawl.
public static let maxSlowDownOffset: Float = SpeechRatePolicy.maxSlowDownOffset
/// Floor for any adjusted rate (`SpeechRatePolicy.minimumSpeechRate`).
public static let minimumSpeechRate: Float = SpeechRatePolicy.minimumSpeechRate
/// Rate assumed when a prosody declares none (`SpeechRatePolicy.defaultSpeechRate`).
static let defaultSpeechRate: Float = SpeechRatePolicy.defaultSpeechRate

/// The standing rate offset (0 until the first slow-down). Read-only; for
/// diagnostics and tests.
public var currentSpeechRateOffset: Float { speechRateOffset }
/// Text of the last voiced turn, or nil before the first one.
public var lastVoicedText: String? { lastVoicedTurn?.text }

/// Recognizes the transport phrases heard in the interaction logs. Exact
/// match after the same normalization `isMemoryResetCommand` applies, plus
/// a leading or trailing "please". Slow-down phrases are checked first so
/// "can you repeat that slower" asks for the slower replay, not a plain
/// repeat. Anything else is a prompt and returns nil.
public static func classifyTransportCommand(_ heard: String) -> TransportCommand? {
var normalized = heard.trimmingCharacters(in: .whitespacesAndNewlines)
.lowercased()
.split(whereSeparator: { $0.isWhitespace })
.map { $0.trimmingCharacters(in: .punctuationCharacters) }
.filter { !$0.isEmpty }
.joined(separator: " ")
if normalized.hasPrefix("please ") { normalized.removeFirst("please ".count) }
if normalized.hasSuffix(" please") { normalized.removeLast(" please".count) }
let slowDownPhrases: Set<String> = [
"slow down", "speak slower", "slower", "say that slower", "say it slower",
"can you repeat that slower", "could you repeat that slower",
"can you say that slower", "could you say that slower", "too fast",
]
if slowDownPhrases.contains(normalized) { return .slowDown }
let repeatPhrases: Set<String> = [
"say that again", "repeat that", "repeat last", "say again", "say it again",
"can you repeat that", "could you repeat that", "can you say that again",
"could you say that again", "what did you say", "come again",
]
if repeatPhrases.contains(normalized) { return .repeatLast }
return nil
}

/// App-side entry for the "Repeat Last Utterance" command: re-speaks the
/// last voiced turn without logging a turn (nothing was heard). No-op
/// before the first voiced turn.
public func repeatLastUtterance() async throws {
try await performTransportCommand(.repeatLast, heard: nil)
}

/// App-side entry for the "Slow Down Speech" command: lowers the standing
/// rate offset and, if a turn has been voiced, re-speaks it at the new rate.
public func slowDownSpeech() async throws {
try await performTransportCommand(.slowDown, heard: nil)
}

/// Shared by the in-turn intercept (`heard` is the transcript, logged as a
/// candidate-less turn so the run's turn count still advances) and the
/// app-side entries (`heard` nil, nothing logged).
private func performTransportCommand(_ command: TransportCommand, heard: String?) async throws {
if command == .slowDown {
speechRateOffset = SpeechRatePolicy.stepDownOffset(speechRateOffset)
}
if let heard {
try await logSilentTurn(heard: heard, advanceSilenceCounter: false)
}
guard let last = lastVoicedTurn else { return }
phase = .speaking
try await speakChunks(last.text, prosody: adjustedForSpeechRate(last.prosody),
grounding: (nodeID: last.nodeID, embedding: last.embedding))
}

/// Applies the standing slow-down offset to a prosody via `SpeechRatePolicy`.
private func adjustedForSpeechRate(_ prosody: SpeechProsody) -> SpeechProsody {
SpeechRatePolicy.offset(prosody, by: speechRateOffset)
}

/// Discourse classification for distinguishing direct speech to NeuralCompose
/// from ambient room side-conversations and backchannels with family/friends.
public enum UtteranceDiscourseKind: String, Sendable, Equatable {
Expand Down
Loading
Loading