Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# NeuralCompose

Privacy-first, fully on-device macOS prototype for **EEG-driven communication**. A Muse headband streams brain signals through BrainFlow, a Core ML classifier on the Apple Neural Engine detects intent (jaw clench / blink / rest / select), and a local MLX LLM suggests next words. The user "types" by letting a cycling 3-token carousel highlight a candidate, then committing with a brain-signal selection. **No network at runtime. No cloud. No telemetry.**
Privacy-first, fully on-device macOS prototype for **EEG-driven communication**. A Muse headband streams brain signals through BrainFlow, a Core ML classifier on the Apple Neural Engine detects intent (jaw clench / blink / rest / select), and a local MLX LLM suggests next words. The user "types" by letting a cycling 3-token carousel highlight a candidate, then committing with a brain-signal selection. **No cloud. No telemetry. Network egress only through the three registered exceptions (registry entries 8, 9, and 10).**

## Stack
- Swift 6.0, macOS 14+, SwiftPM (no Xcode project needed for the default build)
Expand All @@ -22,6 +22,7 @@ Only the non-obvious constraints; the directory tree speaks for itself otherwise
- `Sources/BCIBridge/` — Obj-C++ shim for BrainFlow. Default builds with `BCI_BRIDGE_STUB`; define `BCI_BRAINFLOW_AVAILABLE` + `-lBrainflow` to wire in the real library.
- `Sources/BCICore/` — pure-Swift, **no third-party deps** (protocols, models, ring buffer, intent FSM)
- `Sources/BCILLM/` — **isolated MLX adapter**; ONLY target that imports MLX/transformers
- `Sources/BCIBiofeedback/` : biofeedback egress (ADR-011): `BandPowerEstimator` -> `BiofeedbackFrame` -> `FeatureMapper` (health-gated mappings) / `MindMonitorMirror` (Mind Monitor-compatible OSC) -> `BiofeedbackSink` conformers (`OSCUDPSender`, `CoreMIDIVirtualSource`, recording/null stubs). Off by default; derived features only, never raw EEG unless the explicit raw toggle is on
- `WorldModel/` — **decoupled research spike** (PyTorch, not MLX; no dependency on `Sources/`) — constraints live in `WorldModel/CLAUDE.md` (auto-loads when working there)
- `HARDWARE_SETUP.md`, `MODEL_SETUP.md`, `CALIBRATION.md`, `TROUBLESHOOTING.md` — operational docs (keep up to date)

Expand Down
10 changes: 10 additions & 0 deletions Evaluation/reports/decision_registry.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,3 +101,13 @@ Each entry:
- **Confidence:** Medium — the engineering mirrors an already-reviewed quarantine pattern (entry 8) and the wire format was fixed and agreed by both sessions before either wrote code; the *experiment's* statistical power is explicitly low (3 runs/arm, pre-registered as showing a large effect or nothing reliable) and is out of scope for this entry, which covers the bridge mechanism only.
- **Status:** Accepted for the experimental, opt-in bridge only. Network egress is now possible in the runtime through exactly two quarantined, independently-gated paths — `BCICloudBridge` (entry 8) and `BCIPeerBridge` (this entry) — both opt-in and off by default; the invariant otherwise stands.
<!-- Last updated: 2026-10-11T00:00:00Z (entry 9: two-machine hypnagogic dialectic bridge network-egress exception — BCIPeerBridge, opt-in, text-only, Tailscale-only) -->

### 10. Biofeedback egress (ADR-011) : scoped network-egress exception, derived features only

- **Decision:** A new `BCIBiofeedback` target may emit EEG-derived features (per-channel band powers in Bels, per-channel health, Track A blink/clench events, IMU) over two transports: a CoreMIDI virtual source (in-machine, not network) and OSC/UDP to a user-chosen host, loopback `127.0.0.1:7000` by default. This is a deliberate exception to the "No network at runtime" invariant (CLAUDE.md line 3 / PRINCIPLES.md Principle 5) alongside entries 8 (`BCICloudBridge`) and 9 (`BCIPeerBridge`), and the only one that carries EEG-derived numbers. Accepted 2026-11-07; status follows ADR-011.
- **Evidence / rationale:** (a) The inbound half of the Muse-to-DAW chain already exists (`MindMonitorOSCStream`, ADR-004); the outbound half is what makes NeuralCompose usable as an open biofeedback instrument alongside Mind Monitor + Muse Port. (b) The egress is **tightly scoped**: one target, dependencies `BCICore` + `BCIEEG` only, imported only by `NeuralComposeApp`, never a dependency of another `BCI*` target. (c) **Raw EEG never reaches a frame**: the egress boundary is `BiofeedbackFrame`, a function of band powers, health, events and IMU; raw `/muse/eeg` re-broadcast is a separate explicit toggle through the one named helper `MindMonitorMirror.rawEEG(_:)`. (d) **Health gating is mandatory**: every `BiofeedbackMapping` carries a `HealthPolicy` (default `holdLast`); Mind Monitor's behaviour is available only as the explicit `passthrough` policy. (e) **Deterministic**: pure-Swift periodogram and a clock-free mapper, so a recording through a mapping set yields byte-identical outputs for the replay regression. (f) Off by default; while off the app wires `NullBiofeedbackSink`. (g) No third-party dependency; Network and CoreMIDI are system frameworks behind `canImport`.
- **Scope guard - what this does NOT do:** it does **not** touch `BCICloudBridge` or `BCIPeerBridge` or widen entries 8 and 9; cloud egress stays exclusively in `BCICloudBridge`. It adds no HTTP client. It does not enable macOS Network MIDI (an OS feature the user controls in Audio MIDI Setup). It does not expose World Model or Laya latents as features; that waits for a benchmarked encoder and its own amendment. The `MindMonitorMirror` compatibility output is ungated by design and says so; gating is the mapper's job.
- **Supporting benchmark(s):** `Tests/BCIBiofeedbackTests/BandPowerEstimatorTests.swift` (A^2/2 physical-unit checks, FFT vs naive DFT), `FeatureMapperTests.swift` (all four health policies, smoothing, rate limit, MIDI quantisation, note gates, determinism), `MindMonitorMirrorTests.swift` (address scheme, payload shapes, bundle round trip through `MuseOSCDecoder`), `Tests/BCIEEGTests/OSCEncoderTests.swift`. Golden-recording replay fixture is a follow-up issue and gates live validation.
- **Confidence:** High for the egress core: the design is isolated and all 64 unit tests across `BCIBiofeedbackTests` and `OSCEncoderTests` (including `OSCUDPSender` and `CoreMIDIVirtualSource` compilation) compiled and passed on M4 under Swift 6.0; the `AppContainer` wiring, settings and privacy-indicator strings are follow-up PRs.
- **Status:** Accepted. Network egress of derived features is possible **exclusively** through `BCIBiofeedback` once wired and enabled; raw EEG egress only through the explicit raw toggle; the invariant otherwise stands.
<!-- Last updated: 2026-11-07T08:45:00-06:00 (entry 10: ADR-011 biofeedback egress - accepted, 64/64 tests green on M4) -->
31 changes: 31 additions & 0 deletions Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ let package = Package(
.library(name: "BCIVoice", targets: ["BCIVoice"]),
.library(name: "BCICloudBridge", targets: ["BCICloudBridge"]),
.library(name: "BCIPeerBridge", targets: ["BCIPeerBridge"]),
.library(name: "BCIBiofeedback", targets: ["BCIBiofeedback"]),
.library(name: "WorldModelDemo", targets: ["WorldModelDemo"]),
.executable(name: "EmbeddingBench", targets: ["EmbeddingBench"]),
.executable(name: "SemanticEval", targets: ["SemanticEval"]),
Expand Down Expand Up @@ -167,6 +168,29 @@ let package = Package(
swiftSettings: strictConcurrency
),

// -- Biofeedback egress (OSC + virtual MIDI out) -------------------
// ADR-011. Turns per-channel band powers, channel health and the
// Track A events into health-gated OSC and MIDI for DAWs and
// patchers (Ableton via Max for Live or MIDI Learn, Logic, Reaper,
// Bitwig, Max, TouchDesigner, SuperCollider). Depends on BCICore
// (frames, health) and BCIEEG (OSCMessage + OSCEncoder) only; no
// MLX, no third-party deps. Network (UDP to a user-chosen host,
// loopback by default) and CoreMIDI (in-machine) are system
// frameworks behind `canImport`. This is a scoped network-egress
// exception after BCICloudBridge and BCIPeerBridge, and the only one
// that emits derived EEG features; decision_registry.md entry 10.
// Imported only by NeuralComposeApp; must never become a dependency
// of any other BCI* target.
.target(
name: "BCIBiofeedback",
dependencies: ["BCICore", "BCIEEG"],
path: "Sources/BCIBiofeedback",
swiftSettings: strictConcurrency,
linkerSettings: [
.linkedFramework("CoreMIDI", .when(platforms: [.macOS])),
]
),

// ── World Model demo (synthetic-task JEPA+MPC, CoreML/ANE) ────────
// Deliberately NOT a `BCI*`-prefixed name: every `BCI*` target is a
// real production-pipeline stage; this is a synthetic-task research
Expand Down Expand Up @@ -364,6 +388,13 @@ let package = Package(
dependencies: ["BCIPeerBridge", "BCICore"],
path: "Tests/BCIPeerBridgeTests"
),
.testTarget(
name: "BCIBiofeedbackTests",
// BCIEEG for the OSCMessage type and MuseOSCDecoder, which the
// encoder round-trip and mirror tests decode against.
dependencies: ["BCIBiofeedback", "BCICore", "BCIEEG"],
path: "Tests/BCIBiofeedbackTests"
),
.testTarget(
name: "WorldModelDemoTests",
dependencies: ["WorldModelDemo", "BCICore"],
Expand Down
183 changes: 183 additions & 0 deletions Sources/BCIBiofeedback/BandPowerEstimator.swift
Original file line number Diff line number Diff line change
@@ -0,0 +1,183 @@
import Foundation
import BCICore

/// Per-channel band powers for one window, in linear units (µV²).
///
/// `bels(_:)` is the Mind Monitor convention: the base-10 logarithm of
/// the band power, so 0 Bel is 1 µV² and each Bel is a factor of ten.
/// Mind Monitor's own absolute values are computed from its PSD scaling
/// and will not match these numerically; what matches is the unit, the
/// band edges and the address scheme, which is all a downstream patch
/// can depend on.
public struct BandPowers: Sendable, Hashable {
public let power: [EEGBand: Float]

public init(power: [EEGBand: Float]) {
self.power = power
}

public subscript(band: EEGBand) -> Float {
power[band] ?? 0
}

/// Sum of all band powers present.
public var total: Float {
power.values.reduce(0, +)
}

/// Fraction of total band power in `band`, 0 when the total is 0.
public func relative(_ band: EEGBand) -> Float {
let t = total
return t > 0 ? self[band] / t : 0
}

/// log10 of the band power, floored so a silent channel reads as a
/// large negative number instead of -inf.
public func bels(_ band: EEGBand) -> Float {
BandPowers.bels(self[band])
}

public static let powerFloor: Float = 1e-9

public static func bels(_ linearPower: Float) -> Float {
log10(max(linearPower, powerFloor))
}
}

/// Welch-style single-segment periodogram band power estimator.
///
/// Pure Swift, deterministic, no Accelerate: the same bytes in give the
/// same floats out on every machine, which is what the replay fixtures
/// in `BCIBiofeedbackTests` and the golden-recording regression need.
/// If this ever shows up in a profile, a vDSP path can be added behind
/// the same signature with the fixtures as its acceptance test.
///
/// Method, per channel: subtract the mean (Muse carries an ~800 µV DC
/// offset), apply a Hann window, zero-pad to the next power of two, FFT,
/// one-sided PSD scaled by `2 / (fs * sum(w^2))` so that the integral of
/// the PSD equals the signal's mean-square value, then sum the PSD bins
/// inside each band's half-open range times the bin width. A pure
/// sinusoid of amplitude A therefore reports a band power of about A²/2.
///
/// This is a different estimator from `FeatureExtractor.bandEnergies`,
/// on purpose: that one is a lag-difference proxy summed over channels
/// for the mock classifier; this one is per channel and in physical
/// units, because it is what leaves the machine.
public struct BandPowerEstimator: Sendable {

public let bands: [EEGBand]

public init(bands: [EEGBand] = EEGBand.allCases) {
self.bands = bands
}

/// Band powers for each channel of `window`. `channels` names the
/// window's rows in order; rows beyond `channels.count` are ignored
/// and channels beyond the row count are omitted from the result.
public func powers(for window: EEGWindow, channels: [EEGChannel] = EEGChannel.allCases) -> [EEGChannel: BandPowers] {
var result: [EEGChannel: BandPowers] = [:]
for (index, channel) in channels.enumerated() where index < window.samples.count {
result[channel] = powers(for: window.samples[index], sampleRate: window.sampleRate)
}
return result
}

/// Band powers for one channel's samples.
public func powers(for samples: [Float], sampleRate: Double) -> BandPowers {
guard samples.count >= 4, sampleRate > 0 else {
return BandPowers(power: Dictionary(uniqueKeysWithValues: bands.map { ($0, 0) }))
}

// Demean.
var mean = 0.0
for v in samples { mean += Double(v) }
mean /= Double(samples.count)

// Hann window and its energy.
let n = samples.count
var windowed = [Double](repeating: 0, count: n)
var windowEnergy = 0.0
for i in 0..<n {
let w = 0.5 - 0.5 * cos(2.0 * Double.pi * Double(i) / Double(n - 1))
windowed[i] = (Double(samples[i]) - mean) * w
windowEnergy += w * w
}

// Zero-pad to a power of two.
var fftSize = 1
while fftSize < n { fftSize <<= 1 }
var re = windowed + [Double](repeating: 0, count: fftSize - n)
var im = [Double](repeating: 0, count: fftSize)
BandPowerEstimator.fft(&re, &im)

// One-sided PSD, bins 1 ... fftSize/2 - 1 (DC and Nyquist excluded;
// DC is zero after demeaning and no band reaches Nyquist).
let binWidth = sampleRate / Double(fftSize)
let scale = 2.0 / (sampleRate * windowEnergy)
var power: [EEGBand: Float] = [:]
for band in bands {
var acc = 0.0
let lowBin = Int((band.lowHz / binWidth).rounded(.up))
let highBin = Int((band.highHz / binWidth).rounded(.up)) // exclusive
let upper = min(highBin, fftSize / 2)
if lowBin < upper {
for k in max(lowBin, 1)..<upper {
acc += (re[k] * re[k] + im[k] * im[k]) * scale
}
}
power[band] = Float(acc * binWidth)
}
return BandPowers(power: power)
}

/// In-place iterative radix-2 Cooley-Tukey FFT. `re.count` must be a
/// power of two and equal to `im.count`.
static func fft(_ re: inout [Double], _ im: inout [Double]) {
let n = re.count
precondition(n == im.count && n > 0 && (n & (n - 1)) == 0, "fft size must be a power of two")

// Bit-reversal permutation.
var j = 0
for i in 1..<n {
var bit = n >> 1
while j & bit != 0 {
j ^= bit
bit >>= 1
}
j |= bit
if i < j {
re.swapAt(i, j)
im.swapAt(i, j)
}
}

// Butterflies.
var length = 2
while length <= n {
let angle = -2.0 * Double.pi / Double(length)
let wRe = cos(angle)
let wIm = sin(angle)
var start = 0
while start < n {
var curRe = 1.0
var curIm = 0.0
let half = length / 2
for k in 0..<half {
let a = start + k
let b = a + half
let tRe = re[b] * curRe - im[b] * curIm
let tIm = re[b] * curIm + im[b] * curRe
re[b] = re[a] - tRe
im[b] = im[a] - tIm
re[a] += tRe
im[a] += tIm
let nextRe = curRe * wRe - curIm * wIm
curIm = curRe * wIm + curIm * wRe
curRe = nextRe
}
start += length
}
length <<= 1
}
}
}
Loading
Loading