bledev.midi: BLE-MIDI as a MIDI port, with a portable codec

docs/bledev-internals.md

@@ -19,6 +19,8 @@ standalone module per backend, and an optional selector.
 | `nus.py` | Nordic UART byte stream, built only on the contract. |
 | `repl.py` | The REPL over nus: MicroPython's raw `bluetooth` API from the IRQ on the board, `nus` on the client. |
 | `improv.py` | Improv Wi-Fi setup, both sides, built only on the contract. |
+| `midi.py` | BLE-MIDI as a usbif-style MIDI port, both sides, built only on the contract. |
+| `midi_codec.py` | The BLE-MIDI packet codec. Pure, no imports, shared by every host. |
 | `auto.py` | Picks a backend. Nothing imports it, and backends must not. |
 
 A backend module imports `bledev` and whatever its host provides, and nothing
@@ -181,6 +183,27 @@ no difference to nus throughput.
 After a disconnect bleak sets `client.services` to `None`, so every client call
 checks the connection first and raises `DisconnectedError`.
 
+## Reconnecting fast
+
+The REPL gate once failed one run in five: bleak couldn't subscribe on a
+connection made 1.5 s after the last one closed. A loop of rapid reconnects
+from the laptop (`tests/bledev_board/reconnect_loop.py`: log in, run a line,
+close, wait 1.5 s) caught it at about one cycle in fifty, always the same
+way. bleak's connect returned in 70-440 ms where a real one takes 1.7 s or
+more, Windows reported the device connected and the GATT session active, the
+board never saw a connection (its IRQ log shows it advertising throughout),
+and the first GATT operation hung until Windows reported a drop about nine
+seconds later.
+
+It went away with the REPL's output fixes (below): against `bledev-host`'s
+`repl.py` the loop failed 7 times in 465 cycles; against the fixed one, 0 in
+270, 150 of them with the retry switched off. We haven't found how the
+board's old timer handling produced a connection Windows believed in and the
+board never saw, so `bledev.connect_and_set_up()` stays as the defence: it
+bounds the setup at 4 s and tries again, up to three times. `nus.connect()`
+and `midi.connect()` use it, and the contract checks both the retry and the
+giving up.
+
 ## How the REPL gets there
 
 The REPL has to work when nothing runs asyncio, at the `>>>` prompt, so the
@@ -190,9 +213,26 @@ with `gap_advertise`, and does everything else from an IRQ handler it adds to
 aioble's dispatcher (or sets itself, with no aioble). `os.dupterm` reads a
 `_Stream` whose `readinto` returns `None` when empty, because 0 means end of
 stream and detaches it. Output goes out as notifications of `mtu - 3` bytes,
-and what the controller can't take yet is retried from a one-shot soft timer.
-A program printing faster than the link carries waits in `write()` up to two
-seconds, then drops the excess, because `print()` can't fail.
+and what the controller can't take yet is retried every 5 ms by one periodic
+`machine.Timer`. A program printing faster than the link carries waits in
+`write()` up to two seconds, then drops the excess, because `print()` can't
+fail.
+
+Two ways that retry went wrong, both found by printing a 16,890-character
+list over the link (the REPL gate now does this):
+
+- **A doubled chunk and a lost one.** The timer's callback runs between the
+  main program's bytecodes, so it could land inside a `flush()` that
+  `print()` had started: both sent the same chunk, then both trimmed it, and
+  the next chunk never went out. Five prints out of five came back corrupt.
+  `flush()` now returns at once if it's already running.
+- **A panic.** The retry used to be a one-shot timer re-armed on every write.
+  On the esp32, `Timer.init()` on a virtual timer whose alarm is being
+  dispatched clears its handler first, so `esp_timer`'s task on the other
+  core calls a NULL function (`InstrFetchProhibited` in
+  `timer_process_alarm`). It took the board down twice in about 45 long
+  prints. The timer is now started once and never re-armed; 60 long prints
+  afterwards, no panic and every byte intact.
 
 Ctrl-C works because the IRQ calls `os.dupterm_notify()` whenever a write
 contains 0x03; dupterm then reads it and raises `KeyboardInterrupt`, even in
@@ -234,6 +274,8 @@ The serving scripts have a `PLANT` switch, and a planted run must fail:
 | `gatt_peripheral.py` + `gatt_latency.py` | Per-operation latency of reads and writes-with-response, with the stall count |
 | `nus_server.py` + `nus_client.py` | 16 KB up, down and echoed over nus, byte for byte, timed. On a laptop, `--fast` asks for throughput parameters and `--plant` flips a bit |
 | `repl_server.py` + `repl_client.py` | The REPL: the right password evaluates `123 * 456`, a wrong one sent with code that would create `/pwned` is refused and hung up on and the code never runs, and Ctrl-C stops `while True`. `repl_server.py` runs from `/main.py`, because mpftp soft-resets the board, which turns Bluetooth off |
+| `midi_server.py` + `midi_client.py` | BLE-MIDI. `gate`: 1,000 mixed messages up, checked by the server, then down, checked by the client; `--plant` on the client and `PLANT = "drop"` or `"flip"` on the server must FAIL. `latency`: note-on round trips through an echo, `--fast` for a 7.5 ms interval (throughput parameters on a laptop). A board client reads its arguments from `/midi_client_args.py` |
+| `reconnect_loop.py` (laptop) against `repl_server.py` | Rapid reconnects: log in, run a line, close, wait 1.5 s, N times; `--no-retry` turns `connect_and_set_up`'s retry off |
 | `improv_server.py` + `improv_client.py` | Improv: `--wrong` must get "unable to connect" and a return to "authorized"; without it, on a board, the network comes from that board's own `secrets.py` and must end "provisioned" with a URL |
 | `coex_server.py` + `tcp_pull.py` + `nus_client.py` | Wi-Fi and BLE on one S3: a TCP source on Wi-Fi beside the nus gate |
 
@@ -323,3 +365,48 @@ One early run with BLE advertising read 8.1 KB/s, which looked like a finding
 until the repeats. Measuring BLE's cost to Wi-Fi needs the board at -60 dBm
 or better.
 
+### BLE-MIDI
+
+The LCD-7 ran `midi_server.py` and the T-Embed or the laptop ran
+`midi_client.py`, one desk apart, Wi-Fi off, bledev as `.py` source,
+2026-09-24.
+
+**The gate** passed board to board at both intervals and from the laptop:
+1,000 messages each way, complete, in order and intact, no decode errors, in
+about 0.7 s up and 0.5 s down. The planted runs failed as they must: a bit
+flipped in one packet going up (the server reported the first bad message,
+546), one packet dropped going down (940 of 1,000 arrived), and a bit flipped
+going down.
+
+**Latency**, half a note-on's round trip, each ping sent 0-20 ms after the
+last echo so the phase against the connection events varies:
+
+| Link | Pings | Median | p90 | p99 | Max |
+|---|---|---|---|---|---|
+| S3 to S3, default interval (MicroPython asks 30-50 ms) | 400 notes | 44.3 ms | 44.3 | 49.8 | 69.3 |
+| the same, four-note chords (to the last note) | 100 | 44.4 ms | 44.4 | 49.7 | 49.7 |
+| S3 to S3, 7.5 ms interval | 400 notes | 10.8 ms | 13.6 | 15.9 | 21.1 |
+| the same, chords | 100 | 10.9 ms | 13.7 | 18.8 | 18.8 |
+| Laptop to S3, Windows' defaults | 400 notes | 52.3 ms | 60.0 | 110.7 | 118.1 |
+| the same, chords | 100 | 52.1 ms | 59.8 | 105.7 | 105.7 |
+| Laptop to S3, throughput parameters | 400 notes | 13.4 ms | 16.7 | 23.1 | 24.7 |
+| the same, chords | 100 | 13.3 ms | 16.5 | 23.2 | 23.2 |
+
+No echo was lost in any run. **The codec's own cost** on the S3 (240 MHz,
+`.py` source), splitting, encoding and decoding: 0.51 ms for a note, 1.27 ms
+for a four-note chord, 4.9 ms for a 300-byte SysEx. A round trip crosses the
+codec four times, so it accounts for about 1 ms of the 10.8 ms one way at
+7.5 ms. What the rest splits into, connection events against asyncio's
+scheduling on each board, wasn't measured: that needs a raw GATT echo at the
+same interval to compare.
+
+**Windows** (2026-09-24, Windows 11 build 26200): once the board is paired,
+Windows lists it as a MIDI device, `bledev-midi (Bluetooth MIDI IN)` and
+`(Bluetooth MIDI OUT)`, found through the WinRT MIDI device interfaces. It
+takes pairing: unpaired, nothing appears. winmm doesn't list it, and this
+build's Windows MIDI Services has no Bluetooth transport among its
+`Midi2.*Transport.dll`s, so a DAW that opens ports through winmm can't see
+the board. Pairing needs the board to bond (`ble.config(bond=True, ...)` with
+aioble's `security` module loaded); bledev doesn't do that for you yet.
+Before `midi.serve()` set the GAP name, Windows called the port `MPY ESP32`.
+

docs/bledev.md

@@ -223,6 +223,52 @@ boot is up to you: pass `on_join=lambda ssid, password, ip: ...` to `serve()`.
 (from a button, say) before it accepts credentials. The credentials cross the
 air unencrypted; that's the standard.
 
+## MIDI
+
+`bledev.midi` is BLE-MIDI, the standard every Mac, iPhone, Android phone,
+DAW and Bluetooth MIDI controller speaks. The board advertises it; anything
+else connects:
+
+```python
+import bledev.midi as midi
+
+port = await midi.serve(ble, name="synth")      # on the board
+port = await midi.connect(ble, name="synth")    # on a laptop or another board
+```
+
+What you get is a MIDI port, the same one usbif gives you for a USB MIDI
+function: `port.write(b"\x90\x3c\x64")` sends a note, and `port.read(buf)`
+returns whatever MIDI bytes have arrived, without blocking. So the loop you
+wrote for a USB controller works on a BLE one:
+
+```python
+buf = bytearray(64)
+parser = usbif.MidiParser()                     # running status, SysEx, clock
+while True:
+    n = port.read(buf)
+    parser.feed(buf, n)
+    for status, data in parser.drain():
+        ...
+    await asyncio.sleep(0.001)
+```
+
+When usbif is installed, `port` is a `usbif.MidiPort`. If you'd rather have
+messages than bytes, with the sender's timestamp, use `ts, message = await
+port.receive()` or `async for ts, message in port`. Messages you write go out
+from a task, packed as many to a packet as fit, and `await port.drain()` waits
+for them.
+
+On a board, ask for the shortest connection interval when you connect:
+`midi.connect(ble, name="synth", min_conn_interval_us=7500,
+max_conn_interval_us=7500)`. The latency that buys is in
+[the measurements](#measured-on-two-esp32-s3s).
+
+The packet format lives in `bledev.midi_codec`, pure Python with no
+imports: timestamps, running status, SysEx split across packets, and clock
+slipped into a SysEx. Its test vectors (`tests/bledev_midi_vectors.json`) were
+worked out by hand from the specification, and every interpreter checks
+against the same file.
+
 ## The rules every backend keeps
 
 These hold on every backend. The fake enforces the strict version of each, so
@@ -310,6 +356,23 @@ two minutes. Turning the radio on costs about 64 KB of the S3's internal RAM;
 aioble and bledev together cost about 50 KB of Python heap. The details, and
 how these were measured, are in [bledev-internals.md](bledev-internals.md#measurements).
 
+**MIDI.** How long a note takes to cross the link, one way, as half a
+note-on's round trip through an echo (400 single notes at random moments,
+then 100 four-note chords, which came out the same):
+
+| Link | Median | p90 | p99 | Max |
+|---|---|---|---|---|
+| S3 to S3, default interval | 44.3 ms | 44.3 | 49.8 | 69.3 |
+| S3 to S3, 7.5 ms interval | 10.8 ms | 13.6 | 15.9 | 21.1 |
+| Laptop (bleak) to S3, Windows' defaults | 52.3 ms | 60.0 | 110.7 | 118.1 |
+| Laptop to S3, `priority="throughput"` | 13.4 ms | 16.7 | 23.1 | 24.7 |
+
+Players start to feel delay at about 10 ms, so a MIDI link wants the shortest
+interval, and even then BLE alone uses up that budget. This is BLE's own
+delay; the synth's comes on top. 1,000 mixed messages (notes, controllers,
+pitch bend, a 300-byte SysEx every hundred, clock in between) crossed both
+ways between the boards and to the laptop complete, in order and intact.
+
 ## Writing a backend
 
 A backend subclasses the classes in `bledev/__init__.py` and fills in a few

lib/bledev/__init__.py

@@ -870,3 +870,38 @@ def _want_response(self, response):
 def is_adapter(obj):
     """True for a bledev :class:`BLE`; False for a raw ``bluetooth.BLE`` or anything else."""
     return isinstance(obj, BLE)
+
+
+async def connect_and_set_up(
+    ble, setup, *, name=None, service=None, device=None, timeout_ms=10000, attempts=3, setup_timeout_ms=4000, **options
+):
+    """Find a device, connect, and run ``await setup(connection)``; returns
+    ``(connection, what setup returned)``.
+
+    If the link drops or stalls before ``setup`` finishes, this disconnects
+    and tries again, up to ``attempts`` times. Profiles use it (``nus``,
+    ``midi``), because a link can come up dead: about one reconnect in fifty
+    from Windows, the OS reports a connection the peripheral never saw, and
+    the first GATT operation on it hangs until Windows gives up nine seconds
+    later. ``setup_timeout_ms`` bounds that wait. ``options`` go to
+    ``device.connect()``.
+    """
+    error = None
+    for _ in range(attempts):
+        target = device
+        if target is None:
+            target = await ble.find(name=name, service=service, timeout_ms=timeout_ms)
+        connection = await target.connect(timeout_ms=timeout_ms, **options)
+        try:
+            return connection, await wait_ms(setup(connection), setup_timeout_ms, "setting up the connection")
+        except (DisconnectedError, BLETimeoutError) as e:
+            error = e
+        except BaseException:
+            if connection.is_connected():
+                await connection.disconnect()
+            raise
+        try:
+            await connection.disconnect()
+        except BLEError:
+            pass
+    raise error