bledev.cpble: the CircuitPython backend over _bleio

docs/bledev-internals.md

@@ -1,7 +1,7 @@
 # bledev internals: writing a backend
 
 This page is for you if you're adding a backend (`bledev.bleak`,
-`bledev.webble`, later `bledev.cpble`) or changing one. What an app can rely
+`bledev.webble`, `bledev.cpble`) or changing one. What an app can rely
 on is in [bledev.md](bledev.md#the-rules-every-backend-keeps); this is how a
 backend delivers it.
 
@@ -16,6 +16,7 @@ standalone module per backend, and an optional selector.
 | `fake.py` | In-process loopback. The reference implementation of every hook. |
 | `mpble.py` | MicroPython, over aioble. |
 | `bleak.py` | CPython on Windows, Linux and macOS, over bleak. Central only. |
+| `cpble.py` | CircuitPython, over `_bleio`. Both roles. |
 | `nus.py` | Nordic UART byte stream, built only on the contract. |
 | `repl.py` | The REPL over nus: MicroPython's raw `bluetooth` API from the IRQ on the board, `nus` on the client. |
 | `improv.py` | Improv Wi-Fi setup, both sides, built only on the contract. |
@@ -227,6 +228,77 @@ bounds the setup at 4 s and tries again, up to three times. `nus.connect()`
 and `midi.connect()` use it, and the contract checks both the retry and the
 giving up.
 
+## How cpble gets there
+
+`_bleio` has no events and no async calls. An adapter hands out objects you
+look at: `adapter.connections`, `connection.connected`, and the ring buffers
+that a characteristic's writes and notifications land in. So every wait in
+cpble is a poll of one of those every `POLL_MS` (5 ms), and each blocking
+`_bleio` call (connect, discovery, a read, a write with response, making a
+subscription) holds the event loop until the radio answers, which is one or
+two connection intervals. A scan runs in slices of `SCAN_SLICE_MS` (100 ms),
+each a fresh `start_scan()`, with the scan response merged per address the
+way aioble does.
+
+**Where writes land.** A characteristic with notify or indicate gets a
+`_bleio.PacketBuffer`, which keeps each write whole but takes writes only from
+the central that subscribed, and only after it did (it learns the connection
+from the subscribe event). Any other writable characteristic gets a
+`_bleio.CharacteristicBuffer`, a byte stream: with `capture=True`, each
+`written()` returns what arrived since the last one. Both drop the oldest data
+when full and don't say so; cpble checks the stream buffer's fill and raises
+when it's full, and sizes it at 32 KB (`stream_buffer`), which is nus's whole
+16 KB echo phase in flight twice over. `_bleio` doesn't say which central
+wrote, so `written()` names the most recent live one. Every server value
+buffer is `_bleio`'s largest (512 bytes), because a notification can't be
+longer than the value's buffer; `max_len` is kept in Python (`read()` gives
+the first `max_len` bytes) rather than by `_bleio` refusing the write.
+
+**Notifications, and a stall in CircuitPython 10.3.** `PacketBuffer.write()`
+on the ESP32 keeps a packet it couldn't get an mbuf for and retries it on the
+next `BLE_GAP_EVENT_NOTIFY_TX`. NimBLE sends that event when a notification is
+handed to the stack, not when it leaves the radio, so a packet that never got
+an mbuf never produces one, and the next write that doesn't fit beside it
+waits inside `_bleio` until the central disconnects. Writing 253-byte packets
+back to back from the laptop gate delivered 10 (2,530 bytes) and stopped,
+every time. So cpble passes each notification as the `header=` of an empty
+write: `_bleio` copies a header only into an empty packet, and a write of
+nothing never waits, but it does retry what's pending. A return of 0 means
+the previous packet is still there, and `notify()` raises `BusyError`; nus,
+midi and the other async senders wait and try again. A task retries the last
+packet every 5 ms for 5 s after it was sent, since nothing else will. It
+also keeps notifications whole, where `PacketBuffer` would append the next
+write to a pending packet.
+
+**And a hard fault.** When NimBLE refuses a notification after the mbuf was
+allocated (it needs one more for the ATT header), `PacketBuffer` "undoes" by
+switching to a second outgoing buffer that it allocates only for indications
+and writes, and the next write copies into a null pointer: the board restarts
+in safe mode. cpble can make it rarer but not impossible: notifications are
+held to 244 bytes (the reported MTU is at most 247, one link-layer packet, so
+NimBLE never fragments), and each new connection gets fresh packet buffers.
+The fix is in CircuitPython; the draft for upstream, with the patch, is
+[upstream-reports/cp-packetbuffer-notify-stall.md](upstream-reports/cp-packetbuffer-notify-stall.md).
+
+**Discovery mode, without a hand.** CircuitPython's own file service
+advertises to a new host only in discovery mode, which normally takes a reset
+pressed during the blue blink after boot. The supervisor marks that second in
+an RTC register that survives a reset, and `memorymap` can write that register
+on the S3 (`RTC_CNTL_STORE0_REG`, 0x60008050), so
+`tests/bledev_board/cpfiles_discovery.py` writes the mark and resets. It also
+erases the board's bonds, as the button would.
+
+**What `_bleio` can't do here**, on the ESP32 port: a central can't receive
+indications (they're ignored below Python); pairing is "just works" only
+(`authenticated=True` raises); there are no long reads; the MTU is fixed at
+256 when CircuitPython is built; a service has at most 10 characteristics and
+a characteristic at most 2 descriptors; a peripheral doesn't learn its
+central's address; `Service.deinit()` is the only way to take a service away,
+which `register_services()` uses; and a read that times out inside `_bleio`
+returns a full buffer as if it succeeded. A refused read or write comes back as
+`BluetoothError("Unknown system firmware error: 261")`, NimBLE's 0x100 plus
+the ATT code, and cpble turns that into `GattError(5)`.
+
 ## How the REPL gets there
 
 The REPL has to work when nothing runs asyncio, at the `>>>` prompt, so the
@@ -689,6 +761,59 @@ KB/s), overwriting the same file each time. The planted bit flip on the client
 failed the round-trip check. mpftp's `ble://` transport, and its gate, are in
 [PyDevices/mpftp](https://github.com/PyDevices/mpftp) `docs/plans/ble.md`.
 
+### CircuitPython (cpble)
+
+The LilyGO T-Embed (ESP32-S3) on CircuitPython 10.3.0, bledev as `.py`
+source, against Windows 11 and bleak 3.0.2 one desk apart, 2026-09-24.
+"Official" is circuitpython.org's build; "patched" is the same tag with
+[upstream-reports/cp-packetbuffer-notify-stall.patch](upstream-reports/cp-packetbuffer-notify-stall.patch),
+built here because the official build can't carry a notification stream
+reliably (below).
+
+| Gate | Official 10.3.0 | Patched |
+|---|---|---|
+| nus, 16 KB up, down and echoed (`nus_server.py` on the board) | 9 of 11 passed; 2 restarted the board in safe mode (hard fault) | 12 of 12 |
+| nus throughput, Windows' defaults | 50-90 KB/s up, 31-49 down, 21-37 echo | 50-91 up, 25-49 down, 16-28 echo |
+| nus, `--fast` (throughput parameters) | | 88 up, 44 down, 34 echo |
+| the GATT contract, laptop central (`gatt_central.py --mtu 256`) | pass | 5 of 5 |
+| BLE-MIDI, 1,000 messages each way | pass (0.8 s up, 0.7 s down) | pass |
+| CircuitPython's own file service, 20 KB up and down (`cpfiles_client.py`) | 0 of 2: the download stalls | 7 of 7 round trips in 3 sessions |
+| MicroPython LCD-7 central, CircuitPython serving nus | | pass: 90 up, 37 down, 25 echo |
+
+Every planted fault failed its gate: a bit flipped in the nus stream (either
+side), notification 7 skipped by the contract's peripheral, the three BLE-MIDI
+plants (a flip going up, a dropped and a flipped packet going down), and a bit
+flipped in the file data read back.
+
+**BLE-MIDI latency** against the T-Embed, half a note-on's round trip, 400
+notes: a median of 52.1 ms (p99 103.2) at Windows' defaults and 13.3 ms (p99
+23.4) with throughput parameters, the same as against a MicroPython S3
+(52.3 and 13.4). Polling every 5 ms costs nothing you can see there.
+
+**CircuitPython's file service.** The client paired by itself in 2.1-3.2 s,
+reconnected from the bond with no pairing in 1.0-1.7 s, and moved 20 KB up in
+4.6-10.1 s (2.0-4.4 KB/s: CircuitPython writes flash between 512-byte
+windows) and down in 0.8-1.8 s (11-26 KB/s). Two conditions, both
+CircuitPython's design: the service writes nothing while a computer has the
+CIRCUITPY drive mounted (USB takes the filesystem's lock as soon as the host
+asks whether it may write; `storage.disable_usb_drive()` in `boot.py` for the
+gate), and a command that arrives while a write's automatic reload is
+restarting `code.py` isn't answered until another command arrives
+(`supervisor.runtime.autoreload = False` in `code.py` for the gate). mkdir
+there makes one level, where bledev's server makes the parents too.
+
+**CircuitPython as central**, against the LCD-7 serving nus: found, connected,
+discovered, subscribed and wrote, with every write with a response (making the
+subscription is one) taking 2 s, because CircuitPython's ESP32 port waits out
+its whole timeout for a status to leave 0, and success is 0. The full nus gate
+in that direction didn't run; see ble.md.
+
+**mpftp on CircuitPython**, noticed on the way: a long `exec` or `run` sent
+over the serial REPL arrived garbled now and then (a base64 chunk of 6,000
+characters decoded with 11 wrong bytes, twice), and `put` with the drive
+disabled failed with a syntax error. Copying through the CIRCUITPY drive and
+`exec`-ing `import script` was reliable.
+
 ### Pairing
 
 The laptop (Windows 11, bleak 3.0.2, WinRT custom pairing) and the LCD-7

docs/bledev.md

@@ -31,11 +31,12 @@ transport can talk to it too.
 | Backend | Host | Central (scan, connect) | Peripheral (advertise, serve) |
 |---|---|---|---|
 | `bledev.mpble` | MicroPython with `bluetooth` (ESP32, S3, P4 + C6, Pico W) | yes | yes |
+| `bledev.cpble` | CircuitPython with `_bleio` (ESP32-S3 measured) | yes | yes |
 | `bledev.bleak` | CPython on Windows, Linux, macOS | yes | no |
 | `bledev.webble` | Browsers: PyScript, the Workbench simulator | yes | no |
 | `bledev.fake` | Anywhere: tests | yes | yes |
 
-`bledev.webble` is being written; the other three are here.
+`bledev.webble` is being written; the other four are here.
 
 Roles aren't symmetric. A laptop or a browser can only scan and connect, so
 when a board talks to one, the board advertises. Only board-to-board links put
@@ -86,11 +87,18 @@ import bledev.bleak
 ble = bledev.bleak.BleakBLE()   # central only
 ```
 
+On CircuitPython (it needs the `asyncio` library: `circup install asyncio`):
+
+```python
+import bledev.cpble
+ble = bledev.cpble.get()        # CircuitPython's _bleio.adapter, both roles
+```
+
 Anywhere, if you don't want to name the backend:
 
 ```python
 import bledev.auto
-ble = bledev.auto.adapter()     # mpble, webble or bleak, whichever this host has
+ble = bledev.auto.adapter()     # mpble, cpble, webble or bleak, whichever this host has
 ```
 
 `bledev.auto` is optional, and nothing else in bledev imports it. Set
@@ -438,6 +446,37 @@ USB host can use it the same way. How it numbers axes, what it does with a
 rollover, and what a board central can't read are in
 [bledev-internals.md](bledev-internals.md#how-hid-gets-there).
 
+## On CircuitPython
+
+The same code runs on a CircuitPython board with `bledev.cpble`, in either
+role: nus, MIDI, the contract's checks and the file client all passed over the
+radio there (the numbers are in [the internals](bledev-internals.md#circuitpython-cpble)). A few
+things differ, because `_bleio` works differently from aioble underneath:
+
+- Connecting, discovery, a read, a write with response and subscribing each
+  hold up your other tasks until the radio answers, usually a connection
+  interval or two. `_bleio` has no async calls.
+- It pairs "just works" only. `authenticated=True` and passkeys raise
+  `UnsupportedError`, and it always bonds.
+- A central there can't receive indications, and the MTU is 247 at most.
+- `Characteristic.write()` on a characteristic that notifies sends the value
+  to subscribed centrals even without `send_update`.
+- CircuitPython 10.3's `PacketBuffer` can stall or fault the board when a
+  notification stream outruns the radio. bledev steps around the stall; the
+  fault needs a CircuitPython fix, drafted for upstream with a patch. On the
+  official build, 2 of 11 fast 16 KB streams restarted the board in safe mode;
+  on the patched build, none did.
+- A central there waits 2 s on every read and every write with a response
+  (subscribing is one), so connecting and setting up takes several seconds.
+
+CircuitPython's own BLE file service, the one its supervisor serves, is
+reached with the same `bledev.filetransfer.connect()` a MicroPython board
+uses; it pairs by itself. The service is only public in CircuitPython's
+discovery mode, it won't write while a computer has the CIRCUITPY drive
+mounted, and on the official 10.3.0 build a 20 KB download stalls (the same
+`PacketBuffer` fault). What cpble does about each of these, and why, is in
+[bledev-internals.md](bledev-internals.md#how-cpble-gets-there).
+
 ## The rules every backend keeps
 
 These hold on every backend. The fake enforces the strict version of each, so

docs/upstream-reports/cp-packetbuffer-notify-stall.md

@@ -0,0 +1,89 @@
+# Draft: `_bleio.PacketBuffer` on the ESP32 stalls when NimBLE's buffers run dry, and can hard-fault after a refused notification
+
+**Target:** adafruit/circuitpython, an issue with a PR (the fix is small and
+local to `ports/espressif/common-hal/_bleio/PacketBuffer.c`).
+**Status:** ready to file, awaiting Brad's word. Post only what is below the
+`---`. The patch is `cp-packetbuffer-notify-stall.patch` beside this file,
+against tag 10.3.0. The upstream-emissary agent can turn it into a branch
+and a PR body; nothing has been pushed.
+
+Found while building `bledev.cpble` (pydevices, `lib/bledev/cpble.py`), which
+works around the stall on unpatched firmware (it writes each notification as
+the `header=` of an empty write, which never waits). The workaround doesn't
+avoid the hard fault; only the fix does. The measurements are in
+`docs/bledev-internals.md`, "How cpble gets there".
+
+Verified on a LilyGO T-Embed (ESP32-S3), CircuitPython 10.3.0, the official
+build and the same tag with this patch, against Windows 11 through bleak
+3.0.2, 2026-09-24.
+
+---
+
+### `_bleio.PacketBuffer` on the ESP32: notifications stall for good when NimBLE's mbufs run out, and a refused notification can hard-fault the next write
+
+**CircuitPython 10.3.0, ESP32-S3 (espressif port, NimBLE).** Two faults in
+`ports/espressif/common-hal/_bleio/PacketBuffer.c`, both reachable from the
+BLE file-transfer service without any user code.
+
+**1. A stall.** `queue_next_write()` keeps the pending packet when
+`ble_hs_mbuf_from_flat()` returns `NULL`, expecting the next
+`BLE_GAP_EVENT_NOTIFY_TX` to retry it. For a notification that event fires
+synchronously inside `ble_gatts_notify_custom()`, when the packet is handed
+to the stack (`ble_gattc.c`, `ble_gap_notify_tx_event(rc, conn_handle,
+chr_val_handle, 0)`), not when it leaves the radio. A packet that never got an
+mbuf never produces one, so nothing retries it. The next
+`common_hal_bleio_packet_buffer_write()` that doesn't fit beside it then waits
+in its `while (self->pending_size != 0 ...)` loop until the central
+disconnects.
+
+Seen three ways:
+
+- A user `PacketBuffer` on a notify characteristic: 64 writes of 253 bytes
+  back to back delivered 10 notifications (2,530 bytes), then nothing, every
+  time, until the central gave up.
+- Half-packet writes with an empty `write(b"")` before each (which does retry
+  the pending packet): 3,780 and 4,662 bytes, then the same stall.
+- **The BLE file-transfer service**, paired, listing `/` on a board with a
+  few files: the listing stops part way through the fourth entry (18
+  notifications, because `_process_listdir` writes the path in small pieces),
+  and every later command times out until the host disconnects.
+
+**2. A hard fault.** When `ble_gatts_notify_custom()` itself fails (it
+allocates another mbuf for the ATT header, `ble_att_cmd_get()`, so it can fail
+with `BLE_HS_ENOMEM` right after `ble_hs_mbuf_from_flat()` succeeded), the
+"undo" path flips `pending_index`. Notifications only ever allocate
+`outgoing[0]` (`outgoing2` is allocated only for `CHAR_PROP_WRITE` and
+`CHAR_PROP_INDICATE`), so the next write `memcpy`s into `outgoing[1] ==
+NULL`. The same path also sets `pending_size` back and then zeroes it three
+lines later, so the refused packet is lost as well. On the T-Embed this
+restarted the board in safe mode (`SafeModeReason.HARD_FAULT`) in 2 of 11
+16 KB notify-heavy runs.
+
+**The fix** (patch attached, `PacketBuffer.c` and `PacketBuffer.h`):
+
+- In `queue_next_write()`, on any failure, keep the packet pending (don't
+  zero `pending_size`), and flip `pending_index` back only for indications,
+  which are the only kind that flipped it.
+- Retry a kept packet from a background callback, rescheduled while it keeps
+  failing, since no BLE event will say the buffers are free. Without this, the
+  last packet of a response sits pending until the next command's reply
+  pushes it out, so the host sees each answer one command late. Also retry in
+  `common_hal_bleio_packet_buffer_write()`'s wait loop and in
+  `common_hal_bleio_packet_buffer_flush()`.
+- Drop a pending packet when its central unsubscribes or disconnects, so a
+  kept packet can't reach the next central.
+
+With it (same board, same tests): the 16 KB notify stream passed 12 of 12
+with no safe-mode restart, and the file service listed `/` in 50 ms and moved
+20 KB up and down byte for byte in 7 of 7 round trips, where the official
+build stalled on the download 2 of 2 times.
+
+Not addressed here, noticed on the way: `bleio_packet_buffer_extend()` drops
+the oldest packets when its ring is full, with a `// set an overflow flag?`
+comment; `bleio_gattc_read()` reports a timeout as success with the whole
+buffer's length; `bleio_gattc_read()` and `bleio_gattc_write()` always wait out the whole
+2 s, because `_wait_for_completion()` waits for `_completion_status` to leave
+0 and a success sets it to 0; and ATT errors on reads and writes reach Python as
+`BluetoothError("Unknown system firmware error: 261")` rather than
+`SecurityError`, because `_wait_for_completion()` is checked with
+`CHECK_NIMBLE_ERROR` instead of `CHECK_BLE_ERROR`.