bledev.hid: BLE keyboards and gamepads into events, both ways

docs/bledev-internals.md

@@ -19,6 +19,8 @@ standalone module per backend, and an optional selector.
 | `nus.py` | Nordic UART byte stream, built only on the contract. |
 | `repl.py` | The REPL over nus: MicroPython's raw `bluetooth` API from the IRQ on the board, `nus` on the client. |
 | `improv.py` | Improv Wi-Fi setup, both sides, built only on the contract. |
+| `hidreport.py` | HID report descriptors and reports to `events`. No Bluetooth in it. |
+| `hid.py` | HID over GATT, host and device, built on the contract and `hidreport`. |
 | `auto.py` | Picks a backend. Nothing imports it, and backends must not. |
 
 A backend module imports `bledev` and whatever its host provides, and nothing
@@ -72,6 +74,13 @@ returning a list of your `ClientService` in handle order.
 **`ClientService`**: `_discover_characteristics(uuid, timeout_ms)`, a list of
 your `ClientCharacteristic` in handle order.
 
+**Pairing and descriptors** (for HID): `Connection.encrypted` and
+`Connection.pair(bond, timeout_ms)`; `ClientCharacteristic._discover_descriptors(uuid, timeout_ms)`
+returning your `ClientDescriptor`s, whose `read()` you fill in; and on the
+server, each `Characteristic`'s `descriptors` (fixed values) and `encrypted`
+flag. Report `long_read` in `capabilities()`: False if a read returns only
+one packet. A backend without pairing leaves the base's `UnsupportedError`.
+
 **`ClientCharacteristic`**: `read`, `write`, `subscribe`, `notified`,
 `indicated`. `self._want_response(response)` resolves `response=None` and
 checks the flags for you; `self._check(FLAG_..., what)` checks the rest.
@@ -144,6 +153,18 @@ The laptop found two more (2026-09-24):
   case never showed it because the central there asks for the MTU after
   connecting.
 
+The bless probe found two more on the P4 (2026-09-24), with the board as the
+central and Windows as the peripheral:
+
+- **The same early MTU, the other way round.** Windows exchanges the MTU the
+  moment the board connects, before aioble has recorded the connection, so
+  the board believed 23 on a 256 link. mpble's IRQ now keeps that value too,
+  and the connection's `mtu` picks it up.
+- **`exchange_mtu()` after the peer already exchanged** raised `EALREADY`.
+  NimBLE exchanges once per link, so mpble now returns the settled MTU.
+  `bledev.hid`'s host exchanges before reading the Report Map, so any
+  peripheral that exchanges first (a real keyboard may) would have failed.
+
 **Code that uses the raw API.** A board config's `ble` is now an `MPBLE`,
 and `MPBLE.__getattr__` hands every name it doesn't define to the raw
 `bluetooth.BLE`, so `ble.active()` or `ble.gap_advertise()` still work.
@@ -209,6 +230,66 @@ without using up the attempt.
 The ESP32 has one dupterm slot. The REPL takes it on login and gives back
 whatever held it (WebREPL, say) on disconnect.
 
+## How HID gets there
+
+`bledev.hid` is two halves over one parser. `bledev.hidreport` reads a report
+descriptor into `Report`s of `Field`s (bit offset, size, count, flags,
+usages, logical range) and a `Decoder` diffs successive input reports into
+events. It imports `events` and `keys` and nothing from Bluetooth.
+
+**The keyboard decoder is usbif's, event for event.** Same order (modifier
+releases, key releases, key presses, modifier presses, the new modifier mask
+on all of them), same fields (`scancode` is the HID usage, `window` is
+`None`), same rollover rule. `tests/test_bledev_hid.py` feeds 3,000 random
+boot reports through both decoders when usbif's checkout is beside this one,
+and they must agree. It found one difference on its first run: the ISO "# ~"
+key (usage 0x32), which SDL calls `#` and usbif leaves unmapped. hidreport
+now leaves it unmapped too; the two tables should become one.
+
+**Axes, hats and buttons** come only from joystick, gamepad and multi-axis
+collections, so a mouse's X, Y and buttons produce nothing yet. Axes are
+numbered by usage (desktop X to wheel, then the simulation page), so an Xbox
+controller's sticks are 0 to 3 and its triggers 4 and 5. Every axis starts
+at 0.0 and moves only when its value changes, as SDL's do.
+
+**Descriptors, encryption and pairing** were added to the contract for HID.
+A Report characteristic says which report it carries in its Report Reference
+descriptor (0x2908), and HOGP gives the report ID nowhere else, so the host
+needs `ClientCharacteristic.descriptor()`. A `Characteristic` can be
+`encrypted=True`, and `Connection.pair()` encrypts a link. The host pairs
+only after a read fails with insufficient encryption or authentication,
+which is how a real keyboard asks. mpble pairs through aioble's security
+module ("just works", `io=3`); `ble.enable_bonding()` loads aioble's
+`ble_secrets.json` and turns bonding on, and it has to run on both sides,
+because the side that didn't start pairing stores keys too.
+
+**MicroPython has no long read.** `gattc_read()` is one ATT Read, so a board
+central gets at most `mtu - 1` bytes of any value. The host exchanges the MTU
+up to 247 before reading the Report Map (246 bytes fit), and a map that fails
+to parse at that length raises an error that says why. Our own map is 162
+bytes; an Xbox Series controller's is 283, so a board can't host one until
+MicroPython grows a long read. bleak reads long values itself, and the
+fake's `long_reads=False` models the board.
+
+**The peripheral's identity comes from its functions.** Name, appearance and
+PnP product ID are derived from which of keyboard, consumer control and
+gamepad it serves, so a host that cached one set never sees another set
+under the same identity. The PnP vendor is the Bluetooth SIG's test company
+ID (0xFFFF), because we have no vendor ID of our own.
+
+**A laptop can't see a HID service.** Windows hides 0x1812 from apps: bleak
+lists the board's GAP, GATT, Battery and Device Information services and a
+22-handle gap where the HID service sits (measured against the P4,
+2026-09-24). Chrome's Web Bluetooth blocklist hides it too. So
+`Peripheral(service_uuid=hid.INSPECT_SERVICE)` serves the same
+characteristics under a vendor UUID, and `Host(service_uuid=...)` reads them
+there; that is how the laptop checks a board's HID side, with the same code
+a board host runs. No host treats a board serving that UUID as a keyboard.
+
+The host writes the keyboard's LED report once it has subscribed, as Windows
+does. `Peripheral.leds()` returns those writes, which is how the gate knows
+the host is listening, and how it times a round trip.
+
 ## The checks
 
 **Against the fake**, on both interpreters: `tests/bledev_contract.py`.
@@ -236,6 +317,8 @@ The serving scripts have a `PLANT` switch, and a planted run must fail:
 | `repl_server.py` + `repl_client.py` | The REPL: the right password evaluates `123 * 456`, a wrong one sent with code that would create `/pwned` is refused and hung up on and the code never runs, and Ctrl-C stops `while True`. `repl_server.py` runs from `/main.py`, because mpftp soft-resets the board, which turns Bluetooth off |
 | `improv_server.py` + `improv_client.py` | Improv: `--wrong` must get "unable to connect" and a return to "authorized"; without it, on a board, the network comes from that board's own `secrets.py` and must end "provisioned" with a URL |
 | `coex_server.py` + `tcp_pull.py` + `nus_client.py` | Wi-Fi and BLE on one S3: a TCP source on Wi-Fi beside the nus gate |
+| `hid_peripheral.py` + `hid_central.py` | HID: the device types `hid_script.py`'s text, chords, media keys and gamepad moves; the host's events must equal `hid_script.expected()` and spell the text. Then key-press round trips. `PLANT` drops one Shift. The host also runs on a laptop, where it refuses to pair |
+| `bless_peripheral.py` + `bless_probe.py` | The laptop as a peripheral, through bless, probed by a board |
 
 `gatt_peripheral.py` is also the peripheral to point a host backend at: it
 advertises as `bledev-radio`, and a central steers it by writing commands.
@@ -276,6 +359,42 @@ off, 2026-09-24.
 | registering nus, then advertising | 0.2 KB | 1.8 KB |
 | `active(False)` | all returned | |
 
+### HID, board to board and from the laptop
+
+`hid_peripheral.py` typed `hid_script.py` (30 characters with Shift, Ctrl+C,
+Right Alt + Shift + F5, two media keys, five gamepad reports: 112 events) and
+`hid_central.py` compared every event and the text they spell. MicroPython
+1.29, aioble from mip, 2026-09-24. The planted run drops Shift from the first
+character.
+
+| Device | Host | Plain | Planted | 7.5-15 ms interval | Round trip, default (median / max) | Round trip, 7.5-15 ms |
+|---|---|---|---|---|---|---|
+| LCD-7 (S3) | T-Embed (S3) | PASS, 112/112 | FAIL, event 0 | PASS | 69 / 119 ms | 29 / 49 ms |
+| T-Embed (S3) | LCD-7 (S3) | PASS | FAIL, event 0 | PASS | 68 / 118 ms | 30 / 69 ms |
+| P4 | LCD-7 (S3) | PASS | FAIL, event 0 | PASS | 70 / 220 ms | 30 / 80 ms |
+| LCD-7 (S3) | P4 | PASS 2 of 5 | FAIL, event 0 | FAIL 6 of 6 | 68 / 178 ms | 29 / 69 ms |
+| P4, vendor UUID | laptop (bleak) | PASS | FAIL, event 0 | PASS | 90 / 270 ms | 60 / 90 ms |
+
+The round trip is a key press on the device to the host's LED write arriving
+back, so one way is roughly half: 35 ms at the default interval, 15 ms at
+7.5-15 ms. Decoding on the host adds 1.7 ms (P4) to 2.8 ms (S3) from the
+report's arrival to the app's `events()`, with a p99 of 10 to 23 ms.
+
+**The P4 as host loses reports** below Python, always from around the 26th
+report on (pydevices#87). The S3 never did. `burst_source.py` and
+`burst_sink.py` reproduce it without HID.
+
+**Pairing and bonding** (`ENCRYPTED = True`: every HID characteristic
+encrypted, both sides call `enable_bonding()`), each direction between the S3s
+and with the P4 as device:
+
+| Run | Result |
+|---|---|
+| First connect, no keys on either side | The Report Map read fails, the host pairs "just works" (encrypted, not authenticated, 16-byte key, bonded), 112/112 |
+| Both boards reset, then reconnect | Encryption from the stored keys: the key store's checksum unchanged, connect-to-started 0.8 to 1.4 s faster, 112/112 |
+| The host's `ble_secrets.json` deleted (as an erase would) | The device accepts a fresh pairing (NimBLE's repeat-pairing path drops the old bond), new keys, 112/112 |
+| A host that refuses to pair (planted) | Fails: the device's encrypted Report Map can't be read |
+
 ### The laptop to an S3
 
 Windows 11, bleak 3.0.2, the laptop's Intel radio, to the LCD-7 running

docs/bledev.md

@@ -223,6 +223,62 @@ boot is up to you: pass `on_join=lambda ssid, password, ip: ...` to `serve()`.
 (from a button, say) before it accepts credentials. The credentials cross the
 air unencrypted; that's the standard.
 
+## Keyboards and gamepads (HID)
+
+`bledev.hid` turns a BLE keyboard, media remote or game controller into the
+same `events` a USB keyboard gives you through usbif, or SDL gives you on the
+desktop: `events.Key` with `keys.K_*` codes and modifiers, and
+`events.JoyAxisMotion`, `JoyHatMotion`, `JoyButtonDown` and `JoyButtonUp`.
+Your app can't tell which one it's talking to.
+
+```python
+import bledev.hid as hid
+
+host = await hid.connect(ble, name="Keyboard K380")   # or no name: any HID device
+while True:
+    for event in await host.events():                 # waits for at least one
+        print(event)
+```
+
+`host.poll()` returns whatever has arrived without waiting, the way usbif's
+`poll()` does, so it fits a frame loop. Media keys arrive as `Key` events
+too (`keys.K_VOLUMEUP`, `K_AUDIOPLAY`). Axes run from -1.0 to 1.0 and are
+numbered X, Y, Z, Rx, Ry, Rz, then triggers; buttons count from 0; a hat is
+an `(x, y)` tuple with y up.
+
+A keyboard that wants an encrypted link gets one: the first read it refuses
+pairs the link ("just works", no passkey) and tries again. On a board, call
+`ble.enable_bonding()` once before connecting so the keys are kept, and the
+next connect doesn't pair from scratch. The keys live in `ble_secrets.json`
+on the board's filesystem, so erasing the board forgets them, and a keyboard
+that remembers the board may then need putting back into pairing mode.
+
+**A board can be the keyboard, too.** It advertises as a keyboard, media
+remote and gamepad (or any one or two of them) and sends what you tell it:
+
+```python
+kb = hid.Peripheral(ble)                  # keyboard=True, consumer=True, gamepad=True
+await kb.serve()                          # returns when a host connects
+await kb.keyboard.type("Hello\n")         # US layout
+await kb.keyboard.tap(0x06, hid.MOD_LCTRL)   # Ctrl+C
+await kb.consumer.tap(hid.VOLUME_UP)
+await kb.gamepad.send(axes=(0, 127, 0, 0), buttons=0b101, hat=2)
+```
+
+Its name, appearance and Report Map come from which functions you ask for,
+the same way every time, because hosts cache a device against its identity.
+Pass `encrypted=True` to make hosts pair first, as real keyboards do.
+
+**Don't pair a test board with the computer you're testing from.** Once
+paired, Windows and phones use it as a real keyboard, and it types into
+them.
+
+The parser underneath, `bledev.hidreport`, knows nothing about Bluetooth.
+Give it any HID report descriptor and it decodes that device's reports, so a
+USB host can use it the same way. How it numbers axes, what it does with a
+rollover, and what a board central can't read are in
+[bledev-internals.md](bledev-internals.md#how-hid-gets-there).
+
 ## The rules every backend keeps
 
 These hold on every backend. The fake enforces the strict version of each, so

lib/bledev/__init__.py

@@ -76,6 +76,11 @@ class BusyError(BLEError):
     """
 
 
+#: ATT error codes a :class:`GattError` can carry that apps act on.
+INSUFFICIENT_AUTHENTICATION = 0x05
+INSUFFICIENT_ENCRYPTION = 0x0F
+
+
 class GattError(BLEError):
     """The peer answered a GATT request with an error ``status``."""
 
@@ -353,7 +358,8 @@ def capabilities(self):
 
         ``backend`` (its module name), ``central`` and ``peripheral`` (which
         roles it can take), ``mtu`` (the ATT MTU it asks for), ``max_mtu``,
-        ``indicate``, ``pairing`` and ``l2cap`` (bools).
+        ``indicate``, ``pairing`` and ``l2cap`` (bools), and ``long_read``:
+        whether a read returns values longer than one packet (``mtu - 1``).
         """
         return {
             "backend": self.backend,
@@ -364,6 +370,7 @@ def capabilities(self):
             "indicate": False,
             "pairing": False,
             "l2cap": False,
+            "long_read": True,
         }
 
     def config(self, *names, **settings):
@@ -468,7 +475,8 @@ class Characteristic:
     write; MicroPython's default buffer is 20 bytes, so raise it for anything
     that takes whole MTU-sized writes. With ``capture=True``, ``written()``
     returns every write as ``(connection, data)``, in order, instead of just
-    the last writer.
+    the last writer. ``encrypted=True`` refuses a central that hasn't paired
+    (``GattError`` with status ``INSUFFICIENT_ENCRYPTION``) until it does.
     """
 
     def __init__(
@@ -483,6 +491,7 @@ def __init__(
         initial=None,
         capture=False,
         max_len=20,
+        encrypted=False,
     ):
         service.characteristics.append(self)
         self.service = service
@@ -502,6 +511,11 @@ def __init__(
         self.capture = bool(capture) and bool(flags & (FLAG_WRITE | FLAG_WRITE_NO_RESPONSE))
         self.max_len = max(max_len, len(initial) if initial else 0)
         self._initial = bytes(initial) if initial is not None else None
+        #: True when a central must pair (an encrypted link) before it may
+        #: read, write or subscribe. HID keyboards serve their reports this way.
+        self.encrypted = bool(encrypted)
+        #: :class:`Descriptor` objects, in the order they were made.
+        self.descriptors = []
         # Set by the backend's register_services(); it does the real work.
         self._impl = None
 
@@ -555,6 +569,25 @@ async def written(self, timeout_ms=None):
         return await self._bound().written(timeout_ms)
 
 
+class Descriptor:
+    """A read-only descriptor on a :class:`Characteristic` you serve.
+
+    HID uses these: each Report characteristic carries a Report Reference
+    (0x2908) saying which report it is. The value is fixed once registered.
+    Don't make a CCCD (0x2902); every backend adds that itself for a
+    characteristic with ``notify`` or ``indicate``.
+    """
+
+    def __init__(self, characteristic, uuid, initial=b""):
+        characteristic.descriptors.append(self)
+        self.characteristic = characteristic
+        self.uuid = UUID(uuid)
+        self.value = bytes(initial)
+
+    def __repr__(self):
+        return "Descriptor({!r})".format(self.uuid)
+
+
 # ---------------------------------------------------------------- central side
 
 
@@ -754,6 +787,23 @@ async def disconnected(self, timeout_ms=None):
         """Wait until the link drops, from either side."""
         raise NotImplementedError
 
+    @property
+    def encrypted(self):
+        """True once the link is encrypted (after :meth:`pair`, or a bonded reconnect)."""
+        return False
+
+    async def pair(self, bond=True, timeout_ms=20000):
+        """Pair with the peer and encrypt the link; with ``bond``, keep the keys.
+
+        Either side may call it; in practice the central does, when a read
+        fails with ``INSUFFICIENT_ENCRYPTION``. Pairing here is "just works"
+        (no passkey), which is what a keyboard without a screen does. A bonded
+        peer reconnects without pairing again, for as long as both sides keep
+        their keys. Raises :class:`UnsupportedError` on a backend without
+        pairing.
+        """
+        raise UnsupportedError("{} cannot pair".format(self._ble.backend))
+
     async def exchange_mtu(self, mtu=None, timeout_ms=1000):
         """Negotiate a larger ATT MTU and return the result.
 
@@ -836,8 +886,27 @@ def _check(self, flag, what):
             raise UnsupportedError("{!r} does not support {}".format(self, what))
 
     async def read(self, timeout_ms=1000):
+        """The value. A backend without long reads (``capabilities()["long_read"]``
+        is False: MicroPython) returns at most ``connection.mtu - 1`` bytes."""
         raise NotImplementedError
 
+    def descriptors(self, uuid=None, timeout_ms=2000):
+        """Async-iterate this characteristic's descriptors, optionally only ``uuid``."""
+        uuid = None if uuid is None else UUID(uuid)
+        return _Discovery(lambda: self._discover_descriptors(uuid, timeout_ms))
+
+    async def descriptor(self, uuid, timeout_ms=2000):
+        """The descriptor ``uuid`` on this characteristic, or ``None``."""
+        uuid = UUID(uuid)
+        found = None
+        async for d in self.descriptors(uuid, timeout_ms):
+            if found is None and d.uuid == uuid:
+                found = d
+        return found
+
+    async def _discover_descriptors(self, uuid, timeout_ms):
+        raise UnsupportedError("{} cannot discover descriptors".format(self.connection._ble.backend))
+
     async def write(self, data, response=None, timeout_ms=1000):
         """Write ``data`` (at most ``connection.mtu - 3`` bytes).
 
@@ -867,6 +936,21 @@ def _want_response(self, response):
         return bool(response)
 
 
+class ClientDescriptor:
+    """A descriptor on the peer, found by :meth:`ClientCharacteristic.descriptors`."""
+
+    def __init__(self, characteristic, uuid):
+        self.characteristic = characteristic
+        self.connection = characteristic.connection
+        self.uuid = UUID(uuid)
+
+    def __repr__(self):
+        return "ClientDescriptor({!r})".format(self.uuid)
+
+    async def read(self, timeout_ms=1000):
+        raise NotImplementedError
+
+
 def is_adapter(obj):
     """True for a bledev :class:`BLE`; False for a raw ``bluetooth.BLE`` or anything else."""
     return isinstance(obj, BLE)