diff --git a/docs/bledev-internals.md b/docs/bledev-internals.md index 675f9e34..a4199551 100644 --- a/docs/bledev-internals.md +++ b/docs/bledev-internals.md @@ -21,6 +21,7 @@ standalone module per backend, and an optional selector. | `improv.py` | Improv Wi-Fi setup, both sides, built only on the contract. | | `midi.py` | BLE-MIDI as a usbif-style MIDI port, both sides, built only on the contract. | | `midi_codec.py` | The BLE-MIDI packet codec. Pure, no imports, shared by every host. | +| `filetransfer.py` | CircuitPython's BLE file-transfer protocol: `FileServer` (pure, no radio), served on the board through `repl`'s server, and a client built on the contract. | | `auto.py` | Picks a backend. Nothing imports it, and backends must not. | A backend module imports `bledev` and whatever its host provides, and nothing @@ -249,8 +250,60 @@ without using up the attempt. The ESP32 has one dupterm slot. The REPL takes it on login and gives back whatever held it (WebREPL, say) on disconnect. +## How file transfer gets there + +The protocol is Adafruit's (`supervisor/shared/bluetooth/file_transfer.c` in +CircuitPython; the spec is in Adafruit_CircuitPython_BLE_File_Transfer): +service `0xFEBB`, a version characteristic reading 4, and one transfer +characteristic the client writes commands to and the board answers with +notifications. bledev adds one characteristic, `ADAF0300-...`, for the +password; CircuitPython has none, which is how a client tells the two apart. + +`FileServer` is the protocol without a radio. It takes the client's bytes in +any split, handles every command whose bytes have all arrived (several may +share a write), and queues answers. Its `packet(n)` never lets one +notification carry the end of one answer and the start of the next, because +Adafruit's client expects a header at the start of a notification. A read's +data is streamed from the file one packet at a time rather than loaded. + +On the board it's a second service on `repl`'s raw-API server, registered +in the same `gatts_register_services` call as NUS, because a registration +replaces everything before it. What differs from CircuitPython, and why: + +- **File work runs on the main thread, not in the IRQ.** On the esp32 port + the BLE IRQ runs in NimBLE's host task (`MICROPY_PY_BLUETOOTH_USE_SYNC_EVENTS`), + whose stack is a few KB. The first build handled commands there, and a + recursive delete took the T-Embed down. Now the IRQ only appends the bytes + to an inbox and `micropython.schedule()`s the work; the REPL's 5 ms timer + picks up anything the schedule queue refused. A disconnect in the IRQ only + flags the reset, so the main thread never has a file closed under it. +- **The window is 4 KB, not a 512-byte sector.** Each `WRITE_PACING` offers + `min(remaining, window)` bytes, so 20 KB is five round trips. The transfer + characteristic's append buffer is `2 * window + 64`, room for a window, its + header and the next command, and the board never offers more than that. +- **Truncation.** MicroPython's files can't truncate, so a write at an offset + that leaves the file shorter copies the part that stays. +- **Locked means answered.** Before the password, every command gets its own + answer shape with status `0x80`, so a client can parse the refusal rather + than time out. + +Adafruit's own client (`adafruit_ble_file_transfer.FileTransferClient`, with +`_bleio` stubbed and a stand-in PacketBuffer that hands it one `packet()` per +read) wrote, read, listed, moved and deleted against `FileServer` at MTU 247, +and wrote and read at MTU 23; its `listdir` can't run at 23 against any +server, because it reads a 28-byte header into one 20-byte packet. That was +a one-off check, not a test in the tree. No CircuitPython board has been +tried with bledev's client: CircuitPython's service needs pairing, which +bledev doesn't do yet. + ## The checks +**The file-transfer protocol**, on both interpreters: +`tests/bledev_filetransfer.py` wires the client to a `FileServer` through a +loopback that splits both directions at the MTU (23, 185 and 247, windows +from 512 to 4096), and `tests/test_bledev_filetransfer.py` runs it with a +flipped bit and a lost packet, each of which must fail. + **Against the fake**, on both interpreters: `tests/bledev_contract.py`. Its `pair()` and `nus_pair()` helpers build two fake adapters. A host backend that has a fake peripheral to talk to can reuse most checks by building its @@ -278,6 +331,7 @@ The serving scripts have a `PLANT` switch, and a planted run must fail: | `reconnect_loop.py` (laptop) against `repl_server.py` | Rapid reconnects: log in, run a line, close, wait 1.5 s, N times; `--no-retry` turns `connect_and_set_up`'s retry off | | `improv_server.py` + `improv_client.py` | Improv: `--wrong` must get "unable to connect" and a return to "authorized"; without it, on a board, the network comes from that board's own `secrets.py` and must end "provisioned" with a URL | | `coex_server.py` + `tcp_pull.py` + `nus_client.py` | Wi-Fi and BLE on one S3: a TCP source on Wi-Fi beside the nus gate | +| `files_server.py` + `files_client.py` | File transfer: no password and a wrong one refused; 20 KB up and down byte for byte and timed (`--runs N`, `--fast` for throughput parameters), a read at an offset, mkdir, listdir, move, a recursive delete. `--plant` on the client, or `PLANT = "flip"` on the server, must FAIL. `files_server.py` runs from `/main.py` | `gatt_peripheral.py` is also the peripheral to point a host backend at: it advertises as `bledev-radio`, and a central steers it by writing commands. @@ -410,3 +464,26 @@ the board. Pairing needs the board to bond (`ble.config(bond=True, ...)` with aioble's `security` module loaded); bledev doesn't do that for you yet. Before `midi.serve()` set the GAP name, Windows called the port `MPY ESP32`. +### File transfer + +The laptop (Windows 11, bleak 3.0.2) to the T-Embed running `files_server.py`, +one desk apart, Wi-Fi off, bledev as `.py` source, 2026-09-24. 20 KB each +way, every run byte for byte: + +| Client | Up | Down | +|---|---|---| +| `files_client.py`, Windows' defaults, 5 runs in two sessions | 0.73-1.42 s (14.5-27.9 KB/s) | 0.80-0.98 s (20.8-25.5 KB/s) | +| `files_client.py --fast` (throughput parameters), 3 runs | 0.72-0.82 s (24.9-28.4 KB/s) | 0.32-0.39 s (52.1-64.2 KB/s) | +| mpftp's `ble_bench.py`, throughput parameters, median of 3 | 0.83 s (24.0 KB/s) | 0.30 s (67.5 KB/s) | +| the same over mpremote's raw REPL (256-byte chunks) | 15.6 s (1.3 KB/s) | 10.6 s (1.9 KB/s) | +| the same over the raw REPL, 4 KB chunks | 11.8 s (1.7 KB/s) | 6.3 s (3.2 KB/s) | +| `ble_bench.py`, Windows' defaults, file transfer / raw REPL | 1.37 s / 43.6 s | 0.49 s / 26.6 s | + +So file transfer is the path for files: 19 times faster up and 35 times +down than the raw REPL, which pays a round trip for every raw-paste window +and prints what it reads as text. Downloads run near nus's own notification +rate. Uploads are held to about 25 KB/s by the board writing flash between +windows. Up got slower from run to run at Windows' defaults (25.8, 17.4, 14.5 +KB/s), overwriting the same file each time. The planted bit flip on the client +failed the round-trip check. mpftp's `ble://` transport, and its gate, are in +[PyDevices/mpftp](https://github.com/PyDevices/mpftp) `docs/plans/ble.md`. diff --git a/docs/bledev.md b/docs/bledev.md index 7c5eea0b..86ed3cdc 100644 --- a/docs/bledev.md +++ b/docs/bledev.md @@ -198,6 +198,50 @@ While it runs, the REPL owns the radio: it registers its own service and advertises whenever nobody is connected. Call `bledev.repl.stop()` to give the radio back to your app. +## Files over Bluetooth + +`bledev.filetransfer` is the board side of CircuitPython's BLE file-transfer +protocol, so a MicroPython board serves files the same way a CircuitPython one +does, and one client reaches both. It's opt-in, like the REPL; in `main.py`: + +```python +import bledev.filetransfer +bledev.filetransfer.start(password="correct horse", name="rack") # files and the REPL +``` + +From a laptop or another board: + +```python +import bledev.filetransfer as ft +files = await ft.connect(ble, "correct horse", name="rack") +await files.write("/lib/app.py", source) +print(await files.read("/lib/app.py")) +print(await files.listdir("/lib")) # [(name, size, is_directory, mtime_ns), ...] +await files.mkdir("/data/logs") # parents too +await files.move("/a.txt", "/data/a.txt") +await files.delete("/data") # a directory and everything in it +``` + +`write()` takes an `offset`, and the file ends after what you wrote. +`read()` takes one too. The board paces every write with a free-space count, +so it never receives more than it can hold. + +It's locked the way the REPL is. CircuitPython uses pairing; bledev uses the +REPL's password, sent to one extra characteristic, and until it's right +every command is refused. One attempt per connection, and a wrong one is hung +up on. Logging in to the REPL over the same connection unlocks files too, +which is what [mpftp](https://github.com/PyDevices/mpftp) does. The protocol +has no checksum beyond the radio's own, so check what matters (mpftp compares +SHA-256 after every transfer). + +`start()` serves the REPL beside the files; `console=False` serves files +alone. It's MicroPython only, and like the REPL it owns the radio while it +runs. + +How fast: 20 KB from the laptop to the T-Embed in about 0.8 s and back in +0.3 s, against 16 s and 11 s through the raw REPL. The numbers are in +[the internals](bledev-internals.md#file-transfer). + ## Wi-Fi setup with Improv `bledev.improv` speaks [Improv](https://www.improv-wifi.com/ble/), the BLE diff --git a/lib/bledev/filetransfer.py b/lib/bledev/filetransfer.py new file mode 100644 index 00000000..275178ec --- /dev/null +++ b/lib/bledev/filetransfer.py @@ -0,0 +1,836 @@ +"""Files over Bluetooth: CircuitPython's BLE file-transfer protocol, both sides. + +CircuitPython boards serve this protocol from their supervisor +(``supervisor/shared/bluetooth/file_transfer.c``). This module lets a +MicroPython board serve it too, so one client reaches files on either +interpreter. The protocol is Adafruit's, documented with their client in +`Adafruit_CircuitPython_BLE_File_Transfer +`_: +service ``0xFEBB``, a version characteristic, and one transfer +characteristic that the client writes commands to and the board answers on +with notifications. It reads, writes (at an offset), deletes, makes +directories, lists them and moves, and the board paces every write with a +"free space" count so it never receives more than it can hold. + +On the board, opt in from ``main.py`` (nothing starts it for you):: + + import bledev.filetransfer + bledev.filetransfer.start(password="correct horse", name="rack") + +It's served beside the REPL of :mod:`bledev.repl` by default (``console=False`` +serves files alone), from interrupts, so it keeps working at the ``>>>`` +prompt and while your program runs. From a laptop or another board:: + + import bledev.filetransfer as ft + files = await ft.connect(ble, "correct horse", name="rack") + await files.write("/hello.txt", b"hi") + print(await files.read("/hello.txt"), await files.listdir("/")) + +**The lock.** CircuitPython protects the transfer characteristic with +pairing. bledev uses the REPL's password instead, the same way WebREPL does: +the service carries one more characteristic (``AUTH``), the client writes the +password to it, and until it's right every command is answered with +:data:`STATUS_LOCKED`. One attempt per connection; a wrong password is +hung up on. Logging in to the REPL over the same connection unlocks files +too. The link isn't encrypted, so a sniffer nearby can capture the password; +pairing is the later fix. A client connecting to a CircuitPython board finds +no ``AUTH`` characteristic and skips the step. + +The protocol carries no checksum of its own beyond the link layer's CRC, so a +client that must be sure checks what landed (mpftp compares SHA-256). +""" + +try: + import asyncio +except ImportError: # pragma: no cover + import uasyncio as asyncio + +import struct + +from . import BLEError, BLETimeoutError, DisconnectedError, UnsupportedError, UUID, connect_and_set_up, wait_ms + +SERVICE = UUID(0xFEBB) +VERSION = UUID("adaf0100-4669-6c65-5472-616e73666572") +TRANSFER = UUID("adaf0200-4669-6c65-5472-616e73666572") +#: bledev's addition: write the password here. CircuitPython has no such +#: characteristic (it uses pairing), which is how a client tells them apart. +AUTH = UUID("adaf0300-4669-6c65-5472-616e73666572") + +#: The protocol version this server speaks (CircuitPython 9 and 10 serve 4). +PROTOCOL_VERSION = 4 + +READ = 0x10 +READ_DATA = 0x11 +READ_PACING = 0x12 +WRITE = 0x20 +WRITE_PACING = 0x21 +WRITE_DATA = 0x22 +DELETE = 0x30 +DELETE_STATUS = 0x31 +MKDIR = 0x40 +MKDIR_STATUS = 0x41 +LISTDIR = 0x50 +LISTDIR_ENTRY = 0x51 +MOVE = 0x60 +MOVE_STATUS = 0x61 + +STATUS_OK = 0x01 +STATUS_ERROR = 0x02 +STATUS_ERROR_NO_FILE = 0x03 +STATUS_ERROR_PROTOCOL = 0x04 +STATUS_ERROR_READONLY = 0x05 +#: bledev only: nothing is served until the password is right. +STATUS_LOCKED = 0x80 + +FLAG_DIRECTORY = 0x01 + +# Every header, little-endian and packed. Reserved fields are sent as zero. +_READ = "= total + head = struct.pack(_READ_DATA, READ_DATA, STATUS_OK, 0, offset, total, size) + self._send(head, self._file, offset, size, close=last) + if last: + self._file = None + self._expect = None + else: + self._expect = READ_PACING + + # WRITE + def _do_20(self, buf): + _, _, length, offset, when, total = struct.unpack_from(_WRITE, buf) + path = self._path_at(buf, 20, length) + import os + + try: + try: + size = os.stat(path)[6] + except OSError: + size = -1 + if offset == 0 or size < 0: + f = open(path, "wb") + size = 0 + else: + f = open(path, "r+b") + if offset > size: + f.seek(size) + gap = offset - size + while gap: + n = min(gap, 512) + f.write(bytes(n)) + gap -= n + except OSError as e: + self._send(_answer(WRITE, _status_for(e))) + return + self._file = f + self._path = path + self._total = total + self._time = when + self._offset = offset + self._expect = WRITE_DATA + self._pace(offset) + + # WRITE_DATA + def _do_22(self, buf): + _, status, _, offset, size = struct.unpack_from(_WRITE_DATA, buf) + f = self._file + if status != STATUS_OK or offset != self._offset or size > self.window or offset + size > self._total: + self._send(_answer(WRITE_DATA, STATUS_ERROR_PROTOCOL if status == STATUS_OK else STATUS_ERROR)) + self._abort() + return + try: + f.seek(offset) + f.write(memoryview(buf)[12 : 12 + size]) + except OSError as e: + self._send(_answer(WRITE_DATA, _status_for(e))) + self._abort() + return + self._offset = offset + size + self._pace(self._offset) + + def _pace(self, offset): + free = min(self._total - offset, self.window) + when = self._time + if not free: + try: + self._finish_write() + when = self._stored_time() + except OSError as e: + self._send(_answer(WRITE_DATA, _status_for(e))) + self._abort() + return + self._send(struct.pack(_WRITE_PACING, WRITE_PACING, STATUS_OK, 0, offset, when, free)) + + def _finish_write(self): + import os + + f, self._file, self._expect = self._file, None, None + f.close() + self.changed += 1 + if os.stat(self._path)[6] > self._total: + _truncate(self._path, self._total) + + def _stored_time(self): + import os + + try: + return _ns(os.stat(self._path)[8]) + except OSError: + return self._time + + # DELETE + def _do_30(self, buf): + length = struct.unpack_from(_PATH, buf)[2] + path = self._path_at(buf, 4, length).rstrip("/") or "/" + try: + _remove(path) + status = STATUS_OK + self.changed += 1 + except OSError as e: + status = _status_for(e) + self._send(struct.pack(_STATUS, DELETE_STATUS, status)) + + # MKDIR + def _do_40(self, buf): + _, _, length, _, when = struct.unpack_from(_MKDIR, buf) + path = self._path_at(buf, 16, length).rstrip("/") + import os + + status = STATUS_OK + try: + _makedirs(path) + when = _ns(os.stat(path)[8]) + self.changed += 1 + except OSError as e: + status = _status_for(e) + self._send(struct.pack(_MKDIR_STATUS, MKDIR_STATUS, status, 0, 0, when)) + + # LISTDIR + def _do_50(self, buf): + import os + + length = struct.unpack_from(_PATH, buf)[2] + path = self._path_at(buf, 4, length).rstrip("/") or "/" + try: + if not os.stat(path)[0] & 0x4000: + raise OSError(20) + names = sorted(os.listdir(path)) + except OSError: + self._send(_answer(LISTDIR, STATUS_ERROR_NO_FILE)) + return + count = len(names) + prefix = path if path.endswith("/") else path + "/" + for i, name in enumerate(names): + try: + st = os.stat(prefix + name) + except OSError: + st = (0, 0, 0, 0, 0, 0, 0, 0, 0, 0) + directory = bool(st[0] & 0x4000) + raw = name.encode() + head = struct.pack( + _ENTRY, + LISTDIR_ENTRY, + STATUS_OK, + len(raw), + i, + count, + FLAG_DIRECTORY if directory else 0, + _ns(st[8]), + 0 if directory else st[6], + ) + # One answer per entry: CircuitPython's own client expects an + # entry's header to start a notification. + self._send(head + raw) + self._send(struct.pack(_ENTRY, LISTDIR_ENTRY, STATUS_OK, 0, count, count, 0, 0, 0)) + + # MOVE + def _do_60(self, buf): + import os + + _, _, old, new = struct.unpack_from(_MOVE, buf) + src = self._path_at(buf, 6, old) + dst = self._path_at(buf, 6 + old + 1, new) + try: + os.rename(src, dst) + status = STATUS_OK + self.changed += 1 + except OSError as e: + status = _status_for(e) + self._send(struct.pack(_STATUS, MOVE_STATUS, status)) + + +def _remove(path): + import os + + if os.stat(path)[0] & 0x4000: + prefix = path if path.endswith("/") else path + "/" + for name in os.listdir(path): + _remove(prefix + name) + os.rmdir(path) + else: + os.remove(path) + + +def _makedirs(path): + import os + + try: + mode = os.stat(path)[0] + except OSError: + parent = path[: path.rfind("/")] + if parent: + _makedirs(parent) + os.mkdir(path) + return + if not mode & 0x4000: + raise OSError(17) # a file is in the way + + +def _truncate(path, size): + # MicroPython's files can't truncate, so copy the part that stays. + import os + + tmp = path + ".bledev-tmp" + with open(path, "rb") as src, open(tmp, "wb") as dst: + left = size + while left: + chunk = src.read(min(left, 1024)) + if not chunk: + break + dst.write(chunk) + left -= len(chunk) + os.remove(path) + os.rename(tmp, path) + + +# ---------------------------------------------------------------- the board side + + +def start(password=None, name="mpy-files", *, console=True, window=WINDOW, **options): + """Serve files (and, unless ``console=False``, the REPL) over BLE. MicroPython only. + + The radio, the password and the advertising are :mod:`bledev.repl`'s; + this is ``bledev.repl.start(..., files=True)``. Returns at once; + :func:`stop` or ``bledev.repl.stop()`` ends it. + """ + from . import repl + + return repl.start(password, name, files=True, console=console, window=window, **options) + + +def stop(): + from . import repl + + repl.stop() + + +# ---------------------------------------------------------------- the client side + + +class Client: + """A connection to a board's file service. Made by :func:`connect`. + + Paths are absolute (``/lib/x.py``). Every method raises + :class:`FileTransferError` when the board answers with an error, and + :class:`DisconnectedError` if the link drops. + """ + + def __init__(self, connection, transfer, version, *, chunk=WINDOW): + self.connection = connection + self.version = version + self.chunk = chunk + #: How long to wait for each part of an answer. + self.timeout_ms = 10000 + self._transfer = transfer + self._buf = bytearray() + self._event = asyncio.Event() + self._closed = False + self._no_response = bool(transfer.properties & 0x0004) if transfer is not None else True + self._pump = asyncio.create_task(self._receive()) if transfer is not None else None + self.bytes_in = 0 + self.bytes_out = 0 + + async def _receive(self): + try: + while True: + self._feed(await self._transfer.notified()) + except DisconnectedError: + pass + finally: + self._closed = True + self._event.set() + + def _feed(self, data): + self._buf.extend(data) + self.bytes_in += len(data) + self._event.set() + + async def _send(self, data): + size = max(1, self.connection.mtu - 3) + for i in range(0, len(data), size): + if not self.connection.is_connected(): + raise DisconnectedError("the board hung up") + await self._transfer.write(data[i : i + size], response=not self._no_response) + self.bytes_out += len(data) + + async def _recv(self, n): + while len(self._buf) < n: + if self._closed: + raise DisconnectedError("the board hung up mid-answer") + self._event.clear() + await wait_ms(self._event.wait(), self.timeout_ms, "the board's answer") + data = bytes(self._buf[:n]) + self._buf = self._buf[n:] + return data + + async def _header(self, fmt, code): + data = await self._recv(struct.calcsize(fmt)) + fields = struct.unpack(fmt, data) + if fields[0] != code: + raise FileTransferError(STATUS_ERROR_PROTOCOL, "expected answer 0x{:02x}, got 0x{:02x}".format(code, fields[0])) + return fields + + @staticmethod + def _check(status, what): + if status == STATUS_OK: + return + if status == STATUS_LOCKED: + raise AuthError("{}: the board wants the password first".format(what)) + text = { + STATUS_ERROR_NO_FILE: "no such file or directory", + STATUS_ERROR_PROTOCOL: "protocol error", + STATUS_ERROR_READONLY: "the filesystem is read-only", + }.get(status, "failed") + raise FileTransferError(status, "{}: {} (status 0x{:02x})".format(what, text, status)) + + async def read(self, path, offset=0): + """The file's contents from ``offset`` on.""" + raw = path.encode() + chunk = self.chunk + await self._send(struct.pack(_READ, READ, 0, len(raw), offset, chunk) + raw) + out = bytearray() + while True: + _, status, _, at, total, size = await self._header(_READ_DATA, READ_DATA) + self._check(status, "read " + path) + out.extend(await self._recv(size)) + at += size + if at >= total or not size: + return bytes(out) + await self._send(struct.pack(_READ_PACING, READ_PACING, STATUS_OK, 0, at, min(chunk, total - at))) + + async def write(self, path, data, offset=0, modification_time=None): + """Write ``data`` at ``offset`` (the file ends after it). Returns the stored time in ns.""" + raw = path.encode() + data = memoryview(bytes(data)) + total = offset + len(data) + if modification_time is None: + modification_time = _now_ns() + await self._send(struct.pack(_WRITE, WRITE, 0, len(raw), offset, modification_time, total) + raw) + while True: + _, status, _, at, when, free = await self._header(_WRITE_PACING, WRITE_PACING) + self._check(status, "write " + path) + if at < offset or at > total: + raise FileTransferError(STATUS_ERROR_PROTOCOL, "write {}: the board asked for offset {}".format(path, at)) + if not free: + if at != total: + raise FileTransferError(STATUS_ERROR_PROTOCOL, "write {}: stopped at {} of {}".format(path, at, total)) + return when + n = min(free, total - at) + start = at - offset + await self._send(struct.pack(_WRITE_DATA, WRITE_DATA, STATUS_OK, 0, at, n) + bytes(data[start : start + n])) + + async def delete(self, path): + """Delete a file, or a directory and everything in it.""" + raw = path.encode() + await self._send(struct.pack(_PATH, DELETE, 0, len(raw)) + raw) + _, status = await self._header(_STATUS, DELETE_STATUS) + self._check(status, "delete " + path) + + async def mkdir(self, path, modification_time=None): + """Make a directory and any missing parents. Returns the stored time in ns.""" + raw = path.encode() + if modification_time is None: + modification_time = _now_ns() + await self._send(struct.pack(_MKDIR, MKDIR, 0, len(raw), 0, modification_time) + raw) + _, status, _, _, when = await self._header(_MKDIR_STATUS, MKDIR_STATUS) + self._check(status, "mkdir " + path) + return when + + async def listdir(self, path): + """``[(name, size, is_directory, modification_time_ns), ...]``.""" + raw = path.encode() + await self._send(struct.pack(_PATH, LISTDIR, 0, len(raw)) + raw) + entries = [] + while True: + _, status, length, number, count, flags, when, size = await self._header(_ENTRY, LISTDIR_ENTRY) + self._check(status, "listdir " + path) + if number >= count: + return entries + name = (await self._recv(length)).decode() + entries.append((name, size, bool(flags & FLAG_DIRECTORY), when)) + + async def move(self, old, new): + """Rename or move a file or directory (protocol version 4).""" + if self.version is not None and self.version < 4: + raise UnsupportedError("move needs protocol version 4; the board has {}".format(self.version)) + a = old.encode() + b = new.encode() + await self._send(struct.pack(_MOVE, MOVE, 0, len(a), len(b)) + a + b" " + b) + _, status = await self._header(_STATUS, MOVE_STATUS) + self._check(status, "move {} to {}".format(old, new)) + + async def close(self): + if self.connection is not None and self.connection.is_connected(): + await self.connection.disconnect() + self._closed = True + self._event.set() + + +def _now_ns(): + import time + + try: + return time.time_ns() + except AttributeError: + return _ns(time.time()) + + +async def connect(ble, password=None, name=None, *, device=None, timeout_ms=10000, mtu=247, **connect_options): + """Find a board serving files (by ``name``, or the first advertising the + service), connect, log in, and return a :class:`Client`. + + ``password`` is needed for a bledev board and ignored by a CircuitPython + one, which uses pairing instead (bond with it first). + """ + from . import nus + + ble = nus._adapter(ble) + nus._set_mtu(ble, mtu) + + async def setup(connection): + try: + await connection.exchange_mtu(mtu) + except (UnsupportedError, BLETimeoutError): + pass + service = await connection.service(SERVICE) + if service is None: + raise BLEError("{!r} has no file transfer service".format(connection.device)) + transfer = await service.characteristic(TRANSFER) + if transfer is None: + raise BLEError("{!r}: the file transfer service has no transfer characteristic".format(connection.device)) + version = None + char = await service.characteristic(VERSION) + if char is not None: + try: + version = struct.unpack("> 16) & 0xFF + return bytes(out) + + +async def refused(ble, password, what): + try: + files = await ft.connect(ble, password, name=NAME, timeout_ms=15000) + except ft.AuthError: + check(True, what) + return + check(False, what, "connected") + await files.close() + + +async def main(): + ble = adapter() + options = {"priority": "throughput"} if FAST and not MICROPYTHON else {} + + await refused(ble, None, "no password is refused") + await bledev.sleep_ms(1500) + await refused(ble, "not-the-password", "a wrong password is refused") + await bledev.sleep_ms(1500) + + files = await ft.connect(ble, PASSWORD, name=NAME, timeout_ms=15000, **options) + print("connected, mtu", files.connection.mtu, "protocol version", files.version) + if PLANT: + feed = files._feed + state = {"n": 0} + + def planted(data): + if len(data) > 28: + state["n"] += 1 + if state["n"] == 3: + data = bytes(data[:-1]) + bytes((data[-1] ^ 0x10,)) + feed(data) + + files._feed = planted + + try: + await files.delete("/ftgate") + except ft.FileTransferError: + pass + await files.mkdir("/ftgate/sub") + data = pattern(20 * 1024, 7) + for run in range(RUNS): + t0 = ticks() + await files.write("/ftgate/big.bin", data) + t1 = ticks() + back = await files.read("/ftgate/big.bin") + t2 = ticks() + up = len(data) / (t1 - t0) + down = len(back) / max(1, t2 - t1) + print("run {}: 20 KB up {:.0f} ms ({:.1f} KB/s), down {:.0f} ms ({:.1f} KB/s)".format( + run + 1, t1 - t0, up, t2 - t1, down)) + check(back == data, "20 KB round trip byte for byte", "{} bytes".format(len(back))) + check(await files.read("/ftgate/big.bin", offset=20000) == data[20000:], "read from an offset") + entries = dict((e[0], e) for e in await files.listdir("/ftgate")) + check(entries.get("big.bin", (0, 0))[1] == len(data) and entries["sub"][2], "listdir", repr(sorted(entries))) + await files.move("/ftgate/big.bin", "/ftgate/sub/moved.bin") + names = [e[0] for e in await files.listdir("/ftgate/sub")] + check(names == ["moved.bin"], "move", repr(names)) + await files.delete("/ftgate") + names = [e[0] for e in await files.listdir("/")] + check("ftgate" not in names, "delete a directory and its contents", repr(names)) + await files.close() + + +async def guarded(): + try: + await main() + except Exception as e: + failures.append("exception") + print("EXCEPTION", type(e).__name__, e) + if not MICROPYTHON: + import traceback + + traceback.print_exc() + print("RESULT", "FAIL {}".format(failures) if failures else "PASS") + + +asyncio.run(guarded()) +if not MICROPYTHON: + sys.exit(1 if failures else 0) diff --git a/tests/bledev_board/files_server.py b/tests/bledev_board/files_server.py new file mode 100644 index 00000000..fe6162c6 --- /dev/null +++ b/tests/bledev_board/files_server.py @@ -0,0 +1,36 @@ +# SPDX-FileCopyrightText: 2026 Brad Barnett +# +# SPDX-License-Identifier: MIT +"""The file-transfer gate, board side: serve files and the REPL over BLE and return. + +Run it from ``/main.py`` (mpftp's serial connect soft-resets the board, which +turns Bluetooth off), then drive it over the air with ``files_client.py`` or +``mpftp -d ble://bledev-files``. The password is a test value, not a secret. + +``PLANT = "flip"`` flips one bit in the third packet of every file read, which +the client's byte-for-byte check must catch. ``WINDOW`` is the free space the +board offers per round trip (CircuitPython offers 512). +""" +import bledev.filetransfer as ft + +PASSWORD = "bledev-gate-pw" +NAME = "bledev-files" +PLANT = None +WINDOW = ft.WINDOW + +if PLANT == "flip": + _packet = ft.FileServer.packet + + def packet(self, size): + data = _packet(self, size) + if data is not None and len(data) > 28: + self._plant = getattr(self, "_plant", 0) + 1 + if self._plant == 3: + data = data[:-1] + bytes((data[-1] ^ 0x10,)) + self._pending = data + return data + + ft.FileServer.packet = packet + +ft.start(password=PASSWORD, name=NAME, window=WINDOW) +print("bledev.filetransfer serving as", NAME, "window", WINDOW, "plant", PLANT) diff --git a/tests/bledev_filetransfer.py b/tests/bledev_filetransfer.py new file mode 100644 index 00000000..be807b45 --- /dev/null +++ b/tests/bledev_filetransfer.py @@ -0,0 +1,208 @@ +# SPDX-FileCopyrightText: 2026 Brad Barnett +# +# SPDX-License-Identifier: MIT +"""The file-transfer protocol, board side against client side, without a radio. + +A plain script so it runs on CPython and on MicroPython, like +``bledev_contract.py``: the board's :class:`bledev.filetransfer.FileServer` +serves a scratch directory, and the :class:`bledev.filetransfer.Client` talks +to it through a loopback that splits every write and every notification at +the link's MTU, as a radio does. ``--plant=flip`` flips one bit in one +notification and ``--plant=drop`` loses one; each must make the run fail. + +Ends with ``N passed, M failed``. +""" + +try: + import asyncio +except ImportError: # pragma: no cover + import uasyncio as asyncio + +import os +import sys + +_here = __file__.rsplit("/", 1)[0] if "/" in __file__ else "." +sys.path.insert(0, _here + "/../lib") + +import bledev.filetransfer as ft # noqa: E402 + +PLANT = None +for arg in sys.argv[1:]: + if arg.startswith("--plant="): + PLANT = arg.split("=", 1)[1] +if PLANT: + print("PLANTED", PLANT) + +passed = 0 +failed = 0 + + +def check(ok, what, detail=""): + global passed, failed + if ok: + passed += 1 + else: + failed += 1 + print("PASS" if ok else "FAIL", what, detail) + + +class _Conn: + def __init__(self, mtu): + self.mtu = mtu + + def is_connected(self): + return True + + async def disconnect(self): + pass + + +class Loopback: + """A client wired to a FileServer through MTU-sized pieces both ways.""" + + def __init__(self, server, mtu): + self.server = server + self.mtu = mtu + self.packets = [] # every notification, for the boundary check + self.notified = 0 + self.client = ft.Client(_Conn(mtu), None, ft.PROTOCOL_VERSION) + self.client.timeout_ms = 300 + self.client._send = self._send + + async def _send(self, data): + size = self.mtu - 3 + for i in range(0, len(data), size): + self.server.feed(data[i : i + size]) + while True: + packet = self.server.packet(self.mtu - 3) + if packet is None: + break + self.server.sent() + if len(packet) > 28: + # A full packet of file data (no header is longer than 28 bytes). + self.notified += 1 + if PLANT == "flip" and self.notified == 3: + packet = packet[:-1] + bytes((packet[-1] ^ 0x10,)) + if PLANT == "drop" and self.notified == 3: + continue + self.packets.append(packet) + self.client._feed(packet) + await asyncio.sleep(0) + + +def _scratch(): + base = "/tmp" if sys.implementation.name == "micropython" else os.environ.get("TMPDIR", "/tmp") + root = base + "/bledev-ft-{}".format(os.getpid() if hasattr(os, "getpid") else 1) + try: + ft._remove(root) + except OSError: + pass + os.mkdir(root) + return root + + +def _pattern(n, seed): + out = bytearray(n) + x = seed + for i in range(n): + x = (x * 1103515245 + 12345) & 0x7FFFFFFF + out[i] = (x >> 16) & 0xFF + return bytes(out) + + +async def expect_error(coro, what, status=None, kind=ft.FileTransferError): + try: + await coro + except kind as e: + ok = status is None or getattr(e, "status", None) == status + check(ok, what, repr(e)) + return + except Exception as e: + check(False, what, "raised {!r}".format(e)) + return + check(False, what, "no error") + + +async def run(mtu, window): + root = _scratch() + server = ft.FileServer(root=root, window=window) + loop = Loopback(server, mtu) + files = loop.client + tag = "mtu {} window {}:".format(mtu, window) + + await expect_error(files.listdir("/"), tag + " locked until the password", kind=ft.AuthError) + check(not os.listdir(root), tag + " nothing touched while locked") + server.authed = True + + data = _pattern(20 * 1024, 1) + when = await files.write("/big.bin", data, modification_time=1234 * 10**9) + check(when > 0, tag + " write returns a stored time", str(when)) + with open(root + "/big.bin", "rb") as f: + check(f.read() == data, tag + " 20 KB written byte for byte") + back = await files.read("/big.bin") + check(back == data, tag + " 20 KB read back byte for byte", "{} bytes".format(len(back))) + files.chunk = 1000 + check(await files.read("/big.bin", offset=5000) == data[5000:], tag + " read from an offset, in paced chunks") + files.chunk = ft.WINDOW + + await files.write("/big.bin", b"TAIL", offset=100) + with open(root + "/big.bin", "rb") as f: + check(f.read() == data[:100] + b"TAIL", tag + " write at an offset truncates after it") + await files.write("/gap.bin", b"x", offset=10) + with open(root + "/gap.bin", "rb") as f: + check(f.read() == bytes(10) + b"x", tag + " an offset past the end fills zeros") + await files.write("/empty", b"") + check(os.stat(root + "/empty")[6] == 0, tag + " an empty file") + check(await files.read("/empty") == b"", tag + " read an empty file") + + await files.mkdir("/a/b/c/") + check(os.stat(root + "/a/b/c")[0] & 0x4000 != 0, tag + " mkdir makes parents") + await files.mkdir("/a/b") + check(True, tag + " mkdir of an existing directory is OK") + await expect_error(files.mkdir("/big.bin/x"), tag + " mkdir under a file fails", ft.STATUS_ERROR) + await files.write("/a/b/long-name-for-a-file-{}.txt".format("x" * 40), b"hello") + entries = await files.listdir("/a/b") + names = sorted(e[0] for e in entries) + check(names == sorted(["c", "long-name-for-a-file-{}.txt".format("x" * 40)]), tag + " listdir names", repr(names)) + kinds = dict((e[0], (e[1], e[2])) for e in entries) + check(kinds.get("c") == (0, True), tag + " listdir marks a directory", repr(kinds.get("c"))) + top = dict((e[0], e) for e in await files.listdir("/")) + check(top.get("big.bin", (0, 0))[1] == 104 and top["a"][2], tag + " listdir sizes", repr(top.get("big.bin"))) + await expect_error(files.listdir("/nope"), tag + " listdir of nothing", ft.STATUS_ERROR_NO_FILE) + await expect_error(files.read("/nope"), tag + " read of nothing", ft.STATUS_ERROR) + await expect_error(files.read("/a"), tag + " read of a directory", ft.STATUS_ERROR) + + await files.move("/gap.bin", "/a/moved.bin") + check("moved.bin" in os.listdir(root + "/a") and "gap.bin" not in os.listdir(root), tag + " move") + await expect_error(files.move("/nope", "/x"), tag + " move of nothing", ft.STATUS_ERROR) + await files.delete("/a") + check("a" not in os.listdir(root), tag + " delete takes a directory and its contents") + await expect_error(files.delete("/a"), tag + " delete of nothing", ft.STATUS_ERROR) + + # A continuation with nothing running is a protocol error, and the + # server keeps serving afterwards. + await files._send(bytes((ft.WRITE_DATA, ft.STATUS_OK, 0, 0)) + bytes(8)) + _, status, _, _, _, _ = await files._header(ft._WRITE_PACING, ft.WRITE_PACING) + check(status == ft.STATUS_ERROR_PROTOCOL, tag + " a stray WRITE_DATA is a protocol error", hex(status)) + check(await files.read("/empty") == b"", tag + " still serving after a protocol error") + + # Answers never share a notification: CircuitPython's client needs each + # header at the start of one. + starts = (ft.READ_DATA, ft.WRITE_PACING, ft.DELETE_STATUS, ft.MKDIR_STATUS, ft.LISTDIR_ENTRY, ft.MOVE_STATUS) + check(sum(1 for p in loop.packets if p[0] in starts) > 20, tag + " answers start notifications") + + ft._remove(root) + + +async def main(): + for mtu, window in ((23, 512), (247, 4096), (247, 512), (185, 1000)): + try: + await run(mtu, window) + except Exception as e: + check(False, "mtu {} window {}: the run ended early".format(mtu, window), repr(e)) + print("{} passed, {} failed ({})".format(passed, failed, sys.implementation.name)) + if failed: + sys.exit(1) + + +asyncio.run(main()) diff --git a/tests/test_bledev_filetransfer.py b/tests/test_bledev_filetransfer.py new file mode 100644 index 00000000..57327ea2 --- /dev/null +++ b/tests/test_bledev_filetransfer.py @@ -0,0 +1,55 @@ +# SPDX-FileCopyrightText: 2026 Brad Barnett +# +# SPDX-License-Identifier: MIT +"""Run the file-transfer protocol checks (``bledev_filetransfer.py``) on CPython +and MicroPython, and once per planted link fault, each of which must fail. +The same protocol over a real radio is ``bledev_board/files_client.py``. +""" + +from pathlib import Path +import os +import subprocess +import sys +import unittest + +from test_bledev import _micropython + +_TESTS = Path(__file__).resolve().parent +SCRIPT = _TESTS / "bledev_filetransfer.py" +PLANTS = ("flip", "drop") + + +def _run(interpreter, *args): + return subprocess.run([interpreter, str(SCRIPT), *args], capture_output=True, text=True, timeout=600) + + +class BledevFileTransfer(unittest.TestCase): + def assertPasses(self, result): + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + self.assertIn(" 0 failed", result.stdout) + + def test_cpython(self): + self.assertPasses(_run(sys.executable)) + + def test_micropython(self): + interpreter = _micropython() + if interpreter is None: + message = "no micropython binary: the file-transfer protocol did NOT run on MicroPython" + if os.environ.get("PYDEVICES_REQUIRE_MICROPYTHON") == "1": + self.fail(message) + sys.stderr.write("\n*** " + message + " ***\n") + self.skipTest(message) + result = _run(interpreter) + self.assertPasses(result) + self.assertIn("(micropython)", result.stdout) + + def test_every_plant_fails(self): + for plant in PLANTS: + with self.subTest(plant=plant): + result = _run(sys.executable, "--plant=" + plant) + self.assertIn("PLANTED " + plant, result.stdout) + self.assertNotEqual(result.returncode, 0, "plant {!r} went unnoticed:\n{}".format(plant, result.stdout)) + + +if __name__ == "__main__": + unittest.main()