Skip to content

iOS 27 camera: trace what the preview's pictures hold, in cameracaptured and backboardd (black viewfinder narrowed, not fixed) - #400

Open
claudiusthebot wants to merge 1 commit into
mainfrom
fix/ios27-camera-pixel-trace
Open

claudiusthebot wants to merge 1 commit into
mainfrom
fix/ios27-camera-pixel-trace

Conversation

@claudiusthebot

Copy link
Copy Markdown
Collaborator

What was wrong

The cause is not found yet, and this PR does not fix the camera. It adds the trace that names the step where the
preview turns black.

Phone 1's dylib -30 boot (18 Nov, 24B156, cam_trace=1 cam_metal=1; console scion-phone1/evidence/cd-cam30-metal on the
build Mac) ran the whole preview pipeline, read on 20 Nov at the desk:

  • Verified in the console:
    • cameracaptured did not abort.
    • Every node from the source node to BWImageQueueSinkNode emitted 828x1702 420f IOSurfaces, about 8 a second, and
      none were dropped. The sink rendered them into its CAImageQueue (fences on).
    • At the sink's first sample, backboardd made a format 500 (YCBCR8_420_2P) texture of 828x1702 over the surface.
    • No Metal refusal in either process while the camera ran. backboardd had used 18 of its 64 log lines, so a refusal
      would have shown.
    • No cross-process shared-event request.
    • Camera.app logged DidDisplayFirstPreviewFrame.
    • The screenshot's viewfinder is pure black, (0,0,0) at every point.
  • Inferred:
    • The window server composited the preview with the GPU rather than on a display layer. There is no SWEET FB: layer N
      line and no new framebuffer call. The framebuffer driver would silently drop a 420f layer, but its swap log was
      already used up, so this is not verified.
    • CMCapture's secure-indicator check does not blacken our frames: they carry none of the metadata keys it reads.
  • Not checked by any trace: the pixels. A zeroed 420f picture would show green, not black. Pure black fits three
    places, and the console cannot tell them apart:
    1. the pictures handed to the image queue are black (a node or its GPU writes in cameracaptured);
    2. backboardd's format 500 texture reads black (the YCbCr conversion in metal2vulkan on PanVK, or the pages as
      backboardd sees them);
    3. backboardd never samples the surfaces.

What changed (reading only)

  • tools/ios27/camera/camera_pixeltrace.c (cam.dylib, with cam_trace=1):
    • Reads the 1st, 30th and 300th sample of each node output, and the image queue sink's.
    • The read is a 16x16 grid of luma and an 8x8 grid of chroma, through CoreVideo's read lock.
    • It reads before and after cleaning the picture's cache lines, and gives the IOSurface ID.
    • Called from camera_flowtrace.c.
  • tools/ios27/metal2/YCbCrTrace.h (StarbiteMetal frontend, always on, at most 24 lines a process):
    • Reads the same grid as that process sees the surface.
    • It does so when a two-plane YCbCr texture is made over an IOSurface, and when an encoder first binds one over each
      surface.
    • The frontend gets 3 hook lines. arch-size.allow goes from 2764 to 2767 for them; the helpers are in their own header.
  • docs/ios27/CAMERA.md: what the -30 console shows and what it cannot, and how one boot with both traces names the step.

How it was checked

  • cam.dylib built with tools/ios27/camera/build.sh:
    sha256 4fcb6dabae6e6bc2f7d485607b72198d422fbfa691a8d9442d1fc7979a3bbe5c.
  • StarbiteMetal built with mk-bundle27.sh (every frontend patch applies), using the 12 Nov bridge:
    sha256 8e2757a0c60ab1e2dc4b247cb84e7853b136a7277b5f1ba1a1ee171f1c94b101.
    The frontend also compiles without the patches.
  • Host tests on macOS, both under ASan/UBSan:
    • camera_pixeltrace.c on CoreVideo buffers: black 420f, zeros, 420v with a bright square, BGRA, and an unread
      2vuy;
    • the header on IOSurfaces: 420f, and xf20, which is not read. "Bound" is said once per surface.
  • Repo checks pass: provenance-check.py, arch-check.py --report, docs-check.py, boot-args.py --check,
    drivers-check.py, ios27/builds/builds-test.py and fullroot/build-lock-test.sh.
  • npm run preflight does not apply: this repo has no package.json.

Built, not phone-tested. The phone test (the root and line of dylib -30, the two files swapped, telemetry on) and the
table that turns its lines into the next step are in TEST-ON-PHONE.md of the desk work (~/s9/agents/camera-desk-1120
on the build Mac).

…ameracaptured and in backboardd

Phone 1's dylib -30 boot (18 November) ran the whole preview pipeline: every node
emitted 828x1702 '420f' IOSurfaces, the image queue sink rendered them into its
CAImageQueue, backboardd made a YCBCR8_420_2P (500) texture over one, and nothing
was refused; the viewfinder was pure black. The traces count samples and never
read a pixel, so the console cannot say whether the pictures handed to the image
queue are black, whether backboardd's texture reads black, or whether backboardd
never draws them.

- camera_pixeltrace.c (cam.dylib, cam_trace=1): for each node output's 1st, 30th
  and 300th sample and the image queue sink's, a 16x16 luma / 8x8 chroma grid read
  through CoreVideo's read lock, before and after cleaning the picture's cache
  lines, with the IOSurface ID.
- YCbCrTrace.h (StarbiteMetal frontend, always on, 24 lines a process): the same
  grid as that process reads the surface, when a two-plane YCbCr texture is made
  over an IOSurface and when an encoder first binds one over each surface.
- CAMERA.md: what the -30 console shows, what it cannot, and how one boot with
  both traces names the step. arch-size.allow: the frontend +3 lines (the hooks;
  the helpers are in their own header).

Reading only; nothing else changes. Built, not run on a phone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant