Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions docs/ios27/CAMERA.md
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,58 @@ What phone 1 showed with dylib `-28` (18 November, 24B156, rear camera, `cam_tra
property as Apple's display does (`{PlaneBaseAlignmentLinear = 64}`); not done here, because every process reads that
property (QuartzCore among them) and it is not the camera's to change. **`-30` not run on a phone** when this was written.

### Dylib `-30` on the phone, and where the picture is lost (20 November, desk work)

Phone 1 booted dylib `-30` on 18 November (15:39, rear camera, the line `c272m`: `cam_trace=1 cam_metal=1`; console
`scion-phone1/evidence/cd-cam30-metal`, on the build Mac). Read from that console on 20 November, with no phone:

- **Verified in the console.** cameracaptured did not abort. The preview pipeline was built and connected (`Preview Governor`
to `BWImageQueueSinkNode`), and every node on the path emitted: the source node's `Preview(PrimaryScaler)` 1504 x 1128
`420f`, the stitcher, `Preview Scaler`, `Preview Splitter`, `BWStreamingFilterNode` and `Assistive Overlay` 828 x 1702 `420f`,
each an IOSurface, about 8 a second, none dropped. The image queue sink rendered them (300 by 15:42:06) into its image queue
(`<CAImageQueue ...>`, fences on). At the sink's first sample backboardd made a texture of format 500 (`YCBCR8_420_2P`,
usage ShaderRead) of 828 x 1702 over the surface: `MTL2[57 backboardd]: native-metal: texture memory fmt 500 ...`. Neither
process logged a Metal refusal while the camera ran (backboardd had used 18 of its 64 lines, so a refusal would have
shown), and no `MISSING ... newSharedEventHandle` (a cross-process shared event, which the bridge does not have). Camera.app
logged `DidDisplayFirstPreviewFrame`. The screenshot of the viewfinder is pure black, (0, 0, 0) at every point.
- **Inferred, not seen.** No second display layer: the framebuffer driver (`SweetFramebuffer.cpp`, the xnu fork) takes up to
four layers in a swap but draws only 4-byte (BGRA) surfaces, so a `420f` surface on a layer of its own would be dropped
without a line. The console has no `SWEET FB: layer N surface ... at` line (the driver says each position of a layer above
the first) and no new framebuffer call after the camera began, so the window server most likely composited the preview
with the GPU (hence the format 500 texture), not on a layer of the display. That swap log was used up before the camera
ran, so this is not verified.
- **Inferred: the secure indicator does not blacken it.** iOS 27's CMCapture blackens or counts frames that its ISP marks
(`BWInvalidFramesChecker`, the frame metadata's `InvalidDueToNoSensorAccess` and `SensorAccess`). `camera_device.c`'s
frames carry neither, so the check sees valid frames. The console has no `blackened` line (CAMLOG takes these only when
they name the camera).
- **Not checked by any trace: what the pictures hold.** The flow trace counts samples and says their size and format. Nothing
read a pixel. A zeroed `420f` picture would show green, not black (Y 0 with Cb and Cr 0 converts to green). A black one
is Y 0 or 16 with Cb and Cr 128. A YCbCr texture whose conversion never ran reads (0, 0, 0) with alpha 1. So pure black fits
three places equally well, and the console cannot tell them apart:
1. the pictures handed to the image queue are black (a node in cameracaptured, or its GPU writes not reaching the surfaces);
2. backboardd draws the image queue but its format 500 texture reads black (the conversion in metal2vulkan on PanVK, or the
surface's bytes as backboardd's GPU reads them);
3. backboardd never draws the image queue's surfaces (the layer not drawn, or not by a texture over them).

**The trace that decides it** (this change; trace only, nothing else changes):
- `camera_pixeltrace.c` (cam.dylib, cameracaptured, with `cam_trace=1`): for the 1st, 30th and 300th sample of every node
output, and of the image queue sink, the processor reads the picture through CoreVideo's read lock: a 16 x 16 grid of luma
and an 8 x 8 grid of chroma. It reads them once as the cache has them, and once after cleaning and invalidating the
picture's lines. Said: `trace: pixels: <node>'s output <name>, sample N: 828x1702 '420f' surface S: Y mean M least L most H,
K of 256 above black; Cb B Cr R` (and `; after the cache's clean: ...` where the two differ).
- `tools/ios27/metal2/YCbCrTrace.h`, in `NativeBridgeMetalFrontend.mm` (StarbiteMetal, every Metal client, always on, at most
24 lines a process): the same grid, read in that process, when a two-plane YCbCr texture over an IOSurface is made and
when an encoder first binds a texture over each surface: `native-metal: ycbcr texture made: fmt 500 over surface S '420f'
828x1702 plane 0: Y mean ...` and
`native-metal: ycbcr texture bound (NBRenderEncoder, stage 2, slot N): ...`.

How to read one boot with both: the first `trace: pixels:` line on the path whose grid is black names the node in
cameracaptured (1). If the sink's pictures hold the room, backboardd's `ycbcr texture made` line for the same surface ID says
whether its pages hold it too. No `ycbcr texture bound` line means the surface is never sampled (3). Bound, holding the
picture, and still black on the screen means the conversion or the draw (2). Then the runtime's telemetry
(`/var/local/mtl2-telemetry` present, the rows in `/var/local/mtl2.log`) is the next thing to read. `TEST-ON-PHONE.md` of
the desk work (`~/s9/agents/camera-desk-1120` on the build Mac) has the steps. **Built, not run on a phone.**

### SpringBoard's main-display layout (18 November)

`camera_sbtrace.c` on phone 1 (dylib `-16`, `-17`) shows that SpringBoard does publish its layout: the home screen, the lock
Expand Down
2 changes: 1 addition & 1 deletion tools/arch-size.allow
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ tools/ios27/fullroot/pipeline/cache/aot/fp16_stub.py 1424
tools/ios27/fullroot/pipeline/cache/aot/rewrite-grow.py 1047
tools/ios27/lse/lse-rewrite.py 826
tools/ios27/mesa-ios/starbite-drm/starbite_drm.c 985
tools/ios27/metal2/NativeBridgeMetalFrontend.mm 2764
tools/ios27/metal2/NativeBridgeMetalFrontend.mm 2767
tools/ios27/metal2/bridge/NativeMetalObjects.rs 2440
tools/mesa/air/air_intrinsics.c 2716
tools/mesa/air/air_metal.c 7072
Expand Down
4 changes: 2 additions & 2 deletions tools/ios27/camera/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ SDK=$(xcrun --sdk iphoneos --show-sdk-path)
xcrun --sdk iphoneos clang -arch arm64 -isysroot "$SDK" -miphoneos-version-min=16.0 -O2 -Wall -Wextra -Wno-unused-parameter -Wno-deprecated-declarations \
-dynamiclib -install_name /usr/lib/cam.dylib \
-DHAKIOS_CAMERA_N104 -DHAKIOS_CAMERA_CONSOLE '-DHAKIOS_CAMERA_FW="/private/var/hakios/camera"' -I"$SRC" -I"$HERE" \
-framework CoreFoundation -framework CoreMedia -framework CoreVideo -lobjc \
-o "$OUT" "$HERE/camera27.c" "$HERE/camera_trace.c" "$HERE/camera_oslog.c" "$HERE/camera_formats.c" "$HERE/camera_apptrace.c" "$HERE/camera_statetrace.c" "$HERE/camera_layout.c" "$HERE/camera_refuse.c" "$HERE/camera_flowtrace.c" "$HERE/camera_colour.c" "$HERE/camera_sbtrace.c" "$HERE/camera_tenbit.c" "$HERE/camera_outputs.c" "$HERE/camera_flownext.c" "$HERE/camera_previewtrace.c" "$HERE/camera_stills.c" "$HERE/camera_connecttrace.c" "$HERE/camera_metal.c" \
-framework CoreFoundation -framework CoreMedia -framework CoreVideo -framework IOSurface -lobjc \
-o "$OUT" "$HERE/camera27.c" "$HERE/camera_trace.c" "$HERE/camera_oslog.c" "$HERE/camera_formats.c" "$HERE/camera_apptrace.c" "$HERE/camera_statetrace.c" "$HERE/camera_layout.c" "$HERE/camera_refuse.c" "$HERE/camera_flowtrace.c" "$HERE/camera_colour.c" "$HERE/camera_sbtrace.c" "$HERE/camera_tenbit.c" "$HERE/camera_outputs.c" "$HERE/camera_flownext.c" "$HERE/camera_previewtrace.c" "$HERE/camera_stills.c" "$HERE/camera_connecttrace.c" "$HERE/camera_metal.c" "$HERE/camera_pixeltrace.c" \
"$SRC/camera_device.c" "$SRC/camera_say.c" "$SRC/camera_isp.c"
codesign -f -s - "$OUT" 2>/dev/null
[ "$(lipo -archs "$OUT")" = arm64 ] || { echo "build.sh: $(basename "$OUT") is not plain arm64" >&2; exit 1; }
Expand Down
18 changes: 17 additions & 1 deletion tools/ios27/camera/camera_flowtrace.c
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@
* stamp is from the host clock's now), and every 10 seconds how many each has emitted;
* - every node's dropped samples (the first five of each node, then every 300th), with their reason;
* - the preview stitcher's and the image queue sink's first sample, the sink's image queue (nil: nothing to show), and
* the sink's own dropped samples.
* the sink's own dropped samples;
* - what the pictures hold (camera_pixeltrace.c): each output's 1st, 30th and 300th sample and the image queue sink's.
* Each method is wrapped only if the class itself has it with the number of arguments and return type of iOS 27.0.1's
* CMCapture (24A446's shared cache, read 18 November 2026: names and types only); one that differs is left alone and said.
* The source node's own output methods (_outputConfigurationForStreamingOutputID:nodeOutput: and its kin) are direct
Expand Down Expand Up @@ -168,10 +169,16 @@ scioncam_flow_emit(void *output_object, void *sample_buffer)
if (!scioncam_flow_on) {
return;
}
char where[160] = "";

pthread_mutex_lock(&flow_lock);
f = flow_of(output, &fresh);
if (f != NULL) {
f->emitted++;
if (scioncam_pixels_due(f->emitted)) {
snprintf(where, sizeof(where), "%s's output %s, sample %llu", f->node ? class_getName(f->node) : "?",
f->output, (unsigned long long)f->emitted);
}
}
if (fresh) {
sample_of(sample_buffer, what, sizeof(what));
Expand All @@ -180,6 +187,9 @@ scioncam_flow_emit(void *output_object, void *sample_buffer)
}
summary(mach_absolute_time());
pthread_mutex_unlock(&flow_lock);
if (where[0] != '\0') {
scioncam_pixels(where, sample_buffer); /* camera_pixeltrace.c: what the picture holds */
}
}

/* ---- the methods wrapped ------------------------------------------------------------------------------------------- */
Expand Down Expand Up @@ -229,6 +239,12 @@ w_iq_render(id self, SEL sel, void *sb, id input)
bool first = count == 0;

first_render(self, "the preview's image queue sink", sb, &count);
if (scioncam_pixels_due(count)) {
char where[80];

snprintf(where, sizeof(where), "the image queue sink's sample %llu", (unsigned long long)count);
scioncam_pixels(where, sb); /* what the picture handed to the image queue holds */
}
((void (*)(id, SEL, void *, id))o_iq_render)(self, sel, sb, input);
if (first) {
char q[160];
Expand Down
162 changes: 162 additions & 0 deletions tools/ios27/camera/camera_pixeltrace.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
/*
* SPDX-License-Identifier: MIT-0
*
* What the preview's pictures hold, traced (cam_trace=1; docs/ios27/CAMERA.md, "What the pictures hold"): so that one boot
* with a black viewfinder names the first step whose picture is black, not only the steps that pass a sample on.
*
* Phone 1's last traced boot (18 November, dylib -30, cam_metal=1) had every node of the preview emit its samples, the
* image queue sink hand 828 x 1702 '420f' IOSurfaces to its image queue and backboardd make a YCBCR8_420_2P (500) texture
* over one of them, and the viewfinder pure black. The flow trace counts samples; nothing said what they held. Here, for the
* pictures of the preview's path (each node output's 1st, 30th and 300th sample, and the image queue sink's), the processor
* reads the picture through CoreVideo's read lock and says, from a 16 x 16 grid of luma and an 8 x 8 grid of chroma:
* the luma's mean, least and most, the share of the grid above black, and the chroma's means (128 is grey: a black 420f
* picture is Y 0 or 16, Cb Cr 128; a picture of zeros is Y 0, Cb Cr 0, which the screen would show green). It reads them
* twice: as the processor's cache has them, and again after cleaning and invalidating the picture's lines
* (sys_dcache_flush), so a picture that the GPU wrote and the processor's cache still hides is said as such ("after the
* cache's clean: ..."). The IOSurface's ID is said, so that backboardd's line about the same surface (StarbiteMetal's
* "ycbcr texture ... over surface N") can be put beside it.
*
* Reading is all it does: nothing is written, no picture is held past the call. NOT RUN ON A PHONE.
*/
#include <CoreFoundation/CoreFoundation.h>
#include <CoreMedia/CoreMedia.h>
#include <CoreVideo/CoreVideo.h>
#include <IOSurface/IOSurfaceRef.h>
#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include "camera_trace.h"

void hakios_camera_say(const char *format, ...) __attribute__((format(printf, 1, 2))); /* camera_say.c */
void sys_dcache_flush(void *start, size_t length); /* libSystem */

struct grid {
unsigned mean, least, most, lit; /* luma (or for BGRA, the mean of R, G and B); lit: of 256 points, above 24 */
unsigned cb, cr; /* chroma means (BGRA: 0) */
};

static void
read_grid(CVPixelBufferRef pb, uint32_t pf, struct grid *g)
{
size_t w = CVPixelBufferGetWidth(pb), h = CVPixelBufferGetHeight(pb);
unsigned long sum = 0, n = 0;

memset(g, 0, sizeof(*g));
g->least = 255;
if (pf == kCVPixelFormatType_32BGRA) {
const uint8_t *base = CVPixelBufferGetBaseAddress(pb);
size_t row = CVPixelBufferGetBytesPerRow(pb);

for (unsigned y = 0; base != NULL && y < 16; y++) {
for (unsigned x = 0; x < 16; x++) {
const uint8_t *p = base + (h * (2 * y + 1) / 32) * row + (w * (2 * x + 1) / 32) * 4;
unsigned v = ((unsigned)p[0] + p[1] + p[2]) / 3;

sum += v, n++;
g->least = v < g->least ? v : g->least;
g->most = v > g->most ? v : g->most;
g->lit += v > 24;
}
}
g->mean = n ? (unsigned)(sum / n) : 0;
return;
}
const uint8_t *luma = CVPixelBufferGetBaseAddressOfPlane(pb, 0), *chroma = CVPixelBufferGetBaseAddressOfPlane(pb, 1);
size_t lrow = CVPixelBufferGetBytesPerRowOfPlane(pb, 0), crow = CVPixelBufferGetBytesPerRowOfPlane(pb, 1);
size_t cw = CVPixelBufferGetWidthOfPlane(pb, 1), ch = CVPixelBufferGetHeightOfPlane(pb, 1);
unsigned long cbs = 0, crs = 0, cn = 0;

for (unsigned y = 0; luma != NULL && y < 16; y++) {
for (unsigned x = 0; x < 16; x++) {
unsigned v = luma[(h * (2 * y + 1) / 32) * lrow + w * (2 * x + 1) / 32];

sum += v, n++;
g->least = v < g->least ? v : g->least;
g->most = v > g->most ? v : g->most;
g->lit += v > 24;
}
}
for (unsigned y = 0; chroma != NULL && y < 8; y++) {
for (unsigned x = 0; x < 8; x++) {
const uint8_t *p = chroma + (ch * (2 * y + 1) / 16) * crow + (cw * (2 * x + 1) / 16) * 2;

cbs += p[0], crs += p[1], cn++;
}
}
g->mean = n ? (unsigned)(sum / n) : 0;
g->cb = cn ? (unsigned)(cbs / cn) : 0;
g->cr = cn ? (unsigned)(crs / cn) : 0;
}

static void
grid_text(const struct grid *g, bool bgra, char *out, size_t cap)
{
if (bgra) {
snprintf(out, cap, "RGB mean %u least %u most %u, %u of 256 above black", g->mean, g->least, g->most, g->lit);
} else {
snprintf(out, cap, "Y mean %u least %u most %u, %u of 256 above black; Cb %u Cr %u", g->mean, g->least, g->most,
g->lit, g->cb, g->cr);
}
}

/* the picture of `sample_buffer`, said with `where` (for example "BWPreviewStitcherNode's output ?, sample 30") */
void
scioncam_pixels(const char *where, void *sample_buffer)
{
CVImageBufferRef ib = sample_buffer != NULL ? CMSampleBufferGetImageBuffer((CMSampleBufferRef)sample_buffer) : NULL;
struct grid before, after;
char a[160], b[160];

if (ib == NULL || CFGetTypeID(ib) != CVPixelBufferGetTypeID()) {
return;
}
CVPixelBufferRef pb = (CVPixelBufferRef)ib;
uint32_t pf = CVPixelBufferGetPixelFormatType(pb);
bool bgra = pf == kCVPixelFormatType_32BGRA;

if (!bgra && pf != kCVPixelFormatType_420YpCbCr8BiPlanarFullRange && pf != kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange) {
hakios_camera_say("trace: pixels: %s: '%c%c%c%c' is not read here", where, pf >> 24, pf >> 16 & 255, pf >> 8 & 255,
pf & 255);
return;
}
if (CVPixelBufferLockBaseAddress(pb, kCVPixelBufferLock_ReadOnly) != kCVReturnSuccess) {
hakios_camera_say("trace: pixels: %s: the picture could not be locked for reading", where);
return;
}
IOSurfaceRef surface = CVPixelBufferGetIOSurface(pb);

read_grid(pb, pf, &before);
/* the picture's lines cleaned and invalidated: what the GPU wrote, where the processor's cache held older lines */
if (CVPixelBufferIsPlanar(pb)) {
for (size_t p = 0; p < CVPixelBufferGetPlaneCount(pb); p++) {
void *base = CVPixelBufferGetBaseAddressOfPlane(pb, p);

if (base != NULL) {
sys_dcache_flush(base, CVPixelBufferGetBytesPerRowOfPlane(pb, p) * CVPixelBufferGetHeightOfPlane(pb, p));
}
}
} else if (CVPixelBufferGetBaseAddress(pb) != NULL) {
sys_dcache_flush(CVPixelBufferGetBaseAddress(pb), CVPixelBufferGetBytesPerRow(pb) * CVPixelBufferGetHeight(pb));
}
read_grid(pb, pf, &after);
CVPixelBufferUnlockBaseAddress(pb, kCVPixelBufferLock_ReadOnly);
grid_text(&before, bgra, a, sizeof(a));
if (memcmp(&before, &after, sizeof(before)) == 0) {
hakios_camera_say("trace: pixels: %s: %zux%zu '%c%c%c%c' surface %u: %s", where, CVPixelBufferGetWidth(pb),
CVPixelBufferGetHeight(pb), pf >> 24, pf >> 16 & 255, pf >> 8 & 255, pf & 255,
surface != NULL ? IOSurfaceGetID(surface) : 0, a);
} else {
grid_text(&after, bgra, b, sizeof(b));
hakios_camera_say("trace: pixels: %s: %zux%zu '%c%c%c%c' surface %u: %s; after the cache's clean: %s", where,
CVPixelBufferGetWidth(pb), CVPixelBufferGetHeight(pb), pf >> 24, pf >> 16 & 255, pf >> 8 & 255, pf & 255,
surface != NULL ? IOSurfaceGetID(surface) : 0, a, b);
}
}

/* whether the `count`th sample (from 1) of a picture-carrying output is one to read: the 1st, 30th, 300th, then every 3000th */
bool
scioncam_pixels_due(unsigned long long count)
{
return count == 1 || count == 30 || count == 300 || (count > 300 && count % 3000 == 0);
}
4 changes: 4 additions & 0 deletions tools/ios27/camera/camera_trace.h
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ void scioncam_trace_flow(void);
void scioncam_flow_emit(void *output, void *sample_buffer);
/* camera_flownext.c: at an output's first sample, the node it leads to, said, and that node's render method traced */
void scioncam_flow_next(void *output, const char *node_class, const char *output_name);
/* camera_pixeltrace.c (with the flow trace): what a sample's picture holds, read by the processor from a grid, said with
`where`; scioncam_pixels_due says which samples of an output are read (the 1st, 30th, 300th, then every 3000th) */
void scioncam_pixels(const char *where, void *sample_buffer);
_Bool scioncam_pixels_due(unsigned long long count);
/* camera_previewtrace.c: the session configuration's preview sinks, said (with the flow trace) */
void scioncam_trace_preview(void);

Expand Down
3 changes: 3 additions & 0 deletions tools/ios27/metal2/NativeBridgeMetalFrontend.mm
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ static void noteFormat(uint32_t format,bool made) {
for(unsigned i=0;i<gFormatCount;i++)if(gFormats[i]==format)return;
if(gFormatCount<8)gFormats[gFormatCount++]=format;
}
#include "YCbCrTrace.h"
static NSError *nbError(NSString *why) {
return [NSError errorWithDomain:@"org.blu-sharky.native-metal" code:-1 userInfo:@{NSLocalizedDescriptionKey:why?:@"refused"}];
}
Expand Down Expand Up @@ -932,6 +933,7 @@ - (void)texture:(id<MTLTexture>)texture index:(NSUInteger)index stage:(uint32_t)
[_commandBuffer.surfaces addObject:t];
const MTLTextureUsage u=t.usage;
if(!u || (u&MTLTextureUsageShaderWrite))[_commandBuffer.writtenSurfaces addObject:t];
if(sweetYCbCrFormat(t.descriptor.format))sweetYCbCrBound(object_getClassName(self),stage,index,t.descriptor.format,t.surface,t.surfacePlane);
}
[self check:nmb_encoder_set_texture(_handle,stage,texture?[(NBTexture *)texture handle]:nullptr,(uint32_t)index) what:"setTexture"];
}
Expand Down Expand Up @@ -2260,6 +2262,7 @@ - (id)newTextureLayoutWithDescriptor:(MTLTextureDescriptor *)d isHeapOrBufferBac
NBTexture *texture=[NBTexture new];texture.handle=handle;nmb_texture_describe(handle,&descriptor);texture.descriptor=descriptor;
texture.buffer=buffer;texture.offset=(NSUInteger)offset;texture.rowBytes=(NSUInteger)rowBytes;
texture.surface=(IOSurfaceRef)CFRetain(surface);texture.surfacePlane=plane;
if(sweetYCbCrFormat(descriptor.format))sweetYCbCrMade(descriptor.format,surface,plane);
return (id<MTLTexture>)texture;
}
- (id<MTLLibrary>)newLibraryWithData:(dispatch_data_t)data error:(NSError **)error {
Expand Down
Loading
Loading