Status: on the first console run the sensors answer but no picture data arrives (section 7); diagnostics and fallbacks are in for the next run. In QEMU Camera.app shows a live preview and takes photos through the whole path (QEMU camera model → ARM9 sensor set-up → ARM11 FIFO drain → mediaserverd plug-in → Celestial → Camera.app → Camera Roll). Video recording is not offered (section 8).
Camera.app / UIImagePickerController / AVFoundation (in the app)
| FigRecorderRemote (Mach IPC "com.apple.coremedia.recorder")
mediaserverd: Celestial FigRecorder -> FigLiveSource -> FigCaptureDevice/Stream
| figCreateVideoCaptureDevice loads /System/Library/MediaCapture/vcd.mediacapture
| (CFPreference "vcd" of com.apple.coremedia = true) before any Apple camera
vcd.mediacapture (tools/ios3ds-camera/vcd, ours) YUV422 -> BGRA, CMSampleBuffers
| sysctl ios3ds_cam (CAM_OP_START / FRAME / STOP) in CMBufferQueues
kernel shim, GPU part (kernel/shim/cam.c) 3 frame buffers, IOMallocContiguous
| every 1 ms while a camera runs (audio service wake-ups): drain the CAM FIFO window
| camera mailbox (shim + 0x7E00)
ARM9 monitor (boot/arm9/camera.c): CFG11_CAMERA_CNT clock, Aptina MT9V113 set-up over I2C
3DS: inner camera (I2C 0x10161000 dev 0x7A) and outer left camera (I2C 0x10144000 dev 0x78)
-> CAM0 0x10120000 / CAM1 0x10121000, FIFO windows 0x10320000 / 0x10321000 (ARM11 only)
QEMU evidence (--icount 2, the camera model's pattern): live preview of the
outer camera, a photo, switch to the inner camera (two marker blocks), a photo,
Photos shows "Camera Roll (2)"; 144 frames per 10 s (14.4 fps), 0 FIFO
overruns, ARM9 sensor bring-up 5 ms per camera. Screenshots and the [cam]
log: build/camera-evidence/ (not committed).
iOS sees two cameras: the 3DS outer left camera is the "back" camera
(PortTypeBack), the inner camera the "front" one (PortTypeFront, the
switch button in Camera.app). The outer right camera is kept in standby (it
shares CAM0's parallel bus with the inner one).
| Fact | Where |
|---|---|
Camera.app (/Applications/Camera.app, in the iPod2,1 rootfs) is tagged SBAppTags = (still-camera); SpringBoard hides apps whose tags are not capabilities of the device |
Camera.app Info.plist |
The device capabilities come from SpringBoard.app/N72AP.plist capabilities (no camera keys on the iPod touch 2G); the iPod touch 4G's N81AP.plist has still-camera, front-facing-camera, video-camera, hd-video-capture |
both rootfs images |
Capture runs in mediaserverd: the app talks to FigRecorderRemote; the recorder, live source and capture devices are in Celestial inside mediaserverd (runs as mobile) |
Celestial symbols, mediaserverd launchd plist |
figCreateVideoCaptureDevice (Celestial 0x32e01520): first, if CFPreferencesGetAppBooleanValue("vcd", "com.apple.coremedia"), dlopen("/System/Library/MediaCapture/vcd.mediacapture") + dlsym("FigVirtualCameraDeviceCreate") (FigLoadAspenCaptureDevice 0x32e0133c); then IOKit AppleH4CamIn/H3/H2/H1 → H4ISP...H1ISP.mediacapture. A loaded virtual device wins; create = (allocator, NULL options, &device) |
disassembly |
The capture device / stream interface is Apple's FigBaseObject: vtable {0, base class, derived class}, base class {version 1, storage size, equal, invalidate, finalize, copyDebugDescription, copyProperty, setProperty}, stream class {1, start, stop}; created with FigDerivedObjectCreate(alloc, vtable, FigCapture{Device,Stream}GetClassID(), &obj) |
H3ISP.mediacapture (in the shared cache) 0x3e07e5b8.. |
Device properties used: StreamArray, Clock (host clock), SupportedPropertiesDictionary |
H3ISP device CopyProperty 0x3038c52c, Celestial |
Stream properties used: PortType (PortTypeBack/PortTypeFront), SupportedFormatsArray (dicts of FormatDescription, VideoMaxFrameRate, VideoScaleFactor, VideoIsBinned), FormatDescription, FormatIndex, Min/MaximumFrameRate, BufferQueue and StillImageBufferQueue (CMBufferQueues created by the plug-in, Celestial installs triggers on them), StillImageCaptureNow (the next frame also goes to the still queue), BufferAllocator; notifications StreamStarted/StreamStopped on FigCaptureDeviceGetNotificationCenter() |
H3ISP stream CopyProperty 0x30385a54, Celestial livesource_updateSourceVideoQueues 0x32e250e0 |
Frames: CMSampleBufferCreateForImageBuffer of IOSurface-backed CVPixelBuffers (kIOSurfaceIsGlobal), PTS = host time in ns |
H3ISP MyH3ISPFrameReceivedProc 0x3038afd0 |
Four constants are private to Celestial (bound into Apple's plug-ins by the cache, not visible to dlsym): PortType, PortTypeBack, PortTypeFront, FrameDroppedFromQueue; they are compared with CFEqual, so equal CFStrings work |
cache symbol tables |
Modes and formats come from Celestial.framework/<model>/AVCapture.plist; N72's only has audio recording. Schema taken from the iPod touch 4G (N81) file: AVCaptureMode_PhotoCapture, AVCaptureMode_FrontFacingPhotoCapture with LiveSourceOptions {VideoPort, Sensor, Capture, Preview (Width, Height, PixelFormatType), Min/MaxFrameRate, SensorOrientation, PreviewVideoMirror, FastSwitchCompanion} |
N81 rootfs (public firmware, looked at only) |
CoreAnimation hands 4:2:0 IOSurfaces of a video layer to an AppleM2CLCD overlay window (CADisplayIOMFB swap_queue), which needs physically contiguous memory and is not shown by ios3ds: 420f previews were black (AppleM2CLCD, swap_queue error: e00002c8). BGRA previews are composited like any other layer |
QEMU |
Photos: livesource_workerThread → FigAspenCreateJPEGFromSbuf → no AppleJPEG service on N72 → softwareJPEGEncode (ImageIO CGImageDestination) |
Celestial 0x32e25d40, 0x32e03b28 |
Options considered:
- A fake IOKit camera service (
AppleH3CamInuser client) under Apple's H3ISP.mediacapture: H3ISP drives an ISP with firmware upload, I2C, 3A and a frame receiver through dozens of user-client selectors. Emulating that in the shim is far more code than the camera itself, and none of it matches 3DS hardware. - Interposing CoreMedia/Celestial in mediaserverd (like the Apple80211 interposer): the capture device is created inside Celestial from a function table, not through calls an interposer could catch cleanly.
- Celestial's own virtual-camera slot (
vcd.mediacapture, chosen): Apple left a plug-in hook that is loaded before any hardware camera when one preference is set. The plug-in implements the same FigCaptureDevice interface as Apple's plug-ins, so everything above it (Celestial's live source, preview, still pipeline, JPEG, AVFoundation, UIImagePickerController, Camera.app) is Apple's unmodified code. No byte patch is needed anywhere.
Kernel/hardware side: the CAM data FIFO is ARM11-only, so the ARM11 must move the data. The 3DS's own driver uses CDMA (peripheral 2/3); ios3ds drains the FIFO with the CPU from the shim's interrupt path, every 1 ms while a camera runs, the way the microphone FIFO is drained (docs/audio.md section 7). That needs no DMA microcode and the QEMU model stays simple; the cost is bounded by the frame size (section 6). The I2C buses and the camera clock are the ARM9's (it already owns I2C for the MCU), so the sensor set-up is done there.
| Piece | File |
|---|---|
Plug-in FigVirtualCameraDeviceCreate: device with two streams, frame thread per running stream (sysctl → YUYV → BGRA IOSurface pool buffer → CMSampleBuffer → queues), stills, pattern mode (ios3dsPattern preference) |
tools/ios3ds-camera/vcd/vcd.c → /System/Library/MediaCapture/vcd.mediacapture |
Capabilities still-camera, front-facing-camera |
rootfs/recipe.json plist_merge (new: nested-dictionary merge in scripts/mkrootfs.py) |
Modes/formats: photo back + front (each the other's FastSwitchCompanion), 320x240 BGRA, 15 fps, SensorOrientation 90, no preview mirroring |
rootfs/files/camera/AVCapture.plist → Celestial.framework/N72/AVCapture.plist (our own file; it replaces N72's audio-only one and keeps its audio mode) |
vcd = true |
rootfs/files/camera/com.apple.coremedia.plist → /var/mobile/Library/Preferences/ |
| Shared contracts: sysctl requests, mailbox, register map | boot/common/cam_proto.h |
| Test pattern + YUV conversions (plug-in, QEMU model, host test) | boot/common/cam_pattern.h |
Shim: sysctl ios3ds_cam, FIFO drain, overrun recovery |
kernel/shim/cam.c (GPU part, link.ld *cam*.o), hooks in gpu_glue.c, audio.c |
ARM9: camera clock, I2C (16-bit register index added to i2c.c), Aptina set-up, standby |
boot/arm9/camera.c |
boot.cfg camera=1 (default) / camera=0 |
scripts/mkboot.py, boot/arm9/main.c |
| QEMU model | emu/qemu/hw/arm/ctr_cam.c |
| Host test | tools/ios3ds-camera/tests/test_cam.c, tests/test_camera.py |
First the CAM blocks, in the order the 3DS CAMERA sysmodule uses
(re-log 74): CAM0's CAM_CNT bit 4 (the reset of both blocks; it does not
exist on CAM1) held low while the camera clock (CFG11_CAMERA_CNT bit 0, the
module's pdn:c call) comes on, released with the clock running, clock off;
then on both blocks transfer off, overrun cleared, 256-byte transfer unit,
trimming off, CAM_CNT bits 8-9 = 2, bit 10 on, bit 11 off, bit 14 on; then
the clock on for good. Then the sensor:
GBATEK's DSi Aptina code list (reset, wakeup, PLL as on the DSi, YUV output,
sizes, AE window, sequencer refresh; the cam module's base script is the
same list plus 1 ms/10 ms waits, MCU 0xA111 = 0x0A and flicker-detection MCU
values, which are written too) with output size A = 320x240, then
"activate" (standby off, parallel output on). The read mode is GBATEK's
(0x24 for the 0x7A camera, 0x25 = x-flip for the others); which of them
gives a true or a mirrored picture is unverified (section 7). Each wait gives up after
100 ms; a camera that does not answer or does not come up reports
CAM_ST_FAILED and iOS shows a black preview. Switching cameras puts the
previous sensor into standby first.
- START maps the CAM registers (0x10100000 section, shared with the audio code) and the FIFO windows (0x10300000 → VA 0xBFB00000), allocates three 150 KiB frame buffers once, and posts the request to the ARM9.
- Once the ARM9 reports the sensor streaming, the interrupt path sets
the set-up variant's CAM_CNT with bit 15 (variant 0 = the cam module's
0xC610; bit 4 must stay set on CAM0, stopping clears bit 15 only; the transfer
starts at a frame start) and from then on, every 1 ms, reads
CAM_STAT's fill count and copies that many words from the window into the frame being filled. Window reads always continue at the next address (a word is only consumed when the address changes). - An overrun (stat bit 15) costs the frame: transfer off, overrun cleared, transfer on (next frame start), the partial frame dropped.
- FRAME copies the newest complete frame out (
copyout), never one the interrupt is writing. - The worst interval between two drain services (in shim ticks) is tracked and
published per session (
gap_max, the stats line printsdrain gap max). The FIFO holds about 5.5 ms of the stream: anything beyond that loses words even when the overrun bit stays clear - run 29 saw a FIFO parked at the fill cap 0x500 with CAM_STAT bit 15 still 0 (section 7). - Frame alignment (run 30's dump: one frame shifted by 16 pixels, its
right edge wrapped to the left): the shim cannot see the sensor's VSYNC,
but the vertical blank shows as two or more services in a row with an
empty FIFO (a line arrives every ~0.25 ms, the blank lasts milliseconds).
A frame counts only if its 38400th word is the last before such a blank:
once a blank has been seen, a complete frame waits for it and is published
there if no word came in between; a partial frame at a blank, or words
after a complete one, mean words were lost or doubled, and that frame is
dropped (
resyncs, printed as "frames dropped at the blank" in the 10 s line). The next frame starts aligned. A pause counts as the blank only within 256 words of a frame boundary (QEMU pauses mid-frame). Host test: a lossy FIFO model (8 words vanish every 3rd frame, 4 come twice every 5th) delivers only whole frames. - STOP cleans all three frame buffers; the ARM9 then copies the two newest complete frames out of them (section 4.1).
- Preview: the plug-in delivers BGRA frames (YUYV converted in mediaserverd), which CoreAnimation composites like any layer. 4:2:0 frames go to a CLCD overlay window instead and stay black on ios3ds (section 1).
- The 1-2 s loop (run 30): the frames come from a CVPixelBuffer pool, so
the same IOSurfaces are written again and again. CA binds each surface to
a texture once and keeps it (on Apple's GPU the texture is the surface
memory); ios3ds' PICA200 back end (
tools/ios3ds-gl/pica_be.c) keeps a tiled copy in GPU memory and re-uploaded it only when GL said the image changed. Run 30's[gl] CAline: 0.0 texture uploads per frame while the camera ran at 15 fps, so the preview cycled through the pool's first frames. Now a surface-backed texture is re-uploaded when the IOSurface's seed (IOSurfaceGetSeed, bumped by every write unlock) or a 64-texel sample of it changed. Video and any other producer of recycled surfaces benefit too. - Review image: Camera.app only saves a photo after Celestial has made its
review image. Celestial uses the sample's
ScaledPreviewPixelBufferattachment as it is when there is one; otherwise it scales the frame to 512x384 with VTPixelTransferSession, which on this device refuses to scale up (-12905) and the photo is silently dropped. The plug-in attaches each frame's own pixel buffer. - Photos: every still request is logged (
still requested (frame N), thenstill image #nor why not). Run 30 had none: the request never reached the plug-in (re-log 76); QEMU takes photos upright and face down. - Photos: on
StillImageCaptureNowthe next frame goes to the still queue at 640x480, interpolated from the 320x240 frame (cam_bgra_upscale2x); Celestial JPEG-encodes it in software (no AppleJPEG service on N72) and Camera.app saves it into the Camera Roll (/var/mobile/Media/DCIM). The detail is that of the 320x240 sensor output. SensorOrientation 90: the 3DS sensor delivers landscape frames; iOS's portrait is the console turned 90°, like an iPhone's landscape sensor.- Front/back switch: Celestial creates the device with the mode's stream and
its
FastSwitchCompanionstream and swaps them on the switch button (livesource_swapVideoStreams). The front preview is not mirrored: withPreviewVideoMirrorCelestial needs a mirroring preview scaler (livesource_setupVideoScalerForPreview), which fails without the hardware scaler (-12914), and the switch ended inPreviewFailed(re-log 65). Without a companion Camera.app's switch button does nothing.
Run 29's open question is word-level: the preview in Camera.app shows one
"first frame" that wiggles, while every counter says the data flows
(section 7). The pixel truth decides between the suspects, so at every camera
stop the ARM9 copies the shim's two newest complete 320x240 YUV422 frames
straight out of its buffers to sd:/ios3ds/CAM<seq>.RAW (one file per camera
session, ~300 KiB; [cam] dumped frames seq ... in the log):
header (u32 LE): 0x44554D43 ('CAMD'), 1, 320, 240, cam, port, seq1, seq2
frame 1 (newest), frame 2 - each 320*240*2 bytes, YUV422, no padding
On the PC: python3 tools/ios3ds-camera/analyze_dump.py CAM00001234.RAW
decodes both frames to BMPs (in both YUYV and UYVY byte orders, which settles
the colour question by eye) and prints
- the mean luma difference at zero shift: ~0 = the picture never changes (the kernel delivered the same scene),
- the best (dx, dy) alignment between the two frames: a constant nonzero drift between neighbouring frames = the 38400-word frame window slips through the stream (words lost), the signature of a picture that wanders,
- rows whose best dx disagrees with the global one: words lost mid-frame (horizontal shear).
The shim's drain gap max (worst service interval, ms) and the ARM9's
independent [cam] monitor line (its own CAM_STAT sampling (each monitor-loop turn, so never starved by XNU): max
fill, near-full count) say whether the drain was ever starved long enough to
lose words. Together they tell lost-words (FIFO full behind masked
interrupts; the fix is a CDMA drain, section 6) apart from a display-side
problem, without guessing.
emu/qemu/hw/arm/ctr_cam.c: three MT9V113 register models on I2C (answer only
while CFG11_CAMERA_CNT bit 0 is set), the two CAM blocks, their FIFO windows
(pop on address change, 0x500 units, overrun), and a moving test pattern at 15
fps on the virtual clock (cam_pattern.h: scrolling colour bars, a bouncing
square, one black block for the outer camera and two for the inner one, and
a grey 64x48 frame-counter patch in the middle, Y = 40 + frame mod 160).
tools/ios3ds-camera/tests/preview_counter.py reads that patch off
screenshots of a running session for 10 s: fresh frames make it climb, a
preview looping over old frames steps back (FAIL), a frozen one stays put.
emu/session.py set mcu-orient 5 lays the console face down (4 = face up).
IOS3DS_CAM_LOG=1 logs register writes and transfers.
QEMU's default timing (virtual clock = wall clock while the guest runs about
3x slower than a 3DS) overruns the FIFO on every frame: use --icount 2
(real-hardware instruction timing), where the 1 ms drain keeps up with 0
overruns. The plug-in's pattern mode (ios3dsPattern = true in
com.apple.coremedia's preferences) bypasses the kernel path entirely.
- 320x240 YUV422 is 150 KiB per frame: at 15 fps 288,000 FIFO words a second
through the ARM11, a 150 KiB
copyoutand a YUYV → BGRA conversion per frame in mediaserverd. - The FIFO holds about 5.5 ms of 320x240 video (run 29: 10 KiB against ~1.8 MB/s of sensor data). While XNU masks interrupts longer than that (md0 transfers do, docs/architecture.md) frames are lost (counted as overruns, logged when the camera stops).
- Photos are 320x240. VGA would need 4x the FIFO bandwidth; with the CPU drain that is only reasonable at a few frames per second (or with CDMA).
Run 23 (Old 3DS, re-log 68): all three sensors answer (chip id 2280),
the outer left one comes up in ~135 ms, but 0 frames, 0 FIFO overruns
every 10 s: no word ever reached the CAM1 FIFO. A third start timed out in
the sequencer refresh (init failed (step 5)). The shim's FIFO drain itself
is not in question yet: without overruns the FIFO never filled at all.
From the CAMERA sysmodule (re-log 74): what run 25 lacked is the block
reset with the clock running (CAM0 bit 4 had been set by whatever ran
before, with the clock off), bit 14 on both blocks, a transfer unit of whole
lines, and a stop that leaves bit 4 alone (the shim wrote CAM_CNT = 0, which
holds both blocks in reset). All four are now as the module does them;
variant 0 is the module's set-up (0xC610, 2560-byte unit). The next log
should show after the cam module's block reset/init with CAM0 4610,
CAM1 4600, then data in variant 0.
Run 25 (re-log 70): with these diagnostics: the sensor reports streaming
(SEQ_STATE 3, parallel output on, awake), yet neither block ever sees sync or
data in any variant; CFG11 bit 1 does not stick; CAM_CNT bit 4 sticks on CAM0
but not on CAM1. Now: variants with the DSi's CAM_CNT init bits (8-10) and a
boot self-test (cam_test=1 in boot.cfg, before XNU; until build 60 it ran with diag=1): every sensor x every block set-up
(and two sensor-side variants), one [cam-test] line each, <-- DATA where
anything arrives.
Run 29 (build 43, first hardware data): the build-43 fixes work. The
self-test: all 36 set-ups <-- DATA (each sensor on its mapped block -
outer left on CAM1, inner and outer right on CAM0). At runtime every session
streams: ~12 frames/s (450 FIFO words/ms, words = frames x 38400 exactly),
0 FIFO overruns, the plug-in delivers to Celestial and Celestial dequeues
everything (0 queue drops). But Camera.app's preview shows one "first frame"
whose content wiggles around - and one 5-s diag caught the CAM1 FIFO parked
at the fill cap (1280 units) with the overrun bit still clear, so a full
FIFO does not necessarily flag anything: drain starvation silently loses
words, which would displace the frame window per incident exactly that way.
So the capture stays officially "artifacts" (README) until the build-44 frame
dumps (section 4.1) say whether the delivered frames drift.
What the code does about it (all logged):
- Both CAM blocks are watched until data appears: if the left camera's
data arrives on CAM0 instead of CAM1, the shim moves there
(
CAM0/CAM1in the stats line). - Set-up variants, 2 s each, cycled until data appears
(
CAM_VAR_CNT_LIST/CAM_VAR_CLR_LISTinboot/common/cam_proto.h): the cam module's 0xC610 with +6 = 9 (4 lines of 320 pixels) first, then 0x8010; 0x8610 (the DSi's init bits 8-9 = 2, 10 = 1); 0x8E10 (+ bit 11); 0x8610 with +6 = 0x22 (GBATEK suspects +6 is the DSi's CAM_MCNT, "on" = 0x22); 0xC610 + DRQ with +6 = 0x22. CFG11_CAMERA_CNT= 3, read back (run 25: only bit 0 sticks).- Sensor waits 500 ms instead of 100 ms; the cameras are probed at boot (iOS saw "present 0x0" until the first start).
[cam-diag] lines in sd:/ios3ds/log.txt, in this order:
[cam-diag] boot: CFG11_CAMERA_CNT .. CAM0 cnt .. stat .. +6 .. CAM1 ...and the threechip idlines at boot.[cam-diag] CFG11_CAMERA_CNT xx -> yy: whether bit 1 sticks.- After
streaming ... into CAM1: the sensor registers (STANDBYbit 14 = in standby, should be 0;RESET_MISCbit 9 = parallel output, should be 1;SEQ_STATE: 3 = preview/streaming on the MT9V113 family;OUT_W_A/OUT_H_A= 0140/00f0). - At 0.2 s, 1 s, 5 s: both CAM blocks again (CAM_CNT as the shim set it), and per block over 50 ms: how often CAM_CNT bits 0/1 changed (GBATEK: they toggle during a transfer = the block sees sync from the sensor) and how often the FIFO fill changed.
- Every 10 s:
shim: N services, M with data, stat ..., drain gap max X ms, CAM_CNT ..., variant V, CAMp[, still probing: no data yet], W words.services= 0 means the interrupt-time drain never ran for the camera (a shim problem, not the hardware's).drain gap max> ~5 ms (the FIFO's worth of stream) means words were lost even ifoverrunsstayed 0. [cam] monitor 10 s / at stop: the ARM9's ownCAM_STATsampling (each monitor-loop turn, so it cannot be starved by XNU) - max FIFO fill and near-full count. This observer cannot be starved by XNU's interrupt masking, unlike the shim's.- At stop:
[cam] dumped frames seq ... to ios3ds/CAM....RAW(section 4.1).
How to read it:
- Bits 0/1 toggle and the fill changes on one block → data arrives; the variant/port in the stats line is the working set-up (make it the default).
- No toggles on either block, sensor reports streaming (
STANDBY0, parallel bit set) → the CAM block is not clocked or not enabled: look atCFG11_CAMERA_CNTand the CAM_CNT readback (bits that did not stick). - Sensor in standby /
SEQ_STATEnot 3 → the DSi code list does not bring the 3DS sensor to streaming; the sensor side needs the official sequence.
Other unverified points: the colour byte order (GBATEK: Y0 U Y1 V), the
picture's orientation/mirroring (SensorOrientation, read mode 0x2717/0x272D
bit 0 = x-flip), and whether a 1 ms CPU drain keeps up on the console (many
overruns would mean CDMA is needed). camera=0 in boot.cfg keeps the kernel
away from the cameras (the plug-in then reports no device).
- Video recording: Camera.app's video switch needs
video-cameraand an H.264 encoder; the iPod touch 2G has none (noVideoEncodersfor H.264, no hardware), so the capability stays off. - Focus, exposure, white balance and flash requests are accepted and ignored (the sensors run their own auto exposure/white balance).
- Mirrored front preview (section 4; the sensor's own x-flip could do it).
- The third (outer right) camera and 3D photos.
- VGA stills (section 6).