Delay Lab

See why AR punishes headset delay more than VR, on an ordinary laptop webcam.

🌐 Project site · ⬇️ Download DelayLab.exe · 🤝 Contribute (no webcam needed)

_{Simulated, not a webcam recording: a synthetic head swings side to side, and Delay Lab's real delay buffer, prediction and drawing code produce every frame. A smooth swing like this suits the fix better than a real head does.}

Tip

No headset needed. A laptop with a webcam is enough. Pick a delay with the slider, switch between V VR-style, A AR-style and F AR + fix, and watch the difference.

Important

Status: the app works; the main measurement has not been done yet.

✅ Verified	The app runs live on a laptop webcam (~30 fps, face found in ~100% of frames). All three modes, recording and the sweep run end to end. The packaged exe runs without Python. 26 tests pass.
⏳ Pending	The proper 20-second sweep recording the experiment is built around, and the clap test for the webcam's own delay. Want to be first? See docs/measurements.
⚠️ So far	The only real measurement is one 2.5 s self-test clip. It is a first sign, not a result (details).

Contents: The idea · Quick start · Controls · What you see · The experiment · How it works · Real numbers · Presenting · DelayLab.exe · Help · Contributing

---

💡 The idea in 30 seconds

1. Every headset is a little late. When you move your head, the picture catches up a few milliseconds afterwards.
2. In VR you can't see it. The whole fake world is late together, so there's nothing to compare against. You only feel it, as floatiness or, above about 50 ms, as sickness.
3. In AR you can. The real world is never late, so a virtual object visibly slides off the real thing it was stuck to.

Delay Lab adds a chosen delay to webcam face tracking, shows it both ways, measures the slide in pixels, and then switches on the fix real headsets use (prediction) to show how much of the slide disappears.

slip = head speed × delay. At a normal 100°/s head turn, 20 ms of delay moves an AR object 1 m away by about 3.5 cm. That's easy to spot on a cup on a table. (This chart is calculated from that formula and published latency figures. It is not a Delay Lab measurement.)

---

🚀 Quick start

You need Windows 10/11, a webcam and Python 3.11. MediaPipe can lag behind newer Python releases.

py -3.11 -m venv .venv && .venv\Scripts\activate
pip install -r requirements.txt
python download_model.py        # one-time, ~4 MB face model
python app.py

Check that everything works (self-test)

python app.py --selftest 8

The app drives itself: AR at 0 and 100 ms, AR + fix, VR, a short recording and a sweep. Then it prints the frames per second, how often a face was found, and the gap in each mode. Expect about 25–30 fps and a face in close to 100% of frames. Use --camera 1 if you have more than one webcam.

Prefer no Python at all? See DelayLab.exe.

---

🎮 Controls

Key	Mode / action	What happens
V	VR-style	Camera hidden. A wireframe room moves with your (delayed) head.
A	AR-style	Your live face, with a 🔴 sticker drawn at the delayed nose position. A 🟢 cross marks your real nose.
F	AR + fix	Same as AR, but the sticker is drawn where your head is predicted to be.
slider	delay	Extra delay added on top of the camera's own: 0–150 ms.
R	record	Start or stop recording. Saves head_*.mp4 (raw camera, for the sweep) and demo_*.mp4 (what the window shows).
S	sweep	Replays the newest recording at every delay, without and with the fix, and saves a chart.
W	fullscreen	Toggles fullscreen. The window can also be dragged to any size for a projector.
Q / Esc	quit

---

👀 What you see

_{Illustration of the three modes. The gap numbers in it are examples, not measurements.}

VR-style (V) is a real screenshot of the app. Lean left and the near red box slides further than the far blue one, like looking through a window. At 100 ms the room follows you late, yet nothing looks wrong. The strip at the top shows, from left to right: mode VR-style / AR-style / AR + fix delay the slider value gap 🔴 sticker → 🟢 real nose, in pixels (AR modes only) fps frames per second (aim for 25+)

---

🔬 The experiment

One automatic measurement carries the result. It uses a recording, so every delay sees exactly the same head motion and the chart comes out identical on every run.

flowchart LR
    R["🎥 R<br/>record ~20 s<br/>of head turns"] --> S["▶️ S<br/>start sweep"]
    S --> T["🙂 track<br/>video once"]
    T --> X["🔁 replay<br/>6 delays ×<br/>fix off / on"]
    X --> O["📊 sweep.png<br/>+ sweep.csv"]

Loading

This chart is a prediction, not a measurement: it comes from a synthetic sine-wave head track run through the real sweep code. It shows what the result should look like if the idea holds. The real chart comes from a 20-second recording, which has not been made yet.

What you should see	What it means
🔵 Without fix: a near-straight line on the dotted prediction	The error really is head speed × delay
🟠 With fix: several times smaller, but not zero	Prediction works; it overshoots when the head stops or turns back
▒ Grey band: tracker jitter, well under 1 px	The noise floor. The delay effect is far above it
❌ A clearly bent line, or a fix that doesn't help	Would prove us wrong: tracker jitter beats the delay effect

Note

Only preliminary real data so far. The one real run is a 2.5-second, 68-frame clip recorded during a self-test, with no controlled head motion. It points the same way: 12.5 px at 100 ms without the fix, 4.4 px with it, and 0.4 px of jitter. But it is far too short to count as the result. Still pending: 20 s of smooth side-to-side head turns (R … R, then S). Its chart will replace the prediction above.

---

🔧 How it works

Every camera frame, about 30 times a second:

flowchart LR
    A["📷 webcam<br/>frame"] --> B["🙂 MediaPipe<br/>nose tip"]
    B --> C[("⏱️ delay buffer<br/>nose D ms ago")]
    C --> V["🧊 VR-style<br/>moves the room"]
    C --> AR["🔴 AR-style<br/>late sticker"]
    C --> F["🎯 AR + fix<br/>late + speed × D"]
    AR --> G["📏 gap in px<br/>screen + CSV"]
    F --> G
    V -. "logged only" .-> G

Loading

• Delay buffer: every nose position is saved with its time. To draw, the app asks "where was the nose D ms ago?" and blends the two saved positions either side of that moment.
• The fix: it uses only old data, as a real headset must. It takes the speed from the last two late positions and draws at late position + speed × D.
• Fair from the start: the camera image and the face dots come from the same frame. At 0 ms the sticker sits exactly on your nose, so every gap measured comes only from the delay Delay Lab adds.

Files in this repo

File	Role
app.py	Window, slider, keys, modes, per-frame CSV log, recorder, --selftest
tracker.py	MediaPipe Face Landmarker wrapper (VIDEO mode)
delay.py	Time-stamped look-back buffer and the guess-ahead fix
render.py	Sticker, text strip, wireframe room (pinhole projection, no 3D engine)
sweep.py	Recorded video → every delay × fix → CSV + chart, with the noise floor
make_gif.py	Demo recording → README GIF
chart_style.py	Shared colours and styling for every chart
paths.py	Where the model and output folders live (from source or inside the exe)
build_exe.py	PyInstaller → dist/DelayLab.exe
docs/make_figures.py	Regenerates the images in docs/img/
tests/	python -m pytest: 26 tests, no webcam needed

---

📏 Real-world numbers

Delay builds up along the whole path from motion to photon: the sensor notices the move, software works out where your head is, the picture is drawn, it waits for the next screen refresh, then the pixels light up.

flowchart LR
    M(("🙂 move")) --> S1["1 · sense"] --> S2["2 · locate"] --> S3["3 · draw"] --> S4["4 · wait for<br/>screen"] --> S5["5 · light<br/>pixels"] --> E(("👁️ see"))

Loading

System	Motion-to-photon delay	AR slip at 1 m (100°/s turn)
🎯 Comfortable VR target (3GPP TR 26.918)	< ~20 ms	< 3.5 cm
Apple Vision Pro passthrough (Optofidelity)	~11 ms	~1.9 cm
Meta Quest 3 / Quest Pro / HTC Vive XR Elite	~35–40 ms	~6–7 cm
💻 A laptop webcam pipeline	~60–150 ms (measure yours)	~10–26 cm

Headsets fight delay three ways: faster parts, prediction (Delay Lab's F mode), and last-moment correction ("timewarp"), which shifts the finished picture just before it's shown.

---

🎤 Presenting it

docs/DEMO_SCRIPT.md is a five-minute demo: 1 show the problem → 2 why VR hides it → 3 the measurement → 4 the fix and its overshoot → 5 the link to real headsets.

Make a GIF of your own demo

Press R, show the sticker sliding at 100 ms, switch to F, then press R again. Then run:

python make_gif.py --start 2 --seconds 6     # newest demo_*.mp4 → results/demo.gif

To replace the simulated GIF at the top with a real one, save yours as docs/img/demo.gif and point the README at it. The GIF shows your face, so it stays in results/ (not committed) unless you choose to move it to docs/ and add it here.

---

📦 DelayLab.exe

One file that runs on a Windows PC without Python installed:

pip install -r requirements-build.txt
python build_exe.py                 # → dist/DelayLab.exe (~130 MB, face model included)
dist\DelayLab.exe --selftest 8      # same self-check as above

logs/, recordings/ and results/ are created next to the exe. The first start takes a few seconds while it unpacks.

---

🛠 Troubleshooting

"no face" in the strip

Light your face from the front, sit 40–80 cm from the camera, and avoid a bright window behind you.

Below about 20 fps

Close other apps that use the CPU or the camera. Jerky tracking hides the effect. The camera already runs at 640×480.

The fix looks worse than no fix

Prediction multiplies small tracker jitter when you sit still, and overshoots on jerky motion. Turn your head smoothly for the sweep recording. The overshoot when you stop is real, and worth showing (demo step 4).

Wrong camera, or "Could not read from camera"

Close other apps using the webcam, or try python app.py --camera 1 (or 2, …).

---

🧪 Development

python -m pytest                 # 26 tests on synthetic head tracks: no webcam needed
python docs/make_figures.py      # rebuild the charts and screenshot in docs/img/

CI runs the tests on Windows with Python 3.11 on every push.

🤝 Contributing

Contributions are welcome: bug reports, docs, new modes, and real measurements from other webcams. No webcam? You can still help: the tests run on synthetic head tracks. Look for good first issue and no webcam needed. Taking part in Hacktoberfest? Issues labelled hacktoberfest are ready. Questions go to Discussions.

📘 CONTRIBUTING.md	Dev setup, tests, code style, branch and commit conventions, PR process
🐞 Report a bug · 💡 Request a feature	Issue forms that ask for what's needed to help
🤝 CODE_OF_CONDUCT.md	Contributor Covenant 2.0
🔒 SECURITY.md	Report vulnerabilities privately, not in a public issue

License

MIT. See LICENSE.