A couple of questions about the optical flow ground truth (`_Shake` sequences, and `LM` magnitude)

[ChenHYe]

First off, thanks for putting this dataset together — the amount of per-frame metadata you ship (camera poses, depth, multiple AOVs, alongside the HDR frames) is exactly what's needed for careful low-light optical flow work, and it's rare to see a synthetic dataset this complete. We've been building on it for that reason, which is also why we wanted to flag the following rather than just quietly work around it.

Summary

We've been using the flow ground truth and ran into two things we can't quite explain on our own, so wanted to bring both to you in one place. For context on method: "optical flow" here just means the per-pixel 2D displacement between two consecutive frames, stored in flow/*.exr; one basic sanity check we use throughout is simple — take frame t+1, shift every pixel by the flow vector given for frame t, and see if it lines up with frame t. If the flow is correct, this should reduce the difference between the two frames; if it doesn't (or makes it worse), something's off either in the flow or in our understanding of the format.

1. For _Shake sequences, shifting frame t+1 by the released flow does not consistently reduce the difference from frame t — for the non-shake sequences it does, cleanly.
2. For LM sequences, the flow's direction looks right, but its magnitude looks smaller than what two independent methods estimate.

Very possibly we're missing something on our end for either — details, numbers, and a runnable script for Part 1 below.

---

Part 1: _Shake sequences (FM_Shake 327 + LM_Shake 173)

A model-free check using only your own camera_params + depth

camera_params/XXXX.json records each frame's camera pose (rotation + translation, OpenCV world2cam convention) and depth/XXXX.exr records per-pixel scene depth. For any pixel on a static part of the scene, these two alone are enough to compute where it should land in the next frame, via plain pinhole-camera reprojection — no flow network or image matching involved:

X_cam(t)   = depth(u,v) * K^-1 [u, v, 1]^T
X_world    = R(t)^T (X_cam(t) - T(t))
X_cam(t+1) = R(t+1) X_world + T(t+1)
(u', v')   = project(X_cam(t+1))                    # via K
predicted_flow(u,v) = (u' - u, v' - v)

We then compare predicted_flow to the flow you ship for the same pixel. Agreement would mean flow is geometrically consistent with the camera poses you also ship; a disagreement shaped like a rigid 2D shift would point to flow and camera_params being derived from slightly different per-frame camera poses.

A confound we had to control for: some scenes have independent object motion on top of camera motion — e.g. scene_102_FM has ~0 camera translation/rotation between frame 0 and 1 (measured directly from camera_params), yet nonzero flow, clearly from moving foreground content. A plain reprojection only predicts the camera-induced part, so on its own it will disagree with GT wherever objects are moving, regardless of whether GT is correct there. To avoid that confound for this quick check, we restricted the comparison to the farthest-depth quartile of each frame, as a crude proxy for "probably static background" (moving assets tend to be closer to the camera than the environment). This is not a real segmentation mask, just a cheap filter for a sanity check.

Results (5 scenes, frame 0 -> frame 1 each)

For each scene: the median vector difference between predicted_flow and the released flow over that background mask, and — after subtracting that one median vector — how much scatter is left. A small number means "the two agree up to one constant shift"; a large/incoherent number would mean the disagreement isn't a simple shift.

scene	kind	predicted − released, median (px)	scatter after removing that shift (px)
scene_0_FM	control, camera moves, no shake	(-0.18, +0.01)	0.24
scene_102_FM	control, camera ~static, no shake	(+0.76, -0.73), both signals under 1.1px	0.72
scene_104_FM_Shake	shake	(+11.50, +3.51)	2.04
scene_111_FM_Shake	shake	(+8.83, -2.45)	0.24
scene_10_FM_Shake	shake	(+8.52, -0.47)	0.54

On both non-shake controls, the geometric prediction and the released flow agree closely (residual scatter under 1px). On all three shake scenes, they disagree, but the disagreement is explained almost entirely by a single 2D shift per scene (residual mostly under 1px, one case at 2px) rather than incoherent noise — i.e. it looks like a missing or mismatched rigid camera-pose component, and the size (8-12px) lines up with a second, unrelated method below.

Script (reproduces the table above, no ML framework required)

Only depends on numpy + OpenEXR (pip install numpy OpenEXR). Run as python reprojection_check.py /path/to/scene_XXX_FM_Shake.

reprojection_check.py

"""
reprojection_check.py -- compare S2R-HDR's released `flow` against a purely
geometric prediction built from the scene's own `camera_params` + `depth`.
No optical-flow model, no image matching -- just pinhole-camera reprojection.

For a pixel (u, v) on a static part of the scene at frame t:

    X_cam(t)   = depth(u,v) * K^-1 [u, v, 1]^T
    X_world    = R(t)^T (X_cam(t) - T(t))         # world2cam -> invert
    X_cam(t+1) = R(t+1) X_world + T(t+1)
    (u', v')   = project(X_cam(t+1))              # via K
    predicted_flow(u,v) = (u' - u, v' - v)

Since several scenes have independent object motion on top of camera motion,
this script restricts the comparison to the farthest-depth quartile of the
frame (crude proxy for "static background" -- moving assets tend to be closer
to the camera than the environment). Reports the median (predicted - released)
offset and the residual scatter left after subtracting that one offset: a
small residual means "the two agree up to a single rigid shift".

Usage:
    python reprojection_check.py /path/to/scene_XXX_FM_Shake --frame 0 --bg-pct 75

Requires: numpy, OpenEXR (`pip install OpenEXR`). No PyTorch / flow models.
"""
import argparse
import json
from pathlib import Path

import numpy as np
import OpenEXR


def load_exr_rgba(path):
    with OpenEXR.File(str(path)) as f:
        return np.array(f.parts[0].channels["RGBA"].pixels, dtype=np.float64)


def load_cam(scene_dir, frame_idx):
    with open(scene_dir / "camera_params" / f"{frame_idx:04d}.json") as f:
        d = json.load(f)
    R = np.array(d["extrinsic_r"], dtype=np.float64)   # world2cam rotation
    t = np.array(d["extrinsic_t"], dtype=np.float64)   # world2cam translation
    K = d["intrinsic"]
    fx, fy, cx, cy = K[0][0], K[1][1], K[0][2], K[1][2]
    return R, t, fx, fy, cx, cy, d["width"], d["height"]


def decode_flow(flow_raw, W, H):
    # S2R-HDR convention: R/G channels store flow normalized to [-1, 1]-ish range
    u = flow_raw[..., 0] * W / 2.0
    v = -flow_raw[..., 1] * H / 2.0
    return u, v


def main():
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("scene_dir", type=Path, help="e.g. .../scene_104_FM_Shake")
    ap.add_argument("--frame", type=int, default=0, help="compare frame `--frame` -> `--frame`+1 (default 0)")
    ap.add_argument("--bg-pct", type=float, default=75.0,
                    help="keep only pixels with depth >= this percentile, as a proxy for static background (default 75)")
    ap.add_argument("--sky-depth", type=float, default=65000.0,
                    help="drop pixels with depth >= this (background-at-infinity / sky), default 65000 (float16 max ~65504)")
    args = ap.parse_args()

    f0, f1 = args.frame, args.frame + 1
    R0, t0, fx, fy, cx, cy, W, H = load_cam(args.scene_dir, f0)
    R1, t1, *_ = load_cam(args.scene_dir, f1)

    depth = load_exr_rgba(args.scene_dir / "depth" / f"{f0:04d}.exr")[..., 0]  # R==G==B
    flow_raw = load_exr_rgba(args.scene_dir / "flow" / f"{f0:04d}.exr")
    u_gt, v_gt = decode_flow(flow_raw, W, H)

    valid = depth < args.sky_depth
    thresh = np.percentile(depth[valid], args.bg_pct)
    bg = valid & (depth >= thresh)

    vs, us = np.nonzero(bg)
    d = depth[vs, us]

    # unproject frame-f0 pixels to frame-f0 camera space (z-depth convention)
    x_cam0 = (us - cx) / fx * d
    y_cam0 = (vs - cy) / fy * d
    X_cam0 = np.stack([x_cam0, y_cam0, d], axis=0)  # (3, N)

    # world2cam: X_cam = R @ X_world + t  =>  X_world = R^T (X_cam - t)
    X_world = R0.T @ (X_cam0 - t0[:, None])

    # reproject into frame-f1 camera
    X_cam1 = R1 @ X_world + t1[:, None]
    u1 = fx * X_cam1[0] / X_cam1[2] + cx
    v1 = fy * X_cam1[1] / X_cam1[2] + cy

    u_pred = u1 - us
    v_pred = v1 - vs
    du = u_pred - u_gt[vs, us]
    dv = v_pred - v_gt[vs, us]

    med_du, med_dv = np.median(du), np.median(dv)
    residual = np.median(np.hypot(du - med_du, dv - med_dv))

    print(f"scene: {args.scene_dir.name}  (frame {f0} -> {f1}, background = depth top {100-args.bg_pct:.0f}%, n={len(vs)})")
    print(f"  predicted - released, median offset: ({med_du:+.2f}, {med_dv:+.2f}) px")
    print(f"  residual after removing that one offset: {residual:.2f} px")
    print( "  (small residual = the two agree up to a single rigid shift; large/incoherent = they don't)")


if __name__ == "__main__":
    main()

A second, independent (image-domain) check

Separately, using only the rendered frames and the released flow (no camera_params, no depth, no neural network): warp img[t+1] onto img[t] with flow, then fit a single global 2D translation on the leftover photometric residual and add it back.

scene	warp gain before	warp gain after +translation	recovered offset (px)
scene_0_FM (control)	2.913	2.901	(-0.23, -0.04) — ~0, as expected
scene_102_FM (control)	3.182	3.146	(-0.07, +0.03) — ~0
scene_104_FM_Shake	1.169	3.447	(+11.86, +12.03) — fully recovers to non-shake level
scene_108_LM_Shake	0.520	1.691	(+16.73, +9.18)
scene_10_FM_Shake	0.620	1.495	(+2.66, +4.55)
scene_113_FM_Shake	0.933	1.278	(+7.73, +2.67)
scene_111_FM_Shake	0.774	0.777	(+0.49, +5.44) — a pure translation can't fully fix this one (needs rotation too, matches the geometric check above)

("warp gain" = mean|img[t]-img[t+1]| (no warp) / mean|img[t]-warped(img[t+1])|; gain > 1 means warping reduces the frame-to-frame difference, as correct flow should.)

Two unrelated methods — pure camera-geometry reprojection, and image-domain residual fitting — landing in the same 8-12px range is what makes us suspect a real, physical missing/mismatched component rather than a decoding mistake on our end.

Could this be intentional (flow deliberately excludes camera ego-motion)?

We considered this, since it would be a reasonable design choice. We only hesitated because Appendix B.3 / Figure 12 describe the shake as a perturbation to the camera pose applied before rendering (not a post-render image effect), which we'd naively expect to be visible to every pass rendered from that camera, flow included — but we don't know your renderer's internals, so if flow and camera_params are intentionally allowed to diverge for some reason, we'd love to know, since it'd change how we use this field.

(Small, possibly unrelated notes: the reference dataloader has flow = None with the read call commented out, dataset/dataset_render.py:166-168, 198, 203 — not sure if related, just mentioning in case useful. Also, Appendix B.3 says "30% of the sequences" have Perlin camera shake, while the release has exactly 500/1000 = 50% tagged _Shake — might just be a stale number.)

---

Part 2: LM sequences (173 scenes) — magnitude

LM scenes have a static camera and only foreground/object motion, so the geometric check above doesn't apply (there's no camera motion to reproject) — we need some independent estimate of how far things actually moved. For that we used two neural optical-flow models: RAFT (a well-established estimator using a 4D correlation cost volume) and WAFT (a newer, cost-volume-free architecture, currently at the top of the public Sintel/KITTI/Spring leaderboards). They're independently trained, different architectures — we're using them the way one might use two different measuring instruments, not because we think either is "ground truth" on its own.

Direction of the released flow matches both estimators well in the moving region (cos ~= 1.0), but magnitude looks consistently smaller — median ratio (estimator / GT) ~= 8 (IQR 7-8) across the 131/173 scenes where the moving region has enough texture/displacement to measure at all (the other 42 don't). The ratio isn't a single fixed constant — it varies somewhat scene to scene and even frame to frame within a scene.

Example: scene_134_LM, frame 0, moving/textured region

method	displacement
RAFT (zero-shot on this data)	61.4 px
WAFT (zero-shot, different architecture)	61.5 px
brute-force integer-shift search (no learned model at all)	64 px
our own visual read of the crop	consistent with ~60px
released flow GT	5.67 px (ratio ~= 10.8)

A few more scenes:

scene	|GT| (px)	|WAFT| (px)	ratio
scene_134_LM	5.67	61.49	10.81
scene_139_LM	3.67	39.57	10.69
scene_22_LM	2.12	23.24	11.04
scene_78_LM	2.84	30.29	10.81

(scene_79_LM is one where the ratio comes out ~= 1.0 and GT matches — good to know our measurement isn't just systematically biased high.)

Before raising this we validated both models on FM scenes sharing LM's static-camera + small-object-motion geometry (e.g. scene_14_FM), where both agree with GT (ratio 1.00-1.03) — so we're fairly confident in the estimators themselves here, though of course open to other explanations, e.g. a scale/unit difference in how flow is exported for scenes without camera motion.

---

Questions

• Part 1: does the flow pass use the same per-frame camera pose as camera_params/*.json? Is flow intended to include camera ego-motion, or only scene/object-relative motion? No worries either way, we just want to use the field correctly.
• Part 2: could there be a scale/unit difference in how flow is exported for scenes without camera motion vs. with it?
• The reprojection script for Part 1 is attached above and only needs numpy + OpenEXR. For Part 2 we're happy to share our RAFT/WAFT comparison scripts and full per-scene tables too, just didn't want to dump a PyTorch + external-checkpoint pipeline into this issue unprompted.