Linux export: OPENSCREEN_EXPORT_ENCODER can't open any HW encoder, and the vaMapBuffer2 guard is missing from the export path

[EtienneLescot]

Two related problems in the Linux export path, found while measuring export performance
(#544). One is a defect today; the other is a landmine for the VAAPI work that
pipeline_linux.rs says is planned. The fix for the second already exists in this repo, two
files away.

To be clear up front: the export path does not crash today, because by default it never
touches VA-API. Nothing is on fire.

1. OPENSCREEN_EXPORT_ENCODER cannot do the thing its own comment advertises

crates/compositor/src/pipeline_linux.rs:221 says:

// Liste par defaut, plus l'encodeur force s'il n'y figure pas (ex. h264_vaapi).

The routing works exactly as described — :223-231 appends the forced name and skips the
others, so try_open("h264_vaapi", ...) really is called. But try_open can only ever
configure a software encoder:

• :259 sets (*ctx).pix_fmt = AV_PIX_FMT_YUV420P unconditionally, with no per-candidate format
• :265 calls avcodec_open2(ctx, enc, ptr::null_mut()) with no options
• hw_device_ctx / hw_frames_ctx are never assigned anywhere in the file

So every hardware encoder refuses to open. Measured against the vendored
libavcodec.so.62.28.102, replicating try_open's exact configuration:

encoder	found	avcodec_open2
libopenh264	yes	0 (opens)
libkvazaar	yes	0 (opens)
h264_vaapi	yes	−22 EINVAL
h264_qsv	yes	−22 EINVAL
h264_nvenc	yes	−1
h264_amf	yes	−1313558101

ffmpeg's own diagnostic: "Specified pixel format yuv420p is not supported by the h264_vaapi
encoder. Supported pixel formats: vaapi".

And the failure is fatal rather than a fallback: :229-231 has already continued past
libopenh264, so refused holds only the VAAPI error and :241 bails. Reproduced end to end:

$ OPENSCREEN_EXPORT_ENCODER=h264_vaapi openscreen export project.openscreen -o out.mp4 --json
{"event":"error","message":"OPENSCREEN_EXPORT_ENCODER=h264_vaapi inutilisable :
 h264_vaapi: avcodec_open2(enc) a échoué (ret=-22)"}

A working 74.9 s export becomes a hard error. Either the escape hatch should configure hardware
encoders properly, or it should fall back to the default list rather than aborting, or the
comment should stop naming h264_vaapi as an example of what it accepts. Any of the three is
fine; the present combination is a trap.

Note the encoders are all present — this is not a limitation of the LGPL build. The vendored
libavcodec resolves h264_vaapi, h264_nvenc, h264_qsv and h264_amf by name; only their
configuration is missing.

2. The vaMapBuffer2 guard is missing from the export path

This one is latent, and it is the reason I am filing rather than just noting the above.

electron/native/pipewire-capture/src/encoder.rs:14-21 already documents the hazard exactly:

WHY VAAPI IS GUARDED BY A dlsym. The vendored ffmpeg 8.1 calls vaMapBuffer2, which libva
only grew in 2.22. On a system with an older libva (Ubuntu 24.04 ships 2.20) the call goes
through an implib-generated trampoline that does not return an error — it calls assert(0)
and the process dumps core.

and guards against it with vaapi_is_safe_to_probe() (:124-137, used at :247).

crates/compositor/src/pipeline_linux.rs has no equivalent — zero references to
vaMapBuffer2 or vaapi_is_safe_to_probe. The guard exists in exactly one place in the repo,
and it is the recording helper.

Confirmed on this machine (Ubuntu 24.04, libva2 2.20.0-2ubuntu0.2):

• /lib/x86_64-linux-gnu/libva.so.2 exports vaMapBuffer, and not vaMapBuffer2
• the vendored libavutil.so has no libva DT_NEEDED — it dlopens through implib
  trampolines (strings finds libva.so.2.init.c)

A staged probe against the vendored libraries shows where it lands:

av_hwdevice_ctx_create      ... ok (0)
av_hwframe_ctx_init         ... ok (0)
avcodec_open2(h264_vaapi)   ... ok (0)
av_hwframe_get_buffer       ... ok (0)
av_hwframe_transfer_data    ... implib-gen: failed to resolve symbol 'vaMapBuffer2'
                                Assertion `0 && "Assertion in generated code"' failed  -> SIGABRT

The important detail is where it aborts: av_hwframe_transfer_data, i.e. the CPU→GPU
upload. Encoding from surfaces that are already on the GPU did not abort in the same probe.

So the obvious implementation of VAAPI export — read back RGBA, sws_scale to NV12,
av_hwframe_transfer_data, encode — core-dumps on Ubuntu 24.04 LTS, which is the current LTS.
Whoever picks up the "VAAPI/Vulkan-encode = suivi" work in pipeline_linux.rs:9-10 will hit
this, and the answer is already written in encoder.rs. Worth either lifting
vaapi_is_safe_to_probe() somewhere both paths can call it, or leaving a pointer to it in
pipeline_linux.rs so the next person finds it before the core dump does.

One observation on the existing guard while you are in there: encoder.rs:247 reads

if backend == Backend::Vaapi && forced.is_none() && !vaapi_is_safe_to_probe() {

so an explicitly forced backend skips the safety check. Deliberate, presumably — but it means a
forced VAAPI on an old libva still core-dumps the recorder.

Why this matters beyond tidiness

From #544, on this hardware: libopenh264 is ~27 s of a 74.9 s export, and the RGBA→YUV420P
sws_scale it forces adds ~10 s more — together roughly half the export. A working VAAPI
path is the single biggest lever available on Linux, which makes the two problems above the
things standing between here and it.

Measured on: AMD Ryzen 5 7520U / Radeon 610M (Mendocino), Ubuntu 24.04, Wayland, libva 2.20,
/dev/dri/renderD128. The system ffmpeg (6.1.1) links libva.so.2 directly and is unaffected —
its VAAPI encode works fine, which is what makes the vendored build's behaviour the distinguishing
factor rather than the driver or the hardware.