NvFBC is incorrectly rejected when XRandR is unavailable, despite valid virtual desktop capture support

[uebian]

Is there an existing issue for this?

• I have searched the existing issues

Is your issue described in the documentation?

• I have read the documentation

Is your issue present in the latest beta/pre-release?

This issue is present in the latest pre-release

Describe the Bug

I'm running Sunshine inside an LXC container with an NVIDIA GPU and a headless X server. The X server provides a valid virtual desktop, but does not expose the XRandR extension (I intentionally disabled the XRandR extension to resolve the Xfce desktop environment issue).

In this environment, NvFBC appears to initialize successfully and reports a valid virtual desktop size:

[2026-09-13 23:14:03.366]: Info: Found [0] outputs
[2026-09-13 23:14:03.366]: Info: Virtual Desktop: 1920x1080
[2026-09-13 23:14:03.366]: Info: XrandR: unavailable

However, Sunshine then crashes:

Xlib:  extension "RANDR" missing on display ":0".
Segmentation fault         (core dumped) sunshine

The crash is unexpected because the lack of XRandR should only prevent selecting individual monitors, not prevent NvFBC screen capture entirely.

Expected Behavior

If NvFBC reports (1) capture is possible, (2) a valid non-zero virtual desktop size, and (3) XRandR is unavailable / there are no individual outputs, Sunshine should still consider NvFBC a valid capture backend and capture the entire virtual desktop. Eventually, Sunshine should be able to start in this case.

Additional Context

Below is my X config for your reference:

Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0"
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
EndSection

Section "Files"
EndSection

Section "InputDevice"
    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/mouse"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"
    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Unknown"
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "NVIDIA RTX A6000"
    BusID          "PCI:65:0:0"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "AllowEmptyInitialConfiguration" "True"
    Option         "UseDisplayDevice" "none"
    Option         "MetaModes" "NULL"

    SubSection "Display"
        Depth       24
        Modes       "1920x1080"
        Virtual     1920 1080
    EndSubSection

EndSection

Section "Extensions"
    Option "RANDR" "Disable"
EndSection

Host Operating System

Docker

Operating System Version

Ubuntu 26.04

Architecture

amd64/x86_64

Package

Linux - deb

GPU Type

NVIDIA

GPU Model

RTX A6000

GPU Driver/Mesa Version

615.71.09

Capture Method

NvFBC (Linux)

Apps

The issue is not specific to a certain app

Log output

$ sunshine 
[2026-09-13 23:14:03.102598] [0x00007fccd0c86000] [info]    config: 'output_name' = 0
[2026-09-13 23:14:03.102]: Info: Sunshine version: 2026.912.143204 commit: fbf12d019c16c6f8ff6fb4aaff8519d74e298fdf
[2026-09-13 23:14:03.102]: Info: Package Publisher: LizardByte
[2026-09-13 23:14:03.102]: Info: Publisher Website: https://app.lizardbyte.dev
[2026-09-13 23:14:03.102]: Info: Get support: https://app.lizardbyte.dev/support
[2026-09-13 23:14:03.102]: Info: config: 'output_name' = 0
[2026-09-13 23:14:03.366]: Info: Found [0] outputs
[2026-09-13 23:14:03.366]: Info: Virtual Desktop: 1920x1080
[2026-09-13 23:14:03.366]: Info: XrandR: unavailable
[2026-09-13 23:14:03.389]: Error: Couldn't release NvFBC context from current thread: 
[2026-09-13 23:14:03.389]: Error: Couldn't open: /dev/dri/card1: Permission denied
[2026-09-13 23:14:03.389]: Error: [wayland] Environment variable WAYLAND_DISPLAY has not been defined
[2026-09-13 23:14:03.389]: Info: Detecting displays
Xlib:  extension "RANDR" missing on display ":0".
Segmentation fault         (core dumped) sunshine

Online logs

No response

[uebian]

It looks like this issue might be related to #5644

[psyke83]

Master branch (or if you can't compile, the next release) may fix this. There was a permissions issue with CUDA's DRM fd open call not having sufficient permissions that should be resolved as part of a recent change.