Repository navigation
egl_worker thread never receives CAP_SYS_ADMIN despite correct setcap on the binary — blocks KMS capture entirely (reproducible on both direct-login and xrdp-created sessions) #5803
Description
Is there an existing issue for this?
- I have searched the existing issues
Is your issue described in the documentation?
- I have read the documentation
Is your issue present in the latest beta/pre-release?
None
Describe the Bug
setcap cap_sys_admin,cap_sys_nice=p is correctly applied to the sunshine binary and confirmed present in the file's capability set. However, the dedicated egl_worker thread (introduced by the privileged-EGL-worker-thread security architecture in GHSA-fp6g-27w5-489j / commit 7a987ee) never actually holds cap_sys_admin in its permitted capability set at runtime — only cap_sys_nice is present. This is reproducible identically in two architecturally different session types on the same machine, ruling out a session-specific cause:
- A normal direct physical console login (XFCE, X11)
- A session created via xrdp/xorgxrdp (also XFCE, X11, but a different X driver — xrdpdev rather than nvidia)
As a direct consequence, KMS capture (capture = kms) fails to initialize at all (Unable to initialize capture method, Platform failed to initialize), and every encoder subsequently fails during startup probing with Failed to gain CAP_SYS_ADMIN (nvenc, vulkan, vaapi, and even the software encoder). Sunshine correctly falls back to NvFBC capture when capture is left on auto, so basic streaming still works — but this closes off KMS as a usable capture path, which is otherwise the one path (via its gl_cuda_vram_t encode-device class) known to support AV_PIX_FMT_P010LE and therefore genuine 10-bit output, unlike the NvFBC+CUDA path (cuda::cuda_t, tracked separately as an 8-bit-only limitation).
Expected Behavior
The egl_worker thread (or whichever thread actually calls the KMS/DRM-master-acquiring code) should hold cap_sys_admin in its permitted capability set at the point it attempts to use it, given the binary has the correct file capability and the security-hardening redesign was specifically intended to preserve this capability for exactly this worker thread while dropping it elsewhere.
Additional Context
`$ getcap /usr/local/bin/sunshine
/usr/local/bin/sunshine cap_sys_admin,cap_sys_nice=p
$ getpcaps
: =
$ grep -E "CapPrm|CapEff|CapBnd" /proc//task/<egl_worker tid>/status
CapPrm: 0000000000800000 # cap_sys_nice only — cap_sys_admin (bit 21) absent
CapEff: 0000000000000000
CapBnd: 000001ffffffffff # cap_sys_admin present in bounding set, so it's not a bounding-set restriction`
Confirmed identical on both a direct physical login session and an xrdp-created session on the same host, ruling out session-manager/PAM-stack causes (also checked and ruled out: NoNewPrivs is 0 in both cases; systemd --user's own CapabilityBoundingSet= includes cap_sys_admin; the sunshine.service unit file has no NoNewPrivileges= directive).
This was found while diagnosing a related, separate limitation: NvFBC+CUDA capture (cuda::cuda_t) only ever produces AV_PIX_FMT_NV12/AV_PIX_FMT_YUV444P frames (no P010/10-bit path — already tracked, e.g. PR #5429's description). KMS capture was the natural workaround to test, since its gl_cuda_vram_t encode-device class is documented to support AV_PIX_FMT_P010LE. This capability-inheritance issue is what's actually blocking that workaround from being tested at all, independent of whatever the eventual fix for the NvFBC 8-bit limitation turns out to be. Happy to provide full sunshine.log output, the full systemd unit file, or run further diagnostics on request — this was reproduced carefully across two session types specifically to rule out environmental causes.
I do have a fully documented build log that I am happy to share privately with you.
Host Operating System
Linux
Operating System Version
Rocky Linux 9.6 (Blue Onyx), glibc 2.34
Architecture
amd64/x86_64
Package
other (self built)
GPU Type
NVIDIA
GPU Model
RTX 4000
GPU Driver/Mesa Version
NVIDIA 580.65.06, CUDA 13.3 toolkit
Capture Method
KMS (Linux)
Apps
Log output
[timestamp]: Info: config: 'capture' = kms
[timestamp]: Info: /dev/dri/card0 -> nvidia-drm
[timestamp]: Error: Unable to initialize capture method
[timestamp]: Error: Platform failed to initialize
[timestamp]: Info: Trying encoder [nvenc]
[timestamp]: Error: Failed to gain CAP_SYS_ADMIN
[timestamp]: Info: Encoder [nvenc] failed
[timestamp]: Info: Trying encoder [vulkan]
[timestamp]: Error: Failed to gain CAP_SYS_ADMIN
[timestamp]: Info: Encoder [vulkan] failed
[timestamp]: Info: Trying encoder [vaapi]
[timestamp]: Error: Failed to gain CAP_SYS_ADMIN
[timestamp]: Info: Encoder [vaapi] failed
[timestamp]: Info: Trying encoder [software]
[timestamp]: Error: Failed to gain CAP_SYS_ADMIN
[timestamp]: Info: Encoder [software] failed
[timestamp]: Fatal: Unable to find display or encoder during startup.Online logs
No response
Activity
psyke83 commentedon Sep 25, 2026
Please share a full log and the output of the following two times; once when kms capture is set in the config and startup was attempted, and then once more when NvFBC capture is set and running:
girlypop2000 commentedon Sep 25, 2026
Yeah, I tried to use the KMS method on Linux Mint 22.3 with Rtx3060 and it just doesn't work. The cap_sys_admin is all over in the logs even when using other capture methods. Tried setting cap_sys manually in terminal and even running the program as a service or straight from terminal instead of gui but no luck. Might be nvidia related...
girlypop2000 commentedon Sep 26, 2026
.deb package is what i have just to clarify
psyke83 commentedon Sep 26, 2026
I still need to see full logs and output from the command I posted.
The original bug report also misdiagnoses the problem. There are two privileged worker threads; drm_worker holds cap_sys_admin and cap_sys_nice, but egl_worker only needs to hold cap_sys_nice for creating EGL contexts.
There's a part of the code where the drm_worker drops its privileges if kms or nvfbc capture is not selected/detected. That might be getting triggered erroneously. More logs would help.
girlypop2000 commentedon Sep 26, 2026
Adding one more info tidbit: kms works fine on intel integrated gpu on a latop that i tested (even with the cap_sys errors in the logs)
jessica-atk commentedon Sep 26, 2026
Thanks All,
Sadly I won't be able to test until Monday - the dev machine is off at the moment and unreachable.
Will update here asap.
jessica-atk commentedon Sep 28, 2026
Hi @psyke83,
Here's a summary comparison — physical console login vs. an
xrdp-created session, crossed with default/auto capture vs.capture = kmsforced — all on the same original machine, in response to your question aboutdrm_worker/egl_workercapability state.Summary
capture = kmsdrm_workerhascap_sys_admin,cap_sys_nice=pdrm_workerempty (=)drm_workerempty (=), despite NvFBC working correctlydrm_workerempty (=)The reason for using xrdp as a method for initial login is so that the machine does not need to be consistently logged in to ensure that Sunshine is available and so that multiple people would be able to leverage that machine for streaming, with unqiue secure logins.
1. Direct login, default settings (NvFBC succeeds)
[2026-09-28 09:55:48.141]: Info: Sunshine version: 2026.914.233613 commit: 63d35f7
[2026-09-28 09:55:48.141]: Info: config: 'csrf_allowed_origins' = [redacted]
[2026-09-28 09:55:48.632]: Info: Found [1] outputs
[2026-09-28 09:55:48.633]: Info: Virtual Desktop: 1680x1050
[2026-09-28 09:55:48.708]: Info: Detected display: DP-3 (id: 3)DP-3 connected: true
[2026-09-28 09:55:48.908]: Error: [portalgrab] RemoteDesktop CreateSession failed with response code: 2
[2026-09-28 09:55:48.910]: Error: [portalgrab] ScreenCast CreateSession failed with response code: 2
[2026-09-28 09:55:48.925]: Info: Trying encoder [nvenc]
[2026-09-28 09:55:48.925]: Info: Screencasting with NvFBC
[2026-09-28 09:55:49.086]: Info: Creating encoder [h264_nvenc]
[2026-09-28 09:55:49.087]: Info: Color coding: SDR (Rec. 601)
[2026-09-28 09:55:49.087]: Info: Color depth: 8-bit
[2026-09-28 09:55:49.087]: Info: Color range: JPEG
[2026-09-28 09:55:49.249]: Info: Streaming bitrate is 1000000
[2026-09-28 09:55:49.521]: Info: Creating encoder [hevc_nvenc]
[2026-09-28 09:55:49.787]: Info: Creating encoder [av1_nvenc]
[2026-09-28 09:55:49.796]: Warning: [av1_nvenc @ 0x58f3440] Codec not supported
[2026-09-28 09:55:49.796]: Error: [av1_nvenc @ 0x58f3440] Provided device doesn't support required NVENC features
[2026-09-28 09:55:50.047]: Error: Could not open codec [av1_nvenc]: Function not implemented
[2026-09-28 09:55:50.229]: Info: Creating encoder [h264_nvenc]
[2026-09-28 09:55:50.680]: Info: Creating encoder [hevc_nvenc]
[2026-09-28 09:55:51.135]: Info: Creating encoder [hevc_nvenc]
[2026-09-28 09:55:51.135]: Info: Color coding: SDR (Rec. 709)
[2026-09-28 09:55:51.135]: Info: Color depth: 10-bit
[2026-09-28 09:55:51.166]: Error: cuda::cuda_t doesn't support any format other than AV_PIX_FMT_NV12 and AV_PIX_FMT_YUV444P
[2026-09-28 09:55:51.585]: Info: Creating encoder [hevc_nvenc]
[2026-09-28 09:55:51.618]: Error: cuda::cuda_t doesn't support any format other than AV_PIX_FMT_NV12 and AV_PIX_FMT_YUV444P
[2026-09-28 09:55:52.395]: Info:
[2026-09-28 09:55:52.395]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2026-09-28 09:55:52.395]: Info:
[2026-09-28 09:55:52.395]: Info: Found H.264 encoder: h264_nvenc [nvenc]
[2026-09-28 09:55:52.395]: Info: Found HEVC encoder: hevc_nvenc [nvenc]
[2026-09-28 09:55:52.396]: Info: Starting system tray
[2026-09-28 09:56:19.238]: Info: Executing [Desktop]
[2026-09-28 09:56:19.896]: Info: New streaming session started [active sessions: 1]
[2026-09-28 09:56:20.091]: Info: CLIENT CONNECTED
[2026-09-28 09:56:20.263]: Info: Virtual Desktop: 1680x1050
[2026-09-28 09:56:20.267]: Info: Opus initialized: 48 kHz, 2 channels, 512 kbps (total), LOWDELAY
[2026-09-28 09:56:20.287]: Info: Screencasting with NvFBC
[2026-09-28 09:56:20.411]: Info: Creating encoder [hevc_nvenc]
[2026-09-28 09:56:20.411]: Info: Color coding: SDR (Rec. 601)
[2026-09-28 09:56:20.411]: Info: Color depth: 8-bit
[2026-09-28 09:56:20.411]: Info: Color range: MPEG
[2026-09-28 09:56:20.411]: Info: Streaming bitrate is 30988000
[2026-09-28 09:56:20.517]: Info: Minimum FPS target set to ~30fps (33.3333ms)
2. xrdp-created session, default settings (NvFBC succeeds — but
drm_workerstill empty)24594: = drm_worker
24595: cap_sys_nice=p egl_worker
[2026-09-28 09:48:39.369]: Info: Sunshine version: 2026.914.233613 commit: 63d35f7
[2026-09-28 09:48:39.369]: Info: config: 'csrf_allowed_origins' = [redacted]
[2026-09-28 09:48:39.457]: Error: Failed to create session: Version mismatch between NvFBC and the X driver interface
[2026-09-28 09:48:39.457]: Info: /dev/dri/card0 -> nvidia-drm
[2026-09-28 09:48:39.458]: Info: Detecting displays
[2026-09-28 09:48:39.458]: Info: Detected display: rdp0 (id: 0)rdp0 connected: true
[2026-09-28 09:48:39.462]: Error: [portalgrab] RemoteDesktop CreateSession failed with response code: 2
[2026-09-28 09:48:39.463]: Error: [portalgrab] ScreenCast CreateSession failed with response code: 2
[2026-09-28 09:48:39.496]: Info: Trying encoder [nvenc]
[2026-09-28 09:48:39.496]: Info: [misc] drop_elevated_privileges succeeded in dropping capabilities
[2026-09-28 09:48:39.496]: Error: Failed to drop CAP_SYS_ADMIN
[2026-09-28 09:48:39.496]: Info: Screencasting with X11
[2026-09-28 09:48:39.496]: Info: Creating encoder [h264_nvenc]
[2026-09-28 09:48:39.496]: Info: Color coding: SDR (Rec. 601)
[2026-09-28 09:48:39.496]: Info: Color depth: 8-bit
[2026-09-28 09:48:39.496]: Info: Color range: JPEG
[2026-09-28 09:48:39.641]: Info: Streaming bitrate is 1000000
[2026-09-28 09:48:39.910]: Info: Creating encoder [hevc_nvenc]
[2026-09-28 09:48:40.178]: Info: Creating encoder [av1_nvenc]
[2026-09-28 09:48:40.187]: Warning: [av1_nvenc @ 0x315f7c0] Codec not supported
[2026-09-28 09:48:40.187]: Error: [av1_nvenc @ 0x315f7c0] Provided device doesn't support required NVENC features
[2026-09-28 09:48:40.438]: Error: Could not open codec [av1_nvenc]: Function not implemented
[2026-09-28 09:48:40.439]: Info: Creating encoder [h264_nvenc]
[2026-09-28 09:48:40.707]: Info: Creating encoder [hevc_nvenc]
[2026-09-28 09:48:40.975]: Info: Creating encoder [hevc_nvenc]
[2026-09-28 09:48:40.975]: Info: Color coding: SDR (Rec. 709)
[2026-09-28 09:48:40.975]: Info: Color depth: 10-bit
[2026-09-28 09:48:41.005]: Error: cuda::cuda_t doesn't support any format other than AV_PIX_FMT_NV12 and AV_PIX_FMT_YUV444P
[2026-09-28 09:48:41.242]: Info: Creating encoder [hevc_nvenc]
[2026-09-28 09:48:41.276]: Error: cuda::cuda_t doesn't support any format other than AV_PIX_FMT_NV12 and AV_PIX_FMT_YUV444P
[2026-09-28 09:48:41.509]: Info:
[2026-09-28 09:48:41.509]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2026-09-28 09:48:41.509]: Info:
[2026-09-28 09:48:41.509]: Info: Found H.264 encoder: h264_nvenc [nvenc]
[2026-09-28 09:48:41.509]: Info: Found HEVC encoder: hevc_nvenc [nvenc]
[2026-09-28 09:48:41.510]: Info: Starting system tray
[2026-09-28 09:49:28.311]: Info: Executing [Desktop]
[2026-09-28 09:49:28.953]: Info: New streaming session started [active sessions: 1]
[2026-09-28 09:49:29.009]: Warning: setpriority failed for nice -10: Permission denied
[2026-09-28 09:49:29.149]: Info: CLIENT CONNECTED
[2026-09-28 09:49:29.150]: Info: Detecting displays
[2026-09-28 09:49:29.151]: Info: Detected display: rdp0 (id: 0)rdp0 connected: true
[2026-09-28 09:49:29.151]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-28 09:49:29.151]: Info: [misc] drop_elevated_privileges succeeded in dropping capabilities
[2026-09-28 09:49:29.151]: Info: Screencasting with X11
[2026-09-28 09:49:29.152]: Info: Streaming display: rdp0 with res 1680x1050 offset by 0x0
[2026-09-28 09:49:29.165]: Info: Creating encoder [hevc_nvenc]
[2026-09-28 09:49:29.165]: Info: Color coding: SDR (Rec. 601)
[2026-09-28 09:49:29.165]: Info: Color depth: 8-bit
[2026-09-28 09:49:29.165]: Info: Color range: MPEG
[2026-09-28 09:49:29.165]: Info: Streaming bitrate is 30988000
[2026-09-28 09:49:29.202]: Info: Minimum FPS target set to ~30fps (33.3333ms)
3. Direct login,
capture = kms(fails)4446: = drm_worker
4447: cap_sys_nice=p egl_worker
[2026-09-28 10:04:07.453]: Info: Sunshine version: 2026.914.233613 commit: 63d35f7
[2026-09-28 10:04:07.453]: Info: config: 'capture' = kms
[2026-09-28 10:04:07.453]: Info: config: 'csrf_allowed_origins' = [redacted]
[2026-09-28 10:04:07.455]: Info: /dev/dri/card0 -> nvidia-drm
[2026-09-28 10:04:07.455]: Error: [wayland] Environment variable WAYLAND_DISPLAY has not been defined
[2026-09-28 10:04:07.455]: Error: Unable to initialize capture method
[2026-09-28 10:04:07.455]: Error: Platform failed to initialize
[2026-09-28 10:04:07.456]: Info: Trying encoder [nvenc]
[2026-09-28 10:04:07.456]: Info: [misc] drop_elevated_privileges succeeded in dropping capabilities
[2026-09-28 10:04:07.456]: Error: Failed to drop CAP_SYS_ADMIN
[2026-09-28 10:04:07.656]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-28 10:04:07.856]: Info: Encoder [nvenc] failed
[2026-09-28 10:04:07.856]: Info: Trying encoder [vulkan]
[2026-09-28 10:04:08.056]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-28 10:04:08.256]: Info: Encoder [vulkan] failed
[2026-09-28 10:04:08.256]: Info: Trying encoder [vaapi]
[2026-09-28 10:04:08.456]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-28 10:04:08.656]: Info: Encoder [vaapi] failed
[2026-09-28 10:04:08.656]: Info: Trying encoder [software]
[2026-09-28 10:04:08.857]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-28 10:04:09.057]: Info: Encoder [software] failed
[2026-09-28 10:04:09.057]: Fatal: Unable to find display or encoder during startup.
[2026-09-28 10:04:09.057]: Fatal: Please check that a display is connected and powered on.
[2026-09-28 10:04:09.057]: Error: Video failed to find working encoder
4. xrdp-created session,
capture = kms(fails)24358: = drm_worker
24359: cap_sys_nice=p egl_worker
[2026-09-28 09:43:46.005]: Info: Sunshine version: 2026.914.233613 commit: 63d35f7
[2026-09-28 09:43:46.005]: Info: config: 'capture' = kms
[2026-09-28 09:43:46.005]: Info: config: 'csrf_allowed_origins' = [redacted]
[2026-09-28 09:43:46.006]: Info: /dev/dri/card0 -> nvidia-drm
[2026-09-28 09:43:46.006]: Error: [wayland] Environment variable WAYLAND_DISPLAY has not been defined
[2026-09-28 09:43:46.006]: Error: Unable to initialize capture method
[2026-09-28 09:43:46.006]: Error: Platform failed to initialize
[2026-09-28 09:43:46.008]: Info: Trying encoder [nvenc]
[2026-09-28 09:43:46.008]: Info: [misc] drop_elevated_privileges succeeded in dropping capabilities
[2026-09-28 09:43:46.008]: Error: Failed to drop CAP_SYS_ADMIN
[2026-09-28 09:43:46.208]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-28 09:43:46.408]: Info: Encoder [nvenc] failed
[2026-09-28 09:43:46.408]: Info: Trying encoder [vulkan]
[2026-09-28 09:43:46.608]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-28 09:43:46.808]: Info: Encoder [vulkan] failed
[2026-09-28 09:43:46.808]: Info: Trying encoder [vaapi]
[2026-09-28 09:43:47.008]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-28 09:43:47.208]: Info: Encoder [vaapi] failed
[2026-09-28 09:43:47.208]: Info: Trying encoder [software]
[2026-09-28 09:43:47.408]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-28 09:43:47.608]: Info: Encoder [software] failed
[2026-09-28 09:43:47.608]: Fatal: Unable to find display or encoder during startup.
[2026-09-28 09:43:47.608]: Fatal: Please check that a display is connected and powered on.
[2026-09-28 09:43:47.608]: Error: Video failed to find working encoder
Moonlight's client-side error for this scenario:
Happy to run any further tests that you need.
Thanks
psyke83 commentedon Sep 28, 2026
I believe you're misdiagnosing case 2; NvFBC fails here and falls back to X11 capture. Since you're building yourself, can you try this change? Feel free to test with all configurations, but I'm interested in a log with
capture = kmsfor now. It might be a good idea to update to latest master as well.It might also be helpful to try unmodified master branch with
capture = kmsandencoder = softwareto see if that works.jessica-atk commentedon Sep 28, 2026
Hi,
I tested both scenarios on the latest
master(commit8b2c5247), direct login, as requested.1. Unmodified
master,capture = kms,encoder = softwareWorker thread capabilities:
8269: = drm_worker
8270: = egl_worker
Full log:
[2026-09-28 15:44:23.483]: Info: Sunshine version: master-8b2c5247 commit: 8b2c524
[2026-09-28 15:44:23.483]: Info: Package Publisher: Third Party Publisher
[2026-09-28 15:44:23.483]: Info: Publisher Website:
[2026-09-28 15:44:23.483]: Info: Get support: https://app.lizardbyte.dev/support
[2026-09-28 15:44:23.483]: Info: config: 'capture' = kms
[2026-09-28 15:44:23.483]: Info: config: 'csrf_allowed_origins' = [redacted]
[2026-09-28 15:44:23.483]: Info: config: 'encoder' = software
[2026-09-28 15:44:23.484]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-28 15:44:23.485]: Info: /dev/dri/card0 -> nvidia-drm
[2026-09-28 15:44:23.485]: Error: [wayland] Environment variable WAYLAND_DISPLAY has not been defined
[2026-09-28 15:44:23.485]: Error: Unable to initialize capture method
[2026-09-28 15:44:23.485]: Error: Platform failed to initialize
[2026-09-28 15:44:23.486]: Info: Trying encoder [software]
[2026-09-28 15:44:23.486]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-28 15:44:23.486]: Info: [misc] drop_elevated_privileges succeeded in dropping capabilities
[2026-09-28 15:44:23.886]: Info: Encoder [software] failed
[2026-09-28 15:44:23.886]: Error: Couldn't find any working encoder matching [software]
[2026-09-28 15:44:23.886]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-09-28 15:44:23.886]: Info: Trying encoder [nvenc]
[2026-09-28 15:44:24.286]: Info: Encoder [nvenc] failed
[2026-09-28 15:44:24.286]: Info: Trying encoder [vulkan]
[2026-09-28 15:44:24.686]: Info: Encoder [vulkan] failed
[2026-09-28 15:44:24.686]: Info: Trying encoder [vaapi]
[2026-09-28 15:44:25.086]: Info: Encoder [vaapi] failed
[2026-09-28 15:44:25.087]: Fatal: Unable to find display or encoder during startup.
[2026-09-28 15:44:25.087]: Fatal: Please check that a display is connected and powered on.
[2026-09-28 15:44:25.087]: Error: Video failed to find working encoder
[2026-09-28 15:44:25.087]: Info: Starting system tray
[2026-09-28 15:44:25.088]: Info: Adding avahi service ld1-vfx112lnx
[2026-09-28 15:44:25.092]: Info: Configuration UI available at [https://localhost:47990]
[2026-09-28 15:44:25.103]: Info: System tray created
[2026-09-28 15:44:25.103]: Info: Starting main loop
[2026-09-28 15:44:26.077]: Info: Avahi service ld1-vfx112lnx successfully established.
[2026-09-28 15:45:04.936]: Info: Display device configuration is disabled. Reverting any active display device configuration.
[2026-09-28 15:45:04.936]: Info: Trying encoder [software]
[2026-09-28 15:45:05.336]: Info: Encoder [software] failed
[2026-09-28 15:45:05.336]: Error: Couldn't find any working encoder matching [software]
[2026-09-28 15:45:05.336]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-09-28 15:45:05.336]: Info: Trying encoder [nvenc]
[2026-09-28 15:45:05.736]: Info: Encoder [nvenc] failed
[2026-09-28 15:45:05.736]: Info: Trying encoder [vulkan]
[2026-09-28 15:45:06.136]: Info: Encoder [vulkan] failed
[2026-09-28 15:45:06.136]: Info: Trying encoder [vaapi]
[2026-09-28 15:45:06.537]: Info: Encoder [vaapi] failed
[2026-09-28 15:45:06.537]: Fatal: Unable to find display or encoder during startup.
[2026-09-28 15:45:06.537]: Fatal: Please check that a display is connected and powered on.
2. Patched (
drop_drm_worker_privileges()commented out),capture = kms, encoder on auto/defaultWorker thread capabilities (patch confirmed working — no drop occurred):
9481: cap_sys_admin,cap_sys_nice=p drm_worker
9482: cap_sys_nice=p egl_worker
Full log:
[2026-09-28 15:53:41.893]: Info: Sunshine version: master-8b2c5247 commit: 8b2c524
[2026-09-28 15:53:41.893]: Info: Package Publisher: Third Party Publisher
[2026-09-28 15:53:41.893]: Info: Publisher Website:
[2026-09-28 15:53:41.893]: Info: Get support: https://app.lizardbyte.dev/support
[2026-09-28 15:53:41.893]: Info: config: 'capture' = kms
[2026-09-28 15:53:41.893]: Info: config: 'csrf_allowed_origins' = [redacted]
[2026-09-28 15:53:41.894]: Info: /dev/dri/card0 -> nvidia-drm
[2026-09-28 15:53:41.895]: Error: [wayland] Environment variable WAYLAND_DISPLAY has not been defined
[2026-09-28 15:53:41.895]: Error: Unable to initialize capture method
[2026-09-28 15:53:41.895]: Error: Platform failed to initialize
[2026-09-28 15:53:41.896]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-09-28 15:53:41.896]: Info: Trying encoder [nvenc]
[2026-09-28 15:53:42.296]: Info: Encoder [nvenc] failed
[2026-09-28 15:53:42.296]: Info: Trying encoder [vulkan]
[2026-09-28 15:53:42.696]: Info: Encoder [vulkan] failed
[2026-09-28 15:53:42.696]: Info: Trying encoder [vaapi]
[2026-09-28 15:53:43.096]: Info: Encoder [vaapi] failed
[2026-09-28 15:53:43.096]: Info: Trying encoder [software]
[2026-09-28 15:53:43.496]: Info: Encoder [software] failed
[2026-09-28 15:53:43.496]: Fatal: Unable to find display or encoder during startup.
[2026-09-28 15:53:43.496]: Fatal: Please check that a display is connected and powered on.
[2026-09-28 15:53:43.496]: Error: Video failed to find working encoder
[2026-09-28 15:53:43.497]: Info: Starting system tray
[2026-09-28 15:53:43.498]: Info: Adding avahi service ld1-vfx112lnx
[2026-09-28 15:53:43.502]: Info: Configuration UI available at [https://localhost:47990]
[2026-09-28 15:53:43.514]: Info: System tray created
[2026-09-28 15:53:43.514]: Info: Starting main loop
[2026-09-28 15:53:44.392]: Info: Avahi service ld1-vfx112lnx successfully established.
[2026-09-28 15:57:17.663]: Info: Display device configuration is disabled. Reverting any active display device configuration.
[2026-09-28 15:57:17.663]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-09-28 15:57:17.663]: Info: Trying encoder [nvenc]
[2026-09-28 15:57:18.063]: Info: Encoder [nvenc] failed
[2026-09-28 15:57:18.063]: Info: Trying encoder [vulkan]
[2026-09-28 15:57:18.463]: Info: Encoder [vulkan] failed
[2026-09-28 15:57:18.463]: Info: Trying encoder [vaapi]
[2026-09-28 15:57:18.863]: Info: Encoder [vaapi] failed
[2026-09-28 15:57:18.863]: Info: Trying encoder [software]
[2026-09-28 15:57:19.263]: Info: Encoder [software] failed
[2026-09-28 15:57:19.263]: Fatal: Unable to find display or encoder during startup.
[2026-09-28 15:57:19.263]: Fatal: Please check that a display is connected and powered on.
Summary
The patch works as intended —
drm_workercorrectly retainscap_sys_admin,cap_sys_nice=pthroughout, and (unlike test 1) there's noFailed to gain CAP_SYS_ADMINanywhere in the log this time. However, KMS still appears to fail:Unable to initialize capture method/Platform failed to initialize, in both the unpatched and patched builds.Hope that helps - happy to continue testing with you
psyke83 commentedon Sep 28, 2026
Thanks. KMS is not initializing but there are no useful logs to tells us why. It might help to see a debug log with kms capture attempted using the patched build.
If that doesn't show anything obvious, we might need to instrument more debug logging.
jessica-atk commentedon Sep 28, 2026
Hi - quite a bit of repeated output here - have trimmed in place.
Also I have redacted the hostname and IP
Thanks
Full startup log (
min_log_level = 0,capture = kms,encoder = auto, patch applied):psyke83 commentedon Sep 28, 2026
Please test: #5828
Debug logs would be helpful again. You can probably cut the log off after "Starting main loop" unless you happen to see any kms/drm related errors after that point. This PR as-is may not resolve the issue yet, but it may provide more comprehensive error logs.
If you could also capture an strace log of the ioctl calls it would greatly help me to pin down the potential issue.
In order to allow kmsgrab to work via strace, you need to add capabilities to your strace binary:
Then you can capture an ioctl trace via a direct launch of Sunshine with
capture = kmsset (and making sure that the service or other instances aren't running):Then share the sunshine_ioctl.log that's created.
Don't forget, when you finished capturing logs, restore strace's normal permissions by dropping the capabilities that were added:
Thanks.
jessica-atk commentedon Sep 29, 2026
Good Morning @psyke83,
I just tested PR #5828 (commit
859420e7) on a direct login,capture = kms,min_log_level = 0.Debug log (trimmed per your suggestion — cut after
Starting main loop, kept the second capture attempt since it's directly relevant)[2026-09-29 10:15:54.945]: Info: Sunshine version: pr5828-859420e7 commit: 859420e
[2026-09-29 10:15:54.945]: Info: config: 'capture' = kms
[2026-09-29 10:15:54.945]: Info: config: 'min_log_level' = 0
[2026-09-29 10:15:54.945]: Info: config: 'csrf_allowed_origins' = [redacted]
[2026-09-29 10:15:54.948]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-29 10:15:54.949]: Info: /dev/dri/card0 -> nvidia-drm
[2026-09-29 10:15:54.949]: Debug: Opening render node: /dev/dri/renderD128
[2026-09-29 10:15:54.951]: Error: [wayland] Environment variable WAYLAND_DISPLAY has not been defined
[2026-09-29 10:15:54.951]: Debug: Monitor description
[2026-09-29 10:15:54.951]: Debug: Resolution: 0x0
[2026-09-29 10:15:54.951]: Debug: Offset: 0x0
[2026-09-29 10:15:54.951]: Debug: Desktop resolution: 0x0
[2026-09-29 10:15:54.951]: Debug: Final KMS display_names return list:
[2026-09-29 10:15:54.951]: Error: Unable to initialize capture method
[2026-09-29 10:15:54.951]: Error: Platform failed to initialize
[2026-09-29 10:15:54.984]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-09-29 10:15:54.984]: Info: Trying encoder [nvenc]
[2026-09-29 10:15:54.984]: Error: Failed to gain CAP_SYS_ADMIN
[2026-09-29 10:15:54.984]: Info: [misc] drop_elevated_privileges succeeded in dropping capabilities
[2026-09-29 10:15:55.384]: Info: Encoder [nvenc] failed
[2026-09-29 10:15:55.384]: Info: Trying encoder [vulkan]
[2026-09-29 10:15:55.784]: Info: Encoder [vulkan] failed
[2026-09-29 10:15:55.784]: Info: Trying encoder [vaapi]
[2026-09-29 10:15:56.184]: Info: Encoder [vaapi] failed
[2026-09-29 10:15:56.184]: Info: Trying encoder [software]
[2026-09-29 10:15:56.585]: Info: Encoder [software] failed
[2026-09-29 10:15:56.585]: Fatal: Unable to find display or encoder during startup.
[2026-09-29 10:15:56.585]: Fatal: Please check that a display is connected and powered on.
[2026-09-29 10:15:56.585]: Error: Video failed to find working encoder
[2026-09-29 10:15:56.587]: Info: Starting system tray
[2026-09-29 10:15:56.594]: Info: Adding avahi service
[2026-09-29 10:15:56.610]: Info: Configuration UI available at [https://localhost:47990]
[2026-09-29 10:15:56.762]: Info: System tray created
[2026-09-29 10:15:56.762]: Info: Starting main loop
[2026-09-29 10:15:57.548]: Info: Avahi service successfully established.
[... web UI polling, pairing handshake, and repeated /serverinfo requests omitted ...]
[2026-09-29 10:17:05.116]: Debug: DESTINATION :: /launch
[2026-09-29 10:17:05.116]: Debug: Host -- :47984
[2026-09-29 10:17:05.116]: Debug: appid -- 881448767
[2026-09-29 10:17:05.116]: Debug: mode -- 1920x1080x60
[2026-09-29 10:17:05.117]: Info: Display device configuration is disabled. Reverting any active display device configuration.
[2026-09-29 10:17:05.117]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-09-29 10:17:05.117]: Info: Trying encoder [nvenc]
[2026-09-29 10:17:05.516]: Info: Encoder [nvenc] failed
[2026-09-29 10:17:05.516]: Info: Trying encoder [vulkan]
[2026-09-29 10:17:05.916]: Info: Encoder [vulkan] failed
[2026-09-29 10:17:05.916]: Info: Trying encoder [vaapi]
[2026-09-29 10:17:06.316]: Info: Encoder [vaapi] failed
[2026-09-29 10:17:06.316]: Info: Trying encoder [software]
[2026-09-29 10:17:06.716]: Info: Encoder [software] failed
[2026-09-29 10:17:06.716]: Fatal: Unable to find display or encoder during startup.
[2026-09-29 10:17:06.716]: Fatal: Please check that a display is connected and powered on.
One thing maybe worth flagging: the second capture attempt (triggered by the actual Moonlight
/launchrequest) does not show a freshUnable to initialize capture method/Final KMS display_names return list:sequence — it goes straight to retrying encoders and failing onCAP_SYS_ADMINagain. Wondering if the empty-display-list result from the very first attempt is being cached/reused rather than re-evaluated on each session start?strace ioctl trace
Please see the attached
sunshine_ioctl.log(captured across both the initial startup and the Moonlight connection attempt, stopped with Ctrl+C once the client-side error appeared).sunshine_ioctl.log
Hope this is helpful - happy to continue testing!
psyke83 commentedon Sep 29, 2026
It seems that none of the ioctls that we would expect to fail are reporting failure, so this is concerning. However, looking at the open portion of nvidia's gpu driver source, they implement their own CAP_SYS_ADMIN guards in various parts of the code, so it's possible that there are additional guards compared to what's present in the open drivers (drmModeGetFB, drmModeGetFB2 and a few legacy ioctls that are not used by Sunshine). Depending on how it's implemented, an unprivileged call to a guarded ioctl might just return empty data (similar to the GETFB2 ioctls on open drivers) rather than an error code.
I've updated the PR branch to wrap the remaining drm calls that are shown in your log during startup probe. Please try another build with the new changes, and don't worry about providing any more ioctl logs.
jessica-atk commentedon Sep 29, 2026
Hi,
Thanks for continuing with this.
I've rebuilt with your update. PR #5828 branch (still
pr5828-859420e7, same commit — am assuming this is the intended state, but let me know if I should be on a different commit).Confirmed
setcap cap_sys_admin,cap_sys_nice=pwas correctly present on the freshly rebuilt binary before this run.Result: unfortunately still the same failure, both at initial startup and when triggered again via a real Moonlight connection:
[2026-09-29 17:28:04.089]: Info: Sunshine version: pr5828-859420e7 commit: 859420e
[2026-09-29 17:28:04.089]: Info: config: 'min_log_level' = 0
[2026-09-29 17:28:04.089]: Info: config: 'capture' = kms
[2026-09-29 17:28:04.089]: Info: config: 'csrf_allowed_origins' = [redacted]
[2026-09-29 17:28:04.091]: Info: /dev/dri/card0 -> nvidia-drm
[2026-09-29 17:28:04.091]: Debug: Opening render node: /dev/dri/renderD128
[2026-09-29 17:28:04.092]: Error: [wayland] Environment variable WAYLAND_DISPLAY has not been defined
[2026-09-29 17:28:04.092]: Debug: Monitor description
[2026-09-29 17:28:04.092]: Debug: Resolution: 0x0
[2026-09-29 17:28:04.092]: Debug: Offset: 0x0
[2026-09-29 17:28:04.092]: Debug: Desktop resolution: 0x0
[2026-09-29 17:28:04.092]: Debug: Final KMS display_names return list:
[2026-09-29 17:28:04.092]: Error: Unable to initialize capture method
[2026-09-29 17:28:04.092]: Error: Platform failed to initialize
[2026-09-29 17:28:04.094]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-09-29 17:28:04.094]: Info: Trying encoder [nvenc]
[2026-09-29 17:28:04.094]: Info: [misc] drop_elevated_privileges succeeded in dropping capabilities
[2026-09-29 17:28:04.494]: Info: Encoder [nvenc] failed
[2026-09-29 17:28:04.494]: Info: Trying encoder [vulkan]
[2026-09-29 17:28:04.894]: Info: Encoder [vulkan] failed
[2026-09-29 17:28:04.894]: Info: Trying encoder [vaapi]
[2026-09-29 17:28:05.294]: Info: Encoder [vaapi] failed
[2026-09-29 17:28:05.294]: Info: Trying encoder [software]
[2026-09-29 17:28:05.694]: Info: Encoder [software] failed
[2026-09-29 17:28:05.694]: Fatal: Unable to find display or encoder during startup.
[2026-09-29 17:28:05.694]: Fatal: Please check that a display is connected and powered on.
[2026-09-29 17:28:05.694]: Error: Video failed to find working encoder
[2026-09-29 17:28:05.695]: Info: Starting system tray
[2026-09-29 17:28:05.696]: Info: Adding avahi service
[2026-09-29 17:28:05.700]: Info: Configuration UI available at [https://localhost:47990]
[2026-09-29 17:28:05.712]: Info: System tray created
[2026-09-29 17:28:05.712]: Info: Starting main loop
[... web UI polling / pairing handshake omitted ...]
[2026-09-29 17:28:48.263]: Debug: DESTINATION :: /launch
[2026-09-29 17:28:48.263]: Debug: Host -- :47984
[2026-09-29 17:28:48.263]: Debug: appid -- 881448767
[2026-09-29 17:28:48.263]: Debug: mode -- 1920x1080x60
[2026-09-29 17:28:48.263]: Info: Display device configuration is disabled. Reverting any active display device configuration.
[2026-09-29 17:28:48.263]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-09-29 17:28:48.263]: Info: Trying encoder [nvenc]
[2026-09-29 17:28:48.663]: Info: Encoder [nvenc] failed
[2026-09-29 17:28:48.663]: Info: Trying encoder [vulkan]
[2026-09-29 17:28:49.063]: Info: Encoder [vulkan] failed
[2026-09-29 17:28:49.063]: Info: Trying encoder [vaapi]
[2026-09-29 17:28:49.463]: Info: Encoder [vaapi] failed
[2026-09-29 17:28:49.463]: Info: Trying encoder [software]
[2026-09-29 17:28:49.863]: Info: Encoder [software] failed
[2026-09-29 17:28:49.863]: Fatal: Unable to find display or encoder during startup.
[2026-09-29 17:28:49.863]: Fatal: Please check that a display is connected and powered on.
No
Failed to gain CAP_SYS_ADMINerror appears anywhere in this run at all — butFinal KMS display_names return list:is still empty, andUnable to initialize capture methodis there.Feels like progress!
Thanks
14 remaining items
jessica-atk commentedon Sep 30, 2026
Here's the temp 3 result:
$ grep -E "privileges matrix| connector" ~/.config/sunshine/sunshine.log | cut -b34- | sort | uniq
[misc] has_elevated_privileges matrix: permitted:1 effective:0 platf::kms::card_t::connector_interal_t platf::kms::card_t::connector(uint32_t)
[misc] has_elevated_privileges matrix: permitted:1 effective:0 platf::kms::res_t platf::kms::card_t::res()
[misc] has_elevated_privileges matrix: permitted:1 effective:0 void platf::kms::plane_it_t::load_next_valid_plane()
[misc] has_elevated_privileges matrix: permitted:1 effective:1 int platf::open_drm_card_fd(const std::filesystem::__cxx11::path&, int)
[misc] has_elevated_privileges matrix: permitted:1 effective:1 platf::kms::card_t::init(const char*)::<lambda()>
I changed the URL as there was a typo above - assume it was correct
git fetch https://github.com/psyke83/SunshineStream.git drm_worker_rework_temp3
Will test the pre-worker too
jessica-atk commentedon Sep 30, 2026
Hi,
Tested the pre-DRM-worker commit (
8c74826f2ea28a517cab28c16370ce08333ff9bb)[2026-09-30 21:02:59.569]: Info: /dev/dri/card0 -> nvidia-drm
[2026-09-30 21:02:59.569]: Debug: Opening render node: /dev/dri/renderD128
[2026-09-30 21:02:59.569]: Error: [wayland] Environment variable WAYLAND_DISPLAY has not been defined
[2026-09-30 21:02:59.569]: Debug: Final KMS display_names return list:
[2026-09-30 21:02:59.569]: Error: Unable to initialize capture method
[2026-09-30 21:02:59.569]: Error: Platform failed to initialize
I'll retest this all in the morning when directly in front of the machine, rather than over remote xrdp sessions.
Perhaps the results are being skewed and I don't want to lead us down any false paths
Thanks
jessica-atk commentedon Sep 30, 2026
I just retested over a nomachine session to the machine, removing the virtual seat and reran all three temp branches on a confirmed genuine direct/physical login (verified via
loginctl session-statusshowingSeat: seat0,tty2)Result: identical to the xrdp-session tests in every case. Session type made no difference at all.
temp1
[misc] has_elevated_privileges matrix: permitted:0 effective:0 platf::kms::card_t::connector_interal_t platf::kms::card_t::connector(uint32_t)
[misc] has_elevated_privileges matrix: permitted:0 effective:0 platf::kms::res_t platf::kms::card_t::res()
[misc] has_elevated_privileges matrix: permitted:0 effective:0 void platf::kms::plane_it_t::load_next_valid_plane()
[misc] has_elevated_privileges matrix: permitted:1 effective:1 int platf::open_drm_card_fd(const std::filesystem::__cxx11::path&, int)
[misc] has_elevated_privileges matrix: permitted:1 effective:1 platf::kms::card_t::init(const char*)::<lambda()>
temp2
[misc] has_elevated_privileges matrix: permitted:1 effective:0 platf::kms::card_t::connector_interal_t platf::kms::card_t::connector(uint32_t)
[misc] has_elevated_privileges matrix: permitted:1 effective:0 platf::kms::res_t platf::kms::card_t::res()
[misc] has_elevated_privileges matrix: permitted:1 effective:0 void platf::kms::plane_it_t::load_next_valid_plane()
[misc] has_elevated_privileges matrix: permitted:1 effective:1 int platf::open_drm_card_fd(const std::filesystem::__cxx11::path&, int)
[misc] has_elevated_privileges matrix: permitted:1 effective:1 platf::kms::card_t::init(const char*)::<lambda()>
temp3
[misc] has_elevated_privileges matrix: permitted:1 effective:0 platf::kms::card_t::connector_interal_t platf::kms::card_t::connector(uint32_t)
[misc] has_elevated_privileges matrix: permitted:1 effective:0 platf::kms::res_t platf::kms::card_t::res()
[misc] has_elevated_privileges matrix: permitted:1 effective:0 void platf::kms::plane_it_t::load_next_valid_plane()
[misc] has_elevated_privileges matrix: permitted:1 effective:1 int platf::open_drm_card_fd(const std::filesystem::__cxx11::path&, int)
[misc] has_elevated_privileges matrix: permitted:1 effective:1 platf::kms::card_t::init(const char*)::<lambda()>
Also confirmed separately: the pre-DRM-worker commit (
8c74826f) fails the same way on direct login too.Happy to keep working on this with you! Would love to get to the bottom if it.
If you need any info around more machine specs, installation, how the package was built etc, do let me know.
Thanks
psyke83 commentedon Sep 30, 2026
What version of Sunshine was working with this configuration? On my card, KMS capture doesn't initialize unless a real monitor is connected and active (or a port is forced via an edid override). If you're only ever testing with a monitor turned off or unconnected, it probably won't work on NVIDIA either.
It seems we can rule out the drm worker changes as the culprit, at least.
jessica-atk commentedon Oct 1, 2026
There is always a monitor connected and turned on during testing - was wfh yesterday so conducted testing remotely with xrdp and nomachine in various scenarios - the display was still connected
Sunshine works fine in 8bit mode but we really need the 10bit capture.
The latest releases all work with that 8bit mode. Just not quite understanding why we are not achieving KMS capture.
Colour accuracy is quite critical to our workflow, as is 4:4:4 for pixel accuracy.
psyke83 commentedon Oct 1, 2026
I think the best course of action is to bisect the issue and find the actual culprit causing kms to regress on your setup. I don't think you mentioned what version of Sunshine actually worked; is it possible that other changes to your system outside the scope of Sunshine is causing the issue? It seems fairly clear that the drm worker change is not the culprit.
I'm not sure what procedure you use to build Sunshine, but if you do bisect, make sure to clear your build directory on each rebuild to avoid stale build objects interfering with the result. Finally, make sure that in-place checkouts of older revisions account for changes to submodules; so, if needed, init/sync/update the submodules and verify a clean working directory before building. At the very least you should probably rebuild and retest the last known working revision.
jessica-atk commentedon Oct 1, 2026
Hi,
I haven't had a successful implementation of Sunshine that can leverage KMS.
An example of a successful 8bit capable build was from the latest release - (
v2026.929.125923) using NvFBC/X11 viacapture = autoplus the other tests that we performed.The build directory was also cleaned between tests.
For every build, these commands are used:
git fetch/checkout <branch or commit>git submodule update --init --recursiveFollowing up with something that might help narrow this down further. I built
modetestfromlibdrmsource (not packaged for EL9) to get an independent, Sunshine-free view of the kernel/driver's own KMS state.The single CRTC on this system shows as completely blank/uncommitted, despite the display it should be driving being active:
modetest -M nvidia-drm -p | head -10CRTCs:id fb pos size61 0 (0,0) (0x0)#0 -nan 0 0 0 0 0 0 0 0 0 flags: ; type:Everything is blank and this is the only CRTC object the driver exposes at all.
It fully understands that a monitor is attached and available:
131 0 connected DP-2...and offers the full colour range etc
In our Kernel:
nvidia_modeset=1
Could you please let me know which GPU / Driver configuration you are using for your own testing?
I'm wondering if this issue is something to do with Nvidia's driver/kernel/hardware combination.
I'm happy to continue testing as I'd much rather find a solution to this than leave it after all our efforts - we'd love to be able to use Sunshine!
Thanks
psyke83 commentedon Oct 1, 2026
I'm afraid I don't currently possess any NVIDIA graphics cards to assist you. My setup is a RX 6600 GPU and I'm usually testing Sunshine against up-to-date Arch or Fedora 44. Your distro selection certainly could be a factor given that it's still running kernel 5.14 and a year old NVIDIA driver.
This sounds like a production workstation, but would it be out of the question to boot from an up-to-date Linux live cd that supports NVIDIA drivers in the live session, and temporarily install a Sunshine via binary release to see if KMS works on this hardware when the kernel and driver is more recent? I'm afraid I can't provide much more assistance; aside from only having an AMD GPU, I don't even have any HDR/4:4:4 capable clients to test 10 bit encoding properly.
psyke83 commentedon Oct 1, 2026
I did notice one discrepancy: you wrote "nvidia_modeset=1". Assuming you didn't write in shorthand, this doesn't seem right? If the kernel driver is
nvidia-drmas seen in your prior logs, and if themodesetmodule option name is correct, then you should be appendingnvidia-drm.modeset=1ornvidia_drm.modeset=1to your kernel arguments.Edit: yes, grepping our documentation confirms this (and documents the latter type with an underscore).
jessica-atk commentedon Oct 1, 2026
Hi,
Thanks for pointing out the discrepancy.
Indeed it's nvidia-drm.modeset=1 in the kernel args.
I like the idea of testing on a Live CD to see if our Nvidia hardware is passed through at all. Happy to test that with Fedora 44 and a more recent Nvidia driver (ours needs an update anyway).
Although we're quite locked to very specific kernels etc, it may well provide a valuable insight into Nvidia and Sunshine.
I'd like to keep this discussion going and report back findings.
Whilst we are absolutely bound to RHEL, I'd like to dive deeper and see what's possible.
jessica-atk commentedon Oct 2, 2026
Hi,
I've done a lot more testing.
I built a Fedora 44 machine - still with Nvidia RTX4000.
In that setup, I managed to get Sunshine to stream 4:4:4 10bit HEVC.
The Sunshine settings need to force KMS and Vulkan, but that's not an issue.
Here's the rub - it only works with Wayland display (default with Fedora44 and Gnome).
I installed XFCE desktop to test X11 as we use X11 in house and that's where it stops working.
I think that there is something in the way that X11 interacts with nvidia-drm that the connectors do not get exposed.
So this is some good news and some bad news.
The good news is that this does not appear right now to be a restriction that Nvidia is enforcing in its drivers.
The bad news is that Wayland is still quite new and experimental and many applications that we need to use will only work with X11.
So the next stage for investigation to see if it is possible to understand the Wayland behaviour that allows the connectors to be exposed, and if that behaviour can be replicated into X11 or if it is simply a hard 'NO' for 10bit implementation on X11.
If you have any ideas there, I'm all ears.
I'll report back if I find anything too. Exciting project!
Thanks
jessica-atk commentedon Oct 2, 2026
Hold on!
Got it working!!
What I said above is not entirely correct..........it IS the nvidia driver......or rather how it's utilised in X11.
In X11, nvidia is using its own
nvidia_drv.soThis is confirmed in the xorg log
Wayland's compositor (Mutter) must populate a real CRTC through the standard mechanism to function at all, while the classic X11 nvidia driver can, and apparently does, commit the actual mode-set through its own separate, private path, leaving the generic KMS state (what modetest and Sunshine both read) sitting empty even though the display is genuinely active!
Here's the fix :
lspci | grep -i nvidia2b:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 4000] (rev a1)
2b:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
2b:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)
2b:00.3 Serial bus controller: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)
0x2b = 43 (hex to dec)
sudo tee /etc/X11/xorg.conf.d/20-nvidia-modesetting.conf > /dev/null << 'EOF'Section "Device"Identifier "NVIDIA Card"Driver "modesetting"Option "AccelMethod" "none"BusID "PCI:43:0:0"EndSectionEOFLog out and in again
grep -i "modesetting\|nvidia\|LoadModule" ~/.local/share/xorg/Xorg.*.log | head -20[ 228.741] (**) | |-->Device "NVIDIA Card"
[ 228.754] (II) LoadModule: "glx"
[ 228.755] (II) LoadModule: "modesetting"
[ 228.755] (II) Loading /usr/lib64/xorg/modules/drivers/modesetting_drv.so
[ 228.755] (II) Module modesetting: vendor="X.Org Foundation"
[ 228.755] (II) modesetting: Driver for Modesetting Kernel Drivers: kms
[ 228.756] (II) Applying OutputClass "nvidia" options to /dev/dri/card0
[ 229.098] (II) LoadModule: "fb"
[ 229.098] (II) LoadModule: "shadow"
[ 229.220] (II) LoadModule: "libinput"
[ 229.238] (II) config/udev: Adding input device HDA NVidia HDMI/DP,pcm=8 (/dev/input/event10)
[ 229.239] (II) config/udev: Adding input device HDA NVidia HDMI/DP,pcm=9 (/dev/input/event11)
[ 229.239] (II) config/udev: Adding input device HDA NVidia HDMI/DP,pcm=3 (/dev/input/event8)
[ 229.240] (II) config/udev: Adding input device HDA NVidia HDMI/DP,pcm=7 (/dev/input/event9)
And we can see the modesetting module is now correctly loaded
Confirmed:
modetest -M nvidia-drm -p | head -10opened device NVIDIA DRM driver on driver nvidia-drm (version 0.0.0 at 0)
CRTCs:
id fb pos size
205 847 (0,0) (1680x1050)
#0 59.95 1680 1784 1960 2240 1050 1053 1059 1089 146250 flags: nhsync, pvsync; type:
props:
25 VRR_ENABLED:
flags: range
values: 0 1
value: 0
Sunshine log :
[2026-10-02 15:03:53.942]: Info: Found H.264 encoder: h264_vulkan [vulkan]
[2026-10-02 15:03:53.942]: Info: Found HEVC encoder: hevc_vulkan [vulkan]
[2026-10-02 15:03:53.942]: Info: Executing [Desktop]
[2026-10-02 15:03:54.552]: Info: New streaming session started [active sessions: 1]
[2026-10-02 15:03:54.733]: Info: CLIENT CONNECTED
[2026-10-02 15:03:54.736]: Info: /dev/dri/card0 -> nvidia-drm
[2026-10-02 15:03:54.736]: Info: Screencasting with KMS
[2026-10-02 15:03:54.736]: Info: Mapped 'DP-1' to kmsgrab monitor index 0
[2026-10-02 15:03:54.736]: Info: /dev/dri/card0 -> nvidia-drm
[2026-10-02 15:03:54.736]: Info: Found monitor for DRM screencasting
[2026-10-02 15:03:54.737]: Info: Found connector ID [828]
[2026-10-02 15:03:54.737]: Info: Found cursor plane [199]
[2026-10-02 15:03:54.749]: Info: Creating encoder [hevc_vulkan]
[2026-10-02 15:03:54.749]: Info: Color coding: SDR (Rec. 601)
[2026-10-02 15:03:54.749]: Info: Color depth: 10-bit
[2026-10-02 15:03:54.749]: Info: Color range: MPEG
[2026-10-02 15:03:54.854]: Info: Setting default sink to: [sink-sunshine-stereo]
[2026-10-02 15:03:54.854]: Info: Found default monitor by name: sink-sunshine-stereo.monitor
[2026-10-02 15:03:54.902]: Info: Opus initialized: 48 kHz, 2 channels, 512 kbps (total), LOWDELAY
[2026-10-02 15:03:55.160]: Info: Streaming bitrate is 12588000
[2026-10-02 15:03:55.160]: Info: [vulkan] Encoder quality set to 5 (quality), driver range: 0-5
[2026-10-02 15:03:55.165]: Info: Vulkan encode using GPU: Quadro RTX 4000
[2026-10-02 15:03:55.168]: Info: Minimum FPS target set to ~30fps (33.3333ms)
Again, this testing was performed on the Fedora station running a more recent kernel
uname -r7.2.8-200.fc44.x86_64Nvidia drivers are also latest
NVIDIA-SMI 615.71.09 KMD Version: 615.71.09 CUDA UMD Version: 13.4Now I need to see if this will work on RHEL (specifically Rocky 9) that has a backward 580 nvidia driver and a much older kernel
uname -r5.14.0-570.22.1.el9_6.x86_64Will update!
jessica-atk commentedon Oct 2, 2026
Got it working in Rocky 9!
This is such good news.
Driver:
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64 580.178.04 Release Build
Kernel:
5.14.0-570.22.1.el9_6.x86_64
Phew! What an adventure
[2026-10-02 17:16:26.584]: Info: Found H.264 encoder: h264_vulkan [vulkan]
[2026-10-02 17:16:26.584]: Info: Found HEVC encoder: hevc_vulkan [vulkan]
[2026-10-02 17:16:26.584]: Info: Executing [Desktop]
[2026-10-02 17:16:27.238]: Info: New streaming session started [active sessions: 1]
[2026-10-02 17:16:27.434]: Info: CLIENT CONNECTED
[2026-10-02 17:16:27.437]: Info: /dev/dri/card0 -> nvidia-drm
[2026-10-02 17:16:27.437]: Info: Screencasting with KMS
[2026-10-02 17:16:27.437]: Info: Mapped 'DP-1' to kmsgrab monitor index 0
[2026-10-02 17:16:27.437]: Info: /dev/dri/card0 -> nvidia-drm
[2026-10-02 17:16:27.437]: Info: Found monitor for DRM screencasting
[2026-10-02 17:16:27.438]: Info: Found connector ID [127]
[2026-10-02 17:16:27.438]: Info: Found cursor plane [55]
[2026-10-02 17:16:27.451]: Info: Creating encoder [hevc_vulkan]
[2026-10-02 17:16:27.451]: Info: Color coding: SDR (Rec. 601)
[2026-10-02 17:16:27.451]: Info: Color depth: 10-bit
[2026-10-02 17:16:27.451]: Info: Color range: MPEG
[2026-10-02 17:16:27.553]: Info: Setting default sink to: [sink-sunshine-stereo]
[2026-10-02 17:16:27.554]: Info: Found default monitor by name: sink-sunshine-stereo.monitor
[2026-10-02 17:16:27.623]: Info: Opus initialized: 48 kHz, 2 channels, 512 kbps (total), LOWDELAY
[2026-10-02 17:16:27.831]: Info: Streaming bitrate is 12588000
[2026-10-02 17:16:27.832]: Info: [vulkan] Encoder quality set to 2 (balanced), driver range: 0-5
[2026-10-02 17:16:27.837]: Info: Vulkan encode using GPU: Quadro RTX 4000
[2026-10-02 17:16:27.840]: Info: Minimum FPS target set to ~30fps (33.3333ms)
It's still stating that it's Rec.601
However I did a colour test and it was very very close to true colour.
So perhaps the log itself is not reporting correctly?
10bit 4:4:4 is a game changer - that's PCoIP Ultra level of quality!
girlypop2000 commentedon Oct 2, 2026
Doesn't that basically unload the installed Nvidia driver and falls back to simple software renderer without acceleration or basically any features that the proprietary driver provides? Or did I misunderstood the command contents?
jessica-atk commentedon Oct 3, 2026
Hi @girlypop2000
Thanks for the question, happy to give my perspective on this.
I appreciate why it could seem that way as it does essentially ''change the X driver''.
Here are my thoughts on it:
The
modesettingdriver is an X.Org DDX (the 2D X11 integration layer). Its job is narrowly scoped to mode-setting; telling the kernel's DRM subsystem which connector drives which CRTC, at which resolution, and with which framebuffer. That's the only thing that changed. Everything else in the stack stays exactly as it was:The
nvidia/nvidia_drm/nvidia_modesetkernel modules remain loaded;nvidia-smikept working the whole time.OpenGL, Vulkan, and CUDA all continue to run through NVIDIA's own proprietary userspace libraries. These are a completely separate stack from the X11 DDX
NVENC's hardware encode engine is untouched. The GPU is accessed via NVIDIA's own driver regardless of which DDX is driving the display.
The hard evidence that we are not falling back to software encoding also comes directly out of Sunshine's own logs:
Info: [vulkan] Encoder quality set to 2 (balanced), driver range: 0-5Info: Vulkan encode using GPU: Quadro RTX 4000Those lines would only be present if the GPU's hardware encode path is genuinely active.
The
modesettingdriver's own 2D acceleration path (glamor) is separate from all of this. It only affects basic desktop rendering, not encoding, so doesn't factor into this concern at all.I appreciate that historically Nvidia drivers had a gap in GBM support (which glamor relies upon), but that has been fixed in modern drivers that ship with a GBM backend.
For transparency, we use
580.178.04.I also verified this visually - looking at GPU load and processes during testing as well as the good old ''eyeball test''.
The results were great, the colours rich and the pixel edges sharp.
So for me, I would be comfortable deploying this as a solution. Especially as the results are so demonstrably superior to other remote desktop solutions such as NoMachine, Anydesk etc
Remaining Testing
Whilst it's easy to claim success and sit back, I need to absolutely triple check that disabling
glamordoes not affect the overall desktop experience. Many of the apps we use do not use that path as they leverage their own 3d viewports that are unaffected. Therefore the remaining question is whether the desktop experience is degraded.Will report back!!
Hope that helps, happy to answer any further questions!