virtio/gpu: host-visible blob mappings fail unless the size is a multiple of 16384 on macOS, breaking Mesa's Venus driver in guests

[stewartjeacocke]

Thank you for your work on this project. This is a deeper into debugging that I usually go, so I've relied heavily on Claude to help out. Forgive me if this is not the exact place to open up this ticket, it seemed the most relevant.

Summary

On macOS/Apple Silicon, resource_map_blob fails for any host-visible virtio-gpu blob whose size is not a multiple of 16384, because the size is passed unrounded to hv_vm_map. The guest sees mmap return EINVAL.

This makes Mesa's Venus driver unusable in guests with 4 KiB pages: its ring shmem is 131268 bytes, the guest kernel rounds that to 135168 (33 × 4 KiB), and 144 KiB would be needed for the host mapping to succeed. Venus reports the failure as VK_ERROR_OUT_OF_HOST_MEMORY causing the Vulkan loader to abort vkCreateInstance.

$ vulkaninfo
ERROR at ./vulkaninfo/./vulkaninfo.h:458:vkCreateInstance failed with ERROR_OUT_OF_HOST_MEMORY

I believe this is the unexplained root cause of #102 in krunkit, which was closed by switching to a container image whose Mesa carries a downstream patch.

Environment

• Host: Apple M4 Pro, macOS, libkrun via krunkit (/sys/class/dmi/id/sys_vendor = Libkrun)

• krunkit: 1.1.1

• libkrun: @rpath/libkrun-efi.dylib (compatibility version 1.0.0, current version 1.16.0)

• Guest kernel: 6.19.7-200.fc43.aarch64, page size 4096

• Guest userspace: Debian 13 (trixie), mesa-vulkan-drivers 25.0.7-2+deb13u1, Vulkan loader 1.4.309

• Venus capset negotiates fine: supports_blob_id_0 = 1, VIRTGPU_PARAM_HOST_VISIBLE = 1, CONTEXT_INIT with capset 4 succeeds

Code path

src/devices/src/virtio/gpu/virtio_gpu.rs, the #[cfg(target_os = "macos")]
resource_map_blob, forwards the guest-requested blob size unchanged:

self.map_sender
    .send(WorkerMessage::GpuAddMapping(
        reply_sender,
        map_ptr,
        guest_addr,
        resource.size,      // 135168 for Venus's ring — not 16 KiB aligned
    ))

which reaches HvfVm::map_memory in src/vmm/src/macos/vstate.rs:

hv_vm_map(host_start_addr, guest_start_addr, size, READ | WRITE | EXEC)

hv_vm_map requires page-aligned arguments, and the VM page size on Apple
Silicon is 16384, so the call is rejected. That propagates as:

map_memory Err → GpuAddMapping replies false → Err(ErrUnspec) → guest
virtio_gpu leaves vram->map_state != STATE_OK → guest mmap returns EINVAL
→ Mesa logs failed to allocate/map ring shmem → VK_ERROR_OUT_OF_HOST_MEMORY.

Guest-side confirmation, via an LD_PRELOAD shim over ioctl/mmap64 while
running vulkaninfo with the Venus ICD. Blob creation and VIRTGPU_MAP both
succeed; only the final mmap fails:

[trace] VIRTGPU_RESOURCE_CREATE_BLOB size=135168 blob_mem=2 flags=0x1 blob_id=0 -> 0
[trace] VIRTGPU_MAP                  handle=1 -> 0 (offset=0x100000000)
[trace] mmap64 len=131268 prot=0x3 flags=0x1 fd=3 off=0x100000000 -> FAILED errno=Invalid argument
MESA-VIRTIO: debug: failed to allocate/map ring shmem
MESA-VIRTIO: debug: vn_CreateInstance: VK_ERROR_OUT_OF_HOST_MEMORY

(size reads 135168 because the guest kernel writes back its PAGE_ALIGNed
value; Mesa requested 131268.)

Reproducer

No Vulkan, Mesa, or GPU library needed — just the virtgpu ioctls, on any guest
with a render node. Each size is tested in a forked child, because once a
mapping fails the render context refuses subsequent mappings, which masks later
results.

/*
 * cc -O0 -o virtgpu-blob-map-align virtgpu-blob-map-align.c && ./virtgpu-blob-map-align
 */
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <sys/wait.h>
#include <drm/drm.h>
#include <drm/virtgpu_drm.h>

#define VENUS_CAPSET 4

/* Returns 0 on success, else the failing errno. */
static int try_size(size_t size)
{
   int fd = open("/dev/dri/renderD128", O_RDWR | O_CLOEXEC);
   if (fd < 0)
      return errno;

   uint64_t params[] = { VIRTGPU_CONTEXT_PARAM_CAPSET_ID, VENUS_CAPSET,
                         VIRTGPU_CONTEXT_PARAM_NUM_RINGS, 64 };
   struct drm_virtgpu_context_init ci = {
      .num_params = 2,
      .ctx_set_params = (uint64_t)(uintptr_t)params,
   };
   if (ioctl(fd, DRM_IOCTL_VIRTGPU_CONTEXT_INIT, &ci))
      return errno;

   struct drm_virtgpu_resource_create_blob cb = {
      .blob_mem = VIRTGPU_BLOB_MEM_HOST3D,
      .blob_flags = VIRTGPU_BLOB_FLAG_USE_MAPPABLE,
      .size = size,
      .blob_id = 0,
   };
   if (ioctl(fd, DRM_IOCTL_VIRTGPU_RESOURCE_CREATE_BLOB, &cb))
      return errno;

   struct drm_virtgpu_map m = { .handle = cb.bo_handle };
   if (ioctl(fd, DRM_IOCTL_VIRTGPU_MAP, &m))
      return errno;

   void *p = mmap(NULL, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, m.offset);
   if (p == MAP_FAILED)
      return errno;

   memset(p, 0xab, size);
   munmap(p, size);
   close(fd);
   return 0;
}

int main(void)
{
   static const size_t sizes[] = {
      4096, 8192, 12288, 16384, 32768, 65536, 69632,
      131072, 135168, 139264, 147456, 262144, 266240,
      1048576, 1052672,
   };

   printf("guest page size: %ld\n\n", sysconf(_SC_PAGESIZE));
   printf("%9s  %8s  %8s  %s\n", "size", "n*4KiB", "n*16KiB", "result");

   for (unsigned i = 0; i < sizeof(sizes) / sizeof(*sizes); i++) {
      size_t sz = sizes[i];
      pid_t pid = fork();
      if (pid == 0)
         _exit(try_size(sz));

      int status = 0;
      waitpid(pid, &status, 0);
      printf("%9zu  %8.2f  %8.2f  %s\n", sz, sz / 4096.0, sz / 16384.0,
             WEXITSTATUS(status) ? strerror(WEXITSTATUS(status)) : "ok");
   }
   return 0;
}

Output on this host. Every size that is not a multiple of 16384 fails, and it
tracks 16 KiB rather than the guest's 4 KiB page size:

guest page size: 4096

     size    n*4KiB   n*16KiB  result
     4096      1.00      0.25  Invalid argument
     8192      2.00      0.50  Invalid argument
    12288      3.00      0.75  Invalid argument
    16384      4.00      1.00  ok
    32768      8.00      2.00  ok
    65536     16.00      4.00  ok
    69632     17.00      4.25  Invalid argument
   131072     32.00      8.00  ok
   135168     33.00      8.25  Invalid argument     <- Venus ring shmem
   139264     34.00      8.50  Invalid argument
   147456     36.00      9.00  ok
   262144     64.00     16.00  ok
   266240     65.00     16.25  Invalid argument
  1048576    256.00     64.00  ok
  1052672    257.00     64.25  Invalid argument

Why I don't think the guest is at fault

Nothing in the protocol advertises this constraint. VIRTGPU_PARAM_HOST_VISIBLE
reports that host-visible mapping is available but says nothing about
granularity, so a guest that aligns to its own page size is behaving correctly.
The host is imposing an undocumented requirement and reporting it as a generic
ErrUnspec, which surfaces to userspace as a bare EINVAL with no indication of
the cause.

Fixing it host-side also fixes every existing guest image. The alternative —
every guest shipping a patched Mesa — is what the situation effectively requires
today, and it is why stock Debian/Ubuntu images fail while Fedora images built
against a patched Mesa work.

Current workaround in Mesa for Venus

For anyone who finds this: rebuild guest Mesa with slp's downstream patch, which
forces 16 KiB alignment on Venus allocations —
https://gitlab.freedesktop.org/slp/mesa/-/commit/761ef1ec5ff2aae1cc3dc8bbc22b3d06ef04b549
All four hunks are needed — the vn_ring.c one alone gets past
vkCreateInstance, then device memory allocation fails instead.

With it applied on this host, Venus works.