Host leaking memory when running in a VM with Virtio OpenGL acceleration

[nazar-pc]

Your system information

• Steam Runtime Version: 3.0.20260119.200241
• Distribution (e.g. Ubuntu 18.04): Ubuntu 24.04
• Link to your full system information (Help -> Steam Runtime Diagnostics) in a Gist: https://gist.github.com/nazar-pc/dd88041718abfdeb6be722f6e19a40d9
• Have you checked for system updates?: Yes
• What compatibility tool are you using?: Steam Linux Runtime (various versions), Proton (various versions)

Please describe your issue in as much detail as possible:

I have an Ubuntu 24.04 VM launched with libvirtd using virt-manager that uses Virtio GPU with hardware acceleration. The host is also Ubuntu 24.04 with RX 7600 XT GPU.

VM has the latest HWE kernel 6.17, I tried both Mesa that Ubuntu ships and 25.3.5 from PPA. The host had kernels 6.18/6.19 of various patch versions with Mesa 25.3.5/26.0.x over last few weeks. QEMU on the host was both Ubuntu's stock 8.2.2 and 9.0.2 from their server backports repo.

When playing Ren'Py-based games launched through Steam host starts leaking memory (I have 128G on the host and it is happy to eat all of it), mostly when playing animations, which in most cases (if not all) are just WebM videos. It seems to be leaking from the kernel, not any specific process.

This doesn't happen with QXL GPU without hardware acceleration (though the game usually hangs at some point with QXL). This also doesn't happen when launching games downloaded through Steam directly from the file system without Steam runtime. Only Steam runtime causes it. I tied various versions and even launching through Proton, always the same result.

This happens to all Ren'Py-based games I have tried, unfortunately some of them have a hard requirement on Steam runtime and refuse to launch without it.

Steam was installed through the official Snap on Ubuntu.

I am not sure where exactly the root cause is (seems like the kernel here), but it only happens when launching games through Steam and in no other way. Hopefully someone can look closer into it.

Ubuntu 24.04 doesn't ship newer version of QEMU with Virtio-Venus support, so accelerated OpenGL is the only thing I can use for now.

Steps for reproducing this issue:

1. Install libvirtd and virt-manager on Ubuntu 24.04
2. Launch virt-manager (ideally with vblank_mode=0 environment variable or else the performance will be really bad)
3. Create an Ubuntu 24.04 VM with Virtio GPU and enable hardware acceleration on it and OpenGL Spice display4. Install Steam through App Center (uses Snap)
4. Install any Ren'Py-based game from it
5. Play the game, notice memory is leaking on the host while the game displays animations
6. Shut down the VM, see memory being reclaimed by the host slowly (can be accelerated with echo 3 > /proc/sys/vm/drop_caches

[smcv]

Steam was installed through the official Snap on Ubuntu

Valve doesn't support this package or consider it to be official. Valve's supported way to install Steam on Ubuntu is the steam-launcher .deb package.

For support for Canonical's Snap packages, please use Canonical's support channels.

[Memory] seems to be leaking from the kernel, not any specific process

This sounds like it would have to be a kernel/driver issue on the host: unprivileged user processes shouldn't be able to use up unbounded kernel memory even if they were actively trying to do so, and similarly processes in the VM shouldn't be able to use up unbounded host memory.

This is reminding me of libsdl-org/SDL#14874 which turned out to be a problem with the kernel side of the driver for AMD GPUs.

renpy/renpy#6934 is potentially also related.

Only Steam runtime causes it

What precise thing do you mean by that? What is the minimal difference between steps-to-reproduce that do exhibit this memory leak, and steps that don't?

[nazar-pc]

Linked issues seem somewhat related, thanks.

What precise thing do you mean by that? What is the minimal difference between steps-to-reproduce that do exhibit this memory leak, and steps that don't?

I mean the same game launched through Steam vs directly from the file system, bypassing Steam and its runtime completely.