GPU (AMD IGP) passthrough using QEMU-KVM

Pure qemu-system command line approach

At the end of 2024 I bought a new computer with Ryzen 9 9900X CPU. It has an Integrated Graphics Processor (IGP) that I wanted to pass through to my QEMU-KVM Windows 11 Pro virtual machine from my Gentoo Linux host system.

I got it working, but it was a hassle figuring out the different pieces. Recently I changed my hardware a bit and I now have a Ryzen 9 9950X processor, and my old passthrough setup stopped working, so I did some debugging what's the problem.

I'm writing this article as a memo for myself and others and because I'm so annoyed by the obsolete and wrong information circling around the Internet on the topic, so let's set some things clear.

This concerns specifically passing through an AMD processor's IGP, although passing a discrete GPU should work largely the same way except for a few extra steps required by the IGP.

What is required and what is not

IOMMU

You will need IOMMU. For AMD systems this has basically always been purely a matter of BIOS setup, not kernel options. Discussions like this one are misleading:

"/etc/kernel/cmdline -> add amd_iommu=on iommu=pt to end of options"

First of all, that amd_iommu option has pretty much never been a real option. In this Linux kernel mailing list message dated May 11, 2010, they introduce an option to turn off automatic IOMMU initialization, implying it has been on at least ever since by default. The now legacy Red Hat Virtualization platform documentation unfortunately mentions the same option and is probably one of the reasons people copy paste the same old stuff year after year. Just stop it: you don't need to tell the kernel to specifically turn IOMMU on.

As for Intel platforms, this kernel patch dated August 23, 2011 introduces a renamed config option INTEL_IOMMU_DEFAULT_ON with a default value of on. So again, you probably don't need to specifically turn on IOMMU.

As for the iommu=pt option: unlike some people wrongfully claim on the Internet, this does not turn on IOMMU, either. It sets IOMMU on just for devices that are passed through (hence the "pt"); by default everything would use IOMMU. So this is more like an optimization.

To check IOMMU is working, just check you have IOMMU groups:

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done;

This snippet came from Gentoo documentation, which unfortunately has misleading stuff also, such as this line: dmesg | grep 'IOMMU enabled'. That text is not found in the kernel log, so forget it.

I've never seen a system where the same IOMMU group would have several devices. If this happens, then use the kernel option pcie_acs_override=downstream,multifunction as per Arch Linux docs, but use it only if needed. Some people seem to consider that a requirement by default, which it is not.

In the end, you most probably don't have to add anything to your kernel command line for passthrough to work, but using the iommu=pt option might give you a small performance benefit.

Prevent AMDGPU from claiming the passthrough device

I have both an AMD discrete GPU, and the IGP of the AMD CPU. If I have no configuration at all, the amdgpu driver is automatically loaded for both devices. I then would need to unload the driver for the IGP before passthrough. This, however, kills my X - yes, I'm still using X, not Wayland.

Even if automatically adding devices is on by default, I noticed if in X config I only specify the discrete GPU, the IGP driver is not loaded. Hence I have a file /etc/X11/xorg.conf.d/99-amd.conf with the following contents:

# Use OutputClass so that options also apply to an IGP's amdgpu driver possibly loaded later on.
Section "OutputClass"
    Identifier "AMDGPU settings"
    MatchDriver "amdgpu"
    Option "TearFree" "true"
    Option "VariableRefresh" "true"
    Option "DRI" "3"
EndSection

# Defining just the discrete graphics adapter here skips automatically loading the amdgpu driver for an IGP in case a passthrough for virtual machine use is wanted.
Section "Device"
    Identifier "AMDGPU"
    Driver "amdgpu"
    BusID "PCI:3:0:0"
EndSection

Section "Screen"
    Identifier "Primary"
    Device "AMDGPU"
    DefaultDepth 24
    SubSection "Display"
        Viewport 0 0
        Depth 24
        Modes "3840x2160" "1920x1080" "1024x768" "800x600" "640x480"
    EndSubSection
EndSection

Section "DRI"
    Mode 0666
EndSection

Section "Extensions"
    Option  "Composite" "Enable"
EndSection

As for how Wayland configures devices and drivers I can't say. Perhaps with Wayland it might be possible to unload the driver for the IGP device without the GUI crashing.

A better, more universal way to handle this would be to load VFIO driver for the IGP before amdgpu even tries to claim it. See below.

Kernel modules and their options

You do need VFIO and its related modules. Most likely those are enabled by default on your kernel and you don't have to do anything. If not, see the Gentoo docs about it.

It is advisable to create a file, say, /etc/modprobe.d/vfio.conf and in there put the following:

# This is required in some cases, otherwise KVM fails with an error to set IOMMU for container.
options vfio_iommu_type1 allow_unsafe_interrupts=1

You may try without it first if it works. As mentioned in the previous section, a mechanism to prevent amdgpu driver from claiming the IGP device would be nice. In theory, putting in the vfio.conf file the following should help:

# Make kernel load vfio-pci before amdgpu; load also for the audio device
options vfio-pci ids=1002:13c0,1002:1640
softdep amdgpu pre: vfio-pci

In the module options line, id 1002:13c0 is the IGP's VGA device, and 1002:1640 is its audio device.

However, I was not able to get the module load order working, even with a direct kernel command line option: amdgpu was loaded instead of vfio-pci for the IGP, always. I did double check my dracut-created initial ramdisk had all the necessary modules, but no luck. Alas, I resorted to the X configuration. If you have a non-AMD discrete GPU for your host system, you could just blacklist the amdgpu module and be done with the whole issue.

VBIOS and other ROM files

At least the IGP device requires some VBIOS firmware to be loaded for the device to work on a virtualized Windows client. Otherwise, you will see the AMD IGP in Windows' Device Manager, but it will show an error code 43 with an explanation: "Windows has stopped this device because it has reported problems. (Code 43)". This is the "infamous error 43" you might bump into on the Internet when researching the subject. If you do get that far, however, it's a case of either missing/wrong ROM files, or the also infamous device reset problem, at least if you're trying to pass through and AMD IGP device. See a later section in this article for more info on the device reset problem.

If you're using non-UEFI BIOS such as SeaBIOS, you apparently only need a VBIOS ROM file. However, if you're using UEFI BIOS such as OVMF, you need both a VBIOS ROM file and an AMD GOP driver.

Getting the ROM files is where things get a bit tricky. On my first computer setup, I had the AsRock Phantom Gaming B850M Riptide WiFi motherboard with the Ryzen 9 9900X processor. As the source of the ROM files, I the BIOS file I had downloaded from AsRock's website and flashed to the motherboard.

I used a tool called UEFI Bios Updater which goes around as UBU.bat. Using it to extract everything, I got a bunch of .dat files for the VBIOS, and a few GOP driver files. Running EfiRom.exe -f 0x1002 -i 0xffff -e AMDGopDriver.efi gave me a ROM file where the vendor ID (1002) and product ID (ffff, which apparently can be anything) were set. The tool EfiRom.exe is part of UEFI Bios Updater.

Extracting the ROM files yourself makes logical sense: your computer works, so naturally whatever you extract from the motherboard's BIOS should also work for device passthrough, right? With the AsRock and 9900X it worked just fine. Now I have an Asus TUF Gaming B850M-PLUS II motherboard and Ryzen 9 9950X processor. I did the same extraction procedure to my Asus's BIOS file and eventually had binary-identical VBIOS and GOP ROM files. However, using those resulted in error 43 for the graphics device in the virtual Windows client.

I found a dedicated repository of ROM files specifically for Ryzen IGP passthrough here. Ironically, in the instructions there they again wrongly claim the iommu=pt kernel option would enable IOMMU, which we already concluded is not the case. However, we only need to download the ROM files from the repo. For my 9950X, I tried the 9950X VBIOS along with 9950X3D GOP driver. I have no idea what is the source of these files, but they work for me.

Therefore as for the ROM files I can only say it seems to be a bit random what works and what doesn't when passing through. And, if you use the files provided in the GitHub repo, you don't need a perfect match for your processor - e.g. I used the 9950X3D GOP driver, even if I have a 9950X.

When the passthrough works, the OVMF BIOS graphics are visible on boot.

Putting it all together

Now it's just a matter of putting it all together and forming a functioning qemu-system-x86_64 command line. Most people use libvirt or virsh or whatever, but I'm an old school stubborn purist and I just use a shell script to build the command.

The AMD IGP requires passing through both the VGA device and the audio device. The important prerequisite (handled and checked by my script) is that both devices use the vfio-pci driver. The devices must then be passed through together. The way to do it is to create a PCIe root port:

-device pcie-root-port,id=pcie0,slot=0

I'm calling it pcie0. Apparently under the hood there is already a device called pcie.0, but it doesn't matter. What matters is this new root port device is a different device, and is put on the bus of the existing device. I omit defining the bus and let it be handled automatically. Then, for the actual passthrough devices we define:

-device vfio-pci,host=0000:0f:00.0,bus=pcie0,addr=00.0,x-vga=on,multifunction=on,romfile=vbios.rom
-device vfio-pci,host=0000:0f:00.1,bus=pcie0,addr=00.1,romfile=gopdriver.rom

The 0000:0f:00.0 and 0000:0f:00.1 are the host's IGP VGA and audio devices respectively. Note the bus for the devices: we are using the root port we just defined above. The VGA device is defined as a multifunction device, which essentially makes QEMU-KVM to handle them like one. That's why giving the GOP driver ROM to the audio device satisfies the VGA device's need for it.

When using passthrough, you will also probably want to have no virtualized video device: -vga none. This is because the device will be most likely removed in any case - see the next section for more info. A VNC display might be useful to create to be able to reboot or shutdown the machine. As long as passthrough is not working, you can use RDP to get to the computer to see e.g. if you see the infamous error 43 in device manager. Once passthrough works, I recommend installing a KVM software such as Input Leap (now seems to be recently archived and unmaintained) or Deskflow. With it running as a service you can type in your password in the Windows login screen.

My final command line for starting up the virtual machine looks like this:

qemu-system-x86_64 -daemonize -enable-kvm -name "Windows 11 Pro" -boot order=c -machine q35,accel=kvm -cpu host,kvm=off,+topoext -smp cores=8,threads=1,sockets=1 -m 16384 -k fi -display vnc=localhost:10 -device virtio-net,netdev=netdev10,mac=00:00:00:00:00:10 -netdev tap,id=netdev10,br=vmbridge,script=kvm_tap_vmbridge.sh -drive file="win11pro.img",if=virtio,format=raw -device intel-hda -device hda-duplex,audiodev=snd10 -audiodev pa,id=snd10,server=unix:/run/user/1000/pulse/native -vga none -smbios type=0,uefi=on -bios OVMF/OVMF_CODE.secboot.fd -chardev socket,id=chrtpm,path=/tmp/emulated_tpm/swtpm-sock -tpmdev emulator,id=tpm0,chardev=chrtpm -device tpm-tis,tpmdev=tpm0 -global driver=cfi.pflash01,property=secure,value=on -drive if=pflash,format=raw,unit=0,file=OVMF/OVMF_CODE.secboot.fd,readonly=on -drive if=pflash,format=raw,unit=1,file=win11pro_OVMF_VARS.secboot.fd -global ICH9-LPC.disable_s3=1 -device pcie-root-port,id=pcie0,slot=0 -device vfio-pci,host=0000:0f:00.0,bus=pcie0,addr=00.0,x-vga=on,multifunction=on,romfile=./ROM/proxmox/vbios_9950x.rom -device vfio-pci,host=0000:0f:00.1,bus=pcie0,addr=00.1,romfile=./ROM/proxmox/AMDGopDriver_9950x3d.rom

The script that created the command line handles creating the TPM device automatically, and sets up network bridges and everything. You can find it in my dotfiles GitHub repository under the KVM scripts. It's called startvm. I will also link the file here as it currently stands while I'm writing this article.

AMD device reset problem and fix

There's still one more thing to handle. When passing through an IGP device (or maybe any kind of device I guess), Windows doesn't seem to reset the device correctly on shutdown. I'm not sure if this concerns only AMD GPUs or if also some other devices have similar problems. In any case, what's happening is that the software (Windows, AMD drivers, whoever should be responsible) expects a hardware power cycle when shutting down or resetting the virtual machine. It would reset the graphics device on hardware level. Since that never comes (unless you reboot your host system obviously), it might be that the passthrough device works exactly once. On a second try, the device again might report the infamous error 43 in the Windows client.

Historically, there appears to have been a lot of discussion about the problem on the Internet. Some ask if a PCIe device in general could be reset cycled for power. I remember trying to fiddle around the problem myself. The idea was to use the PCIe device power cycling feature, and then rescan the PCIe devices. However, I did not get it to work. On GitHub, there's also an old project about a kernel module to reset the device. Alas, the list of supported devices has mostly very old hardware.

Even though I've lately seen (on kernel 7.x) some updates related to PCIe power cycling, solving the problem from the virtual machine host's side looks to be practically unsolvable at the moment. However, there's a project called RadeonResetBugFix. It installs a Windows service. What it does is on shutdown it removes all VGA devices, and on startup it adds the AMD graphics device anew. This seems to result in the device being reset correctly, and the virtual machine can be restarted without a hitch. It does slow down the boot process somewhat as it is adding or removing devices, but not by much.

Conclusion

Years go by, but false information keeps on circling the Internet when the subject is such an esoteric one such as passing through an IGP device using QEMU-KVM. It doesn't help that some of the related technical stuff feels illogical, even random. Hopefully this article clears up some things for people who might stumble upon it.