grm blog. Work is copyrighted unless otherwise stated.
2021-05-01 Sat
^

GPU passthrough via libvirt

Notes on gpu passthough to a virtual machine using libvirt.

My host machine runs Arch Linux on an Intel Core i7-8700 @ 3.20GHz with onboard graphics, which is important if you want video from the host machine as well.

I am passing though a Radeon RX 5500 to a windows VM.

This is not a tutorial or a step by step guide and all the knowledge is extracted from this Arch wiki article plus my own tinkering to make this work.

Preparation

Enabling virtualization in the BIOS (for Intel at least) needs two toggles, one for VT-x and another for VT-d. Without the first you can't even run virtualbox and without the second you can't get access to the IOMMUs in order to pass them to a VM. They are not named VT-d and VT-x in the BIOS and use instead more descriptive option names, making them harder to find.

Software changes

As the Arch wiki mentions one can find the IOMMU groups using this script: 

#!/bin/bash
shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

Then just assign every device in the same group with the GPU a vfio driver using kernel parameters. This way if I want the host system to boot with access to the graphics card I just press e on the boot loader screen and delete the last few options. Since I use systemd-boot I added this in /boot/loader/entries/arch.conf

Another small change is needed in the kernel image which is achieved though mkinitcpio. Arch's kernel already loads the necessary vfio modules we just need to make sure they are loaded early. Again the change is described in the Arch wiki and takes place in /etc/mkinitcpio.conf. A mkinitcpio -p linux should be enough to create the new kernel image and be able to boot from it.

The final thing is about sharing the keyboard and mouse on the host and guest OS without having any delay or stuttering. This is achieved though evdev and is of course once again thoroughly described in the Arch wiki. I had to add root the the kvm and input groups for this to work. Note: I have two devices under /dev/input/by-id/ that look like a mouse and both spit out data when I cat them and move the mouse

  • usb-Logitech_USB_Laser_Mouse-mouse and
  • usb-Logitech_USB_Laser_Mouse-event-mouse

The one I need is the one with the event.

Hardware changes

My motherboard has two display outputs, DVI and HDMI, the GPU has two more, DP and HDMI. I use two displays for my linux PC when it loads the Radeon driver, both plugged into the GPU. In order to start a VM with the GPU I need to plug a display to the mobo's outputs (I use DVI -> HDMI) and then reboot the system with the above changes. Thus I unplug the HDMI -> HDMI connection from the GPU to the display and it only has DP -> DP connection to the second screen. This makes sure that the first graphics I see are coming from them CPU's onboard graphics card and then when I start the VM the second screen with the Display Port comes up running windows.

The process is the opposite to revert to using the GPU on the host. I just unplug the DVI -> HDMI cable, replug the HDMI -> HDMI one, restart the computer and make sure to remove the vfio stuff from the kernel parameters.

Other refferences