Setup VGAPassthrough

Aus Ingos Wiki
Wechseln zu: Navigation, Suche

HowTo Setup VGAPassthrough from scratch on Debian 9 (stretch)

Author: Ingo Höft, 2017-08-10

Preparation

You need a CPU with virtualization and IOMMU, two different graphic cards, two keyboards and mice, one for the host, one for the guest. I use one keyboard/mouse with an usb switch.

Install a basic system for testing as described in Setup_KVM_with_console but without Guest Setup.

I have a simple older graphics card for the text console and NVIDIA for the guest. 

~$ lspci | grep VGA
0f:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
37:04.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA 1064SG [Mystique] (rev 03)
~$

The older NVIDIA Quadro 2000 does not support UEFI bios so we will workaround using of debian package ovmf (UEFI firmware for virtual machines). virt-install uses seabios by default. Next time I will look for a newer graphics card supporting UEFI. If your graphics card support UEFI you can look at Video BIOS Collection or check with this rom-parser.

Setup VGAPassthrough on host

It is important to blacklist the graphics card for the guest, here NVIDIA, so the host will never see it. This should be done first to avoid confusion. I had trouble. My linux box does not use the right graphics card on boot and I could not see anything. The nouveau driver is default for NVIDIA. For blacklisting I use kernel parameter. On grub boot menu hit 'e' and append kernel parameter 'modprobe.blacklist=nouveau' (or what your driver is) and boot with F10.

You should know what driver to blacklist before installing the second graphics card. Maybe you can't examine it afterwards because of a black screen. Use 'sudo lspci -v' to show what 'Kernel driver in use:' for the video card to blacklist.

After booting I set in /etc/default/grub for persistence GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on modprobe.blacklist=nouveau" Now I boot with the right graphics card for the console of the host.

The intel_iommu=on is needed for Intel CPUs, amd_iommu=on for AMD CPUs. Don't forget to run 

~$ update-grub

Now we have to load the vfio-pci driver modules for VGAPassthrough. They MUST load before any other video driver so the graphics card can't be occupied bye other drivers. We do it very early in initram, so edit /etc/initramfs-tools/modules like this: 

~$ cat /etc/initramfs-tools/modules
# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax:  module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# the drivers for vfio must load early in THIS order!
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
~$

Now we have to append the physical graphics devices to the vfio-pci driver. This is somewhat tricky. You cannot passthrough simple devices but only complete IOMMU groups. For further information and finding the IOMMU group look here: "An IOMMU group is the smallest set of physical devices that can be passed to a virtual machine".

Using the small bash script from there for my NVIDA card I found: 

IOMMU Group 14 0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF106GL [Quadro 2000] [10de:0dd8] (rev a1)
IOMMU Group 14 0f:00.1 Audio device [0403]: NVIDIA Corporation GF106 High Definition Audio Controller [10de:0be9] (rev a1)

There are two devices in Group 14: the VGA controller and its build in audio device. That's evident.

Now we can add the devices in the group to vfio-pci in /etc/modprobe.d/vfio.conf like this: 

~$ cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:0dd8,10de:0be9
options vfio-pci disable_vga=1
~$ sudo update-initramfs -u

You will find the device numbers from my example and I will not use VGA. To get module parameters I use 'sudo modinfo vfio-pci'.

Reboot.

Now you should find the well set up vfio-pci driver like this: 

~$ lspci -nnk -d 10de:0dd8
0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF106GL [Quadro 2000] [10de:0dd8] (rev a1)
        Subsystem: Hewlett-Packard Company GF106GL [Quadro 2000] [103c:084a]
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau
~$ lspci -nnk -d 10de:0be9
0f:00.1 Audio device [0403]: NVIDIA Corporation GF106 High Definition Audio Controller [10de:0be9] (rev a1)
        Subsystem: Hewlett-Packard Company GF106 High Definition Audio Controller [103c:084a]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
~$

Inspecting 'sudo journalctl -k' is also a good thing.

Install virtual machine

Now it's up to install a guest with all this: 

~$ virsh net-start default
~$ virt-install --virt-type kvm \
                --cpu host \
                --features kvm_hidden=on \
                --name harley \
                --location https://ftp.de.debian.org/debian/dists/stretch/main/installer-amd64/ \
                --os-variant debian9 \
                --extra-args "console=ttyS0,115200" \
                --noautoconsole \
                --disk size=10 \
                --memory 2048 \
                --host-device 0f:00.0 \
                --host-device 0f:00.1 \
;
--cpu host Yes of course, we want hardware accelleration and this guest with its physical device binding runs only on this host.
--features kvm_hidden=on It was told, that Nvidia does not support running consumer-grade cards in VMs.
--noautoconsole virt-install returns from running guest installation, so you can easy connect with virsh. Autoconsole will not give you the guest installation screen. '--graphics none' does not work in any way although we only use text mode in guest installation. I don't know why. Maybe it has something to do with the bound graphics devices.
--host-device Two times - this must be the IOMMU group with its two physical devices as described above.

Connect to guest installation and install as usual 

~$ virsh console harley

Troubleshooting

If you get many error messages from virt-install, try to use temporary --cpu kvm64. This is more robust and may give less and more understandable messages.

On my machine virt-install canceled the installation with error messages [1]. The reason was that the pc has a buggy bios and the kernel disabled interrupt remapping (found with 'sudo journalctl -k') [2]. But IOMMU needs this and there is a way to force this feature with iommu_unsave_interrupts like this: 

~$ cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
~$ sudo update-initramfs -u

Reboot.

Of course this is also needed for machines does not support interrupt remapping at all.

Conclusion

I wanted to setup VGAPassthrough to do my work in an efficient way. I saw it wasn't as simple as I thought so I decided to do it from scratch step by step to learn how it works. After spending about a week I come to this result: I'm able to start the vm and get a login prompt on the screen attached to the passed through graphics card. It works. But when I restart the vm I have no output on the screen. The guests graphics card wasn't reset properly so I have to restart the host to get it for one time again. For me that is a showstopper but it's a known problem and can be fixed by using UEFI. My somewhat older NVIDIA Quadro 2000 doesn't support it. I have looked for nvidia-smi but test it not yet. I should buy a newer graphics card with UEFI support.

Next I wanted to login to the guest on it's login prompt but there is no keybord and mouse support there. Seems that it should also passed through but could only find generic hints in the many HowTos and manuals like: "passthrough your keyboard and mouse to the vm, no problem". I did it and yes, it works. But I have an USB-switch for my keyboard/mouse to switch between host and guest. One switch-port attached to an usb-port for the host, one switch-port attached to an usb-port for the guest. Booting the host with nothing connected to the guests usb-port and starting the guest with keyboard/mouse connected to guests usb-port then I get it to the login prompt. Seems to work but when I switch to the screen of the host without switching the usb-switch back the host will occupy the usb-port for the guest and I loose it for the vm. Seems that I have to hide this port on the host. When looking around for a solution I found this [5].

That was the point I gave up. What will come up with sound? Do I have passthrough everything? It seems there is no generic setup for VGAPassthrough. I found many HowTows and Manuals but each only for a special hardware. I'm not willing to investigate in development to this and just make another HowTo for my machine. I have other projects. Looking in a year what's going on.

References

Footnotes

[1] ERROR internal error: qemu unexpectedly closed the monitor:
          warning: host doesn't support requested feature: CPUID.01H:EDX.ds [bit 21]
          warning: host doesn't support ...

[2] Kernel message:
    "DMAR-IR: This system BIOS has enabled interrupt remapping on a
              chipset that contains an erratum making that feature
              unstable. To maintain system stability interrupt remapping
              is being disabled. Please contact your BIOS vendor for an
              update"
Abgerufen von „http://wiki.hoeft-online.de/index.php?title=Setup_VGAPassthrough&oldid=7“