Ingo: create page

2017-09-20T10:46:45Z

create page

Neue Seite

== HowTo Setup VGAPassthrough from scratch on Debian 9 (stretch) ==
Author: Ingo Höft, 2017-08-10

=== Preparation ===
You need a CPU with virtualization and IOMMU, two different graphic
cards, two keyboards and mice, one for the host, one for the guest. I
use one keyboard/mouse with an usb switch.

Install a basic system for testing as described in
[[Setup_KVM_with_console]] but without Guest Setup.

I have a simple older graphics card for the text console and NVIDIA for
the guest.
~$ lspci | grep VGA
0f:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
37:04.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA 1064SG [Mystique] (rev 03)
~$
The older NVIDIA Quadro 2000 does not support UEFI bios so we will
workaround using of debian package ovmf (UEFI firmware for virtual
machines). virt-install uses seabios by default. Next time I will look
for a newer graphics card supporting UEFI. If your graphics card support
UEFI you can look at [https://www.techpowerup.com/vgabios/ Video BIOS Collection]
or check with this [https://vfio.blogspot.de/2014/08/does-my-graphics-card-rom-support-efi.html rom-parser].

=== Setup VGAPassthrough on host ===
It is important to blacklist the graphics card for the guest,
here NVIDIA, so the host will never see it. This should be done
first to avoid confusion. I had trouble. My linux box does not use
the right graphics card on boot and I could not see anything. The
nouveau driver is default for NVIDIA. For blacklisting I use kernel
parameter. On grub boot menu hit 'e' and append kernel parameter
'modprobe.blacklist=nouveau' (or what your driver is) and boot with F10.

You should know what driver to blacklist before installing the second
graphics card. Maybe you can't examine it afterwards because of a black
screen. Use 'sudo lspci -v' to show what 'Kernel driver in use:' for the
video card to blacklist.

After booting I set in /etc/default/grub for persistence
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on modprobe.blacklist=nouveau"
Now I boot with the right graphics card for the console of the host.

The intel_iommu=on is needed for Intel CPUs, amd_iommu=on for AMD CPUs.
Don't forget to run
~$ update-grub
Now we have to load the vfio-pci driver modules for VGAPassthrough.
They MUST load before any other video driver so the graphics card can't
be occupied bye other drivers. We do it very early in initram, so edit
/etc/initramfs-tools/modules like this:
~$ cat /etc/initramfs-tools/modules
# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax: module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# the drivers for vfio must load early in THIS order!
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
~$

Now we have to append the physical graphics devices to the vfio-pci
driver. This is somewhat tricky. You cannot passthrough simple devices
but only complete IOMMU groups. For further information and finding the
IOMMU group look
[https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Setting_up_IOMMU here]:
"An IOMMU group is the smallest set of physical devices that can be
passed to a virtual machine".

Using the small bash script from there for my NVIDA card I found:
IOMMU Group 14 0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF106GL [Quadro 2000] [10de:0dd8] (rev a1)
IOMMU Group 14 0f:00.1 Audio device [0403]: NVIDIA Corporation GF106 High Definition Audio Controller [10de:0be9] (rev a1)
There are two devices in Group 14: the VGA controller and its build in
audio device. That's evident.

Now we can add the devices in the group to vfio-pci in
/etc/modprobe.d/vfio.conf like this:
~$ cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:0dd8,10de:0be9
options vfio-pci disable_vga=1
~$ sudo update-initramfs -u
You will find the device numbers from my example and I will not use VGA.
To get module parameters I use 'sudo modinfo vfio-pci'.

Reboot.

Now you should find the well set up vfio-pci driver like this:
~$ lspci -nnk -d 10de:0dd8
0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF106GL [Quadro 2000] [10de:0dd8] (rev a1)
Subsystem: Hewlett-Packard Company GF106GL [Quadro 2000] [103c:084a]
Kernel driver in use: vfio-pci
Kernel modules: nouveau
~$ lspci -nnk -d 10de:0be9
0f:00.1 Audio device [0403]: NVIDIA Corporation GF106 High Definition Audio Controller [10de:0be9] (rev a1)
Subsystem: Hewlett-Packard Company GF106 High Definition Audio Controller [103c:084a]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
~$
Inspecting 'sudo journalctl -k' is also a good thing.

=== Install virtual machine ===
Now it's up to install a guest with all this:
~$ virsh net-start default
~$ virt-install --virt-type kvm \
--cpu host \
--features kvm_hidden=on \
--name harley \
--location https://ftp.de.debian.org/debian/dists/stretch/main/installer-amd64/ \
--os-variant debian9 \
--extra-args "console=ttyS0,115200" \
--noautoconsole \
--disk size=10 \
--memory 2048 \
--host-device 0f:00.0 \
--host-device 0f:00.1 \
;
{|cellspacing="10"
| --cpu host
|Yes of course, we want hardware accelleration and this guest with its physical device binding runs only on this host.
|-
| --features kvm_hidden=on
|It was told, that Nvidia does not support running consumer-grade cards in VMs.
|-
| --noautoconsole
|virt-install returns from running guest installation, so you can easy connect with virsh. Autoconsole will not give you the guest installation screen. '--graphics none' does not work in any way although we only use text mode in guest installation. I don't know why. Maybe it has something to do with the bound graphics devices.
|-
| --host-device
|Two times - this must be the IOMMU group with its two physical devices as described above.
|}
Connect to guest installation and install as usual
~$ virsh console harley

=== Troubleshooting ===
If you get many error messages from virt-install, try to use
temporary --cpu kvm64. This is more robust and may give less and more
understandable messages.

On my machine virt-install canceled the installation with error
messages [1]. The reason was that the pc has a buggy bios and the kernel
disabled interrupt remapping (found with 'sudo journalctl -k') [2].
But IOMMU needs this and there is a way to force this feature with
iommu_unsave_interrupts like this:
~$ cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1
~$ sudo update-initramfs -u
Reboot.

Of course this is also needed for machines does not support interrupt
remapping at all.

=== Conclusion ===
I wanted to setup VGAPassthrough to do my work in an efficient way. I
saw it wasn't as simple as I thought so I decided to do it from scratch
step by step to learn how it works. After spending about a week I come
to this result: I'm able to start the vm and get a login prompt on the
screen attached to the passed through graphics card. It works. But when
I restart the vm I have no output on the screen. The guests graphics
card wasn't reset properly so I have to restart the host to get it for
one time again. For me that is a showstopper but it's a known problem
and can be fixed by using UEFI. My somewhat older NVIDIA Quadro 2000
doesn't support it. I have looked for nvidia-smi but test it not
yet. I should buy a newer graphics card with UEFI support.

Next I wanted to login to the guest on it's login prompt but there is
no keybord and mouse support there. Seems that it should also passed
through but could only find generic hints in the many HowTos and manuals
like: "passthrough your keyboard and mouse to the vm, no problem". I did
it and yes, it works. But I have an USB-switch for my keyboard/mouse to
switch between host and guest. One switch-port attached to an usb-port
for the host, one switch-port attached to an usb-port for the guest.
Booting the host with nothing connected to the guests usb-port and
starting the guest with keyboard/mouse connected to guests usb-port then
I get it to the login prompt. Seems to work but when I switch to the
screen of the host without switching the usb-switch back the host will
occupy the usb-port for the guest and I loose it for the vm. Seems that
I have to hide this port on the host. When looking around for a solution
I found this [5].

That was the point I gave up. What will come up with sound? Do I
have passthrough everything? It seems there is no generic setup for
VGAPassthrough. I found many HowTows and Manuals but each only for a
special hardware. I'm not willing to investigate in development to this
and just make another HowTo for my machine. I have other projects.
Looking in a year what's going on.

=== References ===
* https://wiki.debian.org/VGAPassthrough
* https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF
* https://davidyat.es/2016/09/08/gpu-passthrough/
* https://forums.linuxmint.com/viewtopic.php?f=231&t=212692
* https://vfio.blogspot.de/
* https://www.redhat.com/archives/vfio-users/
* http://www.laketide.com/setting-up-gpu-passthrough-with-kvm-on-fedora/
* https://pve.proxmox.com/wiki/Pci_passthrough#IOMMU_interrupt_remapping

=== Footnotes ===
[1] ERROR internal error: qemu unexpectedly closed the monitor:
warning: host doesn't support requested feature: CPUID.01H:EDX.ds [bit 21]
warning: host doesn't support ...

[2] Kernel message:
"DMAR-IR: This system BIOS has enabled interrupt remapping on a
chipset that contains an erratum making that feature
unstable. To maintain system stability interrupt remapping
is being disabled. Please contact your BIOS vendor for an
update"

[[Category:Virtualization]]

Setup VGAPassthrough - Versionsgeschichte

Ingo: create page