Alternating a dedicated GPU between DRM/offloading and passthrough (IOMMU)

PCI-E passthrough¹⁾. But for it to be possible, you need to get your 3D stuff to render at your expensive card and to draw at your integrated GPU. That's whats called “GPU offloading”. And to be able to pull it off, the key software is Bumblebee²⁾, it is, currently, the only one which works for us.

It's quite trivial to install (Debian):

apt install bumblebee-nvidia

You can also install primus-vk-nvidia, which provides primusrun command. A slightly different animal tan optirun, but, in fact, doing pretty much the same stuff.

Then, you just run your 3D application like this:

$ optirun glxgears

Before trying to pass the GPU to the virtual machine, the kernel driver needs to be unloaded or the card needs to be “unbinded”. I've put together a crude script which does more or less that. There it is (stop-nv):

#!/bin/bash

/etc/init.d/bumblebeed stop
/etc/init.d/nvidia-persistenced stop

echo Taking five... 
sleep 5

rmmod nvidia_drm; sleep 1
if [ $? != 0 ]; then echo "DRM's busy. Check it!"; exit 1; fi
rmmod nvidia_modeset; sleep 1
if [ $? != 0 ]; then echo "Modeset's busy. Check it!"; exit 1; fi
rmmod nvidia; sleep 1
if [ $? != 0 ]; then echo "Nvidia driver's busy. Check it!"; exit 1; fi 

# CHANGE THOSE VALUES ACCORDINGLY
# ID: the two groups four hex. digits identifying the particular model, i.e. 10de 0aff
# BDF: the bus in which it sits
# To find those guys, just run “lspci -nn”
ID="IDID IDID"
BDF="0000:0x:00.0"
# Unbinds it from the current driver
[ ! -e /sys/bus/pci/devices/$BDF/driver/unbind ] || \
        echo -n $BDF > /sys/bus/pci/devices/$BDF/driver/
# Binds it to the VFIO driver
echo -n $ID > /sys/bus/pci/drivers/vfio-pci/new_id
echo -n $BDF > /sys/bus/pci/drivers/vfio-pci/bind
# Audio integrado
ID="IDID IDID"
BDF="0000:0x:00.1"
[ ! -e /sys/bus/pci/devices/$BDF/driver/unbind ] || \
        echo -n $BDF > /sys/bus/pci/devices/$BDF/driver/unbind
echo -n $ID > /sys/bus/pci/drivers/vfio-pci/new_id
echo -n $BDF > /sys/bus/pci/drivers/vfio-pci/bind

It is not perfect, and, more often than not, some of those commands end up being reentered manually.

The dedicated card should now be available. In order to avoid potential headaches, make sure your IOMMU groups doesn't contain things like storage controllers or whatever device you might be using. There's a small shell script at ArchWiki that lists valid IOMMU groups:

#!/bin/bash
shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

Example:

[...]
IOMMU Group 1:
	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1)
	01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
[...]

In that example, there's only the root port and the card (plus it's HDMI audio device). Passing it through should not break anything.

At that point, you can just follow the first tutorial noted at that article.

After making sure your VM is unloaded, you can run the following script:

ID="10de IDID"
BDF="0000:0x:00.0"
# Unbind a PCI function from its driver as necessary
[ ! -e /sys/bus/pci/devices/$BDF/driver/unbind ] || \
        echo -n $BDF > /sys/bus/pci/devices/$BDF/driver/unbind

modprobe nvidia
modprobe nvidia_modeset
modprobe nvidia_drm

/etc/init.d/nvidia-persistenced start
/etc/init.d/bumblebeed start

It is important to note that it is also a crude implementation and, as such doesn't deal all that much with the integrated audio device (as it's not of much importance to me). Perhaps you should edit your scripts to include it.

improve that article!

((https://mathiashueber.com/fighting-error-43-nvidia-gpu-virtual-machine/))