The virt-manager video is scrambled on this VM configured to use Intel's GVT-g GPU pass-through. The screen works, I can log in on it, manipulate things on the desktop. I just can't read anything.
Any folk out there knowledgeable on GVT-g that can help me out here?
Background:
Intel's GVT-g is a feature built into 5th through 9th generation core processors that allows virtualization environments like Xen and KVM to present the Intel graphics processors resident on a hosts motherboard to multiple virtual clients, simultaneously. This in turn permits the clients to run 3D accelerated applications at near-hardware speeds.
Explanation:
I have implemented Intel's GVT-g GPU pass-through per the ArchLinux wiki on a Ubuntu 22.04 vm host (Coffee Lake/UHD Graphics 630). The VMs involved are Windows 10 and 2019, I can log into them using RDP (Remmina) despite virt-manager.
On the VMs, the hosts GPU is, indeed, passed-through to the device manager. Driver installation is pretty straight-forward.
The really cool thing is; I get acceleration for my 3D applications even though I am in an RDP session. And I can get it on up to 8 VMs, simultaneously!
Everything works except the video in virt-manager!
The details:
Host configuration
inxi -Fj
System:
Host: cview Kernel: 5.15.0-56-generic x86_64 bits: 64 Desktop: MATE 1.26.0
Distro: Ubuntu 22.04.1 LTS (Jammy Jellyfish)
Machine:
Type: Desktop Mobo: ASUSTeK model: PRIME B360M-A v: Rev X.0x
serial: <superuser required> UEFI: American Megatrends v: 3202
date: 07/10/2021
CPU:
Info: 6-core model: Intel Core i5-9600K bits: 64 type: MCP cache:
L2: 1.5 MiB
Speed (MHz): avg: 800 min/max: 800/4600 cores: 1: 800 2: 800 3: 800
4: 800 5: 800 6: 800
Graphics:
Device-1: Intel CoffeeLake-S GT2 [UHD Graphics 630] driver: i915 v: kernel
Device-2: NVIDIA GM107GL [Quadro K1200] driver: nouveau v: kernel
Display: x11 server: X.Org v: 1.21.1.3 driver: X: loaded: modesetting
unloaded: fbdev,vesa gpu: nouveau resolution: 1: 1920x1080~60Hz
2: 1080x1920~60Hz 3: 1080x1920~60Hz 4: 1920x1080~60Hz
OpenGL: renderer: NV117 v: 4.3 Mesa 22.0.5
Audio:
Device-1: Intel Cannon Lake PCH cAVS driver: snd_hda_intel
Device-2: NVIDIA GM107 High Definition Audio [GeForce 940MX]
driver: snd_hda_intel
Sound Server-1: ALSA v: k5.15.0-56-generic running: yes
Sound Server-2: PulseAudio v: 15.99.1 running: yes
Sound Server-3: PipeWire v: 0.3.48 running: yes
Network:
Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet
driver: r8169
IF: enp4s0 state: up speed: 1000 Mbps duplex: full mac: a8:5e:45:53:72:dd
IF-ID-1: br0 state: up speed: 1000 Mbps duplex: unknown
mac: 86:cf:5b:e2:98:aa
IF-ID-2: br2 state: down mac: 16:e4:81:03:54:b3
IF-ID-3: br257 state: down mac: d6:38:d7:30:2f:c5
IF-ID-4: br258 state: down mac: ae:87:82:f8:88:d7
IF-ID-5: br259 state: down mac: 7a:29:2d:9e:5d:ce
IF-ID-6: br260 state: down mac: da:0b:81:1d:66:c2
IF-ID-7: br261 state: down mac: 16:8f:e2:9c:de:43
IF-ID-8: br262 state: down mac: fa:de:05:71:7a:2b
IF-ID-9: virbr0 state: down mac: 52:54:00:83:4d:49
IF-ID-10: vlan2 state: up speed: 1000 Mbps duplex: full
mac: a8:5e:45:53:72:dd
IF-ID-11: vlan257 state: up speed: 1000 Mbps duplex: full
mac: a8:5e:45:53:72:dd
IF-ID-12: vlan258 state: up speed: 1000 Mbps duplex: full
mac: a8:5e:45:53:72:dd
IF-ID-13: vlan259 state: up speed: 1000 Mbps duplex: full
mac: a8:5e:45:53:72:dd
IF-ID-14: vlan260 state: up speed: 1000 Mbps duplex: full
mac: a8:5e:45:53:72:dd
IF-ID-15: vlan261 state: up speed: 1000 Mbps duplex: full
mac: a8:5e:45:53:72:dd
IF-ID-16: vlan262 state: up speed: 1000 Mbps duplex: full
mac: a8:5e:45:53:72:dd
IF-ID-17: vnet13 state: unknown speed: 10 Mbps duplex: full
mac: fe:54:00:1c:29:2c
IF-ID-18: vnet7 state: unknown speed: 10 Mbps duplex: full
mac: fe:54:00:ab:cb:b0
Drives:
Local Storage: total: 5.92 TiB used: 3.16 TiB (53.3%)
ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 970 EVO Plus 1TB
size: 931.51 GiB
ID-2: /dev/nvme1n1 vendor: Samsung model: SSD 970 EVO Plus 1TB
size: 931.51 GiB
ID-3: /dev/sda vendor: Samsung model: SSD 860 PRO 512GB size: 476.94 GiB
ID-4: /dev/sdb vendor: Western Digital model: WD4003FZEX-00Z4SA0
size: 3.64 TiB
Partition:
ID-1: / size: 447.02 GiB used: 82.6 GiB (18.5%) fs: btrfs dev: /dev/sda3
ID-2: /boot/efi size: 484 MiB used: 5.2 MiB (1.1%) fs: vfat
dev: /dev/sda1
Swap:
ID-1: swap-1 type: partition size: 29.8 GiB used: 17.2 MiB (0.1%)
dev: /dev/sda2
Sensors:
System Temperatures: cpu: 27.8 C mobo: N/A gpu: nouveau temp: 47.0 C
Fan Speeds (RPM): N/A gpu: nouveau fan: 3840
Info:
Processes: 371 Uptime: 1d 4h 19m Memory: 31.2 GiB used: 10.24 GiB (32.8%)
Shell: Bash inxi: 3.3.13
Grub cmdline
GRUB_CMDLINE_LINUX="ipv6.disable=1"
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on i915.enable_gvt=1"
Example vm configuration ("mesa" libvirt configuration)
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>win10</name>
...
<devices>
...
<graphics type='spice'>
<listen type='none'/>
<gl enable='yes' rendernode='/dev/dri/by-path/pci-0000:00:02.0-render'/>
</graphics>
...
<video>
<model type='none'/>
</video>
<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='on'>
<source>
<address uuid='f056b522-1c29-4efe-8032-41affa070017'/>
</source>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</hostdev>
...
</devices>
...
<qemu:commandline>
<qemu:arg value='-set'/>
<qemu:arg value='device.hostdev0.x-igd-opregion=on'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.hostdev0.ramfb=on'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.hostdev0.driver=vfio-pci-nohotplug'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.hostdev0.xres=1920'/>
<qemu:arg value='-set'/>
<qemu:arg value='device.hostdev0.yres=1080'/>
<qemu:env name='MESA_LOADER_DRIVER_OVERRIDE' value='i965'/>
</qemu:commandline>
</domain>
...
Libvirt hook
The virtual GPUs are created/destroyed on VM startup/shutdown using the following libvirt hook:
#!/bin/bash
#
# /etc/libvirt/hooks/qemu
# Automatically creates and destroys virtual GPUs on domain startup and shutdown
# Note: Libvirt will not pick up this file until restarted - reboot your system after creation.
#
GVT_PCI="0000:00:02.0"
MDEV_TYPE="i915-GVTg_V5_4"
fDevType="/sys/bus/pci/devices/$GVT_PCI/mdev_supported_types/$MDEV_TYPE"
domain=$1
cmd1=$2
cmd2=$3
function vGpuConfig {
if [ -e "$fDevType/create" ]; then
if [ "$cmd1" = "prepare" ] && [ "$cmd2" = "begin" ]; then
if [ ! -e "$fDev" ]; then
echo "Starting GPU $GVT_GUID..."
echo "$GVT_GUID" > "$fDevType/create"
else
echo "FAIL!! Device already running: $fDev"
fi
elif [ "$cmd1" = "release" ] && [ "$cmd2" = "end" ]; then
if [ -e "$fDev/remove" ]; then
echo "Stopping GPU $GVT_GUID..."
echo 1 > "$fDev/remove"
else
echo "FAIL!! Device is not running: $fDev"
fi
fi
else
echo "FAIL!! Device type does not exist: $fDevType"
fi
}
if [ $# -ge 3 ]; then
if [ "$domain" = "win10" ]; then
GVT_GUID="cb33ec6d-ad44-4702-b80f-c176f56afea1"
fDev="/sys/bus/pci/devices/$GVT_PCI/$GVT_GUID"
vGpuConfig
elif [ "$domain" = "Win2k19" ]; then
GVT_GUID="f056b522-1c29-4efe-8032-41affa070017"
fDev="/sys/bus/pci/devices/$GVT_PCI/$GVT_GUID"
vGpuConfig
fi
else
echo "usage: $0 <domain> prepare begin | release end"
fi
Guest extensions
Summary:
This is exceeding cool technology even though Intel has moved on (later generations will implement GPU pass-through using SR-IOV).
I would like to publish my procedure on how to implement GVT-g. But virt-manager not working (plus the small matter of a SIGABRT in X when running lightdm) are holding me up. I'd appreciate any assistance I can get.
