[pve-devel] [PATCH pve-docs v2] update the PCI(e) docs

Noel Ullreich n.ullreich at proxmox.com
Thu Mar 16 13:05:13 CET 2023

A little update to the PCI(e) docs with the plan of reworking the PCI
wiki as well.

Along some minor grammar fixes added:
 * how to check if kernelmodules are being loaded
 * how to check which drivers to blacklist
 * how to add softdeps for module loading
 * where to find kernel params

Signed-off-by: Noel Ullreich <n.ullreich at proxmox.com>

changes from v1:
 * fixed spelling mistakes
 * reduced code snippets of how to check iommu groupings to one
 * moved where to find kernel params to kernel cmdline section
 * removed wrong info on display output. will add correct info to
 * changed module names to variable-names, so that people can't
   blindly copy-paste.
 * restructured commit message ;)

 qm-pci-passthrough.adoc | 75 ++++++++++++++++++++++++++++++++++-------
 system-booting.adoc     |  9 +++++
 2 files changed, 71 insertions(+), 13 deletions(-)

diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc
index df6cf21..3f75ed0 100644
--- a/qm-pci-passthrough.adoc
+++ b/qm-pci-passthrough.adoc
@@ -16,16 +16,17 @@ device anymore on the host or in any other VM.
 General Requirements
-Since passthrough is a feature which also needs hardware support, there are
-some requirements to check and preparations to be done to make it work.
+Since passthrough is performed on real hardware, it needs to fulfill some
+requirements. A brief overview of these requirements is given below, for more
+information on specific devices, see
+https://pve.proxmox.com/wiki/PCI_Passthrough[PCI Passthrough Examples].
 Your hardware needs to support `IOMMU` (*I*/*O* **M**emory **M**anagement
 **U**nit) interrupt remapping, this includes the CPU and the mainboard.
-Generally, Intel systems with VT-d, and AMD systems with AMD-Vi support this.
+Generally, Intel systems with VT-d and AMD systems with AMD-Vi support this.
 But it is not guaranteed that everything will work out of the box, due
 to bad hardware implementation and missing or low quality drivers.
@@ -44,8 +45,8 @@ some configuration to enable PCI(e) passthrough.
-First, you have to enable IOMMU support in your BIOS/UEFI. Usually the
-corresponding setting is called `IOMMU` or `VT-d`,but you should find the exact
+First, you will have to enable IOMMU support in your BIOS/UEFI. Usually the
+corresponding setting is called `IOMMU` or `VT-d`, but you should find the exact
 option name in the manual of your motherboard.
 For Intel CPUs, you may also need to enable the IOMMU on the
@@ -72,6 +73,9 @@ hardware IOMMU. To enable these options, add:
 to the xref:sysboot_edit_kernel_cmdline[kernel commandline].
+For a complete list of kernel commandline options (of kernel 5.15), see 
 .Kernel Modules
 You have to make sure the following modules are loaded. This can be achieved by
@@ -92,6 +96,14 @@ After changing anything modules related, you need to refresh your
 # update-initramfs -u -k all
+To check if the modules are being loaded, the output of
+# lsmod | grep vfio
+should include the four modules from above.
 .Finish Configuration
 Finally reboot to bring the changes into effect and check that it is indeed
@@ -105,10 +117,11 @@ should display that `IOMMU`, `Directed I/O` or `Interrupt Remapping` is
 enabled, depending on hardware and kernel the exact message can vary.
 It is also important that the device(s) you want to pass through
-are in a *separate* `IOMMU` group. This can be checked with:
+are in a *separate* `IOMMU` group. This can be checked with a call to the {pve}
-# find /sys/kernel/iommu_groups/ -type l
+# pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""
 It is okay if the device is in an `IOMMU` group together with its functions
@@ -159,8 +172,8 @@ PCI(e) card, for example a GPU or a network card.
 Host Configuration
-In this case, the host must not use the card. There are two methods to achieve
+{pve} tries to automatically make the PCI(e) device unavailable for the host.
+However, if this doesn't work, there are two things that can be done:
 * pass the device IDs to the options of the 'vfio-pci' modules by adding
@@ -175,7 +188,7 @@ the vendor and device IDs obtained by:
 # lspci -nn
-* blacklist the driver completely on the host, ensuring that it is free to bind
+* blacklist the driver on the host completely, ensuring that it is free to bind
 for passthrough, with
@@ -183,11 +196,49 @@ for passthrough, with
 in a .conf file in */etc/modprobe.d/*.
+To find the drivername, execute
+# lspci -k
+for example:
+# lspci -k | grep -A 3 "VGA"
+will output something similar to
+01:00.0 VGA compatible controller: NVIDIA Corporation GP108 [GeForce GT 1030] (rev a1)
+	Subsystem: Micro-Star International Co., Ltd. [MSI] GP108 [GeForce GT 1030]
+	Kernel driver in use: <some-module>
+	Kernel modules: <some-module>
+Now we can blacklist the drivers by writing them into a .conf file:
+echo "blacklist <some-module>" >> /etc/modprobe.d/blacklist.conf
 For both methods you need to
 xref:qm_pci_passthrough_update_initramfs[update the `initramfs`] again and
 reboot after that.
+Should this not work, you might need to set a soft dependency to load the gpu
+modules before loading 'vfio-pci'. This can be done with the 'softdep' flag, see
+also the manpages on 'modprobe.d' for more information.
+For example, if you are using drivers named <some-module>:
+# echo "softdep <some-module> pre: vfio-pci" >> /etc/modprobe.d/<some-module>.conf
 .Verify Configuration
 To check if your changes were successful, you can use
@@ -262,7 +313,6 @@ For example:
 # qm set VMID -hostpci0 02:00,device-id=0x10f6,sub-vendor-id=0x0000
 Other considerations
@@ -288,7 +338,6 @@ Currently, the most common use case for this are NICs (**N**etwork
 physical port. This allows using features such as checksum offloading, etc. to
 be used inside a VM, reducing the (host) CPU overhead.
 Host Configuration
diff --git a/system-booting.adoc b/system-booting.adoc
index 30621a6..d0f847e 100644
--- a/system-booting.adoc
+++ b/system-booting.adoc
@@ -272,6 +272,15 @@ initrd   /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve
 Editing the Kernel Commandline
+A complete list of kernel parameters can be found at 
+replace <YOUR-KERNEL-VERSION> with the major.minor version (e.g. 5.15). You can
+find your kernel version by running
+# uname -r
 You can modify the kernel commandline in the following places, depending on the
 bootloader used:

More information about the pve-devel mailing list