[pve-devel] [PATCH pve-docs v2] update the PCI(e) docs
Noel Ullreich
n.ullreich at proxmox.com
Thu Mar 16 13:05:13 CET 2023
A little update to the PCI(e) docs with the plan of reworking the PCI
wiki as well.
Along some minor grammar fixes added:
* how to check if kernelmodules are being loaded
* how to check which drivers to blacklist
* how to add softdeps for module loading
* where to find kernel params
Signed-off-by: Noel Ullreich <n.ullreich at proxmox.com>
---
changes from v1:
* fixed spelling mistakes
* reduced code snippets of how to check iommu groupings to one
* moved where to find kernel params to kernel cmdline section
* removed wrong info on display output. will add correct info to
Examples-Wiki
* changed module names to variable-names, so that people can't
blindly copy-paste.
* restructured commit message ;)
qm-pci-passthrough.adoc | 75 ++++++++++++++++++++++++++++++++++-------
system-booting.adoc | 9 +++++
2 files changed, 71 insertions(+), 13 deletions(-)
diff --git a/qm-pci-passthrough.adoc b/qm-pci-passthrough.adoc
index df6cf21..3f75ed0 100644
--- a/qm-pci-passthrough.adoc
+++ b/qm-pci-passthrough.adoc
@@ -16,16 +16,17 @@ device anymore on the host or in any other VM.
General Requirements
~~~~~~~~~~~~~~~~~~~~
-Since passthrough is a feature which also needs hardware support, there are
-some requirements to check and preparations to be done to make it work.
-
+Since passthrough is performed on real hardware, it needs to fulfill some
+requirements. A brief overview of these requirements is given below, for more
+information on specific devices, see
+https://pve.proxmox.com/wiki/PCI_Passthrough[PCI Passthrough Examples].
Hardware
^^^^^^^^
Your hardware needs to support `IOMMU` (*I*/*O* **M**emory **M**anagement
**U**nit) interrupt remapping, this includes the CPU and the mainboard.
-Generally, Intel systems with VT-d, and AMD systems with AMD-Vi support this.
+Generally, Intel systems with VT-d and AMD systems with AMD-Vi support this.
But it is not guaranteed that everything will work out of the box, due
to bad hardware implementation and missing or low quality drivers.
@@ -44,8 +45,8 @@ some configuration to enable PCI(e) passthrough.
.IOMMU
-First, you have to enable IOMMU support in your BIOS/UEFI. Usually the
-corresponding setting is called `IOMMU` or `VT-d`,but you should find the exact
+First, you will have to enable IOMMU support in your BIOS/UEFI. Usually the
+corresponding setting is called `IOMMU` or `VT-d`, but you should find the exact
option name in the manual of your motherboard.
For Intel CPUs, you may also need to enable the IOMMU on the
@@ -72,6 +73,9 @@ hardware IOMMU. To enable these options, add:
to the xref:sysboot_edit_kernel_cmdline[kernel commandline].
+For a complete list of kernel commandline options (of kernel 5.15), see
+https://www.kernel.org/doc/html/v5.15/admin-guide/kernel-parameters.html[kernel.org].
+
.Kernel Modules
You have to make sure the following modules are loaded. This can be achieved by
@@ -92,6 +96,14 @@ After changing anything modules related, you need to refresh your
# update-initramfs -u -k all
----
+To check if the modules are being loaded, the output of
+
+----
+# lsmod | grep vfio
+----
+
+should include the four modules from above.
+
.Finish Configuration
Finally reboot to bring the changes into effect and check that it is indeed
@@ -105,10 +117,11 @@ should display that `IOMMU`, `Directed I/O` or `Interrupt Remapping` is
enabled, depending on hardware and kernel the exact message can vary.
It is also important that the device(s) you want to pass through
-are in a *separate* `IOMMU` group. This can be checked with:
+are in a *separate* `IOMMU` group. This can be checked with a call to the {pve}
+API:
----
-# find /sys/kernel/iommu_groups/ -type l
+# pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist ""
----
It is okay if the device is in an `IOMMU` group together with its functions
@@ -159,8 +172,8 @@ PCI(e) card, for example a GPU or a network card.
Host Configuration
^^^^^^^^^^^^^^^^^^
-In this case, the host must not use the card. There are two methods to achieve
-this:
+{pve} tries to automatically make the PCI(e) device unavailable for the host.
+However, if this doesn't work, there are two things that can be done:
* pass the device IDs to the options of the 'vfio-pci' modules by adding
+
@@ -175,7 +188,7 @@ the vendor and device IDs obtained by:
# lspci -nn
----
-* blacklist the driver completely on the host, ensuring that it is free to bind
+* blacklist the driver on the host completely, ensuring that it is free to bind
for passthrough, with
+
----
@@ -183,11 +196,49 @@ for passthrough, with
----
+
in a .conf file in */etc/modprobe.d/*.
++
+To find the drivername, execute
++
+----
+# lspci -k
+----
++
+for example:
++
+----
+# lspci -k | grep -A 3 "VGA"
+----
++
+will output something similar to
++
+----
+01:00.0 VGA compatible controller: NVIDIA Corporation GP108 [GeForce GT 1030] (rev a1)
+ Subsystem: Micro-Star International Co., Ltd. [MSI] GP108 [GeForce GT 1030]
+ Kernel driver in use: <some-module>
+ Kernel modules: <some-module>
+----
++
+Now we can blacklist the drivers by writing them into a .conf file:
++
+----
+echo "blacklist <some-module>" >> /etc/modprobe.d/blacklist.conf
+----
For both methods you need to
xref:qm_pci_passthrough_update_initramfs[update the `initramfs`] again and
reboot after that.
+Should this not work, you might need to set a soft dependency to load the gpu
+modules before loading 'vfio-pci'. This can be done with the 'softdep' flag, see
+also the manpages on 'modprobe.d' for more information.
+
+For example, if you are using drivers named <some-module>:
+
+----
+# echo "softdep <some-module> pre: vfio-pci" >> /etc/modprobe.d/<some-module>.conf
+----
+
+
.Verify Configuration
To check if your changes were successful, you can use
@@ -262,7 +313,6 @@ For example:
# qm set VMID -hostpci0 02:00,device-id=0x10f6,sub-vendor-id=0x0000
----
-
Other considerations
^^^^^^^^^^^^^^^^^^^^
@@ -288,7 +338,6 @@ Currently, the most common use case for this are NICs (**N**etwork
physical port. This allows using features such as checksum offloading, etc. to
be used inside a VM, reducing the (host) CPU overhead.
-
Host Configuration
^^^^^^^^^^^^^^^^^^
diff --git a/system-booting.adoc b/system-booting.adoc
index 30621a6..d0f847e 100644
--- a/system-booting.adoc
+++ b/system-booting.adoc
@@ -272,6 +272,15 @@ initrd /EFI/proxmox/5.0.15-1-pve/initrd.img-5.0.15-1-pve
Editing the Kernel Commandline
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+A complete list of kernel parameters can be found at
+'https://www.kernel.org/doc/html/<YOUR-KERNEL-VERSION>/admin-guide/kernel-parameters.html'.
+replace <YOUR-KERNEL-VERSION> with the major.minor version (e.g. 5.15). You can
+find your kernel version by running
+
+----
+# uname -r
+----
+
You can modify the kernel commandline in the following places, depending on the
bootloader used:
--
2.30.2
More information about the pve-devel
mailing list