[PVE-User] Mellanox ConnectX-5 and SR-IOV

[Uwe Sauter]

Hi all,

I had to change the hardware of one of my Proxmox installations and now have the problem that I cannot configure a 
Mellanox ConnectX-5 card for SR-IOV/passthrough. To be more precise I can boot the VM and it also recognizes the 
Infiniband device but I'm unable to assign a Node GUID and Port GUID for each of the virtual Infiniband devices.
The old host had a ConnectX-3 card that "just worked".

Did anyone run into this, too, and solve this issue?

More details:

I followed these two instructions:

https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-4-connectx-5-with-kvm--ethernet-x

https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connect-ib-connectx-4-with-kvm--infiniband-x

At about halfway through the second site I can't go on because the mentioned paths are not available in /sys:
/sys/class/infiniband/mlx5_0/device/sriov/0/policy
/sys/class/infiniband/mlx5_0/device/sriov/0/node
/sys/class/infiniband/mlx5_0/device/sriov/0/port


I also tried to install Mellanox OFED but the installation routine wants to uninstall many PVE related packages:
pve-qemu-kvm
ceph-mgr
pve-ha-manager
python-cephfs
ceph-base
libpve-storage-perl
ceph
librados2-perl
libradosstriper1
python-rados
librdmacm1:amd64
ceph-mon
libpve-guest-common-perl
rbd-nbd
pve-manager
glusterfs-client
python-rgw
libiscsi7:amd64
glusterfs-common
fio
libosmcomp4
python-ceph
ceph-fuse
pve-container
spiceterm
librgw2
qemu-server
ceph-osd
ceph-mds
python-rbd
proxmox-ve
ceph-common
ibverbs-providers:amd64
libcephfs2
librados2
librbd1

There is also this site

https://docs.mellanox.com/pages/viewpage.action?pageId=12013542#SingleRootIOVirtualization(SR-IOV)-ConfiguringSR-IOVforConnectX-4/Connect-IB/ConnectX-5(InfiniBand)

which mentiones a path that exists on my host: /sys/class/infiniband/mlx5_0/device/sriov_numvfs

But two steps later they use a path that again doesn't exist: /sys/class/infiniband/mlx5_0/device/sriov/0/node


My Proxmox installation details:

# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.34-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-1
pve-kernel-helper: 6.2-1
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.34-1-pve: 5.4.34-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
ceph: 12.2.13-pve1
ceph-fuse: 12.2.13-pve1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.3
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-5
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1


Regards,

	Uwe