[PVE-User] Mellanox ConnectX-5 and SR-IOV

[Uwe Sauter]

Chris,

thanks for taking a look.


Am 14.05.20 um 23:13 schrieb Chris Hofstaedtler | Deduktiva:
> * Uwe Sauter <uwe.sauter.de at gmail.com> [200514 22:23]:
> [...]
>> More details:
>>
>> I followed these two instructions:
>>
>> https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-4-connectx-5-with-kvm--ethernet-x
>>
>> https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connect-ib-connectx-4-with-kvm--infiniband-x
>>
>> At about halfway through the second site I can't go on because the mentioned paths are not available in /sys:
>> /sys/class/infiniband/mlx5_0/device/sriov/0/policy
>> /sys/class/infiniband/mlx5_0/device/sriov/0/node
>> /sys/class/infiniband/mlx5_0/device/sriov/0/port
> 
> Disclaimer: I don't have the hardware you're talking about.
> 
> However, I took a quick look at the mainline kernel drivers and the
> driver sources from mellanox.com - the mainline kernel does _NOT_
> have the code for these files.
> 
> I guess if you want to use that, you'll have to install the ofed
> driver from mellanox.com (preferably by starting with the sources
> for Ubuntu 20.04).

As I mentioned I tried to install Mellanox OFED (for Debian) but it wants to uninstall many PVE related packages. If anyone has
successfully installed MOFED on PVE 6 and can provide instructions, I'd be happy.


> PS: [1] looks like up to date documentation
> [1] https://docs.mellanox.com/pages/viewpage.action?pageId=25146819#SingleRootIOVirtualization(SR-IOV)-settingupsr-iov
>  

This same document also mentions:

"""
Write to the sysfs file the number of Virtual Functions you need to create for the PF. You can use one of the following equivalent
files:
You can use one of the following equivalent files:
- A standard Linux kernel generated file that is available in the new kernels.
echo [num_vfs] > /sys/class/infiniband/mlx5_0/device/sriov_numvfs

Note: This file will be generated only if IOMMU is set in the grub.conf file (by adding intel_iommu=on, as seen in the fourth step
under “Setting Up SR-IOV”).
"""

which I can see and following:

"""
- A file generated by the mlx5_core driver with the same functionality as the kernel generated one.
echo [num_vfs] > /sys/class/infiniband/mlx5_0/device/mlx5_num_vfs

Note: This file is used by old kernels that do not support the standard file. In such kernels, using sriov_numvfs results in the
following error: “bash: echo: write error: Function not implemented”.
"""

which I don't see.


When I follow this instructions with

echo 15 > /sys/class/infiniband/mlx5_0/device/sriov_numvfs

I get entries in the form of

/sys/class/infiniband/mlx5_0/device/virtfn{NUMBER_FROM_0_to_14} -> ../0000:5e:00.1 (and so forth)

but those directories don't have files to set the necessary node and port GUIDs.



Regards,

	Uwe