[PVE-User] Mellanox ConnectX-5 and SR-IOV

[Uwe Sauter]

Gianni, Thomas,

thanks for your time.

I mitigated the issue by exchanging the Infiniband card with an older model that still uses the mlx4 driver. Still had 
some issued finding the correct module settings for this piece of hardware but finally got it working.

Maybe some time, when I have a spare minute I'll try your proposals.

Have a relaxing weekend,

	Uwe


Am 15.05.20 um 19:18 schrieb Gianni Milo:
> I don't own this hardware too, but was able to compile its kernel modules (.ko) from its source package which is 
> provided at Mellanox web site.
> I used a test PVE (VM) with pve-kernel and pve-headers installed on it. Then I extracted the   source package (tar.gz) 
> on temp location and executed install.pl <http://install.pl> script which took a while to complete.
> That produced a couple of kernel modules within /lib/modules/<kernel_version>/updates/dkms.
>   I guess that from that point it's just a case of loading appropriate modules with modprobe command.
> 
> G.
> 
> 
> On Fri, 15 May 2020 at 15:29, Thomas Lamprecht <t.lamprecht at proxmox.com <mailto:t.lamprecht at proxmox.com>> wrote:
> 
>     On 5/15/20 9:00 AM, Uwe Sauter wrote:
>      > Chris,
>      >
>      > thanks for taking a look.
>      >
>      >
>      > Am 14.05.20 um 23:13 schrieb Chris Hofstaedtler | Deduktiva:
>      >> * Uwe Sauter <uwe.sauter.de at gmail.com <mailto:uwe.sauter.de at gmail.com>> [200514 22:23]:
>      >> [...]
>      >>> More details:
>      >>>
>      >>> I followed these two instructions:
>      >>>
>      >>> https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-4-connectx-5-with-kvm--ethernet-x
>      >>>
>      >>> https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connect-ib-connectx-4-with-kvm--infiniband-x
>      >>>
>      >>> At about halfway through the second site I can't go on because the mentioned paths are not available in /sys:
>      >>> /sys/class/infiniband/mlx5_0/device/sriov/0/policy
>      >>> /sys/class/infiniband/mlx5_0/device/sriov/0/node
>      >>> /sys/class/infiniband/mlx5_0/device/sriov/0/port
>      >> Disclaimer: I don't have the hardware you're talking about.
>      >>
>      >> However, I took a quick look at the mainline kernel drivers and the
>      >> driver sources from mellanox.com <http://mellanox.com> - the mainline kernel does _NOT_
>      >> have the code for these files.
>      >>
>      >> I guess if you want to use that, you'll have to install the ofed
>      >> driver from mellanox.com <http://mellanox.com> (preferably by starting with the sources
>      >> for Ubuntu 20.04).
>      > As I mentioned I tried to install Mellanox OFED (for Debian) but it wants to uninstall many PVE related packages.
>     If anyone has
>      > successfully installed MOFED on PVE 6 and can provide instructions, I'd be happy.
>      >
>      >
> 
>     I do not have this HW either but from a quick look you need to do two
>     things:
> 
>     # apt install pve-headers
>     (the script assumes that this is linux-headers-*, and thus fails here already)
> 
> 
>     Open the "install.pl <http://install.pl>" file in an editor and search for the "sub uninstall"
>     There add an early return:
> 
>     return 0;
> 
>     immediately after the opening { after that you should be able to build it.
> 
>     It tries to install it directly with dpkg and that may fail, if so try to
>     install it manually with `apt install /path/to/package.deb ...other.deb`
>     you may want to pass all debs from the DEBS sub directories to that one
> 
>     _______________________________________________
>     pve-user mailing list
>     pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>
>     https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>