[PVE-User] Mellanox ConnectX-5 and SR-IOV

[Gianni Milo]

I don't own this hardware too, but was able to compile its kernel modules
(.ko) from its source package which is provided at Mellanox web site.
I used a test PVE (VM) with pve-kernel and pve-headers installed on it.
Then I extracted the   source package (tar.gz) on temp location and
executed install.pl script which took a while to complete.
That produced a couple of kernel modules within
/lib/modules/<kernel_version>/updates/dkms.
 I guess that from that point it's just a case of loading appropriate
modules with modprobe command.

G.


On Fri, 15 May 2020 at 15:29, Thomas Lamprecht <t.lamprecht at proxmox.com>
wrote:

> On 5/15/20 9:00 AM, Uwe Sauter wrote:
> > Chris,
> >
> > thanks for taking a look.
> >
> >
> > Am 14.05.20 um 23:13 schrieb Chris Hofstaedtler | Deduktiva:
> >> * Uwe Sauter <uwe.sauter.de at gmail.com> [200514 22:23]:
> >> [...]
> >>> More details:
> >>>
> >>> I followed these two instructions:
> >>>
> >>>
> https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-4-connectx-5-with-kvm--ethernet-x
> >>>
> >>>
> https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connect-ib-connectx-4-with-kvm--infiniband-x
> >>>
> >>> At about halfway through the second site I can't go on because the
> mentioned paths are not available in /sys:
> >>> /sys/class/infiniband/mlx5_0/device/sriov/0/policy
> >>> /sys/class/infiniband/mlx5_0/device/sriov/0/node
> >>> /sys/class/infiniband/mlx5_0/device/sriov/0/port
> >> Disclaimer: I don't have the hardware you're talking about.
> >>
> >> However, I took a quick look at the mainline kernel drivers and the
> >> driver sources from mellanox.com - the mainline kernel does _NOT_
> >> have the code for these files.
> >>
> >> I guess if you want to use that, you'll have to install the ofed
> >> driver from mellanox.com (preferably by starting with the sources
> >> for Ubuntu 20.04).
> > As I mentioned I tried to install Mellanox OFED (for Debian) but it
> wants to uninstall many PVE related packages. If anyone has
> > successfully installed MOFED on PVE 6 and can provide instructions, I'd
> be happy.
> >
> >
>
> I do not have this HW either but from a quick look you need to do two
> things:
>
> # apt install pve-headers
> (the script assumes that this is linux-headers-*, and thus fails here
> already)
>
>
> Open the "install.pl" file in an editor and search for the "sub uninstall"
> There add an early return:
>
> return 0;
>
> immediately after the opening { after that you should be able to build it.
>
> It tries to install it directly with dpkg and that may fail, if so try to
> install it manually with `apt install /path/to/package.deb ...other.deb`
> you may want to pass all debs from the DEBS sub directories to that one
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>