[PVE-User] MxGPU with AMD S7150

[Mark Adams]

On Fri, 1 Mar 2019 at 12:52, Dominik Csapak <d.csapak at proxmox.com> wrote:

> On 01.03.19 13:37, Mark Adams wrote:
> > Hi All,
> >
> > I'm trying this out, based on the wiki post and the forum posts:
> >
> >
> https://forum.proxmox.com/threads/amd-s7150-mxgpu-with-proxmox-ve-5-x.50464/
> >
> > https://pve.proxmox.com/wiki/MxGPU_with_AMD_S7150_under_Proxmox_VE_5.x
> >
> > However I'm having issues getting the gim driver working. Was just
> > wondering if the Proxmox staff member that tested this out came across
> this
> > particular issue, or if anyone else had any insights.
>
> Hi, i am the one that tested this.
>

Hi Dominik, Thanks for getting back to me so quickly.


>
> >
> > My hardware is an ASRock EPYCD8-2T motherboard (SR-IOV enabled in bios)
> and
> > an AMD S7150. Proxmox is 5.3-11.
> >
> > When running the modprobe of gim, it crashes out with the following:
> >
> > [Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:299) Enable SRIOV
> > [Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:300) Enable SRIOV vfs
> > count = 16
> > [Fri Mar  1 12:31:49 2019] pci 0000:61:02.0: [1002:692f] type 7f class
> > 0xffffff
> > [Fri Mar  1 12:31:49 2019] pci 0000:61:02.0: unknown header type 7f,
> > ignoring device
> > [Fri Mar  1 12:31:50 2019] gim error:(enable_sriov:311) Fail to enable
> > sriov, status = fffffffb
> > [Fri Mar  1 12:31:50 2019] gim error:(set_new_adapter:668) Failed to
> > properly enable SRIOV
> > [Fri Mar  1 12:31:50 2019] gim info:(gim_probe:91) AMD GIM probe:
> pf_count
> > = 1
> >
>
> mhmm i cannot really remember if that exact error message occured, but
> you have to enable several things in the bios
>
> AMD-Vi/VT-d
> SR-IOV
> ARI
> and possibly above-4g-decoding
>
> also make sure you enable the 'legacy' or non uefi oprom for
> that card
>
> on our supermicro board we could select the oprom for each pcie port
> separately
>

It's the same with this ASRock Rack board. I've set the oprom to legacy for
PCIE slot 1, but it doesn't seem to make any difference. I've also tried
other slots but that doesn't make a difference either.

The only thing I can't find, is any option relating to ARI. Do you recall
at all what the option was called? I think the supermicro and asrock boards
are pretty similar when it comes to options, but maybe this board is
missing ARI.

Also I have ACS enabled but that doesn't help either.


> names may be different in your bios,
> or some options may not exists at all
>
> at last, a different pcie port may be necessary, depending on how
> the mainboard is wired (with epyc all pcie ports should go to the
> cpu, but i do not know about your specific board)
>
> if all else fails, i would open an issue on github for the gim project
> and ask there if anything is known
>

Thanks I will do that.

Regards,
Mark


>
> hope this helps
>
>