[PVE-User] MxGPU with AMD S7150

[Mark Adams]

For posterity, I sorted this by speaking with ASRock Rack and getting an
as-yet unreleased bios that has the ARI forwarding option. Enabled this and
all working now.


On Fri, 1 Mar 2019 at 13:20, Dominik Csapak <d.csapak at proxmox.com> wrote:

> On 01.03.19 14:13, Mark Adams wrote:
> > On Fri, 1 Mar 2019 at 12:52, Dominik Csapak <d.csapak at proxmox.com>
> wrote:
> >
> >> On 01.03.19 13:37, Mark Adams wrote:
> >>> Hi All,
> >>>
> >>> I'm trying this out, based on the wiki post and the forum posts:
> >>>
> >>>
> >>
> https://forum.proxmox.com/threads/amd-s7150-mxgpu-with-proxmox-ve-5-x.50464/
> >>>
> >>> https://pve.proxmox.com/wiki/MxGPU_with_AMD_S7150_under_Proxmox_VE_5.x
> >>>
> >>> However I'm having issues getting the gim driver working. Was just
> >>> wondering if the Proxmox staff member that tested this out came across
> >> this
> >>> particular issue, or if anyone else had any insights.
> >>
> >> Hi, i am the one that tested this.
> >>
> >
> > Hi Dominik, Thanks for getting back to me so quickly.
> >
>
> Hi, no problem
>
> >
> >>
> >>>
> >>> My hardware is an ASRock EPYCD8-2T motherboard (SR-IOV enabled in bios)
> >> and
> >>> an AMD S7150. Proxmox is 5.3-11.
> >>>
> >>> When running the modprobe of gim, it crashes out with the following:
> >>>
> >>> [Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:299) Enable SRIOV
> >>> [Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:300) Enable SRIOV vfs
> >>> count = 16
> >>> [Fri Mar  1 12:31:49 2019] pci 0000:61:02.0: [1002:692f] type 7f class
> >>> 0xffffff
> >>> [Fri Mar  1 12:31:49 2019] pci 0000:61:02.0: unknown header type 7f,
> >>> ignoring device
> >>> [Fri Mar  1 12:31:50 2019] gim error:(enable_sriov:311) Fail to enable
> >>> sriov, status = fffffffb
> >>> [Fri Mar  1 12:31:50 2019] gim error:(set_new_adapter:668) Failed to
> >>> properly enable SRIOV
> >>> [Fri Mar  1 12:31:50 2019] gim info:(gim_probe:91) AMD GIM probe:
> >> pf_count
> >>> = 1
> >>>
> >>
> >> mhmm i cannot really remember if that exact error message occured, but
> >> you have to enable several things in the bios
> >>
> >> AMD-Vi/VT-d
> >> SR-IOV
> >> ARI
> >> and possibly above-4g-decoding
> >>
> >> also make sure you enable the 'legacy' or non uefi oprom for
> >> that card
> >>
> >> on our supermicro board we could select the oprom for each pcie port
> >> separately
> >>
> >
> > It's the same with this ASRock Rack board. I've set the oprom to legacy
> for
> > PCIE slot 1, but it doesn't seem to make any difference. I've also tried
> > other slots but that doesn't make a difference either.
> >
> > The only thing I can't find, is any option relating to ARI. Do you recall
> > at all what the option was called? I think the supermicro and asrock
> boards
> > are pretty similar when it comes to options, but maybe this board is
> > missing ARI.
>
> With ARI i mean Alternative Routing-ID Interpretation, an PCI Extension[1]
>
> >
> > Also I have ACS enabled but that doesn't help either.
> >
> >
> >> names may be different in your bios,
> >> or some options may not exists at all
> >>
> >> at last, a different pcie port may be necessary, depending on how
> >> the mainboard is wired (with epyc all pcie ports should go to the
> >> cpu, but i do not know about your specific board)
> >>
> >> if all else fails, i would open an issue on github for the gim project
> >> and ask there if anything is known
> >>
> >
> > Thanks I will do that.
> >
>
> ok, if i remember anything else, i will answer here on the list
>
> 1:
>
> https://pcisig.com/sites/default/files/specification_documents/ECN-alt-rid-interpretation-070604.pdf
>
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user