[PVE-User] MxGPU with AMD S7150

[Dominik Csapak]

On 01.03.19 14:13, Mark Adams wrote:
> On Fri, 1 Mar 2019 at 12:52, Dominik Csapak <d.csapak at proxmox.com> wrote:
> 
>> On 01.03.19 13:37, Mark Adams wrote:
>>> Hi All,
>>>
>>> I'm trying this out, based on the wiki post and the forum posts:
>>>
>>>
>> https://forum.proxmox.com/threads/amd-s7150-mxgpu-with-proxmox-ve-5-x.50464/
>>>
>>> https://pve.proxmox.com/wiki/MxGPU_with_AMD_S7150_under_Proxmox_VE_5.x
>>>
>>> However I'm having issues getting the gim driver working. Was just
>>> wondering if the Proxmox staff member that tested this out came across
>> this
>>> particular issue, or if anyone else had any insights.
>>
>> Hi, i am the one that tested this.
>>
> 
> Hi Dominik, Thanks for getting back to me so quickly.
> 

Hi, no problem

> 
>>
>>>
>>> My hardware is an ASRock EPYCD8-2T motherboard (SR-IOV enabled in bios)
>> and
>>> an AMD S7150. Proxmox is 5.3-11.
>>>
>>> When running the modprobe of gim, it crashes out with the following:
>>>
>>> [Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:299) Enable SRIOV
>>> [Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:300) Enable SRIOV vfs
>>> count = 16
>>> [Fri Mar  1 12:31:49 2019] pci 0000:61:02.0: [1002:692f] type 7f class
>>> 0xffffff
>>> [Fri Mar  1 12:31:49 2019] pci 0000:61:02.0: unknown header type 7f,
>>> ignoring device
>>> [Fri Mar  1 12:31:50 2019] gim error:(enable_sriov:311) Fail to enable
>>> sriov, status = fffffffb
>>> [Fri Mar  1 12:31:50 2019] gim error:(set_new_adapter:668) Failed to
>>> properly enable SRIOV
>>> [Fri Mar  1 12:31:50 2019] gim info:(gim_probe:91) AMD GIM probe:
>> pf_count
>>> = 1
>>>
>>
>> mhmm i cannot really remember if that exact error message occured, but
>> you have to enable several things in the bios
>>
>> AMD-Vi/VT-d
>> SR-IOV
>> ARI
>> and possibly above-4g-decoding
>>
>> also make sure you enable the 'legacy' or non uefi oprom for
>> that card
>>
>> on our supermicro board we could select the oprom for each pcie port
>> separately
>>
> 
> It's the same with this ASRock Rack board. I've set the oprom to legacy for
> PCIE slot 1, but it doesn't seem to make any difference. I've also tried
> other slots but that doesn't make a difference either.
> 
> The only thing I can't find, is any option relating to ARI. Do you recall
> at all what the option was called? I think the supermicro and asrock boards
> are pretty similar when it comes to options, but maybe this board is
> missing ARI.

With ARI i mean Alternative Routing-ID Interpretation, an PCI Extension[1]

> 
> Also I have ACS enabled but that doesn't help either.
> 
> 
>> names may be different in your bios,
>> or some options may not exists at all
>>
>> at last, a different pcie port may be necessary, depending on how
>> the mainboard is wired (with epyc all pcie ports should go to the
>> cpu, but i do not know about your specific board)
>>
>> if all else fails, i would open an issue on github for the gim project
>> and ask there if anything is known
>>
> 
> Thanks I will do that.
> 

ok, if i remember anything else, i will answer here on the list

1: 
https://pcisig.com/sites/default/files/specification_documents/ECN-alt-rid-interpretation-070604.pdf