[PVE-User] Infiniband Mellanox cards and DDR
Fabrizio Cuseo
f.cuseo at panservice.it
Tue Jul 7 11:46:41 CEST 2015
Hallo.
I am testing a Mellanox card (IBM Voltaire HCA 400Ex-D) with a cisco 4x DDR infiniband switch.
The problem I have is that the cards (that seems to be 4X DDR) have a 10Gbit (SDR) link.
---
03:00.0 InfiniBand: Mellanox Technologies MT25208 [InfiniHost III Ex] (rev 20)
---
root at pve1:/etc/apt/sources.list.d# ibstat
CA 'mthca0'
CA type: MT25208
Number of ports: 2
Firmware version: 5.3.0
Hardware version: 20
Node GUID: 0x0008f104039814d0
System image GUID: 0x0008f104039814d3
Port 1:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 6
LMC: 0
SM lid: 2
Capability mask: 0x02510a68
Port GUID: 0x0008f104039814d1
Link layer: InfiniBand
--------
root at pve1:/etc/apt/sources.list.d# ibportstate 6 0
CA PortInfo:
# Port info: Lid 6 port 0
LinkState:.......................Active
PhysLinkState:...................LinkUp
Lid:.............................6
SMLid:...........................2
LMC:.............................0
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps
LinkSpeedEnabled:................2.5 Gbps
LinkSpeedActive:.................2.5 Gbps
Mkey:............................<not displayed>
MkeyLeasePeriod:.................15
ProtectBits:.....................0
-----
If I try to set the speed at DDR, I have this error:
root at pve1:/etc/apt/sources.list.d# ibportstate 6 0 speed 2
Initial CA PortInfo:
# Port info: Lid 6 port 0
LinkState:.......................Active
PhysLinkState:...................LinkUp
Lid:.............................6
SMLid:...........................2
LMC:.............................0
LinkWidthSupported:..............1X or 4X
LinkWidthEnabled:................1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps
LinkSpeedEnabled:................2.5 Gbps
LinkSpeedActive:.................2.5 Gbps
Mkey:............................<not displayed>
MkeyLeasePeriod:.................15
ProtectBits:.....................0
ibportstate: iberror: failed: smp set portinfo failed
----
The switch reported this card in infiniband topology:
MT25218 InfiniHostEx Mellanox Technologies
Someone have tested the cards ?
Fabrizio Cuseo
--
---
Fabrizio Cuseo - mailto:f.cuseo at panservice.it
Direzione Generale - Panservice InterNetWorking
Servizi Professionali per Internet ed il Networking
Panservice e' associata AIIP - RIPE Local Registry
Phone: +39 0773 410020 - Fax: +39 0773 470219
http://www.panservice.it mailto:info at panservice.it
Numero verde nazionale: 800 901492
More information about the pve-user
mailing list