[PVE-User] Infiniband Mellanox cards and DDR

[Michael Rasmussen]

Another thing is that your cables must support DDR as well.

On July 7, 2015 3:07:00 PM CEST, Fabrizio Cuseo <f.cuseo at panservice.it> wrote:
>I have found that the 400EX is a SDR card, while the 400EX-D is a DDR
>card. I don't know if is a firmware problem.
>
>
>http://sup.xenya.si/sup/info/voltaire/HCAwebfinal.pdf
>
>HCA 400Ex/Ex-F
>• Dual port 4X (10 Gbps) InfiniBand PCI-Express low 
>profile host channel adapter
>HCA 400Ex-D
>• Dual port 4X DDR (20 Gbps) InfiniBand PCI-Express 
>low profile host channel adapter
>HCA 400
>• Dual port 4X (10 Gbps) InfiniBand PCI/PCI-X low 
>profile host channel adapter
>
>
>
>
>----- Messaggio originale -----
>Da: "Michael Rasmussen" <mir at miras.org>
>A: "Fabrizio Cuseo" <f.cuseo at panservice.it>, "pve-user"
><pve-user at pve.proxmox.com>
>Inviato: Martedì, 7 luglio 2015 12:01:03
>Oggetto: Re: [PVE-User] Infiniband Mellanox cards and DDR
>
>AFAIK this card is a SDR card so DDR is not possible. 
>
>
>On July 7, 2015 11:46:41 AM CEST, Fabrizio Cuseo
><f.cuseo at panservice.it> wrote: 
>
>Hallo. 
>
>I am testing a Mellanox card (IBM Voltaire HCA 400Ex-D) with a cisco 4x
>DDR infiniband switch. 
>
>The problem I have is that the cards (that seems to be 4X DDR) have a
>10Gbit (SDR) link. 
>
>--- 
>03:00.0 InfiniBand: Mellanox Technologies MT25208 [InfiniHost III Ex]
>(rev 20) 
>--- 
>
>root at pve1:/etc/apt/sources.list.d# ibstat 
>CA 'mthca0' 
>CA type: MT25208 
>Number of ports: 2 
>Firmware version: 5.3.0 
>Hardware version: 20 
>Node GUID: 0x0008f104039814d0 
>System image GUID: 0x0008f104039814d3 
>Port 1: 
>State: Active 
>Physical state: LinkUp 
>Rate: 10 
>Base lid: 6 
>LMC: 0 
>SM lid: 2 
>Capability mask: 0x02510a68 
>Port GUID: 0x0008f104039814d1 
>Link laye
> r:
>InfiniBand 
>
>-------- 
>
>root at pve1:/etc/apt/sources.list.d# ibportstate 6 0 
>CA PortInfo: 
># Port info: Lid 6 port 0 
>LinkState:.......................Active 
>PhysLinkState:...................LinkUp 
>Lid:.............................6 
>SMLid:...........................2 
>LMC:.............................0 
>LinkWidthSupported:..............1X or 4X 
>LinkWidthEnabled:................1X or 4X 
>LinkWidthActive:.................4X 
>LinkSpeedSupported:..............2.5 Gbps 
>LinkSpeedEnabled:................2.5 Gbps 
>LinkSpeedActive:.................2.5 Gbps 
>Mkey:............................<not displayed> 
>MkeyLeasePeriod:.................15 
>ProtectBits:.....................0 
>
>----- 
>
>
>If I try to set the speed at DDR, I have this error: 
>
>root at pve1:/etc/apt/sources.list.d# ibportstate 6 0 speed 2 
>Initial CA PortInfo: 
># Port info: Lid 6 port 0 
>LinkState:.......................Active 
>PhysLinkState:...................LinkUp 
>Lid:.............................6 
>SMLid:...........................2 
>LMC:.............................0 
>LinkWidthSupported:..............1X or 4X 
>LinkWidthEnabled:................1X or 4X 
>LinkWidthActive:.................4X 
>LinkSpeedSupported:..............2.5 Gbps 
>LinkSpeedEnabled:................2.5 Gbps 
>LinkSpeedActive:.................2.5 Gbps 
>Mkey:............................<not displayed> 
>MkeyLeasePeriod:.................15 
>ProtectBits:.....................0 
>ibportstate: iberror: failed: smp set portinfo failed 
>
>
>---- 
>
>The switch reported this card in infiniband topology: 
>
>MT25218 InfiniHostEx Mellanox Technologies 
>
>
>Someone have tested the cards ? 
>
>Fabrizio Cuseo 
>
>
>
>
>
>
>
>
>-- 
>Sent from my Android phone with K-9 Mail. Please excuse my brevity. 
>---- This mail was virus scanned and spam checked before delivery. This
>mail is also DKIM signed. See header dkim-signature. 
>
>-- 
>---
>Fabrizio Cuseo - mailto:f.cuseo at panservice.it
>Direzione Generale - Panservice InterNetWorking
>Servizi Professionali per Internet ed il Networking
>Panservice e' associata AIIP - RIPE Local Registry
>Phone: +39 0773 410020 - Fax: +39 0773 470219
>http://www.panservice.it  mailto:info at panservice.it
>Numero verde nazionale: 800 901492

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

----

This mail was virus scanned and spam checked before delivery.
This mail is also DKIM signed. See header dkim-signature.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.proxmox.com/pipermail/pve-user/attachments/20150707/58472db1/attachment.htm>