[PVE-User] creating 2 node cluster via crossover cable

[Francesco Ongaro]

On 30/05/19 17:15, Adam Weremczuk wrote:
> Anyway, I've tried creating a cluster using these "crossover" IPs (Ring
> 0 address) but it seems to be persisting of using primary IPs.

Hi Adam,

The part you are referring to IMHO can be resolved by specifying
hostnames when doing the corosync setup (eg: node1-corosync, etc)
and then managing IP resolution in /etc/hosts. This gives maximum
flexibility.

Proxmox nodes chat each other on the IP specified by the hostname (eg:
node1) in /etc/hosts so you will want to use the IP of the direct
attached/dedicated bandwidth network as the main IP.

The usage of the right interface can be verified by migrating a virtual
machine.

BTW did something similar with inexpensive 40GB Mellanox cards:

03:00.0 Network controller: Mellanox Technologies MT27500 Family
[ConnectX-3]

Since such cards are dual port you can connect up to 3 nodes without a
switch.

Alternatively you can bond the two direct attached copper cables (you
can find new and cheap ones online) between two machines if you are
only interested in ZFS replica (it would be nice to have LVM replica
too!).

In the latter, more complex, case my setup is the following:

/etc/network/interfaces

# Infiniband bonding
iface eno1 inet manual
iface eno1d1 inet manual
auto bond0
iface bond0 inet static
        address 192.X.X.X
        netmask 255.255.255.0
        network 192.X.X.0
        slaves eno1 eno1d1
#       bond_mode active-backup
        bond_mode balance-rr
        bond_miimon 100
        bond_downdelay 200
        bond_updelay 200

You can then verify that the bond rate negotiated 40/Gb:

# ethtool bond0
Settings for bond0:
	Supported ports: [ ]
	Supported link modes:   Not reported
	Supported pause frame use: No
	Supports auto-negotiation: No
	Advertised link modes:  Not reported
	Advertised pause frame use: No
	Advertised auto-negotiation: No
	Speed: 40000Mb/s
	Duplex: Full
	Port: Other
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: off
	Link detected: yes

This specific interface is a little choosy after recent kernel updates
and tends to revert to Fibre/IB mode opposed to Ethernet mode (the one
you want for corosync et alia).

If you see something like:

# ethtool eno1d1
Settings for eno1d1:
	Supported ports: [ FIBRE ]

You can try the following:

# cat /etc/modules
mlx4_en

# cat /etc/modprobe.d/mlx4.conf
blacklist mlx4_ib
options mlx4_core port_type_array="2,2"

# update-initramfs -u

# reboot

If anybody has other suggestion on how to force Mellanox Ethernet mode
I'm all ears!

Have a great day,
Francesco

-- 
Francesco Ongaro, Senior Security Researcher
ISGroup: Information Security Group (www.isgroup.it)
Tel       (+39) 045 4853232
Fax       (+39) 045 5111719
Voicemail (+39) 02 320624653

AVVISO PRIVACY

Il contenuto della presente e-mail ed i suoi allegati, sono diretti
esclusivamente al destinatario e devono ritenersi riservati, con
divieto di diffusione o di uso non conforme alle finalità per le quali
la presente e-mail è stata inviata.

Pertanto, ne è vietata la diffusione e la comunicazione da parte di
soggetti diversi dal destinatario, ai sensi degli artt. 616 e ss. c.p.
e D.lgs n. 196/03 Codice Privacy.

Se la presente e-mail ed i suoi allegati sono stati ricevuti per
errore, siete pregati di distruggere quanto ricevuto e di informare il
mittente al seguente recapito: isgroup at isgroup.it