[pve-devel] Quorum problems with NICs Intel of 10 Gb/s and VMsturns off

[Cesar Peschiera]

After several checks, I found the problem in these two servers: a
configuration in the Hardware Bios that isn't compatible with the
pve-kernel-3.10.0-5, and my NICs was getting the link to down and after up.
(i guess that soon i will comunicate my setup of BIOS in Dell R720).
... :-)

But now i have other problem, with the mix of PVE-manager 3.3-5 and 2.3-13
versions in a PVE cluster of 8 nodes: I am losing quorum in several nodes
very often.

Moreover, for now i can not apply a upgrade to my old PVE nodes, so for the
moment i would like to know if is possible to make a quick configuration for
that all my nodes always has quorum.

The strange behaviour is that when i run "pvecm status", i get this message:
Version: 6.2.0
Config Version: 41
Cluster Name: ptrading
Cluster Id: 28503
Cluster Member: Yes
Cluster Generation: 8360
Membership state: Cluster-Member
Nodes: 8
Expected votes: 8
Total votes: 8
Node votes: 1
Quorum: 5
Active subsystems: 6
Flags:
Ports Bound: 0 177
Node name: pve5
Node ID: 5
Multicast addresses: 239.192.111.198
Node addresses: 192.100.100.50

And in the PVE GUI i see the red light in all the others nodes.

Can apply a some kind of temporal solution as "Quorum: 1" for that my nodes
can work well and not has this strange behaviour? (Only until I performed
the updates)
Or, what will be the more simple and quick temporal solution for avoid to do
a upgrade in my nodes?
(something as for example: add to the rc.local file a line that says: "pvecm 
expected 1")

Note about of the Quorum: I don't have any Hardware fence device enabled, so
i do not care that each node always have quorum (i always can turns off the
server manually and brutally if it is necessary).

----- Original Message ----- 
From: "Cesar Peschiera" <brain at click.com.py>
To: "Alexandre DERUMIER" <aderumier at odiso.com>
Cc: "pve-devel" <pve-devel at pve.proxmox.com>
Sent: Saturday, December 20, 2014 9:30 AM
Subject: Re: [pve-devel] Quorum problems with NICs Intel of 10 Gb/s and
VMsturns off


> Hi Alexandre
>
> I put 192.100.100.51 ip address directly to bond0, and i don't have
> network
> enabled (as if the node is totally isolated)
>
> This was my setup:
> -------------------
> auto bond0
> iface bond0 inet static
> address  192.100.100.51
> netmask  255.255.255.0
> gateway  192.100.100.4
> slaves eth0 eth2
> bond_miimon 100
> bond_mode 802.3ad
> bond_xmit_hash_policy layer2
>
> auto vmbr0
> iface vmbr0 inet manual
> bridge_ports bond0
> bridge_stp off
> bridge_fd 0
> post-up echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping
> post-up echo 1 > /sys/class/net/vmbr0/bridge/multicast_querier
>
> ...... :-(
>
> Some other suggestion?
>
> ----- Original Message ----- 
> From: "Alexandre DERUMIER" <aderumier at odiso.com>
> To: "Cesar Peschiera" <brain at click.com.py>
> Cc: "pve-devel" <pve-devel at pve.proxmox.com>
> Sent: Friday, December 19, 2014 7:59 AM
> Subject: Re: [pve-devel] Quorum problems with NICs Intel of 10 Gb/s and
> VMsturns off
>
>
> maybe can you try to put 192.100.100.51 ip address directly to bond0,
>
> to avoid corosync traffic going through to vmbr0.
>
> (I remember some old offloading bugs with 10gbe nic and linux bridge)
>
>
> ----- Mail original -----
> De: "Cesar Peschiera" <brain at click.com.py>
> À: "aderumier" <aderumier at odiso.com>
> Cc: "pve-devel" <pve-devel at pve.proxmox.com>
> Envoyé: Vendredi 19 Décembre 2014 11:08:33
> Objet: Re: [pve-devel] Quorum problems with NICs Intel of 10 Gb/s and
> VMsturns off
>
>>can you post your /etc/network/interfaces of theses 10gb/s nodes ?
>
> This is my configuration:
> Note: The LAN use 192.100.100.0/24
>
> #Network interfaces
> auto lo
> iface lo inet loopback
>
> iface eth0 inet manual
> iface eth1 inet manual
> iface eth2 inet manual
> iface eth3 inet manual
> iface eth4 inet manual
> iface eth5 inet manual
> iface eth6 inet manual
> iface eth7 inet manual
> iface eth8 inet manual
> iface eth9 inet manual
> iface eth10 inet manual
> iface eth11 inet manual
>
> #PVE Cluster and VMs (NICs are of 10 Gb/s):
> auto bond0
> iface bond0 inet manual
> slaves eth0 eth2
> bond_miimon 100
> bond_mode 802.3ad
> bond_xmit_hash_policy layer2
>
> #PVE Cluster and VMs:
> auto vmbr0
> iface vmbr0 inet static
> address 192.100.100.51
> netmask 255.255.255.0
> gateway 192.100.100.4
> bridge_ports bond0
> bridge_stp off
> bridge_fd 0
> post-up echo 0 >
> /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping
> post-up echo 1 > /sys/class/net/vmbr0/bridge/multicast_querier
>
> #A link for DRBD (NICs are of 10 Gb/s):
> auto bond401
> iface bond401 inet static
> address 10.1.1.51
> netmask 255.255.255.0
> slaves eth1 eth3
> bond_miimon 100
> bond_mode balance-rr
> mtu 9000
>
> #Other link for DRBD (NICs are of 10 Gb/s):
> auto bond402
> iface bond402 inet static
> address 10.2.2.51
> netmask 255.255.255.0
> slaves eth4 eth6
> bond_miimon 100
> bond_mode balance-rr
> mtu 9000
>
> #Other link for DRBD (NICs are of 10 Gb/s):
> auto bond403
> iface bond403 inet static
> address 10.3.3.51
> netmask 255.255.255.0
> slaves eth5 eth7
> bond_miimon 100
> bond_mode balance-rr
> mtu 9000
>
> #A link for the NFS-Backups (NICs are of 1 Gb/s):
> auto bond10
> iface bond10 inet static
> address 10.100.100.51
> netmask 255.255.255.0
> slaves eth8 eth10
> bond_miimon 100
> bond_mode balance-rr
> #bond_mode active-backup
> mtu 9000
>