[pve-devel] [PATCH docs v3] pvecm, network: add section on corosync over bonds

[Friedrich Weber]

Testing has shown that running corosync (only) over a bond can be
problematic in some failure scenarios and for certain bond modes. The
documentation only discourages bonds for corosync because corosync can
switch between available networks itself, but does not mention other
caveats when using bonds for corosync.

Hence, extend the documentation with recommendations and caveats
regarding bonds for corosync.

Signed-off-by: Friedrich Weber <f.weber at proxmox.com>
---

Notes:
    Aaron suggested we could expose the bond-lacp-rate in the GUI to
    make it easier to change the setting on the PVE side. I'd open a
    feature report for this.
    
    Changes since v2:
    - fix wording in the failure scenario description
    - explain that load-balancing bond modes are affected and why
    - clarify that the caveats apply whenver a bond is used for Corosync
      traffic (even if only as a redundant link)
    
    Changes since v1:
    - move to its own section under "Cluster Network"
    - reword remarks about bond-lacp-rate fast
    - reword remark under "Requirements"

 pve-network.adoc |  4 +++-
 pvecm.adoc       | 49 ++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 48 insertions(+), 5 deletions(-)

diff --git a/pve-network.adoc b/pve-network.adoc
index 2dec882..b361f97 100644
--- a/pve-network.adoc
+++ b/pve-network.adoc
@@ -495,7 +495,9 @@ use the active-backup mode.
 
 For the cluster network (Corosync) we recommend configuring it with multiple
 networks. Corosync does not need a bond for network redundancy as it can switch
-between networks by itself, if one becomes unusable.
+between networks by itself, if one becomes unusable. Some bond modes are known
+to be problematic for Corosync, see
+xref:pvecm_corosync_over_bonds[Corosync over Bonds].
 
 The following bond configuration can be used as distributed/shared
 storage network. The benefit would be that you get more speed and the
diff --git a/pvecm.adoc b/pvecm.adoc
index 312a26f..60afaaf 100644
--- a/pvecm.adoc
+++ b/pvecm.adoc
@@ -89,10 +89,8 @@ NOTE: To ensure reliable Corosync redundancy, it is essential to have at least
 another link on a different physical network. This enables Corosync to keep the
 cluster communication alive should the dedicated network be down.
 +
-NOTE: A single link backed by a bond is not enough to provide Corosync
-redundancy. When a bonded interface fails and Corosync cannot fall back to
-another link, it can lead to  asymmetric communication in the cluster, which in
-turn can lead to the cluster losing quorum.
+NOTE: A single link backed by a bond can be problematic in certain failure
+scenarios, see xref:pvecm_corosync_over_bonds[Corosync Over Bonds].
 
 * The root password of a cluster node is required for adding nodes.
 
@@ -606,6 +604,49 @@ transport to `udp` or `udpu` in your xref:pvecm_edit_corosync_conf[corosync.conf
 but keep in mind that this will disable all cryptography and redundancy support.
 This is therefore not recommended.
 
+[[pvecm_corosync_over_bonds]]
+Corosync Over Bonds
+~~~~~~~~~~~~~~~~~~~
+
+Using a xref:sysadmin_network_bond[bond] as a Corosync link can be problematic
+in certain failure scenarios. If one of the bonded interfaces fails and stops
+transmitting packets, but its link state stays up, and there are no other
+Corosync links available, some bond modes may cause a state of asymmetric
+connectivity where cluster nodes can only communicate with different subsets of
+other nodes. Affected are bond modes that provide load balancing, as these
+modes may still try to send out a subset of packets via the failed interface.
+In case of asymmetric connectivity, Corosync may not be able to form a stable
+quorum in the cluster. If this state persists and HA is enabled, nodes may
+fence themselves, even if their respective bond is still fully functioning. In
+the worst case, the whole cluster may fence itself.
+
+We recommend at least one dedicated physical NIC for the primary Corosync link,
+see xref:pvecm_cluster_requirements[Requirements]. Bonds may be used as
+additional links for increased redundancy. To avoid fencing in the failure
+scenario outlined above, the following caveats apply whenever a bond is used
+for Corosync traffic:
+
+* We *advise against* using bond modes *balance-rr*, *balance-xor*,
+  *balance-tlb*, or *balance-alb* for Corosync traffic. As explained above,
+  they can cause asymmetric connectivity in certain failure scenarios.
+
+* *IEEE 802.3ad (LACP)*: This bond mode can cause asymmetric connectivity in
+  certain failure scenarios as explained above, but it can recover from this
+  state, as each side of the bond (Proxmox VE node and switch) can stop using a
+  bonded interface if it has not received three LACPDUs in a row on it.
+  However, with default settings, LACPDUs are only sent every 30 seconds,
+  yielding a failover time of 90 seconds. This is too long, as nodes with HA
+  resources will fence themselves already after roughly one minute without a
+  stable quorum. If LACP bonds are used for corosync traffic, we recommend
+  setting `bond-lacp-rate fast` *on the Proxmox VE node and the switch*!
+  Setting this option on one side requests the other side to send an LACPDU
+  every second. Setting this option on both sides can reduce the failover time
+  in the scenario above to 3 seconds and thus prevent fencing.
+
+* Bond mode *active-backup* will not cause asymmetric connectivity in the
+  failure scenario described above. The node whose bond experienced the failure
+  may lose connection to the cluster and, if HA is enabled, fence itself.
+
 Separate Cluster Network
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
-- 
2.47.2