[PVE-User] Cluster network via directly connected interfaces?

[Thomas Lamprecht]

On 11/22/18 7:29 PM, Frank Thommen wrote:
> Please excuse, if this is too basic, but after reading https://pve.proxmox.com/wiki/Cluster_Manager I wondered, if the cluster/corosync network could be built by directly connected network interfaces.  I.e not like this:
> 
>  +-------+
>  | pve01 |----------+
>  +-------+          |
>                     |
>  +-------+     +----------------+
>  | pve02 |-----| network switch |
>  +-------+     +----------------+
>                     |
>  +-------+          |
>  | pve03 |----------+
>  +-------+
> 
> 
> but like this:
> 
>  +-------+
>  | pve01 |---+
>  +-------+   |
>      |       |
>  +-------+   |
>  | pve02 |   |
>  +-------+   |
>      |       |
>  +-------+   |
>  | pve03 |---+
>  +-------+
> 
> (all connections 1Gbit, there are currently not plans to extend over three nodes)
> 
> I can't see any drawback in that solution.  It would remove one layer of hardware dependency and potential spof (the switch).  If we don't trust the interfaces, we might be able to configure a second network with the three remaining interfaces.
> 
> Is such a "direct-connection" topology feasible?  Recommended? Strictly not recommended?

full mesh is certainly not bad. for cluster network (corosync) latency is the key,
bandwidth isn't really much needed. So this surely not bad.
We use also some 10g (or 40G, not sure) full mesh for a ceph cluster network - you
safe a not to cheap switch and get full bandwidth and good latency.
The limiting factor is that this gets quite complex for bigger clusters, but besides
that it doesn't really has any drawbacks for inter cluster connects, AFAICT.

For multicast you need to try, as Uwe said, I'm currently not sure, it could work as
Linux can route multicast just fine (mrouter) but I don't remember exactly anymore -
sorry.

But if you try it it'd be great if you report back. Else unicast is in those cluster
sizes always an option - you really shouldn't have a problem as long as you do not put
storage traffic together with corosync (cluster) on the same net (corosync gets to much
latency spikes then).

> 
> I am currently just planning and thinking and there is no cluster (or even a PROXMOX server) in place.
> 
> Cheers
> frank