[pve-devel] question/idea : managing big proxmox cluster (100nodes), get rid of corosync ?

[Alexandre DERUMIER]

Hi,

I'm looking for the future (1-2year), to manage big proxmox clusters (50-100nodes).

Currently I manage multiple cluster with max 16nodes, because corosync && multicast can be very difficult to tune.
(I'm also managing clusters across datacenters). 

It's become to be difficult to manage, as we can't easily migrate vm across clusters + we need to connect to differents cluster to manage vms.



I'm not an expert in cluster messaging, but I have found some projects which seem interesting:

serf:
https://www.serf.io/intro/index.html

a decentralized cluster messenging , based on gossip protocol



consul:
https://www.consul.io/

cluster messenging (it's use serf) + distributed key-value store + service management
(some beta fuse-fs implementation exist too) 



I don't known how much it could be to switch from corosync, pxmcfs,.... ?

Maybe something like serf is enough to replace corosync (an keep pmxcfs,ha management,...)
consul looks like corosync+ pmxcfs + ha services management.


So,It's just an idea, but I think getting rid of multicast/corosync could be a great step for proxmox.