[PVE-User] BIG cluster questions

[Eneko Lacunza]

Hi Alexandre,

El 26/6/21 a las 13:16, aderumier at odiso.com escribió:
> Le jeudi 24 juin 2021 à 16:30 +0200, Eneko Lacunza via pve-user a écrit :
>> Now for more general questions; do you think a 88-node Proxmox VE
>> cluster is feasible?
>
> Well, corosync is not really done to this amount of node. (I think the 
> hardcoded limit is around 100),
> but in practice, I have seen a lot of users having problem with 
> corosync communication starting around 30 nodes
>
> (Maybe with low-latency switches + fast frequencies cpu, it's possible 
> to keep latency enough low to get it working)

Thanks for this info. Documentation isn't helpful when it says "There’s 
no explicit limit for the number of nodes in a cluster. In practice, the 
actual possible node count may be limited by the host and network 
performance. Currently (2021), there are reports of clusters (using 
high-end enterprise hardware) with over 50 nodes in production." in 
https://pve.proxmox.com/wiki/Cluster_Manager (it's the same wording in 
administration guide)

>
>
>> Those 88 nodes will host about 14.000 VMs. Will HA manager be able to
>> manage them, or are they too many? (HA for those VMs doesn't seem to be
>> a requirement right know).
>
> I have cluster with 2500vms, it's working fine. (on a 15 nodes cluster 
> with around 200vms on each node).
> I don't known with 15000vms, maybe the main pve-crm loop will take 
> more time, I'm not sure about the timeouts.
>
>
> At work, I'm doing nodes 20 nodes cluster (1 by rack) to avoid to have 
> big cluster.
> Multi-cluster central management is not yet on the roadmap, but they 
> are some tool like
> https://maas.io/ <https://maas.io/>
> which allow to manage multi-cluster in a central way.
>
Thanks for this data. In 20 nodes cluster, can you share whether cluster 
network is physically different from VM traffic/storage, and it's speed?

Thanks

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/