[PVE-User] Three node Hyperconverged PVE+Ceph and failure domains...

[storm]

when operating a 3-node cluster, you have to ensure that at least 2 
nodes are up and operational.

If you want the possibility for 2 nodes failing, you need to move to the 
next odd number: 5 - you need at least a 5 node cluster if you want to 
survive the loss of two nodes without problems.

We have a 7 node cluster, so 3 nodes can fail, but we also have to raise 
the Ceph - size to 4, because if three nodes fail you have a high 
possibility, that placement groups will be unavailable because they were 
replicated only to the three nodes which are down.


btw - I think you should look at this hyperconverged solution as if it 
were two different clusters, the proxmox cluster and the ceph cluster 
although it is "all in one node"you are operating two clusters, with 
different preconditions.


best regards


Am 10/03/2021 um 11:47 schrieb Marco Gaiarin:
> One of the most interesting configuration of PVE is the three node,
> switchless (full mesh) configuration, depicted in some PVE docs, most
> notably:
>
> 	https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
> 	https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark-2020-09
>
> But lurking 'ceph-user' mailing list, some weeks ago, lead to an
> interesting discussion about 'failure domains', and many user depicted
> the three node cluster as 'insecure'.
>
> The reasoning are about:
>
> a) 'min_size = 2' is a must if you need to keep your data safe; you can
>   set 'min_size = 1', but clearly there's no scrub/checksumming, so no
> real guarantee against data corruption.
>
> b) but in a three node setup, with 'min_size = 2', if a node goes down,
>   the cluster switch in 'readonly' at the very first subsequent failure,
> eg the cluster does not handle more then a failure.
>
> c) you can change the failure domain, eg:
> 	mon osd down out subtree limit = osd
>   but in this way you have to guarantee (at worst case) room for the
> double of the space on a single node (eg, three node cluster with 2TB of
> space each, to guarantee the 'min_size = 2' you cannot use more then 1TB
> space on overral cluster; so, a 6TB total disk space for a 1TB usable
> space).
>
>
> I'm wrong? If not, the 3-node hyperconverged cluster is suitable only
> for testing?
>
>
> Thanks.
>