[PVE-User] Three node Hyperconverged PVE+Ceph and failure domains...

[Marco Gaiarin]

One of the most interesting configuration of PVE is the three node,
switchless (full mesh) configuration, depicted in some PVE docs, most
notably:

	https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
	https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark-2020-09

But lurking 'ceph-user' mailing list, some weeks ago, lead to an
interesting discussion about 'failure domains', and many user depicted
the three node cluster as 'insecure'.

The reasoning are about:

a) 'min_size = 2' is a must if you need to keep your data safe; you can
 set 'min_size = 1', but clearly there's no scrub/checksumming, so no
real guarantee against data corruption.

b) but in a three node setup, with 'min_size = 2', if a node goes down,
 the cluster switch in 'readonly' at the very first subsequent failure,
eg the cluster does not handle more then a failure.

c) you can change the failure domain, eg:
	mon osd down out subtree limit = osd
 but in this way you have to guarantee (at worst case) room for the
double of the space on a single node (eg, three node cluster with 2TB of
space each, to guarantee the 'min_size = 2' you cannot use more then 1TB
space on overral cluster; so, a 6TB total disk space for a 1TB usable
space).


I'm wrong? If not, the 3-node hyperconverged cluster is suitable only
for testing?


Thanks.

-- 
dott. Marco Gaiarin				        GNUPG Key ID: 240A3D66
  Associazione ``La Nostra Famiglia''          http://www.lanostrafamiglia.it/
  Polo FVG   -   Via della Bontà, 7 - 33078   -   San Vito al Tagliamento (PN)
  marco.gaiarin(at)lanostrafamiglia.it   t +39-0434-842711   f +39-0434-842797

		Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA!
      http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000
	(cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)