[PVE-User] (Very) basic question regarding PVE Ceph integration

[Alwin Antreich]

Hello Frank,

On Mon, Dec 17, 2018 at 11:52:26AM +0100, Frank Thommen wrote:
> Yes, I am aware of PVE and Ceph failover/healing capabilities.  But I always
> liked to separate basic and central services on the hardware level.  This
> way if one server "explodes", only one service is affected.  With PVE+Ceph
> on one node, such an outage would affect two basic services at once.  I
> don't say they wouldn't continue to run productively, but they would run in
> degraded and non-failure-safe mode - assumed we had three such nodes in the
> cluster - until the broken node can be restored.
> 
> But that's probably just my old-fashioned conservative approach.  That's why
> I wanted to ask the list members for their assessment ;-)
In the small constellation of 3x hyper-converged vs 3x PVE + 3x Ceph,
the result will be the same (<50% for quorum).

3x hyper-converged = can sustain one node failure in total
6x (3+3) separated nodes = one failed node (2x total) in each domain

Some thoughts to the separation. While it has its benefits to separate
Ceph from PVE, in that case one node in each domain can fail and they
are not interfering with the other domain. It might also give better
latency (no VM/CT), but this is more depend on the hardware used then
the separation.

On the other hand, when you already consider to use 6x nodes (from
above), then the hyper-converged setup might be more beneficial. As
Ceph's performance grows with the count of nodes (and OSDs). Also the
VM/CT will be more distributed throughout the cluster. A node failure
will have less impact on the count of VM/CT than a failure in a
separated setup.

So each setup has it's pro/con and needs to be weight (distribution vs
separation).

--
Cheers,
Alwin