[PVE-User] Multi Data Center Cluster or Not

[Thomas Lamprecht]

On 28.04.21 04:03, alexandre derumier wrote:
> On 27/04/2021 20:38, JR Richardson wrote:
>> I'm looking for suggestions for geo-diversity using PROXMOX
>> Clustering. I understand running hypervisors in the same cluster in
>> multiple data centers is possible with high capacity/low latency
>> inter-site links. What I'm learning is there could be better ways,
>> like running PROXMOX backup servers (PBS) with Remote Sync. Using PBS
>> is interesting but would require manually restoring nodes should a
>> failure occur.
>>
>> I'm looking for best practice or suggestions in topology that folks
>> are using successfully or even tales of failure for what to avoid.
>
> If you want same cluster on multiple datacenter, you really need low latency (for proxmox && storage), and at least 3 datacenters to keep quorum.
> 
> if you need a 2dc datacenter, with 1 primary && 1 backup as disaster recovery
> 
> you could manually replicate a zfs or ceph storage to the backup dc (with snapshot export/import), or other storage replication feature if you have a san like netapp for example and do an rsync of /etc/pve.
> 

We know of setups which use rbd-mirror to mirror their production Ceph pool
to a second DC for recovery on failure. It's still needs a bit of hands-on
approach on setup and actual recovery can be prepared too (pre-create matching
VMs, maybe lock them by default so no start is done by accident).

We also know some city-gov IT people which run their cluster over multiple
DCs, but they have the luck to be able to run redundant fiber with LAN-like
latency between those DCs, which may not be an option for everyone.

A multi-datacenter management is planned, but we currently are still fleshing
out the basis, albeit some features required for that to happen are in-work.
Nothing ready to soon, though, just mentioning as FYI.

cheers,
Thomas