[PVE-User] Shared storage recommendations

[BSD]

> On lundi, févr. 25, 2019 at 6:22 PM, Frederic Van Espen <frederic.ve at gmail.com (mailto:frederic.ve at gmail.com)> wrote:
> Hi,
>
> We're designing a new datacenter network where we will run proxmox nodes on
> about 30 servers. Of course, shared storage is a part of the design.
>
> What kind of shared storage would anyone recommend based on their
> experience and what kind of network equipment would be essential in that
> design? Let's assume for a bit that budget is not constrained too much. We
> should be able to afford a vendor specific iSCSI device, or be able to
> implement an open source solution like Ceph.
>
> Concerning storage space and IOPS requirements, we're very modest in the
> current setup (about 13TB storage space used, very modest IOPS, about 6500
> write IOPS and 4200 read IOPS currently distributed in the whole network
> according to the prometheus monitoring).
>
> Key in the whole setup is day to day maintainability and scalability.
>
>

We are using NFS shared storage with a ZFS base (using FreeNAS) since many years now with a record "0" problem!

Our architecture is both basic in terms of design and quite sophisticated when It comes to network stack.
Basically all nodes have a shared storage on a dedicated /24 subnet that's only used for the cluster

We are using OpenVSwitch and bonding at this level
All other operational subnets are put on top of that and clearly separated.

This was highlighted here : https://www.osnet.eu/fr/content/tutoriels/mise-en-place-dun-cluster-proxmox-4-avec-nfs-sous-freenas
In french (but schema is easy to figure out).

And yes, we have upgraded our Hypervisors from Proxmox version 3 to 4 to 5 !

We are using pve-zsync to snapshot our CT and some of the KVM.

It has proven to be both resilient and efficient.

ToDay, I would probably invest in more NVMe based infra if I had the opportunity.
Ceph is known to work well also, scope is different though.

What we have here is both :
shared storage for some KVM which are critical and can be moved from one hypervisor to another during upgrades (no downtime)
local storage for less critical nodes (CT) which can accept upgrade downtime with no pb

We are using one backup server (FreeNAS) and one NFS server (also FreeNAS).

You have to design your architecture based on your needs and make sure you balance efficiency with simplicity / maintainability.

Everything is under excellent control - and we do sleep well at night !

P.S. Basic knowledge of ZFS is mandatory in this design.
>
> Cheers,
>
> Frederic
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user