[PVE-User] Storage System/Server question

[Adam Thompson]

The argument could easily be made that hardware RAID is currently the 
second-worst storage option for Proxmox VE.  (The worst being a single, 
non-redundant disk.)

The 3.3 installer now defaults to ZFS, I believe, clearly showing that 
it's a first-class citizen.
You can run CEPH on a single-host cluster, although that would be 
unusual unless you planned to expand later.

Fiber-channel is a viable choice, but if you're prepared to spend on 
10GE, you'd be better served under PVE by buying another half-dozen or 
so servers and running CEPH as a "converged" storage solution. Or even 
iSCSI, based on cost/performance ratios.

Storage latency turns out to be the killer, so in enormous environments, 
FC has an advantage.  However, in enormous environments, you should just 
run CEPH and stop paying obscene amounts of $$$ for FC.  Plus, PVE 
doesn't really scale to more than 16 servers, so we're not talking about 
1000+ IOPS anyway.

If you run ZFS, do not run it on top of hardware RAID.  ZFS is designed 
specifically to run on bare drives without a RAID layer in between.  ZFS 
*replaces* the RAID layer, so it would be exactly like putting, say, 6 
hard drives into a RAID6 array, exporting 6 equal-sized LUNs, and then 
running software RAID6 across the 6 LUNs.  Inefficient, 
performance-killing, and makes diagnosis very difficult.

If you run CEPH or ZFS, you can also use SSDs as a caching layer in 
front of spinning rust, effectively giving you a virtual hybrid disk - 
all the storage of spinning rust, and almost all of the performance of 
flash.

Perhaps the best way to look at it is: look at the explicitly-listed 
storage options on the Storage tab of an existing PVE install.  Pick one 
of those, as they're all fully-supported.  Anything else is a 
second-class citizen.

-Adam


On 05/04/2015 03:30 PM, Steffen Wagner wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>   
> Hi guys!
>
> does someone have experience with storage systems in combination with
> Proxmox? Can someone explain if a hardware Raid is still the "only" and
> "best" way?
> Is ZFS over UFS or JBOD also a possible safe way to go?
>
> Currently I am running 4x240GB Intel Datacenter SSD's with an Adaptec
> Hardware Raid5... but I need more storage and want to buy a storage
> system with 10G Fiber / ...