[PVE-User] Best configuration among these

[Philippe Schwarz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,
i'm going to receive a pack of three servers.
Dell R730
1 X Xeon  E5 2600
16 GB
1 X Gb NIC
1 UPS 1500 W

Going to pay for add-ons:
Second Xeon
+ 48GB (Ensuring it's ECC RAM)
1 Dual 10Gbe SFP+ Intel NIC
+IDRAC Enterprise 8GB (for fencing)
+ Hot plug PSU
+ 2 Raid1 SD Card (OS)
+ 6 CPU Proxmox Licence (base or standard)
5 year NBD warranty

Those 3 servers will be involved in a three-way HA cluster.


I'm wondering about the shared storage solution.
Whether home-made (10Gbe + iSCSI/NFS) or built-in solution (Vendors
DAS bay) :

"home-made"
FreeBSD 10.1 + ZFS + iSCSI (or NFS)
Dell/HP  5U + 32 2,5" bays
2 x Intel Xeon E5-2630
64 GB RAM ECC
2 X  RAID 1GB non-volatile cache
Firmware should'nt block non-certified disks... (Hp reseller already
told me this would be nearly impossible)
IDRAC Enterprise 8GB
2 X Hotplug PSU > 800W
1 X Dual 10GbE SFP+ Intel NIC
1 X Quad 1Gb Intel NIC
2 X SD card 8GB / SSD 64 GB (OS)
+ 1 (not redundant) or 2 (redundant configuration) 8 ports 10Gbe SFP+
+ 1 UPS
5 year NBD warranty

"Built-in"
SAS attachment to servers
Dell/HP  5U + 32 2,5" bays
2 X  RAID 1GB non-volatile cache
Firmware should'nt block non-certified disks.(Hp reseller already told
me this would be nearly impossible)
- - Ciphering of the data pool neither using SED drives nor having to
pay licence fees. (Hp reseller already told me this (not using SED
drives) would be impossible)
- - Ability to use SSD caches without licence fees
- - Ability to replicate to another storage without licence fees
1 UPS
5 year NBD warranty


Despite the fact i love ZFS (snapshot, easy control, high IOPS,..) ,
i'm not against having a look at a black box (In fact i'm not sure,
i'd love to..)

Dell's offer is a MD3200 with 4X900GB disks,3 year NBD
warranty..Cheap, but under our needs (I really don't trust Raid5 and
can't affort RAID10, with 1.8 TB usable space).

Needs :
- - start at 5 TB of usable space .
- - 500 clients
- - HA for
+ 1 openLDAP/Samba 3 (soon 4) file/directory servers
+ few Windows servers (antivirus, WSUS, autocad,solidworks licence,..)
+ 1 proxy squid/Squidguard (100Mb optical fiber Wan link) (this one is
gonna eat a huge amount of IOPS)
+ 1 Web server (very poor IOPS)
+...all the services we're going to set up in the future (Packetfence,
LTSP,...)

BTW, i'm public school, so rich enough to buy thoses equipments,
but... no money for many years after that....
And admin time is far cheaper than any couple of bucks ;-)




Disk Setup:
For 1TB SSD, my ZFS setup would be :
On each controller (redundancy) : 6X1TB RAIDZ1 (Can afford it due to
the tremendous level of IOPS)
So : a mirror of 5TB raidz1
I'm able to loose 1 controller and 1 disk on the last mirror.
Easy to extend, huge IOPS, pretty expensive
12 SSD ~ 7200 €


For 1.2TB HDD, my ZFS setup would be :
A stripe of mirror, each 2ways mirror is split upon the 2 controllers
( 4 X 1.2TB mirror)
I'm able to loose 1 controller or 1 disk on each mirror.
Following the constructor's advices (using it's own brand disks) easy
to extend, a bit cheaper...
8 HDD ~ 4500 €

My questions, at last :
- - Issue with any of these configurations ? Pros&cons
- - Ability to evolve in a near/far future ?
- - Issue with fencing ?
- - How to backup ? (ZFS send/receive or rsync for the first, but wha
about the second ??)
- - Should i use high consumer-grade SSD (Samsung 850 Pro) for huge IOPS
(and high level of raid of course) or entreprise-grade HDD (10Krpm,SAS
in a RAID10 ZFS pool) for roughly the same ratio size/price ? None of
the constructor will assume the replacement of their branded HD by
consumer grade SSD, but i found links that proved it should work....It
"should".. Should i run such a risk ??? If it doesn't work, i'm
getting into big trouble.... The reseller focused on that point..
- - Is ZFS setup wrong ?
- - Forgot something ?


Thanks by advance
Best regards


BCC to admin team
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlTTMO4ACgkQlhqCFkbqHRapsACfehUn6iHFZOQydtUW2pAxWBaf
hx8AnR1v5B9TGejA4TYHJdSodseBOGzw
=6Xu/
-----END PGP SIGNATURE-----