[PVE-User] Supported configuration question

[Pongrácz István]

----------------eredeti üzenet-----------------
Feladó: "Enrico M" enrico at majaglug.net 
Címzett: pve-user at pve.proxmox.com 
Dátum: Sat, 21 Sep 2013 17:24:57 +0200
-------------------------------------------------
 
>> 
>> So, technically you can adjust proxmox as you need. If one not a tech.
>> genie, no problem, just follow the mainstream: a supported hardware raid
>> card + some disks and that's all.
>> 
>> Bye,
>> István
>> 
>> 
> 
> I'm interested in zfs on linux + ssd for zil and l2arc.
> Could you tell some details about your configuration?
> Thanks a lot
> Enrico
>

Hi,

I use 
- one SSD for boot/root filesystem, dedicated to proxmox itself.
- two SSD, partitioned for L2ARC and ZIL (mirrored, some GBs enough)
- 4 WD green drives (anyway, green drives not really recommended) in raid10 equivalent
- Here is the pool layout

 pool: zfsstorage1
 state: ONLINE
 scan: scrub repaired 0 in 5h59m with 0 errors on Sat Sep 21 10:11:02 2013
config:

 NAME STATE READ WRITE CKSUM
 zfsstorage1 ONLINE 0 0 0
 mirror-0 ONLINE 0 0 0
 scsi-SATA_WDC_WD15EARS-00_WD-WMAZA3142219 ONLINE 0 0 0
 scsi-SATA_WDC_WD15EARS-00_WD-WCAYY0094106 ONLINE 0 0 0
 mirror-1 ONLINE 0 0 0
 scsi-SATA_WDC_WD15EARS-00_WD-WCAZA3999177 ONLINE 0 0 0
 scsi-SATA_WDC_WD15EARS-00_WD-WMAZA0845754 ONLINE 0 0 0
 logs
 mirror-2 ONLINE 0 0 0
 scsi-SATA_OCZ-AGILITY3_OCZ-35CM9ZXZ3O5XOC90-part2 ONLINE 0 0 0
 scsi-SATA_OCZ-AGILITY3_OCZ-Z8J00QD2S3RG4SXQ-part2 ONLINE 0 0 0
 cache
 scsi-SATA_OCZ-AGILITY3_OCZ-35CM9ZXZ3O5XOC90-part1 ONLINE 0 0 0
 scsi-SATA_OCZ-AGILITY3_OCZ-Z8J00QD2S3RG4SXQ-part1 ONLINE 0 0 0

- I created the vz layout under zfsstorage1 (private, images, dump, template), including all virtual machines in their own dataset, like this example:
NAME 
zfsstorage1 
zfsstorage1/vz 
zfsstorage1/vz/dump 
zfsstorage1/vz/images 
zfsstorage1/vz/images/102 
zfsstorage1/vz/images/103 
zfsstorage1/vz/private/101 
zfsstorage1/vz/private/107 
zfsstorage1/vz/private/124 

- In proxmox gui I set a new directory, pointed to zfsstorage1/vz layout.

Hints:
- use standard sata controllers without any cache or turn off cache, avoid hardware raid cards
- zfs 6.2 is the latest and greatest, easy to compile
- when you upgrade your kernel, make sure you compile the zfs/spl for it and put into initrd
- limit your ARC (depends on your memory requirements, 1/4 of your physical ram is safe), example: limit to 4GB:
 /etc/modprobe.d/zfs.conf 
 options zfs zfs_arc_max=4294967296
- good balance between performance and compress ratio with lz4 compression
- if you have issues with proxmox populates your vz layout and cannot mount zfs datasets, do the following:
 * stop pve-cluster, pve-manager, apache2, get them out from starting services
 * wipe out the vz layout (be careful and check you do not delete live data)
 * write zfs mount -a and after that start these services stopped before in rc.local
- consider to put swap space to a zvol instead of ssd/hard drive
- use zfs-auto-snapshot and send/recv instead of pve provided backup
- on reliable network, you can replicate your zfs datasets incrementally via ssh using zfs send/recv of snapshots (I use openvpn for off-site backup, very reliable)
- you can play with blocksize, metadata/all cache for arc and l2arc for individual datasets, for example you can select optimal blocksize and cache (primarycache=metadata) for mysql and postgresql to get maximum performance
- you can clone existing dataset to save space, (plan ahead),
- you can use cluster filesystems on top of zfs, like glusterfs, afs, whatever

Bye,
István