[PVE-User] Proxmox + ZFS: Performance issues

[Ralf]

Hi,

On 04/24/16 23:57, Lindsay Mathieson wrote:
> On 25/04/2016 5:03 AM, Ralf wrote:
>> Some Western Digital Green stuff:
>
>  There's a big part of your problem, never ever use WD greens in
> servers, especially with ZFS. They are a desktop drive, not suitable 
I know, I know, but they were available :-)
> for 24/7 operation. They have power saving modes which can't be
> disabled and slow them down, they don't support the ata TLER, which
> necessary for prompt timing out on disk errors - greens will hang for
> every accessing a bad read/write which will freeze your whole zfs pool
> (I've been there). Might be worth scanning your dmesg for disk errors.
No errors, I can dump the whole disk without *any* errors, constantly
100MiB/s. I absolutely understand your argumentation and I really should
buy better disks and migrate data, but I still don't understand that
tremendous performance breakdown. 30-50 MiB/s, yes, that'd be okay, but
5MiB/s. Srsly?
>
> But you should really be using NAS rated drives. I thoroughly
> recommend the Western Digital Reds, they aren't high performance but
> they are very reliable and fast enough. For better performance (and
> more $) the Hitachi NAS drives are also very good.
>
> I had two WD Blacks start failing on me last weekend, less than a year
> in the server, a tense few hours :) And one had failed earlier after 6
> months. They run too hot and aren't up to the 24/7 workout they get in
> a typical VM Server. We've replaced them all with WD Red (3TB) now.
>
> What ZFS disk setup do you have? - could you post your "zpool status"
Sure:

  pool: rpool
 state: ONLINE
  scan: resilvered 4.73M in 0h0m with 0 errors on Sat Apr 23 19:21:58
2016 <- This happened on purpose after off- and onlining sdb2
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdb2    ONLINE       0     0     0
            sda2    ONLINE       0     0     0

>
> Mine are RAID10 with a ssd for log and cache.
>
> zpool status
>   pool: tank
>  state: ONLINE
>   scan: resilvered 1.46T in 11h10m with 0 errors on Fri Apr 22
> 05:04:00 2016
> config:
>
> NAME                                                   STATE READ
> WRITE CKSUM
> tank ONLINE       0     0     0
> mirror-0                                             ONLINE 0     0     0
> ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N7XE8CXN           ONLINE 0     0     0
> ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N5EC7AUN           ONLINE 0     0     0
> mirror-1                                             ONLINE 0     0     0
> ata-WDC_WD30EFRX-68EUZN0_WD-WMC4N0D46M4N           ONLINE 0     0     0
> ata-WDC_WD30EFRX-68EUZN0_WD-WCC4N7DH176P           ONLINE 0     0     0
>         logs
> ata-Samsung_SSD_850_PRO_128GB_S24ZNSAG422885X-part1  ONLINE 0     0     0
>         cache
> ata-Samsung_SSD_850_PRO_128GB_S24ZNSAG422885X-part2  ONLINE 0     0     0
>
>
> I recommend RAID10 for performance and reduncy purposes as well. It
> has twice the write IOPS and 4 times the Read IOPS of a single disk.
> It outperforms raid 5 or 6 (RAIDZ, RAIDZ2)
>     - http://www.zfsbuild.com/2010/05/26/zfs-raid-levels/
>
> The ssd log partition speeds up writes, the cache speeds up reads
> (with time, as its get populated).
>
> Warning: To large a cache can actually kill performance as it uses up
> ZFS ARC memory.
>
> ZFS needs RAM as well, how much have you allocated to it?
There are 10GiB of RAM allocated to it, while Proxmox+three VMs consume
~4GiB and the rest is for ZFS. As dedup is deactivated and I use mirror
raid, I think this should be fair enough. CPU is 8xXeon E5405. So
anything that would consume tons of memory (RaidZ, Dedup, ...) is disabled.

Cheers
  Ralf