[PVE-User] Local XFS storage pool - Fragmentation issues
mvakondios at gmail.com
Wed Oct 18 14:03:04 CEST 2017
My use case, is to provide a large hardware-RAID backed volume (hence no
ZFS) to PVE for continuous decent performance writes by a running container
I have 6 x 6TB SATA drives on a hardware RAID 10 configuration.
It 's formated as a ~10TB XFS device:
/dev/sdb on /10TB type xfs (rw,relatime,attr2,inode64,noquota)
I created a local directory storage pool on my PVE host and assigned a
9.8TB mount point to a running container (DVR)
pve:/10TB/images/101# ls -lah /10TB/images/101/
drwxr----- 2 root root 30 Feb 15 2017 .
drwxr-xr-x 4 root root 38 Feb 15 2017 ..
-rw-r----- 1 root root 9.8T Oct 18 14:03 vm-101-disk-1.raw
After a couple of months, I started receiving the following XFS errors on
Oct 17 06:26:09 pve kernel: [5629844.259800] XFS: loop0(28500) possible
memory allocation deadlock size 72832 in kmem_alloc (mode:0x2400240)
leaving me with an unusable filesystem from inside the container (no
writes), unless I drop the host's page cache:
echo 1 > /proc/sys/vm/drop_caches
or even need to drop all pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches (remember to sync first!)
After research, I concluded that the deadlock is caused from heavy
fragmentation on the XFS image file which has 1.2 million extends (!)
according to xfs_bmap.
Of course, there is no way to defrag this filesystem with xfs_fsr, as the
volume file occupies almost 100% of its space (yes, I won't use more than
80%). Also it wouldn't be practical to do so, as the volume is busy 24/7
(cctv recording) and the available capacity matters.
Please correct me if I am wrong, but the above seemed to be a reasonably
decent layout to employ a local XFS filesystem (which is supposed to have
better handling of large files than EXT4) under PVE. But in practice it's
not! Am I missing something?
Is LVM a more suitable solution?
Any ideas or suggestions would be greatly appreciated.
More information about the pve-user