[PVE-User] Local XFS storage pool - Fragmentation issues
Jean R. Franco
jfranco at maila.net.br
Wed Oct 18 14:26:55 CEST 2017
Take a look /proc/slabinfo to see if xfs_inode is using all your memory.
Please post your pve version with:
----- Mensagem original -----
De: "Markos Vakondios" <mvakondios at gmail.com>
Para: "PVE User List" <pve-user at pve.proxmox.com>
Enviadas: Quarta-feira, 18 de outubro de 2017 10:03:04
Assunto: [PVE-User] Local XFS storage pool - Fragmentation issues
My use case, is to provide a large hardware-RAID backed volume (hence no
ZFS) to PVE for continuous decent performance writes by a running container
I have 6 x 6TB SATA drives on a hardware RAID 10 configuration.
It 's formated as a ~10TB XFS device:
/dev/sdb on /10TB type xfs (rw,relatime,attr2,inode64,noquota)
I created a local directory storage pool on my PVE host and assigned a
9.8TB mount point to a running container (DVR)
pve:/10TB/images/101# ls -lah /10TB/images/101/
drwxr----- 2 root root 30 Feb 15 2017 .
drwxr-xr-x 4 root root 38 Feb 15 2017 ..
-rw-r----- 1 root root 9.8T Oct 18 14:03 vm-101-disk-1.raw
After a couple of months, I started receiving the following XFS errors on
Oct 17 06:26:09 pve kernel: [5629844.259800] XFS: loop0(28500) possible
memory allocation deadlock size 72832 in kmem_alloc (mode:0x2400240)
leaving me with an unusable filesystem from inside the container (no
writes), unless I drop the host's page cache:
echo 1 > /proc/sys/vm/drop_caches
or even need to drop all pagecache, dentries and inodes:
echo 3 > /proc/sys/vm/drop_caches (remember to sync first!)
After research, I concluded that the deadlock is caused from heavy
fragmentation on the XFS image file which has 1.2 million extends (!)
according to xfs_bmap.
Of course, there is no way to defrag this filesystem with xfs_fsr, as the
volume file occupies almost 100% of its space (yes, I won't use more than
80%). Also it wouldn't be practical to do so, as the volume is busy 24/7
(cctv recording) and the available capacity matters.
Please correct me if I am wrong, but the above seemed to be a reasonably
decent layout to employ a local XFS filesystem (which is supposed to have
better handling of large files than EXT4) under PVE. But in practice it's
not! Am I missing something?
Is LVM a more suitable solution?
Any ideas or suggestions would be greatly appreciated.
pve-user mailing list
pve-user at pve.proxmox.com
More information about the pve-user