[PVE-User] Disk full errors but plenty of free space reported (KVM io-error status)

[Francesco Ongaro]

Dear pve-users,

lately on a single node PVE server I'm experiencing what could seem
a full disk error. The issue is that the machine had ~866GB free and
plenty of unused inodes but things were definitely not okay.

Is this something that you ever experienced on your proxmox setup?

For whom who want the full details, here they are :) My mountpoint,
let's call it FOO, is a 1.8T ext3 filesystem, hosting some openvz and
KVM machines, backups, iso, etc. Proxmox version is 3.1-21 on a Debian
Wheezy.

The issue manifests in two ways:

- Backups errors both for KVM (ERROR: vma_co_write write error - Broken
pipe) and OpenVZ (rsync says "No space left on device (28)" and lzop
"No space left on device: <stdout>").

- About two weeks later some KVM machines shown io-error as status, I
was able to reproduce it by starting a defrag on a Windows vm. All the
other vm, regardless their OS, freeze and show "io-error", when writing
to disk.

My mountpoint is pretty standard:

/dev/mapper/FOO on /FOO type ext3
(rw,relatime,errors=continue,barrier=0,data=ordered)

And the disk is half full:

# df -h
Filesystem Size Used Avail Use% Mounted on
udev 10M 0 10M 0% /dev
tmpfs 2.4G 444K 2.4G 1% /run
/dev/mapper/pve-root 95G 36G 55G 40% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 4.8G 9.2M 4.7G 1% /run/shm
/dev/mapper/pve-data 337G 8.2G 329G 3% /var/lib/vz
/dev/sda1 495M 81M 389M 18% /boot
/dev/fuse 30M 32K 30M 1% /etc/pve
/dev/mapper/FOO 1.8T 876G 866G 51% /FOO

I read about others who had seemingly similar issues but disk space was
correctly reported as full:

http://pve.proxmox.com/pipermail/pve-user/2013-July/005863.html

http://pve.proxmox.com/pipermail/pve-user/2014-December/008051.html

Any ideas on how to debug this issue?

Thanks,
Francesco Ongaro
-------------- next part --------------
A non-text attachment was scrubbed...
Name: francesco_ongaro.vcf
Type: text/x-vcard
Size: 375 bytes
Desc: not available
URL: <http://lists.proxmox.com/pipermail/pve-user/attachments/20150827/e75b99bf/attachment.vcf>