[pve-devel] [PVE-User] bug with vzdump very dangerous !!!!!

Mon Mar 16 08:33:55 CET 2015

I have tested with proxmox 3.4, (pve-qemu-kvm: 2.1-12) 

I don't have crash when backup storage is full (local backup storage)

"
Stop
INFO: starting new backup job: vzdump 116 --remove 0 --mode snapshot --compress lzo --storage backup --node kvmtest1
INFO: Starting Backup of VM 116 (qemu)
INFO: status = running
INFO: update VM 116: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating archive '/backup/dump/vzdump-qemu-116-2015_03_16-08_28_41.vma.lzo'
INFO: started backup task '1409d3a4-2637-40ac-8aa6-4c46ee230eb1'
INFO: status: 0% (0/53687091200), sparse 0% (0), duration 3, 0/0 MB/s
lzop: No space left on device: <stdout>
INFO: status: 0% (228065280/53687091200), sparse 0% (93093888), duration 29, 8/5 MB/s
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
ERROR: Backup of VM 116 failed - vma_queue_write: write error - Broken pipe
INFO: Backup job finished with errors
TASK ERROR: job errors
"

I don't known if you can test with more recent pve-qemu-kvm ?

----- Mail original -----
De: "Jean-Laurent Ivars" <jl.ivars at ipgenius.fr>
À: "aderumier" <aderumier at odiso.com>
Cc: "proxmoxve" <pve-user at pve.proxmox.com>, "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Vendredi 13 Mars 2015 14:15:08
Objet: Re: [PVE-User] bug with vzdump very dangerous !!!!!

Thank you for your answer :) 

A check before backup could be a good improvement of the backup system but moreover it would be much important that the vzdump process don’t shutdown the virtual machines even with no left space… don’t you think so ? 

Jean-Laurent Ivars 
Responsable Technique | Technical Manager 
22, rue Robert - 13007 Marseille 
Mobile: 06.52.60.86.47 - Tel: 09 84 56 64 30 - Fax: 09 89 56 64 30 
Linkedin | Viadeo | www.ipgenius.fr 

Le 13 mars 2015 à 13:58, Alexandre DERUMIER < aderumier at odiso.com > a écrit : 

I think we could at least add a check for free space on backup storage. 
(maybe something like : minimum free space = total size of disks vm.). 
We can't known how much compression will be done. 

But of course this bug should be fixed. 

----- Mail original ----- 
De: "Jean-Laurent Ivars" < jl.ivars at ipgenius.fr > 
À: "proxmoxve" < pve-user at pve.proxmox.com > 
Envoyé: Vendredi 13 Mars 2015 12:16:36 
Objet: [PVE-User] bug with vzdump very dangerous !!!!! 

Hi all, 

I am an happy user of Proxmox since a few years but tonight i had a very serious issue : 
I ran out of space on my nfs backup storage and guess what happened ? Of course the backup could not run but the problem is that ALL MY KVM MACHINES HAVE BEEN STOPPED BY THE VZDUMP PROCESS !!! (and of course i have not been warned because my supervision machine was on the same cluster… stupid me) 

This is a very dangerous bug, at the beginning I did not understand what happened but i found this thread where someone had the very same issue : 
http://forum.proxmox.com/threads/21040-VZDUMP-backup-causes-KVM-QEMU-VM-to-shutdown-if-backup-storage-is-full 

I don’t understand why nobody answered him on the forum, in my opinion this is a very serious bug and should be repaired as soon as possible ! 

I give you some technical info (ask me if you need more) 

first precision : I don’t know if it’s relevant but my backup are in snapshot mode 

root @ vha ~ # pveversion -v 
proxmox-ve-2.6.32: 3.3-139 (running kernel: 2.6.32-34-pve) 
pve-manager: 3.3-5 (running version: 3.3-5/bfebec03) 
pve-kernel-2.6.32-33-pve: 2.6.32-138 
pve-kernel-2.6.32-29-pve: 2.6.32-126 
pve-kernel-2.6.32-34-pve: 2.6.32-140 
lvm2: 2.02.98-pve4 
clvm: 2.02.98-pve4 
corosync-pve: 1.4.7-1 
openais-pve: 1.1.4-3 
libqb0: 0.11.1-2 
redhat-cluster-pve: 3.2.0-2 
resource-agents-pve: 3.9.2-4 
fence-agents-pve: 4.0.10-1 
pve-cluster: 3.0-15 
qemu-server: 3.3-3 
pve-firmware: 1.1-3 
libpve-common-perl: 3.0-19 
libpve-access-control: 3.0-15 
libpve-storage-perl: 3.0-25 
pve-libspice-server1: 0.12.4-3 
vncterm: 1.1-8 
vzctl: 4.0-1pve6 
vzprocps: 2.0.11-2 
vzquota: 3.1-2 
pve-qemu-kvm: 2.1-10 
ksm-control-daemon: 1.1-1 
glusterfs-client: 3.5.2-1 
root @ vha ~ # 

exemple of backup log : 
101: mars 12 22:00:03 INFO: Starting Backup of VM 101 (qemu) 
101: mars 12 22:00:03 INFO: status = running 
101: mars 12 22:00:03 INFO: update VM 101: -lock backup 
101: mars 12 22:00:04 INFO: backup mode: snapshot 
101: mars 12 22:00:04 INFO: ionice priority: 7 
101: mars 12 22:00:05 INFO: creating archive '/mnt/pve/stock/dump/vzdump-qemu-101-2015_03_12-22_00_03.vma.lzo' 
101: mars 12 22:00:05 ERROR: client closed connection 101: mars 12 22:00:05 INFO: aborting backup job 
101: mars 12 22:00:05 ERROR: VM 101 not running 101: mars 12 22:00:06 ERROR: Backup of VM 101 failed - client closed connection 

Now that i’m aware of this bug I will be more careful and won’t been catch again but i think it’s something that everyone should be warned about ! 

Best regards, 

Jean-Laurent Ivars 
Responsable Technique | Technical Manager 
22, rue Robert - 13007 Marseille 
Mobile: 06.52.60.86.47 - Tel: 09 84 56 64 30 - Fax: 09 89 56 64 30 
Linkedin | Viadeo | www.ipgenius.fr 

_______________________________________________ 
pve-user mailing list 
pve-user at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user 