[pve-devel] Serious problems with the PVE Cluster

[Eric Blevins]

What you describe does not seem like a Proxmox specific problem to me.

You turned off the NFS server without dismounting the volumes from the 
nodes.
This causes IO to those NFS volumes to stall.

Proxmox does periodically check the backup directories, so there is 
consistent IO to them.

Since that IO cannot complete, it causes processes to hang.
I've even seen Linux not perform IO to local disks when IO to NFS is 
stalled for a period of time.
I am sure you can envision the horrible problems this can cause.

Using NFS soft mount might help prevent this problem but that can also 
cause corrupted data.

My suggestion to help avoid this is to use a vzdump hook script to mount 
the NFS volume only when performing a backup then dismount it at 
completion of backup. Better yet, setup HA NFS.




On 01/29/2014 10:33 PM, Cesar Peschiera wrote:
> Serious problems with the PVE Cluster
> ----------------------------------------
>
> @any developer that can help in the code:
>
> I had problems with 2 of 5 PVE Hosts in a PVE cluster when the NFS 
> Backup Server  was shutdown manually (without  that "KVM Live Backup" 
> is running in the PVE Hosts),
>
> The symptom was:
> 2 PVE Nodes were disconnected suddenly of PVE Cluster
> The PVE GUI shows leds in red for the nodes without connection to PVE 
> Cluster
>
> To return to normal operation:
> Only was necessary start the NFS Backup Server
>
> I mean two stuff:
> 1- If "KVM Live Backup" is running in the PVE hosts while that NFS 
> Backup Server is shutdown suddenly, the problem would have been more 
> serious.
> 2- The PVE Cluster not must depend of NFS Backup Server to run 
> correctly, this situation is "VERY SERIOUS"
>
> For these reasons i think it will be necessary to correct the code of 
> PVE Cluster
>
> Awaiting a answer, i say see you soon
>
> Best regards
> Cesar
>
>
> ----- Part of Original Message ----- From: "Alexandre DERUMIER" 
> <aderumier at odiso.com>
> To: "Cesar Peschiera" <brain at click.com.py>
> Cc: <pve-devel at pve.proxmox.com>
> Sent: Wednesday, January 29, 2014 2:31 AM
> Subject: Re: [pve-devel] KVM Live Backup performance
>
>
>>> And the fifth question:
>>> What will happen if this NFS Server suddenly decomposes while "KVM Live
>>> Backup" is running?
>
> mmm, good question....I don't known what happen when backup job is 
> hanging because of unavailable storage...
>
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel