[PVE-User] Problems with backup process and NFS

[Fabian Grünbichler]

On Fri, May 19, 2017 at 12:49:21PM +0200, Uwe Sauter wrote:
> Am 19.05.2017 um 11:53 schrieb Fabian Grünbichler:
> > On Fri, May 19, 2017 at 11:26:35AM +0200, Uwe Sauter wrote:
> >> Hi Fabian,
> >>
> >> thanks for looking into this.
> >>
> >> As I already mentioned yesterday my NFS setup tries to use TCP as much as possible so the only UDP port used / allowed in the NFS
> >> servers firewall is udp/111 for Portmapper (to allow showmount to work).
> >>
> >>>> Issue 1:
> >>>> Backups failed tonight with "Error: mount error: mount.nfs: /mnt/pve/aurel is busy or already mounted".
> >>>>
> >>>> Question 1:
> >>>> Do NFS shares that should be used for backups need to be "disabled" so that the backup process just mount the share in preparation
> >>>> of the backup? If so, there is a problem with the WebUI as I cannot select a disabled storage location when defining or editing a
> >>>> backup job.
> >>>>
> >>>
> >>> no. I think your NFS share is (wrongly) detected as unmounted, since
> >>> activate_storage in the NFS plugin will try to mount if nfs_is_mounted
> >>> returned undef. in general activate_storage is supposed to detect if the
> >>> storage is already activated and turn into a no-op if needed, and
> >>> calling storage can treat it as idempotent.
> >>>
> >>> what does "grep aurel /proc/mounts" say on this machine?
> >>
> >> # grep aurel /proc/mounts
> >> <ip of server>:/backup/proxmox-infra /mnt/pve/aurel nfs
> >> rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=<ip of
> >> server>,mountvers=3,mountport=892,mountproto=tcp,local_lock=none,addr=141.58.1.15 0 0
> > 
> > that looks okay..
> > 
> >>
> >>>
> >>>>
> >>>> Issue 2:
> >>>> When the backup process failed no email was sent to the specified address though email notification mode is set to "always".
> >>>> Emails in general do work, at least apticron sends update notifications and sending mails from CLI also works.
> >>>>
> >>>> Question 2a:
> >>>> Why was no email sent after the backup process failed?
> >>>>
> >>>
> >>> can you post the complete log?
> >>
> >> Where can I find that?
> > 
> > check the node's task history - should have an entry for the backup job
> > including log.
> 
> Opening the task gives:
> 
> OUTPUT: TASK ERROR: mount error: mount.nfs: /mnt/pve/aurel is busy or already mounted
> 
> STATUS: stopped: mount error: mount.nfs: /mnt/pve/aurel is busy or already mounted
> 
> 
> Opening the log file in /var/log/pve/tasks for that tasks gives the same as above OUTPUT.
> 
> 
> 
> > 
> >>
> >>>
> >>>> Question 2b:
> >>>> Does the WebUI use a different method for sending emails? If so where can this be configured?
> >>>
> >>> different from what?
> >>>
> >>
> >> Different from what apticron and "echo test | mail -s test <recipient address>" use.
> >>
> > 
> > there is /root/.forward which points to pvemailforward which uses
> > "sendmail -bm -N never -f <FROM> <TO>" (using 'root' or the value from
> > datacenter.cfg as sender, and the adress of 'root at pam' as recipient),
> > and PVE::Tools::sendmail() (which is used by vzdump) which uses
> > "sendmail -B 8BITMIME -f <FROM> <TO>" (with configurable sender and
> > recipient). vzdump sets the sender to the one from datacenter.cfg as
> > well, and the recipient to what is configured for the backup job.
> > 
> 
> # cat /etc/pve/datacenter.cfg
> keyboard: de
> 
> # cat /etc/pve/user.cfg
> user:root at pam:1:0:::<my email>::
> 
> I suspect that something just doesn't send emails in that specific error case…

yes, seems like activate_storage is called very early on to retrieve
maxfiles and dumpdir via PVE::API2::VZDump (POST) -> PVE::VZDump->new()
-> PVE::VZDump::storage_info() , and that call is not guarded by an
eval, thus no error handling and sendmail is triggered. can you file a
bug for that? thanks!

the underlying issue is still unclear to me.. can you post the output of
the following snippet (insert your correct IP)

perl -e '
use strict;
use warnings;
use PVE::Storage::NFSPlugin;
my $server = "INSERTNFSSERVERIPHERE";
my $export = "/backup/proxmox-infra";
my $mountpoint = "/mnt/pve/aurel";
print PVE::Storage::NFSPlugin::nfs_is_mounted($server, $export, $mountpoint, undef), "\n";
'

if that does not output anything, the following might also be
interesting (feel free to censor as you see fit):

perl -e 'use strict; use warnings; use PVE::ProcFSTools; use Data::Dumper; print Dumper(PVE::ProcFSTools::parse_proc_mounts());'

> 
> Is there a way to test the mail configuration using PVE's mechanism?

yes, just pipe your mail into the the sendmail command I posted ;) to
check past mails, you can also use something like

journalctl -b | grep 'pvemail\|postfix'