Parallel VM creation/destruction issue

[Eneko Lacunza]

El 5/7/21 a las 12:36, Eneko Lacunza escribió:
> Hi all,
>
> We have split the BIG 88 node cluster in 6 clusters of 15 nodes each 
> (there where some spare servers); now things seem much better :)
>
> Sadly, we are seeing some issues when VDI management system (USD 
> Enterprise) is performing mass (in the order of 100s or even 1000s) 
> destruction and creation of VMs. In a fraction of the clone 
> operations, clone will fail with the following message:
>
> "Error: clone failed. Failed to change directory to 
> '/mnt/pve/vdi-prod1/images/103': No such file or directory at 
> /usr/share/perl5/PVE/Storage/Plugin.pm line 708."
>
> This happens when destroy for that VMID was some seconds before 
> (5s-14s for example). When another clone tries to use that VMID later 
> (as soon as 54s after destruction), it works ok.
>
> PVE version is 6.4 ISO (details below), and storage is NFS 4.2 with 
> pNFS with two pairs of NetApp servers in HA.
>
> Seems like a "race condition" is happening, where the node that is 
> cloning sees the storage directory removed by destruction late (?).
>
> I have checked "qemu-server.git/PVE/QemuServer.pm:sub destroy_vm" and 
> I see first storage disk are freed and after that VM config is 
> removed, which seems quite correct. Could it be the NFS servers that 
> are a bit "late" propagating directory removal to the client nodes?

We seem to have isolated this to the default NFS mount option 
"lookupcach/e"/, it defaults to "all". Changing to "none" seems to fix 
the issue, so it's node's local NFS cache.

UDS Enterprise has released a fix so that VMIDs don't get reused so 
early (they now secuentially use all VMID space until restarting with 
low VMIDs), that fixes the issue without disabling NFS lookupcache.

This may impact other uses of NFS storage, so I suggest a bit more 
checking should be done:

https://git.proxmox.com/?p=pve-storage.git;a=blob;f=PVE/Storage/Plugin.pm;h=26c74304aef46da195a5db1a1d266ccec7c3fb8c;hb=refs/heads/stable-6
  676 sub clone_image {
  677     my ($class, $scfg, $storeid, $volname, $vmid, $snap) = @_;
  678
  679     # this only works for file based storage types
  680     die "storage definintion has no path\n" if !$scfg->{path};
  681
  682     my ($vtype, $basename, $basevmid, undef, undef, $isBase, 
$format) =
  683         $class->parse_volname($volname);
  684
  685     die "clone_image on wrong vtype '$vtype'\n" if $vtype ne 'images';
  686
  687     die "this storage type does not support clone_image on 
snapshot\n" if $snap;
  688
  689     die "this storage type does not support clone_image on 
subvolumes\n" if $format eq 'subvol';
  690
  691     die "clone_image only works on base images\n" if !$isBase;
  692
  693     my $imagedir = $class->get_subdir($scfg, 'images');
  694     $imagedir .= "/$vmid";
  695
*696     mkpath $imagedir;*
  697
  698     my $name = $class->find_free_diskname($imagedir, $scfg, $vmid, 
"qcow2", 1);
  699
  700     warn "clone $volname: $vtype, $name, $vmid to $name 
(base=../$basevmid/$basename)\n";
  701
  702     my $newvol = "$basevmid/$basename/$vmid/$name";
  703
  704     my $path = $class->filesystem_path($scfg, $newvol);
  705
  706     # Note: we use relative paths, so we need to call chdir before 
qemu-img
  707     eval {
*708         local $CWD = $imagedir;*
  709
  710         my $cmd = ['/usr/bin/qemu-img', 'create', '-b', 
"../$basevmid/$basename",
  711                    '-f', 'qcow2', $path];
  712
  713         run_command($cmd);
  714     };
  715     my $err = $@;
  716
  717     die $err if $err;
  718
  719     return $newvol;
  720 }

I'm not a PERL coder but maybe doing "local $CWD = $imagedir" before 
mkpath, or some more explicit test before mkpath that will make it 
create the dir, instead of believing data in lookupcache.

Cheers,

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/