Parallel VM creation/destruction issue

Eneko Lacunza elacunza at
Wed Jul 7 15:44:14 CEST 2021

El 5/7/21 a las 12:36, Eneko Lacunza escribió:
> Hi all,
> We have split the BIG 88 node cluster in 6 clusters of 15 nodes each 
> (there where some spare servers); now things seem much better :)
> Sadly, we are seeing some issues when VDI management system (USD 
> Enterprise) is performing mass (in the order of 100s or even 1000s) 
> destruction and creation of VMs. In a fraction of the clone 
> operations, clone will fail with the following message:
> "Error: clone failed. Failed to change directory to 
> '/mnt/pve/vdi-prod1/images/103': No such file or directory at 
> /usr/share/perl5/PVE/Storage/ line 708."
> This happens when destroy for that VMID was some seconds before 
> (5s-14s for example). When another clone tries to use that VMID later 
> (as soon as 54s after destruction), it works ok.
> PVE version is 6.4 ISO (details below), and storage is NFS 4.2 with 
> pNFS with two pairs of NetApp servers in HA.
> Seems like a "race condition" is happening, where the node that is 
> cloning sees the storage directory removed by destruction late (?).
> I have checked "qemu-server.git/PVE/ destroy_vm" and 
> I see first storage disk are freed and after that VM config is 
> removed, which seems quite correct. Could it be the NFS servers that 
> are a bit "late" propagating directory removal to the client nodes?

We seem to have isolated this to the default NFS mount option 
"lookupcach/e"/, it defaults to "all". Changing to "none" seems to fix 
the issue, so it's node's local NFS cache.

UDS Enterprise has released a fix so that VMIDs don't get reused so 
early (they now secuentially use all VMID space until restarting with 
low VMIDs), that fixes the issue without disabling NFS lookupcache.

This may impact other uses of NFS storage, so I suggest a bit more 
checking should be done:;a=blob;f=PVE/Storage/;h=26c74304aef46da195a5db1a1d266ccec7c3fb8c;hb=refs/heads/stable-6
  676 sub clone_image {
  677     my ($class, $scfg, $storeid, $volname, $vmid, $snap) = @_;
  679     # this only works for file based storage types
  680     die "storage definintion has no path\n" if !$scfg->{path};
  682     my ($vtype, $basename, $basevmid, undef, undef, $isBase, 
$format) =
  683         $class->parse_volname($volname);
  685     die "clone_image on wrong vtype '$vtype'\n" if $vtype ne 'images';
  687     die "this storage type does not support clone_image on 
snapshot\n" if $snap;
  689     die "this storage type does not support clone_image on 
subvolumes\n" if $format eq 'subvol';
  691     die "clone_image only works on base images\n" if !$isBase;
  693     my $imagedir = $class->get_subdir($scfg, 'images');
  694     $imagedir .= "/$vmid";
*696     mkpath $imagedir;*
  698     my $name = $class->find_free_diskname($imagedir, $scfg, $vmid, 
"qcow2", 1);
  700     warn "clone $volname: $vtype, $name, $vmid to $name 
  702     my $newvol = "$basevmid/$basename/$vmid/$name";
  704     my $path = $class->filesystem_path($scfg, $newvol);
  706     # Note: we use relative paths, so we need to call chdir before 
  707     eval {
*708         local $CWD = $imagedir;*
  710         my $cmd = ['/usr/bin/qemu-img', 'create', '-b', 
  711                    '-f', 'qcow2', $path];
  713         run_command($cmd);
  714     };
  715     my $err = $@;
  717     die $err if $err;
  719     return $newvol;
  720 }

I'm not a PERL coder but maybe doing "local $CWD = $imagedir" before 
mkpath, or some more explicit test before mkpath that will make it 
create the dir, instead of believing data in lookupcache.


Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 |
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

More information about the pve-user mailing list