[pve-devel] [V3 PATCH common 1/1] fix #4849: download file from url: add opt parameter for a decompression command

[Fabian Grünbichler]

On July 31, 2023 10:39 am, Philipp Hufnagl wrote:
> Signed-off-by: Philipp Hufnagl <p.hufnagl at proxmox.com>
> ---
>  src/PVE/Tools.pm | 59 +++++++++++++++++++++++++++++++-----------------
>  1 file changed, 38 insertions(+), 21 deletions(-)
> 
> diff --git a/src/PVE/Tools.pm b/src/PVE/Tools.pm
> index 9ffac12..ab97129 100644
> --- a/src/PVE/Tools.pm
> +++ b/src/PVE/Tools.pm

[snipped lots of unrelated changes here!]

> @@ -2013,10 +2013,13 @@ sub download_file_from_url {
>  	}
>      }
>  
> -    my $tmpdest = "$dest.tmp.$$";
> +    my $tmp_download = "$dest.tmp_dwnl.$$";
> +    my $tmp_decomp = "$dest.tmp.dcom.$$";

nit: the naming scheme should be unified ;)

>      eval {
>  	local $SIG{INT} = sub {
> -	    unlink $tmpdest or warn "could not cleanup temporary file: $!";
> +	    unlink $tmp_download or warn "could not cleanup temporary file: $!";
> +	    die "got interrupted by signal\n";

this die here

> +	    unlink $tmp_decomp or warn "could not cleanup temporary file: $!";

means this part never gets to run. also, this part should be guarded by
an if, since for the uncompressed case, $tmp_decomp never gets created
in the first place (see below).

>  	    die "got interrupted by signal\n";
>  	};
>  
> @@ -2029,7 +2032,7 @@ sub download_file_from_url {
>  		$ENV{https_proxy} = $opts->{https_proxy};
>  	    }
>  
> -	    my $cmd = ['wget', '--progress=dot:giga', '-O', $tmpdest, $url];
> +	    my $cmd = ['wget', '--progress=dot:giga', '-O', $tmp_download, $url];
>  
>  	    if (!($opts->{verify_certificates} // 1)) { # default to true
>  		push @$cmd, '--no-check-certificate';
> @@ -2041,7 +2044,7 @@ sub download_file_from_url {
>  	if ($checksum_algorithm) {
>  	    print "calculating checksum...";
>  
> -	    my $checksum_got = get_file_hash($checksum_algorithm, $tmpdest);
> +	    my $checksum_got = get_file_hash($checksum_algorithm, $tmp_download);
>  
>  	    if (lc($checksum_got) eq lc($checksum_expected)) {
>  		print "OK, checksum verified\n";
> @@ -2051,10 +2054,24 @@ sub download_file_from_url {
>  	    }
>  	}
>  
> -	rename($tmpdest, $dest) or die "unable to rename temporary file: $!\n";
> +    if (my $cmd = $opts->{decompression_command}) {

nit: indentation (starting) here is wrong

> +	push @$cmd, $tmp_download;
> +
> +

nit: extra blank line?

> +    my $fh;
> +    if (!open($fh, ">", "$tmp_decomp")) {


> +	die "cant open temporary file $tmp_decomp for decompresson: $!\n";
> +    }
> +	print "decompressing $tmp_download to $tmp_decomp\n";
> +	eval { run_command($cmd, output => '>&'.fileno($fh)); };
> +	my $rerr = $@;

nit: $rerr should just be $err

> +	unlink $tmp_download;
> +	die "$rerr\n" if $rerr;
> +    }
> +	rename($tmp_decomp, $dest) or die "unable to rename temporary file: $!\n";

this needs to be properly handling the no-decompression case - did you
test that? because for me it's broken:

$ pveam download ext4 rockylinux-9-default_20221109_amd64.tar.xz

downloading http://download.proxmox.com/images/system/rockylinux-9-default_20221109_amd64.tar.xz to /mnt/pve/ext4/template/cache/rockylinux-9-default_20221109_amd64.tar.xz
--2023-07-31 15:29:34--  http://download.proxmox.com/images/system/rockylinux-9-default_20221109_amd64.tar.xz
Resolving download.proxmox.com (download.proxmox.com)... 212.224.123.70, 2a01:7e0:0:424::249
Connecting to download.proxmox.com (download.proxmox.com)|212.224.123.70|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 102704656 (98M) [application/octet-stream]
Saving to: '/mnt/pve/ext4/template/cache/rockylinux-9-default_20221109_amd64.tar.xz.tmp_dwnl.665888'
     0K ........ ........ ........ ........ 32% 54.8M 1s
 32768K ........ ........ ........ ........ 65% 45.3M 1s
 65536K ........ ........ ........ ........ 98% 45.1M 0s
 98304K .                                  100% 46.8M=2.0s
2023-07-31 15:29:36 (47.9 MB/s) - '/mnt/pve/ext4/template/cache/rockylinux-9-default_20221109_amd64.tar.xz.tmp_dwnl.665888' saved [102704656/102704656]
calculating checksum...OK, checksum verified
could not cleanup temporary file: No such file or directory at /usr/share/perl5/PVE/Tools.pm line 2073.
unable to rename temporary file: No such file or directory


>      };
>      if (my $err = $@) {
> -	unlink $tmpdest or warn "could not cleanup temporary file: $!";
> +	unlink $tmp_decomp or warn "could not cleanup temporary file: $!";

same here - this should cleanup $tmp_download if uncompressed, or
$tmp_decomp if decompression is enabled..

also, pre-existing (see output above), missing newline in error message.

after the above 'pveam' invocation:

$ ls -lh /mnt/pve/ext4/template/cache
total 98M
-rw-r--r-- 1 root root  98M Nov  9  2022 rockylinux-9-default_20221109_amd64.tar.xz.tmp_dwnl.665888

>  	die $err;
>      }
>  
> -- 
> 2.39.2
> 
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 
>