[pve-devel] [PATCH common] allow longer timeout for cancelling 'vzdump' jobs

[Stefan Reiter]

On 26/01/2021 19:23, Thomas Lamprecht wrote:
> On 14.01.21 16:39, Stefan Reiter wrote:
>> This attempts to solve the issue where on slow network storages,
>> aborting a backup job (which may wait for buffers to flush) could take
>> longer than 5 seconds, and would thus result in the task being killed by
>> SIGKILL, not removing the backup lock in the process.
>>
>> Make the implementation future-proof by using a map from task type to a
>> timeout value. Default stays at 5, so tasks other than 'vzdump' are not
>> affected.
>>
>> Signed-off-by: Stefan Reiter <s.reiter at proxmox.com>
>> ---
>>   src/PVE/RESTEnvironment.pm | 16 ++++++++++++----
>>   1 file changed, 12 insertions(+), 4 deletions(-)
>>
> 
> Not to sure about that map there in pve-common, that module should stay rather
> agnostic of user special treatment.
> 
> Did you thought about passing that explicitly on worker creation, or setting it
> in the RPCEnv inside a worker?

I generally agree that it's a bit misplaced, but I don't see a way to 
encode it in the worker - the only info we have in check_worker and 
stop_task is the UPID, and I don't think it makes sense to encode a 
timeout in that? Or is there a way I'm not seeing to retrieve additional 
info about a worker from the UPID alone?

We could at least put the map in pve-manager, but I'm not sure if that's 
any better.

> 
>> diff --git a/src/PVE/RESTEnvironment.pm b/src/PVE/RESTEnvironment.pm
>> index d5b84d0..8a0cb9a 100644
>> --- a/src/PVE/RESTEnvironment.pm
>> +++ b/src/PVE/RESTEnvironment.pm
>> @@ -365,8 +365,16 @@ sub active_workers  {
>>       return $res;
>>   }
>>   
>> +my $timeout_map = {
>> +    # backup cancellation on slow target storages might take a while, avoid
>> +    # leaving the VM in locked state
>> +    "vzdump" => 60,
>> +};
>> +
>>   my $kill_process_group = sub {
>> -    my ($pid, $pstart) = @_;
>> +    my ($pid, $pstart, $timeout) = @_;
>> +
>> +    $timeout //= 5;
>>   
>>       # send kill to process group (negative pid)
>>       my $kpid = -$pid;
>> @@ -374,8 +382,7 @@ my $kill_process_group = sub {
>>       # always send signal to all pgrp members
>>       kill(15, $kpid); # send TERM signal
>>   
>> -    # give max 5 seconds to shut down
>> -    for (my $i = 0; $i < 5; $i++) {
>> +    for (my $i = 0; $i < $timeout; $i++) {
>>   	return if !PVE::ProcFSTools::check_process_running($pid, $pstart);
>>   	sleep (1);
>>       }
>> @@ -394,7 +401,8 @@ sub check_worker {
>>       return 0 if !$running;
>>   
>>       if ($killit) {
>> -	&$kill_process_group($task->{pid});
>> +	my $type = $task->{type};
>> +	&$kill_process_group($task->{pid}, undef, $timeout_map->{$type});
>>   	return 0;
>>       }
>>   
>>
> 
>