[pve-devel] Bug 1458 - PVE 5 live migration downtime degraded to several seconds (compared to PVE 4)

[Alexandre DERUMIER]

Thanks for the explain Fabian. (I'm always using migration insecure, so I didn't notice this bug)

>>when live-migrating over a unix socket, PVE 5 takes up to a few seconds 
>>between completing the RAM transfer and pausing the source VM, and 
>>resuming the target VM. in PVE 4, the same migration has a downtime of 
>>almost 0. 

few seconds seem so huge ... (user talk about 4s)....

>>AFAICT, the reason for this is a bug fix in PVE 5's qemu-server which 
>>was required to support storage live migration in Qemu 2.9. 

any commit reference ?


>>originally in PVE 4, the target VM in a live migration was started in 
>>incoming migration mode and NOT continued on startup (whereas VMs rolled 
>>back to a RAM snapshot where started in the same mode, but immediately 
>>continued). 

>>in June 2016[3], migration over ssh-forwarded unix sockets was 
>>implemented. the check for skipping the continue command on startup of 
>>the target VM was overlooked, so now VMs migrated over unix sockets were 
>>started in incoming migration mode, but continued on startup. 

But this seem to be a bug, fixed later here ?

https://git.proxmox.com/?p=qemu-server.git;a=commit;h=b37ecfe6ae7f7b557db7712ee6988cb0397306e9


>>I wonder whether going the "immediately cont" route for live migrations 
>>without local storage can cause any issues besides the obvious "moving 
>>the conf file failed and VM is now active on the wrong node" one? 

I don't known if it could be great to have some kind of temporary conf file where a kvm process is running.
(here we could see vm on source host with state running, and vm on source target with state migrating for example).
Like this if something bad happen at the end of migration, user could still stop the target kvm process with gui.

But maybe it's too complex to implement, don't known...


>>if not, I propose doing just that. otherwise, we could think about lowering 
>>the polling interval when waiting for RAM migration to complete (in 
>>phase2) - that should shave off a bit of the downtime as well. 

I wonder where exactly it take so much time..
$downtime seem to be low, but as it's coming from status, maybe are we missing some query migrate .
Also I think we already try to lowering usleep at the end
#reduce sleep if remainig memory if lower than the everage transfert 
$usleep = 300000 if $avglstat && $rem < $avglstat;

maybe this don't work correctly ?

I think a proper way could be catch qemu events, instead pooling status. (but require maybe lot of work)



----- Mail original -----
De: "Fabian Grünbichler" <f.gruenbichler at proxmox.com>
À: "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Jeudi 27 Juillet 2017 14:45:43
Objet: [pve-devel] Bug 1458 - PVE 5 live migration downtime degraded to several seconds (compared to PVE 4)

the following issue was reported on the forum[1] and as bug #1458[2], 
moving this here for further discussion of potential fixes. 

when live-migrating over a unix socket, PVE 5 takes up to a few seconds 
between completing the RAM transfer and pausing the source VM, and 
resuming the target VM. in PVE 4, the same migration has a downtime of 
almost 0. 

AFAICT, the reason for this is a bug fix in PVE 5's qemu-server which 
was required to support storage live migration in Qemu 2.9. 

originally in PVE 4, the target VM in a live migration was started in 
incoming migration mode and NOT continued on startup (whereas VMs rolled 
back to a RAM snapshot where started in the same mode, but immediately 
continued). 

in June 2016[3], migration over ssh-forwarded unix sockets was 
implemented. the check for skipping the continue command on startup of 
the target VM was overlooked, so now VMs migrated over unix sockets were 
started in incoming migration mode, but continued on startup. this does 
not change the behaviour on startup, as a VM in incoming migration mode 
is not actually running until a migration has happened. this does mean 
that the downtime is vastly reduced for such migrations, as Qemu will 
continue the target VM automatically as soon as the migration job is 
completed. 

the only things that happen after this automatic resume is 
- finish tunnel 
- moving the conf file logically between nodes 
- resuming on the target side (which is a no-op in this case) 

so the risk for inconsistencies seems pretty small. 

later on, we introduced live-storage migration. in those cases, we now 
have the following scenario: 
- start storage migration jobs 
- start RAM migration 
- wait for RAM to be completed 
- finish tunnel 
- finish block jobs 
- update conf file 
- move the conf file logically between the nodes 
- resume on target node 

so depending on whether the migration goes over tcp (OK) or over unix 
(not so much) we have very different behaviour and risk for 
inconsistencies. 

with the introduction to PVE 5, this different behaviour was fixed / 
made consistent, by adapting the "manual resume" stance. this was needed 
because Qemu 2.9 does not allow the storage migration over NBD and the 
target VM itself to have write access to the same disks at the same 
time. this fix was not backported to PVE 4, which means that storage 
live-migration is potentially buggy there, but live-migration over unix 
sockets is faster. 

I wonder whether going the "immediately cont" route for live migrations 
without local storage can cause any issues besides the obvious "moving 
the conf file failed and VM is now active on the wrong node" one? if 
not, I propose doing just that. otherwise, we could think about lowering 
the polling interval when waiting for RAM migration to complete (in 
phase2) - that should shave off a bit of the downtime as well. 

in any case, I think we need to backport the manual resume in case of 
local storage live migration fix to PVE 4. 

1: https://forum.proxmox.com/threads/pve-5-live-migration-downtime-degradation-2-4-sec.35890 
2: https://bugzilla.proxmox.com/show_bug.cgi?id=1458 
3: 1c9d54bfd05e0d017a6e2ac5524d75466b1a4455 

_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel