[PVE-User] Live Migration Problem

Fri Jul 15 15:32:04 CEST 2016

this seems to be an qemu bug

see here:
https://forum.proxmox.com/threads/vm-freezeing-with-vcpu-at-100-when-doing-livemigration.25987/

and here:
http://lists.gnu.org/archive/html/qemu-discuss/2014-02/msg00002.html

On 07/15/2016 02:34 PM, Kilian Ries wrote:
> Hi,
>
> today i tested different virtual CPU configurations for the KVM, for example:
>
> - qemu64
> - kvm64
> - 1 core / 1 cpu
> - NUMA active / deactivated
>
> but everytime i migrate from Host1 -> Host2 the KVM freezes. Migration from Host2 -> Host1 works without any problem.
>
> The Host-CPU configuration is:
>
> Host 1:
> 2x AMD Opteron(tm) Processor 4234
>
> Host 2:
> 2x AMD Opteron(tm) Processor 4184
>
>
>
> From the Proxmox Wiki:
>
> "in order to guarantee migration between physical hosts does not result in non-functioning virtual machines, QEMU & KVM disable the guest's ability to directly access some of the features which may be exclusive to the host CPU."
>
> https://pve.proxmox.com/wiki/Allow_Guests_Access_to_Host_CPU
>
> I did a quick comparison of the CPU settings / feature flags on Host1 and Host2 and both were the same (inside the KVM; running with kvm64 CPU type).
>
>
> Does anybody have an idea whats the problem?
>
> ________________________________________
> Von: pve-user <pve-user-bounces at pve.proxmox.com> im Auftrag von Kilian Ries <mail at kilian-ries.de>
> Gesendet: Donnerstag, 14. Juli 2016 16:40
> An: pve-user at pve.proxmox.com
> Betreff: Re: [PVE-User] Live Migration Problem
>
> Thanks for the hint, there was an error with the sources.list. Now i'm really on the latest version:
>
> pveversion -v
>
> proxmox-ve: 4.2-56 (running kernel: 4.4.13-1-pve)
> pve-manager: 4.2-15 (running version: 4.2-15/6669ad2c)
> pve-kernel-4.4.6-1-pve: 4.4.6-48
> pve-kernel-4.4.13-1-pve: 4.4.13-56
> lvm2: 2.02.116-pve2
> corosync-pve: 2.3.5-2
> libqb0: 1.0-1
> pve-cluster: 4.0-42
> qemu-server: 4.0-83
> pve-firmware: 1.1-8
> libpve-common-perl: 4.0-70
> libpve-access-control: 4.0-16
> libpve-storage-perl: 4.0-55
> pve-libspice-server1: 0.12.5-2
> vncterm: 1.2-1
> pve-qemu-kvm: 2.5-19
> pve-container: 1.0-70
> pve-firewall: 2.0-29
> pve-ha-manager: 1.0-32
> ksm-control-daemon: 1.2-1
> glusterfs-client: 3.5.2-2+deb8u2
> lxc-pve: 1.1.5-7
> lxcfs: 2.0.0-pve2
> cgmanager: 0.39-pve1
> criu: 1.6.0-1
> zfsutils: 0.6.5.7-pve10~bpo80
>
>
>
>
> However, after live-migration from host1 to host2 the VM is still frozen:
>
> ###
> Jul 14 16:37:54 starting migration of VM 104 to node 'proxmox2' (192.168.100.253)
> Jul 14 16:37:54 copying disk images
> Jul 14 16:37:55 starting VM 104 on remote node 'proxmox2'
> Jul 14 16:37:57 start remote tunnel
> Jul 14 16:37:58 starting online/live migration on unix:/run/qemu-server/104.migrate
> Jul 14 16:37:58 migrate_set_speed: 8589934592
> Jul 14 16:37:58 migrate_set_downtime: 0.1
> Jul 14 16:37:58 set migration_caps
> Jul 14 16:37:58 set cachesize: 53687091
> Jul 14 16:37:58 start migrate command to unix:/run/qemu-server/104.migrate
> Jul 14 16:38:00 migration status: active (transferred 119489478, remaining 113209344), total 546119680)
> Jul 14 16:38:00 migration xbzrle cachesize: 33554432 transferred 0 pages 0 cachemiss 0 overflow 0
> Jul 14 16:38:02 migration speed: 128.00 MB/s - downtime 92 ms
> Jul 14 16:38:02 migration status: completed
> Jul 14 16:38:05 migration finished successfully (duration 00:00:11)
> TASK OK
> ###
>
> From host2 to host1 everything is fine ...
>
> ________________________________________
> Von: pve-user <pve-user-bounces at pve.proxmox.com> im Auftrag von Dominik Csapak <d.csapak at proxmox.com>
> Gesendet: Donnerstag, 14. Juli 2016 15:37
> An: pve-user at pve.proxmox.com
> Betreff: Re: [PVE-User] Live Migration Problem
>
> On 07/14/2016 02:05 PM, Kilian Ries wrote:
>> Both systems are up to date (apt-get dist-upgrade doesn't show me any package to upgrade).
>>
>> pveversion -v
>>
>> proxmox-ve: 4.2-48 (running kernel: 4.4.6-1-pve)
>> pve-manager: 4.2-2 (running version: 4.2-2/725d76f0)
>> pve-kernel-4.4.6-1-pve: 4.4.6-48
>> lvm2: 2.02.116-pve2
>> corosync-pve: 2.3.5-2
>> libqb0: 1.0-1
>> pve-cluster: 4.0-39
>> qemu-server: 4.0-72
>> pve-firmware: 1.1-8
>> libpve-common-perl: 4.0-59
>> libpve-access-control: 4.0-16
>> libpve-storage-perl: 4.0-50
>> pve-libspice-server1: 0.12.5-2
>> vncterm: 1.2-1
>> pve-qemu-kvm: 2.5-14
>> pve-container: 1.0-62
>> pve-firewall: 2.0-25
>> pve-ha-manager: 1.0-28
>> ksm-control-daemon: 1.2-1
>> glusterfs-client: 3.5.2-2+deb8u2
>> lxc-pve: 1.1.5-7
>> lxcfs: 2.0.0-pve2
>> cgmanager: 0.39-pve1
>> criu: 1.6.0-1
>> zfsutils: 0.6.5-pve9~jessie
>
> you probably have an error in your repositoriy configuration,
> see https://pve.proxmox.com/wiki/Package_repositories
>
>
>>
>> ________________________________________
>> Von: pve-user <pve-user-bounces at pve.proxmox.com> im Auftrag von Thomas Lamprecht <t.lamprecht at proxmox.com>
>> Gesendet: Donnerstag, 14. Juli 2016 13:51
>> An: pve-user at pve.proxmox.com
>> Betreff: Re: [PVE-User] Live Migration Problem
>>
>> Hi
>>
>>
>> On 07/14/2016 12:51 PM, Kilian Ries wrote:
>>> Just tested it, ssh works in both directions.
>>>
>>> As additional information here is the migration output from proxmox:
>>>
>>> ###
>>> Jul 14 12:46:03 starting migration of VM 101 to node 'proxmox2' (192.168.100.253)
>>> Jul 14 12:46:03 copying disk images
>>> Jul 14 12:46:04 starting VM 101 on remote node 'proxmox2'
>>> Jul 14 12:46:05 starting ssh migration tunnel
>>> Jul 14 12:46:06 starting online/live migration on localhost:60000
>>
>> can you do an update, via:
>>
>> apt-get update
>> apt-get dist-upgrade
>>
>> it seems that you use an older package versions than available in the repos.
>> We use UNIX sockets for securely forwarding the migration to the other
>> node, your log shows that it uses TCP ones..
>>
>> And ensure that both nodes are on the same versions, else you may get
>> problems when migrating from new to old..
>> Old to new works.
>>
>> cheers
>>
>>> Jul 14 12:46:06 migrate_set_speed: 8589934592
>>> Jul 14 12:46:06 migrate_set_downtime: 0.1
>>> Jul 14 12:46:08 migration status: active (transferred 127103147, remaining 262205440), total 2156732416)
>>> Jul 14 12:46:08 migration xbzrle cachesize: 134217728 transferred 0 pages 0 cachemiss 0 overflow 0
>>> Jul 14 12:46:10 migration status: active (transferred 248201655, remaining 123899904), total 2156732416)
>>> Jul 14 12:46:10 migration xbzrle cachesize: 134217728 transferred 0 pages 0 cachemiss 0 overflow 0
>>> Jul 14 12:46:12 migration speed: 341.33 MB/s - downtime 54 ms
>>> Jul 14 12:46:12 migration status: completed
>>> Jul 14 12:46:16 migration finished successfully (duration 00:00:13)
>>> TASK OK
>>> ###
>>>
>>> ________________________________________
>>> Von: pve-user <pve-user-bounces at pve.proxmox.com> im Auftrag von Jean-Laurent Ivars <jl.ivars at ipgenius.fr>
>>> Gesendet: Donnerstag, 14. Juli 2016 12:07
>>> An: PVE User List
>>> Betreff: Re: [PVE-User] Live Migration Problem
>>>
>>> hello
>>>
>>> maybe just a silly idea, but did you tried to ssh from host1 -> Host2 maybe it’s just a knowhost issue…
>>>
>>> regards
>>>
>>>
>>> Jean-Laurent Ivars
>>> Responsable Technique | Technical Manager
>>> 22, rue Robert - 13007 Marseille
>>> Tel: 09 84 56 64 30 - Mobile: 06.52.60.86.47
>>> Linkedin <http://fr.linkedin.com/in/jlivars/>   |  Viadeo <http://www.viadeo.com/fr/profile/jean-laurent.ivars>   |  www.ipgenius.fr <https://www.ipgenius.fr/>
>>>> Le 14 juil. 2016 à 12:00, Kilian Ries <mail at kilian-ries.de> a écrit :
>>>>
>>>> Hi,
>>>>
>>>>
>>>> just installed a two-node proxmox 4.2 cluster:
>>>>
>>>>
>>>>
>>>> ###
>>>>
>>>> proxmox-ve: 4.2-48 (running kernel: 4.4.6-1-pve)
>>>>
>>>> pve-manager: 4.2-2 (running version: 4.2-2/725d76f0)
>>>>
>>>> pve-kernel-4.4.6-1-pve: 4.4.6-48
>>>>
>>>> lvm2: 2.02.116-pve2
>>>>
>>>> corosync-pve: 2.3.5-2
>>>>
>>>> libqb0: 1.0-1
>>>>
>>>> pve-cluster: 4.0-39
>>>>
>>>> qemu-server: 4.0-72
>>>>
>>>> pve-firmware: 1.1-8
>>>>
>>>> libpve-common-perl: 4.0-59
>>>>
>>>> libpve-access-control: 4.0-16
>>>>
>>>> libpve-storage-perl: 4.0-50
>>>>
>>>> pve-libspice-server1: 0.12.5-2
>>>>
>>>> vncterm: 1.2-1
>>>>
>>>> pve-qemu-kvm: 2.5-14
>>>>
>>>> pve-container: 1.0-62
>>>>
>>>> pve-firewall: 2.0-25
>>>>
>>>> pve-ha-manager: 1.0-28
>>>>
>>>> ksm-control-daemon: 1.2-1
>>>>
>>>> glusterfs-client: 3.5.2-2+deb8u2
>>>>
>>>> lxc-pve: 1.1.5-7
>>>>
>>>> lxcfs: 2.0.0-pve2
>>>>
>>>> cgmanager: 0.39-pve1
>>>>
>>>> criu: 1.6.0-1
>>>>
>>>> zfsutils: 0.6.5-pve9~jessie
>>>>
>>>> ###
>>>>
>>>>
>>>>
>>>> I'm trying a live Migration via NFS-Storage with a KVM. Migration from Host 2 -> Host 1 always works, Migration from Host 1 -> Host 2 seems to work (no error in live-migration output) but the KVM hangs after migration. I can't ping the VM and VNC output is frozen.
>>>>
>>>>
>>>> Tried it several times, always with the same result. The only difference between the two hosts are the CPUs:
>>>>
>>>>
>>>> Host 1:
>>>>
>>>> AMD Opteron(tm) Processor 4234
>>>>
>>>>
>>>> Host 2:
>>>>
>>>> AMD Opteron(tm) Processor 4184
>>>>
>>>>
>>>>
>>>> Howerver, the KVM is set do Default CPU (KVM 64).
>>>>
>>>>
>>>> How can that happen?
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Greets,
>>>>
>>>> Kilian
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> pve-user mailing list
>>>> pve-user at pve.proxmox.com
>>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>> _______________________________________________
>>> pve-user mailing list
>>> pve-user at pve.proxmox.com
>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>> _______________________________________________
>>> pve-user mailing list
>>> pve-user at pve.proxmox.com
>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>
>>
>>
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>