[pve-devel] migration problems since qemu 1.3

Fri Dec 21 11:43:37 CET 2012

Hi Alexandre,

i've added some debugging / logging code.

The output stops / hangs at query migrate. See here:

Dec 21 11:41:59 starting migration of VM 100 to node 'cloud1-1203' 
(10.255.0.22)
Dec 21 11:41:59 copying disk images
Dec 21 11:41:59 starting VM 100 on remote node 'cloud1-1203'
Dec 21 11:42:02 starting migration tunnel
Dec 21 11:42:03 starting online/live migration on port 60000
Dec 21 11:42:03 migrate-set-capabilities, capabilities => [HASH(0x39a9fb0)]
Dec 21 11:42:03 migrate-set-cache-size, value => 429496729
Dec 21 11:42:03 start migrate tcp:localhost:60000
Dec 21 11:42:05 query-migrate
Dec 21 11:42:05 migration status: active (transferred 468063329, 
remaining 3764068352), total 4303814656)
Dec 21 11:42:07 query-migrate

I can't even ping the VM anymore.

Stefan

Am 21.12.2012 08:58, schrieb Alexandre DERUMIER:
> Hi Stefan, any news ?
>
> I'm trying to reproduce your problem, but it's works fine for me, no crash...
>
> ----- Mail original -----
>
> De: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag>
> À: "Alexandre DERUMIER" <aderumier at odiso.com>
> Cc: pve-devel at pve.proxmox.com
> Envoyé: Jeudi 20 Décembre 2012 16:09:42
> Objet: Re: [pve-devel] migration problems since qemu 1.3
>
> Hi,
> Am 20.12.2012 15:57, schrieb Alexandre DERUMIER:
>> Just an idea (not sure it's the problem),can you try to commment
>>
>> $qmpclient->queue_cmd($vmid, $ballooncb, 'query-balloon');
>>
>> in QemuServer.pm, line 2081.
>>
>> and restart pvedaemon && pvestatd ?
>
> This doesn't change anything.
>
> Right now the kvm process is running on old and new machine.
>
> An strace on the pid on the new machine shows a loop of:
>
> ----------------
> [pid 28351] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 28351] futex(0x7ff8b8025388, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 28351] futex(0x7ff8b8026024,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 11801, {1356016143,
> 843092000}, ffffffff <unfinished ...>
> [pid 28285] mremap(0x7ff77bfe4000, 160378880, 160411648, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160411648, 160448512, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160448512, 160481280, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160481280, 160514048, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160514048, 160546816, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160546816, 160583680, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160583680, 160616448, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160616448, 160649216, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160649216, 160681984, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160681984, 160718848, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160718848, 160751616, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160751616, 160784384, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160784384, 160817152, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160817152, 160854016, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160854016, 160886784, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160886784, 160919552, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160919552, 160952320, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160952320, 160989184, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 160989184, 161021952, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161021952, 161054720, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161054720, 161087488, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161087488, 161124352, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161124352, 161157120, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161157120, 161189888, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161189888, 161222656, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161222656, 161259520, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161259520, 161292288, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161292288, 161325056, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28351] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 28351] futex(0x7ff8b8025388, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 28351] futex(0x7ff8b8026024,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 11803, {1356016144,
> 843283000}, ffffffff <unfinished ...>
> [pid 28285] mremap(0x7ff77bfe4000, 161325056, 161357824, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161357824, 161394688, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161394688, 161427456, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161427456, 161460224, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28345] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid 28345] futex(0x7ff8caa2e274, FUTEX_CMP_REQUEUE_PRIVATE, 1,
> 2147483647, 0x7ff8caa2e1b0, 872) = 1
> [pid 28347] <... futex resumed> ) = 0
> [pid 28345] futex(0x7ff8caa241a8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 28347] futex(0x7ff8caa2e1b0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
> [pid 28345] <... futex resumed> ) = 0
> [pid 28347] <... futex resumed> ) = 0
> [pid 28345] futex(0x7ff8caa2420c,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 799, {1356016153,
> 954319000}, ffffffff <unfinished ...>
> [pid 28347] sendmsg(19, {msg_name(0)=NULL, msg_iov(1)=[{"\t", 1}],
> msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 1
> [pid 28347] futex(0x7ff8caa2e274, FUTEX_WAIT_PRIVATE, 873, NULL
> <unfinished ...>
> [pid 28285] mremap(0x7ff77bfe4000, 161460224, 161492992, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161492992, 161529856, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161529856, 161562624, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161562624, 161595392, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161595392, 161628160, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161628160, 161665024, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161665024, 161697792, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161697792, 161730560, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161730560, 161763328, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161763328, 161800192, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161800192, 161832960, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161832960, 161865728, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161865728, 161898496, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> [pid 28285] mremap(0x7ff77bfe4000, 161898496, 161935360, MREMAP_MAYMOVE)
> = 0x7ff77bfe4000
> -----------------------
>
>
> Stefan
>