[PVE-User] Timeout while migrating VM between 2 proxmox hosts

[Laurent CARON]

Hi,

I'm running a basic proxmox cluster composed of 4 nodes (should it
matter).

Shared storage is coming from a netapp iSCSI box.

Please note those 2 boxes have different kernel versions:
2.6.32-16-pve vs. 2.6.32-14-pve
Kernel versions are different since I'm upgrading the boxes one by 
one without (supposedly) downtime.

While trying to migrate from one host to another I got the following
error:

proxmox-vty-002: node where the machine was (source)
proxmox-siege-001: target node

Nov  6 07:43:51 proxmox-vty-002 pvedaemon[711739]: <root at pam> starting task UPID:proxmox-vty-002:000B6F33:0DE0CB88:5098B1A7:qmigrate:602:root at pam:
Nov  6 07:43:55 proxmox-vty-002 pmxcfs[2384]: [status] notice: received log
Nov  6 07:43:57 proxmox-vty-002 pmxcfs[2384]: [status] notice: received log
Nov  6 07:44:01 proxmox-vty-002 pvedaemon[749363]: VM 602 qmp command failed - VM 602 qmp command 'migrate' failed - got timeout
Nov  6 07:46:33 proxmox-vty-002 kernel: vmbr2000: port 3(tap602i0) entering disabled state
Nov  6 07:46:33 proxmox-vty-002 kernel: vmbr2000: port 3(tap602i0) entering disabled state
Nov  6 07:46:36 proxmox-vty-002 pvedaemon[711739]: <root at pam> end task UPID:proxmox-vty-002:000B6F33:0DE0CB88:5098B1A7:qmigrate:602:root at pam: OK

Nov  6 07:43:55 proxmox-siege-001 qm[4395]: <root at pam> starting task UPID:proxmox-siege-001:0000112C:0000481A:5098B1AB:qmstart:602:root at pam:
Nov  6 07:43:55 proxmox-siege-001 qm[4396]: start VM 602: UPID:proxmox-siege-001:0000112C:0000481A:5098B1AB:qmstart:602:root at pam:
Nov  6 07:43:56 proxmox-siege-001 kernel: device tap602i0 entered promiscuous mode
Nov  6 07:43:56 proxmox-siege-001 kernel: vmbr2000: port 4(tap602i0) entering forwarding state
Nov  6 07:43:57 proxmox-siege-001 qm[4395]: <root at pam> end task UPID:proxmox-siege-001:0000112C:0000481A:5098B1AB:qmstart:602:root at pam: OK
Nov  6 07:44:06 proxmox-siege-001 kernel: tap602i0: no IPv6 routers present
Nov  6 07:44:09 proxmox-siege-001 pvedaemon[3588]: WARNING: unable to connect to VM 602 socket - timeout after 31 retries
Nov  6 07:44:10 proxmox-siege-001 pvestatd[3609]: WARNING: unable to connect to VM 602 socket - timeout after 31 retries
Nov  6 07:44:12 proxmox-siege-001 pvedaemon[3586]: WARNING: unable to connect to VM 602 socket - timeout after 31 retries
Nov  6 07:44:15 proxmox-siege-001 pvedaemon[3587]: WARNING: unable to connect to VM 602 socket - timeout after 31 retries
Nov  6 07:44:19 proxmox-siege-001 pvedaemon[3588]: WARNING: unable to connect to VM 602 socket - timeout after 31 retries
(...snip...)
Nov  6 07:46:23 proxmox-siege-001 pvedaemon[3588]: WARNING: unable to connect to VM 602 socket - timeout after 31 retries
Nov  6 07:46:27 proxmox-siege-001 pvedaemon[3587]: WARNING: unable to connect to VM 602 socket - timeout after 31 retries
Nov  6 07:46:30 proxmox-siege-001 pvestatd[3609]: WARNING: unable to connect to VM 602 socket - timeout after 31 retries
Nov  6 07:46:30 proxmox-siege-001 pvedaemon[3586]: WARNING: unable to connect to VM 602 socket - timeout after 31 retries
Nov  6 07:46:33 proxmox-siege-001 kernel: vmbr2000: port 4(tap602i0) entering disabled state
Nov  6 07:46:33 proxmox-siege-001 kernel: vmbr2000: port 4(tap602i0) entering disabled state
Nov  6 07:46:33 proxmox-siege-001 pvedaemon[3587]: WARNING: unable to connect to VM 602 socket - Connection refused

proxmox-siege-001:~# pveversion -verbose
pve-manager: 2.2-26 (pve-manager/2.2/c1614c8c)
running kernel: 2.6.32-16-pve
proxmox-ve-2.6.32: 2.2-80
pve-kernel-2.6.32-16-pve: 2.6.32-80
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-1
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-28
qemu-server: 2.0-64
pve-firmware: 1.0-21
libpve-common-perl: 1.0-37
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-34
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.2-7
ksm-control-daemon: 1.1-1


proxmox-vty-002:~# pveversion -v
pve-manager: 2.1-14 (pve-manager/2.1/f32f3f46)
running kernel: 2.6.32-14-pve
proxmox-ve-2.6.32: 2.1-74
pve-kernel-2.6.32-14-pve: 2.6.32-74
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.92-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.8-1
pve-cluster: 1.0-27
qemu-server: 2.0-49
pve-firmware: 1.0-18
libpve-common-perl: 1.0-30
libpve-access-control: 1.0-24
libpve-storage-perl: 2.0-31
vncterm: 1.0-3
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.1-8
ksm-control-daemon: 1.1-1

Do any of you guys have a clue ?