[PVE-User] Cannot start VM - timeout waiting on systemd

[Gerald Brandt]

I get a timeout waiting on systemd when I try to start a VM. Any ideas. 
It's not CPU load or memory.


Mar 18 11:11:58 proxmox-1 pvedaemon[5035]: start VM 141: 
UPID:proxmox-1:000013AB:0838CFFC:5E72484E:qmstart:141:root at pam:
Mar 18 11:11:58 proxmox-1 pvedaemon[2463]: <root at pam> starting task 
UPID:proxmox-1:000013AB:0838CFFC:5E72484E:qmstart:141:root at pam:
Mar 18 11:12:00 proxmox-1 systemd[1]: Starting Proxmox VE replication 
runner...
Mar 18 11:12:01 proxmox-1 pmxcfs[3161]: [status] notice: received log
Mar 18 11:12:02 proxmox-1 systemd[1]: Started Session 459 of user root.
Mar 18 11:12:05 proxmox-1 systemd[1]: Started Proxmox VE replication runner.
Mar 18 11:12:06 proxmox-1 pvedaemon[5035]: timeout waiting on systemd
Mar 18 11:12:06 proxmox-1 pvedaemon[2463]: <root at pam> end task 
UPID:proxmox-1:000013AB:0838CFFC:5E72484E:qmstart:141:root at pam: timeout 
waiting on systemd
Mar 18 11:12:07 proxmox-1 qm[5123]: VM 141 qmp command failed - VM 141 
not running
Mar 18 11:12:08 proxmox-1 pmxcfs[3161]: [status] notice: received log
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284039] INFO: task kvm:5081 
blocked for more than 120 seconds.
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284218]       Tainted: 
P           O     4.15.18-24-pve #1
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284358] "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284540] kvm D    0  5081      
1 0x80000006
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284553] Call Trace:
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284579] __schedule+0x3e0/0x870
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284590] schedule+0x36/0x80
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284601] 
schedule_timeout+0x1d4/0x360
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284613]  ? 
call_rcu_sched+0x17/0x20
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284626]  ? 
__percpu_ref_switch_mode+0xd7/0x180
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284635] 
wait_for_completion+0xb4/0x140
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284644]  ? wake_up_q+0x80/0x80
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284657] exit_aio+0xeb/0x100
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284670] mmput+0x2b/0x130
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284696] 
vhost_dev_cleanup+0x382/0x3b0 [vhost]
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284711] 
vhost_net_release+0x53/0xb0 [vhost_net]
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284721] __fput+0xea/0x220
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284731] ____fput+0xe/0x10
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284742] task_work_run+0x9d/0xc0
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284752] do_exit+0x2f6/0xbd0
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284762]  ? 
__switch_to_asm+0x41/0x70
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284771]  ? 
__switch_to_asm+0x41/0x70
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284778]  ? 
__switch_to_asm+0x35/0x70
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284785]  ? 
__switch_to_asm+0x41/0x70
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284793]  ? 
__switch_to_asm+0x35/0x70
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284802] do_group_exit+0x43/0xb0
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284812] get_signal+0x15a/0x7f0
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284825]  ? do_futex+0x7e6/0xd10
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284838] do_signal+0x37/0x710
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284849]  ? 
blk_finish_plug+0x2c/0x40
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284859]  ? 
hrtimer_nanosleep+0xd8/0x1f0
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284866]  ? SyS_futex+0x83/0x180
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284878] 
exit_to_usermode_loop+0x80/0xd0
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284888] do_syscall_64+0x100/0x130
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284899] 
entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284907] RIP: 0033:0x7f6aff09c469
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284912] RSP: 
002b:00007f6af33fc638 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284922] RAX: fffffffffffffe00 
RBX: 000055e6ad1894c8 RCX: 00007f6aff09c469
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284927] RDX: 00000000ffffffff 
RSI: 0000000000000000 RDI: 000055e6ad1894c8
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284931] RBP: 00000000ffffffff 
R08: 0000000000000000 R09: 0000000000000000
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284936] R10: 0000000000000000 
R11: 0000000000000246 R12: 0000000000000000
Mar 18 11:12:08 proxmox-1 kernel: [1379441.284941] R13: 00007ffcf670c33f 
R14: 00007f6af2bff000 R15: 0000000000000003
Mar 18 11:12:08 proxmox-1 pvestatd[3372]: got timeout