[PVE-User] VM's consistently failing to start
Lindsay Mathieson
lindsay.mathieson at gmail.com
Fri Feb 27 01:47:44 CET 2015
I have a reoccurring problem with VM's failing to start. In the task log
I'l see this:
kvm: -netdev
type=tap,id=net0,ifname=tap721i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on:
tap: open vhost char device failed: Cannot allocate memory
kvm: -netdev
type=tap,id=net0,ifname=tap721i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on:
Device 'tap' could not be initialized
TASK ERROR: start failed: command '/usr/bin/kvm -id 721 -chardev
'socket,id=qmp,path=/var/run/qemu-server/721.qmp,server,nowait' -mon
'chardev=qmp,mode=control' -vnc
unix:/var/run/qemu-server/721.vnc,x509,password -pidfile
/var/run/qemu-server/721.pid -daemonize -smbios
'type=1,uuid=5cd965b8-ee6c-4a23-a705-589329833829' -name VM721 -smp
'4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot
'menu=on,strict=on,reboot-timeout=1000' -vga qxl -no-hpet -cpu
'kvm64,hv_spinlocks=0xffff,hv_relaxed,+lahf_lm,+x2apic,+sep' -m 2048 -k
en-us -device
'qxl,id=vga1,ram_size=67108864,vram_size=33554432,bus=pci.0,addr=0x18'
-device 'AC97,addr=0x18' -device
'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -readconfig
/usr/share/qemu-server/pve-usb.cfg -chardev
'spicevmc,id=usbredirchardev0,name=usbredir' -device
'usb-redir,chardev=usbredirchardev0,id=usbredirdev0,bus=ehci.0' -chardev
'spicevmc,id=usbredirchardev1,name=usbredir' -device
'usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=ehci.0' -chardev
'spicevmc,id=usbredirchardev2,name=usbredir' -device
'usb-redir,chardev=usbredirchardev2,id=usbredirdev2,bus=ehci.0' -chardev
'spicevmc,id=usbredirchardev3,name=usbredir' -device
'usb-redir,chardev=usbredirchardev3,id=usbredirdev3,bus=ehci.0' -spice
'tls-port=61004,addr=127.0.0.1,tls-ciphers=DES-CBC3-SHA,seamless-migration=on'
-device 'virtio-serial,id=spice,bus=pci.0,addr=0x9' -chardev
'spicevmc,id=vdagent,name=vdagent' -device
'virtserialport,chardev=vdagent,name=com.redhat.spice.0' -device
'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi
'initiator-name=iqn.1993-08.org.debian:01:46185e44b9c' -drive
'file=/mnt/cephfs/images/721/vm-721-disk-1.qcow2,if=none,id=drive-virtio1,format=qcow2,cache=writeback,aio=native,detect-zeroes=on'
-device
'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb,bootindex=100'
-drive
'file=/mnt/pve/ISO/template/iso/virtio-win-0.1-100.iso,if=none,id=drive-ide0,media=cdrom,aio=native'
-device 'ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200'
-netdev
'type=tap,id=net0,ifname=tap721i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on'
-device
'virtio-net-pci,mac=5A:8D:37:32:57:C6,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'
-rtc 'driftfix=slew,base=localtime' -global
'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
According the the proxmox status, the node has 32GB of ram with 19 GB is
use.
dmesg shows a stack dump occuring as well (attached).
proxmox 3.4
kernel 3.10-7
I've found that if I change the vm network card from PV to E1000 and back
to PV that usually resolves the problem.
--
Lindsay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.proxmox.com/pipermail/pve-user/attachments/20150227/119267fc/attachment.htm>
-------------- next part --------------
[1107653.633478] vmbr0: port 6(tap721i0) entered disabled state
[1107698.992235] device tap721i0 entered promiscuous mode
[1107699.000495] vmbr0: port 6(tap721i0) entered forwarding state
[1107699.000513] vmbr0: port 6(tap721i0) entered forwarding state
[1107699.009935] kvm: page allocation failure: order:4, mode:0x1040d0
[1107699.009942] CPU: 2 PID: 21807 Comm: kvm Tainted: G O-------------- 3.10.0-7-pve #1
[1107699.009944] Hardware name: System manufacturer System Product Name/P9X79 WS, BIOS 4601 03/05/2014
[1107699.009947] 0000000000000000 ffff880123edf8f8 ffffffff81612508 ffff880123edf988
[1107699.009952] ffffffff81141f78 0000000000000000 00000000ffffffff ffff880123edf928
[1107699.009957] ffffffff811446d6 0000004023edf958 0000000400000010 001040d00000000b
[1107699.009961] Call Trace:
[1107699.009969] [<ffffffff81612508>] dump_stack+0x19/0x1b
[1107699.009975] [<ffffffff81141f78>] warn_alloc_failed+0xf8/0x160
[1107699.009979] [<ffffffff811446d6>] ? drain_local_pages+0x16/0x20
[1107699.009984] [<ffffffff811460eb>] __alloc_pages_nodemask+0x94b/0xb70
[1107699.009990] [<ffffffff81183168>] alloc_pages_current+0xb8/0x190
[1107699.009995] [<ffffffff81140ffe>] __get_free_pages+0xe/0x50
[1107699.009999] [<ffffffff8118f009>] kmalloc_order_trace+0x39/0xb0
[1107699.010005] [<ffffffffa09c4f79>] vhost_net_open+0x29/0x1b0 [vhost_net]
[1107699.010010] [<ffffffff8151eb3f>] ? nlmsg_notify+0x4f/0xc0
[1107699.010015] [<ffffffff813a18af>] misc_open+0xaf/0x1c0
[1107699.010020] [<ffffffff811ae57b>] chrdev_open+0x9b/0x1b0
[1107699.010024] [<ffffffff811a7723>] do_dentry_open+0x213/0x2c0
[1107699.010027] [<ffffffff811ae4e0>] ? cdev_put+0x30/0x30
[1107699.010031] [<ffffffff811a7805>] finish_open+0x35/0x50
[1107699.010035] [<ffffffff811b921e>] do_last+0x6fe/0xf10
[1107699.010040] [<ffffffff811b54b8>] ? inode_permission+0x18/0x50
[1107699.010044] [<ffffffff811b5568>] ? link_path_walk+0x78/0x890
[1107699.010047] [<ffffffff811b9ae7>] path_openat+0xb7/0x4b0
[1107699.010053] [<ffffffff8161d9cc>] ? __do_page_fault+0x25c/0x4f0
[1107699.010056] [<ffffffff811ba5c1>] do_filp_open+0x41/0xa0
[1107699.010061] [<ffffffff811c6c93>] ? __alloc_fd+0xd3/0x120
[1107699.010065] [<ffffffff811a8b74>] do_sys_open+0xf4/0x1e0
[1107699.010069] [<ffffffff811bc801>] ? SyS_ioctl+0x91/0xb0
[1107699.010073] [<ffffffff81060a30>] ? task_stopped_code+0x60/0x60
[1107699.010077] [<ffffffff811a8c82>] SyS_open+0x22/0x30
[1107699.010081] [<ffffffff81622619>] system_call_fastpath+0x16/0x1b
[1107699.010083] Mem-Info:
[1107699.010085] Node 0 DMA per-cpu:
[1107699.010088] CPU 0: hi: 0, btch: 1 usd: 0
[1107699.010090] CPU 1: hi: 0, btch: 1 usd: 0
[1107699.010092] CPU 2: hi: 0, btch: 1 usd: 0
[1107699.010094] CPU 3: hi: 0, btch: 1 usd: 0
[1107699.010096] CPU 4: hi: 0, btch: 1 usd: 0
[1107699.010099] CPU 5: hi: 0, btch: 1 usd: 0
[1107699.010101] CPU 6: hi: 0, btch: 1 usd: 0
[1107699.010103] CPU 7: hi: 0, btch: 1 usd: 0
[1107699.010105] CPU 8: hi: 0, btch: 1 usd: 0
[1107699.010107] CPU 9: hi: 0, btch: 1 usd: 0
[1107699.010109] CPU 10: hi: 0, btch: 1 usd: 0
[1107699.010111] CPU 11: hi: 0, btch: 1 usd: 0
[1107699.010113] Node 0 DMA32 per-cpu:
[1107699.010116] CPU 0: hi: 186, btch: 31 usd: 0
[1107699.010118] CPU 1: hi: 186, btch: 31 usd: 0
[1107699.010120] CPU 2: hi: 186, btch: 31 usd: 0
[1107699.010122] CPU 3: hi: 186, btch: 31 usd: 0
[1107699.010124] CPU 4: hi: 186, btch: 31 usd: 0
[1107699.010126] CPU 5: hi: 186, btch: 31 usd: 0
[1107699.010128] CPU 6: hi: 186, btch: 31 usd: 0
[1107699.010130] CPU 7: hi: 186, btch: 31 usd: 0
[1107699.010133] CPU 8: hi: 186, btch: 31 usd: 0
[1107699.010135] CPU 9: hi: 186, btch: 31 usd: 0
[1107699.010137] CPU 10: hi: 186, btch: 31 usd: 0
[1107699.010139] CPU 11: hi: 186, btch: 31 usd: 0
[1107699.010140] Node 0 Normal per-cpu:
[1107699.010143] CPU 0: hi: 186, btch: 31 usd: 30
[1107699.010145] CPU 1: hi: 186, btch: 31 usd: 159
[1107699.010148] CPU 2: hi: 186, btch: 31 usd: 0
[1107699.010150] CPU 3: hi: 186, btch: 31 usd: 1
[1107699.010152] CPU 4: hi: 186, btch: 31 usd: 185
[1107699.010154] CPU 5: hi: 186, btch: 31 usd: 0
[1107699.010156] CPU 6: hi: 186, btch: 31 usd: 0
[1107699.010160] CPU 7: hi: 186, btch: 31 usd: 0
[1107699.010162] CPU 8: hi: 186, btch: 31 usd: 0
[1107699.010164] CPU 9: hi: 186, btch: 31 usd: 0
[1107699.010166] CPU 10: hi: 186, btch: 31 usd: 0
[1107699.010168] CPU 11: hi: 186, btch: 31 usd: 0
[1107699.010174] active_anon:3995861 inactive_anon:486028 isolated_anon:0
[1107699.010174] active_file:1005172 inactive_file:2034828 isolated_file:0
[1107699.010174] unevictable:16175 dirty:1438 writeback:0 unstable:0
[1107699.010174] free:163518 slab_reclaimable:158912 slab_unreclaimable:126081
[1107699.010174] mapped:24526 shmem:16030 pagetables:13844 bounce:0
[1107699.010174] free_cma:0
[1107699.010179] Node 0 DMA free:15772kB min:32kB low:40kB high:48kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[1107699.010186] lowmem_reserve[]: 0 2903 32084 32084
[1107699.010190] Node 0 DMA32 free:489304kB min:6112kB low:7640kB high:9168kB active_anon:1177360kB inactive_anon:416544kB active_file:93896kB inactive_file:464320kB unevictable:4012kB isolated(anon):0kB isolated(file):0kB present:3069508kB managed:2972716kB mlocked:4012kB dirty:336kB writeback:0kB mapped:7152kB shmem:5736kB slab_reclaimable:213004kB slab_unreclaimable:66396kB kernel_stack:1016kB pagetables:4672kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[1107699.010197] lowmem_reserve[]: 0 0 29181 29181
[1107699.010201] Node 0 Normal free:149288kB min:61436kB low:76792kB high:92152kB active_anon:14805792kB inactive_anon:1527568kB active_file:3926792kB inactive_file:7674992kB unevictable:60688kB isolated(anon):0kB isolated(file):0kB present:30408704kB managed:29882156kB mlocked:60688kB dirty:5416kB writeback:0kB mapped:90952kB shmem:57948kB slab_reclaimable:422644kB slab_unreclaimable:437928kB kernel_stack:6720kB pagetables:50704kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[1107699.010208] lowmem_reserve[]: 0 0 0 0
[1107699.010211] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15772kB
[1107699.010227] Node 0 DMA32: 1460*4kB (UEM) 34442*8kB (UEM) 12915*16kB (UEM) 63*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 490032kB
[1107699.010239] Node 0 Normal: 21051*4kB (UEM) 8172*8kB (UEM) 125*16kB (UEM) 3*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 151676kB
[1107699.010252] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[1107699.010255] 3117844 total pagecache pages
[1107699.010256] 59246 pages in swap cache
[1107699.010259] Swap cache stats: add 364213, delete 304967, find 14566406/14584108
[1107699.010260] Free swap = 13959580kB
[1107699.010262] Total swap = 14548988kB
[1107699.125098] 8388607 pages RAM
[1107699.125106] 166400 pages reserved
[1107699.125114] 1245239 pages shared
[1107699.125122] 7306815 pages non-shared
[1107699.269379] vmbr0: port 6(tap721i0) entered disabled state
More information about the pve-user
mailing list