[PVE-User] A less aggressive OOM?
Marco Gaiarin
gaio at lilliput.linux.it
Tue Jul 8 18:31:41 CEST 2025
Mandi! Victor Rodriguez
In chel di` si favelave...
> I would start by analyzing the memory status at the time of the OOM. There
> should be a some lines in journal/syslog were the kernel writes what the
> memory looked like and you can figure out why it had to kill a process.
This is the full OOM log:
Jul 4 20:00:12 pppve1 kernel: [3375931.660119] kvm invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
Jul 4 20:00:12 pppve1 kernel: [3375931.669158] CPU: 1 PID: 4088 Comm: kvm Tainted: P O 6.8.12-10-pve #1
Jul 4 20:00:12 pppve1 kernel: [3375931.677778] Hardware name: Dell Inc. PowerEdge T440/021KCD, BIOS 2.24.0 04/02/2025
Jul 4 20:00:12 pppve1 kernel: [3375931.686211] Call Trace:
Jul 4 20:00:12 pppve1 kernel: [3375931.689504] <TASK>
Jul 4 20:00:12 pppve1 kernel: [3375931.692428] dump_stack_lvl+0x76/0xa0
Jul 4 20:00:12 pppve1 kernel: [3375931.696915] dump_stack+0x10/0x20
Jul 4 20:00:12 pppve1 kernel: [3375931.701057] dump_header+0x47/0x1f0
Jul 4 20:00:12 pppve1 kernel: [3375931.705358] oom_kill_process+0x110/0x240
Jul 4 20:00:12 pppve1 kernel: [3375931.710169] out_of_memory+0x26e/0x560
Jul 4 20:00:12 pppve1 kernel: [3375931.714707] __alloc_pages+0x10ce/0x1320
Jul 4 20:00:12 pppve1 kernel: [3375931.719422] alloc_pages_mpol+0x91/0x1f0
Jul 4 20:00:12 pppve1 kernel: [3375931.724136] alloc_pages+0x54/0xb0
Jul 4 20:00:12 pppve1 kernel: [3375931.728320] __get_free_pages+0x11/0x50
Jul 4 20:00:12 pppve1 kernel: [3375931.732938] __pollwait+0x9e/0xe0
Jul 4 20:00:12 pppve1 kernel: [3375931.737015] eventfd_poll+0x2c/0x70
Jul 4 20:00:12 pppve1 kernel: [3375931.741261] do_sys_poll+0x2f4/0x610
Jul 4 20:00:12 pppve1 kernel: [3375931.745587] ? __pfx___pollwait+0x10/0x10
Jul 4 20:00:12 pppve1 kernel: [3375931.750332] ? __pfx_pollwake+0x10/0x10
Jul 4 20:00:12 pppve1 kernel: [3375931.754900] ? __pfx_pollwake+0x10/0x10
Jul 4 20:00:12 pppve1 kernel: [3375931.759463] ? __pfx_pollwake+0x10/0x10
Jul 4 20:00:12 pppve1 kernel: [3375931.764011] ? __pfx_pollwake+0x10/0x10
Jul 4 20:00:12 pppve1 kernel: [3375931.768617] ? __pfx_pollwake+0x10/0x10
Jul 4 20:00:12 pppve1 kernel: [3375931.773165] ? __pfx_pollwake+0x10/0x10
Jul 4 20:00:12 pppve1 kernel: [3375931.777688] ? __pfx_pollwake+0x10/0x10
Jul 4 20:00:12 pppve1 kernel: [3375931.782156] ? __pfx_pollwake+0x10/0x10
Jul 4 20:00:12 pppve1 kernel: [3375931.786622] ? __pfx_pollwake+0x10/0x10
Jul 4 20:00:12 pppve1 kernel: [3375931.791111] __x64_sys_ppoll+0xde/0x170
Jul 4 20:00:12 pppve1 kernel: [3375931.795656] x64_sys_call+0x1818/0x2480
Jul 4 20:00:12 pppve1 kernel: [3375931.800193] do_syscall_64+0x81/0x170
Jul 4 20:00:12 pppve1 kernel: [3375931.804485] ? __x64_sys_ppoll+0xf2/0x170
Jul 4 20:00:12 pppve1 kernel: [3375931.809100] ? syscall_exit_to_user_mode+0x86/0x260
Jul 4 20:00:12 pppve1 kernel: [3375931.814566] ? do_syscall_64+0x8d/0x170
Jul 4 20:00:12 pppve1 kernel: [3375931.818979] ? syscall_exit_to_user_mode+0x86/0x260
Jul 4 20:00:12 pppve1 kernel: [3375931.824425] ? do_syscall_64+0x8d/0x170
Jul 4 20:00:12 pppve1 kernel: [3375931.828825] ? clear_bhb_loop+0x15/0x70
Jul 4 20:00:12 pppve1 kernel: [3375931.833211] ? clear_bhb_loop+0x15/0x70
Jul 4 20:00:12 pppve1 kernel: [3375931.837579] ? clear_bhb_loop+0x15/0x70
Jul 4 20:00:12 pppve1 kernel: [3375931.841928] entry_SYSCALL_64_after_hwframe+0x78/0x80
Jul 4 20:00:12 pppve1 kernel: [3375931.847482] RIP: 0033:0x765bb1ce8316
Jul 4 20:00:12 pppve1 kernel: [3375931.851577] Code: 7c 24 08 e8 2c 95 f8 ff 4c 8b 54 24 18 48 8b 74 24 10 41 b8 08 00 00 00 41 89 c1 48 8b 7c 24 08 4c 89 e2 b8 0f 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 32 44 89 cf 89 44 24 08 e8 76 95 f8 ff 8b 44
Jul 4 20:00:12 pppve1 kernel: [3375931.871194] RSP: 002b:00007fff2d39ea20 EFLAGS: 00000293 ORIG_RAX: 000000000000010f
Jul 4 20:00:12 pppve1 kernel: [3375931.879298] RAX: ffffffffffffffda RBX: 00006045d3e68470 RCX: 0000765bb1ce8316
Jul 4 20:00:12 pppve1 kernel: [3375931.886963] RDX: 00007fff2d39ea40 RSI: 0000000000000010 RDI: 00006045d4de5f20
Jul 4 20:00:12 pppve1 kernel: [3375931.894630] RBP: 00007fff2d39eaac R08: 0000000000000008 R09: 0000000000000000
Jul 4 20:00:12 pppve1 kernel: [3375931.902299] R10: 0000000000000000 R11: 0000000000000293 R12: 00007fff2d39ea40
Jul 4 20:00:12 pppve1 kernel: [3375931.909951] R13: 00006045d3e68470 R14: 00006045b014d570 R15: 00007fff2d39eab0
Jul 4 20:00:12 pppve1 kernel: [3375931.917656] </TASK>
Jul 4 20:00:12 pppve1 kernel: [3375931.920515] Mem-Info:
Jul 4 20:00:12 pppve1 kernel: [3375931.923465] active_anon:4467063 inactive_anon:2449638 isolated_anon:0
Jul 4 20:00:12 pppve1 kernel: [3375931.923465] active_file:611 inactive_file:303 isolated_file:0
Jul 4 20:00:12 pppve1 kernel: [3375931.923465] unevictable:39551 dirty:83 writeback:237
Jul 4 20:00:12 pppve1 kernel: [3375931.923465] slab_reclaimable:434580 slab_unreclaimable:1792355
Jul 4 20:00:12 pppve1 kernel: [3375931.923465] mapped:571491 shmem:581427 pagetables:26365
Jul 4 20:00:12 pppve1 kernel: [3375931.923465] sec_pagetables:11751 bounce:0
Jul 4 20:00:12 pppve1 kernel: [3375931.923465] kernel_misc_reclaimable:0
Jul 4 20:00:12 pppve1 kernel: [3375931.923465] free:234516 free_pcp:5874 free_cma:0
Jul 4 20:00:12 pppve1 kernel: [3375931.969518] Node 0 active_anon:17033436kB inactive_anon:10633368kB active_file:64kB inactive_file:3196kB unevictable:158204kB isolated(anon):0kB isolated(file):0kB mapped:2285988kB dirty:356kB writeback:948kB shmem:2325708kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:866304kB writeback_tmp:0kB kernel_stack:11520kB pagetables:105460kB sec_pagetables:47004kB all_unreclaimable? no
Jul 4 20:00:12 pppve1 kernel: [3375932.004977] Node 0 DMA free:11264kB boost:0kB min:12kB low:24kB high:36kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 4 20:00:12 pppve1 kernel: [3375932.032646] lowmem_reserve[]: 0 1527 63844 63844 63844
Jul 4 20:00:12 pppve1 kernel: [3375932.038675] Node 0 DMA32 free:252428kB boost:0kB min:1616kB low:3176kB high:4736kB reserved_highatomic:2048KB active_anon:310080kB inactive_anon:986436kB active_file:216kB inactive_file:0kB unevictable:0kB writepending:0kB present:1690624kB managed:1623508kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Jul 4 20:00:12 pppve1 kernel: [3375932.069110] lowmem_reserve[]: 0 0 62317 62317 62317
Jul 4 20:00:12 pppve1 kernel: [3375932.074979] Node 0 Normal free:814396kB boost:290356kB min:356304kB low:420116kB high:483928kB reserved_highatomic:346112KB active_anon:11258684kB inactive_anon:15111580kB active_file:0kB inactive_file:2316kB unevictable:158204kB writepending:1304kB present:65011712kB managed:63820796kB mlocked:155132kB bounce:0kB free_pcp:12728kB local_pcp:0kB free_cma:0kB
Jul 4 20:00:12 pppve1 kernel: [3375932.109188] lowmem_reserve[]: 0 0 0 0 0
Jul 4 20:00:12 pppve1 kernel: [3375932.114119] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB
Jul 4 20:00:12 pppve1 kernel: [3375932.127796] Node 0 DMA32: 5689*4kB (UMH) 1658*8kB (UMH) 381*16kB (UM) 114*32kB (UME) 97*64kB (UME) 123*128kB (UMEH) 87*256kB (MEH) 96*512kB (UMEH) 58*1024kB (UME) 5*2048kB (UME) 11*4096kB (ME) = 253828kB
Jul 4 20:00:12 pppve1 kernel: [3375932.148050] Node 0 Normal: 16080*4kB (UMEH) 36886*8kB (UMEH) 22890*16kB (UMEH) 4687*32kB (MEH) 159*64kB (UMEH) 10*128kB (UE) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 887088kB
Jul 4 20:00:12 pppve1 kernel: [3375932.165899] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jul 4 20:00:12 pppve1 kernel: [3375932.175876] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul 4 20:00:12 pppve1 kernel: [3375932.185569] 586677 total pagecache pages
Jul 4 20:00:12 pppve1 kernel: [3375932.190737] 0 pages in swap cache
Jul 4 20:00:12 pppve1 kernel: [3375932.195285] Free swap = 0kB
Jul 4 20:00:12 pppve1 kernel: [3375932.199404] Total swap = 0kB
Jul 4 20:00:12 pppve1 kernel: [3375932.203513] 16679583 pages RAM
Jul 4 20:00:12 pppve1 kernel: [3375932.207787] 0 pages HighMem/MovableOnly
Jul 4 20:00:12 pppve1 kernel: [3375932.212819] 314667 pages reserved
Jul 4 20:00:12 pppve1 kernel: [3375932.217321] 0 pages hwpoisoned
Jul 4 20:00:12 pppve1 kernel: [3375932.221525] Tasks state (memory values in pages):
Jul 4 20:00:12 pppve1 kernel: [3375932.227400] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Jul 4 20:00:12 pppve1 kernel: [3375932.239680] [ 1959] 106 1959 1971 544 96 448 0 61440 0 0 rpcbind
Jul 4 20:00:12 pppve1 kernel: [3375932.251672] [ 1982] 104 1982 2350 672 160 512 0 57344 0 -900 dbus-daemon
Jul 4 20:00:12 pppve1 kernel: [3375932.264020] [ 1991] 0 1991 1767 275 83 192 0 57344 0 0 ksmtuned
Jul 4 20:00:12 pppve1 kernel: [3375932.276094] [ 1995] 0 1995 69541 480 64 416 0 86016 0 0 pve-lxc-syscall
Jul 4 20:00:12 pppve1 kernel: [3375932.289686] [ 2002] 0 2002 1330 384 32 352 0 53248 0 0 qmeventd
Jul 4 20:00:12 pppve1 kernel: [3375932.301811] [ 2003] 0 2003 55449 727 247 480 0 86016 0 0 rsyslogd
Jul 4 20:00:12 pppve1 kernel: [3375932.313919] [ 2004] 0 2004 3008 928 448 480 0 69632 0 0 smartd
Jul 4 20:00:12 pppve1 kernel: [3375932.325834] [ 2009] 0 2009 6386 992 224 768 0 77824 0 0 systemd-logind
Jul 4 20:00:12 pppve1 kernel: [3375932.339438] [ 2010] 0 2010 584 256 0 256 0 40960 0 -1000 watchdog-mux
Jul 4 20:00:12 pppve1 kernel: [3375932.352936] [ 2021] 0 2021 60174 928 256 672 0 90112 0 0 zed
Jul 4 20:00:12 pppve1 kernel: [3375932.364626] [ 2136] 0 2136 75573 256 64 192 0 86016 0 -1000 lxcfs
Jul 4 20:00:12 pppve1 kernel: [3375932.376485] [ 2397] 0 2397 2208 480 64 416 0 61440 0 0 lxc-monitord
Jul 4 20:00:12 pppve1 kernel: [3375932.389169] [ 2421] 0 2421 40673 454 70 384 0 73728 0 0 apcupsd
Jul 4 20:00:12 pppve1 kernel: [3375932.400685] [ 2426] 0 2426 3338 428 172 256 0 69632 0 0 iscsid
Jul 4 20:00:12 pppve1 kernel: [3375932.412121] [ 2427] 0 2427 3464 3343 431 2912 0 77824 0 -17 iscsid
Jul 4 20:00:12 pppve1 kernel: [3375932.423754] [ 2433] 0 2433 3860 1792 320 1472 0 77824 0 -1000 sshd
Jul 4 20:00:12 pppve1 kernel: [3375932.435208] [ 2461] 0 2461 189627 2688 1344 1344 0 155648 0 0 dsm_ism_srvmgrd
Jul 4 20:00:12 pppve1 kernel: [3375932.448290] [ 2490] 113 2490 4721 750 142 608 0 61440 0 0 chronyd
Jul 4 20:00:12 pppve1 kernel: [3375932.459988] [ 2492] 113 2492 2639 502 118 384 0 61440 0 0 chronyd
Jul 4 20:00:12 pppve1 kernel: [3375932.471684] [ 2531] 0 2531 1469 448 32 416 0 49152 0 0 agetty
Jul 4 20:00:12 pppve1 kernel: [3375932.483269] [ 2555] 0 2555 126545 673 244 429 0 147456 0 0 rrdcached
Jul 4 20:00:12 pppve1 kernel: [3375932.483275] [ 2582] 0 2582 155008 15334 3093 864 11377 434176 0 0 pmxcfs
Jul 4 20:00:12 pppve1 kernel: [3375932.506653] [ 2654] 0 2654 10667 614 134 480 0 77824 0 0 master
Jul 4 20:00:12 pppve1 kernel: [3375932.517986] [ 2656] 107 2656 10812 704 160 544 0 73728 0 0 qmgr
Jul 4 20:00:12 pppve1 kernel: [3375932.529118] [ 2661] 0 2661 139892 41669 28417 2980 10272 405504 0 0 corosync
Jul 4 20:00:12 pppve1 kernel: [3375932.540553] [ 2662] 0 2662 1653 576 32 544 0 53248 0 0 cron
Jul 4 20:00:12 pppve1 kernel: [3375932.551657] [ 2664] 0 2664 1621 480 96 384 0 57344 0 0 proxmox-firewal
Jul 4 20:00:12 pppve1 kernel: [3375932.564093] [ 3164] 0 3164 83332 26227 25203 768 256 360448 0 0 pve-firewall
Jul 4 20:00:12 pppve1 kernel: [3375932.576192] [ 3233] 0 3233 85947 28810 27242 1216 352 385024 0 0 pvestatd
Jul 4 20:00:12 pppve1 kernel: [3375932.587638] [ 3417] 0 3417 93674 36011 35531 480 0 438272 0 0 pvedaemon
Jul 4 20:00:12 pppve1 kernel: [3375932.599167] [ 3421] 0 3421 95913 37068 35884 1120 64 454656 0 0 pvedaemon worke
Jul 4 20:00:12 pppve1 kernel: [3375932.611536] [ 3424] 0 3424 96072 36972 35852 1088 32 454656 0 0 pvedaemon worke
Jul 4 20:00:12 pppve1 kernel: [3375932.623977] [ 3426] 0 3426 96167 37068 35948 1056 64 458752 0 0 pvedaemon worke
Jul 4 20:00:12 pppve1 kernel: [3375932.636698] [ 3558] 0 3558 90342 29540 28676 608 256 385024 0 0 pve-ha-crm
Jul 4 20:00:12 pppve1 kernel: [3375932.648477] [ 3948] 33 3948 94022 37705 35849 1856 0 471040 0 0 pveproxy
Jul 4 20:00:12 pppve1 kernel: [3375932.660083] [ 3954] 33 3954 21688 14368 12736 1632 0 221184 0 0 spiceproxy
Jul 4 20:00:12 pppve1 kernel: [3375932.671862] [ 3956] 0 3956 90222 29321 28521 544 256 397312 0 0 pve-ha-lrm
Jul 4 20:00:12 pppve1 kernel: [3375932.683484] [ 3994] 0 3994 1290140 706601 705993 608 0 6389760 0 0 kvm
Jul 4 20:00:12 pppve1 kernel: [3375932.694551] [ 4088] 0 4088 1271416 1040767 1040223 544 0 8994816 0 0 kvm
Jul 4 20:00:12 pppve1 kernel: [3375932.705624] [ 4160] 0 4160 89394 30149 29541 608 0 380928 0 0 pvescheduler
Jul 4 20:00:12 pppve1 kernel: [3375932.717864] [ 4710] 0 4710 1375 480 32 448 0 57344 0 0 agetty
Jul 4 20:00:12 pppve1 kernel: [3375932.729183] [ 5531] 0 5531 993913 567351 566647 704 0 5611520 0 0 kvm
Jul 4 20:00:12 pppve1 kernel: [3375932.740212] [ 6368] 0 6368 5512483 4229046 4228342 704 0 34951168 0 0 kvm
Jul 4 20:00:12 pppve1 kernel: [3375932.751255] [ 9796] 0 9796 1941 768 64 704 0 57344 0 0 lxc-start
Jul 4 20:00:12 pppve1 kernel: [3375932.762840] [ 9808] 100000 9808 3875 160 32 128 0 77824 0 0 init
Jul 4 20:00:12 pppve1 kernel: [3375932.774063] [ 11447] 100000 11447 9272 192 64 128 0 118784 0 0 rpcbind
Jul 4 20:00:12 pppve1 kernel: [3375932.785534] [ 11620] 100000 11620 45718 240 112 128 0 126976 0 0 rsyslogd
Jul 4 20:00:12 pppve1 kernel: [3375932.797241] [ 11673] 100000 11673 4758 195 35 160 0 81920 0 0 atd
Jul 4 20:00:12 pppve1 kernel: [3375932.808516] [ 11748] 100000 11748 6878 228 36 192 0 98304 0 0 cron
Jul 4 20:00:12 pppve1 kernel: [3375932.819868] [ 11759] 100102 11759 10533 257 65 192 0 122880 0 0 dbus-daemon
Jul 4 20:00:12 pppve1 kernel: [3375932.832328] [ 11765] 100000 11765 13797 315 155 160 0 143360 0 0 sshd
Jul 4 20:00:12 pppve1 kernel: [3375932.843547] [ 11989] 100104 11989 565602 19744 288 160 19296 372736 0 0 postgres
Jul 4 20:00:12 pppve1 kernel: [3375932.855266] [ 12169] 100104 12169 565938 537254 678 192 536384 4517888 0 0 postgres
Jul 4 20:00:12 pppve1 kernel: [3375932.866950] [ 12170] 100104 12170 565859 199654 550 224 198880 4296704 0 0 postgres
Jul 4 20:00:12 pppve1 kernel: [3375932.878525] [ 12171] 100104 12171 565859 4710 358 224 4128 241664 0 0 postgres
Jul 4 20:00:12 pppve1 kernel: [3375932.890252] [ 12172] 100104 12172 565962 7654 518 192 6944 827392 0 0 postgres
Jul 4 20:00:12 pppve1 kernel: [3375932.901845] [ 12173] 100104 12173 20982 742 518 224 0 200704 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375932.913421] [ 13520] 100000 13520 9045 192 128 64 0 114688 0 0 master
Jul 4 20:00:13 pppve1 kernel: [3375932.924809] [ 13536] 100100 13536 9601 320 128 192 0 126976 0 0 qmgr
Jul 4 20:00:13 pppve1 kernel: [3375932.936088] [ 13547] 100000 13547 3168 192 32 160 0 73728 0 0 getty
Jul 4 20:00:13 pppve1 kernel: [3375932.947424] [ 13548] 100000 13548 3168 160 32 128 0 73728 0 0 getty
Jul 4 20:00:13 pppve1 kernel: [3375932.958761] [1302486] 0 1302486 1941 768 96 672 0 53248 0 0 lxc-start
Jul 4 20:00:13 pppve1 kernel: [3375932.970490] [1302506] 100000 1302506 2115 128 32 96 0 65536 0 0 init
Jul 4 20:00:13 pppve1 kernel: [3375932.981999] [1302829] 100001 1302829 2081 128 0 128 0 61440 0 0 portmap
Jul 4 20:00:13 pppve1 kernel: [3375932.993763] [1302902] 100000 1302902 27413 160 64 96 0 122880 0 0 rsyslogd
Jul 4 20:00:13 pppve1 kernel: [3375933.005719] [1302953] 100000 1302953 117996 1654 1366 227 61 450560 0 0 apache2
Jul 4 20:00:13 pppve1 kernel: [3375933.017459] [1302989] 100000 1302989 4736 97 33 64 0 81920 0 0 atd
Jul 4 20:00:13 pppve1 kernel: [3375933.028905] [1303004] 100104 1303004 5843 64 32 32 0 94208 0 0 dbus-daemon
Jul 4 20:00:13 pppve1 kernel: [3375933.041272] [1303030] 100000 1303030 12322 334 110 224 0 139264 0 0 sshd
Jul 4 20:00:13 pppve1 kernel: [3375933.052755] [1303048] 100000 1303048 5664 64 32 32 0 94208 0 0 cron
Jul 4 20:00:13 pppve1 kernel: [3375933.064220] [1303255] 100000 1303255 9322 224 96 128 0 118784 0 0 master
Jul 4 20:00:13 pppve1 kernel: [3375933.075896] [1303284] 100101 1303284 9878 352 128 224 0 122880 0 0 qmgr
Jul 4 20:00:13 pppve1 kernel: [3375933.087405] [1303285] 100000 1303285 1509 32 0 32 0 61440 0 0 getty
Jul 4 20:00:13 pppve1 kernel: [3375933.099008] [1303286] 100000 1303286 1509 64 0 64 0 61440 0 0 getty
Jul 4 20:00:13 pppve1 kernel: [3375933.110571] [1420994] 33 1420994 21749 13271 12759 512 0 204800 0 0 spiceproxy work
Jul 4 20:00:13 pppve1 kernel: [3375933.123378] [1421001] 33 1421001 94055 37044 35892 1152 0 434176 0 0 pveproxy worker
Jul 4 20:00:13 pppve1 kernel: [3375933.136284] [1421002] 33 1421002 94055 36980 35860 1120 0 434176 0 0 pveproxy worker
Jul 4 20:00:13 pppve1 kernel: [3375933.149173] [1421003] 33 1421003 94055 37044 35892 1152 0 434176 0 0 pveproxy worker
Jul 4 20:00:13 pppve1 kernel: [3375933.162040] [2316827] 0 2316827 6820 1088 224 864 0 69632 0 -1000 systemd-udevd
Jul 4 20:00:13 pppve1 kernel: [3375933.174778] [2316923] 0 2316923 51282 2240 224 2016 0 438272 0 -250 systemd-journal
Jul 4 20:00:13 pppve1 kernel: [3375933.187768] [3148356] 0 3148356 32681 21120 19232 1888 0 249856 0 0 glpi-agent (tag
Jul 4 20:00:13 pppve1 kernel: [3375933.200481] [3053571] 0 3053571 19798 480 32 448 0 57344 0 0 pvefw-logger
Jul 4 20:00:13 pppve1 kernel: [3375933.212970] [3498513] 100033 3498513 119792 7207 2632 223 4352 516096 0 0 apache2
Jul 4 20:00:13 pppve1 kernel: [3375933.224713] [3498820] 100104 3498820 575918 235975 9351 160 226464 3424256 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375933.236579] [3500997] 100033 3500997 119889 7202 2594 192 4416 524288 0 0 apache2
Jul 4 20:00:13 pppve1 kernel: [3375933.248240] [3501657] 100104 3501657 571325 199025 6001 160 192864 2945024 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375933.260100] [3502514] 100033 3502514 119119 5907 2004 191 3712 503808 0 0 apache2
Jul 4 20:00:13 pppve1 kernel: [3375933.271772] [3503679] 100104 3503679 575295 211508 6612 192 204704 2953216 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375933.283619] [3515234] 100033 3515234 119042 6568 1960 192 4416 503808 0 0 apache2
Jul 4 20:00:13 pppve1 kernel: [3375933.295362] [3515420] 100104 3515420 569839 97579 4491 160 92928 2293760 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375933.307155] [3520282] 100033 3520282 119129 5416 2056 192 3168 495616 0 0 apache2
Jul 4 20:00:13 pppve1 kernel: [3375933.318923] [3520287] 100033 3520287 119015 5709 1894 167 3648 503808 0 0 apache2
Jul 4 20:00:13 pppve1 kernel: [3375933.330805] [3520288] 100033 3520288 119876 5961 2729 224 3008 507904 0 0 apache2
Jul 4 20:00:13 pppve1 kernel: [3375933.342648] [3521057] 100104 3521057 573824 46069 8341 128 37600 1830912 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375933.354567] [3521067] 100104 3521067 574768 99734 7446 96 92192 2134016 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375933.366512] [3521301] 100104 3521301 569500 174722 4194 160 170368 2482176 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375933.378484] [3532810] 100033 3532810 118740 4127 1727 160 2240 479232 0 0 apache2
Jul 4 20:00:13 pppve1 kernel: [3375933.390140] [3532933] 100033 3532933 118971 5064 1864 160 3040 503808 0 0 apache2
Jul 4 20:00:13 pppve1 kernel: [3375933.401854] [3534151] 100104 3534151 567344 168822 1686 160 166976 2408448 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375933.413852] [3535832] 100104 3535832 569005 41042 2578 128 38336 1150976 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375933.425919] [3550993] 100033 3550993 118029 1768 1544 224 0 425984 0 0 apache2
Jul 4 20:00:13 pppve1 kernel: [3375933.437868] [3560475] 107 3560475 10767 928 160 768 0 77824 0 0 pickup
Jul 4 20:00:13 pppve1 kernel: [3375933.449513] [3563017] 100101 3563017 9838 256 96 160 0 122880 0 0 pickup
Jul 4 20:00:13 pppve1 kernel: [3375933.461255] [3575085] 100100 3575085 9561 288 128 160 0 118784 0 0 pickup
Jul 4 20:00:13 pppve1 kernel: [3375933.473119] [3579986] 0 3579986 1367 384 0 384 0 49152 0 0 sleep
Jul 4 20:00:13 pppve1 kernel: [3375933.484646] [3579996] 100104 3579996 566249 5031 615 128 4288 450560 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375933.496645] [3580020] 0 3580020 91269 30310 29606 704 0 409600 0 0 pvescheduler
Jul 4 20:00:13 pppve1 kernel: [3375933.509585] [3580041] 0 3580041 5005 1920 640 1280 0 81920 0 100 systemd
Jul 4 20:00:13 pppve1 kernel: [3375933.521297] [3580044] 0 3580044 42685 1538 1218 320 0 102400 0 100 (sd-pam)
Jul 4 20:00:13 pppve1 kernel: [3375933.533226] [3580125] 100104 3580125 566119 5607 583 704 4320 446464 0 0 postgres
Jul 4 20:00:13 pppve1 kernel: [3375933.545245] [3580193] 0 3580193 4403 2368 384 1984 0 81920 0 0 sshd
Jul 4 20:00:13 pppve1 kernel: [3375933.556849] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=qemu.slice,mems_allowed=0,global_oom,task_memcg=/qemu.slice/121.scope,task=kvm,pid=6368,uid=0
Jul 4 20:00:13 pppve1 kernel: [3375933.573133] Out of memory: Killed process 6368 (kvm) total-vm:22049932kB, anon-rss:16913368kB, file-rss:2944kB, shmem-rss:0kB, UID:0 pgtables:34132kB oom_score_adj:0
Jul 4 20:00:15 pppve1 kernel: [3375935.378441] zd16: p1 p2 p3 < p5 p6 >
Jul 4 20:00:16 pppve1 kernel: [3375936.735383] oom_reaper: reaped process 6368 (kvm), now anon-rss:0kB, file-rss:32kB, shmem-rss:0kB
Jul 4 20:01:11 pppve1 kernel: [3375991.767379] vmbr0: port 5(tap121i0) entered disabled state
Jul 4 20:01:11 pppve1 kernel: [3375991.778143] tap121i0 (unregistering): left allmulticast mode
Jul 4 20:01:11 pppve1 kernel: [3375991.785976] vmbr0: port 5(tap121i0) entered disabled state
Jul 4 20:01:11 pppve1 kernel: [3375991.791555] zd128: p1
Jul 4 20:01:13 pppve1 kernel: [3375993.594688] zd176: p1 p2
> Makes few sense that OOM triggers in 64GB hosts with just 24GB configured in
> VMs and, probably, less real usage. IMHO it's not VMs what fill your memory
> up to the point of OOM, but some other process, ZFS ARC, maybe even some mem
> leak. Maybe some process is producing severe memory fragmentation.
i can confirm that server was doing some heavy I/O (backup), but AFAIK
nothing more.
Mandi! Roland
> it's a little bit weird that OOM kicks in with VMs <32GB RAM when you have 64GB
> take a closer look why this happens , i.e. why OOM thinks there is ram pressure
effectively server was running:
+ vm 100, 2GB
+ vm 120, 4GB
+ vm 121, 16GB
+ vm 127, 4GB
+ lxc 124, 2GB
+ lxc 125, 4GB
so exactly 32GB of RAM. But most of the VM/LXC barely arrived at half of the
allocated RAM...
Thanks.
--
More information about the pve-user
mailing list