<div style="line-height:1.7;color:#000000;font-size:14px;font-family:arial">Hi<div><br></div><div>Not easy to reproduce it. I traced it for a long time, It seems if the server have a heavy load it will be easier..</div><div><br></div><div>I tested with pve-qemu-kvm-1.3(with backup patches)..</div><div><br></div><div><br>Regards</div><div><br></div><div><br><div></div><div id="divNeteaseMailCard"></div><br>At 2013-02-25 13:28:43,"Dietmar Maurer" <dietmar@proxmox.com> wrote:<br> <blockquote id="isReplyContent" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<style></style>
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Is there a way to reproduce that behavior? You already tested with new pve-qemu-kvm (1.4)?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:pve-devel-bounces@pve.proxmox.com">pve-devel-bounces@pve.proxmox.com</a> [mailto:<a href="mailto:pve-devel-bounces@pve.proxmox.com">pve-devel-bounces@pve.proxmox.com</a>]
<b>On Behalf Of </b>lulin<br>
<b>Sent:</b> Samstag, 23. Februar 2013 11:33<br>
<b>To:</b> <a href="mailto:pve-devel@pve.proxmox.com">pve-devel@pve.proxmox.com</a><br>
<b>Subject:</b> [pve-devel] cpu 100% when get vmstaus<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">Hi<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">I don't know if it is a bug, sometimes when you shudown a VM, pvedaemon will have 100% cpu usage.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">bellow is 'top' output:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">------------------------------<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:black">Tasks: 761 total, 3 running, 758 sleeping, 0 stopped, 0 zombie<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:black">Cpu(s): 14.0%us, 2.8%sy, 0.0%ni, 83.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:black">Mem: 49378500k total, 45560112k used, 3818388k free, 17408k buffers<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:black">Swap: 24764408k total, 4106172k used, 20658236k free, 2291944k cached<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:black"> PID USER PR NI VIRT ;RES SHR S %CPU %MEM TIME+ COMMAND <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:black">31347 root 20 0 209m 41m 3664 R 99.6 0.1 44:36.22 pvedaemon <o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">-----------------------------<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">I print some logs to trace it, and find it is deadloop in QMPclient.pm:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><a href="https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/QMPClient.pm;h=4229e16c3b049572f2a706d9aa50136d694e0e61;hb=HEAD#l240" id="l240"><span style="color:#999999;text-decoration:none">240</span></a>
for (;;)&n<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black">bsp;{<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><a href="https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/QMPClient.pm;h=4229e16c3b049572f2a706d9aa50136d694e0e61;hb=HEAD#l241" id="l241"><span style="color:#999999;text-decoration:none">241</span></a>
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><a href="https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/QMPClient.pm;h=4229e16c3b049572f2a706d9aa50136d694e0e61;hb=HEAD#l242" id="l242"><span style="color:#999999;text-decoration:none"> 242</span></a>
$running = &$check_queue($self);<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><a href="https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/QMPClient.pm;h=4229e16c3b%20049572f2a706d9aa50136d694e0e61;hb=HEAD#l243" id="l243"><span style="color:#999999;text-decoration:none">243</span></a>
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><a href="https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/QMPClient.pm;h=4229e16c3b049572f2a706d9aa50136d694e0e61;hb=HEAD#l244" id="l244"><span style="color:#999999;text-decoration:none"> 244</span></a>
last if !$running;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><a href="https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/QMPClient.pm;h=4229e16c3b049572f2a706d9aa50136d694e0e61;hb=HEAD#l245" id="l245"><span style="color:#999999;text-decoration:none">245</span></a>
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><a href="https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/QMPClient.pm;h=4229e16c3b049572f2a706d9aa50136d694e0e61;hb=HEAD#l246"><span style="color:#999999;text-decoration:none"> 246</span></a>
$self->{mux}->loop;<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><a href="https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/QMPClient.pm;h=4229e16c3b049572f2a706d9aa50136d694e0e61;hb=HEAD#l247" id="l247"><span style="color:#999999;text-decoration:none">247</span></a>
}<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Courier New";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">it seems if a vm is shutdown, the QMP socket can't have a close notify, and use 'netstat' command you can find a connection always stay in CLOSE_WAIT:</span><span style="font-family:"Courier New";color:black"><o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">[root@OVPMAIN ~]# netstat -antop |grep pvedaemon<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">tcp 0 0 127.0.0.1:85 0.0.0.0:* LISTEN 1018/pvedaemon work off (0.00/0/0)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">tcp 1 0 127.0.0.1:85 127.0.0.1:49246 CLOSE_WAIT 31347/pvedaemon wor off (0.00/0/0)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">I tried to change the tcp_keepalive_* param in /proc, but not work since keepalive is disabled by default...<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">Any one have some idea?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">Regards<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black">Lenny<o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Arial","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
</div>
</blockquote></div></div><br><br><span title="neteasefooter"><span id="netease_mail_footer"></span></span>