[PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 released!)

Fri Apr 29 10:31:46 CEST 2016

Ok, so maybe it's a kvm module regression in kernel.

Just to be sure, can you check that vhost_net module is loaded ?

# lsmod|grep vhost_net

and also that you have a process

[vhost-xxxx], where xxxx is the pid of the current qemu-kvm process

and lastly, try to use perf, to see which system calls are using cpu

> #apt-get install linux-tools-4.4
> 
> and do a 
> 
> #perf top

----- Mail original -----
De: "Gilou" <contact+dev at gilouweb.com>
À: "proxmoxve" <pve-user at pve.proxmox.com>
Envoyé: Vendredi 29 Avril 2016 08:48:20
Objet: Re: [PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 released!)

Le 29/04/2016 00:47, Alexandre DERUMIER a écrit : 
> Can you try with previous kernel 4.2 ? (and keep proxmox 4.2 ) 
> 
> I would like to known if it's a regression in qemu or kernel. 

As I said, I did that, the bug is not there (or not that obvious). On a 
4.2.6 kernel, using qemu 2.5, I get ~30% of a core for 1 Gbps, which 
seems reasonnable. 

I also tried on VirtualBox/KVM, on Ubuntu 16.04 host, and the bug does 
not appear there. I can try on qemu probably too, but it seems to me the 
kernel is at fault here. 

> 
> 
> also, you can try to install 
> 
> #apt-get install linux-tools-4.4 
> 
> and do a 
> 
> #perf top 
> 
> 
> you should see which system call use a lot of cpu 

Here's the result, while spitting ~1 Gbps on the LAN: 
https://framapic.org/eT9JbqGsqSEK/txwSUJRpiIV3.png 
On the left, perf top, on the right top with relevant fields, showing 
all threads. 

so read_tsc using ~23%, ktime_get ~10%, and single thread (even if the 
VM has multiple cores) using 100% on the host. So maybe it's not 
virtio-net directly. I have a lot of CPU used on a E1000 as well, but 
that didn't strike me as obvious, as I expected this. 

So you know: 
# uname -a 
Linux px1 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux 

# pveversion -v 
proxmox-ve: 4.2-48 (running kernel: 4.4.6-1-pve) 
pve-manager: 4.2-2 (running version: 4.2-2/725d76f0) 
pve-kernel-4.4.6-1-pve: 4.4.6-48 
pve-kernel-4.2.6-1-pve: 4.2.6-36 
lvm2: 2.02.116-pve2 
corosync-pve: 2.3.5-2 
libqb0: 1.0-1 
pve-cluster: 4.0-39 
qemu-server: 4.0-72 
pve-firmware: 1.1-8 
libpve-common-perl: 4.0-59 
libpve-access-control: 4.0-16 
libpve-storage-perl: 4.0-50 
pve-libspice-server1: 0.12.5-2 
vncterm: 1.2-1 
pve-qemu-kvm: 2.5-14 
pve-container: 1.0-62 
pve-firewall: 2.0-25 
pve-ha-manager: 1.0-28 
ksm-control-daemon: 1.2-1 
glusterfs-client: 3.5.2-2+deb8u1 
lxc-pve: 1.1.5-7 
lxcfs: 2.0.0-pve2 
cgmanager: 0.39-pve1 
criu: 1.6.0-1 
zfsutils: 0.6.5-pve9~jessie 

If I can troubleshoot that further, let me know... 

Cheers, 

Gilles Pietri 

> 
> 
> 
> 
> 
> ----- Mail original ----- 
> De: "Gilou" <contact+dev at gilouweb.com> 
> À: "proxmoxve" <pve-user at pve.proxmox.com> 
> Envoyé: Jeudi 28 Avril 2016 19:31:19 
> Objet: Re: [PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 released!) 
> 
> Le 28/04/2016 19:18, Gilou a écrit : 
>> Le 27/04/2016 15:35, Martin Maurer a écrit : 
>>> Hi all! 
>>> 
>>> We are proud to announce the final release of our Proxmox VE 4.2 with 
>>> the new GUI. 
>>> 
>>> The Sencha Ext JS 6 framework brings a modern 'flat design' look and 
>>> feel to the Proxmox VE GUI with a reworked icon set providing 
>>> consistency and an improved user experience. 
>>> 
>>> Watch our short introduction video - What's new in Proxmox VE 4.2 
>>> http://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-4-2 
>>> 
>>> 
>>> This point release is based on latest and greatest Debian Jessie and a 
>>> 4.4.6 Linux kernel, based on Ubuntu 16.04 LTS kernel. 
>> 
>> Hi, 
>> 
>> I seem to have a regression, and a weird one I see neither on Proxmox 
>> 4.1 4.2.6's kernel, nor on Ubuntu 4.4.0, or 
>> 
>> Context: Ubuntu Guest VM using Ubuntu 16.04 on a virtio_net device 
>> (tested also on Debian 8, and CentOS7), connected on a bridge. 
>> 
>> on 4.1 : 1 Gbps bandwidth, ~1 GHz of CPU 
>> on 4.2 : 1 Gbps bandwidth, ~3 GHz of CPU 
>> 
>> Tested using transfer to /dev/null & iperf, so no disk involved. 
>> 
>> I do not have the issue on ubuntu 4.4.0 using virtualbox/virtio-net. 
>> 
>> Haven't tested further, but this is an isssue. 
> 
> Looked into it, it's really 100% of a core, in the following setups: 
> - single CPU on the VM 
> - local or remote traffic (local goes up to 24 Gbps, remote is 1 Gbps) 
> - no matter what distro (haven't tested Windows) 
> 
> Has anyone noticed something like this or is this something specific? 
> 
> Regards, 
> 
> 
>> 
>> Regards, 
>> Gilles Pietri 
>> 
>> 
>>> 
>>> We included as always countless small improvements and bugfixes, like 
>>> the ability to use Let´s Encrypt certificates. 
>>> 
>>> Release notes 
>>> http://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_4.2 
>>> 
>>> ISO Download 
>>> 
>>> http://www.proxmox.com/en/downloads 
>>> 
>>> Alternate ISO download: 
>>> http://download.proxmox.com/iso/ 
>>> 
>>> Upgrading 
>>> http://pve.proxmox.com/wiki/Downloads 
>>> 
>>> Bugtracker 
>>> https://bugzilla.proxmox.com 
>>> 
>>> A big THANK-YOU to our active community for all feedback, testing, bug 
>>> reporting and patch submissions. 
>>> 
>> 
>> _______________________________________________ 
>> pve-user mailing list 
>> pve-user at pve.proxmox.com 
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user 
>> 
> 
> _______________________________________________ 
> pve-user mailing list 
> pve-user at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user 
> _______________________________________________ 
> pve-user mailing list 
> pve-user at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user 
> 

_______________________________________________ 
pve-user mailing list 
pve-user at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user 