[PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 released!)
Alexandre DERUMIER
aderumier at odiso.com
Sat Apr 30 07:20:56 CEST 2016
ok,
can you check with perf command ?
check both qemu and vhost process
"perf top -p pidofqemu"
"perf top -p pidofvhost"
----- Mail original -----
De: "Gilou" <contact+dev at gilouweb.com>
À: "proxmoxve" <pve-user at pve.proxmox.com>
Envoyé: Vendredi 29 Avril 2016 13:53:17
Objet: Re: [PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 released!)
Le 29/04/2016 10:31, Alexandre DERUMIER a écrit :
> Ok, so maybe it's a kvm module regression in kernel.
>
> Just to be sure, can you check that vhost_net module is loaded ?
>
> # lsmod|grep vhost_net
>
> and also that you have a process
>
> [vhost-xxxx], where xxxx is the pid of the current qemu-kvm process
Hi,
# lsmod | grep vhost_net
vhost_net 20480 1
vhost 32768 1 vhost_net
macvtap 20480 1 vhost_net
root at px3:~# ps aux | grep vhost
root 3308 33.8 3.4 1181136 560584 ? Sl 13:41 1:59
/usr/bin/kvm -id 3004 [...]
net-pci,mac=32:36:65:35:30:62,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
root 3330 1.2 0.0 0 0 ? S 13:41 0:04
[vhost-3308]
So: yes, vhost_net is loaded, and I do have a vhost-3308 associated with
my VM. Yet, the CPU is at 100% for a 1 Gbps traffic :(
@+
Gilles
>
>
>
>
> and lastly, try to use perf, to see which system calls are using cpu
>
>> #apt-get install linux-tools-4.4
>>
>> and do a
>>
>> #perf top
>
>
> ----- Mail original -----
> De: "Gilou" <contact+dev at gilouweb.com>
> À: "proxmoxve" <pve-user at pve.proxmox.com>
> Envoyé: Vendredi 29 Avril 2016 08:48:20
> Objet: Re: [PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 released!)
>
> Le 29/04/2016 00:47, Alexandre DERUMIER a écrit :
>> Can you try with previous kernel 4.2 ? (and keep proxmox 4.2 )
>>
>> I would like to known if it's a regression in qemu or kernel.
>
> As I said, I did that, the bug is not there (or not that obvious). On a
> 4.2.6 kernel, using qemu 2.5, I get ~30% of a core for 1 Gbps, which
> seems reasonnable.
>
> I also tried on VirtualBox/KVM, on Ubuntu 16.04 host, and the bug does
> not appear there. I can try on qemu probably too, but it seems to me the
> kernel is at fault here.
>
>>
>>
>> also, you can try to install
>>
>> #apt-get install linux-tools-4.4
>>
>> and do a
>>
>> #perf top
>>
>>
>> you should see which system call use a lot of cpu
>
>
> Here's the result, while spitting ~1 Gbps on the LAN:
> https://framapic.org/eT9JbqGsqSEK/txwSUJRpiIV3.png
> On the left, perf top, on the right top with relevant fields, showing
> all threads.
>
> so read_tsc using ~23%, ktime_get ~10%, and single thread (even if the
> VM has multiple cores) using 100% on the host. So maybe it's not
> virtio-net directly. I have a lot of CPU used on a E1000 as well, but
> that didn't strike me as obvious, as I expected this.
>
> So you know:
> # uname -a
> Linux px1 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux
>
> # pveversion -v
> proxmox-ve: 4.2-48 (running kernel: 4.4.6-1-pve)
> pve-manager: 4.2-2 (running version: 4.2-2/725d76f0)
> pve-kernel-4.4.6-1-pve: 4.4.6-48
> pve-kernel-4.2.6-1-pve: 4.2.6-36
> lvm2: 2.02.116-pve2
> corosync-pve: 2.3.5-2
> libqb0: 1.0-1
> pve-cluster: 4.0-39
> qemu-server: 4.0-72
> pve-firmware: 1.1-8
> libpve-common-perl: 4.0-59
> libpve-access-control: 4.0-16
> libpve-storage-perl: 4.0-50
> pve-libspice-server1: 0.12.5-2
> vncterm: 1.2-1
> pve-qemu-kvm: 2.5-14
> pve-container: 1.0-62
> pve-firewall: 2.0-25
> pve-ha-manager: 1.0-28
> ksm-control-daemon: 1.2-1
> glusterfs-client: 3.5.2-2+deb8u1
> lxc-pve: 1.1.5-7
> lxcfs: 2.0.0-pve2
> cgmanager: 0.39-pve1
> criu: 1.6.0-1
> zfsutils: 0.6.5-pve9~jessie
>
> If I can troubleshoot that further, let me know...
>
> Cheers,
>
> Gilles Pietri
>
>
>>
>>
>>
>>
>>
>> ----- Mail original -----
>> De: "Gilou" <contact+dev at gilouweb.com>
>> À: "proxmoxve" <pve-user at pve.proxmox.com>
>> Envoyé: Jeudi 28 Avril 2016 19:31:19
>> Objet: Re: [PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 released!)
>>
>> Le 28/04/2016 19:18, Gilou a écrit :
>>> Le 27/04/2016 15:35, Martin Maurer a écrit :
>>>> Hi all!
>>>>
>>>> We are proud to announce the final release of our Proxmox VE 4.2 with
>>>> the new GUI.
>>>>
>>>> The Sencha Ext JS 6 framework brings a modern 'flat design' look and
>>>> feel to the Proxmox VE GUI with a reworked icon set providing
>>>> consistency and an improved user experience.
>>>>
>>>> Watch our short introduction video - What's new in Proxmox VE 4.2
>>>> http://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-4-2
>>>>
>>>>
>>>> This point release is based on latest and greatest Debian Jessie and a
>>>> 4.4.6 Linux kernel, based on Ubuntu 16.04 LTS kernel.
>>>
>>> Hi,
>>>
>>> I seem to have a regression, and a weird one I see neither on Proxmox
>>> 4.1 4.2.6's kernel, nor on Ubuntu 4.4.0, or
>>>
>>> Context: Ubuntu Guest VM using Ubuntu 16.04 on a virtio_net device
>>> (tested also on Debian 8, and CentOS7), connected on a bridge.
>>>
>>> on 4.1 : 1 Gbps bandwidth, ~1 GHz of CPU
>>> on 4.2 : 1 Gbps bandwidth, ~3 GHz of CPU
>>>
>>> Tested using transfer to /dev/null & iperf, so no disk involved.
>>>
>>> I do not have the issue on ubuntu 4.4.0 using virtualbox/virtio-net.
>>>
>>> Haven't tested further, but this is an isssue.
>>
>> Looked into it, it's really 100% of a core, in the following setups:
>> - single CPU on the VM
>> - local or remote traffic (local goes up to 24 Gbps, remote is 1 Gbps)
>> - no matter what distro (haven't tested Windows)
>>
>> Has anyone noticed something like this or is this something specific?
>>
>> Regards,
>>
>>
>>>
>>> Regards,
>>> Gilles Pietri
>>>
>>>
>>>>
>>>> We included as always countless small improvements and bugfixes, like
>>>> the ability to use Let´s Encrypt certificates.
>>>>
>>>> Release notes
>>>> http://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_4.2
>>>>
>>>> ISO Download
>>>>
>>>> http://www.proxmox.com/en/downloads
>>>>
>>>> Alternate ISO download:
>>>> http://download.proxmox.com/iso/
>>>>
>>>> Upgrading
>>>> http://pve.proxmox.com/wiki/Downloads
>>>>
>>>> Bugtracker
>>>> https://bugzilla.proxmox.com
>>>>
>>>> A big THANK-YOU to our active community for all feedback, testing, bug
>>>> reporting and patch submissions.
>>>>
>>>
>>> _______________________________________________
>>> pve-user mailing list
>>> pve-user at pve.proxmox.com
>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>
>>
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
_______________________________________________
pve-user mailing list
pve-user at pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
More information about the pve-user
mailing list