[PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 released!)
Gilou
contact+dev at gilouweb.com
Tue May 3 12:40:08 CEST 2016
Le 30/04/2016 07:20, Alexandre DERUMIER a écrit :
> ok,
>
> can you check with perf command ?
>
> check both qemu and vhost process
>
>
> "perf top -p pidofqemu"
>
> "perf top -p pidofvhost"
OK, I did it, and hit the bug again, on a different hardware.
Context is:
Ubuntu 16.04 VM, single core, host CPU, using virtio-net to blast at 1
Gbps through the physical NIC. I still have 100% of a core used by the
kvm process, and the perf top results are here:
https://gist.github.com/gilou/15b620a7a067fd1d58a7616942e025b4#file-perf_virtionet_4-4-txt
This was done again on a 4.1 updated to a 4.2. I will probably try on a
fresh 4.2 install to check it out, but I doubt it will change much.
If I fall back to the 4.2.6 kernel, bug isn't there.
I can grant you access to the host should you need that, but I'm pretty
sure anybody can reproduce that issue. Which brings quite some worries.
Regards,
Gilles Pietri
>
> ----- Mail original -----
> De: "Gilou" <contact+dev at gilouweb.com>
> À: "proxmoxve" <pve-user at pve.proxmox.com>
> Envoyé: Vendredi 29 Avril 2016 13:53:17
> Objet: Re: [PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 released!)
>
> Le 29/04/2016 10:31, Alexandre DERUMIER a écrit :
>> Ok, so maybe it's a kvm module regression in kernel.
>>
>> Just to be sure, can you check that vhost_net module is loaded ?
>>
>> # lsmod|grep vhost_net
>>
>> and also that you have a process
>>
>> [vhost-xxxx], where xxxx is the pid of the current qemu-kvm process
>
> Hi,
>
> # lsmod | grep vhost_net
> vhost_net 20480 1
> vhost 32768 1 vhost_net
> macvtap 20480 1 vhost_net
>
> root at px3:~# ps aux | grep vhost
> root 3308 33.8 3.4 1181136 560584 ? Sl 13:41 1:59
> /usr/bin/kvm -id 3004 [...]
> net-pci,mac=32:36:65:35:30:62,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
> root 3330 1.2 0.0 0 0 ? S 13:41 0:04
> [vhost-3308]
>
>
> So: yes, vhost_net is loaded, and I do have a vhost-3308 associated with
> my VM. Yet, the CPU is at 100% for a 1 Gbps traffic :(
>
> @+
> Gilles
>
>
>>
>>
>>
>>
>> and lastly, try to use perf, to see which system calls are using cpu
>>
>>> #apt-get install linux-tools-4.4
>>>
>>> and do a
>>>
>>> #perf top
>>
>>
>> ----- Mail original -----
>> De: "Gilou" <contact+dev at gilouweb.com>
>> À: "proxmoxve" <pve-user at pve.proxmox.com>
>> Envoyé: Vendredi 29 Avril 2016 08:48:20
>> Objet: Re: [PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 released!)
>>
>> Le 29/04/2016 00:47, Alexandre DERUMIER a écrit :
>>> Can you try with previous kernel 4.2 ? (and keep proxmox 4.2 )
>>>
>>> I would like to known if it's a regression in qemu or kernel.
>>
>> As I said, I did that, the bug is not there (or not that obvious). On a
>> 4.2.6 kernel, using qemu 2.5, I get ~30% of a core for 1 Gbps, which
>> seems reasonnable.
>>
>> I also tried on VirtualBox/KVM, on Ubuntu 16.04 host, and the bug does
>> not appear there. I can try on qemu probably too, but it seems to me the
>> kernel is at fault here.
>>
>>>
>>>
>>> also, you can try to install
>>>
>>> #apt-get install linux-tools-4.4
>>>
>>> and do a
>>>
>>> #perf top
>>>
>>>
>>> you should see which system call use a lot of cpu
>>
>>
>> Here's the result, while spitting ~1 Gbps on the LAN:
>> https://framapic.org/eT9JbqGsqSEK/txwSUJRpiIV3.png
>> On the left, perf top, on the right top with relevant fields, showing
>> all threads.
>>
>> so read_tsc using ~23%, ktime_get ~10%, and single thread (even if the
>> VM has multiple cores) using 100% on the host. So maybe it's not
>> virtio-net directly. I have a lot of CPU used on a E1000 as well, but
>> that didn't strike me as obvious, as I expected this.
>>
>> So you know:
>> # uname -a
>> Linux px1 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 GNU/Linux
>>
>> # pveversion -v
>> proxmox-ve: 4.2-48 (running kernel: 4.4.6-1-pve)
>> pve-manager: 4.2-2 (running version: 4.2-2/725d76f0)
>> pve-kernel-4.4.6-1-pve: 4.4.6-48
>> pve-kernel-4.2.6-1-pve: 4.2.6-36
>> lvm2: 2.02.116-pve2
>> corosync-pve: 2.3.5-2
>> libqb0: 1.0-1
>> pve-cluster: 4.0-39
>> qemu-server: 4.0-72
>> pve-firmware: 1.1-8
>> libpve-common-perl: 4.0-59
>> libpve-access-control: 4.0-16
>> libpve-storage-perl: 4.0-50
>> pve-libspice-server1: 0.12.5-2
>> vncterm: 1.2-1
>> pve-qemu-kvm: 2.5-14
>> pve-container: 1.0-62
>> pve-firewall: 2.0-25
>> pve-ha-manager: 1.0-28
>> ksm-control-daemon: 1.2-1
>> glusterfs-client: 3.5.2-2+deb8u1
>> lxc-pve: 1.1.5-7
>> lxcfs: 2.0.0-pve2
>> cgmanager: 0.39-pve1
>> criu: 1.6.0-1
>> zfsutils: 0.6.5-pve9~jessie
>>
>> If I can troubleshoot that further, let me know...
>>
>> Cheers,
>>
>> Gilles Pietri
>>
>>
>>>
>>>
>>>
>>>
>>>
>>> ----- Mail original -----
>>> De: "Gilou" <contact+dev at gilouweb.com>
>>> À: "proxmoxve" <pve-user at pve.proxmox.com>
>>> Envoyé: Jeudi 28 Avril 2016 19:31:19
>>> Objet: Re: [PVE-User] Regression on virtio-net ? (Was: Re: Proxmox VE 4.2 released!)
>>>
>>> Le 28/04/2016 19:18, Gilou a écrit :
>>>> Le 27/04/2016 15:35, Martin Maurer a écrit :
>>>>> Hi all!
>>>>>
>>>>> We are proud to announce the final release of our Proxmox VE 4.2 with
>>>>> the new GUI.
>>>>>
>>>>> The Sencha Ext JS 6 framework brings a modern 'flat design' look and
>>>>> feel to the Proxmox VE GUI with a reworked icon set providing
>>>>> consistency and an improved user experience.
>>>>>
>>>>> Watch our short introduction video - What's new in Proxmox VE 4.2
>>>>> http://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-4-2
>>>>>
>>>>>
>>>>> This point release is based on latest and greatest Debian Jessie and a
>>>>> 4.4.6 Linux kernel, based on Ubuntu 16.04 LTS kernel.
>>>>
>>>> Hi,
>>>>
>>>> I seem to have a regression, and a weird one I see neither on Proxmox
>>>> 4.1 4.2.6's kernel, nor on Ubuntu 4.4.0, or
>>>>
>>>> Context: Ubuntu Guest VM using Ubuntu 16.04 on a virtio_net device
>>>> (tested also on Debian 8, and CentOS7), connected on a bridge.
>>>>
>>>> on 4.1 : 1 Gbps bandwidth, ~1 GHz of CPU
>>>> on 4.2 : 1 Gbps bandwidth, ~3 GHz of CPU
>>>>
>>>> Tested using transfer to /dev/null & iperf, so no disk involved.
>>>>
>>>> I do not have the issue on ubuntu 4.4.0 using virtualbox/virtio-net.
>>>>
>>>> Haven't tested further, but this is an isssue.
>>>
>>> Looked into it, it's really 100% of a core, in the following setups:
>>> - single CPU on the VM
>>> - local or remote traffic (local goes up to 24 Gbps, remote is 1 Gbps)
>>> - no matter what distro (haven't tested Windows)
>>>
>>> Has anyone noticed something like this or is this something specific?
>>>
>>> Regards,
>>>
>>>
>>>>
>>>> Regards,
>>>> Gilles Pietri
>>>>
>>>>
>>>>>
>>>>> We included as always countless small improvements and bugfixes, like
>>>>> the ability to use Let´s Encrypt certificates.
>>>>>
>>>>> Release notes
>>>>> http://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_4.2
>>>>>
>>>>> ISO Download
>>>>>
>>>>> http://www.proxmox.com/en/downloads
>>>>>
>>>>> Alternate ISO download:
>>>>> http://download.proxmox.com/iso/
>>>>>
>>>>> Upgrading
>>>>> http://pve.proxmox.com/wiki/Downloads
>>>>>
>>>>> Bugtracker
>>>>> https://bugzilla.proxmox.com
>>>>>
>>>>> A big THANK-YOU to our active community for all feedback, testing, bug
>>>>> reporting and patch submissions.
>>>>>
>>>>
>>>> _______________________________________________
>>>> pve-user mailing list
>>>> pve-user at pve.proxmox.com
>>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>>
>>>
>>> _______________________________________________
>>> pve-user mailing list
>>> pve-user at pve.proxmox.com
>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>> _______________________________________________
>>> pve-user mailing list
>>> pve-user at pve.proxmox.com
>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>
>>
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
More information about the pve-user
mailing list