[PVE-User] high cpu load on 100mbits/sec download with virtio nic

[Maxime AUGER]

Hello proxmox community,

we are currently testing proxmox
the objective is to evaluate if it can support hundred debian8 virtual routers at low speed
(between 10mbits/sec and 50mbits/sec each one)

I'm doubtful about the performance I get.

On proxmox HOST a single wget download consume 12% single CPU load at 11MBytes/sec (1518 packets size)
On proxmox KVM GUEST the same test consume 25% single vcpu load on guest face and 30% of 2x CPU on host face
On our old vmware the same test consume 4% GUEST single vcpu and 7% 1x CPU on HOST side

On proxmox TOP output return a lot of CPU time consumed in HOST kernel space
GUEST run on debian8 with 2 vcpus and a virtio nic
proxmox HOST is a HP DL380G7 2x sockets with a total of 8 cores with 256gb RAM

There are no dropped, replayed packets and no special network queuing stuff
No improvement when increasing receive buffer or incresing IRQ backpressure on host side
no improvement when enabling 2 queues on virtio nic

cross tests :
same behavior on debian8 or 9 with qemu-kvm/libvirt
I have benched with Network bridge (nat and L2 fwd), macvtap, even worse (90% CPU on host) ...
disabling hardware offload TSO/GSO/checksumming, not better ...

at this level of CPU load I can't believe that this is normal behavior.
is this the normal proxmox performance ?