[pve-devel] pve-kernel package : add irqbalance as recommended ? (like debian linux-image package)

[Stefan Priebe - Profihost AG]

Am 20.02.2016 um 09:15 schrieb Alexandre DERUMIER <aderumier at odiso.com>:

>>> Sadly there is various Hardware and chips out there crashing with irqbalance. That's why I removed it from all machines and tune where neede
> Oh ok, didn't knwon that. Never had problem with my dell servers with it.
> So, maybe it's better to not install it by default. (stability first)
> 
> 
> 
>>> By the way how did you get 300000 iops? How many clients and threads involved?
> 1 vm, multiple disk with 1 iothread by disk, with jemalloc
> 
> iops 4k randread
> -----------------
> 
> 1 disk      41278
> 2 disks     75781
> 4 disks     195351
> 8 disks     294241
> 15 disks    298199

Which disk size? Did you fill them with random data? Virtio, virtio scsi or dataplane?

> 
> 
> This is with librbd, jemalloc, debug and cephx disabled (2x 10 cores 3,1ghz E5 v3)
> 
> 
> 
> Without irqbalancer I was stuck around 100000iops (most network irqs going to same core)
> 
> 
> ----- Mail original -----
> De: "Stefan Priebe" <s.priebe at profihost.ag>
> À: "pve-devel" <pve-devel at pve.proxmox.com>
> Envoyé: Samedi 20 Février 2016 08:55:07
> Objet: Re: [pve-devel] pve-kernel package : add irqbalance as    recommended    ? (like debian linux-image package)
> 
> Am 20.02.2016 um 08:25 schrieb Alexandre DERUMIER <aderumier at odiso.com>: 
> 
>>>> Some articles, for instance https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf, 
>>>> explicitly recommend disabling irqbalance when 10GbE is involved. 
>>>> 
>>>> Do you know if this is still true today? After all, the paper is from 2009.
>> 
>> Well, the article is about to disabling irqbalance AND manually binding cpus on network interfaces. 
>> 
>> Manual binding is better because you can fine tuning. 
>> 
>> But using irqbalance vs do nothing, irqbalance wins. 
>> 
>> I have seen a lot of system, using only cpu0 for network interrupts for example. 
>> 
>> Benching ceph, I was stuck around 100000iops 4k (client side/proxmox host) without irqbalance, and 300000 iops with irqbalance. 
>> (This was with 10gbe mellanox cards)
> 
> Sadly there is various Hardware and chips out there crashing with irqbalance. That's why I removed it from all machines and tune where needed. Have a look at red hats tuned which is much better and open source at fedora. 
> 
> By the way how did you get 300000 iops? How many clients and threads involved? 
> 
> Greets, 
> Stefan 
> 
>> 
>> 
>> ----- Mail original ----- 
>> De: "Martin Waschbüsch" <service at waschbuesch.it> 
>> À: "pve-devel" <pve-devel at pve.proxmox.com> 
>> Envoyé: Vendredi 19 Février 2016 15:42:09 
>> Objet: Re: [pve-devel] pve-kernel package : add irqbalance as recommended ? (like debian linux-image package) 
>> 
>> Hi Alexandre, 
>> 
>>> Am 19.02.2016 um 15:06 schrieb Alexandre DERUMIER <aderumier at odiso.com>: 
>>> 
>>> Hi, 
>>> 
>>> I think it could be great to add irqbalance as recommends package for pve-kernel. 
>>> 
>>> I have seen a lot of improvement, mainly with network access (ceph for example), 
>>> when a lot of network interrupts occurs and goes to the same cpu.
>> 
>> Some articles, for instance https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf, 
>> explicitly recommend disabling irqbalance when 10GbE is involved. 
>> 
>> Do you know if this is still true today? After all, the paper is from 2009. 
>> 
>> Best, 
>> 
>> Martin 
>> _______________________________________________ 
>> pve-devel mailing list 
>> pve-devel at pve.proxmox.com 
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
>> 
>> _______________________________________________ 
>> pve-devel mailing list 
>> pve-devel at pve.proxmox.com 
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel