[pve-devel] pve-kernel package : add irqbalance as recommended ? (like debian linux-image package)

[Alexandre DERUMIER]

>>Sadly there is various Hardware and chips out there crashing with irqbalance. That's why I removed it from all machines and tune where neede
Oh ok, didn't knwon that. Never had problem with my dell servers with it.
So, maybe it's better to not install it by default. (stability first)



>>By the way how did you get 300000 iops? How many clients and threads involved?
1 vm, multiple disk with 1 iothread by disk, with jemalloc

iops 4k randread
-----------------

1 disk      41278
2 disks     75781
4 disks     195351
8 disks     294241
15 disks    298199


This is with librbd, jemalloc, debug and cephx disabled (2x 10 cores 3,1ghz E5 v3)



Without irqbalancer I was stuck around 100000iops (most network irqs going to same core)


----- Mail original -----
De: "Stefan Priebe" <s.priebe at profihost.ag>
À: "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Samedi 20 Février 2016 08:55:07
Objet: Re: [pve-devel] pve-kernel package : add irqbalance as	recommended	? (like debian linux-image package)

Am 20.02.2016 um 08:25 schrieb Alexandre DERUMIER <aderumier at odiso.com>: 

>>> Some articles, for instance https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf, 
>>> explicitly recommend disabling irqbalance when 10GbE is involved. 
>>> 
>>> Do you know if this is still true today? After all, the paper is from 2009. 
> 
> Well, the article is about to disabling irqbalance AND manually binding cpus on network interfaces. 
> 
> Manual binding is better because you can fine tuning. 
> 
> But using irqbalance vs do nothing, irqbalance wins. 
> 
> I have seen a lot of system, using only cpu0 for network interrupts for example. 
> 
> Benching ceph, I was stuck around 100000iops 4k (client side/proxmox host) without irqbalance, and 300000 iops with irqbalance. 
> (This was with 10gbe mellanox cards) 

Sadly there is various Hardware and chips out there crashing with irqbalance. That's why I removed it from all machines and tune where needed. Have a look at red hats tuned which is much better and open source at fedora. 

By the way how did you get 300000 iops? How many clients and threads involved? 

Greets, 
Stefan 

> 
> 
> ----- Mail original ----- 
> De: "Martin Waschbüsch" <service at waschbuesch.it> 
> À: "pve-devel" <pve-devel at pve.proxmox.com> 
> Envoyé: Vendredi 19 Février 2016 15:42:09 
> Objet: Re: [pve-devel] pve-kernel package : add irqbalance as recommended ? (like debian linux-image package) 
> 
> Hi Alexandre, 
> 
>> Am 19.02.2016 um 15:06 schrieb Alexandre DERUMIER <aderumier at odiso.com>: 
>> 
>> Hi, 
>> 
>> I think it could be great to add irqbalance as recommends package for pve-kernel. 
>> 
>> I have seen a lot of improvement, mainly with network access (ceph for example), 
>> when a lot of network interrupts occurs and goes to the same cpu. 
> 
> Some articles, for instance https://www.kernel.org/doc/ols/2009/ols2009-pages-169-184.pdf, 
> explicitly recommend disabling irqbalance when 10GbE is involved. 
> 
> Do you know if this is still true today? After all, the paper is from 2009. 
> 
> Best, 
> 
> Martin 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
> 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel