[pve-devel] less cores more iops / speed

[Alexandre DERUMIER]

maybe this is debug perfcounter ? 





----- Mail original ----- 

De: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
À: "Alexandre DERUMIER" <aderumier at odiso.com> 
Cc: pve-devel at pve.proxmox.com, ceph-devel at vger.kernel.org 
Envoyé: Lundi 12 Novembre 2012 16:33:48 
Objet: Re: [pve-devel] less cores more iops / speed 

Adding this to ceph.conf on kvm host adds another 2000 iops (20.000 
iop/s with one VM). I'm sure most of them are useless on a client kvm / 
rbd host but i don't know which one makes sense ;-) 

[global] 
debug ms = 0/0 
debug rbd = 0/0 
debug lockdep = 0/0 
debug context = 0/0 
debug crush = 0/0 
debug buffer = 0/0 
debug timer = 0/0 
debug journaler = 0/0 
debug osd = 0/0 
debug optracker = 0/0 
debug objclass = 0/0 
debug filestore = 0/0 
debug journal = 0/0 
debug ms = 0/0 
debug monc = 0/0 
debug tp = 0/0 
debug auth = 0/0 
debug finisher = 0/0 
debug heartbeatmap = 0/0 
debug perfcounter = 0/0 
debug asok = 0/0 
debug throttle = 0/0 

[client] 
debug ms = 0/0 
debug rbd = 0/0 
debug lockdep = 0/0 
debug context = 0/0 
debug crush = 0/0 
debug buffer = 0/0 
debug timer = 0/0 
debug journaler = 0/0 
debug osd = 0/0 
debug optracker = 0/0 
debug objclass = 0/0 
debug filestore = 0/0 
debug journal = 0/0 
debug ms = 0/0 
debug monc = 0/0 
debug tp = 0/0 
debug auth = 0/0 
debug finisher = 0/0 
debug heartbeatmap = 0/0 
debug perfcounter = 0/0 
debug asok = 0/0 
debug throttle = 0/0 

Stefan 

Am 12.11.2012 15:35, schrieb Alexandre DERUMIER: 
> Another idea, 
> 
> do you have tried to put 
>>>>> debug lockdep = 0/0 
>>>>> debug context = 0/0 
>>>>> debug crush = 0/0 
>>>>> debug buffer = 0/0 
>>>>> debug timer = 0/0 
>>>>> debug journaler = 0/0 
>>>>> debug osd = 0/0 
>>>>> debug optracker = 0/0 
>>>>> debug objclass = 0/0 
>>>>> debug filestore = 0/0 
>>>>> debug journal = 0/0 
>>>>> debug ms = 0/0 
>>>>> debug monc = 0/0 
>>>>> debug tp = 0/0 
>>>>> debug auth = 0/0 
>>>>> debug finisher = 0/0 
>>>>> debug heartbeatmap = 0/0 
>>>>> debug perfcounter = 0/0 
>>>>> debug asok = 0/0 
>>>>> debug throttle = 0/0 
> 
> in a ceph.conf on your kvm host ? 
> 
> 
> ----- Mail original ----- 
> 
> De: "Alexandre DERUMIER" <aderumier at odiso.com> 
> À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
> Cc: pve-devel at pve.proxmox.com 
> Envoyé: Lundi 12 Novembre 2012 15:26:36 
> Objet: Re: [pve-devel] less cores more iops / speed 
> 
> Maybe some tracing on kvm process could give us clues to find where the cpu is used ? 
> 
> Also another idea, can you try with "auth supported=none" ? maybe they are some overhead with ceph authenfication ? 
> 
> 
> 
> 
> ----- Mail original ----- 
> 
> De: "Alexandre DERUMIER" <aderumier at odiso.com> 
> À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
> Cc: pve-devel at pve.proxmox.com 
> Envoyé: Lundi 12 Novembre 2012 15:20:07 
> Objet: Re: [pve-devel] less cores more iops / speed 
> 
> Ok thanks. 
> 
> Seem to use a lot of cpu vs nfs,iscsi ... 
> 
> I hope that ceph dev will work on this soon ! 
> 
> 
> ----- Mail original ----- 
> 
> De: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
> À: "Alexandre DERUMIER" <aderumier at odiso.com> 
> Cc: "eric" <eric at netwalk.com>, pve-devel at pve.proxmox.com 
> Envoyé: Lundi 12 Novembre 2012 15:05:08 
> Objet: Re: [pve-devel] less cores more iops / speed 
> 
> Am 12.11.2012 13:49, schrieb Alexandre DERUMIER: 
>>>> One VM on one Host: 18.000 IOP/s 
>>>> Two VM on one Host: 2x11.000 IOP/s 
>>>> Three VM on one Host: 3x7.000 IOP/s 
>> 
>> And host cpu is 100% ? 
> 
> No. For three VMs yes. For one and two no. I think librbd / rbd 
> implementation in kvm is the bottleneck here. 
> 
> Stefan 
> 
>> ----- Mail original ----- 
>> 
>> De: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag> 
>> À: "Alexandre DERUMIER" <aderumier at odiso.com> 
>> Cc: "eric" <eric at netwalk.com>, pve-devel at pve.proxmox.com 
>> Envoyé: Lundi 12 Novembre 2012 12:58:35 
>> Objet: Re: [pve-devel] less cores more iops / speed 
>> 
>> Am 12.11.2012 08:51, schrieb Alexandre DERUMIER: 
>>>>> Right now RBD in KVM is limited by CPU speed. 
>>> 
>>> Good to known, so it's seem lack of threading, or maybe somes locks. (so faster cpu give more iops). 
>>> 
>>> If you lauch parallel fio on same host on different guest, do you get more total iops ? (for me it's scale) 
>> 
>> One VM on one Host: 18.000 IOP/s 
>> Two VM on one Host: 2x11.000 IOP/s 
>> Three VM on one Host: 3x7.000 IOP/s 
>> 
>>> if you launch 2 parallel fio, on same guest (on differents disk), do you get more iops ? (for me, it doesn't scale, so raid0 in guest doesn't help). 
>> No it doesn't scale. 
>> 
>> Stefan 
>> 
>>> ----- Mail original ----- 
>>> 
>>> De: "Stefan Priebe" <s.priebe at profihost.ag> 
>>> À: "Alexandre DERUMIER" <aderumier at odiso.com> 
>>> Cc: "eric" <eric at netwalk.com>, pve-devel at pve.proxmox.com 
>>> Envoyé: Dimanche 11 Novembre 2012 13:07:36 
>>> Objet: Re: [pve-devel] less cores more iops / speed 
>>> 
>>> Am 11.11.2012 12:12, schrieb Alexandre DERUMIER: 
>>>> If I remember good, stefan can achieve 100.000 iops with iscsi with same kvm host. 
>>> 
>>> Correct but this was always with scsi-generic and I/O multipathing on 
>>> host. rbd does not support scsi-generic ;-( 
>>> 
>>>> I have checked ceph mailing, stefan seem to have resolved his problem with dual core with bios update ! 
>>> Correct. So speed on Dual Xeon is now 14.000 IOP/s and 18.000 IOP/s on 
>>> Single Xeon. But the difference is an issue of the CPU Speed. 3,6Ghz 
>>> Single Xeon vs. 2.5Ghz Dual Xeon. 
>>> 
>>> Right now RBD in KVM is limited by CPU speed. 
>>> 
>>> Greets, 
>>> Stefan 
>>> 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
>