[pve-devel] less cores more iops / speed

[Stefan Priebe - Profihost AG]

Adding this to ceph.conf on kvm host adds another 2000 iops (20.000 
iop/s with one VM). I'm sure most of them are useless on a client kvm / 
rbd host but i don't know which one makes sense ;-)

[global]
         debug ms = 0/0
         debug rbd = 0/0
         debug lockdep = 0/0
         debug context = 0/0
         debug crush = 0/0
         debug buffer = 0/0
         debug timer = 0/0
         debug journaler = 0/0
         debug osd = 0/0
         debug optracker = 0/0
         debug objclass = 0/0
         debug filestore = 0/0
         debug journal = 0/0
         debug ms = 0/0
         debug monc = 0/0
         debug tp = 0/0
         debug auth = 0/0
         debug finisher = 0/0
         debug heartbeatmap = 0/0
         debug perfcounter = 0/0
         debug asok = 0/0
         debug throttle = 0/0

[client]
         debug ms = 0/0
         debug rbd = 0/0
         debug lockdep = 0/0
         debug context = 0/0
         debug crush = 0/0
         debug buffer = 0/0
         debug timer = 0/0
         debug journaler = 0/0
         debug osd = 0/0
         debug optracker = 0/0
         debug objclass = 0/0
         debug filestore = 0/0
         debug journal = 0/0
         debug ms = 0/0
         debug monc = 0/0
         debug tp = 0/0
         debug auth = 0/0
         debug finisher = 0/0
         debug heartbeatmap = 0/0
         debug perfcounter = 0/0
         debug asok = 0/0
         debug throttle = 0/0

Stefan

Am 12.11.2012 15:35, schrieb Alexandre DERUMIER:
> Another idea,
>
> do you have tried to put
>>>>>   debug lockdep = 0/0
>>>>>   debug context = 0/0
>>>>>   debug crush = 0/0
>>>>>   debug buffer = 0/0
>>>>>   debug timer = 0/0
>>>>>   debug journaler = 0/0
>>>>>   debug osd = 0/0
>>>>>   debug optracker = 0/0
>>>>>   debug objclass = 0/0
>>>>>   debug filestore = 0/0
>>>>>   debug journal = 0/0
>>>>>   debug ms = 0/0
>>>>>   debug monc = 0/0
>>>>>   debug tp = 0/0
>>>>>   debug auth = 0/0
>>>>>   debug finisher = 0/0
>>>>>   debug heartbeatmap = 0/0
>>>>>   debug perfcounter = 0/0
>>>>>   debug asok = 0/0
>>>>>   debug throttle = 0/0
>
> in a ceph.conf on your kvm host ?
>
>
> ----- Mail original -----
>
> De: "Alexandre DERUMIER" <aderumier at odiso.com>
> À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag>
> Cc: pve-devel at pve.proxmox.com
> Envoyé: Lundi 12 Novembre 2012 15:26:36
> Objet: Re: [pve-devel] less cores more iops / speed
>
> Maybe some tracing on kvm process could give us clues to find where the cpu is used ?
>
> Also another idea, can you try with "auth supported=none" ? maybe they are some overhead with ceph authenfication ?
>
>
>
>
> ----- Mail original -----
>
> De: "Alexandre DERUMIER" <aderumier at odiso.com>
> À: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag>
> Cc: pve-devel at pve.proxmox.com
> Envoyé: Lundi 12 Novembre 2012 15:20:07
> Objet: Re: [pve-devel] less cores more iops / speed
>
> Ok thanks.
>
> Seem to use a lot of cpu vs nfs,iscsi ...
>
> I hope that ceph dev will work on this soon !
>
>
> ----- Mail original -----
>
> De: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag>
> À: "Alexandre DERUMIER" <aderumier at odiso.com>
> Cc: "eric" <eric at netwalk.com>, pve-devel at pve.proxmox.com
> Envoyé: Lundi 12 Novembre 2012 15:05:08
> Objet: Re: [pve-devel] less cores more iops / speed
>
> Am 12.11.2012 13:49, schrieb Alexandre DERUMIER:
>>>> One VM on one Host: 18.000 IOP/s
>>>> Two VM on one Host: 2x11.000 IOP/s
>>>> Three VM on one Host: 3x7.000 IOP/s
>>
>> And host cpu is 100% ?
>
> No. For three VMs yes. For one and two no. I think librbd / rbd
> implementation in kvm is the bottleneck here.
>
> Stefan
>
>> ----- Mail original -----
>>
>> De: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag>
>> À: "Alexandre DERUMIER" <aderumier at odiso.com>
>> Cc: "eric" <eric at netwalk.com>, pve-devel at pve.proxmox.com
>> Envoyé: Lundi 12 Novembre 2012 12:58:35
>> Objet: Re: [pve-devel] less cores more iops / speed
>>
>> Am 12.11.2012 08:51, schrieb Alexandre DERUMIER:
>>>>> Right now RBD in KVM is limited by CPU speed.
>>>
>>> Good to known, so it's seem lack of threading, or maybe somes locks. (so faster cpu give more iops).
>>>
>>> If you lauch parallel fio on same host on different guest, do you get more total iops ? (for me it's scale)
>>
>> One VM on one Host: 18.000 IOP/s
>> Two VM on one Host: 2x11.000 IOP/s
>> Three VM on one Host: 3x7.000 IOP/s
>>
>>> if you launch 2 parallel fio, on same guest (on differents disk), do you get more iops ? (for me, it doesn't scale, so raid0 in guest doesn't help).
>> No it doesn't scale.
>>
>> Stefan
>>
>>> ----- Mail original -----
>>>
>>> De: "Stefan Priebe" <s.priebe at profihost.ag>
>>> À: "Alexandre DERUMIER" <aderumier at odiso.com>
>>> Cc: "eric" <eric at netwalk.com>, pve-devel at pve.proxmox.com
>>> Envoyé: Dimanche 11 Novembre 2012 13:07:36
>>> Objet: Re: [pve-devel] less cores more iops / speed
>>>
>>> Am 11.11.2012 12:12, schrieb Alexandre DERUMIER:
>>>> If I remember good, stefan can achieve 100.000 iops with iscsi with same kvm host.
>>>
>>> Correct but this was always with scsi-generic and I/O multipathing on
>>> host. rbd does not support scsi-generic ;-(
>>>
>>>> I have checked ceph mailing, stefan seem to have resolved his problem with dual core with bios update !
>>> Correct. So speed on Dual Xeon is now 14.000 IOP/s and 18.000 IOP/s on
>>> Single Xeon. But the difference is an issue of the CPU Speed. 3,6Ghz
>>> Single Xeon vs. 2.5Ghz Dual Xeon.
>>>
>>> Right now RBD in KVM is limited by CPU speed.
>>>
>>> Greets,
>>> Stefan
>>>
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>