[pve-devel] Limited IOP/s on Dual Xeon KVM Host

[Andrey Korolyov]

On Sat, Nov 10, 2012 at 5:49 PM, Stefan Priebe <s.priebe at profihost.ag> wrote:
> Am 10.11.2012 14:41, schrieb Mark Nelson:
>
>> On 11/10/2012 02:03 AM, Stefan Priebe wrote:
>>>
>>> Hello lists,
>>>
>>> on a dual Xeon KVM Host i get max 6000 IOP/s random 4k writes AND reads.
>>> On a Single Xeon KVM Host i get 17.000-18.000 IOP/s write and read. I
>>> already tried to pin the kvm process using numactl and also the fio
>>> process but it doesn't help on the dual xeon.
>>>
>>> 10GBE Network is fine. I get 9.8Gbit/s on both hosts. Kernel is also he
>>> same on both.
>>>
>>> Anybody an idea?
>>
>>
>> When you say KVM host, do you mean the underlying node or the virtual
>> machine instance?
>
> Sorry i'm talking about the vm host regarding the HW. The vm instance is
> always the same.
>
>
>> If you mean underlying node, it could be remote memory access or if you
>> are on a last gen xeon if you have dual io hubs, you could be hitting a
>> remote io hub for the network card.  I wouldn't think that would cause
>> such a big hit, but those are things to look into.
>
>
> I'm on E5-Xeon. What means io hub?

QPI path length, in other terms, numa distance(hope Mark means the
same). Yes, it is impossible to have such degradation even in worst
case on two-head node. I assume two possible things - you have pinned
many processes on the core set which including default core for the
network card` irq, please check it via /proc/interrupts, or you have
not really did pinning and qemu process losing ticks by switching
cores - it may be checked, say, by top and guest cpu bencmark. For the
network card, it may be generally recommended to move its irq affinity
to entire numa node to which it belongs.

>
>
> Stefan
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html