[pve-devel] less cores more iops / speed

[eric]

Not using rbd, just lvm on the local array.

Just wanted to point out that a single vm, regardless of storage used, does seem to be limited on IOPS.

I suspect the cpu time needed to handle each io operation in KVM/qemu is where the limit comes from.  

When I first started testing I got lower results. Then I set some options to get turbo boost to kick in for all cores all the time resulting in better IOPS.

I was testing using crystal diskmark in windows using virtio. Alexandre DERUMIER <aderumier at odiso.com> wrote:Hi Eric,

>>I have been testing IOPS with Areca 1882 pcie 3.0 with SSD disks.
>>A single VM on my system seems to be limited to about 30K IOPS writing
>>and about 70K reading.
>>I can get more total IOPS running benchmarks on two VMs at the same time.

Great Results !
What benchmark do you use ? fio ? can you post the bench command line to compare ?

Also what is your cluster hardware (cpu,ram,disk model, how many nodes, switchs models ?).

Your results seem to be a lot higher than stephan. (This is the best results I ever see with rbd ;)




>>Instructions here:
>>http://forum.proxmox.com/threads/11508-Intel-Turbo-Boost-not-working?p=63058#post63058

I agree with this settings, I always set this options in bios.


Also maybe irqbalance daemon can help:

#apt-get install irqbalance



----- Mail original -----

De: "Eric Blevins" <eric at netwalk.com>
À: pve-devel at pve.proxmox.com
Envoyé: Vendredi 9 Novembre 2012 21:53:21
Objet: Re: [pve-devel] less cores more iops / speed

I have been testing IOPS with Areca 1882 pcie 3.0 with SSD disks.
A single VM on my system seems to be limited to about 30K IOPS writing
and about 70K reading.
I can get more total IOPS running benchmarks on two VMs at the same time.

Xeon E5-2650 2.0Ghz 8 core

These newer Xeon with turboboost make it a little difficult to benchmark
if frequencies change all the time.
With a few teaks I got all the cores in my 2650s to run at 2.4Ghz all
the time.
Instructions here:
http://forum.proxmox.com/threads/11508-Intel-Turbo-Boost-not-working?p=63058#post63058




On 11/09/2012 11:02 AM, Alexandre DERUMIER wrote:
> Oh great,
> so it seem that you can get more ios with more vm. (It's become interesting for me :)
>
> What are you exact xeon models ? same server model,bios options ?
>
> what happen on the dual xeon, if you shutdown some cores ?
>
> echo 0 > /sys/devices/system/cpu/cpuX/online
>
>
> also, does pining help on dual xeon ?
>
>
>
> ----- Mail original -----
>
> De: "Stefan Priebe - Profihost AG" <s.priebe at profihost.ag>
> À: "Alexandre DERUMIER" <aderumier at odiso.com>
> Cc: pve-devel at pve.proxmox.com
> Envoyé: Vendredi 9 Novembre 2012 16:42:09
> Objet: Re: [pve-devel] less cores more iops / speed
>
> Am 09.11.2012 15:49, schrieb Alexandre DERUMIER:
>> Maybe one last question (sorry ;)
>>
>> Do you have tried to launch 2 fio test from 2 differents kvm host at the same time ?
>>
>> If yes, do you have more io or do you have half of speed on each fio test ?
> Only one VM active on a Single Xeon 3.6Ghz:
> write: io=6024MB, bw=68519KB/s, iops=17129, runt= 90025msec
> read : io=6057MB, bw=68891KB/s, iops=17222, runt= 90026msec
> write: io=71540MB, bw=808308KB/s, iops=197, runt= 90630msec
> read : io=147096MB, bw=1631MB/s, iops=407, runt= 90162msec
>
>
> Only one VM active on a Dual Xeon 2.5Ghz:
> write: io=2224MB, bw=25285KB/s, iops=6321, runt= 90070msec
> read : io=2033MB, bw=23108KB/s, iops=5777, runt= 90078msec
> write: io=50900MB, bw=574235KB/s, iops=140, runt= 90767msec
> read : io=84228MB, bw=954499KB/s, iops=233, runt= 90361msec
>
>
> Both VMs active (doing each test in parallel):
>
> VM 1:
> write: io=5376MB, bw=61112KB/s, iops=15277, runt= 90086msec
> read : io=6092MB, bw=69292KB/s, iops=17323, runt= 90028msec
> write: io=67724MB, bw=766012KB/s, iops=187, runt= 90533msec
> read : io=160200MB, bw=1776MB/s, iops=444, runt= 90187msec
>
> VM 2:
> write: io=2351MB, bw=26719KB/s, iops=6679, runt= 90083msec
> read : io=2190MB, bw=24899KB/s, iops=6224, runt= 90081msec
> write: io=48320MB, bw=545652KB/s, iops=133, runt= 90680msec
> read : io=88136MB, bw=999571KB/s, iops=244, runt= 90290msec
>
> Right now i've no idea why the Dual Xeon is slower at all / in general.
>
> Stefan
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel

_______________________________________________
pve-devel mailing list
pve-devel at pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.proxmox.com/pipermail/pve-devel/attachments/20121111/5eedf9a7/attachment.htm>