<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body>Not using rbd, just lvm on the local array.<div><br></div><div>Just wanted to point out that a single vm, regardless of storage used, does seem to be limited on IOPS.</div><div><br></div><div>I suspect the cpu time needed to handle each io operation in KVM/qemu is where the limit comes from. </div><div><br></div><div>When I first started testing I got lower results. Then I set some options to get turbo boost to kick in for all cores all the time resulting in better IOPS.</div><div><br></div><div>I was testing using crystal diskmark in windows using virtio. </div><br>Alexandre DERUMIER <aderumier@odiso.com> wrote:<br>Hi Eric,<br><br>>>I have been testing IOPS with Areca 1882 pcie 3.0 with SSD disks.<br>>>A single VM on my system seems to be limited to about 30K IOPS writing<br>>>and about 70K reading.<br>>>I can get more total IOPS running benchmarks on two VMs at the same time.<br><br>Great Results !<br>What benchmark do you use ? fio ? can you post the bench command line to compare ?<br><br>Also what is your cluster hardware (cpu,ram,disk model, how many nodes, switchs models ?).<br><br>Your results seem to be a lot higher than stephan. (This is the best results I ever see with rbd ;)<br><br><br><br><br>>>Instructions here:<br>>>http://forum.proxmox.com/threads/11508-Intel-Turbo-Boost-not-working?p=63058#post63058<br><br>I agree with this settings, I always set this options in bios.<br><br><br>Also maybe irqbalance daemon can help:<br><br>#apt-get install irqbalance<br><br><br><br>----- Mail original -----<br><br>De: "Eric Blevins" <eric@netwalk.com><br>À: pve-devel@pve.proxmox.com<br>Envoyé: Vendredi 9 Novembre 2012 21:53:21<br>Objet: Re: [pve-devel] less cores more iops / speed<br><br>I have been testing IOPS with Areca 1882 pcie 3.0 with SSD disks.<br>A single VM on my system seems to be limited to about 30K IOPS writing<br>and about 70K reading.<br>I can get more total IOPS running benchmarks on two VMs at the same time.<br><br>Xeon E5-2650 2.0Ghz 8 core<br><br>These newer Xeon with turboboost make it a little difficult to benchmark<br>if frequencies change all the time.<br>With a few teaks I got all the cores in my 2650s to run at 2.4Ghz all<br>the time.<br>Instructions here:<br>http://forum.proxmox.com/threads/11508-Intel-Turbo-Boost-not-working?p=63058#post63058<br><br><br><br><br>On 11/09/2012 11:02 AM, Alexandre DERUMIER wrote:<br>> Oh great,<br>> so it seem that you can get more ios with more vm. (It's become interesting for me :)<br>><br>> What are you exact xeon models ? same server model,bios options ?<br>><br>> what happen on the dual xeon, if you shutdown some cores ?<br>><br>> echo 0 > /sys/devices/system/cpu/cpuX/online<br>><br>><br>> also, does pining help on dual xeon ?<br>><br>><br>><br>> ----- Mail original -----<br>><br>> De: "Stefan Priebe - Profihost AG" <s.priebe@profihost.ag><br>> À: "Alexandre DERUMIER" <aderumier@odiso.com><br>> Cc: pve-devel@pve.proxmox.com<br>> Envoyé: Vendredi 9 Novembre 2012 16:42:09<br>> Objet: Re: [pve-devel] less cores more iops / speed<br>><br>> Am 09.11.2012 15:49, schrieb Alexandre DERUMIER:<br>>> Maybe one last question (sorry ;)<br>>><br>>> Do you have tried to launch 2 fio test from 2 differents kvm host at the same time ?<br>>><br>>> If yes, do you have more io or do you have half of speed on each fio test ?<br>> Only one VM active on a Single Xeon 3.6Ghz:<br>> write: io=6024MB, bw=68519KB/s, iops=17129, runt= 90025msec<br>> read : io=6057MB, bw=68891KB/s, iops=17222, runt= 90026msec<br>> write: io=71540MB, bw=808308KB/s, iops=197, runt= 90630msec<br>> read : io=147096MB, bw=1631MB/s, iops=407, runt= 90162msec<br>><br>><br>> Only one VM active on a Dual Xeon 2.5Ghz:<br>> write: io=2224MB, bw=25285KB/s, iops=6321, runt= 90070msec<br>> read : io=2033MB, bw=23108KB/s, iops=5777, runt= 90078msec<br>> write: io=50900MB, bw=574235KB/s, iops=140, runt= 90767msec<br>> read : io=84228MB, bw=954499KB/s, iops=233, runt= 90361msec<br>><br>><br>> Both VMs active (doing each test in parallel):<br>><br>> VM 1:<br>> write: io=5376MB, bw=61112KB/s, iops=15277, runt= 90086msec<br>> read : io=6092MB, bw=69292KB/s, iops=17323, runt= 90028msec<br>> write: io=67724MB, bw=766012KB/s, iops=187, runt= 90533msec<br>> read : io=160200MB, bw=1776MB/s, iops=444, runt= 90187msec<br>><br>> VM 2:<br>> write: io=2351MB, bw=26719KB/s, iops=6679, runt= 90083msec<br>> read : io=2190MB, bw=24899KB/s, iops=6224, runt= 90081msec<br>> write: io=48320MB, bw=545652KB/s, iops=133, runt= 90680msec<br>> read : io=88136MB, bw=999571KB/s, iops=244, runt= 90290msec<br>><br>> Right now i've no idea why the Dual Xeon is slower at all / in general.<br>><br>> Stefan<br>> _______________________________________________<br>> pve-devel mailing list<br>> pve-devel@pve.proxmox.com<br>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel<br><br>_______________________________________________<br>pve-devel mailing list<br>pve-devel@pve.proxmox.com<br>http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel<br></body>