[PVE-User] [SPAM] Proxmox with ceph storage VM performance strangeness

[Ralf Storm]

Hello Rainer,


same issue here 7nodes Cluster with Proxmox and Ceph on the same nodes 
seperate 10gb for Ceph and 10gb for VMs,

not erasure coded, about 50 ssd. Performance for Backups, recovery etc. 
is almost 1000MByte/s, several vms accessing data at the same time is 
raising the perfomance also to the expected levels, but single vm access 
is never above 150MByte...

Could not exaxctly figure out why in my searches...



regards


Ralf

Am 17/03/2020 um 14:04 schrieb Rainer Krienke:
> Hello,
>
> I run a pve 6.1-7 cluster with 5 nodes that is attached (via 10Gb
> Network) to a ceph nautilus cluster with 9 ceph nodes and 144 magnetic
> disks. The pool with rbd images for disk storage is erasure coded with a
> 4+2 profile.
>
> I ran some performance tests since I noticed that there seems to be a
> strange limit to the disk read/write rate on a single VM even if the
> physical machine hosting the VM as well as cluster is in total capable
> of doing much more.
>
> So what I did was to run a bonnie++ as well as a dd read/write test
> first in parallel on 10 VMs, then on 5 VMs and at last on a single one.
>
> A value of "75" for "bo++rd" in the first line below means that each of
> the 10 bonnie++-processes running on 10 different proxmox VMs in
> parallel reported in average over all the results a value of
> 75MBytes/sec for "block read". The ceph-values are the peaks measured by
> ceph itself during the test run (all rd/wr values in MBytes/sec):
>
> VM-count:  bo++rd: bo++wr: ceph(rd/wr):  dd-rd:  dd-wr:  ceph(rd/wr):
> 10           75      42      540/485       55     58      698/711
>   5           90      62      310/338       47     80      248/421
>   1          108     114      111/120      130    145      337/165
>
>
> What I find a little strange is that running many VMs doing IO in
> parallel I reach a write rate of about 485-711 MBytes/sec. However when
> running a single VM the maximum is at 120-165 MBytes/sec. Since the
> whole networking is based on a 10GB infrastructure and an iperf test
> between a VM and a ceph node reported nearby 10Gb I would expect a
> higher rate for the single VM. Even if I run a test with 5 VMs on *one*
> physical host (values not shown above), the results are not far behind
> the values for 5 VMs on 5 hosts shown above. So the single host seems
> not to be the limiting factor, but the VM itself is limiting IO.
>
> What rates do you find on your proxmox/ceph cluster for single VMs?
> Does any one have any explanation for this rather big difference or
> perhaps an idea what to try in order to get higher IO-rates from a
> single VM?
>
> Thank you very much in advance
> Rainer
>
>
>
> ---------------------------------------------
> Here are the more detailed test results for anyone interested:
>
> Using bonnie++:
> 10 VMs (two on each of the 5 hosts) VMs: 4GB RAM, BTRFS, cd /root;
> bonnie++ -u root
>    Average for each VM:
>    block write: ~42MByte/sec, block read: ~75MByte/sec
>    ceph: total peak: 485MByte/sec write, 540MByte/sec read
>
> 5 VMs (one on each of the 5 hosts) 4GB RAM, BTRFS, cd /root; bonnie++ -u
> root
>    Average for each VM:
>    block write: ~62MByte/sec, block read: ~90MByte/sec
>    ceph: total peak: 338MByte/sec write, 310MByte/sec read
>
> 1 VM  4GB RAM, BTRFS, cd /root; bonnie++ -u root
>    Average for VM:
>    block write: ~114 MByte/sec, block read: ~108MByte/sec
>    ceph: total peak: 120 MByte/sec write, 111MByte/sec read
>
>
> Using dd:
> 10 VMs (two on each of the 5 hosts) VMs: 4GB RAM, write on a ceph based
> vm-disk "sdb" (rbd)
>    write: dd if=/dev/zero of=/dev/sdb bs=nnn count=kkk conv=fsync
> status=progress
>    read:  dd of=/dev/null if=/dev/sdb bs=nnn count=kkk  status=progress
>    Average for each VM:
>    bs=1024k count=12000: dd write: ~58MByte/sec, dd read: ~48MByte/sec
>    bs=4096k count=3000:  dd write: ~59MByte/sec, dd read: ~55MByte/sec
>    ceph: total peak: 711MByte/sec write, 698 MByte/sec read
>
> 5 VMs (two on each of the 5 hosts) VMs: 4GB RAM, write on a ceph based
> vm-disk "sdb" (rbd)
>    write: dd if=/dev/zero of=/dev/sdb bs=4096k count=3000 conv=fsync
> status=progress
>    read:  dd of=/dev/null if=/dev/sdb bs=4096k count=3000  status=progress
>    Average for each VM:
>    bs=4096 count=3000:  dd write: ~80 MByte/sec, dd read: ~47MByte/sec
>    ceph: total peak: 421MByte/sec write, 248 MByte/sec read
>
> 1 VM: 4GB RAM, write on a ceph based vm-disk "sdb" (rbd-device)
>    write: dd if=/dev/zero of=/dev/sdb bs=4096k count=3000 conv=fsync
> status=progress
>    read:  dd of=/dev/null if=/dev/sdb bs=4096k count=3000  status=progress
>    Average for each VM:
>    bs=4096k count=3000:  dd write: ~145 MByte/sec, dd read: ~130 MByte/sec
>    ceph