[PVE-User] Proxmox with ceph storage VM performance strangeness

[Alwin Antreich]

Hallo Rainer,

On Tue, Mar 17, 2020 at 02:04:22PM +0100, Rainer Krienke wrote:
> Hello,
> 
> I run a pve 6.1-7 cluster with 5 nodes that is attached (via 10Gb
> Network) to a ceph nautilus cluster with 9 ceph nodes and 144 magnetic
> disks. The pool with rbd images for disk storage is erasure coded with a
> 4+2 profile.
> 
> I ran some performance tests since I noticed that there seems to be a
> strange limit to the disk read/write rate on a single VM even if the
> physical machine hosting the VM as well as cluster is in total capable
> of doing much more.
> 
> So what I did was to run a bonnie++ as well as a dd read/write test
> first in parallel on 10 VMs, then on 5 VMs and at last on a single one.
> 
> A value of "75" for "bo++rd" in the first line below means that each of
> the 10 bonnie++-processes running on 10 different proxmox VMs in
> parallel reported in average over all the results a value of
> 75MBytes/sec for "block read". The ceph-values are the peaks measured by
> ceph itself during the test run (all rd/wr values in MBytes/sec):
> 
> VM-count:  bo++rd: bo++wr: ceph(rd/wr):  dd-rd:  dd-wr:  ceph(rd/wr):
> 10           75      42      540/485       55     58      698/711
>  5           90      62      310/338       47     80      248/421
>  1          108     114      111/120      130    145      337/165
> 
> 
> What I find a little strange is that running many VMs doing IO in
> parallel I reach a write rate of about 485-711 MBytes/sec. However when
> running a single VM the maximum is at 120-165 MBytes/sec. Since the
> whole networking is based on a 10GB infrastructure and an iperf test
> between a VM and a ceph node reported nearby 10Gb I would expect a
> higher rate for the single VM. Even if I run a test with 5 VMs on *one*
> physical host (values not shown above), the results are not far behind
> the values for 5 VMs on 5 hosts shown above. So the single host seems
> not to be the limiting factor, but the VM itself is limiting IO.
> 
> What rates do you find on your proxmox/ceph cluster for single VMs?
> Does any one have any explanation for this rather big difference or
> perhaps an idea what to try in order to get higher IO-rates from a
> single VM?
> 
> Thank you very much in advance
> Rainer
As Eneko said, single thread vs multiple threads. How are you VMs
configured (qm config <vmid>)?

> 
> 
> 
> ---------------------------------------------
> Here are the more detailed test results for anyone interested:
> 
> Using bonnie++:
> 10 VMs (two on each of the 5 hosts) VMs: 4GB RAM, BTRFS, cd /root;
> bonnie++ -u root
>   Average for each VM:
>   block write: ~42MByte/sec, block read: ~75MByte/sec
>   ceph: total peak: 485MByte/sec write, 540MByte/sec read
> 
> 5 VMs (one on each of the 5 hosts) 4GB RAM, BTRFS, cd /root; bonnie++ -u
> root
>   Average for each VM:
>   block write: ~62MByte/sec, block read: ~90MByte/sec
>   ceph: total peak: 338MByte/sec write, 310MByte/sec read
> 
> 1 VM  4GB RAM, BTRFS, cd /root; bonnie++ -u root
>   Average for VM:
>   block write: ~114 MByte/sec, block read: ~108MByte/sec
>   ceph: total peak: 120 MByte/sec write, 111MByte/sec read
How did you configure bonnie?
And a CoW filesystem on top of Ceph will certainly drop performance.

> 
> 
> Using dd:
> 10 VMs (two on each of the 5 hosts) VMs: 4GB RAM, write on a ceph based
> vm-disk "sdb" (rbd)
>   write: dd if=/dev/zero of=/dev/sdb bs=nnn count=kkk conv=fsync
> status=progress
>   read:  dd of=/dev/null if=/dev/sdb bs=nnn count=kkk  status=progress
>   Average for each VM:
>   bs=1024k count=12000: dd write: ~58MByte/sec, dd read: ~48MByte/sec
>   bs=4096k count=3000:  dd write: ~59MByte/sec, dd read: ~55MByte/sec
>   ceph: total peak: 711MByte/sec write, 698 MByte/sec read
> 
> 5 VMs (two on each of the 5 hosts) VMs: 4GB RAM, write on a ceph based
> vm-disk "sdb" (rbd)
>   write: dd if=/dev/zero of=/dev/sdb bs=4096k count=3000 conv=fsync
> status=progress
>   read:  dd of=/dev/null if=/dev/sdb bs=4096k count=3000  status=progress
>   Average for each VM:
>   bs=4096 count=3000:  dd write: ~80 MByte/sec, dd read: ~47MByte/sec
>   ceph: total peak: 421MByte/sec write, 248 MByte/sec read
> 
> 1 VM: 4GB RAM, write on a ceph based vm-disk "sdb" (rbd-device)
>   write: dd if=/dev/zero of=/dev/sdb bs=4096k count=3000 conv=fsync
> status=progress
>   read:  dd of=/dev/null if=/dev/sdb bs=4096k count=3000  status=progress
>   Average for each VM:
>   bs=4096k count=3000:  dd write: ~145 MByte/sec, dd read: ~130 MByte/sec
>   ceph: total peak: 165 MByte/sec write, 337 MByte/sec read
dd is not well suited for performance benchmarking. Better use bonnie++
or FIO. The later is good for storage benchmarks in general.

As you now have the results from the top-most layer. How do the lower
layers perform? Eg. FIO has a build-in rbd engine and is able to talk
directly to an rbd image. For Ceph pool performance a rados bench can
sure shed some light.

--
Cheers,
Alwin