[PVE-User] Proxmox with ceph storage VM performance strangeness

[Alwin Antreich]

On Tue, Apr 14, 2020 at 03:54:30PM +0200, Rainer Krienke wrote:
> Hello,
> 
> in between I learned a lot from this group (thanks a lot) to solve many
> performance problems I initially faced with proxmox in VMs having their
> storage on CEPH rbds.
> 
> I parallelized access to many disks on a vm where possible, used
> iothreads and activated writeback cache.
> 
> Running a bonnie++ I am now able to get about 300Mbytes/sec block write
> performance, which is a great value because it even scales out with ceph
> if I run the same bonnie++ on eg two machines. In this case I get
> 600MBytes/sec. Great.
> 
> The last strangeness I am experiencing is read performance. The same
> bonnie on a VMs xfs_filesystem that yields 300MB write performance only
> gets a block read of about 90MB/sec.
> 
> So on one of the pxa-hosts and later also on one of the ceph cluster
> nodes (nautilus 14.2.8, 144OSDs) I ran a rados bench test to see if ceph
> is slowing down reads. The results on both systems were very similar. So
> here is the test result from the pxa-host:
> 
> # rados  bench -p my-rbd 60 write --no-cleanup
> Total time run:         60.284332
> Total writes made:      5376
> Write size:             4194304
> Object size:            4194304
> Bandwidth (MB/sec):     356.71
> Stddev Bandwidth:       46.8361
> Max bandwidth (MB/sec): 424
> Min bandwidth (MB/sec): 160
> Average IOPS:           89
> Stddev IOPS:            11
> Max IOPS:               106
> Min IOPS:               40
> Average Latency(s):     0.179274
> Stddev Latency(s):      0.105626
> Max latency(s):         1.00746
> Min latency(s):         0.0656261
> 
> # echo 3 > /proc/sys/vm/drop_caches
> # rados  bench -p pxa-rbd 60 seq
> Total time run:       24.208097
> Total reads made:     5376
> Read size:            4194304
> Object size:          4194304
> Bandwidth (MB/sec):   888.298
> Average IOPS:         222
> Stddev IOPS:          33
> Max IOPS:             249
> Min IOPS:             92
> Average Latency(s):   0.0714553
> Max latency(s):       0.63154
> Min latency(s):       0.0237746
> 
> 
> According to these numbers the relation from write and read  performance
> should be the other way round: writes should be slower than reads, but
> on a VM its exactly the other way round?
Ceph does reads in parallel, while writes are done to the primary OSD by
the client. And that OSD is responsible for distributing the other
copies.

> 
> Any idea why nevertheless writes on a VM are ~3 times faster then reads
> and what I could try to speed up reading?
What is the byte size of bonnie++? If it uses 4 KB and data isn't in the
cache, whole objects need to be requested from the cluster.

--
Cheers,
Alwin