[PVE-User] Proxmox with ceph storage VM performance strangeness

[Rainer Krienke]

Hello,

in between I learned a lot from this group (thanks a lot) to solve many
performance problems I initially faced with proxmox in VMs having their
storage on CEPH rbds.

I parallelized access to many disks on a vm where possible, used
iothreads and activated writeback cache.

Running a bonnie++ I am now able to get about 300Mbytes/sec block write
performance, which is a great value because it even scales out with ceph
if I run the same bonnie++ on eg two machines. In this case I get
600MBytes/sec. Great.

The last strangeness I am experiencing is read performance. The same
bonnie on a VMs xfs_filesystem that yields 300MB write performance only
gets a block read of about 90MB/sec.

So on one of the pxa-hosts and later also on one of the ceph cluster
nodes (nautilus 14.2.8, 144OSDs) I ran a rados bench test to see if ceph
is slowing down reads. The results on both systems were very similar. So
here is the test result from the pxa-host:

# rados  bench -p my-rbd 60 write --no-cleanup
Total time run:         60.284332
Total writes made:      5376
Write size:             4194304
Object size:            4194304
Bandwidth (MB/sec):     356.71
Stddev Bandwidth:       46.8361
Max bandwidth (MB/sec): 424
Min bandwidth (MB/sec): 160
Average IOPS:           89
Stddev IOPS:            11
Max IOPS:               106
Min IOPS:               40
Average Latency(s):     0.179274
Stddev Latency(s):      0.105626
Max latency(s):         1.00746
Min latency(s):         0.0656261

# echo 3 > /proc/sys/vm/drop_caches
# rados  bench -p pxa-rbd 60 seq
Total time run:       24.208097
Total reads made:     5376
Read size:            4194304
Object size:          4194304
Bandwidth (MB/sec):   888.298
Average IOPS:         222
Stddev IOPS:          33
Max IOPS:             249
Min IOPS:             92
Average Latency(s):   0.0714553
Max latency(s):       0.63154
Min latency(s):       0.0237746


According to these numbers the relation from write and read  performance
should be the other way round: writes should be slower than reads, but
on a VM its exactly the other way round?

Any idea why nevertheless writes on a VM are ~3 times faster then reads
and what I could try to speed up reading?

Thanks a lot
Rainer
-- 
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse  1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html,     Fax: +49261287
1001312



-- 
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse  1
56070 Koblenz, Web: http://www.uni-koblenz.de/~krienke, Tel: +49261287 1312
PGP: http://www.uni-koblenz.de/~krienke/mypgp.html,     Fax: +49261287
1001312