[pve-devel] ceph firefly in pve repos

Alexandre DERUMIER aderumier at odiso.com
Thu Sep 11 07:57:07 CEST 2014


forget the ceph tuning
----------------------
          debug_lockdep = 0/0
          debug_context = 0/0
          debug_crush = 0/0
          debug_buffer = 0/0
          debug_timer = 0/0
          debug_filer = 0/0
          debug_objecter = 0/0
          debug_rados = 0/0
          debug_rbd = 0/0
          debug_journaler = 0/0
          debug_objectcatcher = 0/0
          debug_client = 0/0
          debug_osd = 0/0
          debug_optracker = 0/0
          debug_objclass = 0/0
          debug_filestore = 0/0
          debug_journal = 0/0
          debug_ms = 0/0
          debug_monc = 0/0
          debug_tp = 0/0
          debug_auth = 0/0
          debug_finisher = 0/0
          debug_heartbeatmap = 0/0
          debug_perfcounter = 0/0
          debug_asok = 0/0
          debug_throttle = 0/0
          debug_mon = 0/0
          debug_paxos = 0/0
          debug_rgw = 0/0
          osd_op_threads = 5
          filestore_op_threads = 4

         ms_nocrc = true
         cephx sign messages = false
         cephx require signatures = false

         ms_dispatch_throttle_bytes = 0

         #ceph 0.85
         throttler_perf_counter = false
         filestore_fd_cache_size = 64
         filestore_fd_cache_shards = 32
         osd_op_num_threads_per_shard = 1
         osd_op_num_shards = 25
         osd_enable_op_tracker = true


ceph 0.85 results
-----------------
fio write : 1osd
-----------------
bw=11694KB/s, iops=2923

fio read: 1osd
---------------
bw=38642KB/s, iops=9660  (I clear the buffer cache each second to be sure it's coming from disk)



Now enabling :  osd_enable_op_tracker = false
---------------------------------------------


fio read : 1 osd : optracker disable
------------------------------------
bw=80606KB/s, iops=20151,   (ALL cpu 100%) : GREAT !

fio write : 1 osd : optracker disable
------------------------------------
 bw=11630KB/s, iops=2907




Now, I don't understand why write are so slow.


fio on sdb
----------
bw=257806KB/s, iops=64451

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0,00     0,00    0,00 63613,00     0,00 254452,00     8,00    31,24    0,49    0,00    0,49   0,02 100,00

fio on osd through librbd
-------------------------
bw=11630KB/s, iops=2907

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdb               0,00   355,00    0,00 5225,00     0,00 29678,00    11,36    57,63   11,03    0,00   11,03   0,19  99,70



----- Mail original ----- 

De: "Alexandre DERUMIER" <aderumier at odiso.com> 
À: "Dietmar Maurer" <dietmar at proxmox.com> 
Cc: pve-devel at pve.proxmox.com 
Envoyé: Jeudi 11 Septembre 2014 06:22:21 
Objet: Re: [pve-devel] ceph firefly in pve repos 

Hi, here my firsts results (firefly - crucial m550 (1 osd by node - replication x1) 

fio config 
----------- 
[global] 
#logging 
#write_iops_log=write_iops_log 
#write_bw_log=write_bw_log 
#write_lat_log=write_lat_log 
ioengine=rbd 
clientname=admin 
pool=test 
rbdname=test 
invalidate=0 # mandatory 
rw=randwrite 
#rw=randread 
bs=4k 
direct=1 
numjobs=4 
group_reporting=1 

[rbd_iodepth32] 
iodepth=32 



fio write - 3 osd - 1 by node 
-------------------------- 
bw=15068KB/s, iops=3767 


fio read - 3 osd - 1 by node 
----------------------------- 
bw=130792KB/s, iops=32698 


fio write : 1 osd only 
------------------------ 
bw=5009.6KB/s, iops=1252 

fio read (osd) : 1 osd only 
--------------------------- 
bw=57016KB/s, iops=14254 


Reads seem to come from buffer cache, I don't known how to bypass it ? 


about writes: 

iostat show me around 100%util 

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util 
sdb 0,00 355,00 0,00 5225,00 0,00 29678,00 11,36 57,63 11,03 0,00 11,03 0,19 99,70 



But fio on local disk show me a lot more iops 

read 4k : fio --filename=/dev/sdb --direct=1 --rw=randread --bs=4k --iodepth=32 --group_reporting --invalidate=0 --name=abc --ioengine=aio 
bw=294312KB/s, iops=73577 

write 4k : fio --filename=/dev/sdb --direct=1 --rw=randwrite --bs=4k --iodepth=32 --group_reporting --invalidate=0 --name=abc --ioengine=aio 
bw=257806KB/s, iops=64451 




I'm going to try with ceph 0.85 to see improvements 

----- Mail original ----- 

De: "Dietmar Maurer" <dietmar at proxmox.com> 
À: "VELARTIS Philipp Dürhammer" <p.duerhammer at velartis.at>, "Alexandre DERUMIER (aderumier at odiso.com)" <aderumier at odiso.com> 
Cc: pve-devel at pve.proxmox.com 
Envoyé: Mercredi 10 Septembre 2014 13:22:46 
Objet: RE: [pve-devel] ceph firefly in pve repos 

> I would like to know your 4k results. (to see maximal iops performance) With the 
> option -b 4096 (you used the standard 4M with 16 threads) With 4MByte tests I 
> think it should be easy to max out the network... 

OK, here are the results running the fio rbd example benchmark (4K test): 

# ./fio examples/rbd.fio 
rbd_iodepth32: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd, iodepth=32 
fio-2.1.11-20-g9a44 
Starting 1 process 
rbd engine: RBD version: 0.1.8 
Jobs: 1 (f=1): [w(1)] [99.7% done] [0KB/5790KB/0KB /s] [0/1447/0 iops] [eta 00m:01s] 
rbd_iodepth32: (groupid=0, jobs=1): err= 0: pid=494227: Wed Sep 10 13:00:56 2014 
write: io=2048.0MB, bw=5875.2KB/s, iops=1468, runt=356952msec 

So 1468 is not really very high. 

@Alexandre: what values do you get? 
_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 



More information about the pve-devel mailing list