[pve-devel] pve-manager and disk IO monitoring

Fri Sep 30 07:03:00 CEST 2016

"Running a fio test also only shows marginal performance difference
between scsi-block and scsi-generic"

I think that 11% difference is not so marginal.
I'm curious to see difference with full flash array, if we have the same cpu iothread bottleneck like ceph, with scsi-block vs scsi-generic.

Maybe can we add an option to choose between scsi-block && scsi-generic

----- Mail original -----
De: "datanom.net" <mir at datanom.net>
À: "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Vendredi 30 Septembre 2016 01:23:20
Objet: Re: [pve-devel] pve-manager and disk IO monitoring

On Fri, 30 Sep 2016 00:51:06 +0200 
Michael Rasmussen <mir at datanom.net> wrote: 

> 
> So my question is: Why use scsi-generic instead of scsi-block when 
> scsi-generic prevents blockstats? 
> 
Running a fio test also only shows marginal performance difference 
between scsi-block and scsi-generic 

-device scsi-block 
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64 
fio-2.1.11 
Starting 1 process 
iometer: Laying out IO file(s) (1 file(s) / 3072MB) 
Jobs: 1 (f=1): [m(1)] [100.0% done] [73928KB/19507KB/0KB /s] [17.3K/4381/0 iops] [eta 00m:00s] 
iometer: (groupid=0, jobs=1): err= 0: pid=1568: Fri Sep 30 01:17:05 2016 
Description : [Emulation of Intel IOmeter File Server Access Pattern] 
read : io=2454.9MB, bw=87501KB/s, iops=14328, runt= 28728msec 
slat (usec): min=2, max=4703, avg=10.47, stdev=16.99 
clat (usec): min=315, max=1505.6K, avg=3479.55, stdev=8270.22 
lat (usec): min=321, max=1505.6K, avg=3490.40, stdev=8270.14 
clat percentiles (usec): 
| 1.00th=[ 1768], 5.00th=[ 2480], 10.00th=[ 2640], 20.00th=[ 2864], 
| 30.00th=[ 2960], 40.00th=[ 3056], 50.00th=[ 3088], 60.00th=[ 3152], 
| 70.00th=[ 3248], 80.00th=[ 3376], 90.00th=[ 3824], 95.00th=[ 4448], 
| 99.00th=[ 8768], 99.50th=[13120], 99.90th=[52992], 99.95th=[103936], 
| 99.99th=[536576] 
bw (KB /s): min= 7148, max=193016, per=100.00%, avg=87866.39, stdev=28395.12 
write: io=631998KB, bw=21999KB/s, iops=3590, runt= 28728msec 
slat (usec): min=4, max=9301, avg=12.69, stdev=33.41 
clat (usec): min=299, max=778312, avg=3871.08, stdev=7378.66 
lat (usec): min=305, max=778320, avg=3884.17, stdev=7378.66 
clat percentiles (msec): 
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], 
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], 
| 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 7], 
| 99.00th=[ 13], 99.50th=[ 19], 99.90th=[ 55], 99.95th=[ 101], 
| 99.99th=[ 537] 
bw (KB /s): min= 1524, max=46713, per=100.00%, avg=22089.18, stdev=7184.64 
lat (usec) : 500=0.01%, 750=0.03%, 1000=0.06% 
lat (msec) : 2=1.29%, 4=88.78%, 10=8.94%, 20=0.56%, 50=0.22% 
lat (msec) : 100=0.05%, 250=0.05%, 500=0.01%, 750=0.01%, 1000=0.01% 
lat (msec) : 2000=0.01% 
cpu : usr=8.24%, sys=28.49%, ctx=451227, majf=0, minf=8 
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% 
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% 
issued : total=r=411627/w=103162/d=0, short=r=0/w=0/d=0 
latency : target=0, window=0, percentile=100.00%, depth=64 

Run status group 0 (all jobs): 
READ: io=2454.9MB, aggrb=87501KB/s, minb=87501KB/s, maxb=87501KB/s, mint=28728msec, maxt=28728msec 
WRITE: io=631997KB, aggrb=21999KB/s, minb=21999KB/s, maxb=21999KB/s, mint=28728msec, maxt=28728msec 

Disk stats (read/write): 
sda: ios=407383/102110, merge=123/54, ticks=1413272/456272, in_queue=1869620, util=99.71% 

-device scsi-generic 
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64 
fio-2.1.11 
Starting 1 process 
iometer: Laying out IO file(s) (1 file(s) / 3072MB) 
Jobs: 1 (f=1): [m(1)] [100.0% done] [64339KB/16908KB/0KB /s] [15.2K/3816/0 iops] [eta 00m:00s] 
iometer: (groupid=0, jobs=1): err= 0: pid=701: Fri Sep 30 01:20:45 2016 
Description : [Emulation of Intel IOmeter File Server Access Pattern] 
read : io=2454.9MB, bw=88384KB/s, iops=14473, runt= 28441msec 
slat (usec): min=5, max=5814, avg=10.86, stdev=21.71 
clat (usec): min=459, max=885935, avg=3451.71, stdev=3297.21 
lat (usec): min=526, max=885944, avg=3462.97, stdev=3297.14 
clat percentiles (msec): 
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], 
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], 
| 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 4], 95.00th=[ 5], 
| 99.00th=[ 8], 99.50th=[ 11], 99.90th=[ 23], 99.95th=[ 63], 
| 99.99th=[ 153] 
bw (KB /s): min=46295, max=139025, per=100.00%, avg=88833.25, stdev=22609.61 
write: io=631998KB, bw=22221KB/s, iops=3627, runt= 28441msec 
slat (usec): min=6, max=3864, avg=12.96, stdev=24.18 
clat (usec): min=582, max=156777, avg=3801.87, stdev=3128.06 
lat (usec): min=610, max=156789, avg=3815.24, stdev=3128.36 
clat percentiles (msec): 
| 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], 
| 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], 
| 70.00th=[ 4], 80.00th=[ 4], 90.00th=[ 5], 95.00th=[ 7], 
| 99.00th=[ 11], 99.50th=[ 15], 99.90th=[ 49], 99.95th=[ 74], 
| 99.99th=[ 153] 
bw (KB /s): min=11151, max=36378, per=100.00%, avg=22332.46, stdev=5869.71 
lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% 
lat (msec) : 2=0.67%, 4=90.61%, 10=8.03%, 20=0.53%, 50=0.07% 
lat (msec) : 100=0.05%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% 
cpu : usr=8.73%, sys=29.82%, ctx=445178, majf=0, minf=8 
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% 
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% 
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% 
issued : total=r=411627/w=103162/d=0, short=r=0/w=0/d=0 
latency : target=0, window=0, percentile=100.00%, depth=64 

Run status group 0 (all jobs): 
READ: io=2454.9MB, aggrb=88384KB/s, minb=88384KB/s, maxb=88384KB/s, mint=28441msec, maxt=28441msec 
WRITE: io=631997KB, aggrb=22221KB/s, minb=22221KB/s, maxb=22221KB/s, mint=28441msec, maxt=28441msec 

Disk stats (read/write): 
sda: ios=410496/102881, merge=0/5, ticks=1411188/396440, in_queue=1807996, util=99.70% 

-- 
Hilsen/Regards 
Michael Rasmussen 

Get my public GnuPG keys: 
michael <at> rasmussen <dot> cc 
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xD3C9A00E 
mir <at> datanom <dot> net 
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE501F51C 
mir <at> miras <dot> org 
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xE3E80917 
-------------------------------------------------------------- 
/usr/games/fortune -es says: 
We don't know who it was that discovered water, but we're pretty sure 
that it wasn't a fish. 
-- Marshall McLuhan 

_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 