[PVE-User] truenas high i/o wait

[Tonči Stipičević]

Hello to all,

I've been using for years TrueNAS as full VM in Prox host.  No sata 
pass-through.

Host data pool consists of 6 x 4T enterprise sata drives in raid10  and 
that is where TrueNAS  disks images reside. ( 1 x boot, 3 x 2T virtual 
disk stripped)

'Till last week everything was working smoothly (in real time) but after 
upgrade to latest Prox host (community/no-subs)  TrueNAS VM (also on 
latest version)  started generating very high i/o waits (60-70%).

This host hosts 10 VMs and when TrueeNAS VM is not running , i/o wait 
lowers down to 2-3% and CPU %  runs between 20-30% depending on the VM 
load.  Most important -> CPU % utilization is much higher than i/o wait ...

But , as soon as TrueNAS boots  i/o waits jumps up to 60-70% ...   and 
everything slows down ...

I moved all other VMs to another host so to eliminate all possible 
influences , but TrueNAS alone generates the same high i/o wait ...

Is there any way to debug this high i/o wait  problem ? ...

Total DISK READ:       208.71 M/s | Total DISK WRITE:         0.00 B/s
Current DISK READ:     171.33 M/s | Current DISK WRITE:       0.00 B/s
     TID  PRIO  USER    DISK READ>  DISK WRITE  SWAPIN      IO COMMAND
1189075 be/4 root       50.80 M/s    0.00 B/s  ?unavailable?  kvm -id 
100 -name TN02,debug-threads=on -no-shutdown -chardev 
socket,id=qmp,path=/v~d=net0,rx_queue_size=1024,tx_queue_size=1024 
-machine type=pc+pve0 [iou-wrk-876705]
     370 be/0 root       40.69 M/s    0.00 B/s  ?unavailable? [z_rd_int]
1189069 be/4 root       17.61 M/s    0.00 B/s  ?unavailable?  kvm -id 
100 -name TN02,debug-threads=on -no-shutdown -chardev 
socket,id=qmp,path=/v~d=net0,rx_queue_size=1024,tx_queue_size=1024 
-machine type=pc+pve0 [iou-wrk-876705]
1188348 be/0 root       10.78 M/s    0.00 B/s  ?unavailable?  [zvol]

iotop shows that  z_rd_int and kvm (TrueNAS VM)  reads a lot ...   Why ? 
... why is so much reading needed   TrueNAS is doing nothing (VM 
internal 4-5% cpu utilization) , no file transfer or such ....


Till this weekend everything worked seemless and I 've never thought 
about migrating TrueNAS on baremetal or sata-passthrough


Thank you very very much for your help and support


Best regards

Tonči

-- 
srdačan pozdrav / best regards

Tonči Stipičević, dipl. ing. elektr.
direktor / manager

SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263

Podrška / Upravljanje IT sustavima za male i srednje tvrtke
Small & Medium Business IT Support / Management

mob: 091 1234003
www.suma-informatika.hr