[PVE-User] truenas high i/o wait FASLE Alarm

[Tonči Stipičević]

Hello to all,

fortunately it turned out to be that once a month TrueNAS does zfs 
volume scrub  ( I completely forgot that :(   ) and it lasted more than 
24h ...  after that i/o wait lowered to minimum ...

I apologize for disturbance

BR

Tonči

srdačan pozdrav / best regards

Tonči Stipičević, dipl. ing. elektr.
direktor / manager

SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263

Podrška / Upravljanje IT sustavima za male i srednje tvrtke
Small & Medium Business IT Support / Management

mob: 091 1234003
www.suma-informatika.hr

On 06. 10. 2024. 15:01, Tonči Stipičević wrote:
> continued from previous post :
>
> on the other host  TrueNAS VM (TrueNAS-13.0-U6.2)   is causing 1GB/s 
> read ... :    ...   Is this host issue or VM internal issue ?  ... 
> where to "intervene"?
>
>
> IOTOP:
>
> Total DISK READ:      1037.23 M/s | Total DISK WRITE: 1060.48 K/s
> Current DISK READ:     702.02 M/s | Current DISK WRITE:    1679.83 K/s
>     TID  PRIO  USER    DISK READ>  DISK WRITE COMMAND
> 1803753 be/4 root      283.67 M/s    0.00 B/s kvm -id 106 -name 
> TN0103,debug-threads=on -no-shutdown -chardev 
> socket,id=~024,tx_queue_size=256,bootindex=102 -machine type=pc+pve0 
> [iou-wrk-1726849]
> 1803546 be/0 root       59.29 M/s  811.01 B/s [zvol_tq-3]
> 1803545 be/0 root       56.26 M/s    0.00 B/s [zvol_tq-3]
> 1803544 be/0 root       54.22 M/s    0.00 B/s [zvol_tq-3]
>
>
> Thank you
>
> and
>
> BR
>
> srdačan pozdrav / best regards
>
> Tonči Stipičević, dipl. ing. elektr.
> direktor / manager
>
> SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263
>
> Podrška / Upravljanje IT sustavima za male i srednje tvrtke
> Small & Medium Business IT Support / Management
>
> mob: 091 1234003
> www.suma-informatika.hr
>
> On 06. 10. 2024. 14:42, Tonči Stipičević wrote:
>> Hello to all,
>>
>> I've been using for years TrueNAS as full VM in Prox host.  No sata 
>> pass-through.
>>
>> Host data pool consists of 6 x 4T enterprise sata drives in raid10  
>> and that is where TrueNAS  disks images reside. ( 1 x boot, 3 x 2T 
>> virtual disk stripped)
>>
>> 'Till last week everything was working smoothly (in real time) but 
>> after upgrade to latest Prox host (community/no-subs) TrueNAS VM 
>> (also on latest version)  started generating very high i/o waits 
>> (60-70%).
>>
>> This host hosts 10 VMs and when TrueeNAS VM is not running , i/o wait 
>> lowers down to 2-3% and CPU %  runs between 20-30% depending on the 
>> VM load.  Most important -> CPU % utilization is much higher than i/o 
>> wait ...
>>
>> But , as soon as TrueNAS boots  i/o waits jumps up to 60-70% ... and 
>> everything slows down ...
>>
>> I moved all other VMs to another host so to eliminate all possible 
>> influences , but TrueNAS alone generates the same high i/o wait ...
>>
>> Is there any way to debug this high i/o wait  problem ? ...
>>
>> Total DISK READ:       208.71 M/s | Total DISK WRITE: 0.00 B/s
>> Current DISK READ:     171.33 M/s | Current DISK WRITE: 0.00 B/s
>>     TID  PRIO  USER    DISK READ>  DISK WRITE  SWAPIN      IO COMMAND
>> 1189075 be/4 root       50.80 M/s    0.00 B/s  ?unavailable? kvm -id 
>> 100 -name TN02,debug-threads=on -no-shutdown -chardev 
>> socket,id=qmp,path=/v~d=net0,rx_queue_size=1024,tx_queue_size=1024 
>> -machine type=pc+pve0 [iou-wrk-876705]
>>     370 be/0 root       40.69 M/s    0.00 B/s  ?unavailable? [z_rd_int]
>> 1189069 be/4 root       17.61 M/s    0.00 B/s  ?unavailable? kvm -id 
>> 100 -name TN02,debug-threads=on -no-shutdown -chardev 
>> socket,id=qmp,path=/v~d=net0,rx_queue_size=1024,tx_queue_size=1024 
>> -machine type=pc+pve0 [iou-wrk-876705]
>> 1188348 be/0 root       10.78 M/s    0.00 B/s  ?unavailable? [zvol]
>>
>> iotop shows that  z_rd_int and kvm (TrueNAS VM)  reads a lot ... Why 
>> ? ... why is so much reading needed   TrueNAS is doing nothing (VM 
>> internal 4-5% cpu utilization) , no file transfer or such ....
>>
>>
>> Till this weekend everything worked seemless and I 've never thought 
>> about migrating TrueNAS on baremetal or sata-passthrough
>>
>>
>> Thank you very very much for your help and support
>>
>>
>> Best regards
>>
>> Tonči
>>