[PVE-User] truenas high i/o wait

[Tonči Stipičević]

continued from previous post :

on the other host  TrueNAS VM (TrueNAS-13.0-U6.2)   is causing 1GB/s 
read ... :    ...   Is this host issue or VM internal issue ?  ... where 
to "intervene"?


IOTOP:

Total DISK READ:      1037.23 M/s | Total DISK WRITE: 1060.48 K/s
Current DISK READ:     702.02 M/s | Current DISK WRITE:    1679.83 K/s
     TID  PRIO  USER    DISK READ>  DISK WRITE COMMAND
1803753 be/4 root      283.67 M/s    0.00 B/s kvm -id 106 -name 
TN0103,debug-threads=on -no-shutdown -chardev 
socket,id=~024,tx_queue_size=256,bootindex=102 -machine type=pc+pve0 
[iou-wrk-1726849]
1803546 be/0 root       59.29 M/s  811.01 B/s [zvol_tq-3]
1803545 be/0 root       56.26 M/s    0.00 B/s [zvol_tq-3]
1803544 be/0 root       54.22 M/s    0.00 B/s [zvol_tq-3]


Thank you

and

BR

srdačan pozdrav / best regards

Tonči Stipičević, dipl. ing. elektr.
direktor / manager

SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263

Podrška / Upravljanje IT sustavima za male i srednje tvrtke
Small & Medium Business IT Support / Management

mob: 091 1234003
www.suma-informatika.hr

On 06. 10. 2024. 14:42, Tonči Stipičević wrote:
> Hello to all,
>
> I've been using for years TrueNAS as full VM in Prox host.  No sata 
> pass-through.
>
> Host data pool consists of 6 x 4T enterprise sata drives in raid10  
> and that is where TrueNAS  disks images reside. ( 1 x boot, 3 x 2T 
> virtual disk stripped)
>
> 'Till last week everything was working smoothly (in real time) but 
> after upgrade to latest Prox host (community/no-subs)  TrueNAS VM 
> (also on latest version)  started generating very high i/o waits 
> (60-70%).
>
> This host hosts 10 VMs and when TrueeNAS VM is not running , i/o wait 
> lowers down to 2-3% and CPU %  runs between 20-30% depending on the VM 
> load.  Most important -> CPU % utilization is much higher than i/o 
> wait ...
>
> But , as soon as TrueNAS boots  i/o waits jumps up to 60-70% ... and 
> everything slows down ...
>
> I moved all other VMs to another host so to eliminate all possible 
> influences , but TrueNAS alone generates the same high i/o wait ...
>
> Is there any way to debug this high i/o wait  problem ? ...
>
> Total DISK READ:       208.71 M/s | Total DISK WRITE:         0.00 B/s
> Current DISK READ:     171.33 M/s | Current DISK WRITE:       0.00 B/s
>     TID  PRIO  USER    DISK READ>  DISK WRITE  SWAPIN      IO COMMAND
> 1189075 be/4 root       50.80 M/s    0.00 B/s  ?unavailable?  kvm -id 
> 100 -name TN02,debug-threads=on -no-shutdown -chardev 
> socket,id=qmp,path=/v~d=net0,rx_queue_size=1024,tx_queue_size=1024 
> -machine type=pc+pve0 [iou-wrk-876705]
>     370 be/0 root       40.69 M/s    0.00 B/s  ?unavailable? [z_rd_int]
> 1189069 be/4 root       17.61 M/s    0.00 B/s  ?unavailable?  kvm -id 
> 100 -name TN02,debug-threads=on -no-shutdown -chardev 
> socket,id=qmp,path=/v~d=net0,rx_queue_size=1024,tx_queue_size=1024 
> -machine type=pc+pve0 [iou-wrk-876705]
> 1188348 be/0 root       10.78 M/s    0.00 B/s  ?unavailable? [zvol]
>
> iotop shows that  z_rd_int and kvm (TrueNAS VM)  reads a lot ... Why ? 
> ... why is so much reading needed   TrueNAS is doing nothing (VM 
> internal 4-5% cpu utilization) , no file transfer or such ....
>
>
> Till this weekend everything worked seemless and I 've never thought 
> about migrating TrueNAS on baremetal or sata-passthrough
>
>
> Thank you very very much for your help and support
>
>
> Best regards
>
> Tonči
>