[PVE-User] Peak load at 7.30AM...

[dorsy]

Dude, do performance monitoring for the VM and/or host!

For example sar, netdata or whatever You would like, that way You could 
tell what is happening, what processes are consuming resources (cpu, 
ram, disk at least) when this happens.

On 4/26/2023 12:48 PM, Marco Gaiarin wrote:
> Situation: a debian stretch mostly 'samba server' for a 150+ clients, in a
> couple of phisical server; VM get replicated between the two server every 30
> minutes.
>
> Very frequently at 7.30 the VM got a high peak rate, becaming mostly
> irresponsive; after fiddling a bit, i've added a watchdog LOAD limit, and
> the VM now reboot.
>
> Looking at logs, it seems caused by the replica of the 7.30:
>
> 	Apr 26 07:30:12 vdmsv1 qemu-ga: info: guest-ping called
> 	Apr 26 07:30:13 vdmsv1 qemu-ga: info: guest-fsfreeze called
>
> and after some (6 to 8 minutes) watchdog reset it:
>
> 	Apr 26 07:36:11 vdmsv1 watchdog[2525]: loadavg 57 33 14 is higher than the given threshold 80 32 16!
> 	Apr 26 07:36:11 vdmsv1 watchdog[2525]: shutting down the system because of error 253 = 'load average too high'
>
>
> Some notes:
>
> 1) the replica run every 30 minutes; no other of the 47 replicas of the day
>    seems sufficient to trigger a reboot.
>
> 2) the phisical server during the high peak seems totally unaffected (no
>    high iodelay, no sensible load/cpu...).
>
> 3) at 7.30 there's no user (they arrive around 8.15).
>
>
> I'm doing some hypotesis; for example debian by default rotate logs at 6.30,
> but looking at file dates logs are completely rotated before the 7.00, so
> the 7.00 replica could have triggered the reboot...
>
>
> Someone have some hint on how can i debug this!?
>
>
> Thanks.
>