[pbs-devel] Scheduler causing connectivity issues?

[Thomas Lamprecht]

Hi,

On 07/07/2022 17:49, Mark Schouten wrote:
> We’re getting complaints that one of our PBS’es is periodically unreachable. After investigation if the network might be at fault (even though it’s handling about 5.5Gbit at night), we found that PBS is piling up waiting connections every minute, on the minute, as you can see below. You see the output of `date`, combined with `ss -np | grep -c 8007`, the number of active connections.
> 
> At first I thought that pvestatd was ddossing PBS, but pvestatd seems to run more often than once in a minute.
> 
> So stracing the API process, I found that that process is also just waiting for something; must be the proxy-process.
> 
> grepping for ‘minute’ in the code, I stumbled upon the function `next_minute` in ./src/bin/proxmox-backup-proxy.rs. I’m not quite sure if I understand it correctly, but it seems that every minute, the scheduler is going to try and find out if it should be doing something.
> 
> Drilling down on that in my strace-foo, I think I see quite some read/write/rename actions on jobstate-files. Which leads me to conclude that the proxy process is waiting for the scheduler..
> 
> This is just guess-work, but you guys can surely find out better what’s going on than me.
> 
> This PBS is running with 45 users and 67 datastores.
> 
> Hope you guys can find something.. If I need to debug anything, let me know!

Thanks for the info, this already helps quite a bit. We'll look into it and re-check
with you if we need more info.

cheers,
Thomas