[pbs-devel] Scheduler causing connectivity issues?

[Mark Schouten]

Hi,

We’re getting complaints that one of our PBS’es is periodically 
unreachable. After investigation if the network might be at fault (even 
though it’s handling about 5.5Gbit at night), we found that PBS is 
piling up waiting connections every minute, on the minute, as you can 
see below. You see the output of `date`, combined with `ss -np | grep -c 
8007`, the number of active connections.

At first I thought that pvestatd was ddossing PBS, but pvestatd seems to 
run more often than once in a minute.

So stracing the API process, I found that that process is also just 
waiting for something; must be the proxy-process.

grepping for ‘minute’ in the code, I stumbled upon the function 
`next_minute` in ./src/bin/proxmox-backup-proxy.rs. I’m not quite sure 
if I understand it correctly, but it seems that every minute, the 
scheduler is going to try and find out if it should be doing something.

Drilling down on that in my strace-foo, I think I see quite some 
read/write/rename actions on jobstate-files. Which leads me to conclude 
that the proxy process is waiting for the scheduler..

This is just guess-work, but you guys can surely find out better what’s 
going on than me.

This PBS is running with 45 users and 67 datastores.

Hope you guys can find something.. If I need to debug anything, let me 
know!


============
Timestamp                        | `ss -np|grep -c 8007`
Thu 07 Jul 2022 05:38:00 PM CEST | 13
Thu 07 Jul 2022 05:38:00 PM CEST | 24
Thu 07 Jul 2022 05:38:01 PM CEST | 32
Thu 07 Jul 2022 05:38:01 PM CEST | 45
Thu 07 Jul 2022 05:38:02 PM CEST | 58
Thu 07 Jul 2022 05:38:02 PM CEST | 65
Thu 07 Jul 2022 05:38:03 PM CEST | 68
Thu 07 Jul 2022 05:38:03 PM CEST | 76
Thu 07 Jul 2022 05:38:04 PM CEST | 79
Thu 07 Jul 2022 05:38:05 PM CEST | 82
Thu 07 Jul 2022 05:38:05 PM CEST | 88
Thu 07 Jul 2022 05:38:06 PM CEST | 96
Thu 07 Jul 2022 05:38:06 PM CEST | 102
Thu 07 Jul 2022 05:38:07 PM CEST | 104
Thu 07 Jul 2022 05:38:07 PM CEST | 111
Thu 07 Jul 2022 05:38:08 PM CEST | 124
Thu 07 Jul 2022 05:38:08 PM CEST | 130
Thu 07 Jul 2022 05:38:09 PM CEST | 133
Thu 07 Jul 2022 05:38:09 PM CEST | 137
Thu 07 Jul 2022 05:38:10 PM CEST | 22
Thu 07 Jul 2022 05:38:11 PM CEST | 23
Thu 07 Jul 2022 05:39:00 PM CEST | 20
Thu 07 Jul 2022 05:39:01 PM CEST | 36
Thu 07 Jul 2022 05:39:01 PM CEST | 48
Thu 07 Jul 2022 05:39:02 PM CEST | 57
Thu 07 Jul 2022 05:39:02 PM CEST | 64
Thu 07 Jul 2022 05:39:03 PM CEST | 69
Thu 07 Jul 2022 05:39:03 PM CEST | 76
Thu 07 Jul 2022 05:39:04 PM CEST | 78
Thu 07 Jul 2022 05:39:04 PM CEST | 84
Thu 07 Jul 2022 05:39:05 PM CEST | 88
Thu 07 Jul 2022 05:39:06 PM CEST | 96
Thu 07 Jul 2022 05:39:06 PM CEST | 102
Thu 07 Jul 2022 05:39:07 PM CEST | 104
Thu 07 Jul 2022 05:39:07 PM CEST | 111
Thu 07 Jul 2022 05:39:08 PM CEST | 120
Thu 07 Jul 2022 05:39:08 PM CEST | 127
Thu 07 Jul 2022 05:39:09 PM CEST | 131
Thu 07 Jul 2022 05:39:09 PM CEST | 133
Thu 07 Jul 2022 05:39:10 PM CEST | 29
Thu 07 Jul 2022 05:39:10 PM CEST | 24
Thu 07 Jul 2022 05:40:00 PM CEST | 21
Thu 07 Jul 2022 05:40:01 PM CEST | 33
Thu 07 Jul 2022 05:40:01 PM CEST | 45
Thu 07 Jul 2022 05:40:02 PM CEST | 58
Thu 07 Jul 2022 05:40:02 PM CEST | 64
Thu 07 Jul 2022 05:40:03 PM CEST | 70
Thu 07 Jul 2022 05:40:03 PM CEST | 75
Thu 07 Jul 2022 05:40:04 PM CEST | 79
Thu 07 Jul 2022 05:40:04 PM CEST | 83
Thu 07 Jul 2022 05:40:05 PM CEST | 88
Thu 07 Jul 2022 05:40:05 PM CEST | 96
Thu 07 Jul 2022 05:40:06 PM CEST | 102
Thu 07 Jul 2022 05:40:07 PM CEST | 105
Thu 07 Jul 2022 05:40:07 PM CEST | 113
Thu 07 Jul 2022 05:40:08 PM CEST | 122
Thu 07 Jul 2022 05:40:08 PM CEST | 129
Thu 07 Jul 2022 05:40:09 PM CEST | 134
Thu 07 Jul 2022 05:40:09 PM CEST | 135
Thu 07 Jul 2022 05:40:10 PM CEST | 27


—
Mark Schouten, CTO
Tuxis B.V.
mark at tuxis.nl