[pbs-devel] Scheduler causing connectivity issues?

[Jorge Boncompte]

El 8/7/22 a las 9:09, Thomas Lamprecht escribió:
> Hi,
> 
> On 07/07/2022 17:49, Mark Schouten wrote:
>> We’re getting complaints that one of our PBS’es is periodically unreachable. After investigation if the network might be at fault (even though it’s handling about 5.5Gbit at night), we found that PBS is piling up waiting connections every minute, on the minute, as you can see below. You see the output of `date`, combined with `ss -np | grep -c 8007`, the number of active connections.
>>
>> At first I thought that pvestatd was ddossing PBS, but pvestatd seems to run more often than once in a minute.
>>
>> So stracing the API process, I found that that process is also just waiting for something; must be the proxy-process.
>>
>> grepping for ‘minute’ in the code, I stumbled upon the function `next_minute` in ./src/bin/proxmox-backup-proxy.rs. I’m not quite sure if I understand it correctly, but it seems that every minute, the scheduler is going to try and find out if it should be doing something.
>>
>> Drilling down on that in my strace-foo, I think I see quite some read/write/rename actions on jobstate-files. Which leads me to conclude that the proxy process is waiting for the scheduler..
>>
>> This is just guess-work, but you guys can surely find out better what’s going on than me.
>>
>> This PBS is running with 45 users and 67 datastores.
>>
>> Hope you guys can find something.. If I need to debug anything, let me know!
> 
> Thanks for the info, this already helps quite a bit. We'll look into it and re-check
> with you if we need more info.

	Hi, We've been having a problem that resembles this one with several
proxmox-backup-server 2.2.x versions. Our PBS stopped accepting backups
jobs sometimes, but if we retried manully they started fine. The only
message I could find was:

  backup failed: could not activate storage 'XXXXXXX': XXXXXX: error
fetching datastores - 500 Can't connect to XXXXXXXX:8007

	Restarting the proxy seemed to help to get it working one or two days
more. We have reverted proxmox-backup-server to 2.1.8-1 and every is
fine again.

	Regards.

> 
> cheers,
> Thomas
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel