[pbs-devel] Too many open files

[Thomas Lamprecht]

Hi,

Am 07/09/2022 um 14:02 schrieb Mark Schouten:
> On our shared PBS-server, on busy moments, we get the following error:
> Sep  7 00:00:35 pbs001 proxmox-backup-proxy[9288]: error accepting tcp connection: Too many open files (os error 24)
> 

How many FDs are open for the proxy?

ls -1 /proc/$(pidof proxmox-backup-proxy)/fd | wc -l

> This is a box with quite a lot of datastores, but looking at the open files setting:
> root at pbs001:/var/log# cat /proc/sys/fs/file-max
> 9223372036854775807

This is not the relevant limit though, this is the upper limit of the
kernel for file handles
https://sysctl-explorer.net/fs/filemax%20&%20filenr/

The per-process file (hard) limit would be:
https://sysctl-explorer.net/fs/nr_open/

And as that mentions, that again depends on the rlimit, check the default (soft)
imits with

- as root:
  ulimit -a

- for the backup user
  su - backup -s /bin/bash -c "ulimit -n $[1024*1024]"


Or probably even easier, check the /proc entry of the proxy and api daemons:

cat /proc/$(pidof proxmox-backup-proxy)/limits
cat /proc/$(pidof proxmox-backup-api)/limits

> 
> That should be enough. 😄
> 
> Does PBS itself limit the open files somewhere?

No, but it doesn't increase it's soft limit to the hard limit, something we
want to do in the future.

As workaround try increasing soft (and possible hard) limit for root and backup
user, you can do that by editing `/etc/security/limits.conf`, adding lines like:

backup soft nofile 524288
root soft nofile 524288

(should set soft to current hard limit)

Or alternatively use a systemd service override to make it really process specific:

# systemctl edit proxmox-backup-proxy.service

There add a "[Service]" Section, if not already existing and a "LimitNOFILE=X" below,
e.g.:
 
[Service]
LimitNOFILE=524288

Save and close the editor and restart the service, then do the same for the
proxmox-backup.service

As said, we plan to upper that soft limit too in the future.

cheers,
Thomas