[pbs-devel] Authentication performance

[Shannon Sterz]

On Fri Dec 13, 2024 at 5:21 PM CET, Mark Schouten wrote:
> Hi All,
>
> As you may know, we offer free PBS accounts up to 150GB and this is
> quite popular. So popular, that I am seeing performance issues on the
> PBS-end. Yesterday, we rebooted a PBS and directly after booting, the
> load was about 14, without any running tasks.
>
> I think one of the obvious bottlenecks is authentication, as we’re using
> PBS in a slightly different manner than it was probably initially
> intended. Because we’re currently doing about 240 succesful
> authentication-requests per second (on average) on this server.
>
> But we do notice that the PBS is responding slow, even though there are
> not that many tasks. Which leads me to believe that if we can tweak the
> authentication process, OR one of the calls that a PVE cluster is doing
> on the PBS.
>
> I’m thinking one of these two calls:
>
> 1879332 GET /api2/json/admin/datastore (on average 30 per second)
> 14587762 POST /api2/json/access/ticket. (on average 238 per second)
>
> Where the number is the amount of requests today, so far.

Hi,

would you mind sharing either `authkey.pub` or the output of the
following commands:

head --lines=1 /etc/proxmox-backup/authkey.key
cat /etc/proxmox-backup/authkey.key | wc -l

The first should give the PEM header of the authkey whereas the second
provides the amount of lines that the key takes up in the file. Both
give an indication whether you are using the legacy RSA keys or newer
Ed25519 keys. The later should provide more performance, security should
not be affected much by this change. If the output of the commands look
like this:

-----BEGIN PRIVATE KEY-----
3

Then you are using the newer keys. There currently isn't a recommended
way to upgrade the keys. However, in theory you should be able to remove
the old keys, re-start PBS and it should just generate keys in the new
format. Note that this will logout anyone that is currently
authenticated and they'll have to re-authenticate.

In general, tokens should still be fater to authenticate so we'd
recommend that you try to get your users to switch to token-based
authentication where possible. Improving performance there is a bit
trickier though, as it often comes with a security trade-off (in the
background we use yescrypt fo the authentication there, that
delibaretely adds a work factor). However, we may be able to improve
performance a bit via caching methods or similar.

>
> Again, I know this is a different situation from normal, but if we can
> improve PBS even further. My knowledge of Rust is minimal, but if
> someone can point me to the logic here, I may be able to see possible
> improvements myself. Although you guys are probably much more capable at
> that.
>
> Let me know what you think, thanks!
>
> —
> Mark Schouten
> CTO, Tuxis B.V.
> +31 318 200208 / mark at tuxis.nl