[pbs-devel] [PATCH proxmox{, -backup} v2 0/8] shared rate limiter for s3 client instances

[Hannes Laimer]

I did run into some conflicts when applying, mostly control files, other
than that and a small div by zero problem this L(very)GTM!

I did test this with a local S3 storage, worked as advertised.


Reviewed-by: Hannes Laimer <h.laimer at proxmox.com>
Tested-by: Hannes Laimer <h.laimer at proxmox.com>


On 9/16/25 14:42, Christian Ebner wrote:
> This patches implement a shared bandwidth rate limiter for the s3
> client instances, with the goal to allow users to avoid network
> congestion by traffic to the s3 backend.
> 
> The limiter is shared accorss all the clients using the same s3
> endpoint id, setting the limits on client instantiation. To utilize
> the pre-existing shared rate limiter implementation from PBS, factor
> it out into a dedicated crate, together with the non-shared rate limiter
> implementation from proxmox-http.
> 
> Expose the rate limit configuration in the s3 endpoint configuration
> and provide it to the client's https connector. Expose the settings
> also in the advanced options of the s3 endpoint edit/crate window.
> 
> Changes since version 1 (thanks @Thomas for feedback):
> - Move shared/non-shared rate limiter into dedicated crate
> - Use a more generic name for the magic number constant of the mmapped
>    state file
> 
> proxmox:
> 
> Christian Ebner (4):
>    rate-limiter: add crate for traffic rate limiter implementations
>    http: drop factored out rate limiter implementation
>    rest-server: optionally depend on factored out shared rate limiter
>    s3-client: add shared rate limiter via https connector
> 
>   Cargo.toml                                    |   2 +
>   proxmox-http/Cargo.toml                       |   4 +-
>   proxmox-http/debian/control                   |  19 +--
>   proxmox-http/src/client/connector.rs          |   3 +-
>   proxmox-http/src/lib.rs                       |   5 -
>   proxmox-http/src/rate_limited_stream.rs       |   2 +-
>   proxmox-rate-limiter/Cargo.toml               |  29 ++++
>   proxmox-rate-limiter/debian/changelog         |   5 +
>   proxmox-rate-limiter/debian/control           |  70 ++++++++++
>   proxmox-rate-limiter/debian/copyright         |  18 +++
>   proxmox-rate-limiter/debian/debcargo.toml     |   7 +
>   proxmox-rate-limiter/src/lib.rs               |  13 ++
>   .../src/rate_limiter.rs                       |   0
>   .../src/shared_rate_limiter.rs                | 130 ++++++++++++++++++
>   proxmox-rest-server/Cargo.toml                |   2 +
>   proxmox-rest-server/debian/control            |  14 +-
>   proxmox-rest-server/src/connection.rs         |   4 +-
>   proxmox-s3-client/Cargo.toml                  |   7 +-
>   proxmox-s3-client/debian/control              |   7 +-
>   proxmox-s3-client/examples/s3_client.rs       |   1 +
>   proxmox-s3-client/src/api_types.rs            |  29 ++++
>   proxmox-s3-client/src/client.rs               |  63 ++++++++-
>   proxmox-s3-client/src/lib.rs                  |   4 +-
>   23 files changed, 399 insertions(+), 39 deletions(-)
>   create mode 100644 proxmox-rate-limiter/Cargo.toml
>   create mode 100644 proxmox-rate-limiter/debian/changelog
>   create mode 100644 proxmox-rate-limiter/debian/control
>   create mode 100644 proxmox-rate-limiter/debian/copyright
>   create mode 100644 proxmox-rate-limiter/debian/debcargo.toml
>   create mode 100644 proxmox-rate-limiter/src/lib.rs
>   rename {proxmox-http => proxmox-rate-limiter}/src/rate_limiter.rs (100%)
>   create mode 100644 proxmox-rate-limiter/src/shared_rate_limiter.rs
> 
> 
> proxmox-backup:
> 
> Christian Ebner (4):
>    traffic control: use factored out shared rate limiter
>    api: config: update s3 endpoint rate limits in config
>    datastore: s3: set rate limiter options for s3 client
>    ui: expose rate and burst limits for s3 endpoints
> 
>   Cargo.toml                       |   3 +
>   debian/control                   |   1 -
>   pbs-client/Cargo.toml            |   3 +-
>   pbs-client/src/http_client.rs    |   3 +-
>   pbs-datastore/src/datastore.rs   |  18 ++++-
>   src/api2/admin/s3.rs             |   9 ++-
>   src/api2/config/s3.rs            |  34 ++++++++-
>   src/tools/mod.rs                 |   4 -
>   src/tools/shared_rate_limiter.rs | 122 -------------------------------
>   src/traffic_control_cache.rs     |   8 +-
>   www/window/S3ClientEdit.js       |  34 +++++++++
>   11 files changed, 102 insertions(+), 137 deletions(-)
>   delete mode 100644 src/tools/shared_rate_limiter.rs
> 
> 
> Summary over all repositories:
>    34 files changed, 501 insertions(+), 176 deletions(-)
>