[pbs-devel] applied: [PATCH proxmox{, -backup} v11 00/50] fix #2943: S3 storage backend for datastores

[Thomas Lamprecht]

Am 22.07.25 um 12:11 schrieb Christian Ebner:> Disclaimer: These patches are still in an experimental state and not
> intended for production use.
> 
> This patch series aims to add S3 compatible object stores as storage
> backend for PBS datastores. A PBS local cache store using the regular
> datastore layout is used for faster operation, bypassing requests to
> the S3 api when possible. Further, the local cache store allows to
> keep frequently used chunks and is used to avoid expensive metadata
> updates on the object store, e.g. by using local marker file during
> garbage collection.
> 
> Backups are created by upload chunks to the corresponding S3 bucket,
> while keeping the index files in the local cache store, on backup
> finish, the snapshot metadata are persisted to the S3 storage backend.
> 
> Snapshot restores read chunks preferably from the local cache store,
> downloading and insterting them if not present from the S3 object
> store. Listing and snapsoht metadata operation currently rely soly on
> the local cache store.
> 
> Currently chunks use a 1:1 mapping to S3 objects. An advanced packing
> mechanism for chunks to significantly reduce the number of api
> requests and therefore be more cost effective will be implemented as
> followup patches.

applied, huge thanks to you for this quality implementation and the
flexibility for some non-trivial requested changes towards the end,
appreciated!

Also thanks to Lukas and Hannes for their review and testing, that gave
me much more confidence with this and definitively helped to catch
some issues upfront.