[pbs-devel] [RFC v2 proxmox/bookworm-stable proxmox-backup 00/42] S3 storage backend for datastores

[Lukas Wagner]

On  2025-05-29 16:31, Christian Ebner wrote:
> Disclaimer: These patches are in a development state and are not
> intended for production use.
> 
> This patch series aims to add S3 compatible object stores as storage
> backend for PBS datastores. A PBS local cache store using the regular
> datastore layout is used for faster operation, bypassing requests to
> the S3 api when possible. Further, the local cache store allows to
> keep frequently used chunks and is used to avoid expensive metadata
> updates on the object store, e.g. by using local marker file during
> garbage collection.
> 
> Backups are created by upload chunks to the corresponding S3 bucket,
> while keeping the index files in the local cache store, on backup
> finish, the snapshot metadata are persisted to the S3 storage backend.
> 
> Snapshot restores read chunks preferably from the local cache store,
> downloading and insterting them if not present from the S3 object
> store.
> 
> Listing and snapsoht metadata operation currently rely soly on the
> local cache store, with the intention to provide a mechanism to
> re-sync and merge with object stored on the S3 backend if requested.
> 
> Sending this patch series as RFC to get some initial feedback, mostly
> on the S3 client implementation part and the corresponding
> configuration integration with PBS, which is already in an advanced
> stage and warants initial review and real world testing.
> 
> Datastore operations on the S3 backend are still work in progress,
> but feedback on that is appreciated very much as well.
> 
> Among the open points still being worked on are:
> - Consistency between local cache and S3 store.
> - Sync and merge of namespace, group snapshot and index files when
>   required or requested.
> - Advanced packing mechanism for chunks to significantly reduce the
>   number of api requests and therefore be more cost effective.
> - Reduction of in-memory copies for chunks/blobs and recalculation of
>   checksums.
> 

Had some off-list discussions with Christian about a couple of aspects of this
version of the series, here is a quick summary:

With regards to the 'Create Datastore' dialog:
In the current version, the S3 bucket can be selected under 'Advanced'. This might be
a bit hard to find for some users, so I suggested revising the dialog in general.
For example, perhaps we could start by having the user select a type
right away (Normal / Removable / Existing / S3-backed), and then show or hide
the required UI elements accordingly. For the S3-backed store specifically,
my intuitive expectation would be to first select the bucket, and then, as a second
step, choose the location for the local cache.

If we still want to keep S3 a bit hidden for now, we could either add a global setting
or an option within the dialog under 'Advanced' to opt into the experimental S3 feature,
or something along those lines.

Also I mentioned that the 'trash-can' icon - albeit being a bucket - might not be the best
fit for 'S3 Buckets', because it creates the association of 'trash' or 'throwing something away'.
I suggested fa-cloud-upload [1] instead for now, which should be quite fitting for a 'syncing
something to the cloud' feature.

Furthermore, I suggested that maybe the 'bucket' should be a property of the
datastore config, not of the s3 config. That way, the s3 config contains only the
connection info and credentials, which make it easy to use the same s3 config for
multiple datastores which use different buckets as as a backing storage.

Last, we probably should encode the name of the datastore into the key
of the S3 object, unless we want a strict 1:1 relationship between bucket and
datastore. Maybe it could even make sense to allow the user to set custom
prefixes for objects, in case they want PBSs objects not to be stored at the
top level of the bucket.

[1] https://fontawesome.com/v4/icon/cloud-upload

-- 
- Lukas