[pbs-devel] [PATCH proxmox-backup v6 21/37] datastore: implement garbage collection for s3 backend

[Christian Ebner]

On 7/10/25 08:59, Thomas Lamprecht wrote:
> I'm reading through this in not a very orderly fashion, so probably
> won't be a very structured review, but some comments here and there.

Okay thanks, all feedback and additional pair of eyes is very welcome!

> Am 08.07.25 um 19:00 schrieb Christian Ebner:
>> Implements the garbage collection for datastore's backed by an s3
>> object store.
>> Take advantage of the local datastore by placing marker files in the
>> chunk store during phase 1 of the garbage collection, updating their
>> atime if already present. By this expensive api calls can be avoided
>> to update the object metadata (only possible via a copy object
>> operation).
> 
> The last sentence would be IMO slightly easier to understand:
> 
> This allows us to avoid making expensive API calls to update object
> metadata, which would only be possible via a copy object operation.

agreed, will incorporate this

>>
>> The phase 2 is implemented by fetching a list of all the chunks via
>> the ListObjectsV2 api call, filtered by the chunk folder prefix.
>> This operation has to be performed in patches of 1000 objects, given
> 
> s/patches/batches/

will fix

> 
>> by the api's response limits.
>> For each object key, lookup the marker file and decide based on the
>> marker existence and it's atime if the chunk object needs to be
>> removed. Deletion happens via the delete objects operation, allowing
>> to delete multiple chunks by a single request.
>>
>> This allows to efficiently lookup chunks which are not in use
>> anymore while being performant and cost effective.
> 
> Do you got some rough numbers perchance? E.g., something like "a
> datastore with X indexes, Y acutal data and Z deduplication factor
> is garbage collect in T time on:" and then the time numbers for
> e.g. ceph RGW backed S3, AWS/Cloudlflare S3/R2 and file system,
> just to get some idea of the ballpark we're in, and can also help
> to have such numbers as baseline for potential future optimization
> experiments.

I did not actively compare the original version, which did the last 
modified timestamp updates for each object, which was of course orders 
of magnitude slower because of all the copy object API requests.

On it to generate the baseline figures for the different S3 providers as 
compared to a local datastore, and include this here as well.