[pbs-devel] [PATCH proxmox-backup v6 00/21] fix chunk upload/insert, rename corrupt chunks and GC race conditions for s3 backend

[Fabian Grünbichler]

Quoting Christian Ebner (2025-11-14 14:18:40)
> These patches fix possible race conditions on datastores with s3 backend for
> chunk insert, renaming of corrupt chunks during verification and cleanup during
> garbage collection. Further, the patches assure consistency between the chunk
> marker file of the local datastore cache, the s3 object store and the in-memory
> LRU cache during state changes occurring by one of the above mentioned operations.
> 
> Consistency is achieved by using a per-chunk file locking mechanism. File locks
> are stored on the predefined location for datastore file locks, using the same
> `.chunks/prefix/digest` folder layout for consistency and to keep readdir and
> other fs operations performant.
> 
> As part of the series it is now also assured that chunks which are removed from
> the local datastore cache, are also dropped from it's in-memory LRU cache and
> therefore a consistent state is achieved. Further, bad chunks are touched as
> well during GC phase 1 for s3 backed datastores and the creation of missing
> marker files performed conditionally, to avoid consistency issues.

Consider this series:

Reviewed-by: Fabian Grünbichler <f.gruenbichler at proxmox.com>

an additional set of eyes would still be good, given the scope of changes!

I plan to follow up with some significant refactoring after the PBS release is
out the door, as there are two areas where the code is becoming a bit unwieldy:

- make the chunk store mutex a proper lock guard and implement the operations
  that require the lock to be held on top
- make more of the backend handling implicit