[pbs-devel] [PATCH proxmox-backup v6 00/21] fix chunk upload/insert, rename corrupt chunks and GC race conditions for s3 backend

[Thomas Lamprecht]

On Fri, 14 Nov 2025 14:18:40 +0100, Christian Ebner wrote:
> These patches fix possible race conditions on datastores with s3 backend for
> chunk insert, renaming of corrupt chunks during verification and cleanup during
> garbage collection. Further, the patches assure consistency between the chunk
> marker file of the local datastore cache, the s3 object store and the in-memory
> LRU cache during state changes occurring by one of the above mentioned operations.
> 
> Consistency is achieved by using a per-chunk file locking mechanism. File locks
> are stored on the predefined location for datastore file locks, using the same
> `.chunks/prefix/digest` folder layout for consistency and to keep readdir and
> other fs operations performant.
> 
> [...]

Applied, thanks!

[01/21] datastore: GC: drop overly verbose info message during s3 chunk sweep
        commit: d326fc5c055fd2783cff5817aa9c3d1df3bc4092
[02/21] chunk store: implement per-chunk file locking helper for s3 backend
        commit: adcc39540a3ba070d4627f0785dd113427d8c92c
[03/21] datastore: acquire chunk store mutex lock when renaming corrupt chunk
        commit: e35597690a064be6efe78e5df5c196bc3c9ecd7b
[04/21] datastore: get per-chunk file lock for chunk rename on s3 backend
        commit: 5eabe8b9222a108daf335a6eba7fbd64d9329d6b
[05/21] fix #6961: datastore: verify: evict corrupt chunks from in-memory LRU cache
        commit: 34a395e4f95976f8d96253c6cdb0ccc11ba21604
[06/21] datastore: add locking to protect against races on chunk insert for s3
        commit: 59414b6c9c6e5958d0dc5c0356e754095637f8a3
[07/21] GC: fix race with chunk upload/insert on s3 backends
        commit: 86d5d073037132e6e9aeb01f8e6aaf311dadcd10
[08/21] chunk store: reduce exposure of clear_chunk() to crate only
        commit: 9bd3f5a65110e6bafe5921cb4a7fe24da726cc98
[09/21] chunk store: make chunk removal a helper method of the chunk store
        commit: 9312c9de136f3df8001b5b397a51ed69fa3ee818
[10/21] store: split insert_chunk into wrapper + unsafe locked implementation
        commit: 25db75fb4b23356b28717d2bc88b6826486c851a
[11/21] store: cache: move Mutex acquire to cache insertion
        commit: e4905d1cadfb17d85cd350d9815e4787c888eea8
[12/21] chunk store: rename cache-specific helpers
        commit: 76ab83194af389d954241eb4d10126e32236b714
[13/21] GC: cleanup chunk markers from cache in phase 3 on s3 backends
        commit: 77438efecb8ff3d5e51b04650597ba738d3419b6
[14/21] GC: touch bad chunk files independent of backend type
        commit: f46fd187003eb2da355ff8da19aa71cb8d22878f
[15/21] GC: guard missing marker file insertion for s3 backed stores
        commit: 014f22d51df63d3721fb0a96861bd08978bc8064
[16/21] GC: s3: track if a chunk marker file is missing since a bad chunk
        commit: 7548f8f84ff60e8dcb2787f296db6bc34554f8d7
[17/21] chunk store: add helpers marking missing local chunk markers as expected
        commit: abfad2f38cd3bf94c2df5f6b97f8b192417641c0
[18/21] GC: assure chunk exists on s3 store when creating missing chunk marker
        commit: 9510ef1af395b0e744a938155ff796be9ce67ff6
[19/21] datastore: document s3 backend specific locking restrictions
        commit: c1b003afd932cca0d35e319d413382268e112470
[20/21] GC: fix: don't drop bad extension for S3 object to chunk path helper
        commit: 27e3f5b7bb50cc8b419c9b51b560344469629da5
[21/21] GC: clean up bad chunks from the filesystem only
        commit: 36f62da10d1cb40a19e32b8a0d3d4bcb8ac322bf