[pbs-devel] [PATCH proxmox-backup v5 00/19] fix chunk upload/insert, rename corrupt chunks and GC race conditions for s3 backend

[Christian Ebner]

These patches fix possible race conditions on datastores with s3 backend for
chunk insert, renaming of corrupt chunks during verification and cleanup during
garbage collection. Further, the patches assure consistency between the chunk
marker file of the local datastore cache, the s3 object store and the in-memory
LRU cache during state changes occurring by one of the above mentioned operations.

Consistency is achieved by using a per-chunk file locking mechanism. File locks
are stored on the predefined location for datastore file locks, using the same
`.chunks/prefix/digest` folder layout for consistency and to keep readdir and
other fs operations performant.

As part of the series it is now also assured that chunks which are removed from
the local datastore cache, are also dropped from it's in-memory LRU cache and
therefore a consistent state is achieved. Further, bad chunks are touched as
well during GC phase 1 for s3 backed datastores and the creation of missing
marker files performed conditionally, to avoid consistency issues.

Changes since version 4:
- Incorporated patches by Fabian for better handling of the chunk store mutex locking
- Add patches to fix missing marker file creation and keeping of bad chunks during
  garbage collection for s3 backend
- Document locking order restrictions

Changes since version 3:
- Add patches to limit visibility of BackupDir and BackupGroup destroy
- Refactored s3 upload index helper
- Avoid unneeded double stat for GC phase 3 clenaups

Changes since version 2:
- Incorporate additional race fix as discussed in
  https://lore.proxmox.com/pbs-devel/8ab74557-9592-43e7-8706-10fceaae31b7@proxmox.com/T/
  and suggested offlist.

Changes since version 1 (thanks @Fabian for review):
- Fix lock inversion for rename corrup chunk.
- Inline the chunk lock helper, making it explicit and thereby avoid calling the
  helper for regular datastores.
- Pass the backend to the add_blob datastore helper, so it can be reused for the
  backup session and pull sync job.
- Move also the s3 index upload helper from the backup env to the datastore, and
  reuse it for the sync job as well.

This patch series obsoletes two previous patch series with unfortunately
incomplete bugfix attempts found at:
- https://lore.proxmox.com/pbs-devel/8d711a20-b193-47a9-8f38-6ce800e6d0e8@proxmox.com/T/
- https://lore.proxmox.com/pbs-devel/20251015164008.975591-1-c.ebner@proxmox.com/T/

proxmox-backup:

Christian Ebner (16):
  datastore: GC: drop overly verbose info message during s3 chunk sweep
  chunk store: implement per-chunk file locking helper for s3 backend
  datastore: acquire chunk store mutex lock when renaming corrupt chunk
  datastore: get per-chunk file lock for chunk rename on s3 backend
  fix #6961: datastore: verify: evict corrupt chunks from in-memory LRU
    cache
  datastore: add locking to protect against races on chunk insert for s3
  GC: fix race with chunk upload/insert on s3 backends
  chunk store: reduce exposure of clear_chunk() to crate only
  chunk store: make chunk removal a helper method of the chunk store
  GC: cleanup chunk markers from cache in phase 3 on s3 backends
  GC: touch bad chunk files independent of backend type
  GC: guard missing marker file insertion for s3 backed stores
  GC: s3: track if a chunk marker file is missing since a bad chunk
  chunk store: add helpers marking missing local chunk markers as
    expected
  GC: assure chunk exists on s3 store when creating missing chunk marker
  datastore: document s3 backend specific locking restrictions

Fabian Grünbichler (3):
  store: split insert_chunk into wrapper + unsafe locked implementation
  store: cache: move Mutex acquire to cache insertion
  chunk store: rename cache-specific helpers

 pbs-datastore/src/backup_info.rs              |   2 +-
 pbs-datastore/src/chunk_store.rs              | 124 ++++++++++++++--
 pbs-datastore/src/datastore.rs                | 140 ++++++++++++------
 pbs-datastore/src/lib.rs                      |  13 ++
 .../src/local_datastore_lru_cache.rs          |  23 ++-
 5 files changed, 236 insertions(+), 66 deletions(-)


Summary over all repositories:
  5 files changed, 236 insertions(+), 66 deletions(-)

-- 
Generated by git-murpp 0.8.1