[pbs-devel] [PATCH proxmox-backup 00/17] fix chunk upload/insert, rename corrupt chunks and GC race conditions for s3 backend

[Christian Ebner]

These patches fix possible race conditions on datastores with s3 backend for
chunk insert, renaming of corrupt chunks during verification and cleanup during
garbage collection. Further, the patches assure consistency between the chunk
marker file of the local datastore cache, the s3 object store and the in-memory
LRU cache during state changes occurring by one of the above mentioned operations.

Consistency is achieved by using a per-chunk file locking mechanism. File locks
are stored on the predefined location for datastore file locks, using the same
.chunks/prefix/digest folder layout for consistency and to keep readdir and other
fs operations performant.

Before introducing the file locking mechanism, the patches refactor pre-existing
code to move most of the backend related logic away from the api code to the
datastore implementation, in order to have a common interface especially for
chunk insert.

As part of the series it is now also assured that chunks which are removed from
the local datastore cache, are also dropped from it's in-memory LRU cache and'
therefore a consistent state is achieved.

This patch series obsoletes two previous patch series with unfortunately
incomplete bugfix attempts found at:
- https://lore.proxmox.com/pbs-devel/8d711a20-b193-47a9-8f38-6ce800e6d0e8@proxmox.com/T/
- https://lore.proxmox.com/pbs-devel/20251015164008.975591-1-c.ebner@proxmox.com/T/

proxmox-backup:

Christian Ebner (17):
  sync: pull: instantiate backend only once per sync job
  api/datastore: move group notes setting to the datastore
  api/datastore: move snapshot deletion into dedicated datastore helper
  api/datastore: move backup log upload by implementing datastore helper
  api/datastore: add dedicated datastore helper to set snapshot notes
  datastore: refactor chunk insert based on backend
  verify: rename corrupted to corrupt in log output and function names
  verify/datastore: make rename corrupt chunk a datastore helper method
  datastore: refactor rename_corrupt_chunk error handling
  datastore: implement per-chunk file locking helper for s3 backend
  datastore: acquire chunk store mutex lock when renaming corrupt chunk
  datastore: get per-chunk file lock for chunk rename on s3 backend
  fix #6961: datastore: verify: evict corrupt chunks from in-memory LRU
    cache
  datastore: add locking to protect against races on chunk insert for s3
  GC: fix race with chunk upload/insert on s3 backends
  GC: lock chunk marker before cleanup in phase 3 on s3 backends
  datastore: GC: drop overly verbose info message during s3 chunk sweep

 pbs-datastore/src/backup_info.rs |   2 +-
 pbs-datastore/src/chunk_store.rs |  54 ++++++-
 pbs-datastore/src/datastore.rs   | 262 +++++++++++++++++++++++++++++--
 src/api2/admin/datastore.rs      |  74 +++------
 src/api2/backup/upload_chunk.rs  |  64 ++------
 src/api2/tape/restore.rs         |   6 +-
 src/backup/verify.rs             |  83 ++--------
 src/server/pull.rs               |  47 +++---
 8 files changed, 369 insertions(+), 223 deletions(-)


Summary over all repositories:
  8 files changed, 369 insertions(+), 223 deletions(-)

-- 
Generated by git-murpp 0.8.1