[pbs-devel] [PATCH proxmox-backup 0/6] s3 store verify: fix concurrency issues and add missing in-memory cache eviction

[Christian Ebner]

This patch series addresses issues with verification tasks on s3 backends.
Most importantly, it is assured that corrupt chunks are correctly evicted also
from the in-memory LRU cache, as otherwise new, non-corrupt chunk will not be
re-inserted into the chunk store if the digest is still cached as recently used.
Also, it refines the distinction between fetching error and chunk parsing error
handling, as the latter must flag the chunk as bad, the former not.

Further, avoid issues by concurrent chunk inserts and or garbage collection
while renaming corrupt chunks. The chunk store mutex lock was never acquired,
independent of the datastore backend, although being less problematic on
filesystem backends due to the atomic nature of the chunk file rename.

Finally, make sure that the mutex lock guarding the corrupt chunk list is not
held when entering the rename helper, as that performs async api calls to the
object store for datastores backed by S3, which could potentially lead to
deadlocks.

proxmox-backup:

Christian Ebner (6):
  verify/datastore: make rename corrupt chunk a datastore helper method
  datastore: refactor rename_corrupted_chunk error handling
  verify: never hold mutex lock in async scope on corrupt chunk rename
  datastore: acquire chunk store mutex lock when renaming corrupt chunk
  datastore: verify: evict corrupt chunks from in-memory LRU cache
  verify: distinguish s3 object fetching and chunk loading error

 pbs-datastore/src/datastore.rs |  63 ++++++++++++++++++
 src/backup/verify.rs           | 117 ++++++++-------------------------
 2 files changed, 92 insertions(+), 88 deletions(-)


Summary over all repositories:
  2 files changed, 92 insertions(+), 88 deletions(-)

-- 
Generated by git-murpp 0.8.1