[pbs-devel] [PATCH proxmox-backup 0/7] s3 store: fix issues with chunk s3 backend upload and cache eviction

[Christian Ebner]

These patches fix 2 issues with the current s3 backend implementation
and reduce code duplication.

Patch 1 to 3 rework the garbage collection, deduplicating the common atime
check and garbage collection status update logic and refactor the chunk removal
from local datastore cache on s3 backends.

Patch 4 fixes an issue which could lead to backup restores failing
during concurrent backups, if the local datastore cache was small, leading
to chunks being evicted from the cache by truncating the contents, while the
chunk reader still accessed the chunk. This is now circumvented by replacing
the chunk file instead, leaving the contents on the already opened file
handle for the reader still accessible.

The remaining patches fix a possible race condition between s3 backend upload
and garbage collection, which can result in chunk loss. If the chunk upload
finished, garbage collection listed and checked the chunk's in-use marker, just
before it being written by the cache insert after the upload, garbage collection
can incorrectly delete the chunk. This is circumvented by setting and checking
an additional chunk marker file, which is created before starting the upload
and removed after cache insert, assuring that these chunks are not removed.

proxmox-backup:

Christian Ebner (7):
  datastore: gc: inline single callsite method
  gc: chunk store: rework atime check and gc status into common helper
  chunk store: add and use method to remove chunks
  chunk store: fix: replace evicted cache chunks instead of truncate
  api: chunk upload: fix race between chunk backend upload and insert
  api: chunk upload: fix race with garbage collection for no-cache on s3
  pull: guard chunk upload and only insert into cache after upload

 pbs-datastore/src/chunk_store.rs              | 211 +++++++++++++++---
 pbs-datastore/src/datastore.rs                | 155 +++++++------
 .../src/local_datastore_lru_cache.rs          |  32 +--
 src/api2/backup/upload_chunk.rs               |  29 ++-
 src/api2/config/datastore.rs                  |   2 +
 src/server/pull.rs                            |  14 +-
 6 files changed, 299 insertions(+), 144 deletions(-)


Summary over all repositories:
  6 files changed, 299 insertions(+), 144 deletions(-)

-- 
Generated by git-murpp 0.8.1