[pbs-devel] [PATCH proxmox-backup v3 0/3] fix #6750: fix possible deadlock for s3 backed datastore backups

[Christian Ebner]

These patches aim to fix a deadlock which can occur during backup
jobs to datastores backed by S3 backend. The deadlock most likely is
caused by the mutex guard for the backup shared state being held
while entering the tokio::task::block_in_place context and executing
async code, which however can lead to deadlocks as described in [0].

Therefore, these patches avoid holding the mutex guard for the shared
backup state while performing the s3 backend operations, by
prematurely dropping it. To avoid inconsistencies, introduce flags
to keep track of the index writers closing state and add a transient
`Finishing` state to be entered during manifest updates.

Changes since version 2 (thanks @Fabian):
- Avoid unneeded mutex guard during backup removal

Changes since version 1 (thanks @Fabian):
- Use the shared backup state's writers in addition with a closed flag
  instead of counting active backend operations.
- Replace finished flag with BackupState enum to introduce the new,
  transient `Finishing` state to be entered during manifest updates.
- Add missing checks and refactor code to the now mutable reference when
  accessing the shared backup state in the respective close calls.


[0] https://docs.rs/tokio/latest/tokio/sync/struct.Mutex.html#which-kind-of-mutex-should-you-use

Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=6750

Another report in the community forum:
https://forum.proxmox.com/threads/171422/

proxmox-backup:

Christian Ebner (3):
  fix #6750: api: avoid possible deadlock on datastores with s3 backend
  api: backup: never hold mutex guard when doing manifest update
  api: backup: avoid holding mutex and inline backup cleanup method

 src/api2/backup/environment.rs | 181 ++++++++++++++++++++++-----------
 src/api2/backup/mod.rs         |  24 ++++-
 2 files changed, 140 insertions(+), 65 deletions(-)


Summary over all repositories:
  2 files changed, 140 insertions(+), 65 deletions(-)

-- 
Generated by git-murpp 0.8.1