[pbs-devel] [PATCH proxmox-backup v2 0/3] tape/verify: improve multithreaded

[Dominik Csapak]

this is a replacement series for my previous patches[0]

in this version, i reversed the order of patches, so that the tape
improvement comes first. I did this, because for tapes I could measure a
very real and beneficial improvement, while for verification the
situation is more complicated.

There we have currently only the hashing of chunks parallelized, not the
reading of data, but this could also improve throughput (as seen in the
benchmarks of 3/3).

patch 2 and 3 are RFCs because I'm not sure how to proceed with these.
One one hand they can improve the situation (when both are applied) but
makes it worse for other situations.

We could go even further and introduce another
'verification-read-threads' option and have a seperate thread pool for
reading the data and another for verifying them, but I'm not sure if it
would be worth it to spend even more time on it (incl testing, which
takes a lot of time)

changes from v1:
* reordered code from verify,tape to tape,verify
* marked verify patches as rfc
* use a struct for saving the values in the datastore (as thomas
  suggested)
* introduces another verify patch to merge the chunk loading into the
  worker threads

0: https://lists.proxmox.com/pipermail/pbs-devel/2024-April/009162.html

Dominik Csapak (3):
  tape: introduce a tape backup read thread tuning option
  verify: add tuning option for number of threads to use
  verify: move chunk loading into the worker threads

 pbs-api-types/src/datastore.rs              | 15 +++++
 pbs-datastore/src/datastore.rs              | 31 ++++++++++
 src/backup/verify.rs                        | 67 +++++++++------------
 src/tape/pool_writer/new_chunks_iterator.rs | 45 +++++++++-----
 www/Utils.js                                | 10 +++
 www/datastore/OptionView.js                 | 16 +++++
 6 files changed, 130 insertions(+), 54 deletions(-)

-- 
2.39.2