[pbs-devel] superseded: [PATCH proxmox{, -backup} v3 0/9] parallelize chunk reads in verification

[Nicolas Frey]

Superseded-by: https://lore.proxmox.com/pbs-devel/20251113093118.195229-1-n.frey@proxmox.com/T/#t

On 11/10/25 9:44 AM, Nicolas Frey wrote:
> This patch series aims to expand on Dominik's series [0] written for
> pbs 3, parallelizing chunk reads in `VerifyWorker` using a seperate
> thread pool from the verification.
> 
> The number of threads was previously hard-coded, but is now
> configurable via the API and GUI with new properties called
> `{read,verify}-threads`, similarly to tape backups.
> 
> The number of threads should also be configurable through tuning
> options or datastore config (as discussed by Chris & Thomas on list),
> which can be added in a follow up patch series.
> 
> In my local tests I measured the following speed difference:
> verified a single snapshot with ~32 GiB (4x the RAM size) with 4
> cores (just changing the read threads)
> 
> 1 thread:    ~440MiB/s
> 2 threads:   ~780MiB/s
> 4 threads:   ~1140MiB/s
> 
> [0] https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/#t
> 
> Changes since v2:
> * split move to parallel handler into 2 seperate commits
> * add constructor to `IndexVerifyState` and clean up nits
> 
> Changes since v1, thanks to Chris:
> * define dedicated schema for {worker,read,verify}-threads
> * rebase proxmox-backup
> * introduce new state struct `IndexVerifyState` to reduce the amount
>   of Arc clones and overall better bundling
> * adjust update endpoint and UI to the new properties
> 
> proxmox:
> 
> Nicolas Frey (3):
>   pbs-api-types: add schema for {worker,read,verify}-threads
>   pbs-api-types: jobs: add {read,verify}-threads to
>     VerificationJobConfig
>   pbs-api-types: use worker-threads schema for TapeBackupJobSetup
> 
>  pbs-api-types/src/datastore.rs | 14 ++++++++++++++
>  pbs-api-types/src/jobs.rs      | 20 +++++++++++++++-----
>  2 files changed, 29 insertions(+), 5 deletions(-)
> 
> 
> proxmox-backup:
> 
> Nicolas Frey (6):
>   api: verify: move chunk loading into parallel handler
>   api: verify: bundle parameters into new struct
>   api: verify: determine the number of threads to use with
>     {read,verify}-threads
>   api: verify: correct typo in comment
>   api: verify: add {read,verify}-threads to update endpoint
>   ui: verify: add option to set number of threads for job
> 
>  src/api2/admin/datastore.rs    |  18 +++-
>  src/api2/backup/environment.rs |   2 +-
>  src/api2/config/verify.rs      |  16 ++++
>  src/backup/verify.rs           | 164 +++++++++++++++++++++------------
>  src/server/verify_job.rs       |   7 +-
>  www/window/VerifyAll.js        |  18 ++++
>  www/window/VerifyJobEdit.js    |  24 +++++
>  7 files changed, 185 insertions(+), 64 deletions(-)
> 
> 
> Summary over all repositories:
>   9 files changed, 214 insertions(+), 69 deletions(-)
>