[pbs-devel] superseded: [PATCH proxmox{, -backup} v2 0/7] parallelize chunk reads in verification

[Nicolas Frey]

Superseded-by: https://lore.proxmox.com/pbs-devel/20251110084417.173290-1-n.frey@proxmox.com/T/#t

On 11/6/25 5:13 PM, Nicolas Frey wrote:
> This patch series aims to expand on Dominik's series [0] written for
> pbs 3, parallelizing chunk reads in `VerifyWorker` using a seperate
> thread pool from the verification.
> 
> The number of threads was previously hard-coded, but is now
> configurable via the API and GUI with new properties called
> `{read,verify}-threads`, similarly to tape backups.
> 
> In my local tests I measured the following speed difference:
> verified a single snapshot with ~32 GiB (4x the RAM size) with 4
> cores (just changing the read threads)
> 
> 1 thread:    ~440MiB/s
> 2 threads:   ~780MiB/s
> 4 threads:   ~1140MiB/s
> 
> [0] https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/#t
> 
> Changes since v1, thanks to Chris:
> * define dedicated schema for {worker,read,verify}-threads
> * rebase proxmox-backup
> * introduce new state struct `IndexVerifyState` to reduce the amount
>   of Arc clones and overall better bundling
> * adjust update endpoint and UI to the new properties
> 
> proxmox:
> 
> Nicolas Frey (3):
>   pbs-api-types: add schema for {worker,read,verify}-threads
>   pbs-api-types: jobs: add {read,verify}-threads to
>     VerificationJobConfig
>   pbs-api-types: use worker-threads schema for TapeBackupJobSetup
> 
>  pbs-api-types/src/datastore.rs | 14 ++++++++++++++
>  pbs-api-types/src/jobs.rs      | 20 +++++++++++++++-----
>  2 files changed, 29 insertions(+), 5 deletions(-)
> 
> 
> proxmox-backup:
> 
> Nicolas Frey (4):
>   api: verify: move chunk loading into parallel handler
>   api: verify: determine the number of threads to use with
>     {read,verify}-threads
>   api: verify: add {read,verify}-threads to update endpoint
>   ui: verify: add option to set number of threads for job
> 
>  src/api2/admin/datastore.rs    |  18 +++-
>  src/api2/backup/environment.rs |   2 +-
>  src/api2/config/verify.rs      |  16 ++++
>  src/backup/verify.rs           | 149 ++++++++++++++++++++-------------
>  src/server/verify_job.rs       |   7 +-
>  www/window/VerifyAll.js        |  18 ++++
>  www/window/VerifyJobEdit.js    |  24 ++++++
>  7 files changed, 172 insertions(+), 62 deletions(-)
> 
> 
> Summary over all repositories:
>   9 files changed, 201 insertions(+), 67 deletions(-)
>