[pbs-devel] [PATCH proxmox{, -backup} v4 00/12] parallelize chunk reads in verification

[Fabian Grünbichler]

Gave this a spin as well, the code looks good to me and adding more
threads does make things faster, although finding the best combination
is probably a bit tricky:

1:4 90s (baseline)
1:8 91s (verify threads were already not fully utilized in the baseline)
4:4 46s
2:6 58s

these were with a smaller set of snapshots and a warm cache.

with a full datastore that doesn't fit into RAM/cache:

1:4 408s
1:8 349s
4:4 159s
2:6 204s

adding more threads mostly does not help anymore in my case..

On November 13, 2025 10:31 am, Nicolas Frey wrote:
> This patch series aims to expand on Dominik's series [0] written for
> pbs 3, parallelizing chunk reads in `VerifyWorker` using a seperate
> thread pool from the verification.
> 
> The number of threads was previously hard-coded, but is now
> configurable via the API and GUI with new properties called
> `{read,verify}-threads`, similarly to tape backups.
> 
> The number of threads should also be configurable through tuning
> options or datastore config (as discussed by Chris & Thomas on list),
> which can be added in a follow up patch series.
> 
> In my local tests I measured the following speed difference:
> verified a single snapshot with ~32 GiB (4x the RAM size) with 4
> cores (just changing the read threads)
> 
> 1 thread:    ~440MiB/s
> 2 threads:   ~780MiB/s
> 4 threads:   ~1140MiB/s
> 
> [0] https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/#t
> 
> Changes since v3, thanks Chris!:
> * correct the ordering of patches
> * move thread schemas into jobs.rs, rename them and add more specific
>   descriptions
> * add api schema for {read,verify}-threads in backup manager
> * add doc for new parameters
> * rebase
> 
> Changes since v2:
> * split move to parallel handler into 2 seperate commits
> * add constructor to `IndexVerifyState` and clean up nits
> 
> Changes since v1, thanks to Chris:
> * define dedicated schema for {worker,read,verify}-threads
> * rebase proxmox-backup
> * introduce new state struct `IndexVerifyState` to reduce the amount
>   of Arc clones and overall better bundling
> * adjust update endpoint and UI to the new properties
> 
> proxmox:
> 
> Nicolas Frey (3):
>   pbs-api-types: add schema for {worker,read,verify}-threads
>   pbs-api-types: jobs: add {read,verify}-threads to
>     VerificationJobConfig
>   pbs-api-types: use worker-threads schema for TapeBackupJobSetup
> 
>  pbs-api-types/src/jobs.rs | 41 ++++++++++++++++++++++++++++++++++-----
>  1 file changed, 36 insertions(+), 5 deletions(-)
> 
> 
> proxmox-backup:
> 
> Nicolas Frey (9):
>   verify: correct typo in comment
>   verify: introduce new state struct
>   verify: refactor into associated functions to use new state struct
>   verify: move chunk loading into parallel handler
>   verify: determine the number of threads to use with
>     {read,verify}-threads
>   verify: add {read,verify}-threads to update endpoint
>   verify: add {read,verify}-threads to api schema in backup manager
>   ui: verify: add option to set number of threads for job
>   docs: verify: document {read,verify}-threads and update screenshot
> 
>  .../pbs-gui-datastore-verifyjob-add.png       | Bin 15455 -> 41433 bytes
>  docs/maintenance.rst                          |  11 +-
>  src/api2/admin/datastore.rs                   |  16 +-
>  src/api2/backup/environment.rs                |   4 +-
>  src/api2/config/verify.rs                     |  16 ++
>  src/backup/verify.rs                          | 171 +++++++++++-------
>  src/bin/proxmox-backup-manager.rs             |  11 +-
>  src/server/verify_job.rs                      |   7 +-
>  www/window/VerifyAll.js                       |  18 ++
>  www/window/VerifyJobEdit.js                   |  24 +++
>  10 files changed, 210 insertions(+), 68 deletions(-)
> 
> 
> Summary over all repositories:
>   11 files changed, 246 insertions(+), 73 deletions(-)
> 
> -- 
> Generated by git-murpp 0.8.1
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
>