[pbs-devel] [PATCH proxmox{, -backup} v3 0/9] parallelize chunk reads in verification

[Nicolas Frey]

On 11/11/25 11:21 AM, Christian Ebner wrote:
> On 11/10/25 9:44 AM, Nicolas Frey wrote:
>> This patch series aims to expand on Dominik's series [0] written for
>> pbs 3, parallelizing chunk reads in `VerifyWorker` using a seperate
>> thread pool from the verification.
>>
>> The number of threads was previously hard-coded, but is now
>> configurable via the API and GUI with new properties called
>> `{read,verify}-threads`, similarly to tape backups.
>>
>> The number of threads should also be configurable through tuning
>> options or datastore config (as discussed by Chris & Thomas on list),
>> which can be added in a follow up patch series.
>>
>> In my local tests I measured the following speed difference:
>> verified a single snapshot with ~32 GiB (4x the RAM size) with 4
>> cores (just changing the read threads)
>>
>> 1 thread:    ~440MiB/s
>> 2 threads:   ~780MiB/s
>> 4 threads:   ~1140MiB/s
>>
>> [0] https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-
>> d.csapak at proxmox.com/#t
>>
>> Changes since v2:
>> * split move to parallel handler into 2 seperate commits
>> * add constructor to `IndexVerifyState` and clean up nits
>>
>> Changes since v1, thanks to Chris:
>> * define dedicated schema for {worker,read,verify}-threads
>> * rebase proxmox-backup
>> * introduce new state struct `IndexVerifyState` to reduce the amount
>>    of Arc clones and overall better bundling
>> * adjust update endpoint and UI to the new properties
> Series is looking good already, only a few smaller thing need some
> adaption (see individual patches).
> 
> One major thing which should be however included is a short
> documentation of these parameters and what they do.
> 
> Tested the configs are set as expected via cli/ui, default values are
> cleared, the logs expose the number of used reader/verify threads,
> parameters are actually used for the verification job.
> 
> Consider the whole series:
> 
> Tested-by: Christian Ebner <c.ebner at proxmox.com>

Thanks for the Review and Test!

I unfortunately misunderstood your comments on v2 about splitting it
into smaller commits and mixed up the order because of that.

I'll send a v4 with all your suggestions!