[pbs-devel] [PATCH proxmox{, -backup} v4 00/12] parallelize chunk reads in verification

[Christian Ebner]

Gave this another spin, looks good to me apart from the commit tag ordering.

Some additional ballpark benchmark figures from my testing:

Used for tests:
echo 3 > /proc/sys/vm/drop_caches && time proxmox-backup-manager verify 
<datastore> --read-threads 1 --verify-threads 4 --ignore-verified false

S3 store backed by Ceph RGW (6365 chunks with total 10.907 GiB, 42 index 
files)
-------------------------------
#threads | #threads   | runtime
-------------------------------
read: 1  | verify: 4  | 168s
read: 1  | verify: 8  | 166s
read: 2  | verify: 4  |  89s
read: 4  | verify: 4  |  50s
read: 8  | verify: 4  |  32s
read: 16 | verify: 4  |  26s
read: 32 | verify: 4  |  25s
read: 8  | verify: 8  |  30s
read: 16 | verify: 8  |  23s
read: 32 | verify: 8  |  23s
read: 8  | verify: 16 |  30s
read: 16 | verify: 16 |  23s
read: 16 | verify: 32 |  23s
read: 32 | verify: 32 |  23s

so above 8 verify threads CPU bound, below 16 read threads IO bound.

Regular datastore on NVME SSD (6012 chunks with total 8.059 GiB, 37 
index files)
-------------------------------
#threads | #threads   | runtime
-------------------------------
read: 1  | verify: 4  | 11s
read: 1  | verify: 8  |  9s
read: 2  | verify: 4  |  9s
read: 4  | verify: 4  |  8s
read: 8  | verify: 4  |  8s
read: 16 | verify: 4  |  8s
read: 32 | verify: 4  |  9s
read: 8  | verify: 8  |  6s
read: 16 | verify: 8  |  6s
read: 32 | verify: 8  |  6s
read: 8  | verify: 16 |  5s
read: 16 | verify: 16 |  6s
read: 16 | verify: 32 |  6s
read: 32 | verify: 32 |  6s

so mostly CPU bound.

With the comments on the doc patch addressed, consider:

Reviewed-by: Christian Ebner <c.ebner at proxmox.com>
Tested-by: Christian Ebner <c.ebner at proxmox.com>