[pbs-devel] parallelize restore.rs fn restore_image: problems in

[Niko Fellner]

I did some benchmarks now at an Azure L32s_v2 (32 vCPUs, 256 GB memory, 4x 2 TB NVMe; on tests with ZFS I reduced the zfs-arc-max to 8 GiB, because I didn't want to test the memory only)

Overall the speedup of the parallel pbs-restore I encountered was at least 2.1x here, often about 2.6x, but in some tests even up to 6.8x; On big VMs a speedup of 3.6x.

The CPU usage of the most busy pbs-restore process during the 750 GiB restore was often about 300-450% (not great for 32 threads, but we only do parallel reads, so...); the original sync pbs-restore only showed about 98% CPU usage in htop.

First, some tests with my 32 GiB VM (zeroes = 64%):

> Results (relative to 32 GiB):
> Host;          Method;              source;                 target;                 Start;    End;      seconds(End - Start);  speed (End-Start);   speedup;  seconds(PVE log); speed(PVE log); speedup
> Azure L32s_v2; orig. pbs-restore;   nvme disk 1 ZFS pool;   nvme disk 2 ZFS dir;    21:10:38; 21:13:14;  156;                    210.05 MiB/s;      1.0;        58.69;            558.31 MiB/s;  1.0
> Azure L32s_v2; paral.pbs-restore;   nvme disk 1 ZFS pool;   nvme disk 2 ZFS dir;    21:34:25; 21:34:49;   24;                   1365.33 MiB/s;      6.5;        22.85;           1433.99 MiB/s;  2.6

> Azure L32s_v2; orig. pbs-restore;   nvme disk 1 ZFS pool;   nvme disk 2 ZFS dir;    21:19:51; 21:22:27;  156;                    210.05 MiB/s;      1.0;        54.46;            601.66 MiB/s;  1.0
> Azure L32s_v2; paral.pbs-restore;   nvme disk 1 ZFS pool;   nvme disk 2 ZFS dir;    21:41:53; 21:42:16;   23;                   1424.70 MiB/s;      6.8;        22.63;           1447.70 MiB/s;  2.4

> Azure L32s_v2; orig. pbs-restore;   nvme disk 1 ext4 dir;   nvme disk 2 ext4 dir;   02:10:04; 02:10:49;   45;                    728.18 MiB/s;      1.0;        43.97;            745.16 MiB/s;  1.0
> Azure L32s_v2; paral.pbs-restore;   nvme disk 1 ext4 dir;   nvme disk 2 ext4 dir;   02:12:35; 02:12:56;   21;                   1560.38 MiB/s;      2.1;        21.06;           1556.12 MiB/s;  2.1

> Azure L32s_v2; orig. pbs-restore;   2x nvme RAID0 ext4 dir; 2x nvme RAID0 ext4 dir; 01:56:35; 01:57:24;   49;                     668.73 MiB/s;     1.0;        48.94;            669.52 MiB/s;  1.0
> Azure L32s_v2; paral.pbs-restore;   2x nvme RAID0 ext4 dir; 2x nvme RAID0 ext4 dir; 22:24:45; 22:25:04;   19;                    1724.63 MiB/s;     2.6;        18.67;           1754.79 MiB/s;  2.6

Now some tests with my 750 GiB VM (zeroes = 5%):

> Results (relative to 750 GiB):
> Host;          Method;              source;                 target;                 Start;    End;      seconds(End - Start);  speed (End-Start);   speedup;  seconds(PVE log); speed(PVE log); speedup

> Azure L32s_v2; orig. pbs-restore;   2x nvme RAID0 ext4 dir; 2x nvme RAID0 ext4 dir; 00:45:19; 01:47:38; 3739;                     205,40 MiB/s;     1.0;      3736.73;          205.53 MiB/s;   1.0
> Azure L32s_v2; paral.pbs-restore;   2x nvme RAID0 ext4 dir; 2x nvme RAID0 ext4 dir; 00:23:44; 00:41:00; 1036;                     741,31 MiB/s;     3.6;      1034.96;          742.06 MiB/s;   3.6


PVE Logs:

> orig. pbs-restore: restore image complete (bytes=34359738368, duration=58.69s, speed=558.31MB/s)
> paral.pbs-restore: restore image complete (bytes=34359738368, duration=22.85s, speed=1433.99MB/s)

> orig. pbs-restore: restore image complete (bytes=34359738368, duration=54.46s, speed=601.66MB/s)
> paral.pbs-restore: restore image complete (bytes=34359738368, duration=22.63s, speed=1447.70MB/s)

> orig. pbs-restore: restore image complete (bytes=34359738368, duration=43.97s, speed=745.16MB/s)
> paral.pbs-restore: restore image complete (bytes=34359738368, duration=21.06s, speed=1556.12MB/s)

> orig. pbs-restore: restore image complete (bytes=34359738368, duration=48.94s, speed=669.52MB/s)
> paral.pbs-restore: restore image complete (bytes=34359738368, duration=18.67s, speed=1754.79MB/s)

> orig. pbs-restore: restore image complete (bytes=805306368000, duration=3736.73s, speed=205.53MB/s)
> paral.pbs-restore: restore image complete (bytes=805306368000, duration=1034.96s, speed=742.06MB/s)

Maybe anyone is interested in "proxmox-backup-client benchmark --repository chunks2tbnvme":

Host | Datastore<br>(tls bench only) | aes256_gcm<br>MB/s | compress<br>MB/s | decompress<br>MB/s | sha256<br>MB/s | tls<br>MB/s | verify<br>MB/s
-- | -- | --: | --: | --: | --: | --: | --:
Azure L32s_v2 | chunks2tbnvme | 2552.57 | 549.95 | 856.72 | 1479.59 | 588.78 | 545.02