[pbs-devel] [RFC proxmox-backup 0/4] concurrent group pull support for sync jobs

[Christian Ebner]

Pulling contents from a remote source via a sync job suffers from low
throughput on high latency networks because of limitations by the
HTTP/2 connection, as described in [0]. As a workaround, pulling
multiple groups in parallel by establishing multiple reader instances
has been suggested.

This patch series therefore adds a configuration property
`group-sync-tasks` to sync jobs which allows to define the number of
concurrent group pull tasks for each job. This is currently not
exposed on the UI. A valid config would look like this:

```
sync: s-d2755441-cca9
	ns
	owner root at pam
	group-sync-tasks 4
	remote pbs-remote-source
	remote-ns
	remote-store store
	remove-vanished false
	schedule daily
	store pullstore
```

This brings improvements as roughly tested by artificially increasing
the latency on the bridge of the pull target host to 150ms via

`tc qdisc add dev vmbr0 root netem delay 150ms`

and verifying by pinging the remote source host that the latency
applied.

Pulling using 2 concurrent tasks reduced the task runtime by about
-25% as compared to only a single task, 4 configured tasks reduced the
runtime by about -30%.

The current approach however interferes with status logging of a sync
job, as now no sequence is guaranteed anymore. Therefore, the logs are
buffered instead and only shown after the corresponding group pull
tasks has been run to completion.

Sending this as RFC as I am not to happy with how logging is handled,
maybe somebody has a better idea.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=4182

Christian Ebner (4):
  api: config/sync: add optional group-sync-tasks property
  server: pull: factor out group pull task into helper
  fix #4182: server: sync: allow pulling groups concurrently
  server: pull: conditionally buffer parallel tasks log output

 pbs-api-types/src/jobs.rs           |  14 ++
 pbs-datastore/src/store_progress.rs |   2 +-
 src/api2/config/sync.rs             |  10 +
 src/api2/pull.rs                    |  13 +-
 src/server/pull.rs                  | 311 +++++++++++++++++++++-------
 5 files changed, 274 insertions(+), 76 deletions(-)

-- 
2.39.2