[pbs-devel] [PATCH v2 proxmox-backup 00/31] fix #3044: push datastore to remote target

[Christian Ebner]

This patch series implements the functionality to extend the current
sync jobs in pull direction by an additional push direction, allowing
to push contents of a local source datastore to a remote target.

The series implements this by using the REST API of the remote target
for fetching, creating and/or deleting namespaces, groups and backups,
and reuses the clients backup writer functionality to create snapshots
by writing a manifeset on the remote target and sync the fixed index,
dynamic index or blobs contained in the source manifest to the remote,
preserving also encryption information.

Thanks a lot to Thomas and Gabriel for initial feedback on the
previous version, their suggestions have been addressed in this new
version of the patch series.

Most notable changes since version 1 of the patch series include:
- Do not share known chunk state over all backup writer instances.
  This fixes an issue present in the previous series, where a known
  chunk upload might have failed, because seemingly unknown to the
  server. This stems from the fact that the reupload as known chunks
  is only possible for chunks stored in the the sever side's
  `BackupEnvironment`s state, which only lives as long as each
  individual backup writer instance, and only covers chunks from the
  previous snapshot of the backup group (an the chunks seen by the
  current writer instance).
- Instead, download the index file as listed in the previous snapshot
  of the group already present on the server and deduplicate based on
  that. Although this is an orthogonal optimization, it should be the
  better approach as it mimics the current incremental backup.
- Clearly separate sync job configs in pull and push direction, by
  introducing a dedicated config type `sync-push` for sync jobs in
  push direction. This avoids potential misconfiguration.
- Clearly separate sync jobs in pull and push direction in the WebUI,
  by using a separate grid to list the jobs.
- Rename the `ignore-previous` flag to `no-timestamp-check`, which is
  more fitting and now actually only does skip the check, as the
  previous manifest is require to fetch the known chunk list now.

The patch series is structured as follows in this version:
- patch 1 is a cleanup patch fixing typos in api documentation.
- patches 2 to 7 are patches restructuring the current code so that
  functionality of the current pull implementation can be reused for
  the push implementation as well.
- patches 8 and 9 extend the backup writers functionality to be able
  to push snapshots to the target.
- patches 10 to 12 are once again preparatory patches for shared
  implementation of sync jobs in pull and push direction.
- patch 13 defines the required permission acls and roles.
- patch 14 implements almost all of the logic required for the push,
  including pushing of the datastore, namespace, groups and snapshots,
  taking into account also filters and additional sync flags.
- patch 15 extends the current sync job configuration by a new config
  type `sync-push` allowing to configure sync jobs in push direction
  while limiting possible misconfiguration errors.
- patches 16 to 26 expose the new sync job direction via the API, CLI
  and WebUI.
- patches 27 to 31 finally are followup patches, changing the return
  type for the backup group and namespace delete REST API endpoints
  to return statistics on the deleted snapshots, groups and namespaces,
  which are then used to include this information in the task log.
  As this is an API breaking change, the patches are kept independent
  from the other patches.

Link to issue on bugtracker:
https://bugzilla.proxmox.com/show_bug.cgi?id=3044


Christian Ebner (31):
  api: datastore: add missing whitespace in description
  server: sync: move sync related stats to common module
  server: sync: move reader trait to common sync module
  server: sync: move source to common sync module
  client: backup writer: bundle upload stats counters
  client: backup writer: factor out merged chunk stream upload
  client: backup writer: add chunk count and duration stats
  client: backup writer: allow push uploading index and chunks
  api: backup: add no-timestamp-check flag to backup endpoint
  server: sync: move skip info/reason to common sync module
  server: sync: make skip reason message more genenric
  server: sync: factor out namespace depth check into sync module
  api types: define remote permissions and roles for push sync
  fix #3044: server: implement push support for sync operations
  config: jobs: add `sync-push` config type for push sync jobs
  api: push: implement endpoint for sync in push direction
  api: sync: move sync job invocation to common module
  api: sync jobs: expose optional `sync-direction` parameter
  bin: manager: add datastore push cli command
  ui: group filter: allow to set namespace for local datastore
  ui: sync edit: source group filters based on sync direction
  ui: add view with separate grids for pull and push sync jobs
  ui: sync job: adapt edit window to be used for pull and push
  ui: sync: pass sync-direction to allow removing push jobs
  ui: sync view: do not use data model proxy for store
  ui: sync view: set sync direction when invoking run task via api
  datastore: move `BackupGroupDeleteStats` to api types
  api types: implement api type for `BackupGroupDeleteStats`
  datastore: increment deleted group counter when removing group
  api: datastore/namespace: return backup groups delete stats on remove
  server: sync job: use delete stats provided by the api

 examples/upload-speed.rs               |   1 +
 pbs-api-types/src/acl.rs               |  17 +
 pbs-api-types/src/datastore.rs         |  64 ++
 pbs-api-types/src/jobs.rs              |  52 ++
 pbs-client/src/backup_writer.rs        | 232 +++++--
 pbs-config/src/sync.rs                 |  11 +-
 pbs-datastore/src/backup_info.rs       |  34 +-
 pbs-datastore/src/datastore.rs         |  27 +-
 proxmox-backup-client/src/benchmark.rs |   1 +
 proxmox-backup-client/src/main.rs      |   1 +
 src/api2/admin/datastore.rs            |  24 +-
 src/api2/admin/namespace.rs            |  20 +-
 src/api2/admin/sync.rs                 |  27 +-
 src/api2/backup/mod.rs                 |   4 +-
 src/api2/config/datastore.rs           |  22 +-
 src/api2/config/notifications/mod.rs   |  15 +-
 src/api2/config/sync.rs                |  53 +-
 src/api2/mod.rs                        |   3 +
 src/api2/pull.rs                       | 108 ----
 src/api2/push.rs                       | 207 ++++++
 src/api2/sync.rs                       | 131 ++++
 src/bin/proxmox-backup-manager.rs      | 216 +++++--
 src/bin/proxmox-backup-proxy.rs        |  25 +-
 src/server/mod.rs                      |   2 +
 src/server/pull.rs                     | 651 ++-----------------
 src/server/push.rs                     | 833 +++++++++++++++++++++++++
 src/server/sync.rs                     | 570 +++++++++++++++++
 www/Makefile                           |   1 +
 www/config/SyncPullPushView.js         |  60 ++
 www/config/SyncView.js                 |  47 +-
 www/datastore/DataStoreList.js         |   2 +-
 www/datastore/Panel.js                 |   2 +-
 www/form/GroupFilter.js                |  18 +-
 www/window/SyncJobEdit.js              |  41 +-
 34 files changed, 2623 insertions(+), 899 deletions(-)
 create mode 100644 src/api2/push.rs
 create mode 100644 src/api2/sync.rs
 create mode 100644 src/server/push.rs
 create mode 100644 src/server/sync.rs
 create mode 100644 www/config/SyncPullPushView.js

-- 
2.39.2