[pbs-devel] partially-applied: [PATCH v3 proxmox-backup 00/33] fix #3044: push datastore to remote target
Fabian Grünbichler
f.gruenbichler at proxmox.com
Mon Oct 14 13:04:58 CEST 2024
applied the initial seven refactoring patches 1-4 & 9-11, leaving out
the backup writer ones for now since those might still see some changes.
On September 12, 2024 4:32 pm, Christian Ebner wrote:
> This patch series implements the functionality to extend the current
> sync jobs in pull direction by an additional push direction, allowing
> to push contents of a local source datastore to a remote target.
>
> The series implements this by using the REST API of the remote target
> for fetching, creating and/or deleting namespaces, groups and backups,
> and reuses the clients backup writer functionality to create snapshots
> by writing a manifeset on the remote target and sync the fixed index,
> dynamic index or blobs contained in the source manifest to the remote,
> preserving also encryption information.
>
> Thanks to Fabian for further feedback to the previous version of the
> patches, especially regarding users and ACLs.
>
> Most notable changes since version 2 of the patch series include:
> - Add checks and extend roles and privs to allow for restricting a local
> users access to remote datastore operations. In order to perform a
> full sync in push direction, including permissions for namespace
> creation and deleting contents with remove vansished, a acl.cfg looks
> like below:
> ```
> acl:1:/datastore/datastore:syncoperator at pbs:DatastoreAudit
> acl:1:/remote:syncoperator at pbs:RemoteSyncOperator
> acl:1:/remote/local/pushme:syncoperator at pbs:RemoteDatastoreModify,RemoteDatastorePrune,RemoteSyncPushOperator
> ```
> Based on further feedback, privs might get further grouped or an
> additional role containing most of these can be created.
> - Drop patch introducing `no-timestamp-check` flag for backup client, as pointed
> out by Fabian this is not needed, as only backups newer than the currently
> last available will be pushed.
> - Fix read snapshots from source by using the correct namespace.
> - Rename PullParameters `owner` to more fitting `local_user`.
> - Fix typos in remote sync push operator comment.
> - Fix comments not matching the functionality for the cli implementations.
>
> The patch series is structured as follows in this version:
> - patch 1 is a cleanup patch fixing typos in api documentation.
> - patches 2 to 7 are patches restructuring the current code so that
> functionality of the current pull implementation can be reused for
> the push implementation as well.
> - patch 8 extens the backup writers functionality to be able to push
> snapshots to the target.
> - patches 9 to 11 are once again preparatory patches for shared
> implementation of sync jobs in pull and push direction.
> - patches 12 to 14 define the required permission acls and roles.
> - patch 15 implements almost all of the logic required for the push,
> including pushing of the datastore, namespace, groups and snapshots,
> taking into account also filters and additional sync flags.
> - patch 16 extends the current sync job configuration by a new config
> type `sync-push` allowing to configure sync jobs in push direction
> while limiting possible misconfiguration errors.
> - patches 17 to 28 expose the new sync job direction via the API, CLI
> and WebUI.
> - patches 29 to 33 finally are followup patches, changing the return
> type for the backup group and namespace delete REST API endpoints
> to return statistics on the deleted snapshots, groups and namespaces,
> which are then used to include this information in the task log.
> As this is an API breaking change, the patches are kept independent
> from the other patches.
>
> Link to issue on bugtracker:
> https://bugzilla.proxmox.com/show_bug.cgi?id=3044
>
> Christian Ebner (33):
> api: datastore: add missing whitespace in description
> server: sync: move sync related stats to common module
> server: sync: move reader trait to common sync module
> server: sync: move source to common sync module
> client: backup writer: bundle upload stats counters
> client: backup writer: factor out merged chunk stream upload
> client: backup writer: add chunk count and duration stats
> client: backup writer: allow push uploading index and chunks
> server: sync: move skip info/reason to common sync module
> server: sync: make skip reason message more genenric
> server: sync: factor out namespace depth check into sync module
> config: acl: mention optional namespace acl path component
> config: acl: allow namespace components for remote datastores
> api types: define remote permissions and roles for push sync
> fix #3044: server: implement push support for sync operations
> config: jobs: add `sync-push` config type for push sync jobs
> api: push: implement endpoint for sync in push direction
> api: sync: move sync job invocation to server sync module
> api: sync jobs: expose optional `sync-direction` parameter
> api: sync: add permission checks for push sync jobs
> bin: manager: add datastore push cli command
> ui: group filter: allow to set namespace for local datastore
> ui: sync edit: source group filters based on sync direction
> ui: add view with separate grids for pull and push sync jobs
> ui: sync job: adapt edit window to be used for pull and push
> ui: sync: pass sync-direction to allow removing push jobs
> ui: sync view: do not use data model proxy for store
> ui: sync view: set sync direction when invoking run task via api
> datastore: move `BackupGroupDeleteStats` to api types
> api types: implement api type for `BackupGroupDeleteStats`
> datastore: increment deleted group counter when removing group
> api: datastore/namespace: return backup groups delete stats on remove
> server: sync job: use delete stats provided by the api
>
> pbs-api-types/src/acl.rs | 32 +
> pbs-api-types/src/datastore.rs | 64 ++
> pbs-api-types/src/jobs.rs | 52 ++
> pbs-client/src/backup_writer.rs | 228 +++++--
> pbs-config/src/acl.rs | 7 +-
> pbs-config/src/sync.rs | 11 +-
> pbs-datastore/src/backup_info.rs | 34 +-
> pbs-datastore/src/datastore.rs | 27 +-
> src/api2/admin/datastore.rs | 24 +-
> src/api2/admin/namespace.rs | 20 +-
> src/api2/admin/sync.rs | 45 +-
> src/api2/config/datastore.rs | 22 +-
> src/api2/config/notifications/mod.rs | 15 +-
> src/api2/config/sync.rs | 84 ++-
> src/api2/mod.rs | 2 +
> src/api2/pull.rs | 108 ----
> src/api2/push.rs | 182 ++++++
> src/bin/proxmox-backup-manager.rs | 216 +++++--
> src/bin/proxmox-backup-proxy.rs | 25 +-
> src/server/mod.rs | 3 +
> src/server/pull.rs | 658 ++------------------
> src/server/push.rs | 883 +++++++++++++++++++++++++++
> src/server/sync.rs | 700 +++++++++++++++++++++
> www/Makefile | 1 +
> www/config/SyncPullPushView.js | 60 ++
> www/config/SyncView.js | 47 +-
> www/datastore/DataStoreList.js | 2 +-
> www/datastore/Panel.js | 2 +-
> www/form/GroupFilter.js | 18 +-
> www/window/SyncJobEdit.js | 45 +-
> 30 files changed, 2706 insertions(+), 911 deletions(-)
> create mode 100644 src/api2/push.rs
> create mode 100644 src/server/push.rs
> create mode 100644 src/server/sync.rs
> create mode 100644 www/config/SyncPullPushView.js
>
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
More information about the pbs-devel
mailing list