[pbs-devel] [PATCH v4 proxmox 00/31] fix #3044: push datastore to remote target
Christian Ebner
c.ebner at proxmox.com
Thu Oct 17 15:26:45 CEST 2024
This patch series implements the functionality to extend the current
sync jobs in pull direction by an additional push direction, allowing
to push contents of a local source datastore to a remote target.
The series implements this by using the REST API of the remote target
for fetching, creating and/or deleting namespaces, groups and backups,
and reuses the clients backup writer functionality to create snapshots
by writing a manifeset on the remote target and sync the fixed index,
dynamic index or blobs contained in the source manifest to the remote,
preserving also encryption information.
Thanks to Fabian for further feedback to the previous version of the
patches, especially regarding users and ACLs.
Most notable changes since version 3 of the patch series include:
- Rework access control permission checks to resemble the pull based
logic more closely.
In order to perform a full sync in push direction, including
permissions for pruning contents with remove vansished, a acl.cfg
looks like below:
```
acl:1:/datastore/source-store:syncoperator at pbs:DatastoreReader
acl:1:/remote:syncoperator at pbs:RemoteAudit
acl:1:/remote/remote-target/target-store:syncoperator at pbs:RemoteDatastorePrune,RemoteSyncPushOperator
```
- Modify access to sync jobs now requires `DatastoreAudit` for both,
pull and push sync jobs
- Fix previously incorrect privs required for removing target
namespaces
- Fix performance bottleneck by not reading known chunks from source,
by sending `MergedChunkInfo` instead of `ChunkInfo` over to the
upload stream
- Factor upload statistic counters and structs out into their own
module and provide methods for easy conversion
- Implement `map_to_target` helper for easier/more readable source to
target mapping for namespaces
- Optimize namespace creation on target, only try creating non
pre-existing namespace components.
- Avoid temp file for manifest and upload source manifest directly
- Not failing on deletion for protected snapshots is now opt-in
- Refactor api endpoint `version` in order to be able to fetch api
version for target
- Reworked `SyncDirection` api type, use `api` macro to reduce code
Most notable changes since version 2 of the patch series include:
- Add checks and extend roles and privs to allow for restricting a local
users access to remote datastore operations. In order to perform a
full sync in push direction, including permissions for namespace
creation and deleting contents with remove vansished, a acl.cfg looks
like below:
```
acl:1:/datastore/datastore:syncoperator at pbs:DatastoreAudit
acl:1:/remote:syncoperator at pbs:RemoteSyncOperator
acl:1:/remote/local/pushme:syncoperator at pbs:RemoteDatastoreModify,RemoteDatastorePrune,RemoteSyncPushOperator
```
Based on further feedback, privs might get further grouped or an
additional role containing most of these can be created.
- Drop patch introducing `no-timestamp-check` flag for backup client, as pointed
out by Fabian this is not needed, as only backups newer than the currently
last available will be pushed.
- Fix read snapshots from source by using the correct namespace.
- Rename PullParameters `owner` to more fitting `local_user`.
- Fix typos in remote sync push operator comment.
- Fix comments not matching the functionality for the cli implementations.
Link to issue on bugtracker:
https://bugzilla.proxmox.com/show_bug.cgi?id=3044
Christian Ebner (31):
client: backup writer: refactor backup and upload stats counters
client: backup writer: factor out merged chunk stream upload
client: backup writer: allow push uploading index and chunks
config: acl: refactor acl path component check for datastore
config: acl: allow namespace components for remote datastores
api types: implement remote acl path method for sync job
api types: define remote permissions and roles for push sync
fix #3044: server: implement push support for sync operations
api types/config: add `sync-push` config type for push sync jobs
api: push: implement endpoint for sync in push direction
api: sync: move sync job invocation to server sync module
api: sync jobs: expose optional `sync-direction` parameter
api: admin: avoid duplicate name for list sync jobs api method
api: config: Require PRIV_DATASTORE_AUDIT to modify sync job
api: config: factor out sync job owner check
api: config: extend read access check by sync direction
api: config: extend modify access check by sync direction
bin: manager: add datastore push cli command
ui: group filter: allow to set namespace for local datastore
ui: sync edit: source group filters based on sync direction
ui: add view with separate grids for pull and push sync jobs
ui: sync job: adapt edit window to be used for pull and push
ui: sync: pass sync-direction to allow removing push jobs
ui: sync view: do not use data model proxy for store
ui: sync view: set sync direction when invoking run task via api
datastore: move `BackupGroupDeleteStats` to api types
api types: implement api type for `BackupGroupDeleteStats`
api/api-types: refactor api endpoint version, add api types
datastore: increment deleted group counter when removing group
api: datastore/namespace: return backup groups delete stats on remove
server: sync job: use delete stats provided by the api
pbs-api-types/src/acl.rs | 25 +
pbs-api-types/src/datastore.rs | 64 ++
pbs-api-types/src/jobs.rs | 51 ++
pbs-api-types/src/lib.rs | 3 +
pbs-api-types/src/version.rs | 98 +++
pbs-client/src/backup_stats.rs | 130 ++++
pbs-client/src/backup_writer.rs | 242 ++++---
pbs-client/src/lib.rs | 4 +
pbs-config/src/acl.rs | 11 +-
pbs-config/src/sync.rs | 16 +-
pbs-datastore/src/backup_info.rs | 34 +-
pbs-datastore/src/datastore.rs | 27 +-
src/api2/admin/datastore.rs | 29 +-
src/api2/admin/namespace.rs | 31 +-
src/api2/admin/sync.rs | 44 +-
src/api2/config/datastore.rs | 15 +-
src/api2/config/notifications/mod.rs | 21 +-
src/api2/config/sync.rs | 285 ++++++--
src/api2/mod.rs | 2 +
src/api2/pull.rs | 108 ---
src/api2/push.rs | 222 +++++++
src/api2/version.rs | 39 +-
src/bin/proxmox-backup-manager.rs | 216 ++++--
src/bin/proxmox-backup-proxy.rs | 24 +-
src/server/mod.rs | 2 +
src/server/pull.rs | 6 +-
src/server/push.rs | 948 +++++++++++++++++++++++++++
src/server/sync.rs | 128 +++-
www/Makefile | 1 +
www/config/SyncPullPushView.js | 60 ++
www/config/SyncView.js | 47 +-
www/datastore/DataStoreList.js | 2 +-
www/datastore/Panel.js | 2 +-
www/form/GroupFilter.js | 18 +-
www/window/SyncJobEdit.js | 45 +-
35 files changed, 2552 insertions(+), 448 deletions(-)
create mode 100644 pbs-api-types/src/version.rs
create mode 100644 pbs-client/src/backup_stats.rs
create mode 100644 src/api2/push.rs
create mode 100644 src/server/push.rs
create mode 100644 www/config/SyncPullPushView.js
--
2.39.5
More information about the pbs-devel
mailing list