[pbs-devel] [PATCH proxmox-backup v14 00/26] add removable datastores
Hannes Laimer
h.laimer at proxmox.com
Mon Nov 25 07:19:46 CET 2024
On 11/22/24 15:46, Hannes Laimer wrote:
> These patches add support for removable datastores. All removable
> datastores have a backing-device(a UUID) associated with them. Removable
> datastores work like normal ones, just that they can be unplugged. It is
> possible to create a removable datastore, sync backups onto it, unplug
> it and use it on a different PBS.
>
> The datastore path specified is relative to the root of the used device.
> Removable datastores are bind mounted to /mnt/datastore/<NAME>.
> Multiple datastores can be created on a single device, but only device with
> a single datastore on them will be auto-mounted.
>
> When a removable datastore is deleted and 'destroy-data' is set, the
> device has to be mounted. If 'destroy-data' is not set the datastore
> can be deleted even if the device is not present. Removable datastores
> are automatically mounted when plugged in.
>
> v14: thanks @Fabian and @Maximiliano
> * add two functions to get mount status, or ensure mounted for rm.ds.,
> avoiding repeating things
> * use enum for `mount_status` instead of Option<bool>
> * fix problme with unmounting, now check for unmounting maintenance mode before actually unmounting,
> manually chaning the maintenance mode during unmounting will not prevent unmounting
^ *now
> * improve logging for mounting: add context and adjust logging levels
> * improve uuid_mount function: load config file directly and call do_mount function directly without
> going through the API
> * add logging for cleanup on ds deletion
> * move check for nesting into do_create_datastore, and check fir all datastore(not just removable)
> * remove redundant check when deleting through directory
> * use single worker when creating removable datastore through dir endpoint
> * drop get_mount_point function
> * ui: stop loading status after first failed attempt, prevents logs spamming
>
> v13: thanks @Fabian
> * allow multiple datastore on devices
> * replace `is_datastore_available` by a more specific function, it is now
> removable datastore specific and won't be called for normal ones
> * replace removable/is_available in status structs with mount_state,
> which is `None` for normal datastore as it makes it
> less ambiguous what is meant
> * remove notion of 'available' from normal datastores and replace it with
> mounted/mount_status for removable ones, as it never really made sense
> in the first place
> * abort of an unmount task will now reset the maintanance mode
> * add check for race when setting maintenance at end of unmounting task
> * improve documentation and commit messages
> * remove not needed tokio::spawn
> * only auto mount devices with single datastore on them
> * drop ptach that added flag for excluding used partitions
> * make auto mount service not dynamic
> * add debug command to scan devices for datastores they may contain
> * rebase onto master
>
> v12: thanks @Wolfgang
> * use bind mounts, so now
> <DEVICE>/path/to/ds is mounted to /mnt/datastore/<NAME>
> this is a bit cleaner and allows for multiple datastores
> on a single device to be mounted individually, if we
> want to allow that in the future
> * small code improvements
>
>
> v11:
> * rebase onto master
>
> v10: thanks @Gabriel and @Wolfgang
> * make is_datastore_available more robust
> * fix a lot of wording
> * drop format on uuid_mount command for UUID
> * only gather_disk_stats if datastore is available
> * overall code improvements
> * ui: include model in partition selector
> * rebased onto master
>
> v9:
> * change mount point to `/mnt/datastore/<name>`
> * update "Directory" list UI
> * add `absolute_path()` from Dietmar's RFC
> * update docs
>
> v8:
> * still depends on [1]
> * paths for removable datastores are now relative to
> `/mnt/removable_datastore/<UUID>`
> * add support for creation of removable datastore through the
> "create directory" endpoint (last 3 patches)
> * update datastore creation UI
> * update docs
>
> v7:
> * depends on [1]
> * improve logging when waiting for tasks
> * drop `update-datatore-cache` refactoring
> * fix some commit messages
>
> [1] https://lists.proxmox.com/pipermail/pbs-devel/2024-April/008739.html
>
> v6:
> * remove 'drop' flag in datastore cache
> * use maintenance-mode 'unmount' for unmounting process, only for the
> unmounting not for being unmounted
> * rename/simplify update-datastore-cache command
> * ui: integrate new unmounting maintenance mode
> * basically a mix of v3 and v4
>
> v5: thanks @Dietmar and @Christian
> * drop --force for unmount since it'll always fail if tasks are still running, and if
> there are not normal unount will work
> * improve several commit messages
> * improve error message wording
> * add removable datastore section to docs
> * add documentation for is_datastore_available
>
> v4: thanks a lot @Dietmar and @Christian
> * make check if mounted wayyy faster
> * don't keep track of mounting state
> * drop Unplugged maintenance mode
> * use UUID_FORMAT for uuid field
> * a lot of small things, like use of bail!, inline format!, ...
> * include improvement to cache handling
>
> v3:
> * remove lazy unmounting (since 9cba51ac782d04085c0af55128f32178e5132358 is applied)
> * fix CLI (un)mount command, thanks @Gabriel
> * add removable datastore CLI autocomplete helper
> * rebase onto master
> * move ui patches to the end
>
> thanks @Lukas and @Thomas for the feedback
> v2:
> * fix datastore 'add' button in the UI
> * some format!("{}", a) -> format!("{a}")
> * replace `const` with `let` in js code
> * change icon `fa-usb` -> `fa-plug`
> * add some docs
> * add JDoc for parseMaintenanceMode
> * proxmox-schema dep bump
>
> Dietmar Maurer (1):
> maintenance: add 'Unmount' maintenance type
>
> Hannes Laimer (25):
> pbs-api-types: add backing-device to DataStoreConfig
> maintenance: make is_offline more generic
> datastore: add helper for checking if a datastore is mounted
> api: admin: add (un)mount endpoint for removable datastores
> api: removable datastore creation
> api: add check for nested datastores on creation
> pbs-api-types: add mount_status field to DataStoreListItem
> bin: manager: add (un)mount command
> add auto-mounting for removable datastores
> datastore: handle deletion of removable datastore properly
> docs: add removable datastores section
> ui: add partition selector form
> ui: add removable datastore creation support
> ui: add (un)mount button to summary
> ui: tree: render unmounted datastores correctly
> ui: utils: make parseMaintenanceMode more robust
> ui: add datastore status mask for unmounted removable datastores
> ui: maintenance: fix disable msg field if no type is selected
> ui: render 'unmount' maintenance mode correctly
> api: node: allow creation of removable datastore through directory
> endpoint
> api: node: include removable datastores in directory list
> node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
> ui: support create removable datastore through directory creation
> bin: debug: add inspect device command
> api: disks: only return UUID of partitions if it actually is one
>
> debian/proxmox-backup-server.install | 1 +
> debian/proxmox-backup-server.udev | 3 +
> docs/storage.rst | 38 +++
> etc/Makefile | 1 +
> etc/removable-device-attach at .service | 8 +
> pbs-api-types/src/datastore.rs | 47 ++-
> pbs-api-types/src/maintenance.rs | 12 +-
> pbs-config/src/datastore.rs | 14 +
> pbs-datastore/src/datastore.rs | 83 +++++-
> pbs-datastore/src/lib.rs | 4 +-
> src/api2/admin/datastore.rs | 310 +++++++++++++++++++-
> src/api2/config/datastore.rs | 107 ++++++-
> src/api2/node/disks/directory.rs | 74 +++--
> src/api2/status/mod.rs | 30 +-
> src/bin/proxmox_backup_debug/inspect.rs | 149 ++++++++++
> src/bin/proxmox_backup_manager/datastore.rs | 127 +++++++-
> src/server/metric_collection/mod.rs | 4 +
> src/tools/disks/mod.rs | 5 +-
> www/DirectoryList.js | 13 +
> www/Makefile | 1 +
> www/NavigationTree.js | 17 +-
> www/Utils.js | 33 ++-
> www/css/ext6-pbs.css | 20 ++
> www/datastore/DataStoreListSummary.js | 1 +
> www/datastore/Summary.js | 115 +++++++-
> www/form/PartitionSelector.js | 81 +++++
> www/window/CreateDirectory.js | 14 +
> www/window/DataStoreEdit.js | 37 +++
> www/window/MaintenanceOptions.js | 17 +-
> 29 files changed, 1274 insertions(+), 92 deletions(-)
> create mode 100644 etc/removable-device-attach at .service
> create mode 100644 www/form/PartitionSelector.js
>
More information about the pbs-devel
mailing list