[pbs-devel] applied-series: [PATCH proxmox-backup v15 00/26] add removable datastores

[Thomas Lamprecht]

Am 25.11.24 um 17:21 schrieb Hannes Laimer:
> These patches add support for removable datastores. All removable
> datastores have a backing-device(a UUID) associated with them. Removable
> datastores work like normal ones, just that they can be unplugged. It is
> possible to create a removable datastore, sync backups onto it, unplug
> it and use it on a different PBS.
> 
> The datastore path specified is relative to the root of the used device.
> Removable datastores are bind mounted to /mnt/datastore/<NAME>.
> Multiple datastores can be created on a single device, but only device with 
> a single datastore on them will be auto-mounted.
> 
> When a removable datastore is deleted and 'destroy-data' is set, the
> device has to be mounted. If 'destroy-data' is not set the datastore
> can be deleted even if the device is not present. Removable datastores
> are automatically mounted when plugged in. 

> Dietmar Maurer (1):
>   maintenance: add 'Unmount' maintenance type
> 
> Hannes Laimer (25):
>   pbs-api-types: add backing-device to DataStoreConfig
>   maintenance: make is_offline more generic
>   datastore: add helper for checking if a datastore is mounted
>   api: admin: add (un)mount endpoint for removable datastores
>   api: removable datastore creation
>   api: add check for nested datastores on creation
>   pbs-api-types: add mount_status field to DataStoreListItem
>   bin: manager: add (un)mount command
>   add auto-mounting for removable datastores
>   datastore: handle deletion of removable datastore properly
>   docs: add removable datastores section
>   ui: add partition selector form
>   ui: add removable datastore creation support
>   ui: add (un)mount button to summary
>   ui: tree: render unmounted datastores correctly
>   ui: utils: make parseMaintenanceMode more robust
>   ui: add datastore status mask for unmounted removable datastores
>   ui: maintenance: fix disable msg field if no type is selected
>   ui: render 'unmount' maintenance mode correctly
>   api: node: allow creation of removable datastore through directory
>     endpoint
>   api: node: include removable datastores in directory list
>   node: disks: replace BASE_MOUNT_DIR with DATASTORE_MOUNT_DIR
>   ui: support create removable datastore through directory creation
>   bin: debug: add inspect device command
>   api: disks: only return UUID of partitions if it actually is one
> 
>  debian/proxmox-backup-server.install        |   1 +
>  debian/proxmox-backup-server.udev           |   3 +
>  docs/storage.rst                            |  38 +++
>  etc/Makefile                                |   1 +
>  etc/removable-device-attach at .service        |   8 +
>  pbs-api-types/src/datastore.rs              |  47 ++-
>  pbs-api-types/src/maintenance.rs            |  12 +-
>  pbs-config/src/datastore.rs                 |  14 +
>  pbs-datastore/src/datastore.rs              |  83 ++++-
>  pbs-datastore/src/lib.rs                    |   4 +-
>  src/api2/admin/datastore.rs                 | 316 +++++++++++++++++++-
>  src/api2/config/datastore.rs                | 122 +++++++-
>  src/api2/node/disks/directory.rs            |  74 +++--
>  src/api2/status/mod.rs                      |  30 +-
>  src/bin/proxmox_backup_debug/inspect.rs     | 149 +++++++++
>  src/bin/proxmox_backup_manager/datastore.rs | 126 +++++++-
>  src/server/metric_collection/mod.rs         |   4 +
>  src/tools/disks/mod.rs                      |   5 +-
>  www/DirectoryList.js                        |  13 +
>  www/Makefile                                |   1 +
>  www/NavigationTree.js                       |  18 +-
>  www/Utils.js                                |  33 +-
>  www/css/ext6-pbs.css                        |  20 ++
>  www/datastore/DataStoreListSummary.js       |   1 +
>  www/datastore/Summary.js                    | 122 +++++++-
>  www/form/PartitionSelector.js               |  81 +++++
>  www/window/CreateDirectory.js               |  14 +
>  www/window/DataStoreEdit.js                 |  32 ++
>  www/window/MaintenanceOptions.js            |  17 +-
>  29 files changed, 1298 insertions(+), 91 deletions(-)
>  create mode 100644 etc/removable-device-attach at .service
>  create mode 100644 www/form/PartitionSelector.js
> 


applied series, thanks!

I did a handful of follow-up commits for the UI behavior, especially w.r.t. how mount
and unmount are processed. Just check the commits themselves, in general I tried to be
descriptive. Please take a look at them to cross-check for any unforeseen regressions.

What would be still good is adding to the docs how jobs like sync, GC, ... are handled
when the datastore is not plugged and if they are caught up on or not when it's plugged
again. That should have a bit higher priority.

Additionally looking into a "remove" event udev hook to clean up a mount point, e.g. if
a user just pulls out the USB pen drive; while that naturally can never be fully hedged
against, especially in terms of data safety, it would be still nice if re-plugging it
works again. For the use case where one just uses an external drive as cheap physical
offsite copy, i.e. as cheap tape replacement, nothing should be permanently broken, as
one can just reinsert the drive and trigger a new sync and then do a clean unmount.
In my tests with some XFS formatted USB pen drive the mount entry is kept when I suddenly
remove the drive (while the datastore is inactive though) and thus I get an error about
the filesystem UUID already existing when the udev hook tries to mount it again when I
re-plug the USB pen drive.
We do not have to bend backwards for this, but atm. I basically only can resolve this
by unmounting the old mountpoint manually, e.g. through a root shell (or reboot I guess),
both not _that_ nice UX wise.

In any way: thanks for going the distance here, this is definitively one of those
features were initial envisioned amount of work was rather way off due to quite some bike
shedding potential but also all the ugliness of dealing with transient mount points and
resources that are not persistent. Lets fix rough edges and potential issues as follow-ups.