[pve-devel] [PATCH-SERIES v3] Make storage migration more flexible

[Fabian Grünbichler]

On March 12, 2020 1:08 pm, Fabian Ebner wrote:
> This is the second half for the previous series [0].
> 
> This series aims to allow offline migration with '--targetstorage'
> and improve handling unsued/orphaned disks when migrating.
> It also makes it possible to migrate volumes between storages with
> a 'path' and storages without if the target storage uses the standard
> naming scheme and the source format is supported (e.g. migrating raw
> volumes between storages with a path and lvm storages).
> 
> It also adds an apiinfo call to pvesm that can be used to determine
> APIVER and APIAGE of the remote node and does some general refactoring
> regarding volume iterators.
> 
> The series is organised as follows:
> #1-#7 introduce and implement volume related helpers
> Mutual dependencies for both qemu-server and container with guest-common,
> to be precise #1-#3 <-> #5 and #1-#3 <-> #7.
> #8 adds the apiinfo helper
> #9-#12 changes storage_migrate interface
> Another round of mutual dependencies, this time for storage with each
> of guest-common, qemu-server, container.
> #13-#18 implement the goals for this series, and the last inter-package
> dependency is here, because #15 (qemu-server) depends on #13 (storage)
> #19-#22 improve logging and refactor some code with foreach_volume
> 
> Changes from v2:
>     * add apiinfo call and check if remote side supports allow_rename
>     * add valid_volume_keys and implement (and make use of) foreach_volume iterator
>     * make volname_for_storage a local function and avoid
>       requiring a new print_volname for plugins
>     * parse vm_state in parse_volume instead of parse_drive
> 
> [0]: https://pve.proxmox.com/pipermail/pve-devel/2020-February/041910.html

some individual comments (mainly about the refactoring where some areas 
could still be improved) on individual patches.

gave this a limited test run as well, seems to work as expected :) only 
issue that I noticed (possibly also affects the live migration code 
path?) is that --targetstorage does not respect a storage's content type 
limit. we are also still lacking cleanup for failed sync_disks (e.g., 
after M out of N disks have been synced), or failed migration after 
successful sync_disks.

> 
> 
> guest-common:
> 
> Fabian Ebner (5):
>   Add interfaces for volume-related helpers
>   Add foreach_unused_volume
>   foreach_volume: generalize and implement function
>   Use new storage_migrate interface
>   Add update_volume_ids
> 
>  PVE/AbstractConfig.pm | 113 ++++++++++++++++++++++++++++++++++++++----
>  PVE/Replication.pm    |  12 ++++-
>  2 files changed, 112 insertions(+), 13 deletions(-)
> 
> 
> container:
> 
> Fabian Ebner (3):
>   Implement volume-related helpers and use new foreach_volume
>   Use new storage_migrate interface
>   Use foreach_volume instead of foreach_mountpoint_full to avoid
>     duplication
> 
>  src/PVE/API2/LXC.pm        |  4 +--
>  src/PVE/API2/LXC/Config.pm |  2 +-
>  src/PVE/CLI/pct.pm         |  2 +-
>  src/PVE/LXC/Config.pm      | 53 +++++++++++++++++++++-----------------
>  src/PVE/LXC/Migrate.pm     | 10 ++++---
>  5 files changed, 40 insertions(+), 31 deletions(-)
> 
> 
> qemu-server
> 
> Fabian Ebner (10):
>   Implement volume-related helpers and use new foreach_volume
>   Allow parsing vmstate entries
>   Use new storage_migrate interface
>   Take note of changes to the volume IDs when migrating and update the
>     config
>   Allow specifying targetstorage for offline migration
>   Update volume IDs in one go
>   sync_disks: use allow_rename to avoid collisions on the target storage
>   sync_disks: be more verbose if storage_migrate fails
>   sync_disks: log output of storage_migrate
>   update_disk_config: use config volume iterators instead of loops
> 
>  PVE/API2/Qemu.pm        |  3 ---
>  PVE/QemuConfig.pm       | 51 +++++++++++++++++++++++++++++++++++------
>  PVE/QemuMigrate.pm      | 44 +++++++++++++++++++++++++++--------
>  PVE/QemuServer.pm       | 50 +++++++++++++++++++++-------------------
>  PVE/QemuServer/Drive.pm |  1 +
>  5 files changed, 105 insertions(+), 44 deletions(-)
> 
> 
> storage:
> 
> Fabian Ebner (4):
>   Add apiinfo helper to pvesm
>   Collect optional parameters for storage_migrate into $opts
>   Introduce allow_rename parameter for pvesm import and storage_migrate
>   storage_migrate: add volname_for_storage helper
> 
>  PVE/API2/Storage/Content.pm  |  2 +-
>  PVE/CLI/pvesm.pm             | 52 +++++++++++++++++---
>  PVE/Storage.pm               | 94 ++++++++++++++++++++++++++++++------
>  PVE/Storage/LVMPlugin.pm     | 17 ++++---
>  PVE/Storage/Plugin.pm        | 16 ++++--
>  PVE/Storage/ZFSPoolPlugin.pm | 13 +++--
>  6 files changed, 155 insertions(+), 39 deletions(-)
> 
> -- 
> 2.20.1
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
>