[pve-devel] [PATCH-SERIES v3] Make storage migration more flexible

[Fabian Ebner]

On 16.03.20 12:07, Fabian Grünbichler wrote:
> On March 12, 2020 1:08 pm, Fabian Ebner wrote:
>> This is the second half for the previous series [0].
>>
>> This series aims to allow offline migration with '--targetstorage'
>> and improve handling unsued/orphaned disks when migrating.
>> It also makes it possible to migrate volumes between storages with
>> a 'path' and storages without if the target storage uses the standard
>> naming scheme and the source format is supported (e.g. migrating raw
>> volumes between storages with a path and lvm storages).
>>
>> It also adds an apiinfo call to pvesm that can be used to determine
>> APIVER and APIAGE of the remote node and does some general refactoring
>> regarding volume iterators.
>>
>> The series is organised as follows:
>> #1-#7 introduce and implement volume related helpers
>> Mutual dependencies for both qemu-server and container with guest-common,
>> to be precise #1-#3 <-> #5 and #1-#3 <-> #7.
>> #8 adds the apiinfo helper
>> #9-#12 changes storage_migrate interface
>> Another round of mutual dependencies, this time for storage with each
>> of guest-common, qemu-server, container.
>> #13-#18 implement the goals for this series, and the last inter-package
>> dependency is here, because #15 (qemu-server) depends on #13 (storage)
>> #19-#22 improve logging and refactor some code with foreach_volume
>>
>> Changes from v2:
>>      * add apiinfo call and check if remote side supports allow_rename
>>      * add valid_volume_keys and implement (and make use of) foreach_volume iterator
>>      * make volname_for_storage a local function and avoid
>>        requiring a new print_volname for plugins
>>      * parse vm_state in parse_volume instead of parse_drive
>>
>> [0]: https://pve.proxmox.com/pipermail/pve-devel/2020-February/041910.html
> 
> some individual comments (mainly about the refactoring where some areas
> could still be improved) on individual patches.
> 

Ack-ed the comments and nits. Thanks for the review and test!

> gave this a limited test run as well, seems to work as expected :) only
> issue that I noticed (possibly also affects the live migration code
> path?) is that --targetstorage does not respect a storage's content type
> limit. we are also still lacking cleanup for failed sync_disks (e.g.,
> after M out of N disks have been synced), or failed migration after
> successful sync_disks.

Just tested this. It's also present with (current) online migration with 
--targetstorage. The GUI filters target storages for content type, but 
it seems that the backend has no check. I'll add one for the next 
version and also add the disk cleanup.

>>
>>
>> guest-common:
>>
>> Fabian Ebner (5):
>>    Add interfaces for volume-related helpers
>>    Add foreach_unused_volume
>>    foreach_volume: generalize and implement function
>>    Use new storage_migrate interface
>>    Add update_volume_ids
>>
>>   PVE/AbstractConfig.pm | 113 ++++++++++++++++++++++++++++++++++++++----
>>   PVE/Replication.pm    |  12 ++++-
>>   2 files changed, 112 insertions(+), 13 deletions(-)
>>
>>
>> container:
>>
>> Fabian Ebner (3):
>>    Implement volume-related helpers and use new foreach_volume
>>    Use new storage_migrate interface
>>    Use foreach_volume instead of foreach_mountpoint_full to avoid
>>      duplication
>>
>>   src/PVE/API2/LXC.pm        |  4 +--
>>   src/PVE/API2/LXC/Config.pm |  2 +-
>>   src/PVE/CLI/pct.pm         |  2 +-
>>   src/PVE/LXC/Config.pm      | 53 +++++++++++++++++++++-----------------
>>   src/PVE/LXC/Migrate.pm     | 10 ++++---
>>   5 files changed, 40 insertions(+), 31 deletions(-)
>>
>>
>> qemu-server
>>
>> Fabian Ebner (10):
>>    Implement volume-related helpers and use new foreach_volume
>>    Allow parsing vmstate entries
>>    Use new storage_migrate interface
>>    Take note of changes to the volume IDs when migrating and update the
>>      config
>>    Allow specifying targetstorage for offline migration
>>    Update volume IDs in one go
>>    sync_disks: use allow_rename to avoid collisions on the target storage
>>    sync_disks: be more verbose if storage_migrate fails
>>    sync_disks: log output of storage_migrate
>>    update_disk_config: use config volume iterators instead of loops
>>
>>   PVE/API2/Qemu.pm        |  3 ---
>>   PVE/QemuConfig.pm       | 51 +++++++++++++++++++++++++++++++++++------
>>   PVE/QemuMigrate.pm      | 44 +++++++++++++++++++++++++++--------
>>   PVE/QemuServer.pm       | 50 +++++++++++++++++++++-------------------
>>   PVE/QemuServer/Drive.pm |  1 +
>>   5 files changed, 105 insertions(+), 44 deletions(-)
>>
>>
>> storage:
>>
>> Fabian Ebner (4):
>>    Add apiinfo helper to pvesm
>>    Collect optional parameters for storage_migrate into $opts
>>    Introduce allow_rename parameter for pvesm import and storage_migrate
>>    storage_migrate: add volname_for_storage helper
>>
>>   PVE/API2/Storage/Content.pm  |  2 +-
>>   PVE/CLI/pvesm.pm             | 52 +++++++++++++++++---
>>   PVE/Storage.pm               | 94 ++++++++++++++++++++++++++++++------
>>   PVE/Storage/LVMPlugin.pm     | 17 ++++---
>>   PVE/Storage/Plugin.pm        | 16 ++++--
>>   PVE/Storage/ZFSPoolPlugin.pm | 13 +++--
>>   6 files changed, 155 insertions(+), 39 deletions(-)
>>
>> -- 
>> 2.20.1
>>
>>
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel at pve.proxmox.com
>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>>
>>
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>