[pve-devel] [PATCH-SERIES v3] Make storage migration more flexible
Fabian Grünbichler
f.gruenbichler at proxmox.com
Mon Mar 16 16:24:22 CET 2020
On March 16, 2020 2:53 pm, Fabian Ebner wrote:
>
> On 16.03.20 12:07, Fabian Grünbichler wrote:
>> On March 12, 2020 1:08 pm, Fabian Ebner wrote:
>>> This is the second half for the previous series [0].
>>>
>>> This series aims to allow offline migration with '--targetstorage'
>>> and improve handling unsued/orphaned disks when migrating.
>>> It also makes it possible to migrate volumes between storages with
>>> a 'path' and storages without if the target storage uses the standard
>>> naming scheme and the source format is supported (e.g. migrating raw
>>> volumes between storages with a path and lvm storages).
>>>
>>> It also adds an apiinfo call to pvesm that can be used to determine
>>> APIVER and APIAGE of the remote node and does some general refactoring
>>> regarding volume iterators.
>>>
>>> The series is organised as follows:
>>> #1-#7 introduce and implement volume related helpers
>>> Mutual dependencies for both qemu-server and container with guest-common,
>>> to be precise #1-#3 <-> #5 and #1-#3 <-> #7.
>>> #8 adds the apiinfo helper
>>> #9-#12 changes storage_migrate interface
>>> Another round of mutual dependencies, this time for storage with each
>>> of guest-common, qemu-server, container.
>>> #13-#18 implement the goals for this series, and the last inter-package
>>> dependency is here, because #15 (qemu-server) depends on #13 (storage)
>>> #19-#22 improve logging and refactor some code with foreach_volume
>>>
>>> Changes from v2:
>>> * add apiinfo call and check if remote side supports allow_rename
>>> * add valid_volume_keys and implement (and make use of) foreach_volume iterator
>>> * make volname_for_storage a local function and avoid
>>> requiring a new print_volname for plugins
>>> * parse vm_state in parse_volume instead of parse_drive
>>>
>>> [0]: https://pve.proxmox.com/pipermail/pve-devel/2020-February/041910.html
>>
>> some individual comments (mainly about the refactoring where some areas
>> could still be improved) on individual patches.
>>
>
> Ack-ed the comments and nits. Thanks for the review and test!
>
>> gave this a limited test run as well, seems to work as expected :) only
>> issue that I noticed (possibly also affects the live migration code
>> path?) is that --targetstorage does not respect a storage's content type
>> limit. we are also still lacking cleanup for failed sync_disks (e.g.,
>> after M out of N disks have been synced), or failed migration after
>> successful sync_disks.
>
> Just tested this. It's also present with (current) online migration with
> --targetstorage. The GUI filters target storages for content type, but
> it seems that the backend has no check. I'll add one for the next
> version and also add the disk cleanup.
>
that would be great! for the disk cleanup, we need to be really careful
to only delete those disks that we actually allocated (so
differentiation between 'attempted to storage migrate but failed because
of naming conflict' and 'attempted to storage migrate but failed after
allocation' is key, and it's better to err on the side of caution and
skip cleanup than to delete some pre-existing disk).
>>>
>>>
>>> guest-common:
>>>
>>> Fabian Ebner (5):
>>> Add interfaces for volume-related helpers
>>> Add foreach_unused_volume
>>> foreach_volume: generalize and implement function
>>> Use new storage_migrate interface
>>> Add update_volume_ids
>>>
>>> PVE/AbstractConfig.pm | 113 ++++++++++++++++++++++++++++++++++++++----
>>> PVE/Replication.pm | 12 ++++-
>>> 2 files changed, 112 insertions(+), 13 deletions(-)
>>>
>>>
>>> container:
>>>
>>> Fabian Ebner (3):
>>> Implement volume-related helpers and use new foreach_volume
>>> Use new storage_migrate interface
>>> Use foreach_volume instead of foreach_mountpoint_full to avoid
>>> duplication
>>>
>>> src/PVE/API2/LXC.pm | 4 +--
>>> src/PVE/API2/LXC/Config.pm | 2 +-
>>> src/PVE/CLI/pct.pm | 2 +-
>>> src/PVE/LXC/Config.pm | 53 +++++++++++++++++++++-----------------
>>> src/PVE/LXC/Migrate.pm | 10 ++++---
>>> 5 files changed, 40 insertions(+), 31 deletions(-)
>>>
>>>
>>> qemu-server
>>>
>>> Fabian Ebner (10):
>>> Implement volume-related helpers and use new foreach_volume
>>> Allow parsing vmstate entries
>>> Use new storage_migrate interface
>>> Take note of changes to the volume IDs when migrating and update the
>>> config
>>> Allow specifying targetstorage for offline migration
>>> Update volume IDs in one go
>>> sync_disks: use allow_rename to avoid collisions on the target storage
>>> sync_disks: be more verbose if storage_migrate fails
>>> sync_disks: log output of storage_migrate
>>> update_disk_config: use config volume iterators instead of loops
>>>
>>> PVE/API2/Qemu.pm | 3 ---
>>> PVE/QemuConfig.pm | 51 +++++++++++++++++++++++++++++++++++------
>>> PVE/QemuMigrate.pm | 44 +++++++++++++++++++++++++++--------
>>> PVE/QemuServer.pm | 50 +++++++++++++++++++++-------------------
>>> PVE/QemuServer/Drive.pm | 1 +
>>> 5 files changed, 105 insertions(+), 44 deletions(-)
>>>
>>>
>>> storage:
>>>
>>> Fabian Ebner (4):
>>> Add apiinfo helper to pvesm
>>> Collect optional parameters for storage_migrate into $opts
>>> Introduce allow_rename parameter for pvesm import and storage_migrate
>>> storage_migrate: add volname_for_storage helper
>>>
>>> PVE/API2/Storage/Content.pm | 2 +-
>>> PVE/CLI/pvesm.pm | 52 +++++++++++++++++---
>>> PVE/Storage.pm | 94 ++++++++++++++++++++++++++++++------
>>> PVE/Storage/LVMPlugin.pm | 17 ++++---
>>> PVE/Storage/Plugin.pm | 16 ++++--
>>> PVE/Storage/ZFSPoolPlugin.pm | 13 +++--
>>> 6 files changed, 155 insertions(+), 39 deletions(-)
>>>
>>> --
>>> 2.20.1
>>>
>>>
>>> _______________________________________________
>>> pve-devel mailing list
>>> pve-devel at pve.proxmox.com
>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>>>
>>>
>>
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel at pve.proxmox.com
>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>>
>
More information about the pve-devel
mailing list