[pve-devel] [PATCH-SERIES v4] Make storage migration more flexible

[Fabian Ebner]

Previous discussion here: [0].

This series aims to allow offline migration with '--targetstorage'
and improve handling unsued/orphaned disks when migrating.
It also makes it possible to migrate volumes between storages with
a 'path' and storages without if the target storage uses the standard
naming scheme and the source format is supported (e.g. migrating raw
volumes between storages with a path and lvm storages).

It also adds an apiinfo call to pvesm that can be used to determine
APIVER and APIAGE of the remote node and does some general refactoring
regarding volume iterators.

NOTE! that the bump here conflicts with the bump here[1] and either
one would need to be adapted if/when applying both.


The series is organised as follows:

#1 and #2 check for content types
These are independent from the rest.

#3-#11 introduce and make use of volume-related helpers
#7,#8,#10,#11 are not needed for the rest of the series.
Dependencies: qemu-server,container <-> guest-common

#12-#15 storage_migrate interface change
Dependencies: qemu-server,guest-common,container -> storage

#16-#23 do the work to allow offline --targetstorage and to avoid collisions
#22 is optional, but consolidates updating the volume IDs in the config
Dependencies: qemu-server -> guest-common
              qemu-server <-> storage

#24-#27 further improvements
#27 makes migrating raw between dir and lvm possible


Changes from v3:
    * new patches to check for content types
    * new patches to replace foreach_drive/mountpoint with foreach_volume
    * new patch to replace parse_ct_rootfs/mountpoint with parse_volume
    * new patch for cleanup_remotedisks
    * fixed/changed other smaller things pointed out by Fabian G.

[0]: https://pve.proxmox.com/pipermail/pve-devel/2020-March/042007.html
[1]: https://pve.proxmox.com/pipermail/pve-devel/2020-March/042539.html


guest-common:

Fabian Ebner (5):
  Add interfaces for volume-related helpers
  Add foreach_unused_volume
  foreach_volume(_full): generalize and implement function
  Use new storage_migrate interface
  Add update_volume_ids

 PVE/AbstractConfig.pm | 122 ++++++++++++++++++++++++++++++++++++++----
 PVE/Replication.pm    |  12 ++++-
 2 files changed, 121 insertions(+), 13 deletions(-)


container:

Fabian Ebner (4):
  Implement volume-related helpers and use new foreach_volume
  Use foreach_volume instead of foreach_mountpoint-variants
  Use parse_volume instead of parse_ct-variants
  Use new storage_migrate interface

 src/PVE/API2/LXC.pm        | 25 ++++------
 src/PVE/API2/LXC/Config.pm |  2 +-
 src/PVE/API2/LXC/Status.pm |  2 +-
 src/PVE/CLI/pct.pm         |  9 ++--
 src/PVE/LXC.pm             | 21 ++++-----
 src/PVE/LXC/Config.pm      | 95 +++++++++++++++-----------------------
 src/PVE/LXC/Create.pm      |  4 +-
 src/PVE/LXC/Migrate.pm     | 16 ++++---
 src/PVE/VZDump/LXC.pm      |  2 +-
 src/lxc-pve-prestart-hook  |  2 +-
 10 files changed, 75 insertions(+), 103 deletions(-)


qemu-server:

Fabian Ebner (13):
  migration with targetstorage: check if target storage supports images
  Implement volume-related helpers and use new foreach_volume
  update_disk_config: use config volume iterators instead of loops
  Switch to using foreach_volume instead of foreach_drive
  Use new storage_migrate interface
  Allow parsing vmstate entries
  Take note of changes to the volume IDs when migrating and update the
    config
  Allow specifying targetstorage for offline migration
  Update volume IDs in one go
  sync_disks: use allow_rename to avoid collisions on the target storage
  sync_disks: be more verbose if storage_migrate fails
  sync_disks: log output of storage_migrate
  cleanup_remotedisks: also include those migrated with storage_migrate

 PVE/API2/Qemu.pm            |   9 +--
 PVE/QemuConfig.pm           |  53 +++++++++++++---
 PVE/QemuMigrate.pm          |  91 +++++++++++++++++++-------
 PVE/QemuServer.pm           | 123 +++++++++++++++++++++++++-----------
 PVE/QemuServer/Cloudinit.pm |   2 +-
 PVE/QemuServer/Drive.pm     |  61 ------------------
 PVE/VZDump/QemuServer.pm    |   3 +-
 7 files changed, 205 insertions(+), 137 deletions(-)


storage:

Fabian Ebner (5):
  storage_migrate: check if target storage supports content type
  Add apiinfo helper to pvesm
  Collect optional parameters for storage_migrate into $opts
  Introduce allow_rename parameter for pvesm import and storage_migrate
  storage_migrate: add volname_for_storage helper

 PVE/API2/Storage/Content.pm  |  2 +-
 PVE/CLI/pvesm.pm             | 52 ++++++++++++++++---
 PVE/Storage.pm               | 99 ++++++++++++++++++++++++++++++------
 PVE/Storage/LVMPlugin.pm     | 17 ++++---
 PVE/Storage/Plugin.pm        | 16 ++++--
 PVE/Storage/ZFSPoolPlugin.pm | 13 +++--
 6 files changed, 160 insertions(+), 39 deletions(-)

-- 
2.20.1