[pve-devel] [PATCH v3 pve-manager 00/23] storage replication - second try v3

[Dietmar Maurer]

The first series from Wolfgang introduced circular package dependency. So
I moved most of the code to package pve-manager.

Notable changes:

- use a separate replication config file: /etc/pve/replication.cfg
- allow more than one job per guest (as long as targets are different).
- do not add ssh calls to the storage API
- add regression tests
- use systemd like calendar events ('schedule') instead of a simlple integer 'interval'
- more flexible disk handling (add disks is now possible).
- reduce number of ssh invocation on guests with more than one disk
- improved command line tool 'pvesr' can now edit/change configuration
- pvesr status: show additional values like 'NextSync' and 'Duration' of last sync
- save pid in state, so that we can check if replication is still active/running

Please note that actual volume replication is not implemented, but you can run
some interesting regression tests (replication-test5.pl). I just wanted to send
what I have so far because the series already contain 18 patches ...

Changes in v2:
-  add cleanups requested by fabian
-  implement replicate_volume()
-  aquire guest_migration_lock during replication
-  PVE::API2Tools::resolve_proxyto - new helper
-  PVE::API2::ReplicationConfig - implement proxyto_callback for delete

Changes in v3:
- do not use proxyto_callback for delete. Instead, use the new remove_job property to mark a job for removal.
- implement delete
- new regression tests for delete
- use new GuestHelpers class
- minor cleanups


Dietmar Maurer (23):
  pvesr: add pve storage replication tool
  add regression test environment for replication
  replication_test2.pl: test replication scheduler
  PVE/Replication.pm: save pid/ptime to running job state
  PVE::Replication - use new calendar events instead of interval
  replication_test3.pl: regression test for replication to same node
  ReplicationTestEnv.pm: add job tracking log
  replication_test4.pl: Test replication job failure
  pvesr run: add --verbose flag
  pvesr prepare-local-job: new helper
  ReplicationTestEnv.pm: mock storage content methods
  ReplicationTestEnv.pm: mock get_ssh_info and ssh_info_to_command
  pvesr finalize-local-job: add helper to cleanup job
  PVE::Replication::remote_prepare_local_job - new helper
  PVE::Replication::remote_finalize_local_job - new helper
  PVE::Replication::replicate - implement replicate
  bin/test/Makefile: run all replication tests
  ReplicationTestEnv.pm: avoid warning about undefined value
  PVE::Replication - implement replicate_volume()
  PVE::Replication - aquire guest_migration_lock during replication
  PVE::API2Tools::resolve_proxyto - new helper
  PVE::API2::ReplicationConfig - implement delete
  replication_test5.pl: add test for job removal

 Makefile                       |   1 +
 PVE/API2/Cluster.pm            |   7 +
 PVE/API2/Makefile              |   2 +
 PVE/API2/Nodes.pm              |   7 +
 PVE/API2/Replication.pm        |  99 ++++++++
 PVE/API2/ReplicationConfig.pm  | 217 +++++++++++++++++
 PVE/API2Tools.pm               |  18 ++
 PVE/CLI/Makefile               |   2 +-
 PVE/CLI/pvesr.pm               | 350 +++++++++++++++++++++++++++
 PVE/HTTPServer.pm              |   9 +-
 PVE/Makefile                   |   1 +
 PVE/Replication.pm             | 526 +++++++++++++++++++++++++++++++++++++++++
 bin/Makefile                   |   2 +-
 bin/pvesh                      |   8 +-
 bin/pvesr                      |   8 +
 bin/test/Makefile              |   6 +
 bin/test/ReplicationTestEnv.pm | 328 +++++++++++++++++++++++++
 bin/test/replication_test1.pl  |  47 ++++
 bin/test/replication_test2.pl  | 102 ++++++++
 bin/test/replication_test3.pl  |  51 ++++
 bin/test/replication_test4.log |  13 +
 bin/test/replication_test4.pl  |  68 ++++++
 bin/test/replication_test5.log |  59 +++++
 bin/test/replication_test5.pl  | 149 ++++++++++++
 24 files changed, 2069 insertions(+), 11 deletions(-)
 create mode 100644 PVE/API2/Replication.pm
 create mode 100644 PVE/API2/ReplicationConfig.pm
 create mode 100644 PVE/CLI/pvesr.pm
 create mode 100644 PVE/Replication.pm
 create mode 100644 bin/pvesr
 create mode 100755 bin/test/ReplicationTestEnv.pm
 create mode 100755 bin/test/replication_test1.pl
 create mode 100755 bin/test/replication_test2.pl
 create mode 100755 bin/test/replication_test3.pl
 create mode 100644 bin/test/replication_test4.log
 create mode 100755 bin/test/replication_test4.pl
 create mode 100644 bin/test/replication_test5.log
 create mode 100755 bin/test/replication_test5.pl

-- 
2.11.0