[pve-devel] [RFC ha-manager v2] add migrate shutdown policy with an maintenance mode POC
Thomas Lamprecht
t.lamprecht at proxmox.com
Sat Oct 5 21:26:46 CEST 2019
This adds handling for a new shutdown policy, namely "migrate".
If that is set then the LRM doesn't queues stop jobs, but transitions
to a new mode, namely 'maintenance'.
The LRM modes now get passed from the CRM in the NodeStatus update
method, this allows to detect such a mode and make node-status state
transitions. Effectively we only allow to transition if we're
currently online, else this is ignored. 'maintenance' does not
protects from fencing.
The moving then gets done by select service node. A node in
maintenance mode is not in "list_online_nodes" and so also not in
online_node_usage used to re-calculate if a service needs to be
moved. Only started services will get moved, this can be done almost
by leveraging exiting behavior, the next_state_started FSM state
transition method just needs to be thaught to not early return for
nodes which are not online but in maintenance mode.
A few small test adapted from the other policy tests is added to
showcase behavior with reboot, shutdown, and shutdown of the current
manager.
Tested lightly around in a real three node HA cluster, worked quite
good, also with priority groups (failback after the node comes back
online and out of maintenance again).+
Besides some implementation details two question still are not solved
by this approach:
* what if a service cannot be moved away, either by errors or as no
alternative node is found by select_service_node
- a idea would be to track the time and see if we're stuck (this is
not to hard), in such a case we could stop the services after X
minutes and continue.
- waiting indefinitely, the user set this up like this in the first
place. For the real world this can be a bit bad, as SSH,
pveproxy, .. may got already stopped, so we'd need to add some
service ordering to allow admins to still connect and do manual
interventions
* a full cluster shutdown, but that is even without this mode not to
ideal... And as long as it's a central setting and admin has a
single switch to flip to make it work, so not sure how much
handling we want to do here, if we go over the point where we have no
quorum we're dead anyhow, soo.. at least not really an issue of
this series, orthogonal related but not more.
For real world usability the datacenter.cfg schema needs to be
changed to allow the migrate shutdown policy, but that's trivial
Signed-off-by: Thomas Lamprecht <t.lamprecht at proxmox.com>
---
As this really seemed like a path forward, after quite a bit of thinking, I had
to just tinker around a bit more today.. so here it is, more fleshed out and
even a bit tested ;D
changes rfc-v1 -> rfc-v2:
* grammar/typo fixing in commit message (there are surely still some errors)
* a real LRM state, gives a bit nicer handling and more clear code,
ha-manager status shows this then nicely too
* fixed a single small issue in the next_state_started which made all existing
test working as is (very nice)
* added a slight hack to keep the manager working even if it's the one going
into maintenance, we could do a different handling but it doesn't matters
much - the LRM goes into maintenance but is still heartbeating, and the CRM
is a differnt thing anyway, so this can work just fine. Maybe it could be
implemented a bit nicer, though.
* more tests
src/PVE/HA/LRM.pm | 59 ++++++++++++++++++-
src/PVE/HA/Manager.pm | 19 +++---
src/PVE/HA/NodeStatus.pm | 19 +++++-
src/test/test-shutdown-policy3/cmdlist | 4 ++
src/test/test-shutdown-policy3/datacenter.cfg | 5 ++
.../test-shutdown-policy3/hardware_status | 5 ++
src/test/test-shutdown-policy3/log.expect | 59 +++++++++++++++++++
src/test/test-shutdown-policy3/manager_status | 1 +
src/test/test-shutdown-policy3/service_config | 4 ++
src/test/test-shutdown-policy4/cmdlist | 4 ++
src/test/test-shutdown-policy4/datacenter.cfg | 5 ++
.../test-shutdown-policy4/hardware_status | 5 ++
src/test/test-shutdown-policy4/log.expect | 55 +++++++++++++++++
src/test/test-shutdown-policy4/manager_status | 1 +
src/test/test-shutdown-policy4/service_config | 4 ++
src/test/test-shutdown-policy5/cmdlist | 4 ++
src/test/test-shutdown-policy5/datacenter.cfg | 5 ++
.../test-shutdown-policy5/hardware_status | 5 ++
src/test/test-shutdown-policy5/log.expect | 58 ++++++++++++++++++
src/test/test-shutdown-policy5/manager_status | 1 +
src/test/test-shutdown-policy5/service_config | 4 ++
21 files changed, 314 insertions(+), 12 deletions(-)
create mode 100644 src/test/test-shutdown-policy3/cmdlist
create mode 100644 src/test/test-shutdown-policy3/datacenter.cfg
create mode 100644 src/test/test-shutdown-policy3/hardware_status
create mode 100644 src/test/test-shutdown-policy3/log.expect
create mode 100644 src/test/test-shutdown-policy3/manager_status
create mode 100644 src/test/test-shutdown-policy3/service_config
create mode 100644 src/test/test-shutdown-policy4/cmdlist
create mode 100644 src/test/test-shutdown-policy4/datacenter.cfg
create mode 100644 src/test/test-shutdown-policy4/hardware_status
create mode 100644 src/test/test-shutdown-policy4/log.expect
create mode 100644 src/test/test-shutdown-policy4/manager_status
create mode 100644 src/test/test-shutdown-policy4/service_config
create mode 100644 src/test/test-shutdown-policy5/cmdlist
create mode 100644 src/test/test-shutdown-policy5/datacenter.cfg
create mode 100644 src/test/test-shutdown-policy5/hardware_status
create mode 100644 src/test/test-shutdown-policy5/log.expect
create mode 100644 src/test/test-shutdown-policy5/manager_status
create mode 100644 src/test/test-shutdown-policy5/service_config
diff --git a/src/PVE/HA/LRM.pm b/src/PVE/HA/LRM.pm
index 5b986a4..ad1e1cb 100644
--- a/src/PVE/HA/LRM.pm
+++ b/src/PVE/HA/LRM.pm
@@ -16,6 +16,7 @@ use PVE::HA::Resources;
my $valid_states = {
wait_for_agent_lock => "waiting for agent lock",
active => "got agent_lock",
+ maintenance => "going into maintenance",
lost_agent_lock => "lost agent_lock",
};
@@ -61,18 +62,27 @@ sub shutdown_request {
}
my $freeze_all;
+ my $maintenance;
if ($shutdown_policy eq 'conditional') {
$freeze_all = $reboot;
} elsif ($shutdown_policy eq 'freeze') {
$freeze_all = 1;
} elsif ($shutdown_policy eq 'failover') {
$freeze_all = 0;
+ } elsif ($shutdown_policy eq 'migrate') {
+ $maintenance = 1;
} else {
$haenv->log('err', "unknown shutdown policy '$shutdown_policy', fall back to conditional");
$freeze_all = $reboot;
}
- if ($shutdown) {
+ if ($maintenance) {
+ # we get marked as unaivalable by the manager, then all services will
+ # be migrated away, we'll still have the same "can we exit" clause than
+ # a normal shutdown -> no running service on this node
+ # FIXME: after X minutes, add shutdown command for remaining services,
+ # e.g., if they have no alternative node???
+ } elsif ($shutdown) {
# *always* queue stop jobs for all services if the node shuts down,
# independent if it's a reboot or a poweroff, else we may corrupt
# services or hinder node shutdown
@@ -89,7 +99,10 @@ sub shutdown_request {
if ($shutdown) {
my $shutdown_type = $reboot ? 'reboot' : 'shutdown';
- if ($freeze_all) {
+ if ($maintenance) {
+ $haenv->log('info', "$shutdown_type LRM, doing maintenance, removing this node from active list");
+ $self->{mode} = 'maintenance';
+ } elsif ($freeze_all) {
$haenv->log('info', "$shutdown_type LRM, stop and freeze all services");
$self->{mode} = 'restart';
} else {
@@ -300,6 +313,16 @@ sub work {
$self->set_local_status({ state => 'lost_agent_lock'});
} elsif (!$self->get_protected_ha_agent_lock()) {
$self->set_local_status({ state => 'lost_agent_lock'});
+ } elsif ($self->{mode} eq 'maintenance') {
+ $self->set_local_status({ state => 'maintenance'});
+ }
+ } elsif ($state eq 'maintenance') {
+
+ if ($fence_request) {
+ $haenv->log('err', "node need to be fenced during maintenance mode - releasing agent_lock\n");
+ $self->set_local_status({ state => 'lost_agent_lock'});
+ } elsif (!$self->get_protected_ha_agent_lock()) {
+ $self->set_local_status({ state => 'lost_agent_lock'});
}
}
@@ -432,6 +455,38 @@ sub work {
$haenv->sleep(5);
+ } elsif ($state eq 'maintenance') {
+
+ my $startime = $haenv->get_time();
+ return if !$self->update_service_status();
+
+ # wait until all active services moved away
+ my $service_count = $self->active_service_count();
+
+ my $exit_lrm = 0;
+
+ if ($self->{shutdown_request}) {
+ if ($service_count == 0 && $self->run_workers() == 0) {
+ if ($self->{ha_agent_wd}) {
+ $haenv->watchdog_close($self->{ha_agent_wd});
+ delete $self->{ha_agent_wd};
+ }
+
+ $exit_lrm = 1;
+
+ # restart with no or freezed services, release the lock
+ $haenv->release_ha_agent_lock();
+ }
+ }
+
+ $self->manage_resources() if !$exit_lrm;
+
+ $self->update_lrm_status();
+
+ return 0 if $exit_lrm;
+
+ $haenv->sleep_until($startime + 5);
+
} else {
die "got unexpected status '$state'\n";
diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
index 5137de8..f91b852 100644
--- a/src/PVE/HA/Manager.pm
+++ b/src/PVE/HA/Manager.pm
@@ -361,15 +361,16 @@ sub manage {
my ($haenv, $ms, $ns, $ss) = ($self->{haenv}, $self->{ms}, $self->{ns}, $self->{ss});
- $ns->update($haenv->get_node_info());
-
- if (!$ns->node_is_online($haenv->nodename())) {
- $haenv->log('info', "master seems offline");
- return;
- }
-
+ my ($node_info) = $haenv->get_node_info();
my ($lrm_results, $lrm_modes) = $self->read_lrm_status();
+ $ns->update($node_info, $lrm_modes);
+
+ if (!$ns->node_is_operational($haenv->nodename())) {
+ $haenv->log('info', "master seems offline");
+ return;
+ }
+
my $sc = $haenv->read_service_config();
$self->{groups} = $haenv->read_group_config(); # update
@@ -627,7 +628,9 @@ sub next_state_started {
if ($ns->node_is_offline_delayed($sd->{node})) {
&$change_service_state($self, $sid, 'fence');
}
- return;
+ if ($ns->get_node_state($sd->{node}) ne 'maintenance') {
+ return;
+ }
}
if ($cd->{state} eq 'disabled' || $cd->{state} eq 'stopped') {
diff --git a/src/PVE/HA/NodeStatus.pm b/src/PVE/HA/NodeStatus.pm
index 8784110..9d58fa4 100644
--- a/src/PVE/HA/NodeStatus.pm
+++ b/src/PVE/HA/NodeStatus.pm
@@ -24,6 +24,7 @@ sub new {
# possible node state:
my $valid_node_states = {
online => "node online and member of quorate partition",
+ maintenance => "node is a member of quorate partition but currently not able to do work",
unknown => "not member of quorate partition, but possibly still running",
fence => "node needs to be fenced",
gone => "node vanished from cluster members list, possibly deleted"
@@ -38,6 +39,11 @@ sub get_node_state {
return $self->{status}->{$node};
}
+sub node_is_operational {
+ my ($self, $node) = @_;
+ return $self->node_is_online($node) || $self->get_node_state($node) eq 'maintenance';
+}
+
sub node_is_online {
my ($self, $node) = @_;
@@ -117,12 +123,13 @@ my $set_node_state = sub {
};
sub update {
- my ($self, $node_info) = @_;
+ my ($self, $node_info, $lrm_modes) = @_;
my $haenv = $self->{haenv};
foreach my $node (sort keys %$node_info) {
my $d = $node_info->{$node};
+ my $lrm_mode = $lrm_modes->{$node} // 'unkown';
next if !$d->{online};
# record last time the node was online (required to implement fence delay)
@@ -131,11 +138,19 @@ sub update {
my $state = $self->get_node_state($node);
if ($state eq 'online') {
+ if ($lrm_mode eq 'maintenance') {
+ #$haenv->log('info', "update node state maintance");
+ $set_node_state->($self, $node, 'maintenance');
+ }
# &$set_node_state($self, $node, 'online');
} elsif ($state eq 'unknown' || $state eq 'gone') {
&$set_node_state($self, $node, 'online');
} elsif ($state eq 'fence') {
# do nothing, wait until fenced
+ } elsif ($state eq 'maintenance') {
+ if ($lrm_mode ne 'maintenance') {
+ $set_node_state->($self, $node, 'online');
+ }
} else {
die "detected unknown node state '$state";
}
@@ -149,7 +164,7 @@ sub update {
# node is not inside quorate partition, possibly not active
- if ($state eq 'online') {
+ if ($state eq 'online' || $state eq 'maintenance') {
&$set_node_state($self, $node, 'unknown');
} elsif ($state eq 'unknown') {
diff --git a/src/test/test-shutdown-policy3/cmdlist b/src/test/test-shutdown-policy3/cmdlist
new file mode 100644
index 0000000..8558351
--- /dev/null
+++ b/src/test/test-shutdown-policy3/cmdlist
@@ -0,0 +1,4 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on"],
+ [ "reboot node3" ]
+]
diff --git a/src/test/test-shutdown-policy3/datacenter.cfg b/src/test/test-shutdown-policy3/datacenter.cfg
new file mode 100644
index 0000000..de0bf81
--- /dev/null
+++ b/src/test/test-shutdown-policy3/datacenter.cfg
@@ -0,0 +1,5 @@
+{
+ "ha": {
+ "shutdown_policy": "migrate"
+ }
+}
diff --git a/src/test/test-shutdown-policy3/hardware_status b/src/test/test-shutdown-policy3/hardware_status
new file mode 100644
index 0000000..451beb1
--- /dev/null
+++ b/src/test/test-shutdown-policy3/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-shutdown-policy3/log.expect b/src/test/test-shutdown-policy3/log.expect
new file mode 100644
index 0000000..6ecf211
--- /dev/null
+++ b/src/test/test-shutdown-policy3/log.expect
@@ -0,0 +1,59 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'ct:102' on node 'node3'
+info 20 node1/crm: adding new service 'vm:103' on node 'node3'
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service ct:102
+info 25 node3/lrm: service status ct:102 started
+info 25 node3/lrm: starting service vm:103
+info 25 node3/lrm: service status vm:103 started
+info 120 cmdlist: execute reboot node3
+info 120 node3/lrm: got shutdown request with shutdown policy 'migrate'
+info 120 node3/lrm: reboot LRM, doing maintenance, removing this node from active list
+info 120 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+info 120 node1/crm: relocate service 'ct:102' to node 'node1'
+info 120 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node3, target = node1)
+info 120 node1/crm: migrate service 'vm:103' to node 'node1' (running)
+info 120 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 125 node3/lrm: status change active => maintenance
+info 125 node3/lrm: service ct:102 - start relocate to node 'node1'
+info 125 node3/lrm: stopping service ct:102 (relocate)
+info 125 node3/lrm: service status ct:102 stopped
+info 125 node3/lrm: service ct:102 - end relocate to node 'node1'
+info 125 node3/lrm: service vm:103 - start migrate to node 'node1'
+info 125 node3/lrm: service vm:103 - end migrate to node 'node1'
+info 140 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node1)
+info 140 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node1)
+info 141 node1/lrm: got lock 'ha_agent_node1_lock'
+info 141 node1/lrm: status change wait_for_agent_lock => active
+info 141 node1/lrm: starting service ct:102
+info 141 node1/lrm: service status ct:102 started
+info 141 node1/lrm: starting service vm:103
+info 141 node1/lrm: service status vm:103 started
+info 146 node3/lrm: exit (loop end)
+info 146 reboot: execute crm node3 stop
+info 145 node3/crm: server received shutdown request
+info 165 node3/crm: exit (loop end)
+info 165 reboot: execute power node3 off
+info 165 reboot: execute power node3 on
+info 165 node3/crm: status change startup => wait_for_quorum
+info 160 node3/lrm: status change startup => wait_for_agent_lock
+info 180 node1/crm: node 'node3': state changed from 'maintenance' => 'online'
+info 184 node3/crm: status change wait_for_quorum => slave
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-shutdown-policy3/manager_status b/src/test/test-shutdown-policy3/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-shutdown-policy3/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-shutdown-policy3/service_config b/src/test/test-shutdown-policy3/service_config
new file mode 100644
index 0000000..8ee94b5
--- /dev/null
+++ b/src/test/test-shutdown-policy3/service_config
@@ -0,0 +1,4 @@
+{
+ "vm:103": { "node": "node3", "state": "enabled" },
+ "ct:102": { "node": "node3", "state": "enabled" }
+}
diff --git a/src/test/test-shutdown-policy4/cmdlist b/src/test/test-shutdown-policy4/cmdlist
new file mode 100644
index 0000000..a86b9e2
--- /dev/null
+++ b/src/test/test-shutdown-policy4/cmdlist
@@ -0,0 +1,4 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on"],
+ [ "shutdown node3" ]
+]
diff --git a/src/test/test-shutdown-policy4/datacenter.cfg b/src/test/test-shutdown-policy4/datacenter.cfg
new file mode 100644
index 0000000..de0bf81
--- /dev/null
+++ b/src/test/test-shutdown-policy4/datacenter.cfg
@@ -0,0 +1,5 @@
+{
+ "ha": {
+ "shutdown_policy": "migrate"
+ }
+}
diff --git a/src/test/test-shutdown-policy4/hardware_status b/src/test/test-shutdown-policy4/hardware_status
new file mode 100644
index 0000000..451beb1
--- /dev/null
+++ b/src/test/test-shutdown-policy4/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-shutdown-policy4/log.expect b/src/test/test-shutdown-policy4/log.expect
new file mode 100644
index 0000000..2e31059
--- /dev/null
+++ b/src/test/test-shutdown-policy4/log.expect
@@ -0,0 +1,55 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'ct:102' on node 'node3'
+info 20 node1/crm: adding new service 'vm:103' on node 'node3'
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 25 node3/lrm: got lock 'ha_agent_node3_lock'
+info 25 node3/lrm: status change wait_for_agent_lock => active
+info 25 node3/lrm: starting service ct:102
+info 25 node3/lrm: service status ct:102 started
+info 25 node3/lrm: starting service vm:103
+info 25 node3/lrm: service status vm:103 started
+info 120 cmdlist: execute shutdown node3
+info 120 node3/lrm: got shutdown request with shutdown policy 'migrate'
+info 120 node3/lrm: shutdown LRM, doing maintenance, removing this node from active list
+info 120 node1/crm: node 'node3': state changed from 'online' => 'maintenance'
+info 120 node1/crm: relocate service 'ct:102' to node 'node1'
+info 120 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node3, target = node1)
+info 120 node1/crm: migrate service 'vm:103' to node 'node1' (running)
+info 120 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node3, target = node1)
+info 125 node3/lrm: status change active => maintenance
+info 125 node3/lrm: service ct:102 - start relocate to node 'node1'
+info 125 node3/lrm: stopping service ct:102 (relocate)
+info 125 node3/lrm: service status ct:102 stopped
+info 125 node3/lrm: service ct:102 - end relocate to node 'node1'
+info 125 node3/lrm: service vm:103 - start migrate to node 'node1'
+info 125 node3/lrm: service vm:103 - end migrate to node 'node1'
+info 140 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node1)
+info 140 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node1)
+info 141 node1/lrm: got lock 'ha_agent_node1_lock'
+info 141 node1/lrm: status change wait_for_agent_lock => active
+info 141 node1/lrm: starting service ct:102
+info 141 node1/lrm: service status ct:102 started
+info 141 node1/lrm: starting service vm:103
+info 141 node1/lrm: service status vm:103 started
+info 146 node3/lrm: exit (loop end)
+info 146 shutdown: execute crm node3 stop
+info 145 node3/crm: server received shutdown request
+info 165 node3/crm: exit (loop end)
+info 165 shutdown: execute power node3 off
+info 180 node1/crm: node 'node3': state changed from 'maintenance' => 'unknown'
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-shutdown-policy4/manager_status b/src/test/test-shutdown-policy4/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-shutdown-policy4/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-shutdown-policy4/service_config b/src/test/test-shutdown-policy4/service_config
new file mode 100644
index 0000000..8ee94b5
--- /dev/null
+++ b/src/test/test-shutdown-policy4/service_config
@@ -0,0 +1,4 @@
+{
+ "vm:103": { "node": "node3", "state": "enabled" },
+ "ct:102": { "node": "node3", "state": "enabled" }
+}
diff --git a/src/test/test-shutdown-policy5/cmdlist b/src/test/test-shutdown-policy5/cmdlist
new file mode 100644
index 0000000..e84297f
--- /dev/null
+++ b/src/test/test-shutdown-policy5/cmdlist
@@ -0,0 +1,4 @@
+[
+ [ "power node1 on", "power node2 on", "power node3 on"],
+ [ "shutdown node1" ]
+]
diff --git a/src/test/test-shutdown-policy5/datacenter.cfg b/src/test/test-shutdown-policy5/datacenter.cfg
new file mode 100644
index 0000000..de0bf81
--- /dev/null
+++ b/src/test/test-shutdown-policy5/datacenter.cfg
@@ -0,0 +1,5 @@
+{
+ "ha": {
+ "shutdown_policy": "migrate"
+ }
+}
diff --git a/src/test/test-shutdown-policy5/hardware_status b/src/test/test-shutdown-policy5/hardware_status
new file mode 100644
index 0000000..451beb1
--- /dev/null
+++ b/src/test/test-shutdown-policy5/hardware_status
@@ -0,0 +1,5 @@
+{
+ "node1": { "power": "off", "network": "off" },
+ "node2": { "power": "off", "network": "off" },
+ "node3": { "power": "off", "network": "off" }
+}
diff --git a/src/test/test-shutdown-policy5/log.expect b/src/test/test-shutdown-policy5/log.expect
new file mode 100644
index 0000000..15f67c2
--- /dev/null
+++ b/src/test/test-shutdown-policy5/log.expect
@@ -0,0 +1,58 @@
+info 0 hardware: starting simulation
+info 20 cmdlist: execute power node1 on
+info 20 node1/crm: status change startup => wait_for_quorum
+info 20 node1/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node2 on
+info 20 node2/crm: status change startup => wait_for_quorum
+info 20 node2/lrm: status change startup => wait_for_agent_lock
+info 20 cmdlist: execute power node3 on
+info 20 node3/crm: status change startup => wait_for_quorum
+info 20 node3/lrm: status change startup => wait_for_agent_lock
+info 20 node1/crm: got lock 'ha_manager_lock'
+info 20 node1/crm: status change wait_for_quorum => master
+info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
+info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
+info 20 node1/crm: adding new service 'ct:102' on node 'node1'
+info 20 node1/crm: adding new service 'vm:103' on node 'node1'
+info 21 node1/lrm: got lock 'ha_agent_node1_lock'
+info 21 node1/lrm: status change wait_for_agent_lock => active
+info 21 node1/lrm: starting service ct:102
+info 21 node1/lrm: service status ct:102 started
+info 21 node1/lrm: starting service vm:103
+info 21 node1/lrm: service status vm:103 started
+info 22 node2/crm: status change wait_for_quorum => slave
+info 24 node3/crm: status change wait_for_quorum => slave
+info 120 cmdlist: execute shutdown node1
+info 120 node1/lrm: got shutdown request with shutdown policy 'migrate'
+info 120 node1/lrm: shutdown LRM, doing maintenance, removing this node from active list
+info 120 node1/crm: node 'node1': state changed from 'online' => 'maintenance'
+info 120 node1/crm: relocate service 'ct:102' to node 'node2'
+info 120 node1/crm: service 'ct:102': state changed from 'started' to 'relocate' (node = node1, target = node2)
+info 120 node1/crm: migrate service 'vm:103' to node 'node2' (running)
+info 120 node1/crm: service 'vm:103': state changed from 'started' to 'migrate' (node = node1, target = node2)
+info 121 node1/lrm: status change active => maintenance
+info 121 node1/lrm: service ct:102 - start relocate to node 'node2'
+info 121 node1/lrm: stopping service ct:102 (relocate)
+info 121 node1/lrm: service status ct:102 stopped
+info 121 node1/lrm: service ct:102 - end relocate to node 'node2'
+info 121 node1/lrm: service vm:103 - start migrate to node 'node2'
+info 121 node1/lrm: service vm:103 - end migrate to node 'node2'
+info 140 node1/crm: service 'ct:102': state changed from 'relocate' to 'started' (node = node2)
+info 140 node1/crm: service 'vm:103': state changed from 'migrate' to 'started' (node = node2)
+info 142 node1/lrm: exit (loop end)
+info 142 shutdown: execute crm node1 stop
+info 141 node1/crm: server received shutdown request
+info 143 node2/lrm: got lock 'ha_agent_node2_lock'
+info 143 node2/lrm: status change wait_for_agent_lock => active
+info 143 node2/lrm: starting service ct:102
+info 143 node2/lrm: service status ct:102 started
+info 143 node2/lrm: starting service vm:103
+info 143 node2/lrm: service status vm:103 started
+info 160 node1/crm: voluntary release CRM lock
+info 161 node1/crm: exit (loop end)
+info 161 shutdown: execute power node1 off
+info 161 node2/crm: got lock 'ha_manager_lock'
+info 161 node2/crm: status change slave => master
+info 161 node2/crm: node 'node1': state changed from 'maintenance' => 'unknown'
+info 720 hardware: exit simulation - done
diff --git a/src/test/test-shutdown-policy5/manager_status b/src/test/test-shutdown-policy5/manager_status
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/src/test/test-shutdown-policy5/manager_status
@@ -0,0 +1 @@
+{}
diff --git a/src/test/test-shutdown-policy5/service_config b/src/test/test-shutdown-policy5/service_config
new file mode 100644
index 0000000..d37e5b3
--- /dev/null
+++ b/src/test/test-shutdown-policy5/service_config
@@ -0,0 +1,4 @@
+{
+ "vm:103": { "node": "node1", "state": "enabled" },
+ "ct:102": { "node": "node1", "state": "enabled" }
+}
--
2.20.1
More information about the pve-devel
mailing list