[pve-devel] [PATCH ha-manager 2/4] allow use of external fencing devices
Fabian Grünbichler
f.gruenbichler at proxmox.com
Fri Sep 6 11:59:46 CEST 2019
On March 27, 2019 5:42 pm, Thomas Lamprecht wrote:
> A node now can be fenced with the use of external hardware fence
> devices.
> Those device can be configured at /etc/pve/ha/fence.cfg
> also the fencing option in the datacenter configuration file must
> be set to either 'hardware' or 'both', else configured devices
> will *not* be used.
>
> If hardware is selected as mode a valid device config *must* be
> made, the fencing will not be marked as successful even if the CRM
> could theoretical acquire the lock from the failed node!
> This is done as some setups may require a HW fence agent to cut the
> node off and where a watchdog which resets the node may be
> dangerous.
this is worded a big confusingly, but the gist is:
mode => hardware means the fence device must be configured correctly and
work for fencing to succeed, mode => both means a fallback to regular
CRM fencing (which means that 'both' is more like
'fallback_to_watchdog' or 'either') ?
>
> We always *must* acquire the log before we can mark the failed node
> as fenced and place it into the 'unknown' state and recover its
> services.
>
> The CRM bails out in case of an lost manager lock event where
> $manager->cleanup() gets called.
> There we kill all remaining open fence processes, if any,
> and reset the fence status.
>
> The currents masters manager class processes the running fencing
> jobs, this means picking up finished fence workers and evaluating
> their result.
>
> Now regressions test with faked virtual HW fence devices are also
> possible.
> The current virtual devices succeed always, this will be changed
> in a future patch to allow testing of more (dangerous) corner cases.
>
> Device can be configured in the testdir/fence.cfg file and follow
> the exactly same format as the real ones (see man dlm.conf)
some nits and questions inline
missing S-O-B
> ---
> src/PVE/HA/Manager.pm | 4 +-
> src/PVE/HA/NodeStatus.pm | 54 +++++++++++-
> src/test/test-hw-fence1/README | 1 +
> src/test/test-hw-fence1/cmdlist | 4 +
> src/test/test-hw-fence1/fence.cfg | 6 ++
> src/test/test-hw-fence1/hardware_status | 5 ++
> src/test/test-hw-fence1/log.expect | 53 ++++++++++++
> src/test/test-hw-fence1/manager_status | 1 +
> src/test/test-hw-fence1/service_config | 5 ++
> src/test/test-hw-fence2/README | 3 +
> src/test/test-hw-fence2/cmdlist | 5 ++
> src/test/test-hw-fence2/fence.cfg | 8 ++
> src/test/test-hw-fence2/hardware_status | 7 ++
> src/test/test-hw-fence2/log.expect | 110 ++++++++++++++++++++++++
> src/test/test-hw-fence2/manager_status | 1 +
> src/test/test-hw-fence2/service_config | 10 +++
> src/test/test-hw-fence3/README | 5 ++
> src/test/test-hw-fence3/cmdlist | 4 +
> src/test/test-hw-fence3/fence.cfg | 17 ++++
> src/test/test-hw-fence3/hardware_status | 5 ++
> src/test/test-hw-fence3/log.expect | 57 ++++++++++++
> src/test/test-hw-fence3/manager_status | 1 +
> src/test/test-hw-fence3/service_config | 5 ++
> 23 files changed, 369 insertions(+), 2 deletions(-)
> create mode 100644 src/test/test-hw-fence1/README
> create mode 100644 src/test/test-hw-fence1/cmdlist
> create mode 100644 src/test/test-hw-fence1/fence.cfg
> create mode 100644 src/test/test-hw-fence1/hardware_status
> create mode 100644 src/test/test-hw-fence1/log.expect
> create mode 100644 src/test/test-hw-fence1/manager_status
> create mode 100644 src/test/test-hw-fence1/service_config
> create mode 100644 src/test/test-hw-fence2/README
> create mode 100644 src/test/test-hw-fence2/cmdlist
> create mode 100644 src/test/test-hw-fence2/fence.cfg
> create mode 100644 src/test/test-hw-fence2/hardware_status
> create mode 100644 src/test/test-hw-fence2/log.expect
> create mode 100644 src/test/test-hw-fence2/manager_status
> create mode 100644 src/test/test-hw-fence2/service_config
> create mode 100644 src/test/test-hw-fence3/README
> create mode 100644 src/test/test-hw-fence3/cmdlist
> create mode 100644 src/test/test-hw-fence3/fence.cfg
> create mode 100644 src/test/test-hw-fence3/hardware_status
> create mode 100644 src/test/test-hw-fence3/log.expect
> create mode 100644 src/test/test-hw-fence3/manager_status
> create mode 100644 src/test/test-hw-fence3/service_config
>
> diff --git a/src/PVE/HA/Manager.pm b/src/PVE/HA/Manager.pm
> index a6c9b8e..177dfc2 100644
> --- a/src/PVE/HA/Manager.pm
> +++ b/src/PVE/HA/Manager.pm
> @@ -7,6 +7,7 @@ use Digest::MD5 qw(md5_base64);
> use PVE::Tools;
> use PVE::HA::Tools ':exit_codes';
> use PVE::HA::NodeStatus;
> +use PVE::HA::Fence;
>
> sub new {
> my ($this, $haenv) = @_;
> @@ -32,7 +33,8 @@ sub new {
> sub cleanup {
> my ($self) = @_;
>
> - # todo: ?
> + # reset pending fence jobs and node states
> + $self->{ns}->cleanup();
> }
>
> sub flush_master_status {
> diff --git a/src/PVE/HA/NodeStatus.pm b/src/PVE/HA/NodeStatus.pm
> index 940d903..ca13f2f 100644
> --- a/src/PVE/HA/NodeStatus.pm
> +++ b/src/PVE/HA/NodeStatus.pm
> @@ -2,6 +2,7 @@ package PVE::HA::NodeStatus;
>
> use strict;
> use warnings;
> +use PVE::HA::Fence;
>
> use JSON;
>
> @@ -12,8 +13,11 @@ sub new {
>
> my $class = ref($this) || $this;
>
> + my $fencer = PVE::HA::Fence->new($haenv);
> +
> my $self = bless {
> haenv => $haenv,
> + fencer => $fencer,
> status => $status,
> last_online => {},
> }, $class;
> @@ -206,6 +210,7 @@ sub fence_node {
> my ($self, $node) = @_;
>
> my $haenv = $self->{haenv};
> + my $fencer = $self->{fencer};
>
> my $state = $self->get_node_state($node);
>
> @@ -215,16 +220,63 @@ sub fence_node {
I'd like some sort of comment above/in this sub that makes it clear that
this gets called multiple times for a single fence operation. this is
not obvious when looking at this code, and the order of calls to other
modules is rather confusing otherwise (first checking success, then
starting and processing) ;)
> &$send_fence_state_email($self, 'FENCE', $msg, $node);
> }
>
> - my $success = $haenv->get_ha_agent_lock($node);
> + my ($success, $hw_fence_success) = (0, 0);
> +
> + my $fencing_mode = $haenv->fencing_mode();
> +
> + if ($fencing_mode eq 'hardware' || $fencing_mode eq 'both') {
> +
> + $hw_fence_success = $fencer->is_node_fenced($node);
a comment here that explains what the three values mean would be nice to
have as well:
-1 HW fencing failed
0 HW fencing not yet started or still running
1 HW fencing succeeded
> +
> + # bad fence.cfg or no devices and only hardware fencing configured
why not simply
hardware fencing required but failed
?
> + if ($hw_fence_success < 0 && $fencing_mode eq 'hardware') {
> + $haenv->log('err', "Fencing of node '$node' failed and needs " .
> + "manual intervention!");
> + return 0;
Q1: doesn't this mean that fence_node gets called over and over again
without progress?
> + }
> +
> + if ($hw_fence_success > 0) {
> + # we fenced the node, now we're allowed to "steal" its lock
> + $haenv->log('notice', "fencing of node '$node' succeeded, " .
> + "trying to get its agent lock");
> + # This may only be done after successfully fencing node!
> + $haenv->release_ha_agent_lock($node);
> +
> + } else {
> +
> + # start and process fencing
> + $fencer->run_fence_jobs($node);
incorrect indentation
Q2: is this supposed to be called (again and again until regular fencing
succeeds) if $fencing_mode eq 'both' and $hw_success == -1 ?
Q3: why not return 0 here if $mode eq 'hardware' ? we know we can't
proceed in this iteration of fence_node, and (see below)
> +
> + }
> + }
> +
> + # we *always* need the failed nodes lock, it secures that we are allowed to
> + # recover its services and prevents races, e.g. if it's restarting.
s/nodes/node's/
s/secures/ensures/
(continuation of Q3) we could drop this if then AFAICT, and actually
call this "always" if we reach this point ;)
otherwise it's a bit strange to have a comment stating that we always do
something, and then a conditional for doing it. after reading the code
backwards and forwards, I understand that we still always do it (over
multiple iterations, since either the HW fencing is required and we only
get here if it succeeds, or it is not required then we get here anyway).
if we do yield for $hw_success == 0 above, it's probably a good idea to
re-order the conditional branches to have the ones with yielding return
0 first, or use a helper variable to make it more visible..
> + if ($hw_fence_success || $fencing_mode ne 'hardware' ) {
> + $success = $haenv->get_ha_agent_lock($node);
> + }
>
> if ($success) {
> my $msg = "fencing: acknowledged - got agent lock for node '$node'";
> $haenv->log("info", $msg);
> &$set_node_state($self, $node, 'unknown');
> &$send_fence_state_email($self, 'SUCCEED', $msg, $node);
> + $fencer->kill_and_cleanup_jobs($node) if ($fencing_mode ne 'watchdog');
> }
>
> return $success;
> }
>
> +sub cleanup {
> + my ($self) = @_;
> +
> + my $haenv = $self->{haenv};
> + my $fencer = $self->{fencer};
> +
> + if ($fencer->has_fencing_job($haenv->nodename())) {
> + $haenv->log('notice', "bailing out from running fence jobs");
> + $fencer->kill_and_cleanup_jobs();
> + }
> +}
> +
> 1;
> diff --git a/src/test/test-hw-fence1/README b/src/test/test-hw-fence1/README
> new file mode 100644
> index 0000000..d0dea4b
> --- /dev/null
> +++ b/src/test/test-hw-fence1/README
> @@ -0,0 +1 @@
> +Test failover after single node network failure with HW fence devices.
> diff --git a/src/test/test-hw-fence1/cmdlist b/src/test/test-hw-fence1/cmdlist
> new file mode 100644
> index 0000000..eee0e40
> --- /dev/null
> +++ b/src/test/test-hw-fence1/cmdlist
> @@ -0,0 +1,4 @@
> +[
> + [ "power node1 on", "power node2 on", "power node3 on"],
> + [ "network node3 off" ]
> +]
> diff --git a/src/test/test-hw-fence1/fence.cfg b/src/test/test-hw-fence1/fence.cfg
> new file mode 100644
> index 0000000..0bbe096
> --- /dev/null
> +++ b/src/test/test-hw-fence1/fence.cfg
> @@ -0,0 +1,6 @@
> +# see man dlm.conf
> +device virt fence_virt ip="127.0.0.1"
> +connect virt node=node1 plug=100
> +connect virt node=node2 plug=101
> +connect virt node=node3 plug=102
> +
> diff --git a/src/test/test-hw-fence1/hardware_status b/src/test/test-hw-fence1/hardware_status
> new file mode 100644
> index 0000000..451beb1
> --- /dev/null
> +++ b/src/test/test-hw-fence1/hardware_status
> @@ -0,0 +1,5 @@
> +{
> + "node1": { "power": "off", "network": "off" },
> + "node2": { "power": "off", "network": "off" },
> + "node3": { "power": "off", "network": "off" }
> +}
> diff --git a/src/test/test-hw-fence1/log.expect b/src/test/test-hw-fence1/log.expect
> new file mode 100644
> index 0000000..8cd8a40
> --- /dev/null
> +++ b/src/test/test-hw-fence1/log.expect
> @@ -0,0 +1,53 @@
> +info 0 hardware: starting simulation
> +info 20 cmdlist: execute power node1 on
> +info 20 node1/crm: status change startup => wait_for_quorum
> +info 20 node1/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node2 on
> +info 20 node2/crm: status change startup => wait_for_quorum
> +info 20 node2/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node3 on
> +info 20 node3/crm: status change startup => wait_for_quorum
> +info 20 node3/lrm: status change startup => wait_for_agent_lock
> +info 20 node1/crm: got lock 'ha_manager_lock'
> +info 20 node1/crm: status change wait_for_quorum => master
> +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
> +info 20 node1/crm: adding new service 'vm:101' on node 'node1'
> +info 20 node1/crm: adding new service 'vm:102' on node 'node2'
> +info 20 node1/crm: adding new service 'vm:103' on node 'node3'
> +info 21 node1/lrm: got lock 'ha_agent_node1_lock'
> +info 21 node1/lrm: status change wait_for_agent_lock => active
> +info 21 node1/lrm: starting service vm:101
> +info 21 node1/lrm: service status vm:101 started
> +info 22 node2/crm: status change wait_for_quorum => slave
> +info 23 node2/lrm: got lock 'ha_agent_node2_lock'
> +info 23 node2/lrm: status change wait_for_agent_lock => active
> +info 24 node3/crm: status change wait_for_quorum => slave
> +info 25 node3/lrm: got lock 'ha_agent_node3_lock'
> +info 25 node3/lrm: status change wait_for_agent_lock => active
> +info 25 node3/lrm: starting service vm:103
> +info 25 node3/lrm: service status vm:103 started
> +info 40 node1/crm: service 'vm:102': state changed from 'request_stop' to 'stopped'
> +info 120 cmdlist: execute network node3 off
> +info 120 node1/crm: node 'node3': state changed from 'online' => 'unknown'
> +info 124 node3/crm: status change slave => wait_for_quorum
> +info 125 node3/lrm: status change active => lost_agent_lock
> +info 160 node1/crm: service 'vm:103': state changed from 'started' to 'fence'
> +info 160 node1/crm: node 'node3': state changed from 'unknown' => 'fence'
> +emai 160 node1/crm: FENCE: Try to fence node 'node3'
> +noti 160 node1/crm: Start fencing node 'node3'
> +noti 160 node1/crm: [fence 'node3'] execute cmd: fence_virt --ip=127.0.0.1 --plug=102
> +info 160 fence_virt: execute power node3 off
> +info 160 node3/crm: killed by poweroff
> +info 160 node3/lrm: killed by poweroff
> +noti 180 node1/crm: fencing of node 'node3' succeeded, trying to get its agent lock
> +info 180 node1/crm: got lock 'ha_agent_node3_lock'
> +info 180 node1/crm: fencing: acknowledged - got agent lock for node 'node3'
> +info 180 node1/crm: node 'node3': state changed from 'fence' => 'unknown'
> +emai 180 node1/crm: SUCCEED: fencing: acknowledged - got agent lock for node 'node3'
> +info 180 node1/crm: recover service 'vm:103' from fenced node 'node3' to node 'node2'
> +info 180 node1/crm: service 'vm:103': state changed from 'fence' to 'started' (node = node2)
> +info 183 node2/lrm: starting service vm:103
> +info 183 node2/lrm: service status vm:103 started
> +info 720 hardware: exit simulation - done
> diff --git a/src/test/test-hw-fence1/manager_status b/src/test/test-hw-fence1/manager_status
> new file mode 100644
> index 0000000..0967ef4
> --- /dev/null
> +++ b/src/test/test-hw-fence1/manager_status
> @@ -0,0 +1 @@
> +{}
> diff --git a/src/test/test-hw-fence1/service_config b/src/test/test-hw-fence1/service_config
> new file mode 100644
> index 0000000..70f11d6
> --- /dev/null
> +++ b/src/test/test-hw-fence1/service_config
> @@ -0,0 +1,5 @@
> +{
> + "vm:101": { "node": "node1", "state": "enabled" },
> + "vm:102": { "node": "node2" },
> + "vm:103": { "node": "node3", "state": "enabled" }
> +}
> diff --git a/src/test/test-hw-fence2/README b/src/test/test-hw-fence2/README
> new file mode 100644
> index 0000000..a3814ec
> --- /dev/null
> +++ b/src/test/test-hw-fence2/README
> @@ -0,0 +1,3 @@
> +Test HW fencing and failover after the network of two nodes fails.
> +This test if the HW fence mechanism can cope with multiple nodes failing
> +simultaneously, as long there is still quorum.
> diff --git a/src/test/test-hw-fence2/cmdlist b/src/test/test-hw-fence2/cmdlist
> new file mode 100644
> index 0000000..9a8bb59
> --- /dev/null
> +++ b/src/test/test-hw-fence2/cmdlist
> @@ -0,0 +1,5 @@
> +[
> + [ "power node1 on", "power node2 on", "power node3 on",
> + "power node4 on", "power node5 on"],
> + [ "network node3 off", "network node4 off" ]
> +]
> diff --git a/src/test/test-hw-fence2/fence.cfg b/src/test/test-hw-fence2/fence.cfg
> new file mode 100644
> index 0000000..aedcdc3
> --- /dev/null
> +++ b/src/test/test-hw-fence2/fence.cfg
> @@ -0,0 +1,8 @@
> +# see man dlm.conf
> +device virt fence_virt ip="127.0.0.1"
> +connect virt node=node1 plug=100
> +connect virt node=node2 plug=101
> +connect virt node=node3 plug=102
> +connect virt node=node4 plug=103
> +connect virt node=node5 plug=104
> +
> diff --git a/src/test/test-hw-fence2/hardware_status b/src/test/test-hw-fence2/hardware_status
> new file mode 100644
> index 0000000..7b8e961
> --- /dev/null
> +++ b/src/test/test-hw-fence2/hardware_status
> @@ -0,0 +1,7 @@
> +{
> + "node1": { "power": "off", "network": "off" },
> + "node2": { "power": "off", "network": "off" },
> + "node3": { "power": "off", "network": "off" },
> + "node4": { "power": "off", "network": "off" },
> + "node5": { "power": "off", "network": "off" }
> +}
> diff --git a/src/test/test-hw-fence2/log.expect b/src/test/test-hw-fence2/log.expect
> new file mode 100644
> index 0000000..0eadd78
> --- /dev/null
> +++ b/src/test/test-hw-fence2/log.expect
> @@ -0,0 +1,110 @@
> +info 0 hardware: starting simulation
> +info 20 cmdlist: execute power node1 on
> +info 20 node1/crm: status change startup => wait_for_quorum
> +info 20 node1/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node2 on
> +info 20 node2/crm: status change startup => wait_for_quorum
> +info 20 node2/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node3 on
> +info 20 node3/crm: status change startup => wait_for_quorum
> +info 20 node3/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node4 on
> +info 20 node4/crm: status change startup => wait_for_quorum
> +info 20 node4/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node5 on
> +info 20 node5/crm: status change startup => wait_for_quorum
> +info 20 node5/lrm: status change startup => wait_for_agent_lock
> +info 20 node1/crm: got lock 'ha_manager_lock'
> +info 20 node1/crm: status change wait_for_quorum => master
> +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node4': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node5': state changed from 'unknown' => 'online'
> +info 20 node1/crm: adding new service 'vm:101' on node 'node4'
> +info 20 node1/crm: adding new service 'vm:102' on node 'node3'
> +info 20 node1/crm: adding new service 'vm:103' on node 'node3'
> +info 20 node1/crm: adding new service 'vm:104' on node 'node3'
> +info 20 node1/crm: adding new service 'vm:105' on node 'node4'
> +info 22 node2/crm: status change wait_for_quorum => slave
> +info 24 node3/crm: status change wait_for_quorum => slave
> +info 25 node3/lrm: got lock 'ha_agent_node3_lock'
> +info 25 node3/lrm: status change wait_for_agent_lock => active
> +info 25 node3/lrm: starting service vm:102
> +info 25 node3/lrm: service status vm:102 started
> +info 25 node3/lrm: starting service vm:103
> +info 25 node3/lrm: service status vm:103 started
> +info 25 node3/lrm: starting service vm:104
> +info 25 node3/lrm: service status vm:104 started
> +info 26 node4/crm: status change wait_for_quorum => slave
> +info 27 node4/lrm: got lock 'ha_agent_node4_lock'
> +info 27 node4/lrm: status change wait_for_agent_lock => active
> +info 27 node4/lrm: starting service vm:101
> +info 27 node4/lrm: service status vm:101 started
> +info 27 node4/lrm: starting service vm:105
> +info 27 node4/lrm: service status vm:105 started
> +info 28 node5/crm: status change wait_for_quorum => slave
> +info 120 cmdlist: execute network node3 off
> +info 120 cmdlist: execute network node4 off
> +info 120 node1/crm: node 'node3': state changed from 'online' => 'unknown'
> +info 120 node1/crm: node 'node4': state changed from 'online' => 'unknown'
> +info 124 node3/crm: status change slave => wait_for_quorum
> +info 125 node3/lrm: status change active => lost_agent_lock
> +info 126 node4/crm: status change slave => wait_for_quorum
> +info 127 node4/lrm: status change active => lost_agent_lock
> +info 160 node1/crm: service 'vm:101': state changed from 'started' to 'fence'
> +info 160 node1/crm: service 'vm:102': state changed from 'started' to 'fence'
> +info 160 node1/crm: service 'vm:103': state changed from 'started' to 'fence'
> +info 160 node1/crm: service 'vm:104': state changed from 'started' to 'fence'
> +info 160 node1/crm: service 'vm:105': state changed from 'started' to 'fence'
> +info 160 node1/crm: node 'node4': state changed from 'unknown' => 'fence'
> +emai 160 node1/crm: FENCE: Try to fence node 'node4'
> +noti 160 node1/crm: Start fencing node 'node4'
> +noti 160 node1/crm: [fence 'node4'] execute cmd: fence_virt --ip=127.0.0.1 --plug=103
> +info 160 fence_virt: execute power node4 off
> +info 160 node4/crm: killed by poweroff
> +info 160 node4/lrm: killed by poweroff
> +info 160 node1/crm: node 'node3': state changed from 'unknown' => 'fence'
> +emai 160 node1/crm: FENCE: Try to fence node 'node3'
> +noti 160 node1/crm: Start fencing node 'node3'
> +noti 160 node1/crm: [fence 'node3'] execute cmd: fence_virt --ip=127.0.0.1 --plug=102
> +info 160 fence_virt: execute power node3 off
> +info 160 node3/crm: killed by poweroff
> +info 160 node3/lrm: killed by poweroff
> +noti 160 node1/crm: fencing of node 'node3' succeeded, trying to get its agent lock
> +info 160 node1/crm: got lock 'ha_agent_node3_lock'
> +info 160 node1/crm: fencing: acknowledged - got agent lock for node 'node3'
> +info 160 node1/crm: node 'node3': state changed from 'fence' => 'unknown'
> +emai 160 node1/crm: SUCCEED: fencing: acknowledged - got agent lock for node 'node3'
> +info 160 node1/crm: recover service 'vm:102' from fenced node 'node3' to node 'node1'
> +info 160 node1/crm: service 'vm:102': state changed from 'fence' to 'started' (node = node1)
> +info 160 node1/crm: recover service 'vm:103' from fenced node 'node3' to node 'node2'
> +info 160 node1/crm: service 'vm:103': state changed from 'fence' to 'started' (node = node2)
> +info 160 node1/crm: recover service 'vm:104' from fenced node 'node3' to node 'node5'
> +info 160 node1/crm: service 'vm:104': state changed from 'fence' to 'started' (node = node5)
> +info 161 node1/lrm: got lock 'ha_agent_node1_lock'
> +info 161 node1/lrm: status change wait_for_agent_lock => active
> +info 161 node1/lrm: starting service vm:102
> +info 161 node1/lrm: service status vm:102 started
> +info 163 node2/lrm: got lock 'ha_agent_node2_lock'
> +info 163 node2/lrm: status change wait_for_agent_lock => active
> +info 163 node2/lrm: starting service vm:103
> +info 163 node2/lrm: service status vm:103 started
> +info 165 node5/lrm: got lock 'ha_agent_node5_lock'
> +info 165 node5/lrm: status change wait_for_agent_lock => active
> +info 165 node5/lrm: starting service vm:104
> +info 165 node5/lrm: service status vm:104 started
> +noti 180 node1/crm: fencing of node 'node4' succeeded, trying to get its agent lock
> +info 180 node1/crm: got lock 'ha_agent_node4_lock'
> +info 180 node1/crm: fencing: acknowledged - got agent lock for node 'node4'
> +info 180 node1/crm: node 'node4': state changed from 'fence' => 'unknown'
> +emai 180 node1/crm: SUCCEED: fencing: acknowledged - got agent lock for node 'node4'
> +info 180 node1/crm: recover service 'vm:101' from fenced node 'node4' to node 'node1'
> +info 180 node1/crm: service 'vm:101': state changed from 'fence' to 'started' (node = node1)
> +info 180 node1/crm: recover service 'vm:105' from fenced node 'node4' to node 'node2'
> +info 180 node1/crm: service 'vm:105': state changed from 'fence' to 'started' (node = node2)
> +info 181 node1/lrm: starting service vm:101
> +info 181 node1/lrm: service status vm:101 started
> +info 183 node2/lrm: starting service vm:105
> +info 183 node2/lrm: service status vm:105 started
> +info 720 hardware: exit simulation - done
> diff --git a/src/test/test-hw-fence2/manager_status b/src/test/test-hw-fence2/manager_status
> new file mode 100644
> index 0000000..0967ef4
> --- /dev/null
> +++ b/src/test/test-hw-fence2/manager_status
> @@ -0,0 +1 @@
> +{}
> diff --git a/src/test/test-hw-fence2/service_config b/src/test/test-hw-fence2/service_config
> new file mode 100644
> index 0000000..735e219
> --- /dev/null
> +++ b/src/test/test-hw-fence2/service_config
> @@ -0,0 +1,10 @@
> +{
> + "vm:101": { "node": "node3", "state": "enabled" },
> + "vm:102": { "node": "node3", "state": "enabled" },
> + "vm:103": { "node": "node3", "state": "enabled" },
> + "vm:104": { "node": "node3", "state": "enabled" },
> + "vm:101": { "node": "node3", "state": "enabled" },
> + "vm:101": { "node": "node4", "state": "enabled" },
> + "vm:101": { "node": "node4", "state": "enabled" },
> + "vm:105": { "node": "node4", "state": "enabled" }
> +}
> diff --git a/src/test/test-hw-fence3/README b/src/test/test-hw-fence3/README
> new file mode 100644
> index 0000000..206ee5f
> --- /dev/null
> +++ b/src/test/test-hw-fence3/README
> @@ -0,0 +1,5 @@
> +Test failover after single node network failure with parallel HW fence devices.
> +As the simulated Environment is limited to a single powerplug you will see the
> +"killed by poweroff" only once, more important is that the three fence agents
> +get executed and that the node is fenced only after all three successfully
> +finsihed.
s/finsihed/finished
> diff --git a/src/test/test-hw-fence3/cmdlist b/src/test/test-hw-fence3/cmdlist
> new file mode 100644
> index 0000000..eee0e40
> --- /dev/null
> +++ b/src/test/test-hw-fence3/cmdlist
> @@ -0,0 +1,4 @@
> +[
> + [ "power node1 on", "power node2 on", "power node3 on"],
> + [ "network node3 off" ]
> +]
> diff --git a/src/test/test-hw-fence3/fence.cfg b/src/test/test-hw-fence3/fence.cfg
> new file mode 100644
> index 0000000..60645df
> --- /dev/null
> +++ b/src/test/test-hw-fence3/fence.cfg
> @@ -0,0 +1,17 @@
> +# see man dlm.conf
> +device virt:1 fence_virt ip="127.0.0.1"
> +device virt:2 fence_virt ip="127.0.0.2"
> +device virt:3 fence_virt ip="127.0.0.3"
> +
> +connect virt:1 node=node1 plug=100
> +connect virt:2 node=node1 plug=100
> +connect virt:3 node=node1 plug=100
> +
> +connect virt:1 node=node2 plug=101
> +connect virt:2 node=node2 plug=101
> +connect virt:3 node=node2 plug=101
> +
> +connect virt:1 node=node3 plug=102
> +connect virt:2 node=node3 plug=102
> +connect virt:3 node=node3 plug=102
> +
> diff --git a/src/test/test-hw-fence3/hardware_status b/src/test/test-hw-fence3/hardware_status
> new file mode 100644
> index 0000000..451beb1
> --- /dev/null
> +++ b/src/test/test-hw-fence3/hardware_status
> @@ -0,0 +1,5 @@
> +{
> + "node1": { "power": "off", "network": "off" },
> + "node2": { "power": "off", "network": "off" },
> + "node3": { "power": "off", "network": "off" }
> +}
> diff --git a/src/test/test-hw-fence3/log.expect b/src/test/test-hw-fence3/log.expect
> new file mode 100644
> index 0000000..bb0760b
> --- /dev/null
> +++ b/src/test/test-hw-fence3/log.expect
> @@ -0,0 +1,57 @@
> +info 0 hardware: starting simulation
> +info 20 cmdlist: execute power node1 on
> +info 20 node1/crm: status change startup => wait_for_quorum
> +info 20 node1/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node2 on
> +info 20 node2/crm: status change startup => wait_for_quorum
> +info 20 node2/lrm: status change startup => wait_for_agent_lock
> +info 20 cmdlist: execute power node3 on
> +info 20 node3/crm: status change startup => wait_for_quorum
> +info 20 node3/lrm: status change startup => wait_for_agent_lock
> +info 20 node1/crm: got lock 'ha_manager_lock'
> +info 20 node1/crm: status change wait_for_quorum => master
> +info 20 node1/crm: node 'node1': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node2': state changed from 'unknown' => 'online'
> +info 20 node1/crm: node 'node3': state changed from 'unknown' => 'online'
> +info 20 node1/crm: adding new service 'vm:101' on node 'node1'
> +info 20 node1/crm: adding new service 'vm:102' on node 'node2'
> +info 20 node1/crm: adding new service 'vm:103' on node 'node3'
> +info 21 node1/lrm: got lock 'ha_agent_node1_lock'
> +info 21 node1/lrm: status change wait_for_agent_lock => active
> +info 21 node1/lrm: starting service vm:101
> +info 21 node1/lrm: service status vm:101 started
> +info 22 node2/crm: status change wait_for_quorum => slave
> +info 23 node2/lrm: got lock 'ha_agent_node2_lock'
> +info 23 node2/lrm: status change wait_for_agent_lock => active
> +info 24 node3/crm: status change wait_for_quorum => slave
> +info 25 node3/lrm: got lock 'ha_agent_node3_lock'
> +info 25 node3/lrm: status change wait_for_agent_lock => active
> +info 25 node3/lrm: starting service vm:103
> +info 25 node3/lrm: service status vm:103 started
> +info 40 node1/crm: service 'vm:102': state changed from 'request_stop' to 'stopped'
> +info 120 cmdlist: execute network node3 off
> +info 120 node1/crm: node 'node3': state changed from 'online' => 'unknown'
> +info 124 node3/crm: status change slave => wait_for_quorum
> +info 125 node3/lrm: status change active => lost_agent_lock
> +info 160 node1/crm: service 'vm:103': state changed from 'started' to 'fence'
> +info 160 node1/crm: node 'node3': state changed from 'unknown' => 'fence'
> +emai 160 node1/crm: FENCE: Try to fence node 'node3'
> +noti 160 node1/crm: Start fencing node 'node3'
> +noti 160 node1/crm: [fence 'node3'] execute cmd: fence_virt --ip=127.0.0.1 --plug=102
> +info 160 fence_virt: execute power node3 off
> +info 160 node3/crm: killed by poweroff
> +info 160 node3/lrm: killed by poweroff
> +noti 160 node1/crm: [fence 'node3'] execute cmd: fence_virt --ip=127.0.0.2 --plug=102
> +info 160 fence_virt: execute power node3 off
> +noti 160 node1/crm: [fence 'node3'] execute cmd: fence_virt --ip=127.0.0.3 --plug=102
> +info 160 fence_virt: execute power node3 off
> +noti 180 node1/crm: fencing of node 'node3' succeeded, trying to get its agent lock
> +info 180 node1/crm: got lock 'ha_agent_node3_lock'
> +info 180 node1/crm: fencing: acknowledged - got agent lock for node 'node3'
> +info 180 node1/crm: node 'node3': state changed from 'fence' => 'unknown'
> +emai 180 node1/crm: SUCCEED: fencing: acknowledged - got agent lock for node 'node3'
> +info 180 node1/crm: recover service 'vm:103' from fenced node 'node3' to node 'node2'
> +info 180 node1/crm: service 'vm:103': state changed from 'fence' to 'started' (node = node2)
> +info 183 node2/lrm: starting service vm:103
> +info 183 node2/lrm: service status vm:103 started
> +info 720 hardware: exit simulation - done
> diff --git a/src/test/test-hw-fence3/manager_status b/src/test/test-hw-fence3/manager_status
> new file mode 100644
> index 0000000..0967ef4
> --- /dev/null
> +++ b/src/test/test-hw-fence3/manager_status
> @@ -0,0 +1 @@
> +{}
> diff --git a/src/test/test-hw-fence3/service_config b/src/test/test-hw-fence3/service_config
> new file mode 100644
> index 0000000..70f11d6
> --- /dev/null
> +++ b/src/test/test-hw-fence3/service_config
> @@ -0,0 +1,5 @@
> +{
> + "vm:101": { "node": "node1", "state": "enabled" },
> + "vm:102": { "node": "node2" },
> + "vm:103": { "node": "node3", "state": "enabled" }
> +}
> --
> 2.20.1
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
More information about the pve-devel
mailing list