[PVE-User] HA Fencing

[Thomas Lamprecht]

On 12/05/2017 10:25 AM, Mark Adams wrote:
> On 5 December 2017 at 08:52, Thomas Lamprecht <t.lamprecht at proxmox.com>
> wrote:
>> On 12/04/2017 07:51 PM, Mark Adams wrote:
>>> On 17 November 2017 at 10:55, Thomas Lamprecht <t.lamprecht at proxmox.com>
>>>> wrote:
>>>> On 11/16/2017 07:20 PM, Mark Adams wrote:
>>>>> Hi all,
>>>>>
>>>>> It looks like in newer versions of proxmox, the only fencing type
>>>>> advised is watchdog. Is that the case?
>>>>>
>>>>
>>>> Yes, since PVE 4.0 watchdog fencing is the norm.
>>>> There is a patch set of mine which implements the use of external fence
>>>> device, but it has seen no review. I should probably dust it up, look
>>>> over it and re send it again, it's about time we finally get this feature.
>>>>
>>>
>>> I think you should definitely get this feature in - I would even say it
>>> is necessary for an enterprise HA setup?
>>>
>>
>> Not really a necessary. Watchdog based fencing is no less secure than
>> traditional
>> fence devices. In fact, as there's much less to configure, and much less
>> protocols
>> between them I'd say its the opposite. I.e., you do not must fire up a
>> command
>> over TCP/IP to fence a node to a device. Here are multiple problem points,
>> Link problems, high load problems delaying fencing, fence devices whit a
>> setup not
>> well tested, at least not under failure conditions, ...
>> A watchdog, which triggers as soon as the node did not pulled it up,
>> independent
>> of link failures, cluster load is here the safer bet. They are often the
>> norm in
>> highly-secure critical embedded systems to, not without reason.
>> It's the difference between a emergency shutdown button and a
>> dead-man-switch.
>>
> 
> AFAIK It's the only way to know for sure, that your server has actually
> been fenced when it is not contactable by other means, For instance some
> network issue on the host.
> 

Both the Fence devices and a Watchdog can be possibly "wrong", thus we
*always* acquire a cluster wide lock to ensure that we only do anything
HA related if we're in the quorate partition and in an OK state.

With the watchdog you know that it released all resources for sure if the
node went out of the quorate partition for a certain time.
We then try to acquire the nodes local resource manager lock, only then
we start recovery of the fenced services. This lock together with the
watchdog guarantees us that we do not access the same resource twice.

Even if the node starts now up OK again it won't get its lock immediately
and thus won't start any HA service. Only once the recovery had been taken
place and completed it can reintegrate in the cluster and do work again.
If you just power it down with a external fence device it always needs
manual intervention, with the watchdog mechanism you won't need that if
the source of the quorum loss was a temporary switch hiccup or similar -
a bit rare but not unheard of.

> Yes the Watchdog on the machine that goes offline should fence itself, but
> still the only way to know for sure that the machine is dead is to power it
> off right?
> 

Not necessarily (see above). Also network fencing is a thing, i.e. cut all
network links related to shared resources (storage, public network, ...)
This allows to investigate the still running, but fenced off, node for the
failure reason - if wished.

> 
>> Maybe you didn't even meant the reliability stand point but that a better
>> best-case SLA could be possible with fence devices?
>>
> 
> This does make a difference too, it could fail over in seconds with faster
> fencing.
> 

Depends a bit on the fencing devices used, I had some experiences where it was
slower than I expected when testing, but yes still a tad faster than the "wait
for the watchdog+lock" approach, though.

cheers,
Thomas

maybe you can find some more information here, if not read already:
https://pve.proxmox.com/pve-docs/chapter-ha-manager.html