[PVE-User] Locking HA during UPS shutdown

admins at telehouse.solutions admins at telehouse.solutions
Thu Mar 10 14:48:53 CET 2022 

That was actually a really BAD ADVICE…. as when node initiate maintenance mode it will try to migrate hosted vms … and eventually ends up in the same Lock loop..
what you really need is to remove started vms from ha-manager, so when the node initiate shutdown it will do firstly do regular shutdown vm per vm.

So, do something like below as first command in your NUT command sequence:

for a in `ha-manager status | grep started|awk '{print $2}'|sed 's/vm://g'`; do ha-manager remove $a;done


> On Mar 10, 2022, at 2:48 PM, admins at telehouse.solutions wrote:
> 
> I don’t remember, search into pvecm and pve[tab][tab] related commands man pages 
> 
>> On Mar 10, 2022, at 2:19 PM, Stefan Radman <stefan.radman at me.com> wrote:
>> 
>> Hi Sto
>> 
>> Thanks for the suggestions.
>> 
>> The second option is what I was looking for.
>> 
>> How do I initiate “pve node maintenance mode”?
>> 
>> The “Node Maintenance” paragraph in the HA documentation is quite brief and does not refer to any command or GUI component.
>> 
>> Thank you
>> 
>> Stefan
>> 
>> 
>>> On Mar 10, 2022, at 14:50, admins at telehouse.solutions <mailto:admins at telehouse.solutions> wrote:
>>> 
>>> Hi, 
>>> 
>>> here are two ideas: shutdown sequence -and- command sequence
>>> 1: shutdown sequence you may achieve when you set NUT’s on each node to only monitor the UPS power, then configure each node to shutdown itself on a different ups power levels, ex: node1 on 15% battery, node2 on 10% battery and so on
>>> 2: you can set a cmd sequence to firstly execute  pve node maintenance mode , and then execute shutdown -> this way HA will not try to migrate vm to node in maintenance, and the chance all nodes to goes into maintenance in exactly same second seems to be not a risk at all.
>>> 
>>> hope thats helpful.
>>> 
>>> Regards,
>>> Sto.
>>> 
>>>> On Mar 10, 2022, at 1:10 PM, Stefan Radman via pve-user <pve-user at lists.proxmox.com <mailto:pve-user at lists.proxmox.com>> wrote:
>>>> 
>>>> 
>>>> From: Stefan Radman <stefan.radman at me.com <mailto:stefan.radman at me.com>>
>>>> Subject: Locking HA during UPS shutdown
>>>> Date: March 10, 2022 at 1:10:09 PM GMT+2
>>>> To: PVE User List <pve-user at pve.proxmox.com <mailto:pve-user at pve.proxmox.com>>
>>>> 
>>>> 
>>>> Hi 
>>>> 
>>>> I am configuring a 3 node PVE cluster with integrated Ceph storage.
>>>> 
>>>> It is powered by 2 UPS that are monitored by NUT (Network UPS Tools).
>>>> 
>>>> HA is configured with 3 groups:
>>>> group pve1 nodes pve1:1,pve2,pve3
>>>> group pve2 nodes pve1,pve2:1,pve3
>>>> group pve3 nodes pve1,pve2,pve3:1
>>>> 
>>>> That will normally place the VMs in each group on the corresponding node, unless that node fails.
>>>> 
>>>> The cluster is configured to migrate VMs away from a node before shutting it down (Cluster=>Options=>HA Settings: shutdown_policy=migrate).
>>>> 
>>>> NUT is configured to shut down the serves once the last of the two UPS is running low on battery.
>>>> 
>>>> My problem:
>>>> When NUT starts shutting down the 3 nodes, HA will first try to live-migrate them to another node.
>>>> That live migration process gets stuck because all the nodes are shutting down simultaneously.
>>>> It seems that the whole process runs into a timeout, finally “powers off” all the VMs and shuts down the nodes.
>>>> 
>>>> My question:
>>>> Is there a way to “lock” or temporarily de-activate HA before shutting down a node to avoid that deadlock?
>>>> 
>>>> Thank you
>>>> 
>>>> Stefan
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> pve-user mailing list
>>>> pve-user at lists.proxmox.com <mailto:pve-user at lists.proxmox.com>
>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user <https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user>
>>> 
>>> 
>>> Best Regards,
>>> 
>>> Stoyan Stoyanov Sto | Solutions Manager
>>> | Telehouse.Solutions | ICT Department
>>> | phone/viber:  +359 894774934 <tel:+359 894774934>
>>> | telegram:  @prostoSto <https://mysignature.io/redirect/skype:prosto.sto?chat>
>>> | skype:  prosto.sto <https://mysignature.io/redirect/skype:prosto.sto?chat>
>>> | email:  sto at telehouse.solutions <mailto:sto at telehouse.solutions>
>>> | website: www.telehouse.solutions <https://mysig.io/MTRmMTg>
>>> | address: Telepoint #2, Sofia, Bulgaria
>>> <https://mysignature.io/editor/?utm_source=freepixel><356841.png>
>>> 
>>> <https://mysig.io/ZDNkNWY>
>>> Save paper. Don’t print
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Best Regards,
>>> 
>>> Stoyan Stoyanov Sto | Solutions Manager
>>> | Telehouse.Solutions | ICT Department
>>> | phone/viber:  +359 894774934 <tel:+359 894774934>
>>> | telegram:  @prostoSto <https://mysignature.io/redirect/skype:prosto.sto?chat>
>>> | skype:  prosto.sto <https://mysignature.io/redirect/skype:prosto.sto?chat>
>>> | email:  sto at telehouse.solutions <mailto:sto at telehouse.solutions>
>>> | website: www.telehouse.solutions <https://mysig.io/MTRmMTg>
>>> | address: Telepoint #2, Sofia, Bulgaria
>>> <https://mysignature.io/editor/?utm_source=freepixel><356841.png>
>>> 
>>> <https://mysig.io/ZDNkNWY>
>>> Save paper. Don’t print
>> 
> 
> 
> Best Regards,
> 
> Stoyan Stoyanov Sto | Solutions Manager
> | Telehouse.Solutions | ICT Department
> | phone/viber:  +359 894774934 <tel:+359 894774934>
> | telegram:  @prostoSto <https://mysignature.io/redirect/skype:prosto.sto?chat>
> | skype:  prosto.sto <https://mysignature.io/redirect/skype:prosto.sto?chat>
> | email:  sto at telehouse.solutions <mailto:sto at telehouse.solutions>
> | website: www.telehouse.solutions <https://mysig.io/MTRmMTg>
> | address: Telepoint #2, Sofia, Bulgaria
> <https://mysignature.io/editor/?utm_source=freepixel>
> 
> <https://mysig.io/ZDNkNWY>
> Save paper. Don’t print
> 
> 
> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user


Best Regards,

Stoyan Stoyanov Sto | Solutions Manager
| Telehouse.Solutions | ICT Department
| phone/viber:  +359 894774934 <tel:+359 894774934>
| telegram:  @prostoSto <https://mysignature.io/redirect/skype:prosto.sto?chat>
| skype:  prosto.sto <https://mysignature.io/redirect/skype:prosto.sto?chat>
| email:  sto at telehouse.solutions <mailto:sto at telehouse.solutions>
| website: www.telehouse.solutions <https://mysig.io/MTRmMTg>
| address: Telepoint #2, Sofia, Bulgaria
 <https://mysignature.io/editor/?utm_source=freepixel>

 <https://mysig.io/ZDNkNWY>
Save paper. Don’t print

More information about the pve-user mailing list