[pve-devel] HA Migration on shutdown/reboot

[Thomas Lamprecht]

Hi,

On 5/28/19 8:09 AM, Alexandre DERUMIER wrote:
> in our last week training session, students also ask for the same feature,
> 
> with a real problem, when we have 2 hosts in same ha group, with differents priority,
> 
> and user want to do maintenance/reboot, he would like to migrate all vms on other node,
> before reboot. But currently manually migrating vms (with migrate_all for example),
> the HA is migrated them back before the migrate all is finished,
> so it's super difficult to have the node empty for reboot.

I actually planned to dis-allow that, i.e., if a migration to a node is
requested check first if it won't just get migrated immediately away again
and if so then error out.


> 
> Maybe could we first add some kind of "maintenance mode" flag on a node, which mean than vms
> can't be migrated back to it. (we virtually remove it from HA groups until maintenance mode is removed).
> and user could at least migrate them manually.


As Michael Rasmussen noted, nofailback could be enough here, or not?

> 
> 
> 
> For auto migrate on shutdown,
> 
> I think the most difficult part is to how to choose the targetnode where to migrate the vm.
> Maybe could we choose another nodes of a HA group (excluding the node with maintenance mode)

For HA this is not to hard, I would just take it out from the "target node"
calculation, so the HA stack choses automatically based on node's hosted
HA (running, IIRC) service count.

> 
> Another thing which could be great (even with HA), could be to auto choose targetnode based on cpu/ram current usage,
> like  "migrate ... --targetnode  node_with_lower_ram,  node_with_lower_cpu."

This I'd not add initially, but yes, as leveling out based on service count
only works if the VMs have roughly the same load, and not many non-HA VMs
are in the cluster "skewing" the load, we'll want to add something like this
afterwards...


> 
> ----- Mail original -----
> De: "Bastian Sebode" <b.sebode at linet-services.de>
> À: "pve-devel" <pve-devel at pve.proxmox.com>
> Envoyé: Mardi 28 Mai 2019 02:40:03
> Objet: [pve-devel] HA Migration on shutdown/reboot
> 
> Hello Proxmox Team, 
> 
> I'm wondering why there is no option "migrate" for the shutdown_policy 
> available. 
> 
> I looked a bit through the code and found create_migrate_worker() in 
> PVE/API2/Nodes.pm, which is used in mass migration. Can't this be used 
> in PVE/HA/LRM.pm when shutdown_policy = "migrate"? 
> 
> You know that servers take from 1 to 10 minutes to reboot. So the 
> interruption by freezing, shutting down and starting on another node in 
> HA setup seems not logic to me and will take a long time to recover the 
> service. 
> -> shutdown of databases, mounts, guest os 
> -> shutdown of host 
> -> start on other host 
> 
> Migration would keep the service running all the time - I guess in HA 
> activated Environments online migration is mostly possible - and would 
> also accelerate the shutdown of the host, because the VMs don't have to 
> shut down on that host. 
> 
> Defining the Target Node could rely on the HA Groups priority and if not 
> in HA you could still freeze and shutdown a VM or migrate to least used 
> node - I remember Thomas already wrote about this and the problems 
> behind - or even ask where to migrate. 
> 
> Right now the only way to keep services in HA online all the time 
> through a node reboot, is to change the HA Group, so the service gets 
> correctly migrated by crm. But that also has to be undone after the reboot. 
> 
> Okay, right now LXC came to my mind... I know there's no online 
> migration, but am speaking for KVM now. ;-) Freeze is also applicable here. 
> 
> Probably it's not that easy to implement as I think on top, but do you 
> already think of a feature like this? Or is there any other way to 
> update my HA activated Cluster without service downtime and without the 
> HA Group changing? 
> 
> Thanks so far for PVE! <3 
> Peace 
> Bastian 
>