[pve-devel] HA Migration on shutdown/reboot

[Alexandre DERUMIER]

>>I actually planned to dis-allow that, i.e., if a migration to a node is 
>>requested check first if it won't just get migrated immediately away again 
>>and if so then error out

ok, great.


>> 
>>> Maybe could we first add some kind of "maintenance mode" flag on a node, which mean than vms 
>> can't be migrated back to it. (we virtually remove it from HA groups until maintenance mode is removed). 
>> and user could at least migrate them manually. 


>>As Michael Rasmussen noted, nofailback could be enough here, or not? 

I'm not sure.It could be done too with lowering the weight of the node, like -1000, for example.
Then vms will we auto migrated to nodes with a higher weight.


But I think that user don't want to edit each HA groups where node is member before doing
the maintenance.


Maybe simply, when shutdown action is done, change the weight to -1000, wait than vms are migrated,
reboot the node, then rechange the weight when node come back ?






> Another thing which could be great (even with HA), could be to auto choose targetnode based on cpu/ram current usage, 
> like "migrate ... --targetnode node_with_lower_ram, node_with_lower_cpu." 


>>This I'd not add initially, but yes, as leveling out based on service count 
>>only works if the VMs have roughly the same load, and not many non-HA VMs 
>>are in the cluster "skewing" the load, we'll want to add something like this 
>>afterwards... 

Actually, I have some basic code for memory balancing and cpu balancing. (without talking of HA or groups)
(we have a internal script, we launch manually, to generate the qm migrate command)
I'll try to send them to the mailing (need some cleanup).

It's not too much complex if you do the balancing sequentially.
(calculate the ressources of the global cluster, do migration a 1vm, recalcultate the ressources of the global cluster, do a migration a 1 vm,...)
so for manual migration action, or maybe through crm if we want to automate it later.

It's a lot more difficult if each node try to migrate to others nodes. (migrations loops,...)





----- Mail original -----
De: "Thomas Lamprecht" <t.lamprecht at proxmox.com>
À: "pve-devel" <pve-devel at pve.proxmox.com>, "aderumier" <aderumier at odiso.com>
Envoyé: Mardi 28 Mai 2019 08:44:03
Objet: Re: [pve-devel] HA Migration on shutdown/reboot

Hi, 

On 5/28/19 8:09 AM, Alexandre DERUMIER wrote: 
> in our last week training session, students also ask for the same feature, 
> 
> with a real problem, when we have 2 hosts in same ha group, with differents priority, 
> 
> and user want to do maintenance/reboot, he would like to migrate all vms on other node, 
> before reboot. But currently manually migrating vms (with migrate_all for example), 
> the HA is migrated them back before the migrate all is finished, 
> so it's super difficult to have the node empty for reboot. 

I actually planned to dis-allow that, i.e., if a migration to a node is 
requested check first if it won't just get migrated immediately away again 
and if so then error out. 


> 
> Maybe could we first add some kind of "maintenance mode" flag on a node, which mean than vms 
> can't be migrated back to it. (we virtually remove it from HA groups until maintenance mode is removed). 
> and user could at least migrate them manually. 


As Michael Rasmussen noted, nofailback could be enough here, or not? 

> 
> 
> 
> For auto migrate on shutdown, 
> 
> I think the most difficult part is to how to choose the targetnode where to migrate the vm. 
> Maybe could we choose another nodes of a HA group (excluding the node with maintenance mode) 

For HA this is not to hard, I would just take it out from the "target node" 
calculation, so the HA stack choses automatically based on node's hosted 
HA (running, IIRC) service count. 

> 
> Another thing which could be great (even with HA), could be to auto choose targetnode based on cpu/ram current usage, 
> like "migrate ... --targetnode node_with_lower_ram, node_with_lower_cpu." 

This I'd not add initially, but yes, as leveling out based on service count 
only works if the VMs have roughly the same load, and not many non-HA VMs 
are in the cluster "skewing" the load, we'll want to add something like this 
afterwards... 


> 
> ----- Mail original ----- 
> De: "Bastian Sebode" <b.sebode at linet-services.de> 
> À: "pve-devel" <pve-devel at pve.proxmox.com> 
> Envoyé: Mardi 28 Mai 2019 02:40:03 
> Objet: [pve-devel] HA Migration on shutdown/reboot 
> 
> Hello Proxmox Team, 
> 
> I'm wondering why there is no option "migrate" for the shutdown_policy 
> available. 
> 
> I looked a bit through the code and found create_migrate_worker() in 
> PVE/API2/Nodes.pm, which is used in mass migration. Can't this be used 
> in PVE/HA/LRM.pm when shutdown_policy = "migrate"? 
> 
> You know that servers take from 1 to 10 minutes to reboot. So the 
> interruption by freezing, shutting down and starting on another node in 
> HA setup seems not logic to me and will take a long time to recover the 
> service. 
> -> shutdown of databases, mounts, guest os 
> -> shutdown of host 
> -> start on other host 
> 
> Migration would keep the service running all the time - I guess in HA 
> activated Environments online migration is mostly possible - and would 
> also accelerate the shutdown of the host, because the VMs don't have to 
> shut down on that host. 
> 
> Defining the Target Node could rely on the HA Groups priority and if not 
> in HA you could still freeze and shutdown a VM or migrate to least used 
> node - I remember Thomas already wrote about this and the problems 
> behind - or even ask where to migrate. 
> 
> Right now the only way to keep services in HA online all the time 
> through a node reboot, is to change the HA Group, so the service gets 
> correctly migrated by crm. But that also has to be undone after the reboot. 
> 
> Okay, right now LXC came to my mind... I know there's no online 
> migration, but am speaking for KVM now. ;-) Freeze is also applicable here. 
> 
> Probably it's not that easy to implement as I think on top, but do you 
> already think of a feature like this? Or is there any other way to 
> update my HA activated Cluster without service downtime and without the 
> HA Group changing? 
> 
> Thanks so far for PVE! <3 
> Peace 
> Bastian 
>