<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: times new roman,new york,times,serif; font-size: 12pt; color: #000000'>Got it, thanks for the explanation Martin. Did I miss any documentation on this?<div><br></div><div>For a graceful shutdown, I need to use Stop for HA-managed VMs, and Shutdown for those that are not HA-managed.</div><div>Seems to work, just tried it and VM HA status says disabled, as you say.</div><div><br></div><div>But maybe I'm now hitting another bug. Once shut down, I want to offline migrate the VM to another node, and trying, see this:</div><div><br></div><div><span style="font-family: tahoma, arial, verdana, sans-serif; font-size: 11px; line-height: 16px; white-space: pre; background-color: rgb(255, 255, 255); ">Executing HA migrate for VM 103 to node ollie</span><br style="box-sizing: border-box; font-family: tahoma, arial, verdana, sans-serif; font-size: 11px; line-height: 16px; white-space: pre; background-color: rgb(255, 255, 255); "><span style="font-family: tahoma, arial, verdana, sans-serif; font-size: 11px; line-height: 16px; white-space: pre; background-color: rgb(255, 255, 255); ">Trying to migrate pvevm:103 to ollie...Temporary failure; try again</span><br style="box-sizing: border-box; font-family: tahoma, arial, verdana, sans-serif; font-size: 11px; line-height: 16px; white-space: pre; background-color: rgb(255, 255, 255); "><span style="font-family: tahoma, arial, verdana, sans-serif; font-size: 11px; line-height: 16px; white-space: pre; background-color: rgb(255, 255, 255); ">TASK ERROR: command 'clusvcadm -M pvevm:103 -m ollie' failed: exit code 250</span></div><div><span style="font-family: tahoma, arial, verdana, sans-serif; font-size: 11px; line-height: 16px; white-space: pre; background-color: rgb(255, 255, 255); "><br></span></div><div></div><div><span style="font-size: 12pt; ">I then powered up the VM and successfully HA/VM live migrated it. (HA) Stop again - shut down cleanly and try to offline migrate back to the other node, same error as above</span></div><div><span style="font-size: 12pt; "><br></span></div><div><span style="font-size: 12pt; ">Then, with the powered-down VM on a node, I powered off the node (simulate failure), and after fencing, the VM is stuck there, inaccessible at least from the GUI. Is this too expected behavior, or a bug? I've lost access to that VM while the node on which it was last running is down. It seems I have to have it powered up for HA automatic migration to be active.</span></div><div><br></div><div>Last, does the HA autostart parameter supercede the individual VM "start at boot" parameter, or how do they interact? Apologies if this is documented somewhere - I haven't seen it.</div><div><br></div><div>Cheers,<br>Dave<br><br><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">
<style><!--
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
@list l0
{mso-list-id:568539903;
mso-list-type:hybrid;
mso-list-template-ids:151281932 1946980126 201785347 201785349 201785345 201785347 201785349 201785345 201785347 201785349;}
@list l0:level1
{mso-level-start-at:2012;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi,</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If you „Stop“ a HA managed VM you will get the following two actions:</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">A VM acpid shutdown (graceful shutdown)</span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">-<span style="font:7.0pt "Times New Roman"">
</span></span></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">HA stop</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Now, if you go to the Datacenter/Summary tab you will see on “HA Service Status” the VM Status “disabled”.</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">To enable/Start the HA VM/CT again, just start the VM/CT.</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If you shutdown it will restarted again by rgmanager, the expected behavior here.</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If you stop rgmanager as mentioned in the wiki, all VM will get an acpi shutdown (gracefully shutdown) and will be started again on another node.</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Already planned for the future:</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="https://bugzilla.proxmox.com/show_bug.cgi?id=173" target="_blank"><span style="color:#1F497D;text-decoration:none">https://bugzilla.proxmox.com/show_bug.cgi?id=173</span></a></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Martin</span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> David Black [mailto:dave@jamsoft.com]
<br>
<b>Sent:</b> Donnerstag, 11. Oktober 2012 14:40<br>
<b>To:</b> Martin Maurer<br>
<b>Cc:</b> proxmoxve (pve-user@pve.proxmox.com)<br>
<b>Subject:</b> Re: [PVE-User] HA behavior with VMs (PVE 2.2 pre-release)</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal"><span style="color:black">Hi Martin, and I have done almost exactly as you say in the first paragraph.</span></p>
<div>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">The difference: I clicked "Shutdown", not a (forced) "Stop", and the VM gracefully shut down. After a few seconds, the VM is restarted by RGmanager.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">I do not expect RGmanager would restart it, but it does, and this is regardless of the autostart setting. I'll test it a few more times and probably open a bug.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">BTW, I got the idea of stopping RGmanager here: <a href="http://pve.proxmox.com/wiki/High_Availability_Cluster#HA_Cluster_maintenance_.28node_reboots.29" target="_blank">http://pve.proxmox.com/wiki/High_Availability_Cluster#HA_Cluster_maintenance_.28node_reboots.29</a></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">It was to force migrate the VMs off the soon-to-be-rebooted node, not simulate host failure as such - sorry. (I do understand the difference.)</span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">And I just found this thread, with earlier discussion on live migration behavior when stopping RGmanager. It appears just not to have been implemented, and I'll search for an existing RFE before possibly opening
one:</span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"><a href="http://forum.proxmox.com/threads/9285-HA-no-live-migration-when-stopping-rgmanager" target="_blank">http://forum.proxmox.com/threads/9285-HA-no-live-migration-when-stopping-rgmanager</a></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Thanks,</span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:black">Dave</span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="color:black">
<hr size="2" width="100%" align="center" id="zwchr">
</span></div>
<blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi,</span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If you enable HA for a VM or CT, make sure that the VM/CT is off. As soon as you enable HA, the resource manager (rgmanager) takes control and
starts the VM/CT. you can still live migrate manually. If you click “Stop” the VM/CT will be stopped and not started again by rgmanager.</span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If you need to reboot a host, live migrate all VM/CT´s to other node or stop the VM/CT.</span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If your system does not work like this, it’s a bug or configuration issues. (stopping rgmanager is not a suitable testcase to simulate a failed
node)</span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">Best Regards,</span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"> </span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">Martin Maurer</span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"> </span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><a href="mailto:martin@proxmox.com" target="_blank"><span lang="EN-US">martin@proxmox.com</span></a></span><span style="color:black"></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><a href="http://www.proxmox.com" target="_blank"><span lang="EN-US">http://www.proxmox.com</span></a></span><span style="color:black"></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">
<a href="mailto:pve-user-bounces@pve.proxmox.com" target="_blank">pve-user-bounces@pve.proxmox.com</a> [<a href="mailto:pve-user-bounces@pve.proxmox.com" target="_blank">mailto:pve-user-bounces@pve.proxmox.com</a>]
<b>On Behalf Of </b>David Black<br>
<b>Sent:</b> Donnerstag, 11. Oktober 2012 13:39<br>
<b>To:</b> Dietmar Maurer<br>
<b>Cc:</b> proxmoxve (<a href="mailto:pve-user@pve.proxmox.com" target="_blank">pve-user@pve.proxmox.com</a>)<br>
<b>Subject:</b> Re: [PVE-User] HA behavior with VMs (PVE 2.2 pre-release)</span><span style="color:black"></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> </span></p>
<div>
<div>
<p class="MsoNormal"><span style="color:black">If you mean disabling HA allows me to regain full control of the VM, I do understand that, and at the moment consider it a workaround.</span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
<p class="MsoNormal"><span style="color:black">Or do you mean by design, HA enabled = VM unconditionally runs and can't be powered off, regardless of the autostart parameter setting?</span></p>
<div>
<p class="MsoNormal"><span style="color:black">That seems unlikely to me, isn't mentioned in the wiki HA section, and is why I emailed. If that's not how you intend it to work, I'd be happy to open a bug.</span></p>
<div>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="color:black">Dave</span></p>
<div class="MsoNormal" align="center" style="text-align:center"><span style="color:black">
<hr size="2" width="100%" align="center">
</span></div>
<blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Simply remove the VM from HA if you do not want that behavior.</span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Note: HA = High Availability</span><span style="color:black"></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="color:black">Because of #3, it seems I can't have a powered-off VM sitting there, that will be moved to another host in the event whichever host it's on goes down.
</span><span style="color:black">Seems like a bug to me. Maybe the behavior of #3 and 4 are related. I want to make all my VMs "protected" by HA whether they're running or not, so they always remain accessible regardless of their powered off/on status. Also,
why can't I have an HA-protected VM stay powered off? :-)</span></p>
</div>
</div>
</div>
</blockquote>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black"><br>
_______________________________________________<br>
pve-user mailing list<br>
<a href="mailto:pve-user@pve.proxmox.com" target="_blank">pve-user@pve.proxmox.com</a><br>
<a href="http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user" target="_blank">http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user</a></span></p>
</blockquote>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
</div>
</div>
</div>
<br>_______________________________________________<br>pve-user mailing list<br>pve-user@pve.proxmox.com<br>http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user<br></blockquote><br></div></div></body></html>