<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"><base href="x-msg://13338/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Thank you Dietmar.<div><br></div><div>However I just realized that the cluster config refers to the quorum disk by label, not by device.</div><div>So maybe this wasn't my problem in the first place.</div><div><br></div><div>Here's the cluster config I'm trying to apply:</div><div><br></div><div><div><?xml version="1.0"?></div><div><cluster name="sthlm-bb" config_version="3"></div><div><br></div><div><br></div><div><cman expected_votes="3" keyfile="/var/lib/pve-cluster/corosync.authkey"/></div><div> <quorumd votes="1" allow_kill="0" interval="1" label="proxmox1_qdisk" tko="10"/></div><div> <totem token="54000"/></div><div><br></div><div> <clusternodes></div><div> <clusternode name="beast" votes="1" nodeid="1"/></div><div> <clusternode name="beauty" votes="1" nodeid="2"/></clusternodes></div><div><br></div><div></cluster></div><div><br></div><div><br></div><div>Now I'm not really sure what the status is of my cluster. I'm stuck at the line on the wiki that says</div><div>"</div><ul style="line-height: 19.046875px; list-style-type: square; margin: 0.3em 0px 0px 1.5em; padding: 0px; list-style-image: url(http://pve.proxmox.com/mediawiki/skins//pve/bullet.gif); font-family: sans-serif; font-size: 13px; background-color: rgb(255, 255, 255); "><li style="margin-bottom: 0.1em;">Verify that rgmanager is running, if not, start it"</li></ul><div><font face="sans-serif"><span style="font-size: 13px; line-height: 19px;"><br></span></font></div><div><font face="sans-serif"><span style="font-size: 13px; line-height: 19px;">/etc/init.d/rgemanager start</span></font></div><div><font face="sans-serif"><span style="font-size: 13px; line-height: 19px;">doesn't work:</span></font></div><div><font face="sans-serif"><div style="font-size: 13px; line-height: 19px; ">root@beast:/var/log# /etc/init.d/rgmanager status</div><div style="font-size: 13px; line-height: 19px; ">rgmanager is stopped</div><div style="font-size: 13px; line-height: 19px; "><br></div><div style="font-size: 13px; line-height: 19px; ">My /var/log/daemon.log is filled with </div><div style="font-size: 13px; line-height: 19px; ">Oct 2 10:55:11 beast pmxcfs[635625]: [status] crit: cpg_send_message failed: 9</div><div style="font-size: 13px; line-height: 19px; "><br></div><div><div><span style="font-size: 13px; line-height: 19px;">root@beast:/var/log# clustat</span></div><div><span style="font-size: 13px; line-height: 19px;">Cluster Status for sthlm-bb @ Wed Oct 2 10:58:32 2013</span></div><div><span style="font-size: 13px; line-height: 19px;">Member Status: Quorate</span></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><span style="font-size: 13px; line-height: 19px;"> Member Name ID Status</span></div><div><span style="font-size: 13px; line-height: 19px;"> ------ ---- ---- ------</span></div><div><span style="font-size: 13px; line-height: 19px;"> beast 1 Online, Local</span></div><div><span style="font-size: 13px; line-height: 19px;"> beauty 2 Online</span></div><div><span style="font-size: 13px; line-height: 19px;"> /dev/block/8:65 0 Online, Quorum Disk</span></div></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><div><span style="font-size: 13px; line-height: 19px;">root@beauty:~# clustat</span></div><div><span style="font-size: 13px; line-height: 19px;">Cluster Status for sthlm-bb @ Wed Oct 2 10:58:50 2013</span></div><div><span style="font-size: 13px; line-height: 19px;">Member Status: Quorate</span></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><span style="font-size: 13px; line-height: 19px;"> Member Name ID Status</span></div><div><span style="font-size: 13px; line-height: 19px;"> ------ ---- ---- ------</span></div><div><span style="font-size: 13px; line-height: 19px;"> beast 1 Online</span></div><div><span style="font-size: 13px; line-height: 19px;"> beauty 2 Online, Local</span></div><div><span style="font-size: 13px; line-height: 19px;"> /dev/block/8:49 0 Online, Quorum Disk</span></div></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><div style="font-size: 13px; line-height: 19px; ">root@beast:/var/log# pvecm s</div><div style="font-size: 13px; line-height: 19px; ">Version: 6.2.0</div><div style="font-size: 13px; line-height: 19px; ">Config Version: 3</div><div style="font-size: 13px; line-height: 19px; ">Cluster Name: sthlm-bb</div><div style="font-size: 13px; line-height: 19px; ">Cluster Id: 28546</div><div style="font-size: 13px; line-height: 19px; ">Cluster Member: Yes</div><div style="font-size: 13px; line-height: 19px; ">Cluster Generation: 328</div><div style="font-size: 13px; line-height: 19px; ">Membership state: Cluster-Member</div><div style="font-size: 13px; line-height: 19px; ">Nodes: 2</div><div style="font-size: 13px; line-height: 19px; ">Expected votes: 3</div><div style="font-size: 13px; line-height: 19px; ">Quorum device votes: 1</div><div style="font-size: 13px; line-height: 19px; ">Total votes: 3</div><div style="font-size: 13px; line-height: 19px; ">Node votes: 1</div><div style="font-size: 13px; line-height: 19px; ">Quorum: 2</div><div style="font-size: 13px; line-height: 19px; ">Active subsystems: 7</div><div style="font-size: 13px; line-height: 19px; ">Flags:</div><div style="font-size: 13px; line-height: 19px; ">Ports Bound: 0 178</div><div style="font-size: 13px; line-height: 19px; ">Node name: beast</div><div style="font-size: 13px; line-height: 19px; ">Node ID: 1</div><div style="font-size: 13px; line-height: 19px; ">Multicast addresses: 239.192.111.241</div><div style="font-size: 13px; line-height: 19px; ">Node addresses: X.Y.Z.46</div><div style="font-size: 13px; line-height: 19px; "><br></div><div style="font-size: 13px; line-height: 19px; "><div>root@beauty:~# pvecm s</div><div>Version: 6.2.0</div><div>Config Version: 3</div><div>Cluster Name: sthlm-bb</div><div>Cluster Id: 28546</div><div>Cluster Member: Yes</div><div>Cluster Generation: 328</div><div>Membership state: Cluster-Member</div><div>Nodes: 2</div><div>Expected votes: 3</div><div>Quorum device votes: 1</div><div>Total votes: 3</div><div>Node votes: 1</div><div>Quorum: 2</div><div>Active subsystems: 7</div><div>Flags:</div><div>Ports Bound: 0 178</div><div>Node name: beauty</div><div>Node ID: 2</div><div>Multicast addresses: 239.192.111.241</div><div>Node addresses: X.Y.Z.47</div></div><div style="font-size: 13px; line-height: 19px; "><br></div><div style="font-size: 13px; line-height: 19px; "><br></div><div style="font-size: 13px; line-height: 19px; ">My problem seems to be that rgmanager won't start, is this due to some misconfig of the qdisk?</div><div style="font-size: 13px; line-height: 19px; ">I can't find the error log from the rgmanager startup</div><div style="font-size: 13px; line-height: 19px; "><br></div><div style="font-size: 13px; line-height: 19px; ">Here's the info about the qdisk from both nodes:</div><div><div><span style="font-size: 13px; line-height: 19px;">root@beast:/var/log# mkqdisk -f proxmox1_qdisk</span></div><div><span style="font-size: 13px; line-height: 19px;">mkqdisk v1364188437</span></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><span style="font-size: 13px; line-height: 19px;">/dev/block/8:65:</span></div><div><span style="font-size: 13px; line-height: 19px;">/dev/disk/by-id/scsi-1494554000000000028d222db64a9650d05bebe3d5ddef499-part1:</span></div><div><span style="font-size: 13px; line-height: 19px;">/dev/disk/by-path/ip-10.6.6.21:3260-iscsi-iqn.2013-09.se.accumulate.sthlm:qrmdisk1-lun-0-part1:</span></div><div><span style="font-size: 13px; line-height: 19px;">/dev/sde1:</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space:pre"> </span>Magic: eb7a62c2</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space:pre"> </span>Label: proxmox1_qdisk</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space:pre"> </span>Created: Tue Oct 1 16:36:24 2013</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space:pre"> </span>Host: beast</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space:pre"> </span>Kernel Sector Size: 512</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space:pre"> </span>Recorded Sector Size: 512</span></div></div><div style="font-size: 13px; line-height: 19px; "><br></div><div style="font-size: 13px; line-height: 19px; "><br></div><div><div><span style="font-size: 13px; line-height: 19px;">root@beauty:~# mkqdisk -f proxmox1_qdisk</span></div><div><span style="font-size: 13px; line-height: 19px;">mkqdisk v1364188437</span></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><span style="font-size: 13px; line-height: 19px;">/dev/block/8:49:</span></div><div><span style="font-size: 13px; line-height: 19px;">/dev/disk/by-id/scsi-1494554000000000028d222db64a9650d05bebe3d5ddef499-part1:</span></div><div><span style="font-size: 13px; line-height: 19px;">/dev/disk/by-path/ip-10.6.6.21:3260-iscsi-iqn.2013-09.se.accumulate.sthlm:qrmdisk1-lun-0-part1:</span></div><div><span style="font-size: 13px; line-height: 19px;">/dev/sdd1:</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space:pre"> </span>Magic: eb7a62c2</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space:pre"> </span>Label: proxmox1_qdisk</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space:pre"> </span>Created: Tue Oct 1 16:36:24 2013</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space:pre"> </span>Host: beast</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space:pre"> </span>Kernel Sector Size: 512</span></div><div><span style="font-size: 13px; line-height: 19px;"><span class="Apple-tab-span" style="white-space: pre; "> </span>Recorded Sector Size: 512</span></div></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><span style="font-size: 13px; line-height: 19px;">Any idea what I have done wrong?</span></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><span style="font-size: 13px; line-height: 19px;">Thanks,</span></div><div><span style="font-size: 13px; line-height: 19px;">Johan</span></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div style="font-size: 13px; line-height: 19px; "><br></div></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div><div><span style="font-size: 13px; line-height: 19px;"><br></span></div></font></div><div><font face="sans-serif"><span style="font-size: 13px; line-height: 19px;"><br></span></font></div><div><br></div><div> </div>
<br><div><div>On Oct 2, 2013, at 10:30 , Dietmar Maurer <<a href="mailto:dietmar@proxmox.com">dietmar@proxmox.com</a>> wrote:</div><br class="Apple-interchange-newline"><div lang="DE-AT" link="blue" vlink="purple" style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="WordSection1" style="page: WordSection1; "><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">You can use the /dev/disk/by-XYZ device links.<o:p></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="border-style: none none none solid; border-left-width: 1.5pt; border-left-color: blue; padding: 0cm 0cm 0cm 4pt; "><div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0cm 0cm; "><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span lang="EN-US" style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span lang="EN-US" style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span><a href="mailto:pve-user-bounces@pve.proxmox.com">pve-user-bounces@pve.proxmox.com</a> [mailto:pve-<a href="mailto:user-bounces@pve.proxmox.com">user-bounces@pve.proxmox.com</a>]<span class="Apple-converted-space"> </span><b>On Behalf Of<span class="Apple-converted-space"> </span></b>Johan Wikström<br><b>Sent:</b><span class="Apple-converted-space"> </span>Mittwoch, 02. Oktober 2013 10:18<br><b>To:</b><span class="Apple-converted-space"> </span><a href="mailto:pve-user@pve.proxmox.com">pve-user@pve.proxmox.com</a><br><b>Subject:</b><span class="Apple-converted-space"> </span>[PVE-User] Quorom disk on different devices<o:p></o:p></span></div></div></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Hi list!<o:p></o:p></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">I have set up a two node Proxmox cluster on two Dell servers with DRBD. Works well, but I just realized<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">that I need a quorum disk for the HA setup. I created an iSCSI qdisk as per the FAQ (<a href="http://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster#Configuring_Fencing" style="color: purple; text-decoration: underline; ">http://pve.proxmox.com/wiki/Two-Node_High_Availability_Cluster#Configuring_Fencing</a>)<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">but I ran into a problem. For some reason the first server had one extra disk device (floppy, maybe virtual floppy/USB) so now<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">the iSCSI disk is known as /dev/sde on the first server and /dev/sdd on the second server. <o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Since the cluster config file is global I guess this will be a problem when defining the quorum disk?<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Do I need to start creating udev rules for the iSCSI LUN to persist the naming on both hosts, or<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">can I cheat in some way? Maybe it would be enough to remove the "floppy" device link /dev/sdd on the first<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">host and try to re-create the iSCSI target?<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">PVE Version:<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">pve-manager/3.1-3/dc0e9b0e (running kernel: 2.6.32-22-pve)<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Hardware: 2 * Dell T620<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Best regards<o:p></o:p></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span class="apple-style-span"><b><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif; color: rgb(102, 102, 102); ">Johan Wikström</span></b></span><span class="apple-style-span"><span style="font-size: 13.5pt; font-family: Helvetica, sans-serif; "><o:p></o:p></span></span></div><div><p class="MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "></p></div></div></div></div></div></div></div><br></div></body></html>