[PVE-User] TASK ERROR: cluster not ready - no quorum?

Eneko Lacunza elacunza at binovo.es
Mon Mar 9 16:54:14 CET 2015


It seems yesterday something happened at 20:40:53:

Mar 08 20:40:53 corosync [TOTEM ] FAILED TO RECEIVE
Mar 08 20:41:05 corosync [CLM   ] CLM CONFIGURATION CHANGE
Mar 08 20:41:05 corosync [CLM   ] New Configuration:
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.48)
Mar 08 20:41:05 corosync [CLM   ] Members Left:
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.16)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.33)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.49)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.50)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.69)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.75)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.77)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.87)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.141)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.142)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.161)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.163)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.165)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.215)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.216)
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.219)
Mar 08 20:41:05 corosync [CLM   ] Members Joined:
Mar 08 20:41:05 corosync [QUORUM] Members[16]: 1 2 4 5 6 7 8 10 11 12 13 
14 15 16 17 19
Mar 08 20:41:05 corosync [QUORUM] Members[15]: 1 2 4 5 6 7 8 11 12 13 14 
15 16 17 19
Mar 08 20:41:05 corosync [QUORUM] Members[14]: 1 2 4 5 6 7 8 11 12 14 15 
16 17 19
Mar 08 20:41:05 corosync [QUORUM] Members[13]: 1 2 4 5 6 7 8 11 12 15 16 
17 19
Mar 08 20:41:05 corosync [QUORUM] Members[12]: 1 2 4 5 6 7 8 11 12 15 17 19
Mar 08 20:41:05 corosync [QUORUM] Members[11]: 1 2 4 5 6 7 8 11 12 15 17
Mar 08 20:41:05 corosync [QUORUM] Members[10]: 1 2 4 5 6 7 8 11 12 17
Mar 08 20:41:05 corosync [CMAN  ] quorum lost, blocking activity
Mar 08 20:41:05 corosync [QUORUM] This node is within the non-primary 
component and will NOT provide any services.
Mar 08 20:41:05 corosync [QUORUM] Members[9]: 1 2 5 6 7 8 11 12 17
Mar 08 20:41:05 corosync [QUORUM] Members[8]: 1 2 5 6 7 11 12 17
Mar 08 20:41:05 corosync [QUORUM] Members[7]: 1 2 5 6 7 12 17
Mar 08 20:41:05 corosync [QUORUM] Members[6]: 1 2 6 7 12 17
Mar 08 20:41:05 corosync [QUORUM] Members[5]: 1 2 7 12 17
Mar 08 20:41:05 corosync [QUORUM] Members[4]: 1 2 12 17
Mar 08 20:41:05 corosync [QUORUM] Members[3]: 1 12 17
Mar 08 20:41:05 corosync [QUORUM] Members[2]: 1 12
Mar 08 20:41:05 corosync [QUORUM] Members[1]: 12
Mar 08 20:41:05 corosync [CLM   ] CLM CONFIGURATION CHANGE
Mar 08 20:41:05 corosync [CLM   ] New Configuration:
Mar 08 20:41:05 corosync [CLM   ]     r(0) ip(172.31.2.48)
Mar 08 20:41:05 corosync [CLM   ] Members Left:
Mar 08 20:41:05 corosync [CLM   ] Members Joined:
Mar 08 20:41:05 corosync [TOTEM ] A processor joined or left the 
membership and a new membership was formed.
Mar 08 20:41:05 corosync [CPG   ] chosen downlist: sender r(0) 
ip(172.31.2.48) ; members(old:17 left:16)
Mar 08 20:41:05 corosync [MAIN  ] Completed service synchronization, 
ready to provide service

Is the "pvecm nodes" similar in all nodes?

I don't have experience troubleshooting corosync but it seems you have 
to re-estrablish the corosync cluster and quorum.

Check "corosync-quorumtool -l -i" . Also check cman_tool command for 
diagnosing the cluster.

Is corosync service loaded and running? Does restarting it change 
something (service cman restart) ?



On 09/03/15 16:13, Shain Miley wrote:
> Oddly enough...there is nothing in the latest corosync 
> logfile...however the one from last night (when we started seeing the 
> problem) has a lot of info in it.
>
> Here is the link to entire file:
>
> http://717b5bb5f6a032ce28eb-fa7f03050c118691fd4b41bf00a93863.r71.cf1.rackcdn.com/corosync.log.1
>
> Thanks again for your help so far.
>
> Shain
>
> On 3/9/15 10:53 AM, Eneko Lacunza wrote:
>> What about /var/log/cluster/corosync.log ?
>>
>> On 09/03/15 15:34, Shain Miley wrote:
>>> Yes,
>>>
>>> All the nodes are pingable and resolvable via their hostname.
>>>
>>> Here is the ouput of 'pvecm nodes'
>>>
>>>
>>> root at proxmox13:~# pvecm nodes
>>> Node  Sts   Inc   Joined               Name
>>>    1   X    964                        proxmox22
>>>    2   X    964                        proxmox23
>>>    3   X    756                        proxmox24
>>>    4   X    808                        proxmox18
>>>    5   X    964                        proxmox19
>>>    6   X    964                        proxmox20
>>>    7   X    964                        proxmox21
>>>    8   X    964                        proxmox1
>>>    9   X      0                        proxmox2
>>>   10   X    756                        proxmox3
>>>   11   X    964                        proxmox4
>>>   12   M    696   2014-10-20 01:10:09  proxmox13
>>>   13   X    904                        proxmox14
>>>   14   X    848                        proxmox15
>>>   15   X    856                        proxmox16
>>>   16   X    836                        proxmox17
>>>   17   X    964                        proxmox25
>>>   18   X    960                        proxmox26
>>>   19   X    868                        proxmox28
>>>
>>> Thanks,
>>>
>>> Shain
>>>
>>> On 3/9/15 10:23 AM, Eneko Lacunza wrote:
>>>> pvecm nodes
>>>
>>>
>>> -- 
>>> _NPR | Shain Miley| Manager of Systems and Infrastructure, Digital 
>>> Media | smiley at npr.org | p: 202-513-3649
>>
>>
>> -- 
>> Zuzendari Teknikoa / Director Técnico
>> Binovo IT Human Project, S.L.
>> Telf. 943575997
>>        943493611
>> Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
>> www.binovo.es
>
>
> -- 
> _NPR | Shain Miley| Manager of Systems and Infrastructure, Digital 
> Media | smiley at npr.org | p: 202-513-3649


-- 
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943575997
       943493611
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
www.binovo.es

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://pve.proxmox.com/pipermail/pve-user/attachments/20150309/b15f7ce6/attachment-0015.html>


More information about the pve-user mailing list