[pve-devel] corosync bug: cluster break after 1 node clean shutdown
Josef Johansson
josef at oderland.se
Tue Dec 29 15:21:07 CET 2020
Hi,
On 9/4/20 2:29 PM, Alexandre DERUMIER wrote:
> BTW,
>
> do you think it could be possible to add an extra optionnal layer of security check, not related to corosync ?
>
> I'm still afraid of this corosync bug since years, and still don't use HA. (or I have tried to enable it 2months ago,and this give me a disaster yesterday..)
Using corosync to enable HA is a bit scary TBH, not sure how people
solve this, but I'd rather have the logic outside the cluster so it can
make a solid decision whether a node should be rebooted or not, and
choose to reboot via iDRAC if all checks fail. Maybe use the metrics
support to facilitate this?
> Something like an extra heartbeat between nodes daemons, and check if we also have quorum with theses heartbeats ?
>
>
>
> ----- Mail original -----
> De: "aderumier" <aderumier at odiso.com>
> À: "pve-devel" <pve-devel at pve.proxmox.com>
> Envoyé: Jeudi 3 Septembre 2020 16:11:56
> Objet: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
>
> Hi,
>
> I had a problem this morning with corosync, after shutting down cleanly a node from the cluster.
> This is a 14 nodes cluster, node2 was shutted. (with "halt" command)
>
> HA was actived, and all nodes have been rebooted.
> (I have already see this problem some months ago, without HA enabled, and it was like half of the nodes didn't see others nodes, like 2 cluster formations)
>
> Some users have reported similar problems when adding a new node
> https://forum.proxmox.com/threads/quorum-lost-when-adding-new-7th-node.75197/
>
>
> Here the full logs of the nodes, for each node I'm seeing quorum with 13 nodes. (so it's seem to be ok).
> I didn't have time to look live on server, as HA have restarted them.
> So, I don't known if it's a corosync bug or something related to crm,lrm,pmxcs (but I don't see any special logs)
>
> Only node7 have survived a little bit longer,and I had stop lrm/crm to avoid reboot.
>
> libknet1: 1.15-pve1
> corosync: 3.0.3-pve1
>
>
> Any ideas before submiting bug to corosync mailing ? (I'm seeing new libknet && corosync version on their github, I don't have read the changelog yet)
>
>
>
>
> node1
> -----
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm1 corosync[3678]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm1 corosync[3678]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm1 corosync[3678]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 has no active links
>
> --> reboot
>
> node2 : shutdown log
> -----
> Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd received signal 15: Terminated
> Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_00] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302Y9120CGN.ata.state
> Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_01] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302MG120CGN.ata.state
> Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd is exiting (exit status 0)
> Sep 3 10:39:05 m6kvm2 nrpe[20095]: Caught SIGTERM - shutting down...
> Sep 3 10:39:05 m6kvm2 nrpe[20095]: Daemon shutdown
> Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root at pam> starting task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root at pam:
> Sep 3 10:39:07 m6kvm2 pve-guests[38552]: all VMs and CTs stopped
> Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root at pam> end task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root at pam: OK
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: received signal TERM
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server closing
> Sep 3 10:39:07 m6kvm2 spiceproxy[36572]: worker exit
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: worker 36572 finished
> Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server stopped
> Sep 3 10:39:07 m6kvm2 pvestatd[3786]: received signal TERM
> Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server closing
> Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server stopped
> Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: received signal TERM
> Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: got shutdown request with shutdown policy 'conditional'
> Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: shutdown LRM, stop all services
> Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: received terminate request (signal)
> Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: stopping pvefw logger
> Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: watchdog closed (disabled)
> Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: server stopped
> Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: received signal TERM
> Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: server received shutdown request
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: received signal TERM
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server closing
> Sep 3 10:39:10 m6kvm2 pveproxy[29731]: worker exit
> Sep 3 10:39:10 m6kvm2 pveproxy[30873]: worker exit
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 30873 finished
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 31391 finished
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 29731 finished
> Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server stopped
> Sep 3 10:39:14 m6kvm2 pve-ha-crm[12847]: server stopped
>
>
> node3
> -----
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm3 corosync[30580]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm3 corosync[30580]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm3 corosync[30580]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 has no active links
>
>
> node4
> -----
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm4 corosync[4085]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: all data is up to date
> Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:20 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
> Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
>
> node5
> -----
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm5 corosync[41830]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
> Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
>
>
>
>
>
> node6
> -----
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm6 corosync[36694]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 has no active links
>
>
>
> node7
> -----
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:20 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
> ---> here the others nodes reboot almost at the same time
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 3 link: 0 is down
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 14 link: 0 is down
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 has no active links
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 13 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 8 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 6 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 10 link: 0 is down
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 has no active links
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 has no active links
> Sep 3 10:40:28 m6kvm7 corosync[15467]: [TOTEM ] Token has not been received in 4505 ms
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 11 link: 0 is down
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 9 link: 0 is down
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 has no active links
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 has no active links
> Sep 3 10:40:30 m6kvm7 corosync[15467]: [TOTEM ] A processor failed, forming new configuration.
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 4 link: 0 is down
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 12 link: 0 is down
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 1 link: 0 is down
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 has no active links
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 has no active links
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 has no active links
> Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] link: host: 5 link: 0 is down
> Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 (passive) best link: 0 (pri: 1)
> Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 has no active links
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] A new membership (7.82f) was formed. Members left: 1 3 4 5 6 8 9 10 11 12 13 14
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] Failed to receive the leave message. failed: 1 3 4 5 6 8 9 10 11 12 13 14
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 12 received
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] Members[1]: 7
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:40:41 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: dfsm_deliver_queue: queue length 51
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: node lost quorum
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: members: 7/15892
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: received write while not quorate - trigger resync
> Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: leaving CPG group
> Sep 3 10:40:41 m6kvm7 pve-ha-crm[16196]: node 'm6kvm2': state changed from 'online' => 'unknown'
> Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] notice: start cluster connection
> Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: cpg_join failed: 14
> Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: can't initialize service
> Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
> Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during cfs-locked 'domain-ha' operation: no quorum!
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
> Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied
> Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied
> Sep 3 10:40:56 m6kvm7 pve-ha-lrm[16140]: status change active => lost_agent_lock
> Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change master => lost_manager_lock
> Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: watchdog closed (disabled)
> Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change lost_manager_lock => wait_for_quorum
> Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] rx: host: 4 link: 0 is up
> Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [TOTEM ] A new membership (4.834) was formed. Members joined: 4
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 4/3965, 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: members: 4/3965, 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [QUORUM] Members[2]: 4 7
> Sep 3 10:43:24 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 4/3965/00000002)
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 4/3965/00000002)
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 7/15892
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: start sending inode updates
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: sent all (4) updates
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] rx: host: 3 link: 0 is up
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [TOTEM ] A new membership (3.838) was formed. Members joined: 3
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 3/3613, 4/3965, 7/15892
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: members: 3/3613, 4/3965, 7/15892
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [QUORUM] Members[3]: 3 4 7
> Sep 3 10:44:05 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 3/3613/00000002)
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 3/3613/00000002)
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 4/3965
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 4/3965, 7/15892
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: dfsm_deliver_queue: queue length 1
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] rx: host: 1 link: 0 is up
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.83c) was formed. Members joined: 1
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
> Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [QUORUM] Members[4]: 1 3 4 7
> Sep 3 10:44:31 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3552/00000002)
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3552/00000002)
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 3/3613
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 3/3613, 4/3965, 7/15892
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received all states
> Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
> Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
> Sep 3 10:49:18 m6kvm7 pmxcfs[15892]: [status] notice: received log
>
>
>
>
> node8
> -----
>
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm8 corosync[24361]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
> Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
>
>
> --> reboot
>
>
>
> node9
> -----
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm9 corosync[22340]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: all data is up to date
> Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:20 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
> Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
>
>
> --> reboot
>
>
>
> node10
> ------
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm10 corosync[41458]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: all data is up to date
> Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
> Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
>
> --> reboot
>
>
> node11
> ------
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm11 corosync[12455]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
> Sep 3 10:40:20 m6kvm11 pmxcfs[12983]: [status] notice: received log
>
>
>
>
> node12
> ------
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm12 corosync[43716]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:20 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
> Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
>
>
>
>
>
> node13
> ------
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm13 corosync[39182]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: cpg_send_message retried 1 times
> Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: all data is up to date
> Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 has no active links
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
> --> reboot
>
> node14
> ------
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: starting data syncronisation
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: received sync request (epoch 1/3527/00000002)
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
> Sep 3 10:39:16 m6kvm14 corosync[42413]: [MAIN ] Completed service synchronization, ready to provide service.
> Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: received all states
> Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: all data is up to date
> Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] link: host: 2 link: 0 is down
> Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
> Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 has no active links
>
> --> reboot
>
> _______________________________________________
> pve-devel mailing list
> pve-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
--
Med vänliga hälsningar
Josef Johansson
More information about the pve-devel
mailing list