[pve-devel] corosync bug: cluster break after 1 node clean shutdown
Alexandre DERUMIER
aderumier at odiso.com
Fri Sep 4 14:29:40 CEST 2020
BTW,
do you think it could be possible to add an extra optionnal layer of security check, not related to corosync ?
I'm still afraid of this corosync bug since years, and still don't use HA. (or I have tried to enable it 2months ago,and this give me a disaster yesterday..)
Something like an extra heartbeat between nodes daemons, and check if we also have quorum with theses heartbeats ?
----- Mail original -----
De: "aderumier" <aderumier at odiso.com>
À: "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Jeudi 3 Septembre 2020 16:11:56
Objet: [pve-devel] corosync bug: cluster break after 1 node clean shutdown
Hi,
I had a problem this morning with corosync, after shutting down cleanly a node from the cluster.
This is a 14 nodes cluster, node2 was shutted. (with "halt" command)
HA was actived, and all nodes have been rebooted.
(I have already see this problem some months ago, without HA enabled, and it was like half of the nodes didn't see others nodes, like 2 cluster formations)
Some users have reported similar problems when adding a new node
https://forum.proxmox.com/threads/quorum-lost-when-adding-new-7th-node.75197/
Here the full logs of the nodes, for each node I'm seeing quorum with 13 nodes. (so it's seem to be ok).
I didn't have time to look live on server, as HA have restarted them.
So, I don't known if it's a corosync bug or something related to crm,lrm,pmxcs (but I don't see any special logs)
Only node7 have survived a little bit longer,and I had stop lrm/crm to avoid reboot.
libknet1: 1.15-pve1
corosync: 3.0.3-pve1
Any ideas before submiting bug to corosync mailing ? (I'm seeing new libknet && corosync version on their github, I don't have read the changelog yet)
node1
-----
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm1 pmxcfs[3527]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm1 corosync[3678]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm1 corosync[3678]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm1 corosync[3678]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: received all states
Sep 3 10:39:17 m6kvm1 pmxcfs[3527]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm1 corosync[3678]: [KNET ] host: host: 2 has no active links
--> reboot
node2 : shutdown log
-----
Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd received signal 15: Terminated
Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_00] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302Y9120CGN.ata.state
Sep 3 10:39:05 m6kvm2 smartd[27390]: Device: /dev/bus/0 [megaraid_disk_01] [SAT], state written to /var/lib/smartmontools/smartd.INTEL_SSDSC2BB120G6R-PHWA640302MG120CGN.ata.state
Sep 3 10:39:05 m6kvm2 smartd[27390]: smartd is exiting (exit status 0)
Sep 3 10:39:05 m6kvm2 nrpe[20095]: Caught SIGTERM - shutting down...
Sep 3 10:39:05 m6kvm2 nrpe[20095]: Daemon shutdown
Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root at pam> starting task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root at pam:
Sep 3 10:39:07 m6kvm2 pve-guests[38552]: all VMs and CTs stopped
Sep 3 10:39:07 m6kvm2 pve-guests[38550]: <root at pam> end task UPID:m6kvm2:00009698:93A98FFD:5F50ABAB:stopall::root at pam: OK
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: received signal TERM
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server closing
Sep 3 10:39:07 m6kvm2 spiceproxy[36572]: worker exit
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: worker 36572 finished
Sep 3 10:39:07 m6kvm2 spiceproxy[3847]: server stopped
Sep 3 10:39:07 m6kvm2 pvestatd[3786]: received signal TERM
Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server closing
Sep 3 10:39:07 m6kvm2 pvestatd[3786]: server stopped
Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: received signal TERM
Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: got shutdown request with shutdown policy 'conditional'
Sep 3 10:39:07 m6kvm2 pve-ha-lrm[12768]: shutdown LRM, stop all services
Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: received terminate request (signal)
Sep 3 10:39:08 m6kvm2 pvefw-logger[36670]: stopping pvefw logger
Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: watchdog closed (disabled)
Sep 3 10:39:08 m6kvm2 pve-ha-lrm[12768]: server stopped
Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: received signal TERM
Sep 3 10:39:10 m6kvm2 pve-ha-crm[12847]: server received shutdown request
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: received signal TERM
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server closing
Sep 3 10:39:10 m6kvm2 pveproxy[29731]: worker exit
Sep 3 10:39:10 m6kvm2 pveproxy[30873]: worker exit
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 30873 finished
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 31391 finished
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: worker 29731 finished
Sep 3 10:39:10 m6kvm2 pveproxy[24735]: server stopped
Sep 3 10:39:14 m6kvm2 pve-ha-crm[12847]: server stopped
node3
-----
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm3 corosync[30580]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm3 corosync[30580]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm3 corosync[30580]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:16 m6kvm3 pmxcfs[27373]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: received all states
Sep 3 10:39:17 m6kvm3 pmxcfs[27373]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm3 corosync[30580]: [KNET ] host: host: 2 has no active links
node4
-----
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm4 pmxcfs[3903]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm4 corosync[4085]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm4 corosync[4085]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm4 corosync[4085]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: received all states
Sep 3 10:39:17 m6kvm4 pmxcfs[3903]: [status] notice: all data is up to date
Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:21 m6kvm4 corosync[4085]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:19 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:20 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:21 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
Sep 3 10:40:22 m6kvm4 pmxcfs[3903]: [status] notice: received log
node5
-----
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm5 pmxcfs[42272]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm5 corosync[41830]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm5 corosync[41830]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm5 corosync[41830]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: received all states
Sep 3 10:39:17 m6kvm5 pmxcfs[42272]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm5 corosync[41830]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:19 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:20 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:21 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
Sep 3 10:40:22 m6kvm5 pmxcfs[42272]: [status] notice: received log
node6
-----
Sep 3 10:39:16 m6kvm6 corosync[36694]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm6 corosync[36694]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm6 corosync[36694]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm6 corosync[36694]: [KNET ] host: host: 2 has no active links
node7
-----
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm7 corosync[15467]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:39:17 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:19 m6kvm7 corosync[15467]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:19 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:20 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:21 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:40:22 m6kvm7 pmxcfs[15892]: [status] notice: received log
---> here the others nodes reboot almost at the same time
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 3 link: 0 is down
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] link: host: 14 link: 0 is down
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 3 has no active links
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 (passive) best link: 0 (pri: 1)
Sep 3 10:40:25 m6kvm7 corosync[15467]: [KNET ] host: host: 14 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 13 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 8 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 6 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] link: host: 10 link: 0 is down
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 13 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 8 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 6 has no active links
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 (passive) best link: 0 (pri: 1)
Sep 3 10:40:27 m6kvm7 corosync[15467]: [KNET ] host: host: 10 has no active links
Sep 3 10:40:28 m6kvm7 corosync[15467]: [TOTEM ] Token has not been received in 4505 ms
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 11 link: 0 is down
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] link: host: 9 link: 0 is down
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 (passive) best link: 0 (pri: 1)
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 11 has no active links
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 (passive) best link: 0 (pri: 1)
Sep 3 10:40:29 m6kvm7 corosync[15467]: [KNET ] host: host: 9 has no active links
Sep 3 10:40:30 m6kvm7 corosync[15467]: [TOTEM ] A processor failed, forming new configuration.
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 4 link: 0 is down
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 12 link: 0 is down
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] link: host: 1 link: 0 is down
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 4 has no active links
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 (passive) best link: 0 (pri: 1)
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 12 has no active links
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 3 10:40:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 has no active links
Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] link: host: 5 link: 0 is down
Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 (passive) best link: 0 (pri: 1)
Sep 3 10:40:34 m6kvm7 corosync[15467]: [KNET ] host: host: 5 has no active links
Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] A new membership (7.82f) was formed. Members left: 1 3 4 5 6 8 9 10 11 12 13 14
Sep 3 10:40:41 m6kvm7 corosync[15467]: [TOTEM ] Failed to receive the leave message. failed: 1 3 4 5 6 8 9 10 11 12 13 14
Sep 3 10:40:41 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 12 received
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Sep 3 10:40:41 m6kvm7 corosync[15467]: [QUORUM] Members[1]: 7
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:40:41 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: dfsm_deliver_queue: queue length 51
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 1/3527
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 4/3903
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 9/22810
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 6/37248
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 11/12983
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 10/41940
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 5/42272
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 13/39678
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 8/24790
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 3/27373
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 12/44214
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] notice: remove message from non-member 14/42930
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: node lost quorum
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [status] notice: members: 7/15892
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: received write while not quorate - trigger resync
Sep 3 10:40:41 m6kvm7 pmxcfs[15892]: [dcdb] crit: leaving CPG group
Sep 3 10:40:41 m6kvm7 pve-ha-crm[16196]: node 'm6kvm2': state changed from 'online' => 'unknown'
Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] notice: start cluster connection
Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: cpg_join failed: 14
Sep 3 10:40:42 m6kvm7 pmxcfs[15892]: [dcdb] crit: can't initialize service
Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 7/15892
Sep 3 10:40:48 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:40:50 m6kvm7 pve-ha-crm[16196]: got unexpected error - error during cfs-locked 'domain-ha' operation: no quorum!
Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: loop take too long (87 seconds)
Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: loop take too long (92 seconds)
Sep 3 10:40:51 m6kvm7 pve-ha-crm[16196]: lost lock 'ha_manager_lock - cfs lock update failed - Permission denied
Sep 3 10:40:51 m6kvm7 pve-ha-lrm[16140]: lost lock 'ha_agent_m6kvm7_lock - cfs lock update failed - Permission denied
Sep 3 10:40:56 m6kvm7 pve-ha-lrm[16140]: status change active => lost_agent_lock
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change master => lost_manager_lock
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: watchdog closed (disabled)
Sep 3 10:40:56 m6kvm7 pve-ha-crm[16196]: status change lost_manager_lock => wait_for_quorum
Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] rx: host: 4 link: 0 is up
Sep 3 10:43:23 m6kvm7 corosync[15467]: [KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
Sep 3 10:43:24 m6kvm7 corosync[15467]: [TOTEM ] A new membership (4.834) was formed. Members joined: 4
Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:43:24 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 4/3965, 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: members: 4/3965, 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:43:24 m6kvm7 corosync[15467]: [QUORUM] Members[2]: 4 7
Sep 3 10:43:24 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 4/3965/00000002)
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 4/3965/00000002)
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 7/15892
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: start sending inode updates
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: sent all (4) updates
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:43:24 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] rx: host: 3 link: 0 is up
Sep 3 10:44:05 m6kvm7 corosync[15467]: [KNET ] host: host: 3 (passive) best link: 0 (pri: 1)
Sep 3 10:44:05 m6kvm7 corosync[15467]: [TOTEM ] A new membership (3.838) was formed. Members joined: 3
Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:05 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 3/3613, 4/3965, 7/15892
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: members: 3/3613, 4/3965, 7/15892
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:44:05 m6kvm7 corosync[15467]: [QUORUM] Members[3]: 3 4 7
Sep 3 10:44:05 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 3/3613/00000002)
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 3/3613/00000002)
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 4/3965
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 4/3965, 7/15892
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:44:05 m6kvm7 pmxcfs[15892]: [status] notice: dfsm_deliver_queue: queue length 1
Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] rx: host: 1 link: 0 is up
Sep 3 10:44:31 m6kvm7 corosync[15467]: [KNET ] host: host: 1 (passive) best link: 0 (pri: 1)
Sep 3 10:44:31 m6kvm7 corosync[15467]: [TOTEM ] A new membership (1.83c) was formed. Members joined: 1
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 corosync[15467]: [CPG ] downlist left_list: 0 received
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [dcdb] notice: starting data syncronisation
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: members: 1/3552, 3/3613, 4/3965, 7/15892
Sep 3 10:44:31 m6kvm7 pmxcfs[15892]: [status] notice: starting data syncronisation
Sep 3 10:44:31 m6kvm7 corosync[15467]: [QUORUM] Members[4]: 1 3 4 7
Sep 3 10:44:31 m6kvm7 corosync[15467]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received sync request (epoch 1/3552/00000002)
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received sync request (epoch 1/3552/00000002)
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: received all states
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: leader is 3/3613
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: synced members: 3/3613, 4/3965, 7/15892
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [dcdb] notice: all data is up to date
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: received all states
Sep 3 10:44:32 m6kvm7 pmxcfs[15892]: [status] notice: all data is up to date
Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:46:33 m6kvm7 pmxcfs[15892]: [status] notice: received log
Sep 3 10:49:18 m6kvm7 pmxcfs[15892]: [status] notice: received log
node8
-----
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm8 pmxcfs[24790]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm8 corosync[24361]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm8 corosync[24361]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm8 corosync[24361]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: received all states
Sep 3 10:39:17 m6kvm8 pmxcfs[24790]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm8 corosync[24361]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
Sep 3 10:40:19 m6kvm8 pmxcfs[24790]: [status] notice: received log
--> reboot
node9
-----
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm9 pmxcfs[22810]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm9 corosync[22340]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm9 corosync[22340]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm9 corosync[22340]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: received all states
Sep 3 10:39:17 m6kvm9 pmxcfs[22810]: [status] notice: all data is up to date
Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:21 m6kvm9 corosync[22340]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:19 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:20 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:21 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
Sep 3 10:40:22 m6kvm9 pmxcfs[22810]: [status] notice: received log
--> reboot
node10
------
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm10 pmxcfs[41940]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm10 corosync[41458]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm10 corosync[41458]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm10 corosync[41458]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: received all states
Sep 3 10:39:17 m6kvm10 pmxcfs[41940]: [status] notice: all data is up to date
Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:21 m6kvm10 corosync[41458]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
Sep 3 10:40:19 m6kvm10 pmxcfs[41940]: [status] notice: received log
--> reboot
node11
------
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm11 pmxcfs[12983]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm11 corosync[12455]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm11 corosync[12455]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm11 corosync[12455]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: received all states
Sep 3 10:39:17 m6kvm11 pmxcfs[12983]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm11 corosync[12455]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:19 m6kvm11 pmxcfs[12983]: [status] notice: received log
Sep 3 10:40:20 m6kvm11 pmxcfs[12983]: [status] notice: received log
node12
------
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm12 pmxcfs[44214]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm12 corosync[43716]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm12 corosync[43716]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm12 corosync[43716]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: received all states
Sep 3 10:39:17 m6kvm12 pmxcfs[44214]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm12 corosync[43716]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:19 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:20 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:21 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
Sep 3 10:40:22 m6kvm12 pmxcfs[44214]: [status] notice: received log
node13
------
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm13 corosync[39182]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm13 corosync[39182]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm13 corosync[39182]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:16 m6kvm13 pmxcfs[39678]: [dcdb] notice: cpg_send_message retried 1 times
Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: received all states
Sep 3 10:39:17 m6kvm13 pmxcfs[39678]: [status] notice: all data is up to date
Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:19 m6kvm13 corosync[39182]: [KNET ] host: host: 2 has no active links
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
Sep 3 10:40:19 m6kvm13 pmxcfs[39678]: [status] notice: received log
--> reboot
node14
------
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: members: 1/3527, 3/27373, 4/3903, 5/42272, 6/37248, 7/15892, 8/24790, 9/22810, 10/41940, 11/12983, 12/44214, 13/39678, 14/42930
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: starting data syncronisation
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [dcdb] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm14 pmxcfs[42930]: [status] notice: received sync request (epoch 1/3527/00000002)
Sep 3 10:39:16 m6kvm14 corosync[42413]: [TOTEM ] A new membership (1.82b) was formed. Members left: 2
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [CPG ] downlist left_list: 1 received
Sep 3 10:39:16 m6kvm14 corosync[42413]: [QUORUM] Members[13]: 1 3 4 5 6 7 8 9 10 11 12 13 14
Sep 3 10:39:16 m6kvm14 corosync[42413]: [MAIN ] Completed service synchronization, ready to provide service.
Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: received all states
Sep 3 10:39:17 m6kvm14 pmxcfs[42930]: [status] notice: all data is up to date
Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] link: host: 2 link: 0 is down
Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 (passive) best link: 0 (pri: 1)
Sep 3 10:39:20 m6kvm14 corosync[42413]: [KNET ] host: host: 2 has no active links
--> reboot
_______________________________________________
pve-devel mailing list
pve-devel at lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
More information about the pve-devel
mailing list