[PVE-User] Adding a cluster node breaks whole cluster

[Sten Aus]

Okay, I have now updated all nodes to same kernel versions. Thanks to 
the bugfix with latest update, IPv6 now works with newer kernel as well. 
So, all nodes are in the same version and packgages.

Anyway, now I cannot add no new node. Even I can't add this old node, 
which was in the cluster before as a new node.

So I added debug flag to cluster.conf and tried to add new node. Here's 
the output:
> Apr  9 13:03:06 rabaja pmxcfs[673283]: [status] notice: 
> cpg_send_message retry 10 (dfsm.c:215:dfsm_send_message_full)
> Apr  9 13:03:07 rabaja pmxcfs[673283]: [status] notice: 
> cpg_send_message retry 20 (dfsm.c:215:dfsm_send_message_full)
> Apr  9 13:03:08 rabaja pmxcfs[673283]: [status] notice: 
> cpg_send_message retry 30 (dfsm.c:215:dfsm_send_message_full)
> Apr  9 13:03:09 rabaja pmxcfs[673283]: [status] notice: 
> cpg_send_message retry 40 (dfsm.c:215:dfsm_send_message_full)
> Apr  9 13:03:10 rabaja pmxcfs[673283]: [status] notice: 
> cpg_send_message retry 50 (dfsm.c:215:dfsm_send_message_full)
> Apr  9 13:03:11 rabaja pmxcfs[673283]: [status] notice: 
> cpg_send_message retry 60 (dfsm.c:215:dfsm_send_message_full)
> Apr  9 13:03:12 rabaja pmxcfs[673283]: [status] notice: 
> cpg_send_message retry 70 (dfsm.c:215:dfsm_send_message_full)
> Apr  9 13:03:13 rabaja pmxcfs[673283]: [status] notice: 
> cpg_send_message retry 80 (dfsm.c:215:dfsm_send_message_full)
> Apr  9 13:03:14 rabaja pmxcfs[673283]: [status] notice: 
> cpg_send_message retry 90 (dfsm.c:215:dfsm_send_message_full)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [status] notice: 
> cpg_send_message retry 100 (dfsm.c:215:dfsm_send_message_full)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [status] notice: 
> cpg_send_message retried 100 times (dfsm.c:221:dfsm_send_message_full)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [status] crit: cpg_send_message 
> failed: 6 (dfsm.c:329:dfsm_send_message_sync)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process result 0 
> (server.c:310:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process msg:1, 
> size:16 (server.c:160:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process result 0 
> (server.c:310:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process msg:5, 
> size:528 (server.c:160:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process result -2 
> (server.c:310:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process msg:1, 
> size:16 (server.c:160:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process result 0 
> (server.c:310:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process msg:1, 
> size:16 (server.c:160:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process result 0 
> (server.c:310:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process msg:1, 
> size:16 (server.c:160:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process result 0 
> (server.c:310:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process msg:5, 
> size:528 (server.c:160:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process result 0 
> (server.c:310:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [ipcs] debug: process msg:4, 
> size:421 (server.c:160:s1_msg_process_fn)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_fuse_getattr / (pmxcfs.c:126:cfs_fuse_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug start  
> (pmxcfs.c:102:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: 
> cfs_plug_base_lookup_plug (cfs-plug.c:52:cfs_plug_base_lookup_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug end = 
> 0xd9d280 () (pmxcfs.c:109:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_plug_base_getattr  (cfs-plug.c:84:cfs_plug_base_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: leave 
> cfs_plug_base_getattr  (cfs-plug.c:103:cfs_plug_base_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: leave 
> cfs_fuse_getattr / (0) (pmxcfs.c:144:cfs_fuse_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_fuse_getattr /firewall (pmxcfs.c:126:cfs_fuse_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug start 
> firewall (pmxcfs.c:102:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: 
> cfs_plug_base_lookup_plug firewall 
> (cfs-plug.c:52:cfs_plug_base_lookup_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: 
> cfs_plug_base_lookup_plug name = firewall new path = (null) 
> (cfs-plug.c:59:cfs_plug_base_lookup_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug end 
> firewall = 0xd9d280 (firewall) (pmxcfs.c:109:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_plug_base_getattr firewall (cfs-plug.c:84:cfs_plug_base_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: leave 
> cfs_plug_base_getattr firewall (cfs-plug.c:103:cfs_plug_base_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: leave 
> cfs_fuse_getattr /firewall (0) (pmxcfs.c:144:cfs_fuse_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_fuse_getattr /firewall/cluster.fw (pmxcfs.c:126:cfs_fuse_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug start 
> firewall/cluster.fw (pmxcfs.c:102:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: 
> cfs_plug_base_lookup_plug firewall/cluster.fw 
> (cfs-plug.c:52:cfs_plug_base_lookup_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: 
> cfs_plug_base_lookup_plug name = firewall new path = cluster.fw 
> (cfs-plug.c:59:cfs_plug_base_lookup_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug end 
> firewall/cluster.fw = 0xd9d280 (firewall/cluster.fw) 
> (pmxcfs.c:109:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_plug_base_getattr firewall/cluster.fw 
> (cfs-plug.c:84:cfs_plug_base_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: leave 
> cfs_plug_base_getattr firewall/cluster.fw 
> (cfs-plug.c:103:cfs_plug_base_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: leave 
> cfs_fuse_getattr /firewall/cluster.fw (0) (pmxcfs.c:144:cfs_fuse_getattr)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_fuse_open /firewall/cluster.fw (pmxcfs.c:260:cfs_fuse_open)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug start 
> firewall/cluster.fw (pmxcfs.c:102:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: 
> cfs_plug_base_lookup_plug firewall/cluster.fw 
> (cfs-plug.c:52:cfs_plug_base_lookup_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: 
> cfs_plug_base_lookup_plug name = firewall new path = cluster.fw 
> (cfs-plug.c:59:cfs_plug_base_lookup_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug end 
> firewall/cluster.fw = 0xd9d280 (firewall/cluster.fw) 
> (pmxcfs.c:109:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_plug_base_open firewall/cluster.fw 
> (cfs-plug.c:248:cfs_plug_base_open)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: leave 
> cfs_fuse_open /firewall/cluster.fw (0) (pmxcfs.c:276:cfs_fuse_open)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_fuse_read /firewall/cluster.fw 8192 0 (pmxcfs.c:289:cfs_fuse_read)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug start 
> firewall/cluster.fw (pmxcfs.c:102:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: 
> cfs_plug_base_lookup_plug firewall/cluster.fw 
> (cfs-plug.c:52:cfs_plug_base_lookup_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: 
> cfs_plug_base_lookup_plug name = firewall new path = cluster.fw 
> (cfs-plug.c:59:cfs_plug_base_lookup_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug end 
> firewall/cluster.fw = 0xd9d280 (firewall/cluster.fw) 
> (pmxcfs.c:109:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_plug_base_read firewall/cluster.fw 8192 0 
> (cfs-plug.c:271:cfs_plug_base_read)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: leave 
> cfs_fuse_read /firewall/cluster.fw (47) (pmxcfs.c:301:cfs_fuse_read)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_fuse_read /firewall/cluster.fw 8192 47 (pmxcfs.c:289:cfs_fuse_read)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug start 
> firewall/cluster.fw (pmxcfs.c:102:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: 
> cfs_plug_base_lookup_plug firewall/cluster.fw 
> (cfs-plug.c:52:cfs_plug_base_lookup_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: 
> cfs_plug_base_lookup_plug name = firewall new path = cluster.fw 
> (cfs-plug.c:59:cfs_plug_base_lookup_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: find_plug end 
> firewall/cluster.fw = 0xd9d280 (firewall/cluster.fw) 
> (pmxcfs.c:109:find_plug)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: enter 
> cfs_plug_base_read firewall/cluster.fw 8192 47 
> (cfs-plug.c:271:cfs_plug_base_read)
> Apr  9 13:03:15 rabaja pmxcfs[673283]: [main] debug: leave 
> cfs_fuse_read /firewall/cluster.fw (0) (pmxcfs.c:301:cfs_fuse_read)
Is there anything I can provide to you for debug information?

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3242 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.proxmox.com/pipermail/pve-user/attachments/20150409/11b8eaed/attachment.bin>