[PVE-User] Upgaded all nodes. Now two of them will not rejoin cluster

[Ben Hambleton]

I just upgraded all my cluster nodes and rebooted. Upon reboot two of them
will not join the cluster.
When I run pvecm status on the ones that are seeing each other I get:
Quorum information
------------------
Date:  Sun May 22 21:50:10 2016
Quorum provider:  corosync_votequorum
Nodes:  4
Node ID:  0x00000004
Ring ID:  28712
Quorate:  Yes

Votequorum information
----------------------
Expected votes:  6
Highest expected: 6
Total votes:  4
Quorum:  4
Flags:  Quorate

Membership information
----------------------
  Nodeid  Votes Name
0x00000001  1 10.1.1.50
0x00000004  1 10.1.1.51 (local)
0x00000003  1 10.1.1.52
0x00000002  1 10.1.1.54

When I run pvecm status on the ones that aren't seeing each other I get:
Quorum information
------------------
Date:  Sun May 22 21:49:33 2016
Quorum provider:  corosync_votequorum
Nodes:  1
Node ID:  0x00000005
Ring ID:  26872
Quorate:  No

Votequorum information
----------------------
Expected votes:  6
Highest expected: 6
Total votes:  1
Quorum:  4 Activity blocked
Flags:

Membership information
----------------------
  Nodeid  Votes Name
0x00000005  1 10.1.1.53 (local)

I can ssh and ping from all nodes to each other (including the ones that
are not joining the cluster). I have restarted the cluster on the nodes
that can't connect via: /etc/init.d/pve-cluster restart to no avail with no
error. I did check the syslog and got the following error i have never seen
before: Totem is unable to form a cluster because of an operating system or
network fault. The most common cause of this message is that the local
firewall is configured improperly.

Any thoughts or ideas would be greatly appreciated.
Thank you,
Ben




-- 

---------------- Ben Hambleton, Director of Technology ----------------
Tower Hill School, 2813 West 17th Street, Wilmington, Delaware 19806
             phone 302.575.0550   ext. 269       fax 302.657.8368
--------------------- web site:http://www.towerhill.org -----------
-----------