[PVE-User] Cluster won't reform so I can't restart VMs

[Yannis Milios]

Since there were no config changes,  I would have a look on cluster
communication, i.e switch issues ?



On Fri, Aug 11, 2017 at 11:02 AM, Chris Tomkins <christ at brandwatch.com>
wrote:

> Hi Proxmox users,
>
> I have a 4 node cluster. It has been in production for a few months with
> few/no issues.
>
> This morning one of my admins reported that each node appeared isolated
> ("Total votes:      1"). All VMs were up and unaffected. Unfortunately I
> made the mistake of stopping VMs on 3 of the nodes to apply updates and
> reboot as I assumed this would clear the issue. Now the scenario remains
> the same but the VMs are down on 3 of the nodes and it won't allow me to
> start them as I have no quorum.
>
> No config changes were made and this cluster was fine and had quorum last
> time I looked (last week).
>
> I don't want to take the wrong action and make this worse - any advice
> would be greatly appreciated!
>
> hypervisors ar1406/ar1600/ar1601 are up to date and have been rebooted this
> morning. ar1407 has not been rebooted or updated (yet) as the VMs on it are
> critical.
>
> Thanks,
>
> Chris
>
> [LIVE]root at ar1406:~# for i in ar1406 ar1407 ar1600 ar1601; do ssh $i 'cat
> /etc/pve/.members'; done
> {
> "nodename": "ar1406",
> "version": 3,
> "cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate":
> 0 },
> "nodelist": {
>   "ar1407": { "id": 2, "online": 0},
>   "ar1601": { "id": 3, "online": 0},
>   "ar1600": { "id": 4, "online": 0},
>   "ar1406": { "id": 1, "online": 1, "ip": "10.0.6.201"}
>   }
> }
> {
> "nodename": "ar1407",
> "version": 3,
> "cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate":
> 0 },
> "nodelist": {
>   "ar1407": { "id": 2, "online": 1, "ip": "10.0.6.202"},
>   "ar1601": { "id": 3, "online": 0},
>   "ar1600": { "id": 4, "online": 0},
>   "ar1406": { "id": 1, "online": 0}
>   }
> }
> {
> "nodename": "ar1600",
> "version": 3,
> "cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate":
> 0 },
> "nodelist": {
>   "ar1407": { "id": 2, "online": 0},
>   "ar1601": { "id": 3, "online": 0},
>   "ar1600": { "id": 4, "online": 1, "ip": "10.0.6.203"},
>   "ar1406": { "id": 1, "online": 0}
>   }
> }
> {
> "nodename": "ar1601",
> "version": 3,
> "cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate":
> 0 },
> "nodelist": {
>   "ar1407": { "id": 2, "online": 0},
>   "ar1601": { "id": 3, "online": 1, "ip": "10.0.6.204"},
>   "ar1600": { "id": 4, "online": 0},
>   "ar1406": { "id": 1, "online": 0}
>   }
> }
>
> --
>
> Chris Tomkins
>
> Brandwatch | Senior Network Engineer (Linux/Network)
>
> christ at brandwatch.com | (+44) 01273 448 949
>
> @Brandwatch
>
> New York  |  San Francisco  |  Brighton  |  Singapore  |  Berlin |
>  Stuttgart
>
>
> Discover how organizations are using Brandwatch to create their own success
> <https://www.brandwatch.com/customer-success/>
>
>
> Email disclaimer <http://www.brandwatch.com/email-disclaimer/>
>
>
> [image: bw-signature logo.png]
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>