[PVE-User] Cluster won't reform so I can't restart VMs

[Chris Tomkins]

Hi Proxmox users,

I have a 4 node cluster. It has been in production for a few months with
few/no issues.

This morning one of my admins reported that each node appeared isolated
("Total votes:      1"). All VMs were up and unaffected. Unfortunately I
made the mistake of stopping VMs on 3 of the nodes to apply updates and
reboot as I assumed this would clear the issue. Now the scenario remains
the same but the VMs are down on 3 of the nodes and it won't allow me to
start them as I have no quorum.

No config changes were made and this cluster was fine and had quorum last
time I looked (last week).

I don't want to take the wrong action and make this worse - any advice
would be greatly appreciated!

hypervisors ar1406/ar1600/ar1601 are up to date and have been rebooted this
morning. ar1407 has not been rebooted or updated (yet) as the VMs on it are
critical.

Thanks,

Chris

[LIVE]root at ar1406:~# for i in ar1406 ar1407 ar1600 ar1601; do ssh $i 'cat
/etc/pve/.members'; done
{
"nodename": "ar1406",
"version": 3,
"cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate":
0 },
"nodelist": {
  "ar1407": { "id": 2, "online": 0},
  "ar1601": { "id": 3, "online": 0},
  "ar1600": { "id": 4, "online": 0},
  "ar1406": { "id": 1, "online": 1, "ip": "10.0.6.201"}
  }
}
{
"nodename": "ar1407",
"version": 3,
"cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate":
0 },
"nodelist": {
  "ar1407": { "id": 2, "online": 1, "ip": "10.0.6.202"},
  "ar1601": { "id": 3, "online": 0},
  "ar1600": { "id": 4, "online": 0},
  "ar1406": { "id": 1, "online": 0}
  }
}
{
"nodename": "ar1600",
"version": 3,
"cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate":
0 },
"nodelist": {
  "ar1407": { "id": 2, "online": 0},
  "ar1601": { "id": 3, "online": 0},
  "ar1600": { "id": 4, "online": 1, "ip": "10.0.6.203"},
  "ar1406": { "id": 1, "online": 0}
  }
}
{
"nodename": "ar1601",
"version": 3,
"cluster": { "name": "netteamcluster", "version": 4, "nodes": 4, "quorate":
0 },
"nodelist": {
  "ar1407": { "id": 2, "online": 0},
  "ar1601": { "id": 3, "online": 1, "ip": "10.0.6.204"},
  "ar1600": { "id": 4, "online": 0},
  "ar1406": { "id": 1, "online": 0}
  }
}

-- 

Chris Tomkins

Brandwatch | Senior Network Engineer (Linux/Network)

christ at brandwatch.com | (+44) 01273 448 949

@Brandwatch

New York  |  San Francisco  |  Brighton  |  Singapore  |  Berlin |
 Stuttgart


Discover how organizations are using Brandwatch to create their own success
<https://www.brandwatch.com/customer-success/>


Email disclaimer <http://www.brandwatch.com/email-disclaimer/>


[image: bw-signature logo.png]