PVE 6.2 Strange cluster node fence

Wed Apr 14 11:04:10 CEST 2021

Hi all,

Yesterday we had a strange fence happen in a PVE 6.2 cluster.

Cluster has 3 nodes (proxmox1, proxmox2, proxmox3) and has been 
operating normally for a year. Last update was on January 21st 2021. 
Storage is Ceph and nodes are connected to the same network switch with 
active-pasive bonds.

proxmox1 was fenced and automatically rebooted, then everything 
recovered. HA restarted VMs in other nodes too.

proxmox1 syslog: (no network link issues reported at device level)
---
Apr 13 11:35:14 proxmox1 corosync[1410]:   [KNET  ] link: host: 3 link: 
0 is down
Apr 13 11:35:14 proxmox1 corosync[1410]:   [KNET  ] host: host: 3 
(passive) best link: 0 (pri: 1)
Apr 13 11:35:14 proxmox1 corosync[1410]:   [KNET  ] host: host: 3 has no 
active links
Apr 13 11:35:15 proxmox1 corosync[1410]:   [TOTEM ] Token has not been 
received in 61 ms
Apr 13 11:35:15 proxmox1 corosync[1410]:   [TOTEM ] A processor failed, 
forming new configuration.
Apr 13 11:35:18 proxmox1 corosync[1410]:   [KNET  ] rx: host: 3 link: 0 
is up
Apr 13 11:35:18 proxmox1 corosync[1410]:   [KNET  ] host: host: 3 
(passive) best link: 0 (pri: 1)
Apr 13 11:35:18 proxmox1 corosync[1410]:   [TOTEM ] Token has not been 
received in 3069 ms
Apr 13 11:35:19 proxmox1 corosync[1410]:   [TOTEM ] A new membership 
(2.5477) was formed. Members left: 1 3
Apr 13 11:35:19 proxmox1 corosync[1410]:   [TOTEM ] Failed to receive 
the leave message. failed: 1 3
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [dcdb] notice: members: 2/1398
Apr 13 11:35:19 proxmox1 corosync[1410]:   [QUORUM] This node is within 
the non-primary component and will NOT provide any services.
Apr 13 11:35:19 proxmox1 corosync[1410]:   [QUORUM] Members[1]: 2
Apr 13 11:35:19 proxmox1 corosync[1410]:   [MAIN  ] Completed service 
synchronization, ready to provide service.
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [status] notice: members: 2/1398
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [status] notice: node lost quorum
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [dcdb] crit: received write while 
not quorate - trigger resync
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [dcdb] crit: leaving CPG group
Apr 13 11:35:19 proxmox1 corosync[1410]:   [TOTEM ] A new membership 
(1.547b) was formed. Members joined: 1 3
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [status] notice: members: 1/1396, 
2/1398, 3/1457
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [status] notice: starting data 
syncronisation
Apr 13 11:35:19 proxmox1 corosync[1410]:   [QUORUM] This node is within 
the primary component and will provide service.
Apr 13 11:35:19 proxmox1 corosync[1410]:   [QUORUM] Members[3]: 1 2 3
Apr 13 11:35:19 proxmox1 corosync[1410]:   [MAIN  ] Completed service 
synchronization, ready to provide service.
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [status] notice: node has quorum
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [status] notice: received sync 
request (epoch 1/1396/00000006)
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [status] notice: received sync 
request (epoch 1/1396/00000007)
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [status] notice: received all states
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [status] notice: all data is up 
to date
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [dcdb] notice: start cluster 
connection
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_join failed: 14
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [dcdb] crit: can't initialize service
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:19 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:19 proxmox1 pve-ha-lrm[1770]: lost lock 
'ha_agent_proxmox1_lock - cfs lock update failed - Device or resource busy
Apr 13 11:35:20 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:21 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:22 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:23 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:23 proxmox1 pve-ha-lrm[1770]: status change active => 
lost_agent_lock
Apr 13 11:35:24 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 7 more ...]
Apr 13 11:35:25 proxmox1 corosync[1410]:   [QB    ] request returned 
error (/dev/shm/qb-1410-1398-33-nnAQGx/qb): Bad message (74)
Apr 13 11:35:27 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_local_get failed: 2
Apr 13 11:35:28 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:29 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:30 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:31 proxmox1 corosync[1410]:   [QB    ] request returned 
error (/dev/shm/qb-1410-1398-33-JDQj3Z/qb): Bad message (74)
Apr 13 11:35:33 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 2
Apr 13 11:35:33 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 2
Apr 13 11:35:33 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_local_get failed: 2
Apr 13 11:35:33 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 2
Apr 13 11:35:34 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:35 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:35 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:35 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:35 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:36 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 7 more ...]
Apr 13 11:35:37 proxmox1 corosync[1410]:   [QB    ] request returned 
error (/dev/shm/qb-1410-1398-33-jgBffR/qb): Bad message (74)
Apr 13 11:35:39 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_local_get failed: 2
Apr 13 11:35:39 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:39 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:40 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:41 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:42 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:42 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:42 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:42 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:43 proxmox1 corosync[1410]:   [QB    ] request returned 
error (/dev/shm/qb-1410-1398-33-dWqAg7/qb): Bad message (74)
Apr 13 11:35:45 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_local_get failed: 2
Apr 13 11:35:46 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:46 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:46 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:46 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:47 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:47 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:47 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:47 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:48 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 7 more ...]
Apr 13 11:35:49 proxmox1 corosync[1410]:   [QB    ] request returned 
error (/dev/shm/qb-1410-1398-33-LnKe7L/qb): Bad message (74)
Apr 13 11:35:51 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_local_get failed: 2
Apr 13 11:35:52 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:53 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 5 more ...]
Apr 13 11:35:54 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 9 more ...]
Apr 13 11:35:55 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 0
Apr 13 11:35:55 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 0
Apr 13 11:35:55 proxmox1 corosync[1410]:   [QB    ] request returned 
error (/dev/shm/qb-1410-1398-33-dXTlNP/qb): Bad message (74)
Apr 13 11:35:57 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_local_get failed: 2
Apr 13 11:35:58 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:58 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:58 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:58 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:59 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:59 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:59 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:35:59 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:36:00 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[... 9 more ...]
Apr 13 11:36:01 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:36:01 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:36:01 proxmox1 pvesr[938407]: trying to acquire cfs lock 
'file-replication_cfg' ...
Apr 13 11:36:01 proxmox1 corosync[1410]:   [QB    ] request returned 
error (/dev/shm/qb-1410-1398-33-17q0ii/qb): Bad message (74)
Apr 13 11:36:03 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 2
Apr 13 11:36:03 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 2
Apr 13 11:36:03 proxmox1 pvesr[938407]: trying to acquire cfs lock 
'file-replication_cfg' ...
Apr 13 11:36:03 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:36:03 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:36:03 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
Apr 13 11:36:03 proxmox1 pmxcfs[1398]: [dcdb] crit: cpg_send_message 
failed: 9
[reset garbage)
---

proxmox2 log:
---
Apr 13 11:35:15 proxmox2 corosync[1402]:   [KNET  ] link: host: 3 link: 
0 is down
Apr 13 11:35:15 proxmox2 corosync[1402]:   [KNET  ] host: host: 3 
(passive) best link: 0 (pri: 1)
Apr 13 11:35:15 proxmox2 corosync[1402]:   [KNET  ] host: host: 3 has no 
active links
Apr 13 11:35:15 proxmox2 corosync[1402]:   [TOTEM ] Token has not been 
received in 1237 ms
Apr 13 11:35:15 proxmox2 corosync[1402]:   [TOTEM ] A processor failed, 
forming new configuration.
Apr 13 11:35:17 proxmox2 corosync[1402]:   [KNET  ] rx: host: 3 link: 0 
is up
Apr 13 11:35:17 proxmox2 corosync[1402]:   [KNET  ] host: host: 3 
(passive) best link: 0 (pri: 1)
Apr 13 11:35:18 proxmox2 corosync[1402]:   [TOTEM ] Token has not been 
received in 4637 ms
Apr 13 11:35:19 proxmox2 corosync[1402]:   [TOTEM ] A new membership 
(1.5477) was formed. Members left: 2
Apr 13 11:35:19 proxmox2 corosync[1402]:   [TOTEM ] Failed to receive 
the leave message. failed: 2
Apr 13 11:35:19 proxmox2 corosync[1402]:   [TOTEM ] A new membership 
(1.547b) was formed. Members joined: 2
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [dcdb] notice: members: 1/1396, 
3/1457
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [dcdb] notice: starting data 
syncronisation
Apr 13 11:35:19 proxmox2 corosync[1402]:   [QUORUM] Members[3]: 1 2 3
Apr 13 11:35:19 proxmox2 corosync[1402]:   [MAIN  ] Completed service 
synchronization, ready to provide service.
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [dcdb] notice: cpg_send_message 
retried 1 times
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [status] notice: members: 1/1396, 
3/1457
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [status] notice: starting data 
syncronisation
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [status] notice: members: 1/1396, 
2/1398, 3/1457
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [dcdb] notice: received sync 
request (epoch 1/1396/00000006)
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [status] notice: received sync 
request (epoch 1/1396/00000006)
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [status] notice: received sync 
request (epoch 1/1396/00000007)
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [dcdb] notice: received all states
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [dcdb] notice: leader is 1/1396
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [dcdb] notice: synced members: 
1/1396, 3/1457
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [dcdb] notice: start sending 
inode updates
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [dcdb] notice: sent all (0) updates
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [dcdb] notice: all data is up to date
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [dcdb] notice: 
dfsm_deliver_queue: queue length 4
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [status] notice: received all states
Apr 13 11:35:19 proxmox2 pmxcfs[1396]: [status] notice: all data is up 
to date
Apr 13 11:36:00 proxmox2 systemd[1]: Starting Proxmox VE replication 
runner...
Apr 13 11:36:00 proxmox2 systemd[1]: pvesr.service: Succeeded.
Apr 13 11:36:00 proxmox2 systemd[1]: Started Proxmox VE replication runner.
Apr 13 11:36:13 proxmox2 pmxcfs[1396]: [dcdb] notice: members: 1/1396, 
2/1398, 3/1457
Apr 13 11:36:13 proxmox2 pmxcfs[1396]: [dcdb] notice: starting data 
syncronisation
Apr 13 11:36:13 proxmox2 pmxcfs[1396]: [dcdb] notice: received sync 
request (epoch 1/1396/00000007)
Apr 13 11:36:13 proxmox2 pmxcfs[1396]: [dcdb] notice: received all states
Apr 13 11:36:13 proxmox2 pmxcfs[1396]: [dcdb] notice: leader is 1/1396
Apr 13 11:36:13 proxmox2 pmxcfs[1396]: [dcdb] notice: synced members: 
1/1396, 3/1457
Apr 13 11:36:13 proxmox2 pmxcfs[1396]: [dcdb] notice: start sending 
inode updates
Apr 13 11:36:13 proxmox2 pmxcfs[1396]: [dcdb] notice: sent all (8) updates
Apr 13 11:36:13 proxmox2 pmxcfs[1396]: [dcdb] notice: all data is up to date
Apr 13 11:36:25 proxmox2 corosync[1402]:   [KNET  ] link: host: 2 link: 
0 is down
Apr 13 11:36:25 proxmox2 corosync[1402]:   [KNET  ] host: host: 2 
(passive) best link: 0 (pri: 1)
Apr 13 11:36:25 proxmox2 corosync[1402]:   [KNET  ] host: host: 2 has no 
active links
Apr 13 11:36:26 proxmox2 corosync[1402]:   [TOTEM ] Token has not been 
received in 61 ms
Apr 13 11:36:26 proxmox2 corosync[1402]:   [TOTEM ] A processor failed, 
forming new configuration.
Apr 13 11:36:28 proxmox2 corosync[1402]:   [TOTEM ] A new membership 
(1.547f) was formed. Members left: 2
Apr 13 11:36:28 proxmox2 corosync[1402]:   [TOTEM ] Failed to receive 
the leave message. failed: 2
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [dcdb] notice: members: 1/1396, 
3/1457
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [dcdb] notice: starting data 
syncronisation
Apr 13 11:36:28 proxmox2 corosync[1402]:   [QUORUM] Members[2]: 1 3
Apr 13 11:36:28 proxmox2 corosync[1402]:   [MAIN  ] Completed service 
synchronization, ready to provide service.
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [dcdb] notice: cpg_send_message 
retried 1 times
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [status] notice: members: 1/1396, 
3/1457
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [status] notice: starting data 
syncronisation
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [dcdb] notice: received sync 
request (epoch 1/1396/00000008)
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [status] notice: received sync 
request (epoch 1/1396/00000008)
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [dcdb] notice: received all states
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [dcdb] notice: leader is 1/1396
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [dcdb] notice: synced members: 
1/1396, 3/1457
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [dcdb] notice: start sending 
inode updates
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [dcdb] notice: sent all (0) updates
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [dcdb] notice: all data is up to date
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [dcdb] notice: 
dfsm_deliver_queue: queue length 2
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [status] notice: received all states
Apr 13 11:36:28 proxmox2 pmxcfs[1396]: [status] notice: all data is up 
to date
Apr 13 11:36:29 proxmox2 pve-ha-crm[1801]: node 'proxmox1': state 
changed from 'online' => 'unknown'
Apr 13 11:36:38 proxmox2 pvestatd[1553]: got timeout
Apr 13 11:36:38 proxmox2 pvestatd[1553]: status update time (5.090 seconds)
Apr 13 11:36:45 proxmox2 ceph-osd[1424]: 2021-04-13 11:36:45.407 
7f94513df700 -1 osd.2 1166 heartbeat_check: no reply from 
192.168.91.11:6820 osd.0 since back 2021-04-13 11:36:23.684429 front 
2021-04-13 11:36:23.684422 (oldest deadline 2021-04-13 11:36:44.784447)
---

all 3 nodes have the same running Proxmox versions:

root at proxmox1:~# pveversion -v
proxmox-ve: 6.3-1 (running kernel: 5.4.78-2-pve)
pve-manager: 6.3-3 (running version: 6.3-3/eee5f901)
pve-kernel-5.4: 6.3-3
pve-kernel-helper: 6.3-3
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph: 14.2.16-pve1
ceph-fuse: 14.2.16-pve1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.7
libproxmox-backup-qemu0: 1.0.2-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.6-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-3
pve-cluster: 6.2-1
pve-container: 3.3-2
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-7
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

We are upgrading the cluster in the next days as part of our 3-month 
upgrade cycle, but can wait.

Any ideas? Could this be a bug?

Thanks a lot

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/