PVE 7.4 Windows VM hang

[Eneko Lacunza]

Hi all,

We have been experiencing Windows VM hangs during the last weeks, in 
previously stable cluster/VMs.

So far we have seen a Windows 7 (yes I know!) and a Windows 2016 Std 
guest crash with 100% multi-cpu use. A hard stop and start leaves guest 
working again.

- Windows 7 guest has crashed several times:
  * June, 18th
  * June, 25th
  * July 4th

- Windows 2016 guest has crashed once:
  * June 23th

There are more guests in the cluster; 5 linux and 1 Windows 2016; none 
have crashed (all were running in other nodes).

Both guest were running on the same physical node, a Dell T350 server 
with E-2356G and 32GB RAM. We have moved both to another node (with 
E3-1220 v6 CPU) just to see if the issue is with the node; no crashes 
since July, 5th but it's too soon to be sure.

Storage is Ceph v16.2.13, backed by HDDs with SSD partition for 
journal/db (there is a mixture of filestore and bluestore OSDs).

Originally node was running since April, 20th:
* pve-kernel 6.2.9-1-pve
* qemu-server 7.4-3
* pve-qemu-kvm 7.2.0-8

on June 26th we tried upgrading:
* pve-kernel 6.2.11-2-pve
* qemu-server 7.4-4

VM conf:

boot: order=ide0;ide2
cores: 4
ide0: rbd:vm-104-disk-0,cache=writeback,size=30G
ide2: nas-backups2:iso/virtio-win-0.1.126.iso,media=cdrom,size=152204K
machine: pc-i440fx-6.2
memory: 12288
meta: creation-qemu=6.2.0,ctime=1651156825
name: Windows7
net0: virtio=7E:0E:12:3D:0A:89,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win10
parent: Snapshot_02_05_2022
scsi1: rbd:vm-104-disk-2,cache=writeback,size=40G
scsihw: virtio-scsi-pci
smbios1: uuid=1006a846-b056-4ce3-bd6f-7d16cfb5f573
sockets: 1
vmgenid: 27764cc2-dcab-405f-be80-bd6630ba75ba

---

boot: cdn
bootdisk: scsi0
cores: 4
ide2: none,media=cdrom
ide3: none,media=cdrom
memory: 8192
name: Windows2016
net0: virtio=3A:F9:3C:9F:70:FE,bridge=vmbr0
numa: 0
onboot: 1
ostype: win10
scsi0: rbd:vm-110-disk-1,cache=writeback,discard=on,size=110G
scsi1: rbd:vm-110-disk-2,cache=writeback,discard=on,size=30G
scsihw: virtio-scsi-pci
smbios1: uuid=3e82d96a-464a-483d-b178-2359d24333db
sockets: 1


Currently:
# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 6.2.11-2-pve)
pve-manager: 7.4-15 (running version: 7.4-15/a5d2a31e)
pve-kernel-5.15: 7.4-4
pve-kernel-6.2: 7.4-3
pve-kernel-5.13: 7.1-9
pve-kernel-6.2.11-2-pve: 6.2.11-2
pve-kernel-6.2.9-1-pve: 6.2.9-1
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.104-1-pve: 5.15.104-2
pve-kernel-5.15.30-2-pve: 5.15.30-3
pve-kernel-5.13.19-6-pve: 5.13.19-15
ceph: 16.2.13-pve1
ceph-fuse: 16.2.13-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.2
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-4
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1


Any ideas?

Thanks

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 | https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/