[PVE-User] Proxmox and glusterfs: VMs get corupted

[Christian Schoepplein]

Hi,

On Tue, May 30, 2023 at 06:32:11PM +0200, Christian Schoepplein via pve-user wrote:
>we are testing the current proxmox version with a glusterfs storage backend 
>and have a strange issue with file getting corupted inside the virtual 
>machines. For what reason ever from one moment to another binaries can not 
>longer be executed, scripts are damaged and so on. In the logs I get errors 
>like this:
>
>May 30 11:22:36 ns1 dockerd[1234]: time="2023-05-30T11:22:36.874765091+02:00" level=warning msg="Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: could not insert 'bridge': Exec format error\nmodprobe: ERROR: could not insert 'br_netfilter': Exec format error\ninsmod /lib/modules/5.15.0-72-generic/kernel/net/802/stp.ko \ninsmod /lib/modules/5.15.0-72-generic/kernel/net/802/stp.ko \n, error: exit status 1"
>
>On such a broken system a file brings the following:
>
>root at ns1:~# file /lib/modules/5.15.0-72-generic/kernel/net/802/stp.ko
>/lib/modules/5.15.0-72-generic/kernel/net/802/stp.ko: data
>root at ns1:~#
>
>On a normal system it looks like this:
>
>root at gluster1:~# file /lib/modules/5.15.0-72-generic/kernel/net/802/stp.ko
>/lib/modules/5.15.0-72-generic/kernel/net/802/stp.ko: ELF 64-bit LSB 
>relocatable, x86-64, version 1 (SYSV), BuildID[sha1]=1084f7cfcffbd4c607724fba287c0ea7fc5775
>root at gluster1:~#
>
>there are not only kernel modules afected. I saw the same behaviour for 
>scripts, icinga check modules, the sendmail binary and so on, I think it is 
>totaly random :-(.
>
>We have the problems with newly installed VMs, VMs cloned from a template 
>create on our proxmox host and with VMs which we used before with libvirtd 
>and migrated to our new proxmox machine. So IMHO it can not be related to 
>the way we create new virtual machines...

I got the hint to use aio=threads for the attached disks but this did not 
help. The data inside the virtual machines get still corrupted when 
performing much i/o.

Also I read and write many big files directly to our glusterfs and check the 
md5summs and all was fine.

Is proxmox with VMs with qcow2 disks with glusterfs as a storage backend 
working at all? Is anyone using such a setup?

Ciao and thanks,

  Chris