[PVE-User] Service "systemd-tmpfiles-setup" fails since upgrade to PVE 7.x

[Frank Thommen]

Dear all,

since the upgrade of our PVE environment from 6.x to 7.2-4 last week, 
our monitoring (CheckMK) registers, that the service 
"systemd-tmpfiles-setup" fails in all containers (not in the VMs, 
though).  The containers have been created from the provided CentOS 7.9 
templates and are fully updated. The containers have all been created 
under PVE 5.x and 6.x.

I'm not familiar with "systemd-tmpfiles-setup" and I could not relate 
similar failure reports found in the net to our situation.

The status that we get is:
--------------------------------------------------------------
[root at odcf-vm119 ~]# systemctl status systemd-tmpfiles-setup.service
● systemd-tmpfiles-setup.service - Create Volatile Files and Directories
    Loaded: loaded 
(/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static; vendor 
preset: disabled)
    Active: failed (Result: exit-code) since Tue 2022-06-07 12:28:04 
CEST; 23h ago
      Docs: man:tmpfiles.d(5)
            man:systemd-tmpfiles(8)
   Process: 55 ExecStart=/usr/bin/systemd-tmpfiles --create --remove 
--boot --exclude-prefix=/dev (code=exited, status=1/FAILURE)
  Main PID: 55 (code=exited, status=1/FAILURE)

Jun 07 12:28:03 odcf-vm119 systemd[1]: Starting Create Volatile Files 
and Directories...
Jun 07 12:28:03 odcf-vm119 systemd-tmpfiles[55]: Failed to create 
directory or subvolume "/var/lib/machines": Operation not permitted
Jun 07 12:28:04 odcf-vm119 systemd-tmpfiles[55]: Failed to create 
directory or subvolume "/tmp": Operation not permitted
Jun 07 12:28:04 odcf-vm119 systemd-tmpfiles[55]: Failed to create 
directory or subvolume "/var/tmp": Operation not permitted
Jun 07 12:28:04 odcf-vm119 systemd[1]: systemd-tmpfiles-setup.service: 
main process exited, code=exited, status=1/FAILURE
Jun 07 12:28:04 odcf-vm119 systemd[1]: Failed to start Create Volatile 
Files and Directories.
Jun 07 12:28:04 odcf-vm119 systemd[1]: Unit 
systemd-tmpfiles-setup.service entered failed state.
Jun 07 12:28:04 odcf-vm119 systemd[1]: systemd-tmpfiles-setup.service 
failed.
[root at odcf-vm119 ~]#
--------------------------------------------------------------

journalctl tells me (manual extract from `journalctl -t systemd`):
--------------------------------------------------------------
Jun 07 12:28:03 odcf-vm119 systemd[1]: Starting Create Volatile Files 
and Directories...
Jun 07 12:28:04 odcf-vm119 systemd[1]: Started Load/Save Random Seed.
Jun 07 12:28:04 odcf-vm119 systemd[1]: systemd-tmpfiles-setup.service: 
main process exited, code=exited, status=1/FAILURE
Jun 07 12:28:04 odcf-vm119 systemd[1]: Failed to start Create Volatile 
Files and Directories.
Jun 07 12:28:04 odcf-vm119 systemd[1]: Unit 
systemd-tmpfiles-setup.service entered failed state.
Jun 07 12:28:04 odcf-vm119 systemd[1]: systemd-tmpfiles-setup.service 
failed.
--------------------------------------------------------------

and (`journalctl -t systemd-tmpfiles`):
--------------------------------------------------------------
-- Logs begin at Tue 2022-06-07 12:28:03 CEST, end at Wed 2022-06-08 
12:00:47 CEST. --
Jun 07 12:28:03 odcf-vm119 systemd-tmpfiles[55]: Failed to create 
directory or subvolume "/var/lib/machines": Operation not permitted
Jun 07 12:28:04 odcf-vm119 systemd-tmpfiles[55]: Failed to create 
directory or subvolume "/tmp": Operation not permitted
Jun 07 12:28:04 odcf-vm119 systemd-tmpfiles[55]: Failed to create 
directory or subvolume "/var/tmp": Operation not permitted
--------------------------------------------------------------


Can anyone tell, what the problem could be or in which direction we 
should try to search for the problem?


I might also just migrate all containers to VMs, if that is a good or 
recommended solution to get rid of this problem.  I currently don't see 
the benefit of containers over VMs anyway at this moment :-)

Cheers and thanks in advance
Frank