[pve-devel] avoidable writes of pmxcfs to /var/lib/pve-cluster/config.db ?

[Thomas Lamprecht]

Hi,

On 09.03.21 21:45, Roland wrote:
> hello proxmox team,
> 
> i found that pmxcfs process is quite "chatty" and one of the top
> disk-writers on our proxmox nodes.
> 
> i had a closer look, because i was curious, why wearout of our samsung
> EVO is already at 4% .  as disk I/O of our vms is typically very low, so
> we used lower end ssd for those maschines.

FWIW, my crucial MX200 512GB disks is at 3% wear out, running as PVE
root FS since 5.5 years.

> 
> it seems pmxcfs is constantly writing into config.db-wal at a >10 kB/s
> and >10 writes/s rate level, whereas i can only see few changes in config.db
> 
> from my rough calculation, these writes probably sum up to several
> hundreds of gigabytes of disk blocks and >100mio iops written in a year,
> which isn't "just nothing" for lower-end ssd  (small and cheap ssd's may
> only have some tens of TBW lifetime).
> 
> i know that it's recommended to use enterprise ssd for proxmox, but as
> they are expensive i also dislike if they get avoidable wearout on any
> of our systems.
> 
> 
> what makes me raise my eyebrowe is, that it seems that most of the data
> written to the sqlite db seems to be unchanged data, i.e. i don't see
> significant changes in config.db over time, (compared with sqldiff),
> whereas the write-ahead-log at config.db-wal has quite high "flow rate".
> 
> I cannot decide if this really is a must have, but it looks that writing
> of (at least parts of)  the cluster runtime data (like rsa key
> information) is being done in a "just dump it all down into the
> database" way. this may make it easy at the implementation level and
> easy for the programmer.
> 
> i would love to hear a comment on this finding .
> 
> maybe there is will/room for optimisation to avoid unnecessary disk
> wearout, saving avoidable database write/workload (ok it's tiny
> workload)  , but thus probably also lower the risk of database
> corruption in particular problem situations like server crash or whatever.

So the prime candidate for this write load are the PVE HA Local Resource
Manager services on each node, they update their status and that is often
required to signal the current Cluster Resource Manager's master service
that the HA stack on that node is well alive and that commands got
executed with result X. So yes, this is required and intentional.
There maybe some room for optimization, but its not that straight forward,
and (over-)clever solutions are often the wrong ones for an HA stack - as
failure here is something we really want to avoid. But yeah, some easier
to pick fruits could maybe be found here.

The other thing I just noticed when checking out:
# ls -l "/proc/$(pidof pmxcfs)/fd"

to get the FDs for all db related FDs and then watch writes with:
# strace -v -s $[1<<16] -f -p "$(pidof pmxcfs)" -e write=4,5,6

Was seeing additionally some writes for the RSA key files which should just
not be there, but I need to closer investigate this, seemed a bit too odd 
to
me.

I'll see if I can find out a bit more details about above, maybe there's
something to improve lurking there.

FWIW, in general we try to keep stuff rather simple, the main reason is that
simpler systems tend to work more reliable and are easier to maintain, and
the load of even simple services can still get quite complex in sum, like 
in
PVE; But we still try to avoid efficiency trade offs over oversimplification.

cheers,
Thomas