[pve-devel] [PATCH many v4 00/31] Expand and migrate RRD data and add/change summary graphs

[Lukas Wagner]

On Sat Jul 26, 2025 at 3:05 AM CEST, Aaron Lauterer wrote:
> This patch series does a few things. It expands the RRD format for nodes and
>  VMs. For all types (nodes, VMs, storage) we adjust the aggregation to align
>  them with the way they are done on the Backup Server. Therefore, we have new
>  RRD defitions for all 3 types.
>
> New values are added for nodes and VMs. In particular:
>
> Nodes:
> * memfree
> * arcsize
> * pressures:
>   * cpu some
>   * io some
>   * io full
>   * mem some
>   * mem full
>
> VMs:
> * memhost (memory consumption of all processes in the guests cgroup, host view)
> * pressures:
>   * cpu some
>   * cpu full
>   * io some
>   * io full
>   * mem some
>   * mem full
>
> The change in RRD columns and aggregation means, that we need new RRD files. To
> not lose old RRD data, we need to migrate the old RRD files to the ones with
> the new schema. Some initial performance tests showed that migrating 10k VM
> RRD files took ~2m40s single threaded. This is way to long to do it within the
> pmxcfs itself. Therefore this will be a dedicated step:
> The new `proxmox-rrd-migration-tool` migrates the RRD files to the new location
> and aggregation schemas. It is run automatically by the postinst script of the
> pve-manager.
>
> This also means, that we need to handle the situation of new and old RRD
> files and formats. Therefore we introduce new keys by which the metrics
> are broadcast in a cluster. Up until now (pre PVE9), it is in the format of
> 'pve2-{type}/{resource id}'.
> Having the version number this early in the string makes it tough to match
> against newer ones, especially in the C code of the pmxcfs. To make it easier
> in the future, we change the key format to 'pve-{type}-{version}/{resource id}'.
> This way, we can fuzzy match against unknown 'pve-{type}-{version}' in the C
> code too and handle those situations better.
>
> The result is, that to avoid breaking changes, we are only allowed to add new
> columns, but not modify or remove existing columns!
>
>
> To avoid missing data and key errors in the journal, we already bumped 
> changes to PVE 8 so it can handle the new format sent out by pvestatd in the
> latest versions.
>
> On the GUI side, we switch memory graphs to stacked area graphs and for VMs
> we also have a dedicated line for the memory consumption as the host sees it.
> Because the current memory view of a VM will switch to the internal guest view,
> if we get detailed infos via the ballooning device.
> To make those slightly more complicated graphs possible, we need to adapt
> RRDChart.js in the widget-toolkit to allow for detailed overrides.
>
> While we are at it, we can also fix bug #6068 (Node Search tab incorrect Host
> memory usage %) by switching to memhost if available and one wrong if check.
>
>
> As a side note, now that we got pressure graphs, we could start thinking about
> dropping the server load and IO wait graphs. Those are not very specific and
> mash many different metrics into a single one.
>
>
> Release notes:
> We should probably mention in the release notes, that due to the changed
> aggregation settings, it is expected that the resulting RRD files might have
> some data points that the originals didn't have. We observed that in some
> situation we get could get a data point in one time step earlier than before.
> This is most likely due to how RRD recalculates the aggregated data with the
> different resolution.
>
> In the pve8to9 checks, we now have a check that makes sure we do have enough
> free space, as the new RRD files with the new columns and more detailed
> aggeration steps, are quite a bit larger. We also check after install, if any
> RRD files have not yet been migrated, which would warrant another manual run of
> the migration tool.
>
> Plans:
> * add doc patches for the summary pages that explain the different graphs and
> make the help button point to those sections
>
> KNOWN ISSUES:
> * on a live system, renaming the source RRD files to FILE.old doesn't seem to
> work as expected and besides the renamed ones, new ones without the .old prefix
> show up again. I suspect some interaction with rrdached and/or pmxcfs receiving
> new data.
>
> How to test:
> 1. have PVE8 nodes on the latest version (>= 8.4.4)
> 2. Upgrade the first node to PVE9/trixie and install all the other patches
>     to see the automatic upgrade, pve-manager might need to be temporarily
>     bumped to 9.0.0~12!
>     build all the other repositories, copy the .deb files over and then ideally
>     use something like the following to make shure that any dependency will be
>     used from the deb files, and not the apt repositories.
>     ```
>     apt install ./*.deb --reinstall --allow-downgrades -y
>     ```
> 3. you should see, if the pve-manager package calling the
> proxmox-rrd-migration-tool
>

Gave this series a test on a pre-existing 3-node dev cluster.
I first upgraded the cluster to PVE9 and then installed the packages
from this series on top. I could verify that the
proxmox-rrd-migration-tool is executed by pve-manager's d/postinst
script.

Now, unfortunately that was a cluster that I don't use that often, so
the weekly/monthly/yearly RRD data was pretty empty, but I can at least
verify that the hourly data was migrated successfully. Also I could see
the new pressure graphs being populated immediately after the upgrade.
Nice!

During testing I did not really encounter any issues.

Regarding the memleak in pmxcfs that I mentioned during my review of the
pve-cluster patches: I could definitely see RSS creep up quite slowly.
Not sure how much of that is due to the leak and how much is 'normal',
where the heap size slowly converges to some final maximum. I'll keep
the cluster running for a bit more time and see where this goes.

Consider this:

Tested-by: Lukas Wagner <l.wagner at proxmox.com>