[pve-devel] applied: [RFC pve-qemu] disable jemalloc
t.lamprecht at proxmox.com
Sat Mar 11 10:01:28 CET 2023
Am 10/03/2023 um 19:05 schrieb DERUMIER, Alexandre:
> I'm currently benching again qemu with librbd and memory allocator.
> It's seem that they are still performance problem with default glibc
> allocator, around 20-25% less iops and bigger latency.
Are those numbers compared to jemalloc or tcmalloc?
Also, a key problem with allocator tuning is that its heavily dependent on
the workload of each specific library (i.e., not only QEMU itself but also
the specific block backend (library).
> From my bench, i'm around 60k iops vs 80-90k iops with 4k randread.
> Redhat have also notice it
> I known than jemalloc was buggy with rust lib && pbs block driver,
> but did you have evaluated tcmalloc ?
Yes, for PBS once - was way worse in how it generally worked than either
jemalloc and default glibc IIRC, but I don't think I checked for latency,
as then we tracked down freed memory that the allocator did not give back
to the system to how they internally try to keep a pool of available memory
So for latency it might be a win, but IMO not to sure if the other effects
it has are worth that.
> Note that it's possible to load it dynamically with LD_PRELOAD,
> so maybe could we add an option in vm config to enable it ?
I'm not 100% sure if QEMU copes well with preloading it via the dynlinker
as is, or if we need to hard-disable malloc_trim support for it then.
As currently with the "system" allocator (glibc) there's malloc_trim called
(semi-) periodically via call_rcu_thread - and at least qemu's meson build
system config disables malloc_trim for tcmalloc or jemalloc.
Or did you already test this directly on QEMU, not just rbd bench? As then
I'd be open to add some tuning config with a allocator sub-property in there
to our CFGs.
More information about the pve-devel