[pve-devel] qemu-kvm : use jemalloc as default memory allocator

[Alexandre Derumier]

Hi,
I have done a lot of benchmark with ceph these last days,

and I have had some performance problems with tcmalloc 
when increasing number of disks && iothread.

The main problem is that tcmalloc use a shared thread cache of 16MB
by default.
with more threads, this cache is shared, and some bad garbage collection
can occur if the cache is too low.

It's possible to increase it with a env var:
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=256MB


I have done test, with default 16MB and current jessie tcmalloc (2.2.1),
I really bad, with more than 2 disk.
increasing to 256MB, it's helping but still have problem with 16disks.

tcmalloc 2.4 seem to works fine, and more fast, but I think we don't want to maintain
a package.

That's why I have tested with jemalloc, and here no performance problem,
and almost same perforamnce than tcmalloc 2.4.


(They are a lot of discussions on ceph mailing list about this, because this tcmalloc problem
also occur on osds with a lot of iops)


Here the benchmark results of 1 qemu vm randread 4K iodepth=32



libc6
------

1 disk	 29052
2 disks	 55878
4 disks	 127899
8 disks	 240566
15 disks 269976
	
jemaloc
--------	

1 disk	 41278
2 disks	 75781
4 disks	 195351
8 disks	 294241
15 disks 298199
	
	
	
tcmalloc 2.2.1 default 16M cache
------------------------------	

1 disk	 37911
2 disks	 67698
4 disks	 41076
8 disks	 43312
15 disks 37569
	

tcmalloc 2.2.1 : 256M
-----------------------	

1 disk	    33914
2 disks	    58839
4 disks	    148205
8 disks	    213298
15 disks    218383

	
tcmalloc 2.4 :256M cache
-------------------------
1 disk	 42160
2 disks	 83135
4 disks	 194591
8 disks	 306038
15 disks 302278