[pve-devel] [PATCH pve-storage] qcow2 format: enable subcluster allocation by default

[Fiona Ebner]

Am 14.11.24 um 09:31 schrieb DERUMIER, Alexandre:
> Hi Fiona,
> 
> I'm really sorry, I didn't see your reponse, lost in the flood of email
> :(
> 
> 
>>> How does read performance compare for you (with 128 KiB cluster
> size)?
> 
>>> I don't see any noticeable difference in my testing with an ext4
>>> directory storage on an SSD, attaching the qcow2 images as SCSI disks
>>> to
>>> the VM, neither for reading nor writing. I only tested without your
>>> change and with your change using 4k (rand)read and (rand)write.
>>>
>>> I'm not sure we should enable this for everybody, there's always a
>>> risk
>>> to break stuff with added complexity. Maybe it's better to have a
>>> storage configuration option that people can opt-in to, e.g.
>>>
>>> qcow2-create-opts extended_l2=on,cluster_size=128k
>>>
>>> If we get enough positive feedback, we can still change the default
>>> in a
>>> future (major) release.
> 
> What disk size do you use for your bench ? 
> It's really important, because generally, the more bigger, the slower
> read in qcow2 will be , because l2 metadatas need to be cached handle
> in memory. (and qemu have 1MB cache by default)
> 

I don't recall the exact size. Today I retested with a 900 GiB image.

> witout subcluster, for 1TB image, it's around 128MB l2 metadas in qcow2
> file. 
> 
> for write, it's mostly for first block allocation, so it's depend
> of the filesystem behind, if fallocate is supported.
> (I have seen really big difference on shared ocfs2/gfs2 fs)
> 

Okay, I see.

Mine are backed by an ext4 storage. With
'extended_l2=on,cluster_size=128k' I actually get much slower results
for the initial allocations (note that this is with
preallocation=metadata - without preallocation, I still see "without"
being about 1.5 times faster, i.e. 375MB written versus 255MB, however
the usage on the underlying storage is much(!) worse, i.e. 5.71 GiB
versus 373 MiB):

> fio --name=rw --filename=/dev/sdb --ioengine=libaio --direct 1 --bs=4k  --rw=randwrite --numjobs=1 --group_reporting --time_based --runtime 60

With options:

> WRITE: bw=5603KiB/s (5737kB/s), 5603KiB/s-5603KiB/s (5737kB/s-5737kB/s), io=328MiB (344MB), run=60001-60001msec

Without:

> WRITE: bw=27.1MiB/s (28.4MB/s), 27.1MiB/s-27.1MiB/s (28.4MB/s-28.4MB/s), io=1626MiB (1705MB), run=60001-60001msec

For a subsequent randrw:

> fio --name=rw --filename=/dev/sdb --ioengine=libaio --direct 1 --bs=4k  --rw=randrw --numjobs=1 --group_reporting --time_based --runtime 60

With options:

>    READ: bw=7071KiB/s (7241kB/s), 7071KiB/s-7071KiB/s (7241kB/s-7241kB/s), io=414MiB (434MB), run=60001-60001msec
>   WRITE: bw=7058KiB/s (7227kB/s), 7058KiB/s-7058KiB/s (7227kB/s-7227kB/s), io=414MiB (434MB), run=60001-60001msec

Without:

>    READ: bw=11.8MiB/s (12.4MB/s), 11.8MiB/s-11.8MiB/s (12.4MB/s-12.4MB/s), io=708MiB (742MB), run=60001-60001msec
>   WRITE: bw=11.8MiB/s (12.3MB/s), 11.8MiB/s-11.8MiB/s (12.3MB/s-12.3MB/s), io=707MiB (741MB), run=60001-60001msec

Took a snapshot and afterwards I get:

> fio --name=rw --filename=/dev/sdb --ioengine=libaio --direct 1 --bs=4k  --rw=randrw --numjobs=1 --group_reporting --time_based --runtime 60

With options:

>    READ: bw=1250KiB/s (1280kB/s), 1250KiB/s-1250KiB/s (1280kB/s-1280kB/s), io=73.3MiB (76.8MB), run=60003-60003msec
>   WRITE: bw=1251KiB/s (1281kB/s), 1251KiB/s-1251KiB/s (1281kB/s-1281kB/s), io=73.3MiB (76.9MB), run=60003-60003msec

Without:

>    READ: bw=1198KiB/s (1227kB/s), 1198KiB/s-1198KiB/s (1227kB/s-1227kB/s), io=70.2MiB (73.6MB), run=60002-60002msec
>   WRITE: bw=1201KiB/s (1230kB/s), 1201KiB/s-1201KiB/s (1230kB/s-1230kB/s), io=70.4MiB (73.8MB), run=60002-60002msec

But there is a big difference in the read performance, this time in
favor of "with options" (but there is less allocated on the image, as a
consequence of the previous tests):

> fio --name=rw --filename=/dev/sdb --ioengine=libaio --direct 1 --bs=4k  --rw=randread --numjobs=1 --group_reporting --time_based --runtime 60

With options:

> READ: bw=31.2MiB/s (32.7MB/s), 31.2MiB/s-31.2MiB/s (32.7MB/s-32.7MB/s), io=1871MiB (1962MB), run=60001-60001msec

Without:

> READ: bw=19.2MiB/s (20.1MB/s), 19.2MiB/s-19.2MiB/s (20.1MB/s-20.1MB/s), io=1151MiB (1207MB), run=60001-60001msec


> Also, for write, this is helping a lot with backing file. (so for
> linked qcow2 clone, and maybe in the future for external snapshots).
> Because currently, if you write 4k on a overlay , you need to read the
> full cluster 64k on the base write, and rewrite it. (so 8x
> overamplification)
> 

For clone, we should add the options in clone_image() too ;)

> 
> They are good information from the developper :
> https://blogs.igalia.com/berto/2020/12/03/subcluster-allocation-for-qcow2-images/
> 
> 
> I really don't think it's hurt, but maybe a default for 9.0 release to
> be safe. (I'll try to have the external snapshot ready for this date
> too)
> 

It seems to hurt at least in some cases (initial allocation speed can be
worse than without the option, in particular when using
preallocation=metadata). If we have it as opt-in already, users can give
it a try and then we can still think about making it the default for
9.0, should there be enough positive feedback.

We could also think about only using it for linked clones by default
initially.

Independently, you can make it the default for LvmQcow2Plugin of course,
since you see much better results for that use case :)