[pve-devel] [PATCH cluster] fix #5728: pmxcfs: allow bigger writes than 4k for fuse

[Dominik Csapak]

by default libfuse2 limits writes to 4k size, which means that on writes
bigger than that, we do a whole write cycle for each 4k block that comes
in. To avoid that, add the option 'big_writes' to allow writes bigger
than 4k at once (namely up to 128 KiB).

This means that if we update a file with more than 4KiB data, the
following pattern occurs:

* cfs_fuse_write is called with at offset 0 with 4096 size
* sqlite writes the partial file to disk since it's a transaction
* cfs_fuse_write is called with an offset 4096 and with 4096 size
* sqlite updates the data and writes again
* repeat until all data reached cfs_fuse_write

So when cfs_fuse_write accepts bigger chunks, we have less
cfs_fuse_write -> sqlite write cycles, leading to a reduced disk
activity.

Note that sqlite itself uses 4096 byte blocks to write to the file
system layer below.

Most files on pmxcfs are written with `file_set_contents`, which writes
the file into a tmp file and renames it, so we always have some write
overhead.

Previous to pve-common commit
ef0bcc9 (tools: file_set_contents: use syswrite instead of print)

it used `print` to write, which uses an internal 8k buffer, and after
the commit it uses `syswrite`, which writes the file unbuffered in one
go. (Fuse still splits writes at it's defined maximum)

The commit message of that patch includes benchmarks for various sizes
of writes on pmxcfs with this patch included. Results show that we can
reduce the amount of bytes written to disk for files larger than 4 KiB
by a significant amount (with both patches we can reduce the
amplification at 8KiB from ~15x to ~11x, and for 1024KiB from ~360x to
~15x)

When we change to libfuse3, we have to remove this option again, since
it got removed and is the default there.

Signed-off-by: Dominik Csapak <d.csapak at proxmox.com>
---
changes from rfc:
* improve commit message to contain more detail and reference
  Filips commit that improves `file_set_contents`
* add a comment above the option to remove it with change to libfuse3

 src/pmxcfs/pmxcfs.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/src/pmxcfs/pmxcfs.c b/src/pmxcfs/pmxcfs.c
index 1cf8ab8..f0d5102 100644
--- a/src/pmxcfs/pmxcfs.c
+++ b/src/pmxcfs/pmxcfs.c
@@ -945,7 +945,8 @@ int main(int argc, char *argv[])
 
 	mkdir(CFSDIR, 0755);
 
-	char *fa[] = { "-f", "-odefault_permissions", "-oallow_other", NULL};
+	// TODO: remove big_writes with change to libfuse3
+	char *fa[] = { "-f", "-odefault_permissions", "-oallow_other", "-obig_writes", NULL};
 
 	struct fuse_args fuse_args = FUSE_ARGS_INIT(sizeof (fa)/sizeof(gpointer) - 1, fa);
 
-- 
2.39.5