[pbs-devel] [RFC proxmox-backup 2/2] image backup: use 4M input buffer

[Dietmar Maurer]

I get the following from git blame:

db0cb9ce0 src/bin/proxmox-backup-client.rs  (Wolfgang Bumiller    2019-12-12 15:27:07 +0100  289)     let stream = tokio_util::codec::FramedRead::new(file, tokio_util::codec::BytesCodec::new())


So we comited that code in 2019, while FramedRead::with_capacity was
added later in 2020: 

https://github.com/tokio-rs/tokio/pull/2215

So I think that was the reason for using the small buffer.


> On 17.7.2024 15:08 CEST Fabian Grünbichler <f.gruenbichler at proxmox.com> wrote:
> 
>  
> with the default 8k input buffer size, the client will spend most of the time
> polling instead of reading/chunking/uploading.
> 
> tested with 16G random data file from tmpfs to fresh datastore backed by tmpfs,
> without encryption.
> 
> stock:
> 
> Time (mean ± σ):     36.064 s ±  0.655 s    [User: 21.079 s, System: 26.415 s]
>   Range (min … max):   35.663 s … 36.819 s    3 runs
> 
> patched:
> 
>  Time (mean ± σ):     23.591 s ±  0.807 s    [User: 16.532 s, System: 18.629 s]
>   Range (min … max):   22.663 s … 24.125 s    3 runs
> 
> Summary
>   patched ran
>     1.53 ± 0.06 times faster than stock
> 
> Signed-off-by: Fabian Grünbichler <f.gruenbichler at proxmox.com>
> ---
> obviously, this slightly increases memory usage..
> 
> the effect here is less pronounced than for the example because the actual
> reading and chunking part is not all the client has to do for a backup - digest
> calculation and TLS crypto make up the bulk of the rest..
> 
>  proxmox-backup-client/src/main.rs | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
> index 6a7d09047..5edb2a824 100644
> --- a/proxmox-backup-client/src/main.rs
> +++ b/proxmox-backup-client/src/main.rs
> @@ -286,8 +286,12 @@ async fn backup_image<P: AsRef<Path>>(
>  
>      let file = tokio::fs::File::open(path).await?;
>  
> -    let stream = tokio_util::codec::FramedRead::new(file, tokio_util::codec::BytesCodec::new())
> -        .map_err(Error::from);
> +    let stream = tokio_util::codec::FramedRead::with_capacity(
> +        file,
> +        tokio_util::codec::BytesCodec::new(),
> +        4 * 1024 * 1024,
> +    )
> +    .map_err(Error::from);
>  
>      let stream = FixedChunkStream::new(stream, chunk_size.unwrap_or(4 * 1024 * 1024));
>  
> -- 
> 2.39.2
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel