[pbs-devel] [PATCH proxmox-backup 7/7] pull: guard chunk upload and only insert into cache after upload

[Fabian Grünbichler]

On October 6, 2025 12:41 pm, Christian Ebner wrote:
> Inserting the chunk into the local datastore cache leads to it being
> considered as present on the backend. The upload might however still
> fail, leading to missing chunks.
> 
> Therefore, only insert the chunk after the upload completed with
> success and guard the upload by the backend upload marker file to
> avoid races with garbage collection.
> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  src/server/pull.rs | 14 ++++++++++----
>  1 file changed, 10 insertions(+), 4 deletions(-)
> 
> diff --git a/src/server/pull.rs b/src/server/pull.rs
> index 08e6e6b64..639415107 100644
> --- a/src/server/pull.rs
> +++ b/src/server/pull.rs
> @@ -176,14 +176,20 @@ async fn pull_index_chunks<I: IndexFile>(
>                      if target2.cache_contains(&digest) {
>                          return Ok(());
>                      }
> -                    target2.cache_insert(&digest, &chunk)?;
>                      let data = chunk.raw_data().to_vec();
>                      let upload_data = hyper::body::Bytes::from(data);
>                      let object_key = pbs_datastore::s3::object_key_from_digest(&digest)?;
> -                    let _is_duplicate = proxmox_async::runtime::block_on(
> +                    target2.insert_backend_upload_marker(&digest)?;
> +                    match proxmox_async::runtime::block_on(
>                          s3_client.upload_no_replace_with_retry(object_key, upload_data),
> -                    )
> -                    .context("failed to upload chunk to s3 backend")?;
> +                    ) {
> +                        Ok(_is_duplicate) => (),
> +                        Err(err) => {
> +                            target2.cleanup_backend_upload_marker(&digest)?;
> +                            return Err(err.context("failed to upload chunk to s3 backend"));
> +                        }
> +                    }
> +                    target2.cache_insert(&digest, &chunk)?;

same issue as described in patch #5 exists here as well..

>                  }
>              }
>              Ok(())
> -- 
> 2.47.3
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
>