[pbs-devel] [RFC proxmox-backup 00/24] fix #3044: push datastore to remote

[Christian Ebner]

On 7/16/24 16:09, Gabriel Goller wrote:
> Just a high level overview:
>   - In the frontend, there is no way to see the difference between a
>     push/pull sync job when the datastore names are the same. I think
>     it's better if we add a second table for the push jobs (on the same
>     page, something like in the "Prune & GC Jobs" tab.)
>   - Sometimes a group fails to sync, I get an "sync group 'vm/114' 
> failed    pipelined request failed: no such chunk
>     bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8".
>     It seems like some chunks get added to the fixed index but haven't
>     been uploaded yet. (Because sometimes, I also get this error on the
>     receiving end: "POST /fixed_chunk: 400 Bad Request: backup already
>     marked as finished.".)

For the record, the issue causing this seemingly unknown chunks is that 
the server side also keeps track of the known chunks, as stored in the 
`BackupEnvironment` state. This state however only lives as long as the 
backup writer instance connection. Therefore, it is not possible to keep 
all the known chunks for the full sync job.
The upcoming version of the patches will therefore use only the 
orthogonal approach and download the previous snapshot manifest and 
indexes and use that instead to avoid duplicate chunk upload. This also 
mimics more closely a regular backup run.