[pbs-devel] [PATCH v8 proxmox-backup 41/69] upload stream: implement reused chunk injector

[Christian Ebner]

On 6/4/24 10:50, Fabian Grünbichler wrote:
> On May 28, 2024 11:42 am, Christian Ebner wrote:
>> In order to be included in the backups index file, reused payload
>> chunks have to be injected into the payload upload stream at a
>> forced boundary. The chunker forces a chunk boundary and sends the
>> list of reusable dynamic entries to be uploaded.
>>
>> This implements the logic to receive these dynamic entries via the
>> corresponding communication channel from the chunker and inject the
>> entries into the backup upload stream by looking for the matching
>> chunk boundary, already forced by the chunker.
>>
>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>> ---
>> changes since version 7:
>> - no changes
>>
>> changes since version 6:
>> - no changes
>>
>>   pbs-client/src/inject_reused_chunks.rs | 129 +++++++++++++++++++++++++
>>   pbs-client/src/lib.rs                  |   1 +
>>   2 files changed, 130 insertions(+)
>>   create mode 100644 pbs-client/src/inject_reused_chunks.rs
>>
>> diff --git a/pbs-client/src/inject_reused_chunks.rs b/pbs-client/src/inject_reused_chunks.rs
>> new file mode 100644
>> index 000000000..ed147f5fb
>> --- /dev/null
>> +++ b/pbs-client/src/inject_reused_chunks.rs
>> @@ -0,0 +1,129 @@
>> +use std::cmp;
>> +use std::pin::Pin;
>> +use std::sync::atomic::{AtomicUsize, Ordering};
>> +use std::sync::{mpsc, Arc};
>> +use std::task::{Context, Poll};
>> +
>> +use anyhow::{anyhow, Error};
>> +use futures::{ready, Stream};
>> +use pin_project_lite::pin_project;
>> +
>> +use crate::pxar::create::ReusableDynamicEntry;
>> +
>> +pin_project! {
>> +    pub struct InjectReusedChunksQueue<S> {
>> +        #[pin]
>> +        input: S,
>> +        next_injection: Option<InjectChunks>,
>> +        buffer: Option<bytes::BytesMut>,
> 
> we successfully eliminated this buffer I think ;)

True, removed it swiftly!