[pbs-devel] [RFC v2 proxmox-backup 24/36] index: fetch chunk form index by start/end-offset
Fabian Grünbichler
f.gruenbichler at proxmox.com
Tue Mar 12 09:50:40 CET 2024
s/form/from/
but the commit title seems a bit off anyway - this doesn't fetch any
chunk
On March 5, 2024 10:26 am, Christian Ebner wrote:
> In preparation for injecting reused payload chunks in payload streams
> for regular files with unchanged metaddata.
>
> Adds a function to get a list of DynamicEntry's from a chunk index by
> given start and end offset, the range of which will be contained
> within these returned chunks.
>
> In addition to the list of index entries, the padding to the start of
> the requested start offset from the first chunk is returned, as well
> as the end padding following the requested payload to the actual
> chunk end.
>
> The padding is used for calculation of the payload reference offset
> written to the metadata archive, required for payload access during
> decoding.
>
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
> changes since version 1:
> - s/Appendable/Reusable/ incorrect naming leftover from previous approach
>
> pbs-datastore/src/dynamic_index.rs | 55 ++++++++++++++++++++++++++++++
> 1 file changed, 55 insertions(+)
>
> diff --git a/pbs-datastore/src/dynamic_index.rs b/pbs-datastore/src/dynamic_index.rs
> index 71a5082e..85c86b32 100644
> --- a/pbs-datastore/src/dynamic_index.rs
> +++ b/pbs-datastore/src/dynamic_index.rs
> @@ -74,6 +74,26 @@ impl DynamicEntry {
> }
> }
>
> +/// Dynamic Entry reusable by payload references
> +#[derive(Clone, Debug)]
> +#[repr(C)]
> +pub struct ReusableDynamicEntry {
> + size_le: u64,
> + digest: [u8; 32],
> +}
> +
> +impl ReusableDynamicEntry {
> + #[inline]
> + pub fn size(&self) -> u64 {
> + u64::from_le(self.size_le)
> + }
> +
> + #[inline]
> + pub fn digest(&self) -> [u8; 32] {
> + self.digest.clone()
> + }
> +}
> +
> pub struct DynamicIndexReader {
> _file: File,
> pub size: usize,
> @@ -188,6 +208,41 @@ impl DynamicIndexReader {
> self.binary_search(middle_idx + 1, middle_end, end_idx, end, offset)
> }
> }
> +
> + /// List of chunk indices containing the data from start_offset to end_offset
> + pub fn indices(
and tbh, this also seems like a very generic name ;) there's also only a
single call site, so maybe this could live as a helper there? or do we
expect to need this somewhere else in the near future?
> + &self,
> + start_offset: u64,
> + end_offset: u64,
> + ) -> Result<(Vec<ReusableDynamicEntry>, u64, u64), Error> {
> + let end_idx = self.index.len() - 1;
this would then be self.index_count() - 1;
> + let chunk_end = self.chunk_end(end_idx);
> + let start = self.binary_search(0, 0, end_idx, chunk_end, start_offset)?;
> + let end = self.binary_search(0, 0, end_idx, chunk_end, end_offset - 1)?;
these three are pub fn
> +
> + let offset_first = if start == 0 {
> + 0
> + } else {
> + self.index[start - 1].end()
this would be self.chunk_end(start - 1)
> + };
> +
> + let padding_start = start_offset - offset_first;
> + let padding_end = self.index[end].end() - end_offset;
> +
also chunk_end
> + let mut indices = Vec::new();
> + let mut prev_end = offset_first;
> + for dynamic_entry in &self.index[start..end + 1] {
also pub via index()
> + let size = dynamic_entry.end() - prev_end;
> + let reusable_dynamic_entry = ReusableDynamicEntry {
> + size_le: size.to_le(),
> + digest: dynamic_entry.digest.clone(),
> + };
> + prev_end += size;
> + indices.push(reusable_dynamic_entry);
> + }
> +
> + Ok((indices, padding_start, padding_end))
the end padding is currently not even used at the call site?
> + }
> }
>
> impl IndexFile for DynamicIndexReader {
> --
> 2.39.2
>
>
>
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>
>
>
More information about the pbs-devel
mailing list