[pbs-devel] [RFC proxmox-backup] api: datastore: determine size and blob type if not in manifest

[Dietmar Maurer]

This is a critical path, and I do not really want an additional delay here.
We need a better solution for this.


> On 10/14/2020 5:04 PM Stefan Reiter <s.reiter at proxmox.com> wrote:
> 
>  
> Try to read the first 8 byte (DataBlobHeader magic) from any .blob file
> not mentioned in the manifest. This allows classifying the
> client.log.blob file as encrypted if it is, and showing the correct
> symbol in the GUI for it.
> 
> While already open, also determine the file size by seeking to End(0).
> 
> Signed-off-by: Stefan Reiter <s.reiter at proxmox.com>
> ---
> 
> I noticed that the GUI would show the "download" icon enabled for
> 'client.log.blob' files, even if they were encrypted. The button wouldn't work
> of course, since the server can't decode encrypted blobs, so nothing would
> happen except an error logged to the console.
> 
> The problem is that currently we don't know if a blob file not listed in the
> manifest is encrypted - but we can find out, since it's encoded into the header.
> Not sure if this is worth the open() and read() for every unknown blob file, but
> it would solve the problem...
> 
> 
>  src/api2/admin/datastore.rs | 25 +++++++++++++++++++++++--
>  src/backup/data_blob.rs     | 12 ++++++++++++
>  2 files changed, 35 insertions(+), 2 deletions(-)
> 
> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> index 11223e6a..2a9ac469 100644
> --- a/src/api2/admin/datastore.rs
> +++ b/src/api2/admin/datastore.rs
> @@ -2,6 +2,7 @@ use std::collections::{HashSet, HashMap};
>  use std::ffi::OsStr;
>  use std::os::unix::ffi::OsStrExt;
>  use std::sync::{Arc, Mutex};
> +use std::io::{Read, Seek, SeekFrom};
>  
>  use anyhow::{bail, format_err, Error};
>  use futures::*;
> @@ -91,10 +92,30 @@ fn get_all_snapshot_files(
>  
>      for file in &info.files {
>          if file_set.contains(file) { continue; }
> +
> +        // Try to determine size and crypt mode for unknown blobs
> +        let mut crypt_mode = None;
> +        let mut size = None;
> +        if file.ends_with(".blob") {
> +            let mut path = store.snapshot_path(&info.backup_dir);
> +            path.push(file);
> +            if let Ok(mut file) = std::fs::File::open(path) {
> +                let mut buffer = [0u8; 8];
> +                if let Ok(n) = file.read(&mut buffer[..]) {
> +                    if n == buffer.len() {
> +                        crypt_mode = DataBlob::is_blob_magic(&buffer);
> +                    }
> +                }
> +                if let Ok(pos) = file.seek(SeekFrom::End(0)) {
> +                    size = Some(pos);
> +                }
> +            }
> +        }
> +
>          files.push(BackupContent {
>              filename: file.to_string(),
> -            size: None,
> -            crypt_mode: None,
> +            size,
> +            crypt_mode,
>          });
>      }
>  
> diff --git a/src/backup/data_blob.rs b/src/backup/data_blob.rs
> index 284dc243..626a5fe4 100644
> --- a/src/backup/data_blob.rs
> +++ b/src/backup/data_blob.rs
> @@ -321,6 +321,18 @@ impl DataBlob {
>  
>          Ok(())
>      }
> +
> +    /// Determine if the given value is a valid blob magic number.
> +    /// Returns CryptMode::Encrypt or CryptMode::None depending on type.
> +    pub fn is_blob_magic(magic: &[u8; 8]) -> Option<CryptMode> {
> +        if magic == &UNCOMPRESSED_BLOB_MAGIC_1_0 || magic == &COMPRESSED_BLOB_MAGIC_1_0 {
> +            Some(CryptMode::None)
> +        } else if magic == &ENCRYPTED_BLOB_MAGIC_1_0 || magic == &ENCR_COMPR_BLOB_MAGIC_1_0 {
> +            Some(CryptMode::Encrypt)
> +        } else {
> +            None
> +        }
> +    }
>  }
>  
>  /// Builder for chunk DataBlobs
> -- 
> 2.20.1
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel