[pbs-devel] [PATCH proxmox-backup v3 3/5] datastore: data blob: add helper and test for checking zstd_safe error code
Dominik Csapak
d.csapak at proxmox.com
Mon Aug 5 11:24:12 CEST 2024
We want to check the error code of zstd not to be 'Destination buffer to
small', but there is no practical api at the moment. So we introduce a
helper that uses the internal logic of zstd to determine the error.
Since this is not guaranteed to be a stable api, add a test for that so
we catch that error early on build. This should be fine, as long as this
zstd behavior only changes with e.g. major debian upgrades. (Which means
a new version)
Signed-off-by: Dominik Csapak <d.csapak at proxmox.com>
---
new in v3
note that this probably throws a warning until the next patch due to the
unused function. it would also be possible to merge this patch into
the next if that's wanted
Cargo.toml | 1 +
pbs-datastore/Cargo.toml | 1 +
pbs-datastore/src/data_blob.rs | 28 +++++++++++++++++++++++++++-
3 files changed, 29 insertions(+), 1 deletion(-)
diff --git a/Cargo.toml b/Cargo.toml
index 2b51ec82..275e3c95 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -158,6 +158,7 @@ url = "2.1"
walkdir = "2"
xdg = "2.2"
zstd = { version = "0.12", features = [ "bindgen" ] }
+zstd-safe = "6.0"
[dependencies]
anyhow.workspace = true
diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index d8997e1d..494c231b 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -23,6 +23,7 @@ tokio = { workspace = true, features = [] }
tracing.workspace = true
walkdir.workspace = true
zstd.workspace = true
+zstd-safe.workspace = true
pathpatterns.workspace = true
pxar.workspace = true
diff --git a/pbs-datastore/src/data_blob.rs b/pbs-datastore/src/data_blob.rs
index a3a41c5e..adf5a932 100644
--- a/pbs-datastore/src/data_blob.rs
+++ b/pbs-datastore/src/data_blob.rs
@@ -12,6 +12,17 @@ use super::file_formats::*;
const MAX_BLOB_SIZE: usize = 128 * 1024 * 1024;
+// tests if the error code was 'Destination buffer is too small'
+// by subtracting the error code from 0 (with underflow)
+// see https://github.com/facebook/zstd/blob/dev/lib/common/error_private.h
+// we test for this below so we catch errors if the interface changes
+fn zstd_error_is_target_too_small(err: usize) -> bool {
+ let (real_code, _) = 0usize.overflowing_sub(err);
+ // ZSTD_error_dstSize_tooSmall = 70,
+ // see https://github.com/facebook/zstd/blob/dev/lib/zstd_errors.h
+ real_code == 70
+}
+
/// Encoded data chunk with digest and positional information
pub struct ChunkInfo {
pub chunk: DataBlob,
@@ -567,7 +578,7 @@ impl<'a, 'b> DataChunkBuilder<'a, 'b> {
mod test {
use pbs_tools::crypt_config::CryptConfig;
- use super::DataChunkBuilder;
+ use super::{zstd_error_is_target_too_small, DataChunkBuilder};
const TEST_DATA_LEN: usize = 50;
@@ -640,4 +651,19 @@ mod test {
.expect("cannot decode encrypted, compressed chunk");
assert_eq!(data, data_decoded);
}
+
+ #[test]
+ fn zstd_error_code_test() {
+ // test for the error code internal logic of zstd so we catch interface changes here
+ let data = &build_test_data();
+ let mut target = Vec::new();
+ match zstd_safe::compress(&mut target, data, 1) {
+ Ok(_) => panic!("unexpected success with zero-sized buffer"),
+ Err(err) => {
+ if !zstd_error_is_target_too_small(err) {
+ panic!("unexpected error code");
+ }
+ }
+ }
+ }
}
--
2.39.2
More information about the pbs-devel
mailing list