From w.bumiller at proxmox.com  Tue May  6 13:57:08 2025
From: w.bumiller at proxmox.com (Wolfgang Bumiller)
Date: Tue, 6 May 2025 13:57:08 +0200
Subject: [pbs-devel] applied: [PATCH proxmox v2] proxmox-client: add query
 builder
In-Reply-To: <20250416113601.256829-1-m.sandoval@proxmox.com>
References: <20250416113601.256829-1-m.sandoval@proxmox.com>
Message-ID: <vntqovqrtcfj6ioi5drrqq35tkdf4rnxi5cnt5y473l66hvy52@bj6xo3phe3vm>

applied, thanks



From c.ebner at proxmox.com  Wed May  7 17:38:34 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Wed,  7 May 2025 17:38:34 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 2/2] fix #6358: remove group note
 file if present on group destroy
In-Reply-To: <20250507153834.758840-1-c.ebner@proxmox.com>
References: <20250507153834.758840-1-c.ebner@proxmox.com>
Message-ID: <20250507153834.758840-3-c.ebner@proxmox.com>

Removing the group directory when forgetting a backup group or
removing the final backup snapshot of a group did not take into
consideration a potentially present group note file, leading for it
to fail.

Further, since the owner file is removed before trying to remove the
(not empty) group directory, the group will not be usable anymore as
the owner check will fail as well.

To fix this, remove the backup group's note file first, if present
and only after that try to cleanup the rest.

Fixes: https://bugzilla.proxmox.com/show_bug.cgi?id=6358
Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index d4732fdd9..22b4eddf3 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -244,6 +244,13 @@ impl BackupGroup {
 
     /// Helper function, assumes that no more snapshots are present in the group.
     fn remove_group_dir(&self) -> Result<(), Error> {
+        let note_path = self.store.group_notes_path(&self.ns, &self.group);
+        if let Err(err) = std::fs::remove_file(&note_path) {
+            if err.kind() != std::io::ErrorKind::NotFound {
+                bail!("removing the note file '{note_path:?}' failed - {err}")
+            }
+        }
+
         let owner_path = self.store.owner_path(&self.ns, &self.group);
 
         std::fs::remove_file(&owner_path).map_err(|err| {
-- 
2.39.5




From c.ebner at proxmox.com  Wed May  7 17:38:33 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Wed,  7 May 2025 17:38:33 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 1/2] api: datastore: make group
 notes path helper a DataStore method
In-Reply-To: <20250507153834.758840-1-c.ebner@proxmox.com>
References: <20250507153834.758840-1-c.ebner@proxmox.com>
Message-ID: <20250507153834.758840-2-c.ebner@proxmox.com>

Move and make the helper function to get a backup groups notes file
path a `DataStore` method instead. This allows it to be reused when
access to the notes path is required from the datastore itself.

Further, use the plural `notes` wording also in the helper to be
consistent with the rest of the codebase.

In preparation for correctly removing the notes file from the backup
group on destruction.

No functional changes intended.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 11 +++++++++++
 src/api2/admin/datastore.rs    | 26 +++++++-------------------
 2 files changed, 18 insertions(+), 19 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index cbf78ecb6..91c7e76be 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -40,6 +40,8 @@ use crate::DataBlob;
 static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
     LazyLock::new(|| Mutex::new(HashMap::new()));
 
+const GROUP_NOTES_FILE_NAME: &str = "notes";
+
 /// checks if auth_id is owner, or, if owner is a token, if
 /// auth_id is the user of the token
 pub fn check_backup_owner(owner: &Authid, auth_id: &Authid) -> Result<(), Error> {
@@ -524,6 +526,15 @@ impl DataStore {
         full_path
     }
 
+    /// Returns the absolute path of a backup groups notes file
+    pub fn group_notes_path(
+        &self,
+        ns: &BackupNamespace,
+        group: &pbs_api_types::BackupGroup,
+    ) -> PathBuf {
+        self.group_path(ns, group).join(GROUP_NOTES_FILE_NAME)
+    }
+
     /// Returns the absolute path for backup_dir
     pub fn snapshot_path(
         &self,
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 392494488..cc7e17a29 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -4,7 +4,7 @@ use std::collections::HashSet;
 use std::ffi::OsStr;
 use std::ops::Deref;
 use std::os::unix::ffi::OsStrExt;
-use std::path::{Path, PathBuf};
+use std::path::Path;
 use std::sync::Arc;
 
 use anyhow::{bail, format_err, Context, Error};
@@ -77,18 +77,6 @@ use crate::backup::{
 
 use crate::server::jobstate::{compute_schedule_status, Job, JobState};
 
-const GROUP_NOTES_FILE_NAME: &str = "notes";
-
-fn get_group_note_path(
-    store: &DataStore,
-    ns: &BackupNamespace,
-    group: &pbs_api_types::BackupGroup,
-) -> PathBuf {
-    let mut note_path = store.group_path(ns, group);
-    note_path.push(GROUP_NOTES_FILE_NAME);
-    note_path
-}
-
 // helper to unify common sequence of checks:
 // 1. check privs on NS (full or limited access)
 // 2. load datastore
@@ -244,8 +232,8 @@ pub fn list_groups(
                 })
                 .to_owned();
 
-            let note_path = get_group_note_path(&datastore, &ns, group.as_ref());
-            let comment = file_read_firstline(note_path).ok();
+            let notes_path = datastore.group_notes_path(&ns, group.as_ref());
+            let comment = file_read_firstline(notes_path).ok();
 
             group_info.push(GroupListItem {
                 backup: group.into(),
@@ -2053,8 +2041,8 @@ pub fn get_group_notes(
         &backup_group,
     )?;
 
-    let note_path = get_group_note_path(&datastore, &ns, &backup_group);
-    Ok(file_read_optional_string(note_path)?.unwrap_or_else(|| "".to_owned()))
+    let notes_path = datastore.group_notes_path(&ns, &backup_group);
+    Ok(file_read_optional_string(notes_path)?.unwrap_or_else(|| "".to_owned()))
 }
 
 #[api(
@@ -2101,8 +2089,8 @@ pub fn set_group_notes(
         &backup_group,
     )?;
 
-    let note_path = get_group_note_path(&datastore, &ns, &backup_group);
-    replace_file(note_path, notes.as_bytes(), CreateOptions::new(), false)?;
+    let notes_path = datastore.group_notes_path(&ns, &backup_group);
+    replace_file(notes_path, notes.as_bytes(), CreateOptions::new(), false)?;
 
     Ok(())
 }
-- 
2.39.5




From c.ebner at proxmox.com  Wed May  7 17:38:32 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Wed,  7 May 2025 17:38:32 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 0/2] fix #6358: group removal
 fails if group notes exist
Message-ID: <20250507153834.758840-1-c.ebner@proxmox.com>

These patches fix an issue with the backup group removal failing and
leaving behind an unusable backup group, due to not taking a possibly
present backup group's notes file into account.

The series consists of 2 patches, the first being preparatory,
allowing to reuse the group notes filepath helper, the second fixing
the actual issue by conditionally removing the notes file before
further group directory cleanup.

Link to the bugtracker issue:
https://bugzilla.proxmox.com/show_bug.cgi?id=6358

Christian Ebner (2):
  api: datastore: make group notes path helper a DataStore method
  fix #6358: remove group note file if present on group destroy

 pbs-datastore/src/backup_info.rs |  7 +++++++
 pbs-datastore/src/datastore.rs   | 11 +++++++++++
 src/api2/admin/datastore.rs      | 26 +++++++-------------------
 3 files changed, 25 insertions(+), 19 deletions(-)

-- 
2.39.5




From l.leahu-vladucu at proxmox.com  Wed May  7 17:36:39 2025
From: l.leahu-vladucu at proxmox.com (=?UTF-8?q?Lauren=C8=9Biu=20Leahu-Vl=C4=83ducu?=)
Date: Wed,  7 May 2025 17:36:39 +0200
Subject: [pbs-devel] [PATCH proxmox] proxmox-product-config: fix code
 documentation on permissions
Message-ID: <20250507153639.46774-1-l.leahu-vladucu@proxmox.com>

This patch fixes the documentation of some functions being
inconsistent with the actual code. While such inconsistencies are
never good, when it comes to permissions, they might have even worse
consequences.

To be precise, this patch fixes the following:
- replace_config() actually uses permissions 0640 (docs stated 0660)
- although the possibility of setting a privileged user (usually root,
but possibly different) has been added in the past, the docs still
stated "root" or "superuser". However, some functions also explicitly
use "root", which made it even more confusing. It is now clear which
functions use the API user, which use the privileged user, and which
explicitly use root.
- fixed some small style inconsistencies (e.g. priv-user instead
of priv_user)

Signed-off-by: Lauren?iu Leahu-Vl?ducu <l.leahu-vladucu at proxmox.com>
---
 .../src/filesystem_helpers.rs                 | 26 +++++++++----------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/proxmox-product-config/src/filesystem_helpers.rs b/proxmox-product-config/src/filesystem_helpers.rs
index 9aa8b1a4..d9f0e827 100644
--- a/proxmox-product-config/src/filesystem_helpers.rs
+++ b/proxmox-product-config/src/filesystem_helpers.rs
@@ -17,9 +17,9 @@ pub fn default_create_options() -> CreateOptions {
         .group(api_user.gid)
 }
 
-/// Return [CreateOptions] for files owned by `priv_user.uid:api-user.gid` with permission `0640`.
+/// Return [CreateOptions] for files owned by `priv_user.uid:api_user.gid` with permission `0640`.
 ///
-/// Only the superuser can write those files, but group `api-user.gid` can read them.
+/// Only `priv_user` can write those files, but group `api_user.gid` can read them.
 pub fn privileged_create_options() -> CreateOptions {
     let api_user = get_api_user();
     let priv_user = get_priv_user();
@@ -30,9 +30,9 @@ pub fn privileged_create_options() -> CreateOptions {
         .group(api_user.gid)
 }
 
-/// Return [CreateOptions] for files owned by `priv_user.uid: priv_user.gid` with permission `0600`.
+/// Return [CreateOptions] for files owned by `priv_user.uid:priv_user.gid` with permission `0600`.
 ///
-/// Only the superuser can read and write those files.
+/// Only `priv_user` can read and write those files.
 pub fn secret_create_options() -> CreateOptions {
     let priv_user = get_priv_user();
     let mode = Mode::from_bits_truncate(0o0600);
@@ -63,16 +63,16 @@ pub fn lockfile_create_options() -> CreateOptions {
         .group(api_user.gid)
 }
 
-/// Atomically write data to file owned by `priv_user.uid:api-user.gid` with permission `0640`
+/// Atomically write data to file owned by `priv_user.uid:api_user.gid` with permission `0640`
 ///
-/// Only the superuser can write those files, but group 'api-user' can read them.
+/// Only `priv_user` can write those files, but group 'api_user' can read them.
 pub fn replace_privileged_config<P: AsRef<Path>>(path: P, data: &[u8]) -> Result<(), Error> {
     let options = privileged_create_options();
     proxmox_sys::fs::replace_file(path, data, options, true)?;
     Ok(())
 }
 
-/// Atomically write data to file owned by `api-user.uid:api-user.gid` with permission `0660`.
+/// Atomically write data to file owned by `api_user.uid:api_user.gid` with permission `0640`.
 pub fn replace_config<P: AsRef<Path>>(path: P, data: &[u8]) -> Result<(), Error> {
     let options = default_create_options();
     proxmox_sys::fs::replace_file(path, data, options, true)?;
@@ -81,7 +81,7 @@ pub fn replace_config<P: AsRef<Path>>(path: P, data: &[u8]) -> Result<(), Error>
 
 /// Atomically write data to file owned by `priv_user.uid:priv_user.gid` with permission `0600`.
 ///
-/// Only the superuser can read and write those files.
+/// Only `priv_user` can read and write those files.
 pub fn replace_secret_config<P: AsRef<Path>>(path: P, data: &[u8]) -> Result<(), Error> {
     let options = secret_create_options();
     proxmox_sys::fs::replace_file(path, data, options, true)?;
@@ -119,15 +119,15 @@ pub unsafe fn create_mocked_lock() -> ApiLockGuard {
     ApiLockGuard(None)
 }
 
-/// Open or create a lock file owned by user `api-user` and lock it.
+/// Open or create a lock file owned by user `api_user` and lock it.
 ///
-/// Owner/Group of the file is set to `api-user.uid/api-user.gid`.
+/// Owner/Group of the file is set to `api_user.uid/api_user.gid`.
 /// File mode is `0660`.
 /// Default timeout is 10 seconds.
 ///
 /// The lock is released as soon as you drop the returned lock guard.
 ///
-/// Note: This method needs to be called by user `root` or `api-user`.
+/// Note: This method needs to be called by `priv_user` or `api_user`.
 pub fn open_api_lockfile<P: AsRef<Path>>(
     path: P,
     timeout: Option<std::time::Duration>,
@@ -139,14 +139,14 @@ pub fn open_api_lockfile<P: AsRef<Path>>(
     Ok(ApiLockGuard(Some(file)))
 }
 ///
-/// Open or create a lock file owned by root and lock it.
+/// Open or create a lock file owned by `priv_user` and lock it.
 ///
 /// File mode is `0600`.
 /// Default timeout is 10 seconds.
 ///
 /// The lock is released as soon as you drop the returned lock guard.
 ///
-/// Note: This method needs to be called by user `root`.
+/// Note: This method needs to be called by user `priv_user`.
 pub fn open_secret_lockfile<P: AsRef<Path>>(
     path: P,
     timeout: Option<std::time::Duration>,
-- 
2.39.5




From g.goller at proxmox.com  Wed May  7 18:31:14 2025
From: g.goller at proxmox.com (Gabriel Goller)
Date: Wed,  7 May 2025 18:31:14 +0200
Subject: [pbs-devel] [RFC PATCH] schema: allow serializing rust Schema to
 perl JsonSchema
Message-ID: <20250507163114.1162300-1-g.goller@proxmox.com>

Implement serde::Serialize on the rust Schema, so that we can serialize
it and use it as a JsonSchema in perl. This allows us to write a single
Schema in rust and reuse it in perl for the api properties.

The interesting bits (custom impls) are:
 * Recursive oneOf type-property resolver
 * oneOf and allOf implementation
 * ApiStringFormat skip of ApiStringVerifyFn (which won't work obviously)

Signed-off-by: Gabriel Goller <g.goller at proxmox.com>
---

This is kinda hard to test, because nothing actually fails when the properties
are wrong and the whole allOf, oneOf and ApiStringFormat is a bit
untransparent. So some properties could be wrongly serialized, but I
think I got everything right. Looking over all the properties would be
appreciated!

 Cargo.toml                        |   1 +
 proxmox-schema/Cargo.toml         |   4 +-
 proxmox-schema/src/const_regex.rs |  12 ++
 proxmox-schema/src/schema.rs      | 242 ++++++++++++++++++++++++++++--
 4 files changed, 246 insertions(+), 13 deletions(-)

diff --git a/Cargo.toml b/Cargo.toml
index 2ca0ea618707..a0d760ae8fc9 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -104,6 +104,7 @@ regex = "1.5"
 serde = "1.0"
 serde_cbor = "0.11.1"
 serde_json = "1.0"
+serde_with = "3.8.1"
 serde_plain = "1.0"
 syn = { version = "2", features = [ "full", "visit-mut" ] }
 tar = "0.4"
diff --git a/proxmox-schema/Cargo.toml b/proxmox-schema/Cargo.toml
index c8028aa52bd0..48ebf3a9005e 100644
--- a/proxmox-schema/Cargo.toml
+++ b/proxmox-schema/Cargo.toml
@@ -15,8 +15,9 @@ rust-version.workspace = true
 anyhow.workspace = true
 const_format = { workspace = true, optional = true }
 regex.workspace = true
-serde.workspace = true
+serde = { workspace = true, features = ["derive"] }
 serde_json.workspace = true
+serde_with.workspace = true
 textwrap = "0.16"
 
 # the upid type needs this for 'getpid'
@@ -27,7 +28,6 @@ proxmox-api-macro = { workspace = true, optional = true }
 
 [dev-dependencies]
 url.workspace = true
-serde = { workspace = true, features = [ "derive" ] }
 proxmox-api-macro.workspace = true
 
 [features]
diff --git a/proxmox-schema/src/const_regex.rs b/proxmox-schema/src/const_regex.rs
index 8ddc41abedeb..56f6c27fa1de 100644
--- a/proxmox-schema/src/const_regex.rs
+++ b/proxmox-schema/src/const_regex.rs
@@ -1,5 +1,7 @@
 use std::fmt;
 
+use serde::Serialize;
+
 /// Helper to represent const regular expressions
 ///
 /// The current Regex::new() function is not `const_fn`. Unless that
@@ -13,6 +15,16 @@ pub struct ConstRegexPattern {
     pub regex_obj: fn() -> &'static regex::Regex,
 }
 
+impl Serialize for ConstRegexPattern {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: serde::Serializer,
+    {
+        // Get the compiled regex and serialize its pattern as a string
+        serializer.serialize_str((self.regex_obj)().as_str())
+    }
+}
+
 impl fmt::Debug for ConstRegexPattern {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         write!(f, "{:?}", self.regex_string)
diff --git a/proxmox-schema/src/schema.rs b/proxmox-schema/src/schema.rs
index ddbbacd462a4..11461eaf6ace 100644
--- a/proxmox-schema/src/schema.rs
+++ b/proxmox-schema/src/schema.rs
@@ -4,10 +4,12 @@
 //! completely static API definitions that can be included within the programs read-only text
 //! segment.
 
-use std::collections::HashSet;
+use std::collections::{HashMap, HashSet};
 use std::fmt;
 
 use anyhow::{bail, format_err, Error};
+use serde::ser::{SerializeMap, SerializeStruct};
+use serde::{Serialize, Serializer};
 use serde_json::{json, Value};
 
 use crate::ConstRegexPattern;
@@ -181,7 +183,8 @@ impl<'a> FromIterator<(&'a str, Error)> for ParameterError {
 }
 
 /// Data type to describe boolean values
-#[derive(Debug)]
+#[serde_with::skip_serializing_none]
+#[derive(Debug, Serialize)]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 #[non_exhaustive]
 pub struct BooleanSchema {
@@ -222,7 +225,8 @@ impl BooleanSchema {
 }
 
 /// Data type to describe integer values.
-#[derive(Debug)]
+#[serde_with::skip_serializing_none]
+#[derive(Debug, Serialize)]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 #[non_exhaustive]
 pub struct IntegerSchema {
@@ -304,7 +308,8 @@ impl IntegerSchema {
 }
 
 /// Data type to describe (JSON like) number value
-#[derive(Debug)]
+#[serde_with::skip_serializing_none]
+#[derive(Debug, Serialize)]
 #[non_exhaustive]
 pub struct NumberSchema {
     pub description: &'static str,
@@ -406,7 +411,8 @@ impl PartialEq for NumberSchema {
 }
 
 /// Data type to describe string values.
-#[derive(Debug)]
+#[serde_with::skip_serializing_none]
+#[derive(Debug, Serialize)]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 #[non_exhaustive]
 pub struct StringSchema {
@@ -418,6 +424,7 @@ pub struct StringSchema {
     /// Optional maximal length.
     pub max_length: Option<usize>,
     /// Optional microformat.
+    #[serde(flatten)]
     pub format: Option<&'static ApiStringFormat>,
     /// A text representation of the format/type (used to generate documentation).
     pub type_text: Option<&'static str>,
@@ -534,7 +541,8 @@ impl StringSchema {
 ///
 /// All array elements are of the same type, as defined in the `items`
 /// schema.
-#[derive(Debug)]
+#[serde_with::skip_serializing_none]
+#[derive(Debug, Serialize)]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 #[non_exhaustive]
 pub struct ArraySchema {
@@ -634,6 +642,43 @@ pub type SchemaPropertyEntry = (&'static str, bool, &'static Schema);
 /// This is a workaround unless RUST can const_fn `Hash::new()`
 pub type SchemaPropertyMap = &'static [SchemaPropertyEntry];
 
+/// A wrapper struct to hold the [`SchemaPropertyMap`] and serialize it nicely.
+///
+/// [`SchemaPropertyMap`] holds [`SchemaPropertyEntry`]s which are tuples. Tuples are serialized to
+/// arrays, but we need a Map with the name (first item in the tuple) as a key and the optional
+/// (second item in the tuple) as a property of the value.
+pub struct SerializableSchemaProperties<'a>(&'a [SchemaPropertyEntry]);
+
+impl Serialize for SerializableSchemaProperties<'_> {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        let mut seq = serializer.serialize_map(Some(self.0.len()))?;
+
+        for (name, optional, schema) in self.0 {
+            let schema_with_metadata = OptionalSchema {
+                optional: *optional,
+                schema,
+            };
+
+            seq.serialize_entry(&name, &schema_with_metadata)?;
+        }
+
+        seq.end()
+    }
+}
+
+/// A schema with a optional bool property.
+///
+/// The schema gets flattened, so it looks just like a normal Schema but with a optional property.
+#[derive(Serialize)]
+struct OptionalSchema<'a> {
+    optional: bool,
+    #[serde(flatten)]
+    schema: &'a Schema,
+}
+
 const fn assert_properties_sorted(properties: SchemaPropertyMap) {
     use std::cmp::Ordering;
 
@@ -656,7 +701,7 @@ const fn assert_properties_sorted(properties: SchemaPropertyMap) {
 /// Legacy property strings may contain shortcuts where the *value* of a specific key is used as a
 /// *key* for yet another option. Most notably, PVE's `netX` properties use `<model>=<macaddr>`
 /// instead of `model=<model>,macaddr=<macaddr>`.
-#[derive(Clone, Copy, Debug)]
+#[derive(Clone, Copy, Debug, Serialize)]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 pub struct KeyAliasInfo {
     pub key_alias: &'static str,
@@ -700,6 +745,77 @@ pub struct ObjectSchema {
     pub key_alias_info: Option<KeyAliasInfo>,
 }
 
+impl Serialize for ObjectSchema {
+    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
+    where
+        S: Serializer,
+    {
+        let mut s = serializer.serialize_struct("ObjectSchema", 5)?;
+
+        s.serialize_field("description", self.description)?;
+        s.serialize_field("additional_properties", &self.additional_properties)?;
+
+        // Collect all OneOf type properties recursively
+        let mut oneofs: Vec<SchemaPropertyEntry> = Vec::new();
+        for (_, _, schema) in self.properties {
+            collect_oneof_type_properties(schema, &mut oneofs);
+        }
+
+        if !oneofs.is_empty() {
+            // Extend the oneOf type-properties with the actual properties
+            oneofs.extend_from_slice(self.properties);
+            s.serialize_field("properties", &SerializableSchemaProperties(&oneofs))?;
+        } else {
+            s.serialize_field("properties", &SerializableSchemaProperties(self.properties))?;
+        }
+
+        if let Some(default_key) = self.default_key {
+            s.serialize_field("default_key", default_key)?;
+        } else {
+            s.skip_field("default_key")?;
+        }
+        if let Some(key_alias_info) = self.key_alias_info {
+            s.serialize_field("key_alias_info", &key_alias_info)?;
+        } else {
+            s.skip_field("key_alias_info")?;
+        }
+
+        s.end()
+    }
+}
+
+// Recursive function to find all OneOf type properties in a schema
+fn collect_oneof_type_properties(schema: &Schema, result: &mut Vec<SchemaPropertyEntry>) {
+    match schema {
+        Schema::OneOf(oneof) => {
+            result.push(*oneof.type_property_entry);
+        }
+        Schema::Array(array) => {
+            // Recursively check the array schema
+            collect_oneof_type_properties(array.items, result);
+        }
+        Schema::String(string) => {
+            // Check the PropertyString Schema
+            if let Some(ApiStringFormat::PropertyString(schema)) = string.format {
+                collect_oneof_type_properties(schema, result);
+            }
+        }
+        Schema::Object(obj) => {
+            // Check all properties in the object
+            for (_, _, prop_schema) in obj.properties {
+                collect_oneof_type_properties(prop_schema, result);
+            }
+        }
+        Schema::AllOf(all_of) => {
+            // Check all schemas in the allOf list
+            for &schema in all_of.list {
+                collect_oneof_type_properties(schema, result);
+            }
+        }
+        _ => {}
+    }
+}
+
 impl ObjectSchema {
     /// Create a new `object` schema.
     ///
@@ -811,7 +927,7 @@ impl ObjectSchema {
 ///
 /// Technically this could also contain an `additional_properties` flag, however, in the JSON
 /// Schema, this is not supported, so here we simply assume additional properties to be allowed.
-#[derive(Debug)]
+#[derive(Debug, Serialize)]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 #[non_exhaustive]
 pub struct AllOfSchema {
@@ -864,7 +980,7 @@ impl AllOfSchema {
 /// In serde-language, we use an internally tagged enum representation.
 ///
 /// Note that these are limited to object schemas. Other schemas will produce errors.
-#[derive(Debug)]
+#[derive(Debug, Serialize)]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 #[non_exhaustive]
 pub struct OneOfSchema {
@@ -880,6 +996,30 @@ pub struct OneOfSchema {
     pub list: &'static [(&'static str, &'static Schema)],
 }
 
+fn serialize_oneof_schema<S>(one_of: &OneOfSchema, serializer: S) -> Result<S::Ok, S::Error>
+where
+    S: Serializer,
+{
+    use serde::ser::SerializeMap;
+
+    let mut map = serializer.serialize_map(Some(3))?;
+
+    map.serialize_entry("description", &one_of.description)?;
+
+    let variants = one_of
+        .list
+        .iter()
+        .map(|(_, schema)| schema)
+        .collect::<Vec<_>>();
+
+    map.serialize_entry("oneOf", &variants)?;
+
+    // The schema gets inserted into the parent properties
+    map.serialize_entry("type-property", &one_of.type_property_entry.0)?;
+
+    map.end()
+}
+
 const fn assert_one_of_list_is_sorted(list: &[(&str, &Schema)]) {
     use std::cmp::Ordering;
 
@@ -1360,7 +1500,8 @@ impl Iterator for OneOfPropertyIterator {
 ///     ],
 /// ).schema();
 /// ```
-#[derive(Debug)]
+#[derive(Debug, Serialize)]
+#[serde(tag = "type", rename_all = "lowercase")]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 pub enum Schema {
     Null,
@@ -1370,10 +1511,81 @@ pub enum Schema {
     String(StringSchema),
     Object(ObjectSchema),
     Array(ArraySchema),
+    #[serde(serialize_with = "serialize_allof_schema")]
     AllOf(AllOfSchema),
+    #[serde(untagged)]
+    #[serde(serialize_with = "serialize_oneof_schema", rename = "oneOf")]
     OneOf(OneOfSchema),
 }
 
+/// Serialize the AllOf Schema
+///
+/// This will create one ObjectSchema and merge the properties of all the children.
+fn serialize_allof_schema<S>(all_of: &AllOfSchema, serializer: S) -> Result<S::Ok, S::Error>
+where
+    S: Serializer,
+{
+    use serde::ser::SerializeMap;
+
+    let mut map = serializer.serialize_map(Some(4))?;
+
+    // Add the top-level description
+    map.serialize_entry("description", &all_of.description)?;
+
+    // The type is always object
+    map.serialize_entry("type", "object")?;
+
+    let mut all_properties = HashMap::new();
+    let mut additional_properties = false;
+
+    for &schema in all_of.list {
+        if let Some(object_schema) = schema.object() {
+            // If any schema allows additional properties, the merged schema will too
+            if object_schema.additional_properties {
+                additional_properties = true;
+            }
+
+            // Add all properties from this schema
+            for (name, optional, prop_schema) in object_schema.properties {
+                all_properties.insert(*name, (*optional, *prop_schema));
+            }
+        } else if let Some(nested_all_of) = schema.all_of() {
+            // For nested AllOf schemas go through recursively
+            for &nested_schema in nested_all_of.list {
+                if let Some(object_schema) = nested_schema.object() {
+                    if object_schema.additional_properties {
+                        additional_properties = true;
+                    }
+
+                    for (name, optional, prop_schema) in object_schema.properties {
+                        all_properties.insert(*name, (*optional, *prop_schema));
+                    }
+                }
+            }
+        }
+    }
+
+    // Add the merged properties
+    let properties_entry = all_properties
+        .iter()
+        .map(|(name, (optional, schema))| {
+            (
+                *name,
+                OptionalSchema {
+                    optional: *optional,
+                    schema,
+                },
+            )
+        })
+        .collect::<HashMap<_, _>>();
+
+    map.serialize_entry("properties", &properties_entry)?;
+
+    map.serialize_entry("additional_properties", &additional_properties)?;
+
+    map.end()
+}
+
 impl Schema {
     /// Verify JSON value with `schema`.
     pub fn verify_json(&self, data: &Value) -> Result<(), Error> {
@@ -1694,10 +1906,12 @@ impl Schema {
 }
 
 /// A string enum entry. An enum entry must have a value and a description.
-#[derive(Clone, Debug)]
+#[derive(Clone, Debug, Serialize)]
+#[serde(transparent)]
 #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
 pub struct EnumEntry {
     pub value: &'static str,
+    #[serde(skip)]
     pub description: &'static str,
 }
 
@@ -1776,14 +1990,20 @@ impl EnumEntry {
 /// let data = PRODUCT_LIST_SCHEMA.parse_property_string("1,2"); // parse as Array
 /// assert!(data.is_ok());
 /// ```
+#[derive(Serialize)]
 pub enum ApiStringFormat {
     /// Enumerate all valid strings
+    #[serde(rename = "enum")]
     Enum(&'static [EnumEntry]),
     /// Use a regular expression to describe valid strings.
+    #[serde(rename = "pattern")]
     Pattern(&'static ConstRegexPattern),
     /// Use a schema to describe complex types encoded as string.
+    #[serde(rename = "format")]
     PropertyString(&'static Schema),
     /// Use a verification function.
+    /// Note: we can't serialize this, panic if we encounter this.
+    #[serde(skip)]
     VerifyFn(ApiStringVerifyFn),
 }
 
-- 
2.39.5




From w.bumiller at proxmox.com  Thu May  8 10:24:12 2025
From: w.bumiller at proxmox.com (Wolfgang Bumiller)
Date: Thu, 8 May 2025 10:24:12 +0200
Subject: [pbs-devel] [RFC PATCH] schema: allow serializing rust Schema
 to perl JsonSchema
In-Reply-To: <20250507163114.1162300-1-g.goller@proxmox.com>
References: <20250507163114.1162300-1-g.goller@proxmox.com>
Message-ID: <t26nlnfp2kclqkotha76a5vz23kjkl6as4aawkj2fw2rkkesox@mpbsb24dhxst>

On Wed, May 07, 2025 at 06:31:14PM +0200, Gabriel Goller wrote:
> Implement serde::Serialize on the rust Schema, so that we can serialize
> it and use it as a JsonSchema in perl. This allows us to write a single
> Schema in rust and reuse it in perl for the api properties.
> 
> The interesting bits (custom impls) are:
>  * Recursive oneOf type-property resolver
>  * oneOf and allOf implementation
>  * ApiStringFormat skip of ApiStringVerifyFn (which won't work obviously)

I'd like the commit to explain where we actually want to use/need this.
Long ago (before we had the `OneOf` schema I already started this, but
figured we didn't really need it anyway at the time.

> 
> Signed-off-by: Gabriel Goller <g.goller at proxmox.com>
> ---
> 
> This is kinda hard to test, because nothing actually fails when the properties
> are wrong and the whole allOf, oneOf and ApiStringFormat is a bit
> untransparent. So some properties could be wrongly serialized, but I
> think I got everything right. Looking over all the properties would be
> appreciated!

allOf and regular object-schema can use the same startegy, you serialize
a map and you can use the iterator you get from the `.properties()`
method to iterate over all properties.

One thing you can take out of my `staff/old-perl-serializing` commit is
the `ApiStringFormat::VerifyFn` handling, it uses perlmod to serialize
the function as an xsub with some magic to call the rust verifier code.

We *may* also consider using such a wrapper for regex patterns in case
we run into any significant differences between the perl & regex-crate
engines...

Also, technically the perl `JSONSchema` should be able to verify a
schema... (At least partially...)

> 
>  Cargo.toml                        |   1 +
>  proxmox-schema/Cargo.toml         |   4 +-
>  proxmox-schema/src/const_regex.rs |  12 ++
>  proxmox-schema/src/schema.rs      | 242 ++++++++++++++++++++++++++++--
>  4 files changed, 246 insertions(+), 13 deletions(-)
> 
> diff --git a/Cargo.toml b/Cargo.toml
> index 2ca0ea618707..a0d760ae8fc9 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -104,6 +104,7 @@ regex = "1.5"
>  serde = "1.0"
>  serde_cbor = "0.11.1"
>  serde_json = "1.0"
> +serde_with = "3.8.1"

^ Is it really worth introducing this for `None` handling :S

>  serde_plain = "1.0"
>  syn = { version = "2", features = [ "full", "visit-mut" ] }
>  tar = "0.4"
> diff --git a/proxmox-schema/Cargo.toml b/proxmox-schema/Cargo.toml
> index c8028aa52bd0..48ebf3a9005e 100644
> --- a/proxmox-schema/Cargo.toml
> +++ b/proxmox-schema/Cargo.toml
> @@ -15,8 +15,9 @@ rust-version.workspace = true
>  anyhow.workspace = true
>  const_format = { workspace = true, optional = true }
>  regex.workspace = true
> -serde.workspace = true
> +serde = { workspace = true, features = ["derive"] }
>  serde_json.workspace = true
> +serde_with.workspace = true
>  textwrap = "0.16"
>  
>  # the upid type needs this for 'getpid'
> @@ -27,7 +28,6 @@ proxmox-api-macro = { workspace = true, optional = true }
>  
>  [dev-dependencies]
>  url.workspace = true
> -serde = { workspace = true, features = [ "derive" ] }
>  proxmox-api-macro.workspace = true
>  
>  [features]
> diff --git a/proxmox-schema/src/const_regex.rs b/proxmox-schema/src/const_regex.rs
> index 8ddc41abedeb..56f6c27fa1de 100644
> --- a/proxmox-schema/src/const_regex.rs
> +++ b/proxmox-schema/src/const_regex.rs
> @@ -1,5 +1,7 @@
>  use std::fmt;
>  
> +use serde::Serialize;
> +
>  /// Helper to represent const regular expressions
>  ///
>  /// The current Regex::new() function is not `const_fn`. Unless that
> @@ -13,6 +15,16 @@ pub struct ConstRegexPattern {
>      pub regex_obj: fn() -> &'static regex::Regex,
>  }
>  
> +impl Serialize for ConstRegexPattern {

I'm not a huge fan of adding this trait to `ConstRegexPattern`...

> +    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
> +    where
> +        S: serde::Serializer,
> +    {
> +        // Get the compiled regex and serialize its pattern as a string
> +        serializer.serialize_str((self.regex_obj)().as_str())
> +    }
> +}
> +
>  impl fmt::Debug for ConstRegexPattern {
>      fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
>          write!(f, "{:?}", self.regex_string)
> diff --git a/proxmox-schema/src/schema.rs b/proxmox-schema/src/schema.rs
> index ddbbacd462a4..11461eaf6ace 100644
> --- a/proxmox-schema/src/schema.rs
> +++ b/proxmox-schema/src/schema.rs
> @@ -4,10 +4,12 @@
>  //! completely static API definitions that can be included within the programs read-only text
>  //! segment.
>  
> -use std::collections::HashSet;
> +use std::collections::{HashMap, HashSet};
>  use std::fmt;
>  
>  use anyhow::{bail, format_err, Error};
> +use serde::ser::{SerializeMap, SerializeStruct};
> +use serde::{Serialize, Serializer};
>  use serde_json::{json, Value};
>  
>  use crate::ConstRegexPattern;
> @@ -181,7 +183,8 @@ impl<'a> FromIterator<(&'a str, Error)> for ParameterError {
>  }
>  
>  /// Data type to describe boolean values
> -#[derive(Debug)]
> +#[serde_with::skip_serializing_none]
> +#[derive(Debug, Serialize)]
>  #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
>  #[non_exhaustive]
>  pub struct BooleanSchema {
> @@ -222,7 +225,8 @@ impl BooleanSchema {
>  }
>  
>  /// Data type to describe integer values.
> -#[derive(Debug)]
> +#[serde_with::skip_serializing_none]
> +#[derive(Debug, Serialize)]
>  #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
>  #[non_exhaustive]
>  pub struct IntegerSchema {
> @@ -304,7 +308,8 @@ impl IntegerSchema {
>  }
>  
>  /// Data type to describe (JSON like) number value
> -#[derive(Debug)]
> +#[serde_with::skip_serializing_none]
> +#[derive(Debug, Serialize)]
>  #[non_exhaustive]
>  pub struct NumberSchema {
>      pub description: &'static str,
> @@ -406,7 +411,8 @@ impl PartialEq for NumberSchema {
>  }
>  
>  /// Data type to describe string values.
> -#[derive(Debug)]
> +#[serde_with::skip_serializing_none]
> +#[derive(Debug, Serialize)]
>  #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
>  #[non_exhaustive]
>  pub struct StringSchema {

^ needs a `#[serde(rename_all = "camelCase")]`

> @@ -418,6 +424,7 @@ pub struct StringSchema {
>      /// Optional maximal length.
>      pub max_length: Option<usize>,
>      /// Optional microformat.
> +    #[serde(flatten)]
>      pub format: Option<&'static ApiStringFormat>,
>      /// A text representation of the format/type (used to generate documentation).
>      pub type_text: Option<&'static str>,

^ while this needs a `rename = "typetext"`

But also, if the type text is None, property strings get the type text
generated via `format::get_property_string_type_text`.

You can take this from my staff branch.

> @@ -534,7 +541,8 @@ impl StringSchema {
>  ///
>  /// All array elements are of the same type, as defined in the `items`
>  /// schema.
> -#[derive(Debug)]
> +#[serde_with::skip_serializing_none]
> +#[derive(Debug, Serialize)]
>  #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
>  #[non_exhaustive]
>  pub struct ArraySchema {

^ needs a `#[serde(rename_all = "camelCase")]`

> @@ -634,6 +642,43 @@ pub type SchemaPropertyEntry = (&'static str, bool, &'static Schema);
>  /// This is a workaround unless RUST can const_fn `Hash::new()`
>  pub type SchemaPropertyMap = &'static [SchemaPropertyEntry];
>  
> +/// A wrapper struct to hold the [`SchemaPropertyMap`] and serialize it nicely.
> +///
> +/// [`SchemaPropertyMap`] holds [`SchemaPropertyEntry`]s which are tuples. Tuples are serialized to
> +/// arrays, but we need a Map with the name (first item in the tuple) as a key and the optional
> +/// (second item in the tuple) as a property of the value.
> +pub struct SerializableSchemaProperties<'a>(&'a [SchemaPropertyEntry]);
> +
> +impl Serialize for SerializableSchemaProperties<'_> {
> +    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
> +    where
> +        S: Serializer,
> +    {
> +        let mut seq = serializer.serialize_map(Some(self.0.len()))?;
> +
> +        for (name, optional, schema) in self.0 {
> +            let schema_with_metadata = OptionalSchema {
> +                optional: *optional,
> +                schema,
> +            };
> +
> +            seq.serialize_entry(&name, &schema_with_metadata)?;
> +        }
> +
> +        seq.end()
> +    }
> +}
> +
> +/// A schema with a optional bool property.
> +///
> +/// The schema gets flattened, so it looks just like a normal Schema but with a optional property.
> +#[derive(Serialize)]
> +struct OptionalSchema<'a> {
> +    optional: bool,
> +    #[serde(flatten)]
> +    schema: &'a Schema,
> +}
> +
>  const fn assert_properties_sorted(properties: SchemaPropertyMap) {
>      use std::cmp::Ordering;
>  
> @@ -656,7 +701,7 @@ const fn assert_properties_sorted(properties: SchemaPropertyMap) {
>  /// Legacy property strings may contain shortcuts where the *value* of a specific key is used as a
>  /// *key* for yet another option. Most notably, PVE's `netX` properties use `<model>=<macaddr>`
>  /// instead of `model=<model>,macaddr=<macaddr>`.
> -#[derive(Clone, Copy, Debug)]
> +#[derive(Clone, Copy, Debug, Serialize)]
>  #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
>  pub struct KeyAliasInfo {
>      pub key_alias: &'static str,
> @@ -700,6 +745,77 @@ pub struct ObjectSchema {
>      pub key_alias_info: Option<KeyAliasInfo>,
>  }
>  
> +impl Serialize for ObjectSchema {
> +    fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
> +    where
> +        S: Serializer,

Move the contents of this into a:

    fn serialize_object_schema<S, T: ObjectSchemaType>(schema: &T, serializer: S) -> Result<S::Ok, S::Error>

> +    {
> +        let mut s = serializer.serialize_struct("ObjectSchema", 5)?;

(^ May as well just use `_map()` instead of `_struct()`, not sure
there's much of a benefit here.)

> +
> +        s.serialize_field("description", self.description)?;
> +        s.serialize_field("additional_properties", &self.additional_properties)?;
> +
> +        // Collect all OneOf type properties recursively

The `ObjectSchemaType` trait gives you a `.properties()` Iterator.

> +        let mut oneofs: Vec<SchemaPropertyEntry> = Vec::new();
> +        for (_, _, schema) in self.properties {
> +            collect_oneof_type_properties(schema, &mut oneofs);
> +        }
> +
> +        if !oneofs.is_empty() {
> +            // Extend the oneOf type-properties with the actual properties
> +            oneofs.extend_from_slice(self.properties);
> +            s.serialize_field("properties", &SerializableSchemaProperties(&oneofs))?;
> +        } else {
> +            s.serialize_field("properties", &SerializableSchemaProperties(self.properties))?;
> +        }
> +
> +        if let Some(default_key) = self.default_key {
> +            s.serialize_field("default_key", default_key)?;

^ Like `optional`, in perl `default_key` is part of the *property*, not
part of the `Object` schema.

> +        } else {
> +            s.skip_field("default_key")?;
> +        }
> +        if let Some(key_alias_info) = self.key_alias_info {

This does not exist in perl.

The key alias info works like this:
- All keys listed in the `KeyAliasInfo::values` alias to
  `KeyAliasInfo::alias`, and *which* key has been used is stored in the
  property named `KeyAliasInfo::key_alias`.

Assume we have

    key_alias_info = {
        keyAlias = "model",
        alias = "macaddr",
        values = [ "virtio", "e1000", ... ]
    }

This transforms:

    virtio=aa:aa:aa:aa:aa:aa,bridge=vmbr0

into

    model=virtio,macaddr=aa:aa:aa:aa:aa:aa,bridge=vmbr0

So you need to pass the key alias info (KAI from here on out) along to a
helper which serializes the property list which then:
- first finds the schema for the `KAI.alias`
- whenever a property is serialized which is listed in the `KAI.values`
  array it merges: `keyAlias = KAI.keyAlias, alias = KAI.alias` *into* the
  property
- iterate through all the `KAI.values` and serialize them with the
  schema found for the `KAI.alias` property
  

> +            s.serialize_field("key_alias_info", &key_alias_info)?;
> +        } else {
> +            s.skip_field("key_alias_info")?;
> +        }
> +
> +        s.end()
> +    }
> +}
> +
> +// Recursive function to find all OneOf type properties in a schema
> +fn collect_oneof_type_properties(schema: &Schema, result: &mut Vec<SchemaPropertyEntry>) {

Why are you recursing into arrays, property strings etc. here? The contents of arrays aren't
part of the containing object's property list?

> +    match schema {
> +        Schema::OneOf(oneof) => {
> +            result.push(*oneof.type_property_entry);
> +        }
> +        Schema::Array(array) => {
> +            // Recursively check the array schema
> +            collect_oneof_type_properties(array.items, result);
> +        }
> +        Schema::String(string) => {
> +            // Check the PropertyString Schema
> +            if let Some(ApiStringFormat::PropertyString(schema)) = string.format {
> +                collect_oneof_type_properties(schema, result);
> +            }
> +        }
> +        Schema::Object(obj) => {
> +            // Check all properties in the object
> +            for (_, _, prop_schema) in obj.properties {
> +                collect_oneof_type_properties(prop_schema, result);
> +            }
> +        }
> +        Schema::AllOf(all_of) => {
> +            // Check all schemas in the allOf list
> +            for &schema in all_of.list {
> +                collect_oneof_type_properties(schema, result);
> +            }
> +        }
> +        _ => {}
> +    }
> +}
> +
>  impl ObjectSchema {
>      /// Create a new `object` schema.
>      ///
> @@ -811,7 +927,7 @@ impl ObjectSchema {
>  ///
>  /// Technically this could also contain an `additional_properties` flag, however, in the JSON
>  /// Schema, this is not supported, so here we simply assume additional properties to be allowed.
> -#[derive(Debug)]
> +#[derive(Debug, Serialize)]
>  #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
>  #[non_exhaustive]
>  pub struct AllOfSchema {
> @@ -864,7 +980,7 @@ impl AllOfSchema {
>  /// In serde-language, we use an internally tagged enum representation.
>  ///
>  /// Note that these are limited to object schemas. Other schemas will produce errors.
> -#[derive(Debug)]
> +#[derive(Debug, Serialize)]
>  #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
>  #[non_exhaustive]
>  pub struct OneOfSchema {
> @@ -880,6 +996,30 @@ pub struct OneOfSchema {
>      pub list: &'static [(&'static str, &'static Schema)],
>  }
>  
> +fn serialize_oneof_schema<S>(one_of: &OneOfSchema, serializer: S) -> Result<S::Ok, S::Error>
> +where
> +    S: Serializer,
> +{

I'll first have to refresh my memory on the perl-side oneOf weirdness
for this.
Maybe @Dominik can take a look as well.
If we need rust->perl support here then the perl oneOf may very well
need some changes...

> +    use serde::ser::SerializeMap;
> +
> +    let mut map = serializer.serialize_map(Some(3))?;
> +
> +    map.serialize_entry("description", &one_of.description)?;
> +
> +    let variants = one_of
> +        .list
> +        .iter()
> +        .map(|(_, schema)| schema)
> +        .collect::<Vec<_>>();
> +
> +    map.serialize_entry("oneOf", &variants)?;
> +
> +    // The schema gets inserted into the parent properties
> +    map.serialize_entry("type-property", &one_of.type_property_entry.0)?;
> +
> +    map.end()
> +}
> +
>  const fn assert_one_of_list_is_sorted(list: &[(&str, &Schema)]) {
>      use std::cmp::Ordering;
>  
> @@ -1360,7 +1500,8 @@ impl Iterator for OneOfPropertyIterator {
>  ///     ],
>  /// ).schema();
>  /// ```
> -#[derive(Debug)]
> +#[derive(Debug, Serialize)]
> +#[serde(tag = "type", rename_all = "lowercase")]
>  #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
>  pub enum Schema {
>      Null,
> @@ -1370,10 +1511,81 @@ pub enum Schema {
>      String(StringSchema),
>      Object(ObjectSchema),
>      Array(ArraySchema),
> +    #[serde(serialize_with = "serialize_allof_schema")]

^ This should use `#[serde(rename = "object")]`. We don't have explicit
AllOfs in perl, because in perl we can easily just merge properties. In
rust we were unable to do that, since our schemas are all
compile-time-const.

>      AllOf(AllOfSchema),
> +    #[serde(untagged)]
> +    #[serde(serialize_with = "serialize_oneof_schema", rename = "oneOf")]
>      OneOf(OneOfSchema),
>  }
>  
> +/// Serialize the AllOf Schema
> +///
> +/// This will create one ObjectSchema and merge the properties of all the children.
> +fn serialize_allof_schema<S>(all_of: &AllOfSchema, serializer: S) -> Result<S::Ok, S::Error>

drop this and call the above mentioned `serialize_object_schema()`
instead.

> +where
> +    S: Serializer,
> +{
> +    use serde::ser::SerializeMap;
> +
> +    let mut map = serializer.serialize_map(Some(4))?;
> +
> +    // Add the top-level description
> +    map.serialize_entry("description", &all_of.description)?;
> +
> +    // The type is always object
> +    map.serialize_entry("type", "object")?;
> +
> +    let mut all_properties = HashMap::new();
> +    let mut additional_properties = false;
> +
> +    for &schema in all_of.list {
> +        if let Some(object_schema) = schema.object() {
> +            // If any schema allows additional properties, the merged schema will too
> +            if object_schema.additional_properties {
> +                additional_properties = true;
> +            }
> +
> +            // Add all properties from this schema
> +            for (name, optional, prop_schema) in object_schema.properties {
> +                all_properties.insert(*name, (*optional, *prop_schema));
> +            }
> +        } else if let Some(nested_all_of) = schema.all_of() {
> +            // For nested AllOf schemas go through recursively
> +            for &nested_schema in nested_all_of.list {
> +                if let Some(object_schema) = nested_schema.object() {
> +                    if object_schema.additional_properties {
> +                        additional_properties = true;
> +                    }
> +
> +                    for (name, optional, prop_schema) in object_schema.properties {
> +                        all_properties.insert(*name, (*optional, *prop_schema));
> +                    }
> +                }
> +            }
> +        }
> +    }
> +
> +    // Add the merged properties
> +    let properties_entry = all_properties
> +        .iter()
> +        .map(|(name, (optional, schema))| {
> +            (
> +                *name,
> +                OptionalSchema {
> +                    optional: *optional,
> +                    schema,
> +                },
> +            )
> +        })
> +        .collect::<HashMap<_, _>>();
> +
> +    map.serialize_entry("properties", &properties_entry)?;
> +
> +    map.serialize_entry("additional_properties", &additional_properties)?;
> +
> +    map.end()
> +}
> +
>  impl Schema {
>      /// Verify JSON value with `schema`.
>      pub fn verify_json(&self, data: &Value) -> Result<(), Error> {
> @@ -1694,10 +1906,12 @@ impl Schema {
>  }
>  
>  /// A string enum entry. An enum entry must have a value and a description.
> -#[derive(Clone, Debug)]
> +#[derive(Clone, Debug, Serialize)]
> +#[serde(transparent)]
>  #[cfg_attr(feature = "test-harness", derive(Eq, PartialEq))]
>  pub struct EnumEntry {
>      pub value: &'static str,
> +    #[serde(skip)]
>      pub description: &'static str,
>  }
>  
> @@ -1776,14 +1990,20 @@ impl EnumEntry {
>  /// let data = PRODUCT_LIST_SCHEMA.parse_property_string("1,2"); // parse as Array
>  /// assert!(data.is_ok());
>  /// ```
> +#[derive(Serialize)]
>  pub enum ApiStringFormat {
>      /// Enumerate all valid strings
> +    #[serde(rename = "enum")]
>      Enum(&'static [EnumEntry]),
>      /// Use a regular expression to describe valid strings.
> +    #[serde(rename = "pattern")]
>      Pattern(&'static ConstRegexPattern),
>      /// Use a schema to describe complex types encoded as string.
> +    #[serde(rename = "format")]
>      PropertyString(&'static Schema),
>      /// Use a verification function.
> +    /// Note: we can't serialize this, panic if we encounter this.
> +    #[serde(skip)]

^ as mentioned, see my old-perl-serializing branch...

>      VerifyFn(ApiStringVerifyFn),
>  }
>  
> -- 
> 2.39.5



From g.goller at proxmox.com  Thu May  8 11:52:27 2025
From: g.goller at proxmox.com (Gabriel Goller)
Date: Thu, 8 May 2025 11:52:27 +0200
Subject: [pbs-devel] [RFC PATCH] schema: allow serializing rust Schema
 to perl JsonSchema
In-Reply-To: <t26nlnfp2kclqkotha76a5vz23kjkl6as4aawkj2fw2rkkesox@mpbsb24dhxst>
References: <20250507163114.1162300-1-g.goller@proxmox.com>
 <t26nlnfp2kclqkotha76a5vz23kjkl6as4aawkj2fw2rkkesox@mpbsb24dhxst>
Message-ID: <qtifqo37bg47fb3uqsn452tgdnehqnnmp47kowqcx776nl75ap@tjgoaf5ucbhn>

On 08.05.2025 10:24, Wolfgang Bumiller wrote:
>On Wed, May 07, 2025 at 06:31:14PM +0200, Gabriel Goller wrote:
>> Implement serde::Serialize on the rust Schema, so that we can serialize
>> it and use it as a JsonSchema in perl. This allows us to write a single
>> Schema in rust and reuse it in perl for the api properties.
>>
>> The interesting bits (custom impls) are:
>>  * Recursive oneOf type-property resolver
>>  * oneOf and allOf implementation
>>  * ApiStringFormat skip of ApiStringVerifyFn (which won't work obviously)
>
>I'd like the commit to explain where we actually want to use/need this.
>Long ago (before we had the `OneOf` schema I already started this, but
>figured we didn't really need it anyway at the time.

We would primarily use this for the fabrics feature where SectionConfig
types are defined in rust using the api macro. In pve-network, we
retrieve these rust structs and expose them through the api. The current
issue is that we have to manually write the schema twice ? once in rust
and again in the perl api properties. We could eliminate this
duplication by getting the api schema generated in rust and use it in
the perl api properties.

Thanks for the thorough review!
I'll go through all the other stuff now!



From c.ebner at proxmox.com  Thu May  8 15:05:36 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:36 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 02/21] datastore: mark groups as
 trash on destroy
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-3-c.ebner@proxmox.com>

In order to implement the trash can functionality, mark all the
snapshots of the group and the group itself as trash instead of
deleting them right away. Cleanup of the group is deferred to the
garbage collection.

Groups and snapshots are marked by the trash marker file. New backups
to this group will check for the marker file (see subsequent
commits), clearing the whole group and all of the snapshots to
create a new snapshot within that group. Otherwise ownership
conflicts could arise. This implies that a new backup clears the
whole trashed group.

Snapshots already marked as trash within the same backup group will
be cleared as well when the group is requested to be destroyed with
skip trash.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 19 ++++++++++++++++---
 pbs-datastore/src/datastore.rs   |  4 ++--
 2 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index 76bcd15f5..9ce4cb0f8 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -215,7 +215,7 @@ impl BackupGroup {
     ///
     /// Returns `BackupGroupDeleteStats`, containing the number of deleted snapshots
     /// and number of protected snaphsots, which therefore were not removed.
-    pub fn destroy(&self) -> Result<BackupGroupDeleteStats, Error> {
+    pub fn destroy(&self, skip_trash: bool) -> Result<BackupGroupDeleteStats, Error> {
         let _guard = self
             .lock()
             .with_context(|| format!("while destroying group '{self:?}'"))?;
@@ -229,14 +229,20 @@ impl BackupGroup {
                 delete_stats.increment_protected_snapshots();
                 continue;
             }
-            snap.destroy(false, false)?;
+            snap.destroy(false, skip_trash)?;
             delete_stats.increment_removed_snapshots();
         }
 
         // Note: make sure the old locking mechanism isn't used as `remove_dir_all` is not safe in
         // that case
         if delete_stats.all_removed() && !*OLD_LOCKING {
-            self.remove_group_dir()?;
+            if skip_trash {
+                self.remove_group_dir()?;
+            } else {
+                let path = self.full_group_path().join(TRASH_MARKER_FILENAME);
+                let _trash_file =
+                    std::fs::File::create(path).context("failed to set trash file")?;
+            }
             delete_stats.increment_removed_groups();
         }
 
@@ -245,6 +251,13 @@ impl BackupGroup {
 
     /// Helper function, assumes that no more snapshots are present in the group.
     fn remove_group_dir(&self) -> Result<(), Error> {
+        let trash_path = self.full_group_path().join(TRASH_MARKER_FILENAME);
+        if let Err(err) = std::fs::remove_file(&trash_path) {
+            if err.kind() != std::io::ErrorKind::NotFound {
+                bail!("removing the trash file '{trash_path:?}' failed - {err}")
+            }
+        }
+
         let owner_path = self.store.owner_path(&self.ns, &self.group);
 
         std::fs::remove_file(&owner_path).map_err(|err| {
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 6df26e825..e546bc532 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -581,7 +581,7 @@ impl DataStore {
         let mut stats = BackupGroupDeleteStats::default();
 
         for group in self.iter_backup_groups(ns.to_owned())? {
-            let delete_stats = group?.destroy()?;
+            let delete_stats = group?.destroy(true)?;
             stats.add(&delete_stats);
             removed_all_groups = removed_all_groups && delete_stats.all_removed();
         }
@@ -674,7 +674,7 @@ impl DataStore {
     ) -> Result<BackupGroupDeleteStats, Error> {
         let backup_group = self.backup_group(ns.clone(), backup_group.clone());
 
-        backup_group.destroy()
+        backup_group.destroy(true)
     }
 
     /// Remove a backup directory including all content
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:39 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:39 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 05/21] sync: ignore trashed
 snapshots when reading from local source
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-6-c.ebner@proxmox.com>

Trashed snapshots should never be synced, so filter them out when
listing the snapshots backup directories to be synced.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/server/sync.rs | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/server/sync.rs b/src/server/sync.rs
index 09814ef0c..3de2ec9a4 100644
--- a/src/server/sync.rs
+++ b/src/server/sync.rs
@@ -461,6 +461,7 @@ impl SyncSource for LocalSource {
             .backup_group(namespace.clone(), group.clone())
             .iter_snapshots()?
             .filter_map(Result::ok)
+            .filter(|snapshot| !snapshot.is_trashed())
             .map(|snapshot| snapshot.dir().to_owned())
             .collect::<Vec<BackupDir>>())
     }
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:35 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:35 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 01/21] datastore/api: mark
 snapshots as trash on destroy
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-2-c.ebner@proxmox.com>

In order to implement the trash can functionality, mark snapshots
as trash instead of removing them by default. However, provide a
`skip-trash` flag to opt-out and destroy the snapshot including it's
contents immediately.

Trashed snapshots are marked by creating a `.trashed` marker file
inside the snapshot folder. Actual removal of the snapshot will be
deferred to the garbage collection task.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 66 ++++++++++++++++++--------------
 pbs-datastore/src/datastore.rs   |  2 +-
 src/api2/admin/datastore.rs      | 18 ++++++++-
 3 files changed, 55 insertions(+), 31 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index d4732fdd9..76bcd15f5 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -21,6 +21,7 @@ use crate::manifest::{BackupManifest, MANIFEST_LOCK_NAME};
 use crate::{DataBlob, DataStore};
 
 pub const DATASTORE_LOCKS_DIR: &str = "/run/proxmox-backup/locks";
+pub const TRASH_MARKER_FILENAME: &str = ".trashed";
 
 // TODO: Remove with PBS 5
 // Note: The `expect()` call here will only happen if we can neither confirm nor deny the existence
@@ -228,7 +229,7 @@ impl BackupGroup {
                 delete_stats.increment_protected_snapshots();
                 continue;
             }
-            snap.destroy(false)?;
+            snap.destroy(false, false)?;
             delete_stats.increment_removed_snapshots();
         }
 
@@ -575,7 +576,8 @@ impl BackupDir {
     /// Destroy the whole snapshot, bails if it's protected
     ///
     /// Setting `force` to true skips locking and thus ignores if the backup is currently in use.
-    pub fn destroy(&self, force: bool) -> Result<(), Error> {
+    /// Setting `skip_trash` to true will remove the snapshot instead of marking it as trash.
+    pub fn destroy(&self, force: bool, skip_trash: bool) -> Result<(), Error> {
         let (_guard, _manifest_guard);
         if !force {
             _guard = self
@@ -588,37 +590,45 @@ impl BackupDir {
             bail!("cannot remove protected snapshot"); // use special error type?
         }
 
-        let full_path = self.full_path();
-        log::info!("removing backup snapshot {:?}", full_path);
-        std::fs::remove_dir_all(&full_path).map_err(|err| {
-            format_err!("removing backup snapshot {:?} failed - {}", full_path, err,)
-        })?;
+        let mut full_path = self.full_path();
+        log::info!("removing backup snapshot {full_path:?}");
+        if skip_trash {
+            std::fs::remove_dir_all(&full_path).map_err(|err| {
+                format_err!("removing backup snapshot {full_path:?} failed - {err}")
+            })?;
+        } else {
+            full_path.push(TRASH_MARKER_FILENAME);
+            let _trash_file =
+                std::fs::File::create(full_path).context("failed to set trash file")?;
+        }
 
         // remove no longer needed lock files
         let _ = std::fs::remove_file(self.manifest_lock_path()); // ignore errors
         let _ = std::fs::remove_file(self.lock_path()); // ignore errors
 
-        let group = BackupGroup::from(self);
-        let guard = group.lock().with_context(|| {
-            format!("while checking if group '{group:?}' is empty during snapshot destruction")
-        });
-
-        // Only remove the group if all of the following is true:
-        //
-        // - we can lock it: if we can't lock the group, it is still in use (either by another
-        //   backup process or a parent caller (who needs to take care that empty groups are
-        //   removed themselves).
-        // - it is now empty: if the group isn't empty, removing it will fail (to avoid removing
-        //   backups that might still be used).
-        // - the new locking mechanism is used: if the old mechanism is used, a group removal here
-        //   could lead to a race condition.
-        //
-        // Do not error out, as we have already removed the snapshot, there is nothing a user could
-        // do to rectify the situation.
-        if guard.is_ok() && group.list_backups()?.is_empty() && !*OLD_LOCKING {
-            group.remove_group_dir()?;
-        } else if let Err(err) = guard {
-            log::debug!("{err:#}");
+        if skip_trash {
+            let group = BackupGroup::from(self);
+            let guard = group.lock().with_context(|| {
+                format!("while checking if group '{group:?}' is empty during snapshot destruction")
+            });
+
+            // Only remove the group if all of the following is true:
+            //
+            // - we can lock it: if we can't lock the group, it is still in use (either by another
+            //   backup process or a parent caller (who needs to take care that empty groups are
+            //   removed themselves).
+            // - it is now empty: if the group isn't empty, removing it will fail (to avoid removing
+            //   backups that might still be used).
+            // - the new locking mechanism is used: if the old mechanism is used, a group removal here
+            //   could lead to a race condition.
+            //
+            // Do not error out, as we have already removed the snapshot, there is nothing a user could
+            // do to rectify the situation.
+            if guard.is_ok() && group.list_backups()?.is_empty() && !*OLD_LOCKING {
+                group.remove_group_dir()?;
+            } else if let Err(err) = guard {
+                log::debug!("{err:#}");
+            }
         }
 
         Ok(())
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index cbf78ecb6..6df26e825 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -686,7 +686,7 @@ impl DataStore {
     ) -> Result<(), Error> {
         let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
 
-        backup_dir.destroy(force)
+        backup_dir.destroy(force, true)
     }
 
     /// Returns the time of the last successful backup
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 392494488..aafd1bbd7 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -402,6 +402,12 @@ pub async fn list_snapshot_files(
                 type: pbs_api_types::BackupDir,
                 flatten: true,
             },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                default: false,
+                description: "Immediately remove the snapshot, not marking it as trash.",
+            },
         },
     },
     access: {
@@ -415,6 +421,7 @@ pub async fn delete_snapshot(
     store: String,
     ns: Option<BackupNamespace>,
     backup_dir: pbs_api_types::BackupDir,
+    skip_trash: bool,
     _info: &ApiMethod,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Value, Error> {
@@ -435,7 +442,7 @@ pub async fn delete_snapshot(
 
         let snapshot = datastore.backup_dir(ns, backup_dir)?;
 
-        snapshot.destroy(false)?;
+        snapshot.destroy(false, skip_trash)?;
 
         Ok(Value::Null)
     })
@@ -979,6 +986,12 @@ pub fn verify(
                 optional: true,
                 description: "Spins up an asynchronous task that does the work.",
             },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                default: false,
+                description: "Immediately remove the snapshot, not marking it as trash.",
+            },
         },
     },
     returns: pbs_api_types::ADMIN_DATASTORE_PRUNE_RETURN_TYPE,
@@ -995,6 +1008,7 @@ pub fn prune(
     keep_options: KeepOptions,
     store: String,
     ns: Option<BackupNamespace>,
+    skip_trash: bool,
     param: Value,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Value, Error> {
@@ -1098,7 +1112,7 @@ pub fn prune(
             });
 
             if !keep {
-                if let Err(err) = backup_dir.destroy(false) {
+                if let Err(err) = backup_dir.destroy(false, skip_trash) {
                     warn!(
                         "failed to remove dir {:?}: {}",
                         backup_dir.relative_path(),
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:41 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:41 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 07/21] sync: ignore trashed
 groups in local source reader
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-8-c.ebner@proxmox.com>

Check and exclude backup groups which have been marked as trash from
sync.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 7 +++++++
 src/server/sync.rs               | 1 +
 2 files changed, 8 insertions(+)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index 189ed28ad..b4fabb2cc 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -105,6 +105,13 @@ impl BackupGroup {
         self.full_group_path().exists()
     }
 
+    /// Check if the group is currently marked as trash by checking the presence of the trash
+    /// marker file in the group's directory
+    pub fn is_trashed(&self) -> bool {
+        let path = self.full_group_path().join(TRASH_MARKER_FILENAME);
+        path.exists()
+    }
+
     pub fn list_backups(&self, filter: ListBackupFilter) -> Result<Vec<BackupInfo>, Error> {
         let mut list = vec![];
 
diff --git a/src/server/sync.rs b/src/server/sync.rs
index 3de2ec9a4..ce338fbbe 100644
--- a/src/server/sync.rs
+++ b/src/server/sync.rs
@@ -447,6 +447,7 @@ impl SyncSource for LocalSource {
             Some(owner),
         )?
         .filter_map(Result::ok)
+        .filter(|backup_group| !backup_group.is_trashed())
         .map(|backup_group| backup_group.group().clone())
         .collect::<Vec<pbs_api_types::BackupGroup>>())
     }
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:40 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:40 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 06/21] api: tape: check trash
 marker when trying to write snapshot
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-7-c.ebner@proxmox.com>

Since snapshots might be marked as trash, the snapshot directory
can still be present until cleaned up by garbage collection.

Therefore, check for the presence of the trash marker after acquiring
the locked snapshot reader and skip over marked ones.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/tape/backup.rs | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
index 923cb7834..17c8bc605 100644
--- a/src/api2/tape/backup.rs
+++ b/src/api2/tape/backup.rs
@@ -574,7 +574,13 @@ fn backup_snapshot(
     info!("backup snapshot {snapshot_path:?}");
 
     let snapshot_reader = match snapshot.locked_reader() {
-        Ok(reader) => reader,
+        Ok(reader) => {
+            if snapshot.is_trashed() {
+                info!("snapshot {snapshot_path:?} trashed, skipping");
+                return Ok(SnapshotBackupResult::Ignored);
+            }
+            reader
+        }
         Err(err) => {
             if !snapshot.full_path().exists() {
                 // we got an error and the dir does not exist,
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:45 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:45 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 11/21] datastore: check for
 trash marker in namespace exists check
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-12-c.ebner@proxmox.com>

Namespaces which have been marked as trash are not considered
existing. This makes sure that sync jobs or tape backup jobs try to
newly create the namespace, thereby clearing all pre-existing
contents in it.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index b1d81e199..ffc6a7039 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -567,7 +567,11 @@ impl DataStore {
     pub fn namespace_exists(&self, ns: &BackupNamespace) -> bool {
         let mut path = self.base_path();
         path.push(ns.path());
-        path.exists()
+        if !path.exists() {
+            return false;
+        }
+        path.push(TRASH_MARKER_FILENAME);
+        !path.exists()
     }
 
     /// Remove the namespace and all it's parent components from the trash by removing the trash or
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:37 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:37 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 03/21] datastore: allow
 filtering of backups by their trash status
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-4-c.ebner@proxmox.com>

Extends the BackupGroup::list_backups method by an enum parameter to
filter backup snapshots based on their trash status.

This allows to reuse the same logic for listing active, trashed or
all snapshots.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 33 +++++++++++++++++++++++++++++---
 pbs-datastore/src/datastore.rs   |  4 ++--
 src/api2/admin/datastore.rs      | 10 +++++-----
 src/api2/tape/backup.rs          |  4 ++--
 src/backup/verify.rs             |  4 ++--
 src/server/prune_job.rs          |  3 ++-
 src/server/pull.rs               |  3 ++-
 7 files changed, 45 insertions(+), 16 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index 9ce4cb0f8..a8c864ac8 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -52,6 +52,12 @@ impl fmt::Debug for BackupGroup {
     }
 }
 
+pub enum ListBackupFilter {
+    Active,
+    All,
+    Trashed,
+}
+
 impl BackupGroup {
     pub(crate) fn new(
         store: Arc<DataStore>,
@@ -99,7 +105,7 @@ impl BackupGroup {
         self.full_group_path().exists()
     }
 
-    pub fn list_backups(&self) -> Result<Vec<BackupInfo>, Error> {
+    pub fn list_backups(&self, filter: ListBackupFilter) -> Result<Vec<BackupInfo>, Error> {
         let mut list = vec![];
 
         let path = self.full_group_path();
@@ -117,6 +123,19 @@ impl BackupGroup {
                 let files = list_backup_files(l2_fd, backup_time)?;
 
                 let protected = backup_dir.is_protected();
+                match filter {
+                    ListBackupFilter::All => (),
+                    ListBackupFilter::Trashed => {
+                        if !backup_dir.is_trashed() {
+                            return Ok(());
+                        }
+                    }
+                    ListBackupFilter::Active => {
+                        if backup_dir.is_trashed() {
+                            return Ok(());
+                        }
+                    }
+                }
 
                 list.push(BackupInfo {
                     backup_dir,
@@ -132,7 +151,7 @@ impl BackupGroup {
 
     /// Finds the latest backup inside a backup group
     pub fn last_backup(&self, only_finished: bool) -> Result<Option<BackupInfo>, Error> {
-        let backups = self.list_backups()?;
+        let backups = self.list_backups(ListBackupFilter::Active)?;
         Ok(backups
             .into_iter()
             .filter(|item| !only_finished || item.is_finished())
@@ -480,6 +499,11 @@ impl BackupDir {
         path.exists()
     }
 
+    pub fn is_trashed(&self) -> bool {
+        let path = self.full_path().join(TRASH_MARKER_FILENAME);
+        path.exists()
+    }
+
     pub fn backup_time_to_string(backup_time: i64) -> Result<String, Error> {
         // fixme: can this fail? (avoid unwrap)
         proxmox_time::epoch_to_rfc3339_utc(backup_time)
@@ -637,7 +661,10 @@ impl BackupDir {
             //
             // Do not error out, as we have already removed the snapshot, there is nothing a user could
             // do to rectify the situation.
-            if guard.is_ok() && group.list_backups()?.is_empty() && !*OLD_LOCKING {
+            if guard.is_ok()
+                && group.list_backups(ListBackupFilter::All)?.is_empty()
+                && !*OLD_LOCKING
+            {
                 group.remove_group_dir()?;
             } else if let Err(err) = guard {
                 log::debug!("{err:#}");
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index e546bc532..867324380 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -28,7 +28,7 @@ use pbs_api_types::{
 };
 use pbs_config::BackupLockGuard;
 
-use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, OLD_LOCKING};
+use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, ListBackupFilter, OLD_LOCKING};
 use crate::chunk_store::ChunkStore;
 use crate::dynamic_index::{DynamicIndexReader, DynamicIndexWriter};
 use crate::fixed_index::{FixedIndexReader, FixedIndexWriter};
@@ -1158,7 +1158,7 @@ impl DataStore {
                         _ => bail!("exhausted retries and unexpected counter overrun"),
                     };
 
-                    let mut snapshots = match group.list_backups() {
+                    let mut snapshots = match group.list_backups(ListBackupFilter::All) {
                         Ok(snapshots) => snapshots,
                         Err(err) => {
                             if group.exists() {
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index aafd1bbd7..133a6d658 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -51,7 +51,7 @@ use pbs_api_types::{
 };
 use pbs_client::pxar::{create_tar, create_zip};
 use pbs_config::CachedUserInfo;
-use pbs_datastore::backup_info::BackupInfo;
+use pbs_datastore::backup_info::{BackupInfo, ListBackupFilter};
 use pbs_datastore::cached_chunk_reader::CachedChunkReader;
 use pbs_datastore::catalog::{ArchiveEntry, CatalogReader};
 use pbs_datastore::data_blob::DataBlob;
@@ -223,7 +223,7 @@ pub fn list_groups(
                 return Ok(group_info);
             }
 
-            let snapshots = match group.list_backups() {
+            let snapshots = match group.list_backups(ListBackupFilter::Active) {
                 Ok(snapshots) => snapshots,
                 Err(_) => return Ok(group_info),
             };
@@ -624,7 +624,7 @@ unsafe fn list_snapshots_blocking(
             return Ok(snapshots);
         }
 
-        let group_backups = group.list_backups()?;
+        let group_backups = group.list_backups(ListBackupFilter::Active)?;
 
         snapshots.extend(
             group_backups
@@ -657,7 +657,7 @@ async fn get_snapshots_count(
                 Ok(group) => group,
                 Err(_) => return Ok(counts), // TODO: add this as error counts?
             };
-            let snapshot_count = group.list_backups()?.len() as u64;
+            let snapshot_count = group.list_backups(ListBackupFilter::Active)?.len() as u64;
 
             // only include groups with snapshots, counting/displaying empty groups can confuse
             if snapshot_count > 0 {
@@ -1042,7 +1042,7 @@ pub fn prune(
     }
     let mut prune_result: Vec<PruneResult> = Vec::new();
 
-    let list = group.list_backups()?;
+    let list = group.list_backups(ListBackupFilter::Active)?;
 
     let mut prune_info = compute_prune_info(list, &keep_options)?;
 
diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
index 31293a9a9..923cb7834 100644
--- a/src/api2/tape/backup.rs
+++ b/src/api2/tape/backup.rs
@@ -17,7 +17,7 @@ use pbs_api_types::{
 };
 
 use pbs_config::CachedUserInfo;
-use pbs_datastore::backup_info::{BackupDir, BackupInfo};
+use pbs_datastore::backup_info::{BackupDir, BackupInfo, ListBackupFilter};
 use pbs_datastore::{DataStore, StoreProgress};
 
 use crate::tape::TapeNotificationMode;
@@ -433,7 +433,7 @@ fn backup_worker(
         progress.done_snapshots = 0;
         progress.group_snapshots = 0;
 
-        let snapshot_list = group.list_backups()?;
+        let snapshot_list = group.list_backups(ListBackupFilter::Active)?;
 
         // filter out unfinished backups
         let mut snapshot_list: Vec<_> = snapshot_list
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 3d2cba8ac..1b5e8564b 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -14,7 +14,7 @@ use pbs_api_types::{
     CryptMode, SnapshotVerifyState, VerifyState, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_VERIFY,
     UPID,
 };
-use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo};
+use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo, ListBackupFilter};
 use pbs_datastore::index::IndexFile;
 use pbs_datastore::manifest::{BackupManifest, FileInfo};
 use pbs_datastore::{DataBlob, DataStore, StoreProgress};
@@ -411,7 +411,7 @@ pub fn verify_backup_group(
     filter: Option<&dyn Fn(&BackupManifest) -> bool>,
 ) -> Result<Vec<String>, Error> {
     let mut errors = Vec::new();
-    let mut list = match group.list_backups() {
+    let mut list = match group.list_backups(ListBackupFilter::Active) {
         Ok(list) => list,
         Err(err) => {
             info!(
diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
index 1c86647a0..596afe086 100644
--- a/src/server/prune_job.rs
+++ b/src/server/prune_job.rs
@@ -1,6 +1,7 @@
 use std::sync::Arc;
 
 use anyhow::Error;
+use pbs_datastore::backup_info::ListBackupFilter;
 use tracing::{info, warn};
 
 use pbs_api_types::{
@@ -54,7 +55,7 @@ pub fn prune_datastore(
     )? {
         let group = group?;
         let ns = group.backup_ns();
-        let list = group.list_backups()?;
+        let list = group.list_backups(ListBackupFilter::Active)?;
 
         let mut prune_info = compute_prune_info(list, &prune_options.keep)?;
         prune_info.reverse(); // delete older snapshots first
diff --git a/src/server/pull.rs b/src/server/pull.rs
index b1724c142..50d7b0806 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -7,6 +7,7 @@ use std::sync::{Arc, Mutex};
 use std::time::SystemTime;
 
 use anyhow::{bail, format_err, Error};
+use pbs_datastore::backup_info::ListBackupFilter;
 use proxmox_human_byte::HumanByte;
 use tracing::info;
 
@@ -660,7 +661,7 @@ async fn pull_group(
             .target
             .store
             .backup_group(target_ns.clone(), group.clone());
-        let local_list = group.list_backups()?;
+        let local_list = group.list_backups(ListBackupFilter::Active)?;
         for info in local_list {
             let snapshot = info.backup_dir;
             if source_snapshots.contains(&snapshot.backup_time()) {
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:46 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:46 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 12/21] datastore: clear trashed
 snapshot dir if re-creation requested
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-13-c.ebner@proxmox.com>

If a previously trashed snapshot has been requested for re-creation
(e.g. by a sync job in push direction), drop the contents of the
currently trashed snapshot.
The snapshot directory itself is already locked at that point, either
by the old locking mechanism acting on the directory itself or by the
new locking mechanism. Therefore, concurrent operations can be
excluded.

For the call site this acts as if the snapshot directory has been
newly created.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index ffc6a7039..4f7766c36 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -951,8 +951,9 @@ impl DataStore {
     ) -> Result<(PathBuf, bool, BackupLockGuard), Error> {
         let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
         let relative_path = backup_dir.relative_path();
+        let full_path = backup_dir.full_path();
 
-        match std::fs::create_dir(backup_dir.full_path()) {
+        match std::fs::create_dir(&full_path) {
             Ok(_) => {
                 let guard = backup_dir.lock().with_context(|| {
                     format!("while creating new locked snapshot '{backup_dir:?}'")
@@ -963,6 +964,32 @@ impl DataStore {
                 let guard = backup_dir
                     .lock()
                     .with_context(|| format!("while creating locked snapshot '{backup_dir:?}'"))?;
+
+                if backup_dir.is_trashed() {
+                    info!("clear trashed backup snapshot {full_path:?}");
+                    let dir_entries = std::fs::read_dir(&full_path).context(
+                        "failed to read directory contents during cleanup of trashed snapshot",
+                    )?;
+                    for entry in dir_entries {
+                        let entry = entry.context(
+                            "failed to read directory entry during clenup of trashed snapshot",
+                        )?;
+                        // Only expect regular file entries
+                        std::fs::remove_file(entry.path()).context(
+                            "failed to remove directory entry during clenup of trashed snapshot",
+                        )?;
+                    }
+                    let group = BackupGroup::from(backup_dir);
+                    let group_trash_file = group.full_group_path().join(TRASH_MARKER_FILENAME);
+                    if let Err(err) = std::fs::remove_file(&group_trash_file) {
+                        if err.kind() != std::io::ErrorKind::NotFound {
+                            bail!("failed to remove group trash file of trashed snapshot");
+                        }
+                    }
+                    self.untrash_namespace(ns)?;
+                    return Ok((relative_path, true, guard));
+                }
+
                 Ok((relative_path, false, guard))
             }
             Err(e) => Err(e.into()),
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:50 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:50 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 16/21] api: datastore: add flag
 to list trashed snapshots only
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-17-c.ebner@proxmox.com>

Allows to conditionally show either active or trashed backup
snapshots, the latter being used when displaying the contents of the
trash for given datastore.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 84c0bf5b4..cbd24c729 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -473,6 +473,12 @@ pub async fn delete_snapshot(
                 optional: true,
                 schema: BACKUP_ID_SCHEMA,
             },
+            "trashed": {
+                type: bool,
+                optional: true,
+                default: false,
+                description: "List trashed snapshots only."
+            },
         },
     },
     returns: pbs_api_types::ADMIN_DATASTORE_LIST_SNAPSHOTS_RETURN_TYPE,
@@ -488,6 +494,7 @@ pub async fn list_snapshots(
     ns: Option<BackupNamespace>,
     backup_type: Option<BackupType>,
     backup_id: Option<String>,
+    trashed: bool,
     _param: Value,
     _info: &ApiMethod,
     rpcenv: &mut dyn RpcEnvironment,
@@ -495,7 +502,7 @@ pub async fn list_snapshots(
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
 
     tokio::task::spawn_blocking(move || unsafe {
-        list_snapshots_blocking(store, ns, backup_type, backup_id, auth_id)
+        list_snapshots_blocking(store, ns, backup_type, backup_id, auth_id, trashed)
     })
     .await
     .map_err(|err| format_err!("failed to await blocking task: {err}"))?
@@ -508,6 +515,7 @@ unsafe fn list_snapshots_blocking(
     backup_type: Option<BackupType>,
     backup_id: Option<String>,
     auth_id: Authid,
+    trashed: bool,
 ) -> Result<Vec<SnapshotListItem>, Error> {
     let ns = ns.unwrap_or_default();
 
@@ -631,7 +639,12 @@ unsafe fn list_snapshots_blocking(
             return Ok(snapshots);
         }
 
-        let group_backups = group.list_backups(ListBackupFilter::Active)?;
+        let filter = if trashed {
+            ListBackupFilter::Trashed
+        } else {
+            ListBackupFilter::Active
+        };
+        let group_backups = group.list_backups(filter)?;
 
         snapshots.extend(
             group_backups
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:54 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:54 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 20/21] ui: drop 'permanent' in
 group/snapshot forget, default is to trash
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-21-c.ebner@proxmox.com>

Soften the message as the snapshots and groups will not be deleted
immediately anymore, but rather moved to the trash, from where they
still can be restored until the next garbage collection run.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/datastore/Content.js | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/www/datastore/Content.js b/www/datastore/Content.js
index fffd8c160..a6e28a773 100644
--- a/www/datastore/Content.js
+++ b/www/datastore/Content.js
@@ -1029,9 +1029,9 @@ Ext.define('PBS.DataStoreContent', {
 			if (data.ty === 'ns') {
 			    tip = gettext("Remove namespace '{0}'");
 			} else if (data.ty === 'dir') {
-			    tip = gettext("Permanently forget snapshot '{0}'");
+			    tip = gettext("Forget snapshot '{0}'");
 			} else if (data.ty === 'group') {
-			    tip = gettext("Permanently forget group '{0}'");
+			    tip = gettext("Forget group '{0}'");
 			}
 			return Ext.String.format(tip, v);
 		    },
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:55 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:55 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 21/21] ui: allow to skip trash
 on namespace deletion
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-22-c.ebner@proxmox.com>

In order to bypass the trash, add a check box to the delete namespace
dialog and set the `skip-trash` api call parameter accordingly.

Also, extend the warning to mention that when the trash is skipped,
also already trashed items in the namespace are removed permanently.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/window/NamespaceEdit.js | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/www/window/NamespaceEdit.js b/www/window/NamespaceEdit.js
index a9a440bbf..79a045823 100644
--- a/www/window/NamespaceEdit.js
+++ b/www/window/NamespaceEdit.js
@@ -83,11 +83,30 @@ Ext.define('PBS.window.NamespaceDelete', {
 		},
 	    },
 	},
+	{
+	    xtype: 'proxmoxcheckbox',
+	    name: 'skip-trash',
+	    boxLabel: gettext('Skip Trash (delete content immediately)'),
+	    value: false,
+	    listeners: {
+		change: function(field, value) {
+		    let win = field.up('proxmoxSafeDestroy');
+		    if (value) {
+			win.params['skip-trash'] = value;
+		    } else {
+			delete win.params['skip-trash'];
+		    }
+		},
+	    },
+	    bind: {
+		disabled: '{!rmGroups.checked}',
+	    },
+	},
 	{
 	    xtype: 'box',
 	    padding: '5 0 0 0',
 	    html: `<span class="pmx-hint">${gettext('Note')}</span>: `
-	      + gettext('This will permanently remove all backups from the current namespace and all namespaces below it!'),
+	      + gettext('This will remove all backups from the current namespace and all namespaces below it! If the trash is skipped, this will remove also previously trashed items'),
 	    bind: {
 		hidden: '{!rmGroups.checked}',
 	    },
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:42 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:42 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 08/21] datastore: namespace: add
 filter for trash status
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-9-c.ebner@proxmox.com>

As for snapshots, allow to filter namespaces based on their trash
status while iterating.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/examples/ls-snapshots.rs |  8 ++-
 pbs-datastore/src/datastore.rs         | 24 ++++---
 pbs-datastore/src/hierarchy.rs         | 91 ++++++++++++++++++++++++--
 pbs-datastore/src/lib.rs               |  1 +
 src/api2/admin/namespace.rs            | 16 +++--
 src/api2/tape/backup.rs                |  8 ++-
 src/backup/hierarchy.rs                | 26 +++++---
 src/server/pull.rs                     |  8 ++-
 src/server/sync.rs                     |  5 +-
 9 files changed, 149 insertions(+), 38 deletions(-)

diff --git a/pbs-datastore/examples/ls-snapshots.rs b/pbs-datastore/examples/ls-snapshots.rs
index 2eeea4892..1d3707a17 100644
--- a/pbs-datastore/examples/ls-snapshots.rs
+++ b/pbs-datastore/examples/ls-snapshots.rs
@@ -2,7 +2,7 @@ use std::path::PathBuf;
 
 use anyhow::{bail, Error};
 
-use pbs_datastore::DataStore;
+use pbs_datastore::{DataStore, NamespaceListFilter};
 
 fn run() -> Result<(), Error> {
     let base: PathBuf = match std::env::args().nth(1) {
@@ -20,7 +20,11 @@ fn run() -> Result<(), Error> {
 
     let store = unsafe { DataStore::open_path("", base, None)? };
 
-    for ns in store.recursive_iter_backup_ns_ok(Default::default(), max_depth)? {
+    for ns in store.recursive_iter_backup_ns_ok(
+        Default::default(),
+        max_depth,
+        NamespaceListFilter::Active,
+    )? {
         println!("found namespace store:/{}", ns);
 
         for group in store.iter_backup_groups(ns)? {
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 867324380..aee69768f 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -32,7 +32,9 @@ use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, ListBackupFilter, O
 use crate::chunk_store::ChunkStore;
 use crate::dynamic_index::{DynamicIndexReader, DynamicIndexWriter};
 use crate::fixed_index::{FixedIndexReader, FixedIndexWriter};
-use crate::hierarchy::{ListGroups, ListGroupsType, ListNamespaces, ListNamespacesRecursive};
+use crate::hierarchy::{
+    ListGroups, ListGroupsType, ListNamespaces, ListNamespacesRecursive, NamespaceListFilter,
+};
 use crate::index::IndexFile;
 use crate::task_tracking::{self, update_active_operations};
 use crate::DataBlob;
@@ -617,7 +619,7 @@ impl DataStore {
         let mut stats = BackupGroupDeleteStats::default();
         if delete_groups {
             log::info!("removing whole namespace recursively below {store}:/{ns}",);
-            for ns in self.recursive_iter_backup_ns(ns.to_owned())? {
+            for ns in self.recursive_iter_backup_ns(ns.to_owned(), NamespaceListFilter::Active)? {
                 let (removed_ns_groups, delete_stats) = self.remove_namespace_groups(&ns?)?;
                 stats.add(&delete_stats);
                 removed_all_requested = removed_all_requested && removed_ns_groups;
@@ -629,7 +631,7 @@ impl DataStore {
         // now try to delete the actual namespaces, bottom up so that we can use safe rmdir that
         // will choke if a new backup/group appeared in the meantime (but not on an new empty NS)
         let mut children = self
-            .recursive_iter_backup_ns(ns.to_owned())?
+            .recursive_iter_backup_ns(ns.to_owned(), NamespaceListFilter::All)?
             .collect::<Result<Vec<BackupNamespace>, Error>>()?;
 
         children.sort_by_key(|b| std::cmp::Reverse(b.depth()));
@@ -851,8 +853,9 @@ impl DataStore {
     pub fn iter_backup_ns(
         self: &Arc<DataStore>,
         ns: BackupNamespace,
+        filter: NamespaceListFilter,
     ) -> Result<ListNamespaces, Error> {
-        ListNamespaces::new(Arc::clone(self), ns)
+        ListNamespaces::new(Arc::clone(self), ns, filter)
     }
 
     /// Get a streaming iter over single-level backup namespaces of a datatstore, filtered by Ok
@@ -862,10 +865,11 @@ impl DataStore {
     pub fn iter_backup_ns_ok(
         self: &Arc<DataStore>,
         ns: BackupNamespace,
+        filter: NamespaceListFilter,
     ) -> Result<impl Iterator<Item = BackupNamespace> + 'static, Error> {
         let this = Arc::clone(self);
         Ok(
-            ListNamespaces::new(Arc::clone(self), ns)?.filter_map(move |ns| match ns {
+            ListNamespaces::new(Arc::clone(self), ns, filter)?.filter_map(move |ns| match ns {
                 Ok(ns) => Some(ns),
                 Err(err) => {
                     log::error!("list groups error on datastore {} - {}", this.name(), err);
@@ -882,8 +886,9 @@ impl DataStore {
     pub fn recursive_iter_backup_ns(
         self: &Arc<DataStore>,
         ns: BackupNamespace,
+        filter: NamespaceListFilter,
     ) -> Result<ListNamespacesRecursive, Error> {
-        ListNamespacesRecursive::new(Arc::clone(self), ns)
+        ListNamespacesRecursive::new(Arc::clone(self), ns, filter)
     }
 
     /// Get a streaming iter over single-level backup namespaces of a datatstore, filtered by Ok
@@ -894,12 +899,13 @@ impl DataStore {
         self: &Arc<DataStore>,
         ns: BackupNamespace,
         max_depth: Option<usize>,
+        filter: NamespaceListFilter,
     ) -> Result<impl Iterator<Item = BackupNamespace> + 'static, Error> {
         let this = Arc::clone(self);
         Ok(if let Some(depth) = max_depth {
-            ListNamespacesRecursive::new_max_depth(Arc::clone(self), ns, depth)?
+            ListNamespacesRecursive::new_max_depth(Arc::clone(self), ns, depth, filter)?
         } else {
-            ListNamespacesRecursive::new(Arc::clone(self), ns)?
+            ListNamespacesRecursive::new(Arc::clone(self), ns, filter)?
         }
         .filter_map(move |ns| match ns {
             Ok(ns) => Some(ns),
@@ -1136,7 +1142,7 @@ impl DataStore {
 
         let arc_self = Arc::new(self.clone());
         for namespace in arc_self
-            .recursive_iter_backup_ns(BackupNamespace::root())
+            .recursive_iter_backup_ns(BackupNamespace::root(), NamespaceListFilter::All)
             .context("creating namespace iterator failed")?
         {
             let namespace = namespace.context("iterating namespaces failed")?;
diff --git a/pbs-datastore/src/hierarchy.rs b/pbs-datastore/src/hierarchy.rs
index e0bf84419..f6385ba6a 100644
--- a/pbs-datastore/src/hierarchy.rs
+++ b/pbs-datastore/src/hierarchy.rs
@@ -8,7 +8,7 @@ use anyhow::{bail, format_err, Error};
 use pbs_api_types::{BackupNamespace, BackupType, BACKUP_DATE_REGEX, BACKUP_ID_REGEX};
 use proxmox_sys::fs::get_file_type;
 
-use crate::backup_info::{BackupDir, BackupGroup};
+use crate::backup_info::{BackupDir, BackupGroup, TRASH_MARKER_FILENAME};
 use crate::DataStore;
 
 /// A iterator for all BackupDir's (Snapshots) in a BackupGroup
@@ -268,31 +268,49 @@ where
     }
 }
 
+#[derive(Clone, Copy, Debug)]
+pub enum NamespaceListFilter {
+    Active,
+    All,
+    Trashed,
+}
+
 /// A iterator for a (single) level of Namespaces
 pub struct ListNamespaces {
     ns: BackupNamespace,
     base_path: PathBuf,
     ns_state: Option<proxmox_sys::fs::ReadDir>,
+    filter: NamespaceListFilter,
 }
 
 impl ListNamespaces {
     /// construct a new single-level namespace iterator on a datastore with an optional anchor ns
-    pub fn new(store: Arc<DataStore>, ns: BackupNamespace) -> Result<Self, Error> {
+    pub fn new(
+        store: Arc<DataStore>,
+        ns: BackupNamespace,
+        filter: NamespaceListFilter,
+    ) -> Result<Self, Error> {
         Ok(ListNamespaces {
             ns,
             base_path: store.base_path(),
             ns_state: None,
+            filter,
         })
     }
 
     /// to allow constructing the iter directly on a path, e.g., provided by section config
     ///
     /// NOTE: it's recommended to use the datastore one constructor or go over the recursive iter
-    pub fn new_from_path(path: PathBuf, ns: Option<BackupNamespace>) -> Result<Self, Error> {
+    pub fn new_from_path(
+        path: PathBuf,
+        ns: Option<BackupNamespace>,
+        filter: NamespaceListFilter,
+    ) -> Result<Self, Error> {
         Ok(ListNamespaces {
             ns: ns.unwrap_or_default(),
             base_path: path,
             ns_state: None,
+            filter,
         })
     }
 }
@@ -328,6 +346,50 @@ impl Iterator for ListNamespaces {
                 };
                 if let Ok(name) = entry.file_name().to_str() {
                     if name != "." && name != ".." {
+                        use nix::fcntl::OFlag;
+                        use nix::sys::stat::Mode;
+
+                        match self.filter {
+                            NamespaceListFilter::All => (),
+                            NamespaceListFilter::Active => {
+                                let mut trash_path = self.base_path.to_owned();
+                                if !self.ns.is_root() {
+                                    trash_path.push(self.ns.path());
+                                }
+                                trash_path.push("ns");
+                                trash_path.push(name);
+                                trash_path.push(TRASH_MARKER_FILENAME);
+                                match proxmox_sys::fd::openat(
+                                    &libc::AT_FDCWD,
+                                    &trash_path,
+                                    OFlag::O_PATH | OFlag::O_CLOEXEC | OFlag::O_NOFOLLOW,
+                                    Mode::empty(),
+                                ) {
+                                    Ok(_) => continue,
+                                    Err(nix::errno::Errno::ENOENT) => (),
+                                    Err(err) => return Some(Err(err.into())),
+                                }
+                            }
+                            NamespaceListFilter::Trashed => {
+                                let mut trash_path = self.base_path.to_owned();
+                                if !self.ns.is_root() {
+                                    trash_path.push(self.ns.path());
+                                }
+                                trash_path.push("ns");
+                                trash_path.push(name);
+                                trash_path.push(TRASH_MARKER_FILENAME);
+                                match proxmox_sys::fd::openat(
+                                    &libc::AT_FDCWD,
+                                    &trash_path,
+                                    OFlag::O_PATH | OFlag::O_CLOEXEC | OFlag::O_NOFOLLOW,
+                                    Mode::empty(),
+                                ) {
+                                    Ok(_) => (),
+                                    Err(nix::errno::Errno::ENOENT) => continue,
+                                    Err(err) => return Some(Err(err.into())),
+                                }
+                            }
+                        }
                         return Some(BackupNamespace::from_parent_ns(&self.ns, name.to_string()));
                     }
                 }
@@ -368,12 +430,17 @@ pub struct ListNamespacesRecursive {
     /// the maximal recursion depth from the anchor start ns (depth == 0) downwards
     max_depth: u8,
     state: Option<Vec<ListNamespaces>>, // vector to avoid code recursion
+    filter: NamespaceListFilter,
 }
 
 impl ListNamespacesRecursive {
     /// Creates an recursive namespace iterator.
-    pub fn new(store: Arc<DataStore>, ns: BackupNamespace) -> Result<Self, Error> {
-        Self::new_max_depth(store, ns, pbs_api_types::MAX_NAMESPACE_DEPTH)
+    pub fn new(
+        store: Arc<DataStore>,
+        ns: BackupNamespace,
+        filter: NamespaceListFilter,
+    ) -> Result<Self, Error> {
+        Self::new_max_depth(store, ns, pbs_api_types::MAX_NAMESPACE_DEPTH, filter)
     }
 
     /// Creates an recursive namespace iterator that iterates recursively until depth is reached.
@@ -386,6 +453,7 @@ impl ListNamespacesRecursive {
         store: Arc<DataStore>,
         ns: BackupNamespace,
         max_depth: usize,
+        filter: NamespaceListFilter,
     ) -> Result<Self, Error> {
         if max_depth > pbs_api_types::MAX_NAMESPACE_DEPTH {
             let limit = pbs_api_types::MAX_NAMESPACE_DEPTH + 1;
@@ -399,6 +467,7 @@ impl ListNamespacesRecursive {
             ns,
             max_depth: max_depth as u8,
             state: None,
+            filter,
         })
     }
 }
@@ -418,7 +487,11 @@ impl Iterator for ListNamespacesRecursive {
                 match iter.next() {
                     Some(Ok(ns)) => {
                         if state.len() < self.max_depth as usize {
-                            match ListNamespaces::new(Arc::clone(&self.store), ns.to_owned()) {
+                            match ListNamespaces::new(
+                                Arc::clone(&self.store),
+                                ns.to_owned(),
+                                self.filter,
+                            ) {
                                 Ok(iter) => state.push(iter),
                                 Err(err) => log::error!("failed to create child ns iter {err}"),
                             }
@@ -434,7 +507,11 @@ impl Iterator for ListNamespacesRecursive {
                 // first next call ever: initialize state vector and start iterating at our level
                 let mut state = Vec::with_capacity(pbs_api_types::MAX_NAMESPACE_DEPTH);
                 if self.max_depth as usize > 0 {
-                    match ListNamespaces::new(Arc::clone(&self.store), self.ns.to_owned()) {
+                    match ListNamespaces::new(
+                        Arc::clone(&self.store),
+                        self.ns.to_owned(),
+                        self.filter,
+                    ) {
                         Ok(list_ns) => state.push(list_ns),
                         Err(err) => {
                             // yield the error but set the state to Some to avoid re-try, a future
diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index 5014b6c09..7390c6df8 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -208,6 +208,7 @@ pub use datastore::{
 mod hierarchy;
 pub use hierarchy::{
     ListGroups, ListGroupsType, ListNamespaces, ListNamespacesRecursive, ListSnapshots,
+    NamespaceListFilter,
 };
 
 mod snapshot_reader;
diff --git a/src/api2/admin/namespace.rs b/src/api2/admin/namespace.rs
index 6cf88d89e..e5463524a 100644
--- a/src/api2/admin/namespace.rs
+++ b/src/api2/admin/namespace.rs
@@ -9,7 +9,7 @@ use pbs_api_types::{
     DATASTORE_SCHEMA, NS_MAX_DEPTH_SCHEMA, PROXMOX_SAFE_ID_FORMAT,
 };
 
-use pbs_datastore::DataStore;
+use pbs_datastore::{DataStore, NamespaceListFilter};
 
 use crate::backup::{check_ns_modification_privs, check_ns_privs, NS_PRIVS_OK};
 
@@ -99,12 +99,14 @@ pub fn list_namespaces(
 
     let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
 
-    let iter = match datastore.recursive_iter_backup_ns_ok(parent, max_depth) {
-        Ok(iter) => iter,
-        // parent NS doesn't exists and user has no privs on it, avoid info leakage.
-        Err(_) if parent_access.is_err() => http_bail!(FORBIDDEN, "permission check failed"),
-        Err(err) => return Err(err),
-    };
+    let iter =
+        match datastore.recursive_iter_backup_ns_ok(parent, max_depth, NamespaceListFilter::Active)
+        {
+            Ok(iter) => iter,
+            // parent NS doesn't exists and user has no privs on it, avoid info leakage.
+            Err(_) if parent_access.is_err() => http_bail!(FORBIDDEN, "permission check failed"),
+            Err(err) => return Err(err),
+        };
 
     let ns_to_item =
         |ns: BackupNamespace| -> NamespaceListItem { NamespaceListItem { ns, comment: None } };
diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
index 17c8bc605..53554c54b 100644
--- a/src/api2/tape/backup.rs
+++ b/src/api2/tape/backup.rs
@@ -18,7 +18,7 @@ use pbs_api_types::{
 
 use pbs_config::CachedUserInfo;
 use pbs_datastore::backup_info::{BackupDir, BackupInfo, ListBackupFilter};
-use pbs_datastore::{DataStore, StoreProgress};
+use pbs_datastore::{DataStore, NamespaceListFilter, StoreProgress};
 
 use crate::tape::TapeNotificationMode;
 use crate::{
@@ -392,7 +392,11 @@ fn backup_worker(
     }
 
     let mut group_list = Vec::new();
-    let namespaces = datastore.recursive_iter_backup_ns_ok(root_namespace, setup.max_depth)?;
+    let namespaces = datastore.recursive_iter_backup_ns_ok(
+        root_namespace,
+        setup.max_depth,
+        NamespaceListFilter::Active,
+    )?;
     for ns in namespaces {
         group_list.extend(datastore.list_backup_groups(ns)?);
     }
diff --git a/src/backup/hierarchy.rs b/src/backup/hierarchy.rs
index 8dd71fcf7..ec8d4b9c8 100644
--- a/src/backup/hierarchy.rs
+++ b/src/backup/hierarchy.rs
@@ -7,7 +7,9 @@ use pbs_api_types::{
     PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_READ,
 };
 use pbs_config::CachedUserInfo;
-use pbs_datastore::{backup_info::BackupGroup, DataStore, ListGroups, ListNamespacesRecursive};
+use pbs_datastore::{
+    backup_info::BackupGroup, DataStore, ListGroups, ListNamespacesRecursive, NamespaceListFilter,
+};
 
 /// Asserts that `privs` are fulfilled on datastore + (optional) namespace.
 pub fn check_ns_privs(
@@ -75,12 +77,15 @@ pub fn can_access_any_namespace(
 ) -> bool {
     // NOTE: traversing the datastore could be avoided if we had an "ACL tree: is there any priv
     // below /datastore/{store}" helper
-    let mut iter =
-        if let Ok(iter) = store.recursive_iter_backup_ns_ok(BackupNamespace::root(), None) {
-            iter
-        } else {
-            return false;
-        };
+    let mut iter = if let Ok(iter) = store.recursive_iter_backup_ns_ok(
+        BackupNamespace::root(),
+        None,
+        NamespaceListFilter::Active,
+    ) {
+        iter
+    } else {
+        return false;
+    };
     let wanted =
         PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_MODIFY | PRIV_DATASTORE_READ | PRIV_DATASTORE_BACKUP;
     let name = store.name();
@@ -129,7 +134,12 @@ impl<'a> ListAccessibleBackupGroups<'a> {
         owner_and_priv: Option<u64>,
         auth_id: Option<&'a Authid>,
     ) -> Result<Self, Error> {
-        let ns_iter = ListNamespacesRecursive::new_max_depth(Arc::clone(store), ns, max_depth)?;
+        let ns_iter = ListNamespacesRecursive::new_max_depth(
+            Arc::clone(store),
+            ns,
+            max_depth,
+            NamespaceListFilter::Active,
+        )?;
         Ok(ListAccessibleBackupGroups {
             auth_id,
             ns_iter,
diff --git a/src/server/pull.rs b/src/server/pull.rs
index 50d7b0806..d3c6fcf6a 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -25,7 +25,7 @@ use pbs_datastore::fixed_index::FixedIndexReader;
 use pbs_datastore::index::IndexFile;
 use pbs_datastore::manifest::{BackupManifest, FileInfo};
 use pbs_datastore::read_chunk::AsyncReadChunk;
-use pbs_datastore::{check_backup_owner, DataStore, StoreProgress};
+use pbs_datastore::{check_backup_owner, DataStore, NamespaceListFilter, StoreProgress};
 use pbs_tools::sha::sha256;
 
 use super::sync::{
@@ -750,7 +750,11 @@ fn check_and_remove_vanished_ns(
     let mut local_ns_list: Vec<BackupNamespace> = params
         .target
         .store
-        .recursive_iter_backup_ns_ok(params.target.ns.clone(), Some(max_depth))?
+        .recursive_iter_backup_ns_ok(
+            params.target.ns.clone(),
+            Some(max_depth),
+            NamespaceListFilter::Active,
+        )?
         .filter(|ns| {
             let user_privs =
                 user_info.lookup_privs(&params.owner, &ns.acl_path(params.target.store.name()));
diff --git a/src/server/sync.rs b/src/server/sync.rs
index ce338fbbe..308a03977 100644
--- a/src/server/sync.rs
+++ b/src/server/sync.rs
@@ -26,7 +26,9 @@ use pbs_api_types::{
 use pbs_client::{BackupReader, BackupRepository, HttpClient, RemoteChunkReader};
 use pbs_datastore::data_blob::DataBlob;
 use pbs_datastore::read_chunk::AsyncReadChunk;
-use pbs_datastore::{BackupManifest, DataStore, ListNamespacesRecursive, LocalChunkReader};
+use pbs_datastore::{
+    BackupManifest, DataStore, ListNamespacesRecursive, LocalChunkReader, NamespaceListFilter,
+};
 
 use crate::backup::ListAccessibleBackupGroups;
 use crate::server::jobstate::Job;
@@ -425,6 +427,7 @@ impl SyncSource for LocalSource {
             self.store.clone(),
             self.ns.clone(),
             max_depth.unwrap_or(MAX_NAMESPACE_DEPTH),
+            NamespaceListFilter::Active,
         )?
         .collect();
 
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:38 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:38 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 04/21] datastore: ignore trashed
 snapshots for last successful backup
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-5-c.ebner@proxmox.com>

Exclude trahed snapshots from looking up the last successful backup
of the group, as trashed items are marked for deletion by the next
garbage collection run and must be considered as if not present
anymore.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index a8c864ac8..189ed28ad 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -172,8 +172,13 @@ impl BackupGroup {
                     return Ok(());
                 }
 
-                let mut manifest_path = PathBuf::from(backup_time);
-                manifest_path.push(MANIFEST_BLOB_NAME.as_ref());
+                let path = PathBuf::from(backup_time);
+                let trash_marker_path = path.join(TRASH_MARKER_FILENAME);
+                if trash_marker_path.exists() {
+                    return Ok(());
+                }
+
+                let manifest_path = path.join(MANIFEST_BLOB_NAME.as_ref());
 
                 use nix::fcntl::{openat, OFlag};
                 match openat(
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:34 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:34 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 00/21] implement trash bin
 functionality
Message-ID: <20250508130555.494782-1-c.ebner@proxmox.com>

This patch series implements a trash bin functionality, marking
backup snapshots, groups and namespaces as trashed on prune and
forget instead of deleting them immediately. Cleanup is deferred to
the garbage collection job, allowing to recover the trashed items if
removed by accident.

In contrast to the previous version 1 of the patch series [0] which
moved trashed items to a separate folder structure, this patch series
utilizes the proposed `.trashed` marker file approach to mark and
filter content items. These marker files have to be set and removed
accordingly in a consistent manner and are used to filter in
iterators based on the trash state.

Sending this series as RFC in order to gain some initial feedback on
the chosen implementation approach and possible
shortcomings/oversights. Further, feedback especially for the design
of the WebUI (which still has some rough edges) is very welcome!

[0] https://lore.proxmox.com/pbs-devel/7db77767-ed54-4d7f-8caa-24b7216b159c at proxmox.com/T/

Christian Ebner (21):
  datastore/api: mark snapshots as trash on destroy
  datastore: mark groups as trash on destroy
  datastore: allow filtering of backups by their trash status
  datastore: ignore trashed snapshots for last successful backup
  sync: ignore trashed snapshots when reading from local source
  api: tape: check trash marker when trying to write snapshot
  sync: ignore trashed groups in local source reader
  datastore: namespace: add filter for trash status
  datastore: refactor recursive namespace removal
  datastore: mark namespace as trash instead of deleting it
  datastore: check for trash marker in namespace exists check
  datastore: clear trashed snapshot dir if re-creation requested
  datastore: recreate trashed backup groups if requested
  datastore: GC: clean-up trashed snapshots, groups and namespaces
  client: expose skip trash flags for cli commands
  api: datastore: add flag to list trashed snapshots only
  api: namespace: add option to list all namespaces, including trashed
  api: admin: implement endpoints to restore trashed contents
  ui: add recover for trashed items tab to datastore panel
  ui: drop 'permanent' in group/snapshot forget, default is to trash
  ui: allow to skip trash on namespace deletion

 pbs-datastore/examples/ls-snapshots.rs  |   8 +-
 pbs-datastore/src/backup_info.rs        | 130 +++-
 pbs-datastore/src/datastore.rs          | 320 ++++++++-
 pbs-datastore/src/hierarchy.rs          |  91 ++-
 pbs-datastore/src/lib.rs                |   1 +
 proxmox-backup-client/src/group.rs      |  14 +-
 proxmox-backup-client/src/namespace.rs  |  14 +-
 proxmox-backup-client/src/snapshot.rs   |  16 +-
 src/api2/admin/datastore.rs             | 225 +++++-
 src/api2/admin/namespace.rs             |  28 +-
 src/api2/backup/environment.rs          |   1 +
 src/api2/tape/backup.rs                 |  20 +-
 src/backup/hierarchy.rs                 |  26 +-
 src/backup/verify.rs                    |   4 +-
 src/bin/proxmox-backup-manager.rs       |  12 +-
 src/bin/proxmox_backup_manager/prune.rs |   2 +-
 src/server/prune_job.rs                 |   7 +-
 src/server/pull.rs                      |  25 +-
 src/server/sync.rs                      |   7 +-
 www/Makefile                            |   1 +
 www/datastore/Content.js                |   4 +-
 www/datastore/Panel.js                  |   8 +
 www/datastore/RecoverTrashed.js         | 876 ++++++++++++++++++++++++
 www/window/NamespaceEdit.js             |  21 +-
 24 files changed, 1737 insertions(+), 124 deletions(-)
 create mode 100644 www/datastore/RecoverTrashed.js

-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:43 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:43 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 09/21] datastore: refactor
 recursive namespace removal
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-10-c.ebner@proxmox.com>

Split off the recursive destruction logic for the namespace folder
hierarchy into its own helper function.
This will allow to separate the marking of namespaces as trashed
from the actually destruction an namespace cleanup by garbage
collection or when bypassing the trash.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index aee69768f..023a6a12e 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -628,6 +628,18 @@ impl DataStore {
             log::info!("pruning empty namespace recursively below {store}:/{ns}");
         }
 
+        removed_all_requested =
+            removed_all_requested && self.destroy_namespace_recursive(ns, delete_groups)?;
+
+        Ok((removed_all_requested, stats))
+    }
+
+    fn destroy_namespace_recursive(
+        self: &Arc<Self>,
+        ns: &BackupNamespace,
+        delete_groups: bool,
+    ) -> Result<bool, Error> {
+        let mut removed_all_requested = true;
         // now try to delete the actual namespaces, bottom up so that we can use safe rmdir that
         // will choke if a new backup/group appeared in the meantime (but not on an new empty NS)
         let mut children = self
@@ -662,7 +674,7 @@ impl DataStore {
             }
         }
 
-        Ok((removed_all_requested, stats))
+        Ok(removed_all_requested)
     }
 
     /// Remove a complete backup group including all snapshots.
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:47 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:47 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 13/21] datastore: recreate
 trashed backup groups if requested
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-14-c.ebner@proxmox.com>

A whole backup group might have been marked as trashed, including all
of the contained snapshots.

Since a new backup to that group (even as different user/owner)
should still work, permanently clear the whole trashed group before
recreation. This will limit the trash lifetime as now the group is
not recoverable until next garbage collection.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 4f7766c36..ca05e1bea 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -934,6 +934,32 @@ impl DataStore {
                 let guard = backup_group.lock().with_context(|| {
                     format!("while creating locked backup group '{backup_group:?}'")
                 })?;
+                if backup_group.is_trashed() {
+                    info!("clear trashed backup group {full_path:?}");
+                    let dir_entries = std::fs::read_dir(&full_path).context(
+                        "failed to read directory contents during cleanup of trashed group",
+                    )?;
+                    for entry in dir_entries {
+                        let entry = entry.context(
+                            "failed to read directory entry during clenup of trashed group",
+                        )?;
+                        let file_type = entry.file_type().context(
+                            "failed to get entry file type during clenup of trashed group",
+                        )?;
+                        if file_type.is_dir() {
+                            std::fs::remove_dir_all(entry.path())
+                                .context("failed to remove directory entry during clenup of trashed snapshot")?;
+                        } else {
+                            std::fs::remove_file(entry.path())
+                                .context("failed to remove directory entry during clenup of trashed snapshot")?;
+                        }
+                    }
+                    self.set_owner(ns, backup_group.group(), auth_id, false)?;
+                    let owner = self.get_owner(ns, backup_group.group())?; // just to be sure
+                    self.untrash_namespace(ns)?;
+                    return Ok((owner, guard));
+                }
+
                 let owner = self.get_owner(ns, backup_group.group())?; // just to be sure
                 Ok((owner, guard))
             }
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:48 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:48 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 14/21] datastore: GC: clean-up
 trashed snapshots, groups and namespaces
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-15-c.ebner@proxmox.com>

Cleanup trashed items during phase 1 of garbage collection. If
encountered, index files located within trashed snapshots are touched
as well, deferring chunk cleanup to the next run

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 84 +++++++++++++++++++++++++++++++++-
 1 file changed, 83 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index ca05e1bea..d88af4c68 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -574,6 +574,18 @@ impl DataStore {
         !path.exists()
     }
 
+    /// Checks if the namespace trash marker file exists,
+    /// does not imply that the namespace itself exists.
+    pub fn namespace_is_trashed(&self, namespace: &BackupNamespace) -> bool {
+        if namespace.is_root() {
+            return false;
+        }
+        let mut path = self.base_path();
+        path.push(namespace.path());
+        path.push(TRASH_MARKER_FILENAME);
+        path.exists()
+    }
+
     /// Remove the namespace and all it's parent components from the trash by removing the trash or
     /// trash-pending marker file for each namespace level from deepest to shallowest. Missing files
     /// are ignored.
@@ -1322,7 +1334,7 @@ impl DataStore {
             .context("creating namespace iterator failed")?
         {
             let namespace = namespace.context("iterating namespaces failed")?;
-            for group in arc_self.iter_backup_groups(namespace)? {
+            for group in arc_self.iter_backup_groups(namespace.clone())? {
                 let group = group.context("iterating backup groups failed")?;
 
                 // Avoid race between listing/marking of snapshots by GC and pruning the last
@@ -1403,10 +1415,80 @@ impl DataStore {
                             }
                             processed_index_files += 1;
                         }
+
+                        // Only try to lock a trashed snapshots and continue if that is not
+                        // possible, as then most likely this is in the process of being untrashed.
+                        // Check trash state before and after locking to avoid otherwise possible
+                        // races.
+                        if snapshot.backup_dir.is_trashed() {
+                            if let Ok(_lock) = snapshot.backup_dir.lock() {
+                                if snapshot.backup_dir.is_trashed() {
+                                    let path = snapshot.backup_dir.full_path();
+                                    log::info!("removing trashed backup snapshot {path:?}");
+                                    std::fs::remove_dir_all(&path).with_context(|| {
+                                        format!("removing trashed backup snapshot {path:?} failed")
+                                    })?;
+                                }
+                            } else {
+                                let path = snapshot.backup_dir.full_path();
+                                warn!("failed to lock trashed backup snapshot can {path:?}");
+                            }
+                        }
                     }
 
                     break;
                 }
+                if group.is_trashed() {
+                    if let Ok(_lock) = group.lock() {
+                        if group.is_trashed() {
+                            let trash_path = group.full_group_path().join(".trashed");
+                            std::fs::remove_file(&trash_path).map_err(|err| {
+                                format_err!(
+                                    "removing the trash file '{trash_path:?}' failed - {err}"
+                                )
+                            })?;
+
+                            let owner_path = group.full_group_path().join("owner");
+                            std::fs::remove_file(&owner_path).map_err(|err| {
+                                format_err!(
+                                    "removing the owner file '{owner_path:?}' failed - {err}"
+                                )
+                            })?;
+
+                            let path = group.full_group_path();
+
+                            std::fs::remove_dir(&path).map_err(|err| {
+                                format_err!("removing group directory {path:?} failed - {err}")
+                            })?;
+
+                            // Remove any now empty backup type directory
+                            let base_file = std::fs::File::open(self.base_path())?;
+                            let base_fd = base_file.as_raw_fd();
+                            for ty in BackupType::iter() {
+                                let mut ty_dir = namespace.path();
+                                ty_dir.push(ty.to_string());
+                                match unlinkat(Some(base_fd), &ty_dir, UnlinkatFlags::RemoveDir) {
+                                    Ok(_) => (),
+                                    Err(nix::errno::Errno::ENOENT) |
+                                    Err(nix::errno::Errno::ENOTEMPTY) => (),
+                                    Err(err) => info!("failed to remove backup type directory for {namespace} - {err}"),
+                                }
+                            }
+                        } else {
+                            let path = group.full_group_path();
+                            warn!("failed to lock trashed backup group {path:?}");
+                        }
+                    }
+                }
+            }
+            if self.namespace_is_trashed(&namespace) {
+                // Remove the namespace, but only if it was empty (as the GC already cleared child
+                // items and no new ones have been created since).
+                match arc_self.destroy_namespace_recursive(&namespace, false) {
+                    Ok(true) => info!("removed trashed namespace {namespace}"),
+                    Ok(false) => info!("failed to remove trashed namespace {namespace}, not empty"),
+                    Err(err) => warn!("removing trashed namespace failed: {err:#}"),
+                }
             }
         }
 
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:51 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:51 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 17/21] api: namespace: add
 option to list all namespaces, including trashed
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-18-c.ebner@proxmox.com>

Add an optional parameter so all namespaces can be listed, including
ones which are marked as trash. This allows to display all namespace
in the UI, independent of their current state.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/namespace.rs             | 28 +++++++++++++++++--------
 src/bin/proxmox-backup-manager.rs       | 12 ++++++++---
 src/bin/proxmox_backup_manager/prune.rs |  2 +-
 3 files changed, 29 insertions(+), 13 deletions(-)

diff --git a/src/api2/admin/namespace.rs b/src/api2/admin/namespace.rs
index 6f70497a6..098e69ee8 100644
--- a/src/api2/admin/namespace.rs
+++ b/src/api2/admin/namespace.rs
@@ -75,6 +75,12 @@ pub fn create_namespace(
                 schema: NS_MAX_DEPTH_SCHEMA,
                 optional: true,
             },
+            "include-trashed": {
+                type: bool,
+                optional: true,
+                default: false,
+                description: "List also namespaces marked as trash.",
+            }
         },
     },
     returns: pbs_api_types::ADMIN_DATASTORE_LIST_NAMESPACE_RETURN_TYPE,
@@ -89,6 +95,7 @@ pub fn list_namespaces(
     store: String,
     parent: Option<BackupNamespace>,
     max_depth: Option<usize>,
+    include_trashed: bool,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Vec<NamespaceListItem>, Error> {
     let parent = parent.unwrap_or_default();
@@ -99,14 +106,17 @@ pub fn list_namespaces(
 
     let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
 
-    let iter =
-        match datastore.recursive_iter_backup_ns_ok(parent, max_depth, NamespaceListFilter::Active)
-        {
-            Ok(iter) => iter,
-            // parent NS doesn't exists and user has no privs on it, avoid info leakage.
-            Err(_) if parent_access.is_err() => http_bail!(FORBIDDEN, "permission check failed"),
-            Err(err) => return Err(err),
-        };
+    let filter = if include_trashed {
+        NamespaceListFilter::All
+    } else {
+        NamespaceListFilter::Active
+    };
+    let iter = match datastore.recursive_iter_backup_ns_ok(parent, max_depth, filter) {
+        Ok(iter) => iter,
+        // parent NS doesn't exists and user has no privs on it, avoid info leakage.
+        Err(_) if parent_access.is_err() => http_bail!(FORBIDDEN, "permission check failed"),
+        Err(err) => return Err(err),
+    };
 
     let ns_to_item =
         |ns: BackupNamespace| -> NamespaceListItem { NamespaceListItem { ns, comment: None } };
@@ -148,7 +158,7 @@ pub fn list_namespaces(
             "skip-trash": {
                 type: bool,
                 optional: true,
-                default: true,
+                default: false,
                 description: "Remove and namespace immediately, skip moving to trash",
             },
         },
diff --git a/src/bin/proxmox-backup-manager.rs b/src/bin/proxmox-backup-manager.rs
index d4363e717..0c4d70b41 100644
--- a/src/bin/proxmox-backup-manager.rs
+++ b/src/bin/proxmox-backup-manager.rs
@@ -857,8 +857,14 @@ pub fn complete_remote_datastore_namespace(
         Some((None, source_store)) => {
             let mut rpcenv = CliEnvironment::new();
             rpcenv.set_auth_id(Some(String::from("root at pam")));
-            crate::api2::admin::namespace::list_namespaces(source_store, None, None, &mut rpcenv)
-                .ok()
+            crate::api2::admin::namespace::list_namespaces(
+                source_store,
+                None,
+                None,
+                false,
+                &mut rpcenv,
+            )
+            .ok()
         }
         _ => None,
     } {
@@ -893,7 +899,7 @@ pub fn complete_sync_local_datastore_namespace(
 
     if let Some(store) = store {
         if let Ok(data) =
-            crate::api2::admin::namespace::list_namespaces(store, None, None, &mut rpcenv)
+            crate::api2::admin::namespace::list_namespaces(store, None, None, false, &mut rpcenv)
         {
             for item in data {
                 list.push(item.ns.name());
diff --git a/src/bin/proxmox_backup_manager/prune.rs b/src/bin/proxmox_backup_manager/prune.rs
index 923eb6f51..b57c39bd7 100644
--- a/src/bin/proxmox_backup_manager/prune.rs
+++ b/src/bin/proxmox_backup_manager/prune.rs
@@ -164,7 +164,7 @@ fn complete_prune_local_datastore_namespace(
 
     if let Some(store) = store {
         if let Ok(data) =
-            crate::api2::admin::namespace::list_namespaces(store, None, None, &mut rpcenv)
+            crate::api2::admin::namespace::list_namespaces(store, None, None, false, &mut rpcenv)
         {
             for item in data {
                 list.push(item.ns.name());
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:52 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:52 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 18/21] api: admin: implement
 endpoints to restore trashed contents
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-19-c.ebner@proxmox.com>

Implements the api endpoints to restore trashed contents contained
within namespaces, backup groups or individual snapshots.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs | 173 +++++++++++++++++++++++++++++++++++-
 1 file changed, 172 insertions(+), 1 deletion(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index cbd24c729..eb033c3fc 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -51,7 +51,7 @@ use pbs_api_types::{
 };
 use pbs_client::pxar::{create_tar, create_zip};
 use pbs_config::CachedUserInfo;
-use pbs_datastore::backup_info::{BackupInfo, ListBackupFilter};
+use pbs_datastore::backup_info::{BackupInfo, ListBackupFilter, TRASH_MARKER_FILENAME};
 use pbs_datastore::cached_chunk_reader::CachedChunkReader;
 use pbs_datastore::catalog::{ArchiveEntry, CatalogReader};
 use pbs_datastore::data_blob::DataBlob;
@@ -2727,6 +2727,165 @@ pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<V
     Ok(json!(upid))
 }
 
+#[api(
+    input: {
+        properties: {
+            store: { schema: DATASTORE_SCHEMA },
+            ns: { type: BackupNamespace, },
+        },
+    },
+    access: {
+        permission: &Permission::Anybody,
+        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
+            or DATASTORE_BACKUP and being the owner of the group",
+    },
+)]
+/// Recover trashed contents of a namespace.
+pub fn recover_namespace(
+    store: String,
+    ns: BackupNamespace,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let limited = check_ns_privs_full(
+        &store,
+        &ns,
+        &auth_id,
+        PRIV_DATASTORE_MODIFY,
+        PRIV_DATASTORE_BACKUP,
+    )?;
+
+    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
+
+    for backup_group in datastore.iter_backup_groups(ns.clone())? {
+        let backup_group = backup_group?;
+        if limited {
+            let owner = datastore.get_owner(&ns, backup_group.group())?;
+            if check_backup_owner(&owner, &auth_id).is_err() {
+                continue;
+            }
+        }
+        do_recover_group(&backup_group)?;
+    }
+
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            store: { schema: DATASTORE_SCHEMA },
+            group: {
+                type: pbs_api_types::BackupGroup,
+                flatten: true,
+            },
+            ns: {
+                type: BackupNamespace,
+                optional: true,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Anybody,
+        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
+            or DATASTORE_BACKUP and being the owner of the group",
+    },
+)]
+/// Recover trashed contents of a backup group.
+pub fn recover_group(
+    store: String,
+    group: pbs_api_types::BackupGroup,
+    ns: Option<BackupNamespace>,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let ns = ns.unwrap_or_default();
+    let datastore = check_privs_and_load_store(
+        &store,
+        &ns,
+        &auth_id,
+        PRIV_DATASTORE_MODIFY,
+        PRIV_DATASTORE_BACKUP,
+        Some(Operation::Write),
+        &group,
+    )?;
+
+    let backup_group = datastore.backup_group(ns, group);
+    do_recover_group(&backup_group)?;
+
+    Ok(())
+}
+
+fn do_recover_group(backup_group: &BackupGroup) -> Result<(), Error> {
+    let trashed_snapshots = backup_group.list_backups(ListBackupFilter::Trashed)?;
+    for snapshot in trashed_snapshots {
+        do_recover_snapshot(&snapshot.backup_dir)?;
+    }
+
+    let group_trash_path = backup_group.full_group_path().join(TRASH_MARKER_FILENAME);
+    if let Err(err) = std::fs::remove_file(&group_trash_path) {
+        if err.kind() != std::io::ErrorKind::NotFound {
+            bail!("failed to remove group trash file {group_trash_path:?} - {err}");
+        }
+    }
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            store: { schema: DATASTORE_SCHEMA },
+            backup_dir: {
+                type: pbs_api_types::BackupDir,
+                flatten: true,
+            },
+            ns: {
+                type: BackupNamespace,
+                optional: true,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Anybody,
+        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
+            or DATASTORE_BACKUP and being the owner of the group",
+    },
+)]
+/// Recover trashed contents of a backup snapshot.
+pub fn recover_snapshot(
+    store: String,
+    backup_dir: pbs_api_types::BackupDir,
+    ns: Option<BackupNamespace>,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let ns = ns.unwrap_or_default();
+    let datastore = check_privs_and_load_store(
+        &store,
+        &ns,
+        &auth_id,
+        PRIV_DATASTORE_MODIFY,
+        PRIV_DATASTORE_BACKUP,
+        Some(Operation::Write),
+        &backup_dir.group,
+    )?;
+
+    let snapshot = datastore.backup_dir(ns, backup_dir)?;
+    do_recover_snapshot(&snapshot)?;
+
+    Ok(())
+}
+
+fn do_recover_snapshot(snapshot_dir: &BackupDir) -> Result<(), Error> {
+    let trash_path = snapshot_dir.full_path().join(TRASH_MARKER_FILENAME);
+    if let Err(err) = std::fs::remove_file(&trash_path) {
+        if err.kind() != std::io::ErrorKind::NotFound {
+            bail!("failed to remove trash file {trash_path:?} - {err}");
+        }
+    }
+    Ok(())
+}
+
 #[sortable]
 const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
     (
@@ -2792,6 +2951,18 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
         "pxar-file-download",
         &Router::new().download(&API_METHOD_PXAR_FILE_DOWNLOAD),
     ),
+    (
+        "recover-group",
+        &Router::new().post(&API_METHOD_RECOVER_GROUP),
+    ),
+    (
+        "recover-namespace",
+        &Router::new().post(&API_METHOD_RECOVER_NAMESPACE),
+    ),
+    (
+        "recover-snapshot",
+        &Router::new().post(&API_METHOD_RECOVER_SNAPSHOT),
+    ),
     ("rrd", &Router::new().get(&API_METHOD_GET_RRD_STATS)),
     (
         "snapshots",
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:44 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:44 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 10/21] datastore: mark namespace
 as trash instead of deleting it
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-11-c.ebner@proxmox.com>

As for backup snapshots and groups, mark the namespace as trash
instead of removing it and the contents right away, if the trash
should not be bypassed.

Actual removal of the hirarchical folder structure has to be taken
care of by the garbage collection.

In order to avoid races during removal, first mark the namespaces as
trash pending, mark the snapshots and groups as trash and only after
rename the pending marker file to the trash marker file. By this,
concurrent backups can remove the trash pending marker to avoid the
namespace being trashed.

On re-creation of a trashed namespace remove the marker file on itself
and any parent component from deepest to shallowest. As trashing a full
namespace can also set the trash pending state for recursive namespace
cleanup, remove encounters of that marker file as well to avoid the
namespace or its parent being trashed.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 135 +++++++++++++++++++++++++++++----
 src/api2/admin/namespace.rs    |   2 +-
 src/server/pull.rs             |   2 +-
 3 files changed, 123 insertions(+), 16 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 023a6a12e..b1d81e199 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -28,7 +28,9 @@ use pbs_api_types::{
 };
 use pbs_config::BackupLockGuard;
 
-use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, ListBackupFilter, OLD_LOCKING};
+use crate::backup_info::{
+    BackupDir, BackupGroup, BackupInfo, ListBackupFilter, OLD_LOCKING, TRASH_MARKER_FILENAME,
+};
 use crate::chunk_store::ChunkStore;
 use crate::dynamic_index::{DynamicIndexReader, DynamicIndexWriter};
 use crate::fixed_index::{FixedIndexReader, FixedIndexWriter};
@@ -42,6 +44,8 @@ use crate::DataBlob;
 static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
     LazyLock::new(|| Mutex::new(HashMap::new()));
 
+const TRASH_PENDING_MARKER_FILENAME: &str = ".pending";
+
 /// checks if auth_id is owner, or, if owner is a token, if
 /// auth_id is the user of the token
 pub fn check_backup_owner(owner: &Authid, auth_id: &Authid) -> Result<(), Error> {
@@ -554,6 +558,7 @@ impl DataStore {
         ns_full_path.push(ns.path());
 
         std::fs::create_dir_all(ns_full_path)?;
+        self.untrash_namespace(&ns)?;
 
         Ok(ns)
     }
@@ -565,6 +570,34 @@ impl DataStore {
         path.exists()
     }
 
+    /// Remove the namespace and all it's parent components from the trash by removing the trash or
+    /// trash-pending marker file for each namespace level from deepest to shallowest. Missing files
+    /// are ignored.
+    pub fn untrash_namespace(&self, namespace: &BackupNamespace) -> Result<(), Error> {
+        let mut namespace = namespace.clone();
+        while namespace.depth() > 0 {
+            let mut trash_file_path = self.base_path();
+            trash_file_path.push(namespace.path());
+            let mut pending_file_path = trash_file_path.clone();
+            pending_file_path.push(TRASH_PENDING_MARKER_FILENAME);
+            if let Err(err) = std::fs::remove_file(&pending_file_path) {
+                // ignore not found, either not trashed or un-trashed by concurrent operation
+                if err.kind() != std::io::ErrorKind::NotFound {
+                    bail!("failed to remove trash-pending file {trash_file_path:?}: {err}");
+                }
+            }
+            trash_file_path.push(TRASH_MARKER_FILENAME);
+            if let Err(err) = std::fs::remove_file(&trash_file_path) {
+                // ignore not found, either not trashed or un-trashed by concurrent operation
+                if err.kind() != std::io::ErrorKind::NotFound {
+                    bail!("failed to remove trash file {trash_file_path:?}: {err}");
+                }
+            }
+            namespace.pop();
+        }
+        Ok(())
+    }
+
     /// Remove all backup groups of a single namespace level but not the namespace itself.
     ///
     /// Does *not* descends into child-namespaces and doesn't remoes the namespace itself either.
@@ -574,6 +607,7 @@ impl DataStore {
     pub fn remove_namespace_groups(
         self: &Arc<Self>,
         ns: &BackupNamespace,
+        skip_trash: bool,
     ) -> Result<(bool, BackupGroupDeleteStats), Error> {
         // FIXME: locking? The single groups/snapshots are already protected, so may not be
         // necessary (depends on what we all allow to do with namespaces)
@@ -583,20 +617,22 @@ impl DataStore {
         let mut stats = BackupGroupDeleteStats::default();
 
         for group in self.iter_backup_groups(ns.to_owned())? {
-            let delete_stats = group?.destroy(true)?;
+            let delete_stats = group?.destroy(skip_trash)?;
             stats.add(&delete_stats);
             removed_all_groups = removed_all_groups && delete_stats.all_removed();
         }
 
-        let base_file = std::fs::File::open(self.base_path())?;
-        let base_fd = base_file.as_raw_fd();
-        for ty in BackupType::iter() {
-            let mut ty_dir = ns.path();
-            ty_dir.push(ty.to_string());
-            // best effort only, but we probably should log the error
-            if let Err(err) = unlinkat(Some(base_fd), &ty_dir, UnlinkatFlags::RemoveDir) {
-                if err != nix::errno::Errno::ENOENT {
-                    log::error!("failed to remove backup type {ty} in {ns} - {err}");
+        if skip_trash {
+            let base_file = std::fs::File::open(self.base_path())?;
+            let base_fd = base_file.as_raw_fd();
+            for ty in BackupType::iter() {
+                let mut ty_dir = ns.path();
+                ty_dir.push(ty.to_string());
+                // best effort only, but we probably should log the error
+                if let Err(err) = unlinkat(Some(base_fd), &ty_dir, UnlinkatFlags::RemoveDir) {
+                    if err != nix::errno::Errno::ENOENT {
+                        log::error!("failed to remove backup type {ty} in {ns} - {err}");
+                    }
                 }
             }
         }
@@ -613,6 +649,7 @@ impl DataStore {
         self: &Arc<Self>,
         ns: &BackupNamespace,
         delete_groups: bool,
+        skip_trash: bool,
     ) -> Result<(bool, BackupGroupDeleteStats), Error> {
         let store = self.name();
         let mut removed_all_requested = true;
@@ -620,16 +657,68 @@ impl DataStore {
         if delete_groups {
             log::info!("removing whole namespace recursively below {store}:/{ns}",);
             for ns in self.recursive_iter_backup_ns(ns.to_owned(), NamespaceListFilter::Active)? {
-                let (removed_ns_groups, delete_stats) = self.remove_namespace_groups(&ns?)?;
+                let namespace = ns?;
+
+                if !skip_trash {
+                    let mut path = self.base_path();
+                    path.push(namespace.path());
+                    path.push(TRASH_PENDING_MARKER_FILENAME);
+                    if let Err(err) = std::fs::File::create(&path) {
+                        if err.kind() != std::io::ErrorKind::AlreadyExists {
+                            return Err(err).context("failed to set trash pending marker file");
+                        }
+                    }
+                }
+
+                let (removed_ns_groups, delete_stats) =
+                    self.remove_namespace_groups(&namespace, skip_trash)?;
                 stats.add(&delete_stats);
                 removed_all_requested = removed_all_requested && removed_ns_groups;
+
+                if !skip_trash && !removed_ns_groups {
+                    self.untrash_namespace(&namespace)?;
+                }
             }
         } else {
             log::info!("pruning empty namespace recursively below {store}:/{ns}");
         }
 
-        removed_all_requested =
-            removed_all_requested && self.destroy_namespace_recursive(ns, delete_groups)?;
+        if skip_trash {
+            removed_all_requested =
+                removed_all_requested && self.destroy_namespace_recursive(ns, delete_groups)?;
+            return Ok((removed_all_requested, stats));
+        }
+
+        let mut children = self
+            .recursive_iter_backup_ns(ns.to_owned(), NamespaceListFilter::Active)?
+            .collect::<Result<Vec<BackupNamespace>, Error>>()?;
+
+        children.sort_by_key(|b| std::cmp::Reverse(b.depth()));
+
+        let mut all_trashed = true;
+        for ns in children.iter() {
+            let mut ns_dir = ns.path();
+            ns_dir.push("ns");
+
+            if !ns.is_root() {
+                let mut path = self.base_path();
+                path.push(ns.path());
+
+                let pending_path = path.join(TRASH_PENDING_MARKER_FILENAME);
+                path.push(TRASH_MARKER_FILENAME);
+                if let Err(err) = std::fs::rename(pending_path, path) {
+                    if err.kind() == std::io::ErrorKind::NotFound {
+                        all_trashed = false;
+                    } else {
+                        return Err(err).context("Renaming pending marker to trash marker failed");
+                    }
+                }
+            }
+        }
+
+        if !all_trashed {
+            bail!("failed to prune namespace, not empty");
+        }
 
         Ok((removed_all_requested, stats))
     }
@@ -657,6 +746,24 @@ impl DataStore {
             let _ = unlinkat(Some(base_fd), &ns_dir, UnlinkatFlags::RemoveDir);
 
             if !ns.is_root() {
+                let rel_trash_path = ns.path().join(TRASH_MARKER_FILENAME);
+                if let Err(err) =
+                    unlinkat(Some(base_fd), &rel_trash_path, UnlinkatFlags::NoRemoveDir)
+                {
+                    if err != nix::errno::Errno::ENOENT {
+                        bail!("removing the trash file '{rel_trash_path:?}' failed - {err}")
+                    }
+                }
+                let rel_pending_path = ns.path().join(TRASH_PENDING_MARKER_FILENAME);
+                if let Err(err) =
+                    unlinkat(Some(base_fd), &rel_pending_path, UnlinkatFlags::NoRemoveDir)
+                {
+                    if err != nix::errno::Errno::ENOENT {
+                        bail!(
+                            "removing the trash pending file '{rel_pending_path:?}' failed - {err}"
+                        )
+                    }
+                }
                 match unlinkat(Some(base_fd), &ns.path(), UnlinkatFlags::RemoveDir) {
                     Ok(()) => log::debug!("removed namespace {ns}"),
                     Err(nix::errno::Errno::ENOENT) => {
diff --git a/src/api2/admin/namespace.rs b/src/api2/admin/namespace.rs
index e5463524a..a12d97883 100644
--- a/src/api2/admin/namespace.rs
+++ b/src/api2/admin/namespace.rs
@@ -166,7 +166,7 @@ pub fn delete_namespace(
 
     let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
 
-    let (removed_all, stats) = datastore.remove_namespace_recursive(&ns, delete_groups)?;
+    let (removed_all, stats) = datastore.remove_namespace_recursive(&ns, delete_groups, true)?;
     if !removed_all {
         let err_msg = if delete_groups {
             if datastore.old_locking() {
diff --git a/src/server/pull.rs b/src/server/pull.rs
index d3c6fcf6a..a60ccbf10 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -729,7 +729,7 @@ fn check_and_remove_ns(params: &PullParameters, local_ns: &BackupNamespace) -> R
     let (removed_all, _delete_stats) = params
         .target
         .store
-        .remove_namespace_recursive(local_ns, true)?;
+        .remove_namespace_recursive(local_ns, true, true)?;
 
     Ok(removed_all)
 }
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:49 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:49 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 15/21] client: expose skip trash
 flags for cli commands
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-16-c.ebner@proxmox.com>

Allows to explicitly set/clear the `skip-trash` flag when pruning
namespaces, groups or snapshots via the client cli command.

Set defaults for `skip-trash` to false in order to use the trash.

Further, never add the flag to the api call parameters in the client
if not explicitly set, in order to keep backwards compatibility to
older PBS instances.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs         |  6 ++++--
 proxmox-backup-client/src/group.rs     | 14 +++++++++++++-
 proxmox-backup-client/src/namespace.rs | 14 +++++++++++++-
 proxmox-backup-client/src/snapshot.rs  | 16 ++++++++++++----
 src/api2/admin/datastore.rs            | 11 +++++++++--
 src/api2/admin/namespace.rs            | 12 ++++++++++--
 src/api2/backup/environment.rs         |  1 +
 src/server/prune_job.rs                |  4 +++-
 src/server/pull.rs                     | 12 +++++++-----
 9 files changed, 72 insertions(+), 18 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index d88af4c68..0f86754bb 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -808,10 +808,11 @@ impl DataStore {
         self: &Arc<Self>,
         ns: &BackupNamespace,
         backup_group: &pbs_api_types::BackupGroup,
+        skip_trash: bool,
     ) -> Result<BackupGroupDeleteStats, Error> {
         let backup_group = self.backup_group(ns.clone(), backup_group.clone());
 
-        backup_group.destroy(true)
+        backup_group.destroy(skip_trash)
     }
 
     /// Remove a backup directory including all content
@@ -820,10 +821,11 @@ impl DataStore {
         ns: &BackupNamespace,
         backup_dir: &pbs_api_types::BackupDir,
         force: bool,
+        skip_trash: bool,
     ) -> Result<(), Error> {
         let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
 
-        backup_dir.destroy(force, true)
+        backup_dir.destroy(force, skip_trash)
     }
 
     /// Returns the time of the last successful backup
diff --git a/proxmox-backup-client/src/group.rs b/proxmox-backup-client/src/group.rs
index 67f26e261..42f8e1e61 100644
--- a/proxmox-backup-client/src/group.rs
+++ b/proxmox-backup-client/src/group.rs
@@ -37,11 +37,20 @@ pub fn group_mgmt_cli() -> CliCommandMap {
                 type: BackupNamespace,
                 optional: true,
             },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                description: "Immediately remove the group, not marking contents as trash.",
+            },
         }
     }
 )]
 /// Forget (remove) backup snapshots.
-async fn forget_group(group: String, mut param: Value) -> Result<(), Error> {
+async fn forget_group(
+    group: String,
+    skip_trash: Option<bool>,
+    mut param: Value,
+) -> Result<(), Error> {
     let backup_group: BackupGroup = group.parse()?;
     let repo = remove_repository_from_value(&mut param)?;
     let client = connect(&repo)?;
@@ -63,6 +72,9 @@ async fn forget_group(group: String, mut param: Value) -> Result<(), Error> {
     )?;
     if confirmation.is_yes() {
         let path = format!("api2/json/admin/datastore/{}/groups", repo.store());
+        if let Some(skip_trash) = skip_trash {
+            api_param["skip-trash"] = Value::from(skip_trash);
+        }
         if let Err(err) = client.delete(&path, Some(api_param)).await {
             // "ENOENT: No such file or directory" is part of the error returned when the group
             // has not been found. The full error contains the full datastore path and we would
diff --git a/proxmox-backup-client/src/namespace.rs b/proxmox-backup-client/src/namespace.rs
index 2929e394b..204a11b1d 100644
--- a/proxmox-backup-client/src/namespace.rs
+++ b/proxmox-backup-client/src/namespace.rs
@@ -136,11 +136,20 @@ async fn create_namespace(param: Value) -> Result<(), Error> {
                 description: "Destroys all groups in the hierarchy.",
                 optional: true,
             },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                description: "Immediately remove the namespace, not marking contents as trash.",
+            },
         }
     },
 )]
 /// Delete an existing namespace.
-async fn delete_namespace(param: Value, delete_groups: Option<bool>) -> Result<(), Error> {
+async fn delete_namespace(
+    param: Value,
+    delete_groups: Option<bool>,
+    skip_trash: Option<bool>,
+) -> Result<(), Error> {
     let repo = extract_repository_from_value(&param)?;
     let backup_ns = optional_ns_param(&param)?;
 
@@ -150,6 +159,9 @@ async fn delete_namespace(param: Value, delete_groups: Option<bool>) -> Result<(
 
     let path = format!("api2/json/admin/datastore/{}/namespace", repo.store());
     let mut param = json!({ "ns": backup_ns });
+    if let Some(skip_trash) = skip_trash {
+        param["skip-trash"] = Value::from(skip_trash);
+    }
 
     if let Some(value) = delete_groups {
         param["delete-groups"] = serde_json::to_value(value)?;
diff --git a/proxmox-backup-client/src/snapshot.rs b/proxmox-backup-client/src/snapshot.rs
index f195c23b7..a9b46726a 100644
--- a/proxmox-backup-client/src/snapshot.rs
+++ b/proxmox-backup-client/src/snapshot.rs
@@ -173,11 +173,16 @@ async fn list_snapshot_files(param: Value) -> Result<Value, Error> {
                 type: String,
                 description: "Snapshot path.",
             },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                description: "Immediately remove the snapshot, not marking it as trash.",
+            },
         }
     }
 )]
 /// Forget (remove) backup snapshots.
-async fn forget_snapshots(param: Value) -> Result<(), Error> {
+async fn forget_snapshots(skip_trash: Option<bool>, param: Value) -> Result<(), Error> {
     let repo = extract_repository_from_value(&param)?;
 
     let backup_ns = optional_ns_param(&param)?;
@@ -188,9 +193,12 @@ async fn forget_snapshots(param: Value) -> Result<(), Error> {
 
     let path = format!("api2/json/admin/datastore/{}/snapshots", repo.store());
 
-    client
-        .delete(&path, Some(snapshot_args(&backup_ns, &snapshot)?))
-        .await?;
+    let mut args = snapshot_args(&backup_ns, &snapshot)?;
+    if let Some(skip_trash) = skip_trash {
+        args["skip-trash"] = Value::from(skip_trash);
+    }
+
+    client.delete(&path, Some(args)).await?;
 
     record_repository(&repo);
 
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 133a6d658..84c0bf5b4 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -277,7 +277,13 @@ pub fn list_groups(
                 optional: true,
                 default: true,
                 description: "Return error when group cannot be deleted because of protected snapshots",
-            }
+            },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                default: false,
+                description: "Immediately remove the snapshot, not marking it as trash.",
+            },
         },
     },
     returns: {
@@ -295,6 +301,7 @@ pub async fn delete_group(
     ns: Option<BackupNamespace>,
     error_on_protected: bool,
     group: pbs_api_types::BackupGroup,
+    skip_trash: bool,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<BackupGroupDeleteStats, Error> {
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
@@ -312,7 +319,7 @@ pub async fn delete_group(
             &group,
         )?;
 
-        let delete_stats = datastore.remove_backup_group(&ns, &group)?;
+        let delete_stats = datastore.remove_backup_group(&ns, &group, skip_trash)?;
 
         let error_msg = if datastore.old_locking() {
             "could not remove empty groups directories due to old locking mechanism.\n\
diff --git a/src/api2/admin/namespace.rs b/src/api2/admin/namespace.rs
index a12d97883..6f70497a6 100644
--- a/src/api2/admin/namespace.rs
+++ b/src/api2/admin/namespace.rs
@@ -144,7 +144,13 @@ pub fn list_namespaces(
                 optional: true,
                 default: true,
                 description: "Return error when namespace cannot be deleted because of protected snapshots",
-            }
+            },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                default: true,
+                description: "Remove and namespace immediately, skip moving to trash",
+            },
         },
     },
     access: {
@@ -157,6 +163,7 @@ pub fn delete_namespace(
     ns: BackupNamespace,
     delete_groups: bool,
     error_on_protected: bool,
+    skip_trash: bool,
     _info: &ApiMethod,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<BackupGroupDeleteStats, Error> {
@@ -166,7 +173,8 @@ pub fn delete_namespace(
 
     let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
 
-    let (removed_all, stats) = datastore.remove_namespace_recursive(&ns, delete_groups, true)?;
+    let (removed_all, stats) =
+        datastore.remove_namespace_recursive(&ns, delete_groups, skip_trash)?;
     if !removed_all {
         let err_msg = if delete_groups {
             if datastore.old_locking() {
diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 3d541b461..b5619eb8c 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -730,6 +730,7 @@ impl BackupEnvironment {
             self.backup_dir.backup_ns(),
             self.backup_dir.as_ref(),
             true,
+            true,
         )?;
 
         Ok(())
diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
index 596afe086..2794dc3ab 100644
--- a/src/server/prune_job.rs
+++ b/src/server/prune_job.rs
@@ -76,7 +76,9 @@ pub fn prune_datastore(
                 info.backup_dir.backup_time_string()
             );
             if !keep && !dry_run {
-                if let Err(err) = datastore.remove_backup_dir(ns, info.backup_dir.as_ref(), false) {
+                if let Err(err) =
+                    datastore.remove_backup_dir(ns, info.backup_dir.as_ref(), false, true)
+                {
                     let path = info.backup_dir.relative_path();
                     warn!("failed to remove dir {path:?}: {err}");
                 }
diff --git a/src/server/pull.rs b/src/server/pull.rs
index a60ccbf10..0d5845e85 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -504,6 +504,7 @@ async fn pull_snapshot_from<'a>(
                     snapshot.backup_ns(),
                     snapshot.as_ref(),
                     true,
+                    true,
                 ) {
                     info!("cleanup error - {cleanup_err}");
                 }
@@ -678,7 +679,7 @@ async fn pull_group(
             params
                 .target
                 .store
-                .remove_backup_dir(&target_ns, snapshot.as_ref(), false)?;
+                .remove_backup_dir(&target_ns, snapshot.as_ref(), false, true)?;
             sync_stats.add(SyncStats::from(RemovedVanishedStats {
                 snapshots: 1,
                 groups: 0,
@@ -997,10 +998,11 @@ pub(crate) async fn pull_ns(
                     continue;
                 }
                 info!("delete vanished group '{local_group}'");
-                let delete_stats_result = params
-                    .target
-                    .store
-                    .remove_backup_group(&target_ns, local_group);
+                let delete_stats_result =
+                    params
+                        .target
+                        .store
+                        .remove_backup_group(&target_ns, local_group, false);
 
                 match delete_stats_result {
                     Ok(stats) => {
-- 
2.39.5




From c.ebner at proxmox.com  Thu May  8 15:05:53 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu,  8 May 2025 15:05:53 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 19/21] ui: add recover for
 trashed items tab to datastore panel
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <20250508130555.494782-20-c.ebner@proxmox.com>

Display a dedicated recover trashed tab which allows to inspect and
recover trashed items.

This is based on the pre-existing contents tab but drops any actions
which make no sense for the given context, such as editing of group
ownership, notes, verification, ecc.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/Makefile                    |   1 +
 www/datastore/Panel.js          |   8 +
 www/datastore/RecoverTrashed.js | 876 ++++++++++++++++++++++++++++++++
 3 files changed, 885 insertions(+)
 create mode 100644 www/datastore/RecoverTrashed.js

diff --git a/www/Makefile b/www/Makefile
index 44c5fa133..aa8955460 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -115,6 +115,7 @@ JSSRC=							\
 	datastore/Panel.js				\
 	datastore/DataStoreListSummary.js		\
 	datastore/DataStoreList.js			\
+	datastore/RecoverTrashed.js			\
 	ServerStatus.js					\
 	ServerAdministration.js				\
 	NodeNotes.js				        \
diff --git a/www/datastore/Panel.js b/www/datastore/Panel.js
index ad9fc10fe..386b62284 100644
--- a/www/datastore/Panel.js
+++ b/www/datastore/Panel.js
@@ -99,6 +99,14 @@ Ext.define('PBS.DataStorePanel', {
 		datastore: '{datastore}',
 	    },
 	},
+	{
+	    xtype: 'pbsDataStoreRecoverTrashed',
+	    itemId: 'trashed',
+	    iconCls: 'fa fa-rotate-left',
+	    cbind: {
+		datastore: '{datastore}',
+	    },
+	},
     ],
 
     initComponent: function() {
diff --git a/www/datastore/RecoverTrashed.js b/www/datastore/RecoverTrashed.js
new file mode 100644
index 000000000..2257a8cd3
--- /dev/null
+++ b/www/datastore/RecoverTrashed.js
@@ -0,0 +1,876 @@
+Ext.define('PBS.DataStoreRecoverTrashed', {
+    extend: 'Ext.tree.Panel',
+    alias: 'widget.pbsDataStoreRecoverTrashed',
+    mixins: ['Proxmox.Mixin.CBind'],
+
+    rootVisible: false,
+
+    title: gettext('Recover Trashed'),
+
+    controller: {
+	xclass: 'Ext.app.ViewController',
+
+	init: function(view) {
+	    if (!view.datastore) {
+		throw "no datastore specified";
+	    }
+
+	    this.store = Ext.create('Ext.data.Store', {
+		model: 'pbs-data-store-snapshots',
+		groupField: 'backup-group',
+	    });
+	    this.store.on('load', this.onLoad, this);
+
+	    view.getStore().setSorters([
+		'sortWeight',
+		'text',
+		'backup-time',
+	    ]);
+	},
+
+	control: {
+	    '#': { // view
+		rowdblclick: 'rowDoubleClicked',
+	    },
+	    'pbsNamespaceSelector': {
+		change: 'nsChange',
+	    },
+	},
+
+	rowDoubleClicked: function(table, rec, el, rowId, ev) {
+	    if (rec?.data?.ty === 'ns' && !rec.data.root) {
+		this.nsChange(null, rec.data.ns);
+	    }
+	},
+
+	nsChange: function(field, value) {
+	    let view = this.getView();
+	    if (field === null) {
+		field = view.down('pbsNamespaceSelector');
+		field.setValue(value);
+		return;
+	    }
+	    view.namespace = value;
+	    this.reload();
+	},
+
+	reload: function() {
+	    let view = this.getView();
+
+	    if (!view.store || !this.store) {
+		console.warn('cannot reload, no store(s)');
+		return;
+	    }
+
+	    let url = `/api2/json/admin/datastore/${view.datastore}/snapshots?trashed=1`;
+	    if (view.namespace && view.namespace !== '') {
+		url += `&ns=${encodeURIComponent(view.namespace)}`;
+	    }
+	    this.store.setProxy({
+		type: 'proxmox',
+		timeout: 300*1000, // 5 minutes, we should make that api call faster
+		url: url,
+	    });
+
+	    this.store.load();
+	},
+
+	getRecordGroups: function(records) {
+	    let groups = {};
+
+	    for (const item of records) {
+		var btype = item.data["backup-type"];
+		let group = btype + "/" + item.data["backup-id"];
+
+		if (groups[group] !== undefined) {
+		    continue;
+		}
+
+		var cls = PBS.Utils.get_type_icon_cls(btype);
+		if (cls === "") {
+		    console.warn(`got unknown backup-type '${btype}'`);
+		    continue; // FIXME: auto render? what do?
+		}
+
+		groups[group] = {
+		    text: group,
+		    leaf: false,
+		    iconCls: "fa " + cls,
+		    expanded: false,
+		    backup_type: item.data["backup-type"],
+		    backup_id: item.data["backup-id"],
+		    children: [],
+		};
+	    }
+
+	    return groups;
+	},
+
+	loadNamespaceFromSameLevel: async function() {
+	    let view = this.getView();
+	    try {
+		let url = `/api2/extjs/admin/datastore/${view.datastore}/namespace?max-depth=1`;
+		if (view.namespace && view.namespace !== '') {
+		    url += `&parent=${encodeURIComponent(view.namespace)}`;
+		}
+		url += '&include-trashed=1';
+		let { result: { data: ns } } = await Proxmox.Async.api2({ url });
+		return ns;
+	    } catch (err) {
+		console.debug(err);
+	    }
+	    return [];
+	},
+
+	onLoad: async function(store, records, success, operation) {
+	    let me = this;
+	    let view = this.getView();
+
+	    let namespaces = await me.loadNamespaceFromSameLevel();
+
+	    if (!success) {
+		// TODO also check error code for != 403 ?
+		if (namespaces.length === 0) {
+		    let error = Proxmox.Utils.getResponseErrorMessage(operation.getError());
+		    Proxmox.Utils.setErrorMask(view.down('treeview'), error);
+		    return;
+		} else {
+		    records = [];
+		}
+	    } else {
+		Proxmox.Utils.setErrorMask(view.down('treeview'));
+	    }
+
+	    let groups = this.getRecordGroups(records);
+
+	    let selected;
+	    let expanded = {};
+
+	    view.getSelection().some(function(item) {
+		let id = item.data.text;
+		if (item.data.leaf) {
+		    id = item.parentNode.data.text + id;
+		}
+		selected = id;
+		return true;
+	    });
+
+	    view.getRootNode().cascadeBy({
+		before: item => {
+		    if (item.isExpanded() && !item.data.leaf) {
+			let id = item.data.text;
+			expanded[id] = true;
+			return true;
+		    }
+		    return false;
+		},
+		after: Ext.emptyFn,
+	    });
+
+	    for (const item of records) {
+		let group = item.data["backup-type"] + "/" + item.data["backup-id"];
+		let children = groups[group].children;
+
+		let data = item.data;
+
+		data.text = group + '/' + PBS.Utils.render_datetime_utc(data["backup-time"]);
+		data.leaf = false;
+		data.cls = 'no-leaf-icons';
+		data.matchesFilter = true;
+		data.ty = 'dir';
+
+		data.expanded = !!expanded[data.text];
+
+		data.children = [];
+		for (const file of data.files) {
+		    file.text = file.filename;
+		    file['crypt-mode'] = PBS.Utils.cryptmap.indexOf(file['crypt-mode']);
+		    file.fingerprint = data.fingerprint;
+		    file.leaf = true;
+		    file.matchesFilter = true;
+		    file.ty = 'file';
+
+		    data.children.push(file);
+		}
+
+		children.push(data);
+	    }
+
+	    let children = [];
+	    for (const [name, group] of Object.entries(groups)) {
+		let last_backup = 0;
+		let crypt = {
+		    none: 0,
+		    mixed: 0,
+		    'sign-only': 0,
+		    encrypt: 0,
+		};
+		for (let item of group.children) {
+		    crypt[PBS.Utils.cryptmap[item['crypt-mode']]]++;
+		    if (item["backup-time"] > last_backup && item.size !== null) {
+			last_backup = item["backup-time"];
+			group["backup-time"] = last_backup;
+			group["last-comment"] = item.comment;
+			group.files = item.files;
+			group.size = item.size;
+			group.owner = item.owner;
+		    }
+		}
+		group.count = group.children.length;
+		group.matchesFilter = true;
+		crypt.count = group.count;
+		group['crypt-mode'] = PBS.Utils.calculateCryptMode(crypt);
+		group.expanded = !!expanded[name];
+		group.sortWeight = 0;
+		group.ty = 'group';
+		children.push(group);
+	    }
+
+	    for (const item of namespaces) {
+		if (item.ns === view.namespace || (!view.namespace && item.ns === '')) {
+		    continue;
+		}
+		children.push({
+		    text: item.ns,
+		    iconCls: 'fa fa-object-group',
+		    expanded: true,
+		    expandable: false,
+		    ns: (view.namespaces ?? '') !== '' ? `/${item.ns}` : item.ns,
+		    ty: 'ns',
+		    sortWeight: 10,
+		    leaf: true,
+		});
+	    }
+
+	    let isRootNS = !view.namespace || view.namespace === '';
+	    let rootText = isRootNS
+		? gettext('Root Namespace')
+		: Ext.String.format(gettext("Namespace '{0}'"), view.namespace);
+
+	    let topNodes = [];
+	    if (!isRootNS) {
+		let parentNS = view.namespace.split('/').slice(0, -1).join('/');
+		topNodes.push({
+		    text: `.. (${parentNS === '' ? gettext('Root') : parentNS})`,
+		    iconCls: 'fa fa-level-up',
+		    ty: 'ns',
+		    ns: parentNS,
+		    sortWeight: -10,
+		    leaf: true,
+		});
+	    }
+	    topNodes.push({
+		text: rootText,
+		iconCls: "fa fa-" + (isRootNS ? 'database' : 'object-group'),
+		expanded: true,
+		expandable: false,
+		sortWeight: -5,
+		root: true, // fake root
+		isRootNS,
+		ty: 'ns',
+		children: children,
+	    });
+
+	    view.setRootNode({
+		expanded: true,
+		children: topNodes,
+	    });
+
+	    if (!children.length) {
+		view.setEmptyText(Ext.String.format(
+		    gettext('No accessible snapshots found in namespace {0}'),
+		    view.namespace && view.namespace !== '' ? `'${view.namespace}'`: gettext('Root'),
+		));
+	    }
+
+	    if (selected !== undefined) {
+		let selection = view.getRootNode().findChildBy(function(item) {
+		    let id = item.data.text;
+		    if (item.data.leaf) {
+			id = item.parentNode.data.text + id;
+		    }
+		    return selected === id;
+		}, undefined, true);
+		if (selection) {
+		    view.setSelection(selection);
+		    view.getView().focusRow(selection);
+		}
+	    }
+
+	    Proxmox.Utils.setErrorMask(view, false);
+	    if (view.getStore().getFilters().length > 0) {
+		let searchBox = me.lookup("searchbox");
+		let searchvalue = searchBox.getValue();
+		me.search(searchBox, searchvalue);
+	    }
+	},
+
+	recoverNamespace: function(data) {
+	    let me = this;
+	    let view = me.getView();
+	    if (!view.namespace || view.namespace === '') {
+		console.warn('recoverNamespace called with root NS!');
+		return;
+	    }
+	    Ext.Msg.show({
+		title: gettext('Confirm'),
+		icon: Ext.Msg.WARNING,
+		message: gettext('Are you sure you want to recover all namespace contents?'),
+		buttons: Ext.Msg.YESNO,
+		defaultFocus: 'yes',
+		callback: function(btn) {
+		    if (btn !== 'yes') {
+		        return;
+		    }
+		    let params = { "ns": view.namespace };
+
+		    Proxmox.Utils.API2Request({
+			url: `/admin/datastore/${view.datastore}/recover-namespace`,
+			params,
+			method: 'POST',
+			waitMsgTarget: view,
+			failure: function(response, opts) {
+			    Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+			},
+			callback: me.reload.bind(me),
+		    });
+		},
+	    });
+	},
+
+	forgetNamespace: function(data) {
+	    let me = this;
+	    let view = me.getView();
+	    if (!view.namespace || view.namespace === '') {
+		console.warn('forgetNamespace called with root NS!');
+		return;
+	    }
+	    let nsParts = view.namespace.split('/');
+	    let nsName = nsParts.pop();
+	    let parentNS = nsParts.join('/');
+
+	    Ext.create('PBS.window.NamespaceDelete', {
+		datastore: view.datastore,
+		namespace: view.namespace,
+		item: { id: nsName },
+		apiCallDone: success => {
+		    if (success) {
+			view.namespace = parentNS; // move up before reload to avoid "ENOENT" error
+			me.reload();
+		    }
+		},
+	    });
+	},
+
+	recoverGroup: function(data) {
+	    let me = this;
+	    let view = me.getView();
+
+	    let params = {
+		"backup-type": data.backup_type,
+		"backup-id": data.backup_id,
+	    };
+	    if (view.namespace && view.namespace !== '') {
+		params.ns = view.namespace;
+	    }
+
+	    Ext.Msg.show({
+		title: gettext('Confirm'),
+		icon: Ext.Msg.WARNING,
+		message: Ext.String.format(gettext('Are you sure you want to recover group {0}'), `'${data.text}'`),
+		buttons: Ext.Msg.YESNO,
+		defaultFocus: 'yes',
+		callback: function(btn) {
+		    if (btn !== 'yes') {
+		        return;
+		    }
+
+		    Proxmox.Utils.API2Request({
+			url: `/admin/datastore/${view.datastore}/recover-group`,
+			params,
+			method: 'POST',
+			waitMsgTarget: view,
+			failure: function(response, opts) {
+			    Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+			},
+			callback: me.reload.bind(me),
+		    });
+		},
+	    });
+	},
+
+	forgetGroup: function(data) {
+	    let me = this;
+	    let view = me.getView();
+
+	    let params = {
+		"backup-type": data.backup_type,
+		"backup-id": data.backup_id,
+		"skip-trash": true,
+	    };
+	    if (view.namespace && view.namespace !== '') {
+		params.ns = view.namespace;
+	    }
+
+	    Ext.create('Proxmox.window.SafeDestroy', {
+		url: `/admin/datastore/${view.datastore}/groups`,
+		params,
+		item: {
+		    id: data.text,
+		},
+		autoShow: true,
+		taskName: 'forget-group',
+		listeners: {
+		    destroy: () => me.reload(),
+		},
+	    });
+	},
+
+	recoverSnapshot: function(data) {
+	    let me = this;
+	    let view = me.getView();
+
+	    Ext.Msg.show({
+		title: gettext('Confirm'),
+		icon: Ext.Msg.WARNING,
+		message: Ext.String.format(gettext('Are you sure you want to recover snapshot {0}'), `'${data.text}'`),
+		buttons: Ext.Msg.YESNO,
+		defaultFocus: 'yes',
+		callback: function(btn) {
+		    if (btn !== 'yes') {
+		        return;
+		    }
+		    let params = {
+			"backup-type": data["backup-type"],
+			"backup-id": data["backup-id"],
+			"backup-time": (data['backup-time'].getTime()/1000).toFixed(0),
+		    };
+		    if (view.namespace && view.namespace !== '') {
+			params.ns = view.namespace;
+		    }
+
+		    //TODO adapt to recover api endpoint
+		    Proxmox.Utils.API2Request({
+			url: `/admin/datastore/${view.datastore}/recover-snapshot`,
+			params,
+			method: 'POST',
+			waitMsgTarget: view,
+			failure: function(response, opts) {
+			    Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+			},
+			callback: me.reload.bind(me),
+		    });
+		},
+	    });
+	},
+
+	forgetSnapshot: function(data) {
+	    let me = this;
+	    let view = me.getView();
+
+	    Ext.Msg.show({
+		title: gettext('Confirm'),
+		icon: Ext.Msg.WARNING,
+		message: Ext.String.format(gettext('Are you sure you want to remove snapshot {0}'), `'${data.text}'`),
+		buttons: Ext.Msg.YESNO,
+		defaultFocus: 'no',
+		callback: function(btn) {
+		    if (btn !== 'yes') {
+		        return;
+		    }
+		    let params = {
+			"backup-type": data["backup-type"],
+			"backup-id": data["backup-id"],
+			"backup-time": (data['backup-time'].getTime()/1000).toFixed(0),
+			"skip-trash": true,
+		    };
+		    if (view.namespace && view.namespace !== '') {
+			params.ns = view.namespace;
+		    }
+
+		    Proxmox.Utils.API2Request({
+			url: `/admin/datastore/${view.datastore}/snapshots`,
+			params,
+			method: 'DELETE',
+			waitMsgTarget: view,
+			failure: function(response, opts) {
+			    Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+			},
+			callback: me.reload.bind(me),
+		    });
+		},
+	    });
+	},
+
+	onRecover: function(table, rI, cI, item, e, { data }) {
+	    let me = this;
+	    let view = this.getView();
+	    if ((data.ty !== 'group' && data.ty !== 'dir' && data.ty !== 'ns') || !view.datastore) {
+		return;
+	    }
+
+	    if (data.ty === 'ns') {
+		me.recoverNamespace(data);
+	    } else if (data.ty === 'dir') {
+		me.recoverSnapshot(data);
+	    } else {
+		me.recoverGroup(data);
+	    }
+	},
+
+	onForget: function(table, rI, cI, item, e, { data }) {
+	    let me = this;
+	    let view = this.getView();
+	    if ((data.ty !== 'group' && data.ty !== 'dir' && data.ty !== 'ns') || !view.datastore) {
+		return;
+	    }
+
+	    if (data.ty === 'ns') {
+		me.forgetNamespace(data);
+	    } else if (data.ty === 'dir') {
+		me.forgetSnapshot(data);
+	    } else {
+		me.forgetGroup(data);
+	    }
+	},
+
+	// opens a namespace browser
+	openBrowser: function(tv, rI, Ci, item, e, rec) {
+	    let me = this;
+	    if (rec.data.ty === 'ns') {
+		me.nsChange(null, rec.data.ns);
+	    }
+	},
+
+	filter: function(item, value) {
+	    if (item.data.text.indexOf(value) !== -1) {
+		return true;
+	    }
+
+	    if (item.data.owner && item.data.owner.indexOf(value) !== -1) {
+		return true;
+	    }
+
+	    return false;
+	},
+
+	search: function(tf, value) {
+	    let me = this;
+	    let view = me.getView();
+	    let store = view.getStore();
+	    if (!value && value !== 0) {
+		store.clearFilter();
+		// only collapse the children below our toplevel namespace "root"
+		store.getRoot().lastChild.collapseChildren(true);
+		tf.triggers.clear.setVisible(false);
+		return;
+	    }
+	    tf.triggers.clear.setVisible(true);
+	    if (value.length < 2) return;
+	    Proxmox.Utils.setErrorMask(view, true);
+	    // we do it a little bit later for the error mask to work
+	    setTimeout(function() {
+		store.clearFilter();
+		store.getRoot().collapseChildren(true);
+
+		store.beginUpdate();
+		store.getRoot().cascadeBy({
+		    before: function(item) {
+			if (me.filter(item, value)) {
+			    item.set('matchesFilter', true);
+			    if (item.parentNode && item.parentNode.id !== 'root') {
+				item.parentNode.childmatches = true;
+			    }
+			    return false;
+			}
+			return true;
+		    },
+		    after: function(item) {
+			if (me.filter(item, value) || item.id === 'root' || item.childmatches) {
+			    item.set('matchesFilter', true);
+			    if (item.parentNode && item.parentNode.id !== 'root') {
+				item.parentNode.childmatches = true;
+			    }
+			    if (item.childmatches) {
+				item.expand();
+			    }
+			} else {
+			    item.set('matchesFilter', false);
+			}
+			delete item.childmatches;
+		    },
+		});
+		store.endUpdate();
+
+		store.filter((item) => !!item.get('matchesFilter'));
+		Proxmox.Utils.setErrorMask(view, false);
+	    }, 10);
+	},
+    },
+
+    listeners: {
+	activate: function() {
+	    let me = this;
+	    me.getController().reload();
+	},
+	itemcontextmenu: function(panel, record, item, index, event) {
+	    event.stopEvent();
+	    let title;
+	    let view = panel.up('pbsDataStoreRecoverTrashed');
+	    let controller = view.getController();
+	    let createControllerCallback = function(name) {
+		return function() {
+		    controller[name](view, undefined, undefined, undefined, undefined, record);
+		};
+	    };
+	    if (record.data.ty === 'group') {
+		title = gettext('Group');
+	    } else if (record.data.ty === 'dir') {
+		title = gettext('Snapshot');
+	    } else if (record.data.ty === 'ns') {
+		title = gettext('Namespace');
+	    }
+	    if (title) {
+		let menu = Ext.create('PBS.datastore.RecoverTrashedContextMenu', {
+		    title: title,
+		    onRecover: createControllerCallback('onRecover'),
+		    onForget: createControllerCallback('onForget'),
+		});
+		menu.showAt(event.getXY());
+	    }
+	},
+    },
+
+    columns: [
+	{
+	    xtype: 'treecolumn',
+	    header: gettext("Backup Group"),
+	    dataIndex: 'text',
+	    renderer: (value, meta, record) => {
+		if (record.data.protected) {
+		    return `${value} (${gettext('protected')})`;
+		}
+		return value;
+	    },
+	    flex: 1,
+	},
+	{
+	    text: gettext('Comment'),
+	    dataIndex: 'comment',
+	    flex: 1,
+	    renderer: (v, meta, record) => {
+		let data = record.data;
+		if (!data || data.leaf || data.root) {
+		    return '';
+		}
+
+		let additionalClasses = "";
+		if (!v) {
+		    if (!data.expanded) {
+			v = data['last-comment'] ?? '';
+			additionalClasses = 'pmx-opacity-75';
+		    } else {
+			v = '';
+		    }
+		}
+		v = Ext.String.htmlEncode(v);
+		return `<span class="snapshot-comment-column ${additionalClasses}">${v}</span>`;
+	    },
+	},
+	{
+	    header: gettext('Actions'),
+	    xtype: 'actioncolumn',
+	    dataIndex: 'text',
+	    width: 80,
+	    items: [
+		{
+		    handler: 'onRecover',
+		    getTip: (v, m, { data }) => {
+			let tip = '{0}';
+			if (data.ty === 'ns') {
+			    tip = gettext("Recover all namespace contents");
+			} else if (data.ty === 'dir') {
+			    tip = gettext("Recover snapshot '{0}'");
+			} else if (data.ty === 'group') {
+			    tip = gettext("Recover group '{0}'");
+			}
+			return Ext.String.format(tip, v);
+		    },
+		    getClass: (v, m, { data }) =>
+		        (data.ty === 'ns' && !data.isRootNS && data.ns === undefined) ||
+		           data.ty === 'group' || data.ty === 'dir'
+		        ? 'fa fa-rotate-left'
+		        : 'pmx-hidden',
+		    isActionDisabled: (v, r, c, i, { data }) => false,
+		},
+		'->',
+		{
+		    handler: 'onForget',
+		    getTip: (v, m, { data }) => {
+			let tip = '{0}';
+			if (data.ty === 'ns') {
+			    tip = gettext("Permanently forget namespace contents '{0}'");
+			} else if (data.ty === 'dir') {
+			    tip = gettext("Permanently forget snapshot '{0}'");
+			} else if (data.ty === 'group') {
+			    tip = gettext("Permanently forget group '{0}'");
+			}
+			return Ext.String.format(tip, v);
+		    },
+		    getClass: (v, m, { data }) =>
+		        (data.ty === 'ns' && !data.isRootNS && data.ns === undefined) ||
+		           data.ty === 'group' || data.ty === 'dir'
+		        ? 'fa critical fa-trash-o'
+		        : 'pmx-hidden',
+		    isActionDisabled: (v, r, c, i, { data }) => false,
+		},
+		{
+		    handler: 'openBrowser',
+		    tooltip: gettext('Browse'),
+		    getClass: (v, m, { data }) => data.ty === 'ns' && !data.root
+			? 'fa fa-folder-open-o'
+			: 'pmx-hidden',
+		    isActionDisabled: (v, r, c, i, { data }) => data.ty !== 'ns',
+		},
+	    ],
+	},
+	{
+	    xtype: 'datecolumn',
+	    header: gettext('Backup Time'),
+	    sortable: true,
+	    dataIndex: 'backup-time',
+	    format: 'Y-m-d H:i:s',
+	    width: 150,
+	},
+	{
+	    header: gettext("Size"),
+	    sortable: true,
+	    dataIndex: 'size',
+	    renderer: (v, meta, { data }) => {
+		if ((data.text === 'client.log.blob' && v === undefined) || (data.ty !== 'dir' && data.ty !== 'file')) {
+		    return '';
+		}
+		if (v === undefined || v === null) {
+		    meta.tdCls = "x-grid-row-loading";
+		    return '';
+		}
+		return Proxmox.Utils.format_size(v);
+	    },
+	},
+	{
+	    xtype: 'numbercolumn',
+	    format: '0',
+	    header: gettext("Count"),
+	    sortable: true,
+	    width: 75,
+	    align: 'right',
+	    dataIndex: 'count',
+	},
+	{
+	    header: gettext("Owner"),
+	    sortable: true,
+	    dataIndex: 'owner',
+	},
+	{
+	    header: gettext('Encrypted'),
+	    dataIndex: 'crypt-mode',
+	    renderer: (v, meta, record) => {
+		if (record.data.size === undefined || record.data.size === null) {
+		    return '';
+		}
+		if (v === -1) {
+		    return '';
+		}
+		let iconCls = PBS.Utils.cryptIconCls[v] || '';
+		let iconTxt = "";
+		if (iconCls) {
+		    iconTxt = `<i class="fa fa-fw fa-${iconCls}"></i> `;
+		}
+		let tip;
+		if (v !== PBS.Utils.cryptmap.indexOf('none') && record.data.fingerprint !== undefined) {
+		    tip = "Key: " + PBS.Utils.renderKeyID(record.data.fingerprint);
+		}
+		let txt = (iconTxt + PBS.Utils.cryptText[v]) || Proxmox.Utils.unknownText;
+		if (record.data.ty === 'group' || tip === undefined) {
+		    return txt;
+		} else {
+		    return `<span data-qtip="${tip}">${txt}</span>`;
+		}
+	    },
+	},
+    ],
+
+    tbar: [
+	{
+	    text: gettext('Reload'),
+	    iconCls: 'fa fa-refresh',
+	    handler: 'reload',
+	},
+	'->',
+	{
+	    xtype: 'tbtext',
+	    html: gettext('Namespace') + ':',
+	},
+	{
+	    xtype: 'pbsNamespaceSelector',
+	    width: 200,
+	    cbind: {
+		datastore: '{datastore}',
+	    },
+	},
+	'-',
+	{
+	    xtype: 'tbtext',
+	    html: gettext('Search'),
+	},
+	{
+	    xtype: 'textfield',
+	    reference: 'searchbox',
+	    emptyText: gettext('group, date or owner'),
+	    triggers: {
+		clear: {
+		    cls: 'pmx-clear-trigger',
+		    weight: -1,
+		    hidden: true,
+		    handler: function() {
+			this.triggers.clear.setVisible(false);
+			this.setValue('');
+		    },
+		},
+	    },
+	    listeners: {
+		change: {
+		    fn: 'search',
+		    buffer: 500,
+		},
+	    },
+	},
+    ],
+});
+
+Ext.define('PBS.datastore.RecoverTrashedContextMenu', {
+    extend: 'Ext.menu.Menu',
+    mixins: ['Proxmox.Mixin.CBind'],
+
+    onRecover: undefined,
+    onForget: undefined,
+
+    items: [
+	{
+	    text: gettext('Recover'),
+	    iconCls: 'fa critical fa-rotate-left',
+	    handler: function() { this.up('menu').onRecover(); },
+	    cbind: {
+		hidden: '{!onRecover}',
+	    },
+	},
+	{
+	    text: gettext('Remove'),
+	    iconCls: 'fa critical fa-trash-o',
+	    handler: function() { this.up('menu').onForget(); },
+	    cbind: {
+		hidden: '{!onForget}',
+	    },
+	},
+    ],
+});
-- 
2.39.5




From f.gruenbichler at proxmox.com  Fri May  9 14:27:15 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Fri, 09 May 2025 14:27:15 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 18/21] api: admin: implement
 endpoints to restore trashed contents
In-Reply-To: <20250508130555.494782-19-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-19-c.ebner@proxmox.com>
Message-ID: <1746793013.k8qdvp27bh.astroid@yuna.none>

On May 8, 2025 3:05 pm, Christian Ebner wrote:
> Implements the api endpoints to restore trashed contents contained
> within namespaces, backup groups or individual snapshots.
> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  src/api2/admin/datastore.rs | 173 +++++++++++++++++++++++++++++++++++-
>  1 file changed, 172 insertions(+), 1 deletion(-)
> 
> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> index cbd24c729..eb033c3fc 100644
> --- a/src/api2/admin/datastore.rs
> +++ b/src/api2/admin/datastore.rs
> @@ -51,7 +51,7 @@ use pbs_api_types::{
>  };
>  use pbs_client::pxar::{create_tar, create_zip};
>  use pbs_config::CachedUserInfo;
> -use pbs_datastore::backup_info::{BackupInfo, ListBackupFilter};
> +use pbs_datastore::backup_info::{BackupInfo, ListBackupFilter, TRASH_MARKER_FILENAME};
>  use pbs_datastore::cached_chunk_reader::CachedChunkReader;
>  use pbs_datastore::catalog::{ArchiveEntry, CatalogReader};
>  use pbs_datastore::data_blob::DataBlob;
> @@ -2727,6 +2727,165 @@ pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<V
>      Ok(json!(upid))
>  }
>  
> +#[api(
> +    input: {
> +        properties: {
> +            store: { schema: DATASTORE_SCHEMA },
> +            ns: { type: BackupNamespace, },
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Anybody,
> +        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
> +            or DATASTORE_BACKUP and being the owner of the group",
> +    },
> +)]
> +/// Recover trashed contents of a namespace.
> +pub fn recover_namespace(
> +    store: String,
> +    ns: BackupNamespace,
> +    rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> +    let limited = check_ns_privs_full(
> +        &store,
> +        &ns,
> +        &auth_id,
> +        PRIV_DATASTORE_MODIFY,
> +        PRIV_DATASTORE_BACKUP,
> +    )?;
> +
> +    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
> +
> +    for backup_group in datastore.iter_backup_groups(ns.clone())? {
> +        let backup_group = backup_group?;
> +        if limited {
> +            let owner = datastore.get_owner(&ns, backup_group.group())?;
> +            if check_backup_owner(&owner, &auth_id).is_err() {
> +                continue;
> +            }
> +        }
> +        do_recover_group(&backup_group)?;
> +    }
> +
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            store: { schema: DATASTORE_SCHEMA },
> +            group: {
> +                type: pbs_api_types::BackupGroup,
> +                flatten: true,
> +            },
> +            ns: {
> +                type: BackupNamespace,
> +                optional: true,
> +            },
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Anybody,
> +        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
> +            or DATASTORE_BACKUP and being the owner of the group",
> +    },
> +)]
> +/// Recover trashed contents of a backup group.
> +pub fn recover_group(
> +    store: String,
> +    group: pbs_api_types::BackupGroup,
> +    ns: Option<BackupNamespace>,
> +    rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> +    let ns = ns.unwrap_or_default();
> +    let datastore = check_privs_and_load_store(
> +        &store,
> +        &ns,
> +        &auth_id,
> +        PRIV_DATASTORE_MODIFY,
> +        PRIV_DATASTORE_BACKUP,
> +        Some(Operation::Write),
> +        &group,
> +    )?;
> +
> +    let backup_group = datastore.backup_group(ns, group);
> +    do_recover_group(&backup_group)?;
> +
> +    Ok(())
> +}
> +
> +fn do_recover_group(backup_group: &BackupGroup) -> Result<(), Error> {

missing locking for the group?

> +    let trashed_snapshots = backup_group.list_backups(ListBackupFilter::Trashed)?;
> +    for snapshot in trashed_snapshots {
> +        do_recover_snapshot(&snapshot.backup_dir)?;
> +    }
> +
> +    let group_trash_path = backup_group.full_group_path().join(TRASH_MARKER_FILENAME);
> +    if let Err(err) = std::fs::remove_file(&group_trash_path) {
> +        if err.kind() != std::io::ErrorKind::NotFound {
> +            bail!("failed to remove group trash file {group_trash_path:?} - {err}");
> +        }
> +    }
> +    Ok(())
> +}
> +
> +#[api(
> +    input: {
> +        properties: {
> +            store: { schema: DATASTORE_SCHEMA },
> +            backup_dir: {
> +                type: pbs_api_types::BackupDir,
> +                flatten: true,
> +            },
> +            ns: {
> +                type: BackupNamespace,
> +                optional: true,
> +            },
> +        },
> +    },
> +    access: {
> +        permission: &Permission::Anybody,
> +        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
> +            or DATASTORE_BACKUP and being the owner of the group",
> +    },
> +)]
> +/// Recover trashed contents of a backup snapshot.
> +pub fn recover_snapshot(
> +    store: String,
> +    backup_dir: pbs_api_types::BackupDir,
> +    ns: Option<BackupNamespace>,
> +    rpcenv: &mut dyn RpcEnvironment,
> +) -> Result<(), Error> {
> +    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
> +    let ns = ns.unwrap_or_default();
> +    let datastore = check_privs_and_load_store(
> +        &store,
> +        &ns,
> +        &auth_id,
> +        PRIV_DATASTORE_MODIFY,
> +        PRIV_DATASTORE_BACKUP,
> +        Some(Operation::Write),
> +        &backup_dir.group,
> +    )?;
> +
> +    let snapshot = datastore.backup_dir(ns, backup_dir)?;
> +    do_recover_snapshot(&snapshot)?;
> +
> +    Ok(())
> +}
> +
> +fn do_recover_snapshot(snapshot_dir: &BackupDir) -> Result<(), Error> {

missing locking for the snapshot?

> +    let trash_path = snapshot_dir.full_path().join(TRASH_MARKER_FILENAME);
> +    if let Err(err) = std::fs::remove_file(&trash_path) {
> +        if err.kind() != std::io::ErrorKind::NotFound {
> +            bail!("failed to remove trash file {trash_path:?} - {err}");
> +        }
> +    }
> +    Ok(())
> +}
> +
>  #[sortable]
>  const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
>      (
> @@ -2792,6 +2951,18 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
>          "pxar-file-download",
>          &Router::new().download(&API_METHOD_PXAR_FILE_DOWNLOAD),
>      ),
> +    (
> +        "recover-group",
> +        &Router::new().post(&API_METHOD_RECOVER_GROUP),

I am not sure whether those should be POST or PUT, they are modifying an
existing (trashed) group/snapshot/.. after all?

> +    ),
> +    (
> +        "recover-namespace",
> +        &Router::new().post(&API_METHOD_RECOVER_NAMESPACE),
> +    ),
> +    (
> +        "recover-snapshot",
> +        &Router::new().post(&API_METHOD_RECOVER_SNAPSHOT),
> +    ),
>      ("rrd", &Router::new().get(&API_METHOD_GET_RRD_STATS)),
>      (
>          "snapshots",
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 



From f.gruenbichler at proxmox.com  Fri May  9 14:27:17 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Fri, 09 May 2025 14:27:17 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 16/21] api: datastore: add
 flag to list trashed snapshots only
In-Reply-To: <20250508130555.494782-17-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-17-c.ebner@proxmox.com>
Message-ID: <1746792915.yfm4snbghn.astroid@yuna.none>

On May 8, 2025 3:05 pm, Christian Ebner wrote:
> Allows to conditionally show either active or trashed backup
> snapshots, the latter being used when displaying the contents of the
> trash for given datastore.

and what if I want to list both/all?

> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  src/api2/admin/datastore.rs | 17 +++++++++++++++--
>  1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> index 84c0bf5b4..cbd24c729 100644
> --- a/src/api2/admin/datastore.rs
> +++ b/src/api2/admin/datastore.rs
> @@ -473,6 +473,12 @@ pub async fn delete_snapshot(
>                  optional: true,
>                  schema: BACKUP_ID_SCHEMA,
>              },
> +            "trashed": {
> +                type: bool,
> +                optional: true,
> +                default: false,
> +                description: "List trashed snapshots only."
> +            },
>          },
>      },
>      returns: pbs_api_types::ADMIN_DATASTORE_LIST_SNAPSHOTS_RETURN_TYPE,
> @@ -488,6 +494,7 @@ pub async fn list_snapshots(
>      ns: Option<BackupNamespace>,
>      backup_type: Option<BackupType>,
>      backup_id: Option<String>,
> +    trashed: bool,
>      _param: Value,
>      _info: &ApiMethod,
>      rpcenv: &mut dyn RpcEnvironment,
> @@ -495,7 +502,7 @@ pub async fn list_snapshots(
>      let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
>  
>      tokio::task::spawn_blocking(move || unsafe {
> -        list_snapshots_blocking(store, ns, backup_type, backup_id, auth_id)
> +        list_snapshots_blocking(store, ns, backup_type, backup_id, auth_id, trashed)
>      })
>      .await
>      .map_err(|err| format_err!("failed to await blocking task: {err}"))?
> @@ -508,6 +515,7 @@ unsafe fn list_snapshots_blocking(
>      backup_type: Option<BackupType>,
>      backup_id: Option<String>,
>      auth_id: Authid,
> +    trashed: bool,
>  ) -> Result<Vec<SnapshotListItem>, Error> {
>      let ns = ns.unwrap_or_default();
>  
> @@ -631,7 +639,12 @@ unsafe fn list_snapshots_blocking(
>              return Ok(snapshots);
>          }
>  
> -        let group_backups = group.list_backups(ListBackupFilter::Active)?;
> +        let filter = if trashed {
> +            ListBackupFilter::Trashed
> +        } else {
> +            ListBackupFilter::Active
> +        };
> +        let group_backups = group.list_backups(filter)?;
>  
>          snapshots.extend(
>              group_backups
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 



From f.gruenbichler at proxmox.com  Fri May  9 14:27:25 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Fri, 09 May 2025 14:27:25 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 14/21] datastore: GC:
 clean-up trashed snapshots, groups and namespaces
In-Reply-To: <20250508130555.494782-15-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-15-c.ebner@proxmox.com>
Message-ID: <1746792635.lyqxegtia2.astroid@yuna.none>

On May 8, 2025 3:05 pm, Christian Ebner wrote:
> Cleanup trashed items during phase 1 of garbage collection. If
> encountered, index files located within trashed snapshots are touched
> as well, deferring chunk cleanup to the next run
> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  pbs-datastore/src/datastore.rs | 84 +++++++++++++++++++++++++++++++++-
>  1 file changed, 83 insertions(+), 1 deletion(-)
> 
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index ca05e1bea..d88af4c68 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -574,6 +574,18 @@ impl DataStore {
>          !path.exists()
>      }
>  
> +    /// Checks if the namespace trash marker file exists,
> +    /// does not imply that the namespace itself exists.
> +    pub fn namespace_is_trashed(&self, namespace: &BackupNamespace) -> bool {
> +        if namespace.is_root() {
> +            return false;
> +        }
> +        let mut path = self.base_path();
> +        path.push(namespace.path());
> +        path.push(TRASH_MARKER_FILENAME);
> +        path.exists()
> +    }
> +
>      /// Remove the namespace and all it's parent components from the trash by removing the trash or
>      /// trash-pending marker file for each namespace level from deepest to shallowest. Missing files
>      /// are ignored.
> @@ -1322,7 +1334,7 @@ impl DataStore {
>              .context("creating namespace iterator failed")?
>          {
>              let namespace = namespace.context("iterating namespaces failed")?;
> -            for group in arc_self.iter_backup_groups(namespace)? {
> +            for group in arc_self.iter_backup_groups(namespace.clone())? {
>                  let group = group.context("iterating backup groups failed")?;
>  
>                  // Avoid race between listing/marking of snapshots by GC and pruning the last
> @@ -1403,10 +1415,80 @@ impl DataStore {
>                              }
>                              processed_index_files += 1;
>                          }
> +
> +                        // Only try to lock a trashed snapshots and continue if that is not
> +                        // possible, as then most likely this is in the process of being untrashed.
> +                        // Check trash state before and after locking to avoid otherwise possible
> +                        // races.
> +                        if snapshot.backup_dir.is_trashed() {
> +                            if let Ok(_lock) = snapshot.backup_dir.lock() {
> +                                if snapshot.backup_dir.is_trashed() {
> +                                    let path = snapshot.backup_dir.full_path();
> +                                    log::info!("removing trashed backup snapshot {path:?}");
> +                                    std::fs::remove_dir_all(&path).with_context(|| {
> +                                        format!("removing trashed backup snapshot {path:?} failed")
> +                                    })?;
> +                                }
> +                            } else {
> +                                let path = snapshot.backup_dir.full_path();
> +                                warn!("failed to lock trashed backup snapshot can {path:?}");
> +                            }
> +                        }
>                      }
>  
>                      break;
>                  }
> +                if group.is_trashed() {
> +                    if let Ok(_lock) = group.lock() {
> +                        if group.is_trashed() {

shouldn't this use some helper to reduce code duplication?

> +                            let trash_path = group.full_group_path().join(".trashed");
> +                            std::fs::remove_file(&trash_path).map_err(|err| {
> +                                format_err!(
> +                                    "removing the trash file '{trash_path:?}' failed - {err}"
> +                                )
> +                            })?;
> +
> +                            let owner_path = group.full_group_path().join("owner");
> +                            std::fs::remove_file(&owner_path).map_err(|err| {
> +                                format_err!(
> +                                    "removing the owner file '{owner_path:?}' failed - {err}"
> +                                )
> +                            })?;
> +
> +                            let path = group.full_group_path();
> +
> +                            std::fs::remove_dir(&path).map_err(|err| {
> +                                format_err!("removing group directory {path:?} failed - {err}")
> +                            })?;
> +
> +                            // Remove any now empty backup type directory

is this needed here? if we remove the whole namespace below, it would be
done anyway..

> +                            let base_file = std::fs::File::open(self.base_path())?;
> +                            let base_fd = base_file.as_raw_fd();
> +                            for ty in BackupType::iter() {
> +                                let mut ty_dir = namespace.path();
> +                                ty_dir.push(ty.to_string());
> +                                match unlinkat(Some(base_fd), &ty_dir, UnlinkatFlags::RemoveDir) {
> +                                    Ok(_) => (),
> +                                    Err(nix::errno::Errno::ENOENT) |
> +                                    Err(nix::errno::Errno::ENOTEMPTY) => (),
> +                                    Err(err) => info!("failed to remove backup type directory for {namespace} - {err}"),
> +                                }
> +                            }
> +                        } else {
> +                            let path = group.full_group_path();
> +                            warn!("failed to lock trashed backup group {path:?}");
> +                        }
> +                    }
> +                }
> +            }
> +            if self.namespace_is_trashed(&namespace) {
> +                // Remove the namespace, but only if it was empty (as the GC already cleared child
> +                // items and no new ones have been created since).
> +                match arc_self.destroy_namespace_recursive(&namespace, false) {
> +                    Ok(true) => info!("removed trashed namespace {namespace}"),
> +                    Ok(false) => info!("failed to remove trashed namespace {namespace}, not empty"),
> +                    Err(err) => warn!("removing trashed namespace failed: {err:#}"),
> +                }
>              }
>          }
>  
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 



From f.gruenbichler at proxmox.com  Fri May  9 14:27:39 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Fri, 09 May 2025 14:27:39 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 07/21] sync: ignore trashed
 groups in local source reader
In-Reply-To: <20250508130555.494782-8-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-8-c.ebner@proxmox.com>
Message-ID: <1746790767.6u1tbrjgn7.astroid@yuna.none>

On May 8, 2025 3:05 pm, Christian Ebner wrote:
> Check and exclude backup groups which have been marked as trash from
> sync.

could be grouped with patch 5?

> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  pbs-datastore/src/backup_info.rs | 7 +++++++
>  src/server/sync.rs               | 1 +
>  2 files changed, 8 insertions(+)
> 
> diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
> index 189ed28ad..b4fabb2cc 100644
> --- a/pbs-datastore/src/backup_info.rs
> +++ b/pbs-datastore/src/backup_info.rs
> @@ -105,6 +105,13 @@ impl BackupGroup {
>          self.full_group_path().exists()
>      }
>  
> +    /// Check if the group is currently marked as trash by checking the presence of the trash
> +    /// marker file in the group's directory
> +    pub fn is_trashed(&self) -> bool {
> +        let path = self.full_group_path().join(TRASH_MARKER_FILENAME);
> +        path.exists()
> +    }

and this hunk moved to patch 2, or the is_trashed helpers extracted into
their own patch?

> +
>      pub fn list_backups(&self, filter: ListBackupFilter) -> Result<Vec<BackupInfo>, Error> {
>          let mut list = vec![];
>  
> diff --git a/src/server/sync.rs b/src/server/sync.rs
> index 3de2ec9a4..ce338fbbe 100644
> --- a/src/server/sync.rs
> +++ b/src/server/sync.rs
> @@ -447,6 +447,7 @@ impl SyncSource for LocalSource {
>              Some(owner),
>          )?
>          .filter_map(Result::ok)
> +        .filter(|backup_group| !backup_group.is_trashed())
>          .map(|backup_group| backup_group.group().clone())
>          .collect::<Vec<pbs_api_types::BackupGroup>>())
>      }
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 



From f.gruenbichler at proxmox.com  Fri May  9 14:27:09 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Fri, 09 May 2025 14:27:09 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 10/21] datastore: mark
 namespace as trash instead of deleting it
In-Reply-To: <20250508130555.494782-11-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-11-c.ebner@proxmox.com>
Message-ID: <1746793154.56dzxckyhs.astroid@yuna.none>

On May 8, 2025 3:05 pm, Christian Ebner wrote:
> As for backup snapshots and groups, mark the namespace as trash
> instead of removing it and the contents right away, if the trash
> should not be bypassed.
> 
> Actual removal of the hirarchical folder structure has to be taken
> care of by the garbage collection.
> 
> In order to avoid races during removal, first mark the namespaces as
> trash pending, mark the snapshots and groups as trash and only after
> rename the pending marker file to the trash marker file. By this,
> concurrent backups can remove the trash pending marker to avoid the
> namespace being trashed.
> 
> On re-creation of a trashed namespace remove the marker file on itself
> and any parent component from deepest to shallowest. As trashing a full
> namespace can also set the trash pending state for recursive namespace
> cleanup, remove encounters of that marker file as well to avoid the
> namespace or its parent being trashed.

this is fairly involved since we don't have locks on namespaces..

should we have them for creation/removal/trashing/untrashing?

I assume those are fairly rare occurrences, I haven't yet analyzed the
interactions here to see whether the two-marker approach is actually
race-free..

OTOH, do we really need to (be able to) trash namespaces?

> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  pbs-datastore/src/datastore.rs | 135 +++++++++++++++++++++++++++++----
>  src/api2/admin/namespace.rs    |   2 +-
>  src/server/pull.rs             |   2 +-
>  3 files changed, 123 insertions(+), 16 deletions(-)
> 
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 023a6a12e..b1d81e199 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -28,7 +28,9 @@ use pbs_api_types::{
>  };
>  use pbs_config::BackupLockGuard;
>  
> -use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, ListBackupFilter, OLD_LOCKING};
> +use crate::backup_info::{
> +    BackupDir, BackupGroup, BackupInfo, ListBackupFilter, OLD_LOCKING, TRASH_MARKER_FILENAME,
> +};
>  use crate::chunk_store::ChunkStore;
>  use crate::dynamic_index::{DynamicIndexReader, DynamicIndexWriter};
>  use crate::fixed_index::{FixedIndexReader, FixedIndexWriter};
> @@ -42,6 +44,8 @@ use crate::DataBlob;
>  static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
>      LazyLock::new(|| Mutex::new(HashMap::new()));
>  
> +const TRASH_PENDING_MARKER_FILENAME: &str = ".pending";
> +
>  /// checks if auth_id is owner, or, if owner is a token, if
>  /// auth_id is the user of the token
>  pub fn check_backup_owner(owner: &Authid, auth_id: &Authid) -> Result<(), Error> {
> @@ -554,6 +558,7 @@ impl DataStore {
>          ns_full_path.push(ns.path());
>  
>          std::fs::create_dir_all(ns_full_path)?;
> +        self.untrash_namespace(&ns)?;
>  
>          Ok(ns)
>      }
> @@ -565,6 +570,34 @@ impl DataStore {
>          path.exists()
>      }
>  
> +    /// Remove the namespace and all it's parent components from the trash by removing the trash or
> +    /// trash-pending marker file for each namespace level from deepest to shallowest. Missing files
> +    /// are ignored.
> +    pub fn untrash_namespace(&self, namespace: &BackupNamespace) -> Result<(), Error> {
> +        let mut namespace = namespace.clone();
> +        while namespace.depth() > 0 {
> +            let mut trash_file_path = self.base_path();
> +            trash_file_path.push(namespace.path());
> +            let mut pending_file_path = trash_file_path.clone();
> +            pending_file_path.push(TRASH_PENDING_MARKER_FILENAME);
> +            if let Err(err) = std::fs::remove_file(&pending_file_path) {
> +                // ignore not found, either not trashed or un-trashed by concurrent operation
> +                if err.kind() != std::io::ErrorKind::NotFound {
> +                    bail!("failed to remove trash-pending file {trash_file_path:?}: {err}");
> +                }
> +            }
> +            trash_file_path.push(TRASH_MARKER_FILENAME);
> +            if let Err(err) = std::fs::remove_file(&trash_file_path) {
> +                // ignore not found, either not trashed or un-trashed by concurrent operation
> +                if err.kind() != std::io::ErrorKind::NotFound {
> +                    bail!("failed to remove trash file {trash_file_path:?}: {err}");
> +                }
> +            }
> +            namespace.pop();
> +        }
> +        Ok(())
> +    }
> +
>      /// Remove all backup groups of a single namespace level but not the namespace itself.
>      ///
>      /// Does *not* descends into child-namespaces and doesn't remoes the namespace itself either.
> @@ -574,6 +607,7 @@ impl DataStore {
>      pub fn remove_namespace_groups(
>          self: &Arc<Self>,
>          ns: &BackupNamespace,
> +        skip_trash: bool,
>      ) -> Result<(bool, BackupGroupDeleteStats), Error> {
>          // FIXME: locking? The single groups/snapshots are already protected, so may not be
>          // necessary (depends on what we all allow to do with namespaces)
> @@ -583,20 +617,22 @@ impl DataStore {
>          let mut stats = BackupGroupDeleteStats::default();
>  
>          for group in self.iter_backup_groups(ns.to_owned())? {
> -            let delete_stats = group?.destroy(true)?;
> +            let delete_stats = group?.destroy(skip_trash)?;
>              stats.add(&delete_stats);
>              removed_all_groups = removed_all_groups && delete_stats.all_removed();
>          }
>  
> -        let base_file = std::fs::File::open(self.base_path())?;
> -        let base_fd = base_file.as_raw_fd();
> -        for ty in BackupType::iter() {
> -            let mut ty_dir = ns.path();
> -            ty_dir.push(ty.to_string());
> -            // best effort only, but we probably should log the error
> -            if let Err(err) = unlinkat(Some(base_fd), &ty_dir, UnlinkatFlags::RemoveDir) {
> -                if err != nix::errno::Errno::ENOENT {
> -                    log::error!("failed to remove backup type {ty} in {ns} - {err}");
> +        if skip_trash {
> +            let base_file = std::fs::File::open(self.base_path())?;
> +            let base_fd = base_file.as_raw_fd();
> +            for ty in BackupType::iter() {
> +                let mut ty_dir = ns.path();
> +                ty_dir.push(ty.to_string());
> +                // best effort only, but we probably should log the error
> +                if let Err(err) = unlinkat(Some(base_fd), &ty_dir, UnlinkatFlags::RemoveDir) {
> +                    if err != nix::errno::Errno::ENOENT {
> +                        log::error!("failed to remove backup type {ty} in {ns} - {err}");
> +                    }
>                  }
>              }
>          }
> @@ -613,6 +649,7 @@ impl DataStore {
>          self: &Arc<Self>,
>          ns: &BackupNamespace,
>          delete_groups: bool,
> +        skip_trash: bool,
>      ) -> Result<(bool, BackupGroupDeleteStats), Error> {
>          let store = self.name();
>          let mut removed_all_requested = true;
> @@ -620,16 +657,68 @@ impl DataStore {
>          if delete_groups {
>              log::info!("removing whole namespace recursively below {store}:/{ns}",);
>              for ns in self.recursive_iter_backup_ns(ns.to_owned(), NamespaceListFilter::Active)? {
> -                let (removed_ns_groups, delete_stats) = self.remove_namespace_groups(&ns?)?;
> +                let namespace = ns?;
> +
> +                if !skip_trash {
> +                    let mut path = self.base_path();
> +                    path.push(namespace.path());
> +                    path.push(TRASH_PENDING_MARKER_FILENAME);
> +                    if let Err(err) = std::fs::File::create(&path) {

pending marker created here iff delete_groups && !skip_trash

> +                        if err.kind() != std::io::ErrorKind::AlreadyExists {
> +                            return Err(err).context("failed to set trash pending marker file");
> +                        }
> +                    }
> +                }
> +
> +                let (removed_ns_groups, delete_stats) =
> +                    self.remove_namespace_groups(&namespace, skip_trash)?;
>                  stats.add(&delete_stats);
>                  removed_all_requested = removed_all_requested && removed_ns_groups;
> +
> +                if !skip_trash && !removed_ns_groups {
> +                    self.untrash_namespace(&namespace)?;
> +                }
>              }
>          } else {
>              log::info!("pruning empty namespace recursively below {store}:/{ns}");
>          }
>  
> -        removed_all_requested =
> -            removed_all_requested && self.destroy_namespace_recursive(ns, delete_groups)?;
> +        if skip_trash {
> +            removed_all_requested =
> +                removed_all_requested && self.destroy_namespace_recursive(ns, delete_groups)?;
> +            return Ok((removed_all_requested, stats));

early return here iff skip_trash

> +        }
> +
> +        let mut children = self
> +            .recursive_iter_backup_ns(ns.to_owned(), NamespaceListFilter::Active)?
> +            .collect::<Result<Vec<BackupNamespace>, Error>>()?;
> +
> +        children.sort_by_key(|b| std::cmp::Reverse(b.depth()));
> +
> +        let mut all_trashed = true;
> +        for ns in children.iter() {
> +            let mut ns_dir = ns.path();
> +            ns_dir.push("ns");
> +
> +            if !ns.is_root() {
> +                let mut path = self.base_path();
> +                path.push(ns.path());
> +
> +                let pending_path = path.join(TRASH_PENDING_MARKER_FILENAME);
> +                path.push(TRASH_MARKER_FILENAME);
> +                if let Err(err) = std::fs::rename(pending_path, path) {

pending marker assumed to exist here iff !skip_trash

> +                    if err.kind() == std::io::ErrorKind::NotFound {
> +                        all_trashed = false;
> +                    } else {
> +                        return Err(err).context("Renaming pending marker to trash marker failed");
> +                    }
> +                }
> +            }
> +        }
> +
> +        if !all_trashed {
> +            bail!("failed to prune namespace, not empty");

wrong error returned as a result..

> +        }
>  
>          Ok((removed_all_requested, stats))
>      }
> @@ -657,6 +746,24 @@ impl DataStore {
>              let _ = unlinkat(Some(base_fd), &ns_dir, UnlinkatFlags::RemoveDir);
>  
>              if !ns.is_root() {
> +                let rel_trash_path = ns.path().join(TRASH_MARKER_FILENAME);
> +                if let Err(err) =
> +                    unlinkat(Some(base_fd), &rel_trash_path, UnlinkatFlags::NoRemoveDir)
> +                {
> +                    if err != nix::errno::Errno::ENOENT {
> +                        bail!("removing the trash file '{rel_trash_path:?}' failed - {err}")
> +                    }
> +                }
> +                let rel_pending_path = ns.path().join(TRASH_PENDING_MARKER_FILENAME);
> +                if let Err(err) =
> +                    unlinkat(Some(base_fd), &rel_pending_path, UnlinkatFlags::NoRemoveDir)
> +                {
> +                    if err != nix::errno::Errno::ENOENT {
> +                        bail!(
> +                            "removing the trash pending file '{rel_pending_path:?}' failed - {err}"
> +                        )
> +                    }
> +                }
>                  match unlinkat(Some(base_fd), &ns.path(), UnlinkatFlags::RemoveDir) {
>                      Ok(()) => log::debug!("removed namespace {ns}"),
>                      Err(nix::errno::Errno::ENOENT) => {
> diff --git a/src/api2/admin/namespace.rs b/src/api2/admin/namespace.rs
> index e5463524a..a12d97883 100644
> --- a/src/api2/admin/namespace.rs
> +++ b/src/api2/admin/namespace.rs
> @@ -166,7 +166,7 @@ pub fn delete_namespace(
>  
>      let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
>  
> -    let (removed_all, stats) = datastore.remove_namespace_recursive(&ns, delete_groups)?;
> +    let (removed_all, stats) = datastore.remove_namespace_recursive(&ns, delete_groups, true)?;
>      if !removed_all {
>          let err_msg = if delete_groups {
>              if datastore.old_locking() {
> diff --git a/src/server/pull.rs b/src/server/pull.rs
> index d3c6fcf6a..a60ccbf10 100644
> --- a/src/server/pull.rs
> +++ b/src/server/pull.rs
> @@ -729,7 +729,7 @@ fn check_and_remove_ns(params: &PullParameters, local_ns: &BackupNamespace) -> R
>      let (removed_all, _delete_stats) = params
>          .target
>          .store
> -        .remove_namespace_recursive(local_ns, true)?;
> +        .remove_namespace_recursive(local_ns, true, true)?;
>  
>      Ok(removed_all)
>  }
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 



From f.gruenbichler at proxmox.com  Fri May  9 14:27:42 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Fri, 09 May 2025 14:27:42 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 06/21] api: tape: check
 trash marker when trying to write snapshot
In-Reply-To: <20250508130555.494782-7-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-7-c.ebner@proxmox.com>
Message-ID: <1746790719.mae7zgxkos.astroid@yuna.none>

On May 8, 2025 3:05 pm, Christian Ebner wrote:
> Since snapshots might be marked as trash, the snapshot directory
> can still be present until cleaned up by garbage collection.
> 
> Therefore, check for the presence of the trash marker after acquiring
> the locked snapshot reader and skip over marked ones.
> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  src/api2/tape/backup.rs | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
> index 923cb7834..17c8bc605 100644
> --- a/src/api2/tape/backup.rs
> +++ b/src/api2/tape/backup.rs
> @@ -574,7 +574,13 @@ fn backup_snapshot(
>      info!("backup snapshot {snapshot_path:?}");
>  
>      let snapshot_reader = match snapshot.locked_reader() {
> -        Ok(reader) => reader,
> +        Ok(reader) => {
> +            if snapshot.is_trashed() {
> +                info!("snapshot {snapshot_path:?} trashed, skipping");

not sure why we log this, but don't log this in other places?

> +                return Ok(SnapshotBackupResult::Ignored);
> +            }
> +            reader
> +        }
>          Err(err) => {
>              if !snapshot.full_path().exists() {
>                  // we got an error and the dir does not exist,
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 



From f.gruenbichler at proxmox.com  Fri May  9 14:27:51 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Fri, 09 May 2025 14:27:51 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 02/21] datastore: mark
 groups as trash on destroy
In-Reply-To: <20250508130555.494782-3-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-3-c.ebner@proxmox.com>
Message-ID: <1746790386.nzu2zcr38q.astroid@yuna.none>

On May 8, 2025 3:05 pm, Christian Ebner wrote:
> In order to implement the trash can functionality, mark all the
> snapshots of the group and the group itself as trash instead of
> deleting them right away. Cleanup of the group is deferred to the
> garbage collection.
> 
> Groups and snapshots are marked by the trash marker file. New backups
> to this group will check for the marker file (see subsequent
> commits), clearing the whole group and all of the snapshots to
> create a new snapshot within that group. Otherwise ownership
> conflicts could arise. This implies that a new backup clears the
> whole trashed group.

this seems a bit surprising.. couldn't we check the new and older owner,
and abort if there's a mismatch, but proceed otherwise?

the implementation of this also doesn't happen in this patch though, so
maybe drop it here in any case - this just implements trashing groups..

> Snapshots already marked as trash within the same backup group will
> be cleared as well when the group is requested to be destroyed with
> skip trash.

this makes sense

> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  pbs-datastore/src/backup_info.rs | 19 ++++++++++++++++---
>  pbs-datastore/src/datastore.rs   |  4 ++--
>  2 files changed, 18 insertions(+), 5 deletions(-)
> 
> diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
> index 76bcd15f5..9ce4cb0f8 100644
> --- a/pbs-datastore/src/backup_info.rs
> +++ b/pbs-datastore/src/backup_info.rs
> @@ -215,7 +215,7 @@ impl BackupGroup {
>      ///
>      /// Returns `BackupGroupDeleteStats`, containing the number of deleted snapshots
>      /// and number of protected snaphsots, which therefore were not removed.
> -    pub fn destroy(&self) -> Result<BackupGroupDeleteStats, Error> {
> +    pub fn destroy(&self, skip_trash: bool) -> Result<BackupGroupDeleteStats, Error> {
>          let _guard = self
>              .lock()
>              .with_context(|| format!("while destroying group '{self:?}'"))?;
> @@ -229,14 +229,20 @@ impl BackupGroup {
>                  delete_stats.increment_protected_snapshots();
>                  continue;
>              }
> -            snap.destroy(false, false)?;
> +            snap.destroy(false, skip_trash)?;
>              delete_stats.increment_removed_snapshots();
>          }
>  
>          // Note: make sure the old locking mechanism isn't used as `remove_dir_all` is not safe in
>          // that case
>          if delete_stats.all_removed() && !*OLD_LOCKING {
> -            self.remove_group_dir()?;
> +            if skip_trash {
> +                self.remove_group_dir()?;
> +            } else {
> +                let path = self.full_group_path().join(TRASH_MARKER_FILENAME);
> +                let _trash_file =
> +                    std::fs::File::create(path).context("failed to set trash file")?;
> +            }
>              delete_stats.increment_removed_groups();
>          }
>  
> @@ -245,6 +251,13 @@ impl BackupGroup {
>  
>      /// Helper function, assumes that no more snapshots are present in the group.
>      fn remove_group_dir(&self) -> Result<(), Error> {
> +        let trash_path = self.full_group_path().join(TRASH_MARKER_FILENAME);
> +        if let Err(err) = std::fs::remove_file(&trash_path) {
> +            if err.kind() != std::io::ErrorKind::NotFound {
> +                bail!("removing the trash file '{trash_path:?}' failed - {err}")
> +            }
> +        }
> +
>          let owner_path = self.store.owner_path(&self.ns, &self.group);
>  
>          std::fs::remove_file(&owner_path).map_err(|err| {
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 6df26e825..e546bc532 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -581,7 +581,7 @@ impl DataStore {
>          let mut stats = BackupGroupDeleteStats::default();
>  
>          for group in self.iter_backup_groups(ns.to_owned())? {
> -            let delete_stats = group?.destroy()?;
> +            let delete_stats = group?.destroy(true)?;
>              stats.add(&delete_stats);
>              removed_all_groups = removed_all_groups && delete_stats.all_removed();
>          }
> @@ -674,7 +674,7 @@ impl DataStore {
>      ) -> Result<BackupGroupDeleteStats, Error> {
>          let backup_group = self.backup_group(ns.clone(), backup_group.clone());
>  
> -        backup_group.destroy()
> +        backup_group.destroy(true)
>      }
>  
>      /// Remove a backup directory including all content
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 



From f.gruenbichler at proxmox.com  Fri May  9 14:27:30 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Fri, 09 May 2025 14:27:30 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 13/21] datastore: recreate
 trashed backup groups if requested
In-Reply-To: <20250508130555.494782-14-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-14-c.ebner@proxmox.com>
Message-ID: <1746792270.885af032mi.astroid@yuna.none>

On May 8, 2025 3:05 pm, Christian Ebner wrote:
> A whole backup group might have been marked as trashed, including all
> of the contained snapshots.
> 
> Since a new backup to that group (even as different user/owner)
> should still work, permanently clear the whole trashed group before
> recreation. This will limit the trash lifetime as now the group is
> not recoverable until next garbage collection.

IMHO this is phrased in a way that makes it hard to parse, and in any
case, such things should go into the docs..

> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  pbs-datastore/src/datastore.rs | 26 ++++++++++++++++++++++++++
>  1 file changed, 26 insertions(+)
> 
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index 4f7766c36..ca05e1bea 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -934,6 +934,32 @@ impl DataStore {
>                  let guard = backup_group.lock().with_context(|| {
>                      format!("while creating locked backup group '{backup_group:?}'")
>                  })?;
> +                if backup_group.is_trashed() {
> +                    info!("clear trashed backup group {full_path:?}");

I think we should only do this if the new and old owner are not
identical..

> +                    let dir_entries = std::fs::read_dir(&full_path).context(
> +                        "failed to read directory contents during cleanup of trashed group",
> +                    )?;
> +                    for entry in dir_entries {
> +                        let entry = entry.context(
> +                            "failed to read directory entry during clenup of trashed group",
> +                        )?;
> +                        let file_type = entry.file_type().context(
> +                            "failed to get entry file type during clenup of trashed group",
> +                        )?;
> +                        if file_type.is_dir() {
> +                            std::fs::remove_dir_all(entry.path())
> +                                .context("failed to remove directory entry during clenup of trashed snapshot")?;
> +                        } else {
> +                            std::fs::remove_file(entry.path())
> +                                .context("failed to remove directory entry during clenup of trashed snapshot")?;
> +                        }
> +                    }
> +                    self.set_owner(ns, backup_group.group(), auth_id, false)?;
> +                    let owner = self.get_owner(ns, backup_group.group())?; // just to be sure

sure about that? we are holding a lock here, nobody is allowed to change
the owner but us..

> +                    self.untrash_namespace(ns)?;
> +                    return Ok((owner, guard));
> +                }
> +
>                  let owner = self.get_owner(ns, backup_group.group())?; // just to be sure
>                  Ok((owner, guard))
>              }
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 



From f.gruenbichler at proxmox.com  Fri May  9 14:27:36 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Fri, 09 May 2025 14:27:36 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 12/21] datastore: clear
 trashed snapshot dir if re-creation requested
In-Reply-To: <20250508130555.494782-13-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-13-c.ebner@proxmox.com>
Message-ID: <1746791755.kc17x5cte7.astroid@yuna.none>

On May 8, 2025 3:05 pm, Christian Ebner wrote:
> If a previously trashed snapshot has been requested for re-creation
> (e.g. by a sync job in push direction), drop the contents of the
> currently trashed snapshot.
> The snapshot directory itself is already locked at that point, either
> by the old locking mechanism acting on the directory itself or by the
> new locking mechanism. Therefore, concurrent operations can be
> excluded.
> 
> For the call site this acts as if the snapshot directory has been
> newly created.
> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  pbs-datastore/src/datastore.rs | 29 ++++++++++++++++++++++++++++-
>  1 file changed, 28 insertions(+), 1 deletion(-)
> 
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index ffc6a7039..4f7766c36 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -951,8 +951,9 @@ impl DataStore {
>      ) -> Result<(PathBuf, bool, BackupLockGuard), Error> {
>          let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
>          let relative_path = backup_dir.relative_path();
> +        let full_path = backup_dir.full_path();
>  
> -        match std::fs::create_dir(backup_dir.full_path()) {
> +        match std::fs::create_dir(&full_path) {
>              Ok(_) => {
>                  let guard = backup_dir.lock().with_context(|| {
>                      format!("while creating new locked snapshot '{backup_dir:?}'")
> @@ -963,6 +964,32 @@ impl DataStore {
>                  let guard = backup_dir
>                      .lock()
>                      .with_context(|| format!("while creating locked snapshot '{backup_dir:?}'"))?;
> +
> +                if backup_dir.is_trashed() {
> +                    info!("clear trashed backup snapshot {full_path:?}");
> +                    let dir_entries = std::fs::read_dir(&full_path).context(
> +                        "failed to read directory contents during cleanup of trashed snapshot",
> +                    )?;
> +                    for entry in dir_entries {
> +                        let entry = entry.context(
> +                            "failed to read directory entry during clenup of trashed snapshot",
> +                        )?;
> +                        // Only expect regular file entries
> +                        std::fs::remove_file(entry.path()).context(
> +                            "failed to remove directory entry during clenup of trashed snapshot",
> +                        )?;
> +                    }
> +                    let group = BackupGroup::from(backup_dir);
> +                    let group_trash_file = group.full_group_path().join(TRASH_MARKER_FILENAME);
> +                    if let Err(err) = std::fs::remove_file(&group_trash_file) {
> +                        if err.kind() != std::io::ErrorKind::NotFound {
> +                            bail!("failed to remove group trash file of trashed snapshot");
> +                        }
> +                    }

this shouldn't be possible to hit, right? as creating a backup dir
entails first creating the backup group (guarded by the group lock), and
that would already clear the group's trash marker..

> +                    self.untrash_namespace(ns)?;
> +                    return Ok((relative_path, true, guard));
> +                }
> +
>                  Ok((relative_path, false, guard))
>              }
>              Err(e) => Err(e.into()),
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 



From f.gruenbichler at proxmox.com  Fri May  9 14:27:48 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Fri, 09 May 2025 14:27:48 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 03/21] datastore: allow
 filtering of backups by their trash status
In-Reply-To: <20250508130555.494782-4-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-4-c.ebner@proxmox.com>
Message-ID: <1746790489.fw746s24j7.astroid@yuna.none>

On May 8, 2025 3:05 pm, Christian Ebner wrote:
> Extends the BackupGroup::list_backups method by an enum parameter to
> filter backup snapshots based on their trash status.
> 
> This allows to reuse the same logic for listing active, trashed or
> all snapshots.
> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  pbs-datastore/src/backup_info.rs | 33 +++++++++++++++++++++++++++++---
>  pbs-datastore/src/datastore.rs   |  4 ++--
>  src/api2/admin/datastore.rs      | 10 +++++-----
>  src/api2/tape/backup.rs          |  4 ++--
>  src/backup/verify.rs             |  4 ++--
>  src/server/prune_job.rs          |  3 ++-
>  src/server/pull.rs               |  3 ++-
>  7 files changed, 45 insertions(+), 16 deletions(-)
> 
> diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
> index 9ce4cb0f8..a8c864ac8 100644
> --- a/pbs-datastore/src/backup_info.rs
> +++ b/pbs-datastore/src/backup_info.rs
> @@ -52,6 +52,12 @@ impl fmt::Debug for BackupGroup {
>      }
>  }
>  
> +pub enum ListBackupFilter {
> +    Active,

active sounds like there's currently a backup going on..

> +    All,
> +    Trashed,
> +}
> +
>  impl BackupGroup {
>      pub(crate) fn new(
>          store: Arc<DataStore>,
> @@ -99,7 +105,7 @@ impl BackupGroup {
>          self.full_group_path().exists()
>      }
>  
> -    pub fn list_backups(&self) -> Result<Vec<BackupInfo>, Error> {
> +    pub fn list_backups(&self, filter: ListBackupFilter) -> Result<Vec<BackupInfo>, Error> {
>          let mut list = vec![];
>  
>          let path = self.full_group_path();
> @@ -117,6 +123,19 @@ impl BackupGroup {
>                  let files = list_backup_files(l2_fd, backup_time)?;
>  
>                  let protected = backup_dir.is_protected();
> +                match filter {
> +                    ListBackupFilter::All => (),
> +                    ListBackupFilter::Trashed => {
> +                        if !backup_dir.is_trashed() {
> +                            return Ok(());
> +                        }
> +                    }
> +                    ListBackupFilter::Active => {
> +                        if backup_dir.is_trashed() {
> +                            return Ok(());
> +                        }
> +                    }
> +                }
>  
>                  list.push(BackupInfo {
>                      backup_dir,
> @@ -132,7 +151,7 @@ impl BackupGroup {
>  
>      /// Finds the latest backup inside a backup group
>      pub fn last_backup(&self, only_finished: bool) -> Result<Option<BackupInfo>, Error> {
> -        let backups = self.list_backups()?;
> +        let backups = self.list_backups(ListBackupFilter::Active)?;
>          Ok(backups
>              .into_iter()
>              .filter(|item| !only_finished || item.is_finished())
> @@ -480,6 +499,11 @@ impl BackupDir {
>          path.exists()
>      }
>  
> +    pub fn is_trashed(&self) -> bool {
> +        let path = self.full_path().join(TRASH_MARKER_FILENAME);
> +        path.exists()
> +    }
> +
>      pub fn backup_time_to_string(backup_time: i64) -> Result<String, Error> {
>          // fixme: can this fail? (avoid unwrap)
>          proxmox_time::epoch_to_rfc3339_utc(backup_time)
> @@ -637,7 +661,10 @@ impl BackupDir {
>              //
>              // Do not error out, as we have already removed the snapshot, there is nothing a user could
>              // do to rectify the situation.
> -            if guard.is_ok() && group.list_backups()?.is_empty() && !*OLD_LOCKING {
> +            if guard.is_ok()
> +                && group.list_backups(ListBackupFilter::All)?.is_empty()
> +                && !*OLD_LOCKING
> +            {
>                  group.remove_group_dir()?;
>              } else if let Err(err) = guard {
>                  log::debug!("{err:#}");
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index e546bc532..867324380 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -28,7 +28,7 @@ use pbs_api_types::{
>  };
>  use pbs_config::BackupLockGuard;
>  
> -use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, OLD_LOCKING};
> +use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, ListBackupFilter, OLD_LOCKING};
>  use crate::chunk_store::ChunkStore;
>  use crate::dynamic_index::{DynamicIndexReader, DynamicIndexWriter};
>  use crate::fixed_index::{FixedIndexReader, FixedIndexWriter};
> @@ -1158,7 +1158,7 @@ impl DataStore {
>                          _ => bail!("exhausted retries and unexpected counter overrun"),
>                      };
>  
> -                    let mut snapshots = match group.list_backups() {
> +                    let mut snapshots = match group.list_backups(ListBackupFilter::All) {
>                          Ok(snapshots) => snapshots,
>                          Err(err) => {
>                              if group.exists() {
> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> index aafd1bbd7..133a6d658 100644
> --- a/src/api2/admin/datastore.rs
> +++ b/src/api2/admin/datastore.rs
> @@ -51,7 +51,7 @@ use pbs_api_types::{
>  };
>  use pbs_client::pxar::{create_tar, create_zip};
>  use pbs_config::CachedUserInfo;
> -use pbs_datastore::backup_info::BackupInfo;
> +use pbs_datastore::backup_info::{BackupInfo, ListBackupFilter};
>  use pbs_datastore::cached_chunk_reader::CachedChunkReader;
>  use pbs_datastore::catalog::{ArchiveEntry, CatalogReader};
>  use pbs_datastore::data_blob::DataBlob;
> @@ -223,7 +223,7 @@ pub fn list_groups(
>                  return Ok(group_info);
>              }
>  
> -            let snapshots = match group.list_backups() {
> +            let snapshots = match group.list_backups(ListBackupFilter::Active) {
>                  Ok(snapshots) => snapshots,
>                  Err(_) => return Ok(group_info),
>              };
> @@ -624,7 +624,7 @@ unsafe fn list_snapshots_blocking(
>              return Ok(snapshots);
>          }
>  
> -        let group_backups = group.list_backups()?;
> +        let group_backups = group.list_backups(ListBackupFilter::Active)?;
>  
>          snapshots.extend(
>              group_backups
> @@ -657,7 +657,7 @@ async fn get_snapshots_count(
>                  Ok(group) => group,
>                  Err(_) => return Ok(counts), // TODO: add this as error counts?
>              };
> -            let snapshot_count = group.list_backups()?.len() as u64;
> +            let snapshot_count = group.list_backups(ListBackupFilter::Active)?.len() as u64;
>  
>              // only include groups with snapshots, counting/displaying empty groups can confuse
>              if snapshot_count > 0 {
> @@ -1042,7 +1042,7 @@ pub fn prune(
>      }
>      let mut prune_result: Vec<PruneResult> = Vec::new();
>  
> -    let list = group.list_backups()?;
> +    let list = group.list_backups(ListBackupFilter::Active)?;
>  
>      let mut prune_info = compute_prune_info(list, &keep_options)?;
>  
> diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
> index 31293a9a9..923cb7834 100644
> --- a/src/api2/tape/backup.rs
> +++ b/src/api2/tape/backup.rs
> @@ -17,7 +17,7 @@ use pbs_api_types::{
>  };
>  
>  use pbs_config::CachedUserInfo;
> -use pbs_datastore::backup_info::{BackupDir, BackupInfo};
> +use pbs_datastore::backup_info::{BackupDir, BackupInfo, ListBackupFilter};
>  use pbs_datastore::{DataStore, StoreProgress};
>  
>  use crate::tape::TapeNotificationMode;
> @@ -433,7 +433,7 @@ fn backup_worker(
>          progress.done_snapshots = 0;
>          progress.group_snapshots = 0;
>  
> -        let snapshot_list = group.list_backups()?;
> +        let snapshot_list = group.list_backups(ListBackupFilter::Active)?;
>  
>          // filter out unfinished backups
>          let mut snapshot_list: Vec<_> = snapshot_list
> diff --git a/src/backup/verify.rs b/src/backup/verify.rs
> index 3d2cba8ac..1b5e8564b 100644
> --- a/src/backup/verify.rs
> +++ b/src/backup/verify.rs
> @@ -14,7 +14,7 @@ use pbs_api_types::{
>      CryptMode, SnapshotVerifyState, VerifyState, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_VERIFY,
>      UPID,
>  };
> -use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo};
> +use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo, ListBackupFilter};
>  use pbs_datastore::index::IndexFile;
>  use pbs_datastore::manifest::{BackupManifest, FileInfo};
>  use pbs_datastore::{DataBlob, DataStore, StoreProgress};
> @@ -411,7 +411,7 @@ pub fn verify_backup_group(
>      filter: Option<&dyn Fn(&BackupManifest) -> bool>,
>  ) -> Result<Vec<String>, Error> {
>      let mut errors = Vec::new();
> -    let mut list = match group.list_backups() {
> +    let mut list = match group.list_backups(ListBackupFilter::Active) {
>          Ok(list) => list,
>          Err(err) => {
>              info!(
> diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
> index 1c86647a0..596afe086 100644
> --- a/src/server/prune_job.rs
> +++ b/src/server/prune_job.rs
> @@ -1,6 +1,7 @@
>  use std::sync::Arc;
>  
>  use anyhow::Error;
> +use pbs_datastore::backup_info::ListBackupFilter;
>  use tracing::{info, warn};
>  
>  use pbs_api_types::{
> @@ -54,7 +55,7 @@ pub fn prune_datastore(
>      )? {
>          let group = group?;
>          let ns = group.backup_ns();
> -        let list = group.list_backups()?;
> +        let list = group.list_backups(ListBackupFilter::Active)?;
>  
>          let mut prune_info = compute_prune_info(list, &prune_options.keep)?;
>          prune_info.reverse(); // delete older snapshots first
> diff --git a/src/server/pull.rs b/src/server/pull.rs
> index b1724c142..50d7b0806 100644
> --- a/src/server/pull.rs
> +++ b/src/server/pull.rs
> @@ -7,6 +7,7 @@ use std::sync::{Arc, Mutex};
>  use std::time::SystemTime;
>  
>  use anyhow::{bail, format_err, Error};
> +use pbs_datastore::backup_info::ListBackupFilter;
>  use proxmox_human_byte::HumanByte;
>  use tracing::info;
>  
> @@ -660,7 +661,7 @@ async fn pull_group(
>              .target
>              .store
>              .backup_group(target_ns.clone(), group.clone());
> -        let local_list = group.list_backups()?;
> +        let local_list = group.list_backups(ListBackupFilter::Active)?;
>          for info in local_list {
>              let snapshot = info.backup_dir;
>              if source_snapshots.contains(&snapshot.backup_time()) {
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 



From f.gruenbichler at proxmox.com  Fri May  9 14:27:59 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Fri, 09 May 2025 14:27:59 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 01/21] datastore/api: mark
 snapshots as trash on destroy
In-Reply-To: <20250508130555.494782-2-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-2-c.ebner@proxmox.com>
Message-ID: <1746790319.z1qzufxguu.astroid@yuna.none>

On May 8, 2025 3:05 pm, Christian Ebner wrote:
> In order to implement the trash can functionality, mark snapshots
> as trash instead of removing them by default. However, provide a
> `skip-trash` flag to opt-out and destroy the snapshot including it's
> contents immediately.
> 
> Trashed snapshots are marked by creating a `.trashed` marker file
> inside the snapshot folder. Actual removal of the snapshot will be
> deferred to the garbage collection task.
> 
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
>  pbs-datastore/src/backup_info.rs | 66 ++++++++++++++++++--------------
>  pbs-datastore/src/datastore.rs   |  2 +-
>  src/api2/admin/datastore.rs      | 18 ++++++++-
>  3 files changed, 55 insertions(+), 31 deletions(-)
> 
> diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
> index d4732fdd9..76bcd15f5 100644
> --- a/pbs-datastore/src/backup_info.rs
> +++ b/pbs-datastore/src/backup_info.rs
> @@ -21,6 +21,7 @@ use crate::manifest::{BackupManifest, MANIFEST_LOCK_NAME};
>  use crate::{DataBlob, DataStore};
>  
>  pub const DATASTORE_LOCKS_DIR: &str = "/run/proxmox-backup/locks";
> +pub const TRASH_MARKER_FILENAME: &str = ".trashed";
>  
>  // TODO: Remove with PBS 5
>  // Note: The `expect()` call here will only happen if we can neither confirm nor deny the existence
> @@ -228,7 +229,7 @@ impl BackupGroup {
>                  delete_stats.increment_protected_snapshots();
>                  continue;
>              }
> -            snap.destroy(false)?;
> +            snap.destroy(false, false)?;
>              delete_stats.increment_removed_snapshots();
>          }
>  
> @@ -575,7 +576,8 @@ impl BackupDir {
>      /// Destroy the whole snapshot, bails if it's protected
>      ///
>      /// Setting `force` to true skips locking and thus ignores if the backup is currently in use.
> -    pub fn destroy(&self, force: bool) -> Result<(), Error> {
> +    /// Setting `skip_trash` to true will remove the snapshot instead of marking it as trash.
> +    pub fn destroy(&self, force: bool, skip_trash: bool) -> Result<(), Error> {
>          let (_guard, _manifest_guard);
>          if !force {
>              _guard = self
> @@ -588,37 +590,45 @@ impl BackupDir {
>              bail!("cannot remove protected snapshot"); // use special error type?
>          }
>  
> -        let full_path = self.full_path();
> -        log::info!("removing backup snapshot {:?}", full_path);
> -        std::fs::remove_dir_all(&full_path).map_err(|err| {
> -            format_err!("removing backup snapshot {:?} failed - {}", full_path, err,)
> -        })?;
> +        let mut full_path = self.full_path();
> +        log::info!("removing backup snapshot {full_path:?}");
> +        if skip_trash {
> +            std::fs::remove_dir_all(&full_path).map_err(|err| {
> +                format_err!("removing backup snapshot {full_path:?} failed - {err}")
> +            })?;
> +        } else {
> +            full_path.push(TRASH_MARKER_FILENAME);
> +            let _trash_file =
> +                std::fs::File::create(full_path).context("failed to set trash file")?;
> +        }
>  
>          // remove no longer needed lock files
>          let _ = std::fs::remove_file(self.manifest_lock_path()); // ignore errors
>          let _ = std::fs::remove_file(self.lock_path()); // ignore errors
>  
> -        let group = BackupGroup::from(self);
> -        let guard = group.lock().with_context(|| {
> -            format!("while checking if group '{group:?}' is empty during snapshot destruction")
> -        });
> -
> -        // Only remove the group if all of the following is true:
> -        //
> -        // - we can lock it: if we can't lock the group, it is still in use (either by another
> -        //   backup process or a parent caller (who needs to take care that empty groups are
> -        //   removed themselves).
> -        // - it is now empty: if the group isn't empty, removing it will fail (to avoid removing
> -        //   backups that might still be used).
> -        // - the new locking mechanism is used: if the old mechanism is used, a group removal here
> -        //   could lead to a race condition.
> -        //
> -        // Do not error out, as we have already removed the snapshot, there is nothing a user could
> -        // do to rectify the situation.
> -        if guard.is_ok() && group.list_backups()?.is_empty() && !*OLD_LOCKING {
> -            group.remove_group_dir()?;
> -        } else if let Err(err) = guard {
> -            log::debug!("{err:#}");
> +        if skip_trash {
> +            let group = BackupGroup::from(self);
> +            let guard = group.lock().with_context(|| {
> +                format!("while checking if group '{group:?}' is empty during snapshot destruction")
> +            });
> +
> +            // Only remove the group if all of the following is true:
> +            //
> +            // - we can lock it: if we can't lock the group, it is still in use (either by another
> +            //   backup process or a parent caller (who needs to take care that empty groups are
> +            //   removed themselves).
> +            // - it is now empty: if the group isn't empty, removing it will fail (to avoid removing
> +            //   backups that might still be used).
> +            // - the new locking mechanism is used: if the old mechanism is used, a group removal here
> +            //   could lead to a race condition.
> +            //
> +            // Do not error out, as we have already removed the snapshot, there is nothing a user could
> +            // do to rectify the situation.
> +            if guard.is_ok() && group.list_backups()?.is_empty() && !*OLD_LOCKING {
> +                group.remove_group_dir()?;
> +            } else if let Err(err) = guard {
> +                log::debug!("{err:#}");
> +            }
>          }
>  
>          Ok(())
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index cbf78ecb6..6df26e825 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -686,7 +686,7 @@ impl DataStore {
>      ) -> Result<(), Error> {
>          let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
>  
> -        backup_dir.destroy(force)
> +        backup_dir.destroy(force, true)
>      }
>  
>      /// Returns the time of the last successful backup
> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> index 392494488..aafd1bbd7 100644
> --- a/src/api2/admin/datastore.rs
> +++ b/src/api2/admin/datastore.rs
> @@ -402,6 +402,12 @@ pub async fn list_snapshot_files(
>                  type: pbs_api_types::BackupDir,
>                  flatten: true,
>              },
> +            "skip-trash": {
> +                type: bool,
> +                optional: true,
> +                default: false,

should this default to false in the backend? wouldn't that be a bit
surprising for scripted access? or is this 4.0 material anyway? ;)

> +                description: "Immediately remove the snapshot, not marking it as trash.",
> +            },
>          },
>      },
>      access: {
> @@ -415,6 +421,7 @@ pub async fn delete_snapshot(
>      store: String,
>      ns: Option<BackupNamespace>,
>      backup_dir: pbs_api_types::BackupDir,
> +    skip_trash: bool,
>      _info: &ApiMethod,
>      rpcenv: &mut dyn RpcEnvironment,
>  ) -> Result<Value, Error> {
> @@ -435,7 +442,7 @@ pub async fn delete_snapshot(
>  
>          let snapshot = datastore.backup_dir(ns, backup_dir)?;
>  
> -        snapshot.destroy(false)?;
> +        snapshot.destroy(false, skip_trash)?;
>  
>          Ok(Value::Null)
>      })
> @@ -979,6 +986,12 @@ pub fn verify(
>                  optional: true,
>                  description: "Spins up an asynchronous task that does the work.",
>              },
> +            "skip-trash": {
> +                type: bool,
> +                optional: true,
> +                default: false,
> +                description: "Immediately remove the snapshot, not marking it as trash.",
> +            },
>          },
>      },
>      returns: pbs_api_types::ADMIN_DATASTORE_PRUNE_RETURN_TYPE,
> @@ -995,6 +1008,7 @@ pub fn prune(
>      keep_options: KeepOptions,
>      store: String,
>      ns: Option<BackupNamespace>,
> +    skip_trash: bool,
>      param: Value,
>      rpcenv: &mut dyn RpcEnvironment,
>  ) -> Result<Value, Error> {
> @@ -1098,7 +1112,7 @@ pub fn prune(
>              });
>  
>              if !keep {
> -                if let Err(err) = backup_dir.destroy(false) {
> +                if let Err(err) = backup_dir.destroy(false, skip_trash) {
>                      warn!(
>                          "failed to remove dir {:?}: {}",
>                          backup_dir.relative_path(),
> -- 
> 2.39.5
> 
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 
> 



From c.ebner at proxmox.com  Fri May  9 14:59:16 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 9 May 2025 14:59:16 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 18/21] api: admin: implement
 endpoints to restore trashed contents
In-Reply-To: <1746793013.k8qdvp27bh.astroid@yuna.none>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-19-c.ebner@proxmox.com>
 <1746793013.k8qdvp27bh.astroid@yuna.none>
Message-ID: <39b85c49-8a09-4702-8a76-2d7bdbc500e5@proxmox.com>

Thanks for feedback, will have a closer look next week.

Allow me two quick questions inline though...

On 5/9/25 14:27, Fabian Gr?nbichler wrote:
> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>> Implements the api endpoints to restore trashed contents contained
>> within namespaces, backup groups or individual snapshots.
>>
>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>> ---
>>   src/api2/admin/datastore.rs | 173 +++++++++++++++++++++++++++++++++++-
>>   1 file changed, 172 insertions(+), 1 deletion(-)
>>
>> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
>> index cbd24c729..eb033c3fc 100644
>> --- a/src/api2/admin/datastore.rs
>> +++ b/src/api2/admin/datastore.rs
>> @@ -51,7 +51,7 @@ use pbs_api_types::{
>>   };
>>   use pbs_client::pxar::{create_tar, create_zip};
>>   use pbs_config::CachedUserInfo;
>> -use pbs_datastore::backup_info::{BackupInfo, ListBackupFilter};
>> +use pbs_datastore::backup_info::{BackupInfo, ListBackupFilter, TRASH_MARKER_FILENAME};
>>   use pbs_datastore::cached_chunk_reader::CachedChunkReader;
>>   use pbs_datastore::catalog::{ArchiveEntry, CatalogReader};
>>   use pbs_datastore::data_blob::DataBlob;
>> @@ -2727,6 +2727,165 @@ pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<V
>>       Ok(json!(upid))
>>   }
>>   
>> +#[api(
>> +    input: {
>> +        properties: {
>> +            store: { schema: DATASTORE_SCHEMA },
>> +            ns: { type: BackupNamespace, },
>> +        },
>> +    },
>> +    access: {
>> +        permission: &Permission::Anybody,
>> +        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
>> +            or DATASTORE_BACKUP and being the owner of the group",
>> +    },
>> +)]
>> +/// Recover trashed contents of a namespace.
>> +pub fn recover_namespace(
>> +    store: String,
>> +    ns: BackupNamespace,
>> +    rpcenv: &mut dyn RpcEnvironment,
>> +) -> Result<(), Error> {
>> +    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
>> +    let limited = check_ns_privs_full(
>> +        &store,
>> +        &ns,
>> +        &auth_id,
>> +        PRIV_DATASTORE_MODIFY,
>> +        PRIV_DATASTORE_BACKUP,
>> +    )?;
>> +
>> +    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
>> +
>> +    for backup_group in datastore.iter_backup_groups(ns.clone())? {
>> +        let backup_group = backup_group?;
>> +        if limited {
>> +            let owner = datastore.get_owner(&ns, backup_group.group())?;
>> +            if check_backup_owner(&owner, &auth_id).is_err() {
>> +                continue;
>> +            }
>> +        }
>> +        do_recover_group(&backup_group)?;
>> +    }
>> +
>> +    Ok(())
>> +}
>> +
>> +#[api(
>> +    input: {
>> +        properties: {
>> +            store: { schema: DATASTORE_SCHEMA },
>> +            group: {
>> +                type: pbs_api_types::BackupGroup,
>> +                flatten: true,
>> +            },
>> +            ns: {
>> +                type: BackupNamespace,
>> +                optional: true,
>> +            },
>> +        },
>> +    },
>> +    access: {
>> +        permission: &Permission::Anybody,
>> +        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
>> +            or DATASTORE_BACKUP and being the owner of the group",
>> +    },
>> +)]
>> +/// Recover trashed contents of a backup group.
>> +pub fn recover_group(
>> +    store: String,
>> +    group: pbs_api_types::BackupGroup,
>> +    ns: Option<BackupNamespace>,
>> +    rpcenv: &mut dyn RpcEnvironment,
>> +) -> Result<(), Error> {
>> +    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
>> +    let ns = ns.unwrap_or_default();
>> +    let datastore = check_privs_and_load_store(
>> +        &store,
>> +        &ns,
>> +        &auth_id,
>> +        PRIV_DATASTORE_MODIFY,
>> +        PRIV_DATASTORE_BACKUP,
>> +        Some(Operation::Write),
>> +        &group,
>> +    )?;
>> +
>> +    let backup_group = datastore.backup_group(ns, group);
>> +    do_recover_group(&backup_group)?;
>> +
>> +    Ok(())
>> +}
>> +
>> +fn do_recover_group(backup_group: &BackupGroup) -> Result<(), Error> {
> 
> missing locking for the group?

Not sure about that one. After all the group is trashed at least as long 
as all the snapshots are trashed. And GC will only ever clean up the 
group folder if the trash marker is not set. So I do not see a reason 
why this should be locked.

> 
>> +    let trashed_snapshots = backup_group.list_backups(ListBackupFilter::Trashed)?;
>> +    for snapshot in trashed_snapshots {
>> +        do_recover_snapshot(&snapshot.backup_dir)?;
>> +    }
>> +
>> +    let group_trash_path = backup_group.full_group_path().join(TRASH_MARKER_FILENAME);
>> +    if let Err(err) = std::fs::remove_file(&group_trash_path) {
>> +        if err.kind() != std::io::ErrorKind::NotFound {
>> +            bail!("failed to remove group trash file {group_trash_path:?} - {err}");
>> +        }
>> +    }
>> +    Ok(())
>> +}
>> +
>> +#[api(
>> +    input: {
>> +        properties: {
>> +            store: { schema: DATASTORE_SCHEMA },
>> +            backup_dir: {
>> +                type: pbs_api_types::BackupDir,
>> +                flatten: true,
>> +            },
>> +            ns: {
>> +                type: BackupNamespace,
>> +                optional: true,
>> +            },
>> +        },
>> +    },
>> +    access: {
>> +        permission: &Permission::Anybody,
>> +        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
>> +            or DATASTORE_BACKUP and being the owner of the group",
>> +    },
>> +)]
>> +/// Recover trashed contents of a backup snapshot.
>> +pub fn recover_snapshot(
>> +    store: String,
>> +    backup_dir: pbs_api_types::BackupDir,
>> +    ns: Option<BackupNamespace>,
>> +    rpcenv: &mut dyn RpcEnvironment,
>> +) -> Result<(), Error> {
>> +    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
>> +    let ns = ns.unwrap_or_default();
>> +    let datastore = check_privs_and_load_store(
>> +        &store,
>> +        &ns,
>> +        &auth_id,
>> +        PRIV_DATASTORE_MODIFY,
>> +        PRIV_DATASTORE_BACKUP,
>> +        Some(Operation::Write),
>> +        &backup_dir.group,
>> +    )?;
>> +
>> +    let snapshot = datastore.backup_dir(ns, backup_dir)?;
>> +    do_recover_snapshot(&snapshot)?;
>> +
>> +    Ok(())
>> +}
>> +
>> +fn do_recover_snapshot(snapshot_dir: &BackupDir) -> Result<(), Error> {
> 
> missing locking for the snapshot?

Why? remove_file() should be atomic?

> 
>> +    let trash_path = snapshot_dir.full_path().join(TRASH_MARKER_FILENAME);
>> +    if let Err(err) = std::fs::remove_file(&trash_path) {
>> +        if err.kind() != std::io::ErrorKind::NotFound {
>> +            bail!("failed to remove trash file {trash_path:?} - {err}");
>> +        }
>> +    }
>> +    Ok(())
>> +}
>> +
>>   #[sortable]
>>   const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
>>       (
>> @@ -2792,6 +2951,18 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
>>           "pxar-file-download",
>>           &Router::new().download(&API_METHOD_PXAR_FILE_DOWNLOAD),
>>       ),
>> +    (
>> +        "recover-group",
>> +        &Router::new().post(&API_METHOD_RECOVER_GROUP),
> 
> I am not sure whether those should be POST or PUT, they are modifying an
> existing (trashed) group/snapshot/.. after all?
> 
>> +    ),
>> +    (
>> +        "recover-namespace",
>> +        &Router::new().post(&API_METHOD_RECOVER_NAMESPACE),
>> +    ),
>> +    (
>> +        "recover-snapshot",
>> +        &Router::new().post(&API_METHOD_RECOVER_SNAPSHOT),
>> +    ),
>>       ("rrd", &Router::new().get(&API_METHOD_GET_RRD_STATS)),
>>       (
>>           "snapshots",
>> -- 
>> 2.39.5
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel at lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 




From c.ebner at proxmox.com  Mon May 12 09:47:59 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 12 May 2025 09:47:59 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 10/21] datastore: mark
 namespace as trash instead of deleting it
In-Reply-To: <1746793154.56dzxckyhs.astroid@yuna.none>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-11-c.ebner@proxmox.com>
 <1746793154.56dzxckyhs.astroid@yuna.none>
Message-ID: <8c214e1c-9878-41e6-a988-706eab6601a1@proxmox.com>

On 5/9/25 14:27, Fabian Gr?nbichler wrote:
> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>> As for backup snapshots and groups, mark the namespace as trash
>> instead of removing it and the contents right away, if the trash
>> should not be bypassed.
>>
>> Actual removal of the hirarchical folder structure has to be taken
>> care of by the garbage collection.
>>
>> In order to avoid races during removal, first mark the namespaces as
>> trash pending, mark the snapshots and groups as trash and only after
>> rename the pending marker file to the trash marker file. By this,
>> concurrent backups can remove the trash pending marker to avoid the
>> namespace being trashed.
>>
>> On re-creation of a trashed namespace remove the marker file on itself
>> and any parent component from deepest to shallowest. As trashing a full
>> namespace can also set the trash pending state for recursive namespace
>> cleanup, remove encounters of that marker file as well to avoid the
>> namespace or its parent being trashed.
> 
> this is fairly involved since we don't have locks on namespaces..
> 
> should we have them for creation/removal/trashing/untrashing?

That is what I did try to avoid at all cost with the two-marker 
approach, as locking the namespace might be rather invasive. But if that 
does not work out as intended, I see no other way as to add exclusive 
locking for namespaces as well, yes.

> 
> I assume those are fairly rare occurrences, I haven't yet analyzed the
> interactions here to see whether the two-marker approach is actually
> race-free..
> 
> OTOH, do we really need to (be able to) trash namespaces?

Yes, I think we do need that as well since the datastore's hierarchy 
should remain in place, and the namespace iterator requires a way to 
distinguish between a namespace which has been trashed/deleted and a 
namespace which has not, but might contain trashed items. Otherwise a 
user requesting to forget a namespace, still sees the (empty as only 
trashed contents) namespace tree after the operation. Which would be 
rather unexpected?

> 
>>
>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>> ---
>>   pbs-datastore/src/datastore.rs | 135 +++++++++++++++++++++++++++++----
>>   src/api2/admin/namespace.rs    |   2 +-
>>   src/server/pull.rs             |   2 +-
>>   3 files changed, 123 insertions(+), 16 deletions(-)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 023a6a12e..b1d81e199 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -28,7 +28,9 @@ use pbs_api_types::{
>>   };
>>   use pbs_config::BackupLockGuard;
>>   
>> -use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, ListBackupFilter, OLD_LOCKING};
>> +use crate::backup_info::{
>> +    BackupDir, BackupGroup, BackupInfo, ListBackupFilter, OLD_LOCKING, TRASH_MARKER_FILENAME,
>> +};
>>   use crate::chunk_store::ChunkStore;
>>   use crate::dynamic_index::{DynamicIndexReader, DynamicIndexWriter};
>>   use crate::fixed_index::{FixedIndexReader, FixedIndexWriter};
>> @@ -42,6 +44,8 @@ use crate::DataBlob;
>>   static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
>>       LazyLock::new(|| Mutex::new(HashMap::new()));
>>   
>> +const TRASH_PENDING_MARKER_FILENAME: &str = ".pending";
>> +
>>   /// checks if auth_id is owner, or, if owner is a token, if
>>   /// auth_id is the user of the token
>>   pub fn check_backup_owner(owner: &Authid, auth_id: &Authid) -> Result<(), Error> {
>> @@ -554,6 +558,7 @@ impl DataStore {
>>           ns_full_path.push(ns.path());
>>   
>>           std::fs::create_dir_all(ns_full_path)?;
>> +        self.untrash_namespace(&ns)?;
>>   
>>           Ok(ns)
>>       }
>> @@ -565,6 +570,34 @@ impl DataStore {
>>           path.exists()
>>       }
>>   
>> +    /// Remove the namespace and all it's parent components from the trash by removing the trash or
>> +    /// trash-pending marker file for each namespace level from deepest to shallowest. Missing files
>> +    /// are ignored.
>> +    pub fn untrash_namespace(&self, namespace: &BackupNamespace) -> Result<(), Error> {
>> +        let mut namespace = namespace.clone();
>> +        while namespace.depth() > 0 {
>> +            let mut trash_file_path = self.base_path();
>> +            trash_file_path.push(namespace.path());
>> +            let mut pending_file_path = trash_file_path.clone();
>> +            pending_file_path.push(TRASH_PENDING_MARKER_FILENAME);
>> +            if let Err(err) = std::fs::remove_file(&pending_file_path) {
>> +                // ignore not found, either not trashed or un-trashed by concurrent operation
>> +                if err.kind() != std::io::ErrorKind::NotFound {
>> +                    bail!("failed to remove trash-pending file {trash_file_path:?}: {err}");
>> +                }
>> +            }
>> +            trash_file_path.push(TRASH_MARKER_FILENAME);
>> +            if let Err(err) = std::fs::remove_file(&trash_file_path) {
>> +                // ignore not found, either not trashed or un-trashed by concurrent operation
>> +                if err.kind() != std::io::ErrorKind::NotFound {
>> +                    bail!("failed to remove trash file {trash_file_path:?}: {err}");
>> +                }
>> +            }
>> +            namespace.pop();
>> +        }
>> +        Ok(())
>> +    }
>> +
>>       /// Remove all backup groups of a single namespace level but not the namespace itself.
>>       ///
>>       /// Does *not* descends into child-namespaces and doesn't remoes the namespace itself either.
>> @@ -574,6 +607,7 @@ impl DataStore {
>>       pub fn remove_namespace_groups(
>>           self: &Arc<Self>,
>>           ns: &BackupNamespace,
>> +        skip_trash: bool,
>>       ) -> Result<(bool, BackupGroupDeleteStats), Error> {
>>           // FIXME: locking? The single groups/snapshots are already protected, so may not be
>>           // necessary (depends on what we all allow to do with namespaces)
>> @@ -583,20 +617,22 @@ impl DataStore {
>>           let mut stats = BackupGroupDeleteStats::default();
>>   
>>           for group in self.iter_backup_groups(ns.to_owned())? {
>> -            let delete_stats = group?.destroy(true)?;
>> +            let delete_stats = group?.destroy(skip_trash)?;
>>               stats.add(&delete_stats);
>>               removed_all_groups = removed_all_groups && delete_stats.all_removed();
>>           }
>>   
>> -        let base_file = std::fs::File::open(self.base_path())?;
>> -        let base_fd = base_file.as_raw_fd();
>> -        for ty in BackupType::iter() {
>> -            let mut ty_dir = ns.path();
>> -            ty_dir.push(ty.to_string());
>> -            // best effort only, but we probably should log the error
>> -            if let Err(err) = unlinkat(Some(base_fd), &ty_dir, UnlinkatFlags::RemoveDir) {
>> -                if err != nix::errno::Errno::ENOENT {
>> -                    log::error!("failed to remove backup type {ty} in {ns} - {err}");
>> +        if skip_trash {
>> +            let base_file = std::fs::File::open(self.base_path())?;
>> +            let base_fd = base_file.as_raw_fd();
>> +            for ty in BackupType::iter() {
>> +                let mut ty_dir = ns.path();
>> +                ty_dir.push(ty.to_string());
>> +                // best effort only, but we probably should log the error
>> +                if let Err(err) = unlinkat(Some(base_fd), &ty_dir, UnlinkatFlags::RemoveDir) {
>> +                    if err != nix::errno::Errno::ENOENT {
>> +                        log::error!("failed to remove backup type {ty} in {ns} - {err}");
>> +                    }
>>                   }
>>               }
>>           }
>> @@ -613,6 +649,7 @@ impl DataStore {
>>           self: &Arc<Self>,
>>           ns: &BackupNamespace,
>>           delete_groups: bool,
>> +        skip_trash: bool,
>>       ) -> Result<(bool, BackupGroupDeleteStats), Error> {
>>           let store = self.name();
>>           let mut removed_all_requested = true;
>> @@ -620,16 +657,68 @@ impl DataStore {
>>           if delete_groups {
>>               log::info!("removing whole namespace recursively below {store}:/{ns}",);
>>               for ns in self.recursive_iter_backup_ns(ns.to_owned(), NamespaceListFilter::Active)? {
>> -                let (removed_ns_groups, delete_stats) = self.remove_namespace_groups(&ns?)?;
>> +                let namespace = ns?;
>> +
>> +                if !skip_trash {
>> +                    let mut path = self.base_path();
>> +                    path.push(namespace.path());
>> +                    path.push(TRASH_PENDING_MARKER_FILENAME);
>> +                    if let Err(err) = std::fs::File::create(&path) {
> 
> pending marker created here iff delete_groups && !skip_trash
> 
>> +                        if err.kind() != std::io::ErrorKind::AlreadyExists {
>> +                            return Err(err).context("failed to set trash pending marker file");
>> +                        }
>> +                    }
>> +                }
>> +
>> +                let (removed_ns_groups, delete_stats) =
>> +                    self.remove_namespace_groups(&namespace, skip_trash)?;
>>                   stats.add(&delete_stats);
>>                   removed_all_requested = removed_all_requested && removed_ns_groups;
>> +
>> +                if !skip_trash && !removed_ns_groups {
>> +                    self.untrash_namespace(&namespace)?;
>> +                }
>>               }
>>           } else {
>>               log::info!("pruning empty namespace recursively below {store}:/{ns}");
>>           }
>>   
>> -        removed_all_requested =
>> -            removed_all_requested && self.destroy_namespace_recursive(ns, delete_groups)?;
>> +        if skip_trash {
>> +            removed_all_requested =
>> +                removed_all_requested && self.destroy_namespace_recursive(ns, delete_groups)?;
>> +            return Ok((removed_all_requested, stats));
> 
> early return here iff skip_trash
> 
>> +        }
>> +
>> +        let mut children = self
>> +            .recursive_iter_backup_ns(ns.to_owned(), NamespaceListFilter::Active)?
>> +            .collect::<Result<Vec<BackupNamespace>, Error>>()?;
>> +
>> +        children.sort_by_key(|b| std::cmp::Reverse(b.depth()));
>> +
>> +        let mut all_trashed = true;
>> +        for ns in children.iter() {
>> +            let mut ns_dir = ns.path();
>> +            ns_dir.push("ns");
>> +
>> +            if !ns.is_root() {
>> +                let mut path = self.base_path();
>> +                path.push(ns.path());
>> +
>> +                let pending_path = path.join(TRASH_PENDING_MARKER_FILENAME);
>> +                path.push(TRASH_MARKER_FILENAME);
>> +                if let Err(err) = std::fs::rename(pending_path, path) {
> 
> pending marker assumed to exist here iff !skip_trash
> 
>> +                    if err.kind() == std::io::ErrorKind::NotFound {
>> +                        all_trashed = false;
>> +                    } else {
>> +                        return Err(err).context("Renaming pending marker to trash marker failed");
>> +                    }
>> +                }
>> +            }
>> +        }
>> +
>> +        if !all_trashed {
>> +            bail!("failed to prune namespace, not empty");
> 
> wrong error returned as a result..

Ah good catch! Indeed the logic does not work when `delete_groups` and 
`skip_trash` is false, and the namespace to delete being empty. Will fix 
this in an upcoming iteration of the patches.

> 
>> +        }
>>   
>>           Ok((removed_all_requested, stats))
>>       }
>> @@ -657,6 +746,24 @@ impl DataStore {
>>               let _ = unlinkat(Some(base_fd), &ns_dir, UnlinkatFlags::RemoveDir);
>>   
>>               if !ns.is_root() {
>> +                let rel_trash_path = ns.path().join(TRASH_MARKER_FILENAME);
>> +                if let Err(err) =
>> +                    unlinkat(Some(base_fd), &rel_trash_path, UnlinkatFlags::NoRemoveDir)
>> +                {
>> +                    if err != nix::errno::Errno::ENOENT {
>> +                        bail!("removing the trash file '{rel_trash_path:?}' failed - {err}")
>> +                    }
>> +                }
>> +                let rel_pending_path = ns.path().join(TRASH_PENDING_MARKER_FILENAME);
>> +                if let Err(err) =
>> +                    unlinkat(Some(base_fd), &rel_pending_path, UnlinkatFlags::NoRemoveDir)
>> +                {
>> +                    if err != nix::errno::Errno::ENOENT {
>> +                        bail!(
>> +                            "removing the trash pending file '{rel_pending_path:?}' failed - {err}"
>> +                        )
>> +                    }
>> +                }
>>                   match unlinkat(Some(base_fd), &ns.path(), UnlinkatFlags::RemoveDir) {
>>                       Ok(()) => log::debug!("removed namespace {ns}"),
>>                       Err(nix::errno::Errno::ENOENT) => {
>> diff --git a/src/api2/admin/namespace.rs b/src/api2/admin/namespace.rs
>> index e5463524a..a12d97883 100644
>> --- a/src/api2/admin/namespace.rs
>> +++ b/src/api2/admin/namespace.rs
>> @@ -166,7 +166,7 @@ pub fn delete_namespace(
>>   
>>       let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
>>   
>> -    let (removed_all, stats) = datastore.remove_namespace_recursive(&ns, delete_groups)?;
>> +    let (removed_all, stats) = datastore.remove_namespace_recursive(&ns, delete_groups, true)?;
>>       if !removed_all {
>>           let err_msg = if delete_groups {
>>               if datastore.old_locking() {
>> diff --git a/src/server/pull.rs b/src/server/pull.rs
>> index d3c6fcf6a..a60ccbf10 100644
>> --- a/src/server/pull.rs
>> +++ b/src/server/pull.rs
>> @@ -729,7 +729,7 @@ fn check_and_remove_ns(params: &PullParameters, local_ns: &BackupNamespace) -> R
>>       let (removed_all, _delete_stats) = params
>>           .target
>>           .store
>> -        .remove_namespace_recursive(local_ns, true)?;
>> +        .remove_namespace_recursive(local_ns, true, true)?;
>>   
>>       Ok(removed_all)
>>   }
>> -- 
>> 2.39.5
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel at lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 




From c.ebner at proxmox.com  Mon May 12 09:57:17 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 12 May 2025 09:57:17 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 16/21] api: datastore: add
 flag to list trashed snapshots only
In-Reply-To: <1746792915.yfm4snbghn.astroid@yuna.none>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-17-c.ebner@proxmox.com>
 <1746792915.yfm4snbghn.astroid@yuna.none>
Message-ID: <60443ef0-6ef5-4dd8-997f-fbd9ee00fb04@proxmox.com>

On 5/9/25 14:27, Fabian Gr?nbichler wrote:
> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>> Allows to conditionally show either active or trashed backup
>> snapshots, the latter being used when displaying the contents of the
>> trash for given datastore.
> 
> and what if I want to list both/all?

Did not include that option/variant as not required for the WebUI and 
possible to achieve by 2 API calls instead of a single one with the 
respective flags.

Further, listing both of them with the same call would require to add a 
trash flag to the response item type, which I tried to avoid.

Do we require to be able to list both with the same api call?



From c.ebner at proxmox.com  Mon May 12 10:05:28 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 12 May 2025 10:05:28 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 13/21] datastore: recreate
 trashed backup groups if requested
In-Reply-To: <1746792270.885af032mi.astroid@yuna.none>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-14-c.ebner@proxmox.com>
 <1746792270.885af032mi.astroid@yuna.none>
Message-ID: <af19bffe-5e5b-4d27-b015-1b7560afb4e5@proxmox.com>

On 5/9/25 14:27, Fabian Gr?nbichler wrote:
> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>> A whole backup group might have been marked as trashed, including all
>> of the contained snapshots.
>>
>> Since a new backup to that group (even as different user/owner)
>> should still work, permanently clear the whole trashed group before
>> recreation. This will limit the trash lifetime as now the group is
>> not recoverable until next garbage collection.
> 
> IMHO this is phrased in a way that makes it hard to parse, and in any
> case, such things should go into the docs..

Acked, will add a section to the docs for handling and implications of 
trashed items.

> 
>>
>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>> ---
>>   pbs-datastore/src/datastore.rs | 26 ++++++++++++++++++++++++++
>>   1 file changed, 26 insertions(+)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index 4f7766c36..ca05e1bea 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -934,6 +934,32 @@ impl DataStore {
>>                   let guard = backup_group.lock().with_context(|| {
>>                       format!("while creating locked backup group '{backup_group:?}'")
>>                   })?;
>> +                if backup_group.is_trashed() {
>> +                    info!("clear trashed backup group {full_path:?}");
> 
> I think we should only do this if the new and old owner are not
> identical..

Hmm, not sure if that would not introduce other possible issues/confusions?
E.g. a PVE host creates snapshots for a VM/CT with given ID to the 
corresponding backup group. The group get's pruned as not required 
anymore, the VM/CT destroyed. A new VM/CT is created on the PVE host and 
backups created to the (trashed) group...

> 
>> +                    let dir_entries = std::fs::read_dir(&full_path).context(
>> +                        "failed to read directory contents during cleanup of trashed group",
>> +                    )?;
>> +                    for entry in dir_entries {
>> +                        let entry = entry.context(
>> +                            "failed to read directory entry during clenup of trashed group",
>> +                        )?;
>> +                        let file_type = entry.file_type().context(
>> +                            "failed to get entry file type during clenup of trashed group",
>> +                        )?;
>> +                        if file_type.is_dir() {
>> +                            std::fs::remove_dir_all(entry.path())
>> +                                .context("failed to remove directory entry during clenup of trashed snapshot")?;
>> +                        } else {
>> +                            std::fs::remove_file(entry.path())
>> +                                .context("failed to remove directory entry during clenup of trashed snapshot")?;
>> +                        }
>> +                    }
>> +                    self.set_owner(ns, backup_group.group(), auth_id, false)?;
>> +                    let owner = self.get_owner(ns, backup_group.group())?; // just to be sure
> 
> sure about that? we are holding a lock here, nobody is allowed to change
> the owner but us..

Not really, opted for staying on the safe side here, because the 
per-exsiting code does it as well, but without mentioning why exactly.

> 
>> +                    self.untrash_namespace(ns)?;
>> +                    return Ok((owner, guard));
>> +                }
>> +
>>                   let owner = self.get_owner(ns, backup_group.group())?; // just to be sure
>>                   Ok((owner, guard))
>>               }
>> -- 
>> 2.39.5
>>
>>
>>
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel at lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>
>>
>>
> 
> 
> _______________________________________________
> pbs-devel mailing list
> pbs-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
> 
> 




From c.ebner at proxmox.com  Mon May 12 10:31:25 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 12 May 2025 10:31:25 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 12/21] datastore: clear
 trashed snapshot dir if re-creation requested
In-Reply-To: <1746791755.kc17x5cte7.astroid@yuna.none>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-13-c.ebner@proxmox.com>
 <1746791755.kc17x5cte7.astroid@yuna.none>
Message-ID: <b1476069-8238-4318-ba3d-0cab1295a251@proxmox.com>

On 5/9/25 14:27, Fabian Gr?nbichler wrote:
> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>> If a previously trashed snapshot has been requested for re-creation
>> (e.g. by a sync job in push direction), drop the contents of the
>> currently trashed snapshot.
>> The snapshot directory itself is already locked at that point, either
>> by the old locking mechanism acting on the directory itself or by the
>> new locking mechanism. Therefore, concurrent operations can be
>> excluded.
>>
>> For the call site this acts as if the snapshot directory has been
>> newly created.
>>
>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>> ---
>>   pbs-datastore/src/datastore.rs | 29 ++++++++++++++++++++++++++++-
>>   1 file changed, 28 insertions(+), 1 deletion(-)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index ffc6a7039..4f7766c36 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -951,8 +951,9 @@ impl DataStore {
>>       ) -> Result<(PathBuf, bool, BackupLockGuard), Error> {
>>           let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
>>           let relative_path = backup_dir.relative_path();
>> +        let full_path = backup_dir.full_path();
>>   
>> -        match std::fs::create_dir(backup_dir.full_path()) {
>> +        match std::fs::create_dir(&full_path) {
>>               Ok(_) => {
>>                   let guard = backup_dir.lock().with_context(|| {
>>                       format!("while creating new locked snapshot '{backup_dir:?}'")
>> @@ -963,6 +964,32 @@ impl DataStore {
>>                   let guard = backup_dir
>>                       .lock()
>>                       .with_context(|| format!("while creating locked snapshot '{backup_dir:?}'"))?;
>> +
>> +                if backup_dir.is_trashed() {
>> +                    info!("clear trashed backup snapshot {full_path:?}");
>> +                    let dir_entries = std::fs::read_dir(&full_path).context(
>> +                        "failed to read directory contents during cleanup of trashed snapshot",
>> +                    )?;
>> +                    for entry in dir_entries {
>> +                        let entry = entry.context(
>> +                            "failed to read directory entry during clenup of trashed snapshot",
>> +                        )?;
>> +                        // Only expect regular file entries
>> +                        std::fs::remove_file(entry.path()).context(
>> +                            "failed to remove directory entry during clenup of trashed snapshot",
>> +                        )?;
>> +                    }
>> +                    let group = BackupGroup::from(backup_dir);
>> +                    let group_trash_file = group.full_group_path().join(TRASH_MARKER_FILENAME);
>> +                    if let Err(err) = std::fs::remove_file(&group_trash_file) {
>> +                        if err.kind() != std::io::ErrorKind::NotFound {
>> +                            bail!("failed to remove group trash file of trashed snapshot");
>> +                        }
>> +                    }
> 
> this shouldn't be possible to hit, right? as creating a backup dir
> entails first creating the backup group (guarded by the group lock), and
> that would already clear the group's trash marker..

Yes, you are right: The whole group and namespace un-trashing logic is 
already performed by `create_locked_backup_group` and redundant at this 
point. So I will drop this and add a comment mentioning this fact instead.




From c.ebner at proxmox.com  Mon May 12 11:19:10 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 12 May 2025 11:19:10 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 06/21] api: tape: check
 trash marker when trying to write snapshot
In-Reply-To: <1746790719.mae7zgxkos.astroid@yuna.none>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-7-c.ebner@proxmox.com>
 <1746790719.mae7zgxkos.astroid@yuna.none>
Message-ID: <b26b95c3-2722-4458-a050-2c64ba9f6aae@proxmox.com>

On 5/9/25 14:27, Fabian Gr?nbichler wrote:
> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>> Since snapshots might be marked as trash, the snapshot directory
>> can still be present until cleaned up by garbage collection.
>>
>> Therefore, check for the presence of the trash marker after acquiring
>> the locked snapshot reader and skip over marked ones.
>>
>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>> ---
>>   src/api2/tape/backup.rs | 8 +++++++-
>>   1 file changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
>> index 923cb7834..17c8bc605 100644
>> --- a/src/api2/tape/backup.rs
>> +++ b/src/api2/tape/backup.rs
>> @@ -574,7 +574,13 @@ fn backup_snapshot(
>>       info!("backup snapshot {snapshot_path:?}");
>>   
>>       let snapshot_reader = match snapshot.locked_reader() {
>> -        Ok(reader) => reader,
>> +        Ok(reader) => {
>> +            if snapshot.is_trashed() {
>> +                info!("snapshot {snapshot_path:?} trashed, skipping");
> 
> not sure why we log this, but don't log this in other places?

The intention was to keep the pre-existing logging as for vanished 
snapshots (since pruned in the mean time), but make the 2 cases 
distinguishable.

So I think that either the logging should be dropped for both cases, or 
this should be logged as is. Opinions?




From c.ebner at proxmox.com  Mon May 12 11:32:16 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 12 May 2025 11:32:16 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 03/21] datastore: allow
 filtering of backups by their trash status
In-Reply-To: <1746790489.fw746s24j7.astroid@yuna.none>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-4-c.ebner@proxmox.com>
 <1746790489.fw746s24j7.astroid@yuna.none>
Message-ID: <ac00a151-5981-4af8-a9ea-8a6449b1dd05@proxmox.com>

On 5/9/25 14:27, Fabian Gr?nbichler wrote:
> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>> Extends the BackupGroup::list_backups method by an enum parameter to
>> filter backup snapshots based on their trash status.
>>
>> This allows to reuse the same logic for listing active, trashed or
>> all snapshots.
>>
>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>> ---
>>   pbs-datastore/src/backup_info.rs | 33 +++++++++++++++++++++++++++++---
>>   pbs-datastore/src/datastore.rs   |  4 ++--
>>   src/api2/admin/datastore.rs      | 10 +++++-----
>>   src/api2/tape/backup.rs          |  4 ++--
>>   src/backup/verify.rs             |  4 ++--
>>   src/server/prune_job.rs          |  3 ++-
>>   src/server/pull.rs               |  3 ++-
>>   7 files changed, 45 insertions(+), 16 deletions(-)
>>
>> diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
>> index 9ce4cb0f8..a8c864ac8 100644
>> --- a/pbs-datastore/src/backup_info.rs
>> +++ b/pbs-datastore/src/backup_info.rs
>> @@ -52,6 +52,12 @@ impl fmt::Debug for BackupGroup {
>>       }
>>   }
>>   
>> +pub enum ListBackupFilter {
>> +    Active,
> 
> active sounds like there's currently a backup going on..
> 
>> +    All,
>> +    Trashed,
>> +}

True, I might rename the enum and it's variants to

pub enum TrashStateFilter {
     All,
     NotTrashed,
     Trashed,
}

and use that for both, snapshot and namespace filtering.

Although a bit thorn, I do dislike the `NotTrashed` but fail to come up 
with a more striking name...



From f.gruenbichler at proxmox.com  Mon May 12 11:38:02 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Mon, 12 May 2025 11:38:02 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 06/21] api: tape: check
 trash marker when trying to write snapshot
In-Reply-To: <b26b95c3-2722-4458-a050-2c64ba9f6aae@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-7-c.ebner@proxmox.com>
 <1746790719.mae7zgxkos.astroid@yuna.none>
 <b26b95c3-2722-4458-a050-2c64ba9f6aae@proxmox.com>
Message-ID: <1747042630.pyemyxspmp.astroid@yuna.none>

On May 12, 2025 11:19 am, Christian Ebner wrote:
> On 5/9/25 14:27, Fabian Gr?nbichler wrote:
>> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>>> Since snapshots might be marked as trash, the snapshot directory
>>> can still be present until cleaned up by garbage collection.
>>>
>>> Therefore, check for the presence of the trash marker after acquiring
>>> the locked snapshot reader and skip over marked ones.
>>>
>>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>>> ---
>>>   src/api2/tape/backup.rs | 8 +++++++-
>>>   1 file changed, 7 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
>>> index 923cb7834..17c8bc605 100644
>>> --- a/src/api2/tape/backup.rs
>>> +++ b/src/api2/tape/backup.rs
>>> @@ -574,7 +574,13 @@ fn backup_snapshot(
>>>       info!("backup snapshot {snapshot_path:?}");
>>>   
>>>       let snapshot_reader = match snapshot.locked_reader() {
>>> -        Ok(reader) => reader,
>>> +        Ok(reader) => {
>>> +            if snapshot.is_trashed() {
>>> +                info!("snapshot {snapshot_path:?} trashed, skipping");
>> 
>> not sure why we log this, but don't log this in other places?
> 
> The intention was to keep the pre-existing logging as for vanished 
> snapshots (since pruned in the mean time), but make the 2 cases 
> distinguishable.
> 
> So I think that either the logging should be dropped for both cases, or 
> this should be logged as is. Opinions?

iff we make trashing the default at some point, this would become very
noisy.. the vanished logging is different, since it "just" triggers on
the time window between listing snapshots and attempting to read from
them..



From f.gruenbichler at proxmox.com  Mon May 12 11:46:25 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Mon, 12 May 2025 11:46:25 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 10/21] datastore: mark
 namespace as trash instead of deleting it
In-Reply-To: <8c214e1c-9878-41e6-a988-706eab6601a1@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-11-c.ebner@proxmox.com>
 <1746793154.56dzxckyhs.astroid@yuna.none>
 <8c214e1c-9878-41e6-a988-706eab6601a1@proxmox.com>
Message-ID: <1747042727.5oy8el2pyl.astroid@yuna.none>

On May 12, 2025 9:47 am, Christian Ebner wrote:
> On 5/9/25 14:27, Fabian Gr?nbichler wrote:
>> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>>> As for backup snapshots and groups, mark the namespace as trash
>>> instead of removing it and the contents right away, if the trash
>>> should not be bypassed.
>>>
>>> Actual removal of the hirarchical folder structure has to be taken
>>> care of by the garbage collection.
>>>
>>> In order to avoid races during removal, first mark the namespaces as
>>> trash pending, mark the snapshots and groups as trash and only after
>>> rename the pending marker file to the trash marker file. By this,
>>> concurrent backups can remove the trash pending marker to avoid the
>>> namespace being trashed.
>>>
>>> On re-creation of a trashed namespace remove the marker file on itself
>>> and any parent component from deepest to shallowest. As trashing a full
>>> namespace can also set the trash pending state for recursive namespace
>>> cleanup, remove encounters of that marker file as well to avoid the
>>> namespace or its parent being trashed.
>> 
>> this is fairly involved since we don't have locks on namespaces..
>> 
>> should we have them for creation/removal/trashing/untrashing?
> 
> That is what I did try to avoid at all cost with the two-marker 
> approach, as locking the namespace might be rather invasive. But if that 
> does not work out as intended, I see no other way as to add exclusive 
> locking for namespaces as well, yes.
> 
>> 
>> I assume those are fairly rare occurrences, I haven't yet analyzed the
>> interactions here to see whether the two-marker approach is actually
>> race-free..
>> 
>> OTOH, do we really need to (be able to) trash namespaces?
> 
> Yes, I think we do need that as well since the datastore's hierarchy 
> should remain in place, and the namespace iterator requires a way to 
> distinguish between a namespace which has been trashed/deleted and a 
> namespace which has not, but might contain trashed items. Otherwise a 
> user requesting to forget a namespace, still sees the (empty as only 
> trashed contents) namespace tree after the operation. Which would be 
> rather unexpected?

we could also require emptying the trash as part of forgetting a
namespace? we already have the `delete_groups` option and only remove a
non-empty namespace is set, we could re-use that or add a new
`empty_trash` option next to it, if we want double-opt-in ;)

else, we'd also need to support trashing whole datastores if
we follow this line of thinking..

like I said, I don't think forgetting namespaces is something that is
done very often, as namespaces are normally fairly static.. and if I
want to remove a namespace, I probably also want to remove all its
contents (trash or regular ;)).



From c.ebner at proxmox.com  Mon May 12 11:46:51 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 12 May 2025 11:46:51 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 06/21] api: tape: check
 trash marker when trying to write snapshot
In-Reply-To: <1747042630.pyemyxspmp.astroid@yuna.none>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-7-c.ebner@proxmox.com>
 <1746790719.mae7zgxkos.astroid@yuna.none>
 <b26b95c3-2722-4458-a050-2c64ba9f6aae@proxmox.com>
 <1747042630.pyemyxspmp.astroid@yuna.none>
Message-ID: <7199e702-5bb0-4917-8f22-018ca12ff8cb@proxmox.com>

On 5/12/25 11:38, Fabian Gr?nbichler wrote:
> On May 12, 2025 11:19 am, Christian Ebner wrote:
>> On 5/9/25 14:27, Fabian Gr?nbichler wrote:
>>> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>>>> Since snapshots might be marked as trash, the snapshot directory
>>>> can still be present until cleaned up by garbage collection.
>>>>
>>>> Therefore, check for the presence of the trash marker after acquiring
>>>> the locked snapshot reader and skip over marked ones.
>>>>
>>>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>>>> ---
>>>>    src/api2/tape/backup.rs | 8 +++++++-
>>>>    1 file changed, 7 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
>>>> index 923cb7834..17c8bc605 100644
>>>> --- a/src/api2/tape/backup.rs
>>>> +++ b/src/api2/tape/backup.rs
>>>> @@ -574,7 +574,13 @@ fn backup_snapshot(
>>>>        info!("backup snapshot {snapshot_path:?}");
>>>>    
>>>>        let snapshot_reader = match snapshot.locked_reader() {
>>>> -        Ok(reader) => reader,
>>>> +        Ok(reader) => {
>>>> +            if snapshot.is_trashed() {
>>>> +                info!("snapshot {snapshot_path:?} trashed, skipping");
>>>
>>> not sure why we log this, but don't log this in other places?
>>
>> The intention was to keep the pre-existing logging as for vanished
>> snapshots (since pruned in the mean time), but make the 2 cases
>> distinguishable.
>>
>> So I think that either the logging should be dropped for both cases, or
>> this should be logged as is. Opinions?
> 
> iff we make trashing the default at some point, this would become very
> noisy.. the vanished logging is different, since it "just" triggers on
> the time window between listing snapshots and attempting to read from
> them..

Good point, so will drop it then!

Although this opens up another thing to consider a bit closer: the 
current patches already favor the use of the trash over immediately 
deleting contents. I feel it might be best to only use it by default for 
manual operations via cli or ui, but not for actions taken by jobs... 
and for the first one provide the opt-out flags.



From c.ebner at proxmox.com  Mon May 12 11:55:29 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 12 May 2025 11:55:29 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 06/21] api: tape: check
 trash marker when trying to write snapshot
In-Reply-To: <1747042630.pyemyxspmp.astroid@yuna.none>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-7-c.ebner@proxmox.com>
 <1746790719.mae7zgxkos.astroid@yuna.none>
 <b26b95c3-2722-4458-a050-2c64ba9f6aae@proxmox.com>
 <1747042630.pyemyxspmp.astroid@yuna.none>
Message-ID: <3f0e6a0f-424a-494b-b114-e059b01e8e73@proxmox.com>

On 5/12/25 11:38, Fabian Gr?nbichler wrote:
> On May 12, 2025 11:19 am, Christian Ebner wrote:
>> On 5/9/25 14:27, Fabian Gr?nbichler wrote:
>>> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>>>> Since snapshots might be marked as trash, the snapshot directory
>>>> can still be present until cleaned up by garbage collection.
>>>>
>>>> Therefore, check for the presence of the trash marker after acquiring
>>>> the locked snapshot reader and skip over marked ones.
>>>>
>>>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>>>> ---
>>>>    src/api2/tape/backup.rs | 8 +++++++-
>>>>    1 file changed, 7 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
>>>> index 923cb7834..17c8bc605 100644
>>>> --- a/src/api2/tape/backup.rs
>>>> +++ b/src/api2/tape/backup.rs
>>>> @@ -574,7 +574,13 @@ fn backup_snapshot(
>>>>        info!("backup snapshot {snapshot_path:?}");
>>>>    
>>>>        let snapshot_reader = match snapshot.locked_reader() {
>>>> -        Ok(reader) => reader,
>>>> +        Ok(reader) => {
>>>> +            if snapshot.is_trashed() {
>>>> +                info!("snapshot {snapshot_path:?} trashed, skipping");
>>>
>>> not sure why we log this, but don't log this in other places?
>>
>> The intention was to keep the pre-existing logging as for vanished
>> snapshots (since pruned in the mean time), but make the 2 cases
>> distinguishable.
>>
>> So I think that either the logging should be dropped for both cases, or
>> this should be logged as is. Opinions?
> 
> iff we make trashing the default at some point, this would become very
> noisy.. the vanished logging is different, since it "just" triggers on
> the time window between listing snapshots and attempting to read from
> them..

On second thought, the logging here would act just the same as for the 
vanished case, as the snapshot list generate at the start of the job is 
pre-filtered already, only considering not trashed snapshots...



From f.gruenbichler at proxmox.com  Mon May 12 12:01:54 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Mon, 12 May 2025 12:01:54 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 16/21] api: datastore: add
 flag to list trashed snapshots only
In-Reply-To: <60443ef0-6ef5-4dd8-997f-fbd9ee00fb04@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-17-c.ebner@proxmox.com>
 <1746792915.yfm4snbghn.astroid@yuna.none>
 <60443ef0-6ef5-4dd8-997f-fbd9ee00fb04@proxmox.com>
Message-ID: <1747044000.m8pvlkpa0b.astroid@yuna.none>

On May 12, 2025 9:57 am, Christian Ebner wrote:
> On 5/9/25 14:27, Fabian Gr?nbichler wrote:
>> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>>> Allows to conditionally show either active or trashed backup
>>> snapshots, the latter being used when displaying the contents of the
>>> trash for given datastore.
>> 
>> and what if I want to list both/all?
> 
> Did not include that option/variant as not required for the WebUI and 
> possible to achieve by 2 API calls instead of a single one with the 
> respective flags.

well, doing two API calls means I then have to check for changed state
and merge the results as a client.. 

> Further, listing both of them with the same call would require to add a 
> trash flag to the response item type, which I tried to avoid.

we could only include the trash marker in the serialized result if it is
set?

> Do we require to be able to list both with the same api call?

I think it makes for a better interface..



From f.gruenbichler at proxmox.com  Mon May 12 12:02:54 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Mon, 12 May 2025 12:02:54 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 13/21] datastore: recreate
 trashed backup groups if requested
In-Reply-To: <af19bffe-5e5b-4d27-b015-1b7560afb4e5@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-14-c.ebner@proxmox.com>
 <1746792270.885af032mi.astroid@yuna.none>
 <af19bffe-5e5b-4d27-b015-1b7560afb4e5@proxmox.com>
Message-ID: <1747043245.m22bmacx8h.astroid@yuna.none>

On May 12, 2025 10:05 am, Christian Ebner wrote:
> On 5/9/25 14:27, Fabian Gr?nbichler wrote:
>> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>>> A whole backup group might have been marked as trashed, including all
>>> of the contained snapshots.
>>>
>>> Since a new backup to that group (even as different user/owner)
>>> should still work, permanently clear the whole trashed group before
>>> recreation. This will limit the trash lifetime as now the group is
>>> not recoverable until next garbage collection.
>> 
>> IMHO this is phrased in a way that makes it hard to parse, and in any
>> case, such things should go into the docs..
> 
> Acked, will add a section to the docs for handling and implications of 
> trashed items.
> 
>> 
>>>
>>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>>> ---
>>>   pbs-datastore/src/datastore.rs | 26 ++++++++++++++++++++++++++
>>>   1 file changed, 26 insertions(+)
>>>
>>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>>> index 4f7766c36..ca05e1bea 100644
>>> --- a/pbs-datastore/src/datastore.rs
>>> +++ b/pbs-datastore/src/datastore.rs
>>> @@ -934,6 +934,32 @@ impl DataStore {
>>>                   let guard = backup_group.lock().with_context(|| {
>>>                       format!("while creating locked backup group '{backup_group:?}'")
>>>                   })?;
>>> +                if backup_group.is_trashed() {
>>> +                    info!("clear trashed backup group {full_path:?}");
>> 
>> I think we should only do this if the new and old owner are not
>> identical..
> 
> Hmm, not sure if that would not introduce other possible issues/confusions?
> E.g. a PVE host creates snapshots for a VM/CT with given ID to the 
> corresponding backup group. The group get's pruned as not required 
> anymore, the VM/CT destroyed. A new VM/CT is created on the PVE host and 
> backups created to the (trashed) group...

I think the downside of too aggressively clearing trashed snapshot
(which might still be valuable) is far bigger than the downside of this
potential footgun. especially if the gist of how trashing works is
"trash will be cleared on next GC run", then "trash group; scheduled
backup runs" clearing all trashed snapshots would be potentially
disastrous - e.g., if I don't have GC configured at the moment and use
"trash group" as a sort of bulk action before recovering a few
individual snapshots..

if I give a user access to a VMID on the PVE side, then I need to ensure
any traces of old usage of that VMID is gone if I don't want that user
to see those traces. this doesn't change with the trash feature at all
(there's also things like old task logs, RRD data etc that are hard to
clear, so you *should never reuse a VMID between users* anyway).

as long as the owner stays the same, a user with access that allows
creating a new snapshot could have already recovered and read all those
trashed snapshot before creating a new snapshot, so nothing is leaked
that is not already accessible..

I am not even 100% sure if clearing the trash on owner change is
sensible/expected behaviour (the alternative being to block new
snapshots until the trash is cleared).

>> 
>>> +                    let dir_entries = std::fs::read_dir(&full_path).context(
>>> +                        "failed to read directory contents during cleanup of trashed group",
>>> +                    )?;
>>> +                    for entry in dir_entries {
>>> +                        let entry = entry.context(
>>> +                            "failed to read directory entry during clenup of trashed group",
>>> +                        )?;
>>> +                        let file_type = entry.file_type().context(
>>> +                            "failed to get entry file type during clenup of trashed group",
>>> +                        )?;
>>> +                        if file_type.is_dir() {
>>> +                            std::fs::remove_dir_all(entry.path())
>>> +                                .context("failed to remove directory entry during clenup of trashed snapshot")?;
>>> +                        } else {
>>> +                            std::fs::remove_file(entry.path())
>>> +                                .context("failed to remove directory entry during clenup of trashed snapshot")?;
>>> +                        }
>>> +                    }
>>> +                    self.set_owner(ns, backup_group.group(), auth_id, false)?;
>>> +                    let owner = self.get_owner(ns, backup_group.group())?; // just to be sure
>> 
>> sure about that? we are holding a lock here, nobody is allowed to change
>> the owner but us..
> 
> Not really, opted for staying on the safe side here, because the 
> per-exsiting code does it as well, but without mentioning why exactly.

AFAICT that pre-dates locking of groups or snapshots, I think it doesn't
make sense there either since the introduction of those - all calls to
set_owner are guarded by the group lock now..

>>> +                    self.untrash_namespace(ns)?;
>>> +                    return Ok((owner, guard));
>>> +                }
>>> +
>>>                   let owner = self.get_owner(ns, backup_group.group())?; // just to be sure
>>>                   Ok((owner, guard))
>>>               }
>>> -- 
>>> 2.39.5
>>>
>>>
>>>
>>> _______________________________________________
>>> pbs-devel mailing list
>>> pbs-devel at lists.proxmox.com
>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>>
>>>
>>>
>> 
>> 
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel at lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>> 
>> 
> 
> 



From f.gruenbichler at proxmox.com  Mon May 12 12:03:03 2025
From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=)
Date: Mon, 12 May 2025 12:03:03 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 18/21] api: admin: implement
 endpoints to restore trashed contents
In-Reply-To: <39b85c49-8a09-4702-8a76-2d7bdbc500e5@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-19-c.ebner@proxmox.com>
 <1746793013.k8qdvp27bh.astroid@yuna.none>
 <39b85c49-8a09-4702-8a76-2d7bdbc500e5@proxmox.com>
Message-ID: <1747042049.qdgxw6i7os.astroid@yuna.none>

On May 9, 2025 2:59 pm, Christian Ebner wrote:
> Thanks for feedback, will have a closer look next week.
> 
> Allow me two quick questions inline though...
> 
> On 5/9/25 14:27, Fabian Gr?nbichler wrote:
>> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>>> Implements the api endpoints to restore trashed contents contained
>>> within namespaces, backup groups or individual snapshots.
>>>
>>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>>> ---
>>>   src/api2/admin/datastore.rs | 173 +++++++++++++++++++++++++++++++++++-
>>>   1 file changed, 172 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
>>> index cbd24c729..eb033c3fc 100644
>>> --- a/src/api2/admin/datastore.rs
>>> +++ b/src/api2/admin/datastore.rs
>>> @@ -51,7 +51,7 @@ use pbs_api_types::{
>>>   };
>>>   use pbs_client::pxar::{create_tar, create_zip};
>>>   use pbs_config::CachedUserInfo;
>>> -use pbs_datastore::backup_info::{BackupInfo, ListBackupFilter};
>>> +use pbs_datastore::backup_info::{BackupInfo, ListBackupFilter, TRASH_MARKER_FILENAME};
>>>   use pbs_datastore::cached_chunk_reader::CachedChunkReader;
>>>   use pbs_datastore::catalog::{ArchiveEntry, CatalogReader};
>>>   use pbs_datastore::data_blob::DataBlob;
>>> @@ -2727,6 +2727,165 @@ pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<V
>>>       Ok(json!(upid))
>>>   }
>>>   
>>> +#[api(
>>> +    input: {
>>> +        properties: {
>>> +            store: { schema: DATASTORE_SCHEMA },
>>> +            ns: { type: BackupNamespace, },
>>> +        },
>>> +    },
>>> +    access: {
>>> +        permission: &Permission::Anybody,
>>> +        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
>>> +            or DATASTORE_BACKUP and being the owner of the group",
>>> +    },
>>> +)]
>>> +/// Recover trashed contents of a namespace.
>>> +pub fn recover_namespace(
>>> +    store: String,
>>> +    ns: BackupNamespace,
>>> +    rpcenv: &mut dyn RpcEnvironment,
>>> +) -> Result<(), Error> {
>>> +    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
>>> +    let limited = check_ns_privs_full(
>>> +        &store,
>>> +        &ns,
>>> +        &auth_id,
>>> +        PRIV_DATASTORE_MODIFY,
>>> +        PRIV_DATASTORE_BACKUP,
>>> +    )?;
>>> +
>>> +    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
>>> +
>>> +    for backup_group in datastore.iter_backup_groups(ns.clone())? {
>>> +        let backup_group = backup_group?;
>>> +        if limited {
>>> +            let owner = datastore.get_owner(&ns, backup_group.group())?;
>>> +            if check_backup_owner(&owner, &auth_id).is_err() {
>>> +                continue;
>>> +            }
>>> +        }
>>> +        do_recover_group(&backup_group)?;
>>> +    }
>>> +
>>> +    Ok(())
>>> +}
>>> +
>>> +#[api(
>>> +    input: {
>>> +        properties: {
>>> +            store: { schema: DATASTORE_SCHEMA },
>>> +            group: {
>>> +                type: pbs_api_types::BackupGroup,
>>> +                flatten: true,
>>> +            },
>>> +            ns: {
>>> +                type: BackupNamespace,
>>> +                optional: true,
>>> +            },
>>> +        },
>>> +    },
>>> +    access: {
>>> +        permission: &Permission::Anybody,
>>> +        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
>>> +            or DATASTORE_BACKUP and being the owner of the group",
>>> +    },
>>> +)]
>>> +/// Recover trashed contents of a backup group.
>>> +pub fn recover_group(
>>> +    store: String,
>>> +    group: pbs_api_types::BackupGroup,
>>> +    ns: Option<BackupNamespace>,
>>> +    rpcenv: &mut dyn RpcEnvironment,
>>> +) -> Result<(), Error> {
>>> +    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
>>> +    let ns = ns.unwrap_or_default();
>>> +    let datastore = check_privs_and_load_store(
>>> +        &store,
>>> +        &ns,
>>> +        &auth_id,
>>> +        PRIV_DATASTORE_MODIFY,
>>> +        PRIV_DATASTORE_BACKUP,
>>> +        Some(Operation::Write),
>>> +        &group,
>>> +    )?;
>>> +
>>> +    let backup_group = datastore.backup_group(ns, group);
>>> +    do_recover_group(&backup_group)?;
>>> +
>>> +    Ok(())
>>> +}
>>> +
>>> +fn do_recover_group(backup_group: &BackupGroup) -> Result<(), Error> {
>> 
>> missing locking for the group?
> 
> Not sure about that one. After all the group is trashed at least as long 
> as all the snapshots are trashed. And GC will only ever clean up the 
> group folder if the trash marker is not set. So I do not see a reason 
> why this should be locked.

because logically, this is the inverse of BackupGroup::destroy with
skip_trash=false, and that locks the group..

else you could have a recovery and a full deletion running concurrently
for the same group. also, while you are recovering the group you
probably don't want to start a new backup snapshot, which would also be
possible with the missing lock.

>>> +    let trashed_snapshots = backup_group.list_backups(ListBackupFilter::Trashed)?;
>>> +    for snapshot in trashed_snapshots {
>>> +        do_recover_snapshot(&snapshot.backup_dir)?;
>>> +    }
>>> +
>>> +    let group_trash_path = backup_group.full_group_path().join(TRASH_MARKER_FILENAME);
>>> +    if let Err(err) = std::fs::remove_file(&group_trash_path) {
>>> +        if err.kind() != std::io::ErrorKind::NotFound {
>>> +            bail!("failed to remove group trash file {group_trash_path:?} - {err}");
>>> +        }
>>> +    }
>>> +    Ok(())
>>> +}
>>> +
>>> +#[api(
>>> +    input: {
>>> +        properties: {
>>> +            store: { schema: DATASTORE_SCHEMA },
>>> +            backup_dir: {
>>> +                type: pbs_api_types::BackupDir,
>>> +                flatten: true,
>>> +            },
>>> +            ns: {
>>> +                type: BackupNamespace,
>>> +                optional: true,
>>> +            },
>>> +        },
>>> +    },
>>> +    access: {
>>> +        permission: &Permission::Anybody,
>>> +        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
>>> +            or DATASTORE_BACKUP and being the owner of the group",
>>> +    },
>>> +)]
>>> +/// Recover trashed contents of a backup snapshot.
>>> +pub fn recover_snapshot(
>>> +    store: String,
>>> +    backup_dir: pbs_api_types::BackupDir,
>>> +    ns: Option<BackupNamespace>,
>>> +    rpcenv: &mut dyn RpcEnvironment,
>>> +) -> Result<(), Error> {
>>> +    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
>>> +    let ns = ns.unwrap_or_default();
>>> +    let datastore = check_privs_and_load_store(
>>> +        &store,
>>> +        &ns,
>>> +        &auth_id,
>>> +        PRIV_DATASTORE_MODIFY,
>>> +        PRIV_DATASTORE_BACKUP,
>>> +        Some(Operation::Write),
>>> +        &backup_dir.group,
>>> +    )?;
>>> +
>>> +    let snapshot = datastore.backup_dir(ns, backup_dir)?;
>>> +    do_recover_snapshot(&snapshot)?;
>>> +
>>> +    Ok(())
>>> +}
>>> +
>>> +fn do_recover_snapshot(snapshot_dir: &BackupDir) -> Result<(), Error> {
>> 
>> missing locking for the snapshot?
> 
> Why? remove_file() should be atomic?

but a skip_trash=true deletion might be going on already or some other
operation holding a lock on the snapshot that doesn't want the 'trash'
status being changed underneath it?

>> 
>>> +    let trash_path = snapshot_dir.full_path().join(TRASH_MARKER_FILENAME);
>>> +    if let Err(err) = std::fs::remove_file(&trash_path) {
>>> +        if err.kind() != std::io::ErrorKind::NotFound {
>>> +            bail!("failed to remove trash file {trash_path:?} - {err}");
>>> +        }
>>> +    }
>>> +    Ok(())
>>> +}
>>> +
>>>   #[sortable]
>>>   const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
>>>       (
>>> @@ -2792,6 +2951,18 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
>>>           "pxar-file-download",
>>>           &Router::new().download(&API_METHOD_PXAR_FILE_DOWNLOAD),
>>>       ),
>>> +    (
>>> +        "recover-group",
>>> +        &Router::new().post(&API_METHOD_RECOVER_GROUP),
>> 
>> I am not sure whether those should be POST or PUT, they are modifying an
>> existing (trashed) group/snapshot/.. after all?
>> 
>>> +    ),
>>> +    (
>>> +        "recover-namespace",
>>> +        &Router::new().post(&API_METHOD_RECOVER_NAMESPACE),
>>> +    ),
>>> +    (
>>> +        "recover-snapshot",
>>> +        &Router::new().post(&API_METHOD_RECOVER_SNAPSHOT),
>>> +    ),
>>>       ("rrd", &Router::new().get(&API_METHOD_GET_RRD_STATS)),
>>>       (
>>>           "snapshots",
>>> -- 
>>> 2.39.5
>>>
>>>
>>>
>>> _______________________________________________
>>> pbs-devel mailing list
>>> pbs-devel at lists.proxmox.com
>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>>>
>>>
>>>
>> 
>> 
>> _______________________________________________
>> pbs-devel mailing list
>> pbs-devel at lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pbs-devel
>> 
>> 
> 
> 



From f.gruenbichler at proxmox.com  Mon May 12 12:08:34 2025
From: f.gruenbichler at proxmox.com (=?UTF-8?Q?Fabian_Gr=C3=BCnbichler?=)
Date: Mon, 12 May 2025 12:08:34 +0200 (CEST)
Subject: [pbs-devel] [RFC v2 proxmox-backup 03/21] datastore: allow
 filtering of backups by their trash status
In-Reply-To: <ac00a151-5981-4af8-a9ea-8a6449b1dd05@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-4-c.ebner@proxmox.com>
 <1746790489.fw746s24j7.astroid@yuna.none>
 <ac00a151-5981-4af8-a9ea-8a6449b1dd05@proxmox.com>
Message-ID: <1770180182.13719.1747044514683@webmail.proxmox.com>


> Christian Ebner <c.ebner at proxmox.com> hat am 12.05.2025 11:32 CEST geschrieben:
> 
>  
> On 5/9/25 14:27, Fabian Gr?nbichler wrote:
> > On May 8, 2025 3:05 pm, Christian Ebner wrote:
> >> Extends the BackupGroup::list_backups method by an enum parameter to
> >> filter backup snapshots based on their trash status.
> >>
> >> This allows to reuse the same logic for listing active, trashed or
> >> all snapshots.
> >>
> >> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> >> ---
> >>   pbs-datastore/src/backup_info.rs | 33 +++++++++++++++++++++++++++++---
> >>   pbs-datastore/src/datastore.rs   |  4 ++--
> >>   src/api2/admin/datastore.rs      | 10 +++++-----
> >>   src/api2/tape/backup.rs          |  4 ++--
> >>   src/backup/verify.rs             |  4 ++--
> >>   src/server/prune_job.rs          |  3 ++-
> >>   src/server/pull.rs               |  3 ++-
> >>   7 files changed, 45 insertions(+), 16 deletions(-)
> >>
> >> diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
> >> index 9ce4cb0f8..a8c864ac8 100644
> >> --- a/pbs-datastore/src/backup_info.rs
> >> +++ b/pbs-datastore/src/backup_info.rs
> >> @@ -52,6 +52,12 @@ impl fmt::Debug for BackupGroup {
> >>       }
> >>   }
> >>   
> >> +pub enum ListBackupFilter {
> >> +    Active,
> > 
> > active sounds like there's currently a backup going on..
> > 
> >> +    All,
> >> +    Trashed,
> >> +}
> 
> True, I might rename the enum and it's variants to
> 
> pub enum TrashStateFilter {
>      All,
>      NotTrashed,
>      Trashed,
> }
> 
> and use that for both, snapshot and namespace filtering.
> 
> Although a bit thorn, I do dislike the `NotTrashed` but fail to come up 
> with a more striking name...

IncludeTrash (list all snapshots, including trash)
ExcludeTrash (the backwards compat default?)
OnlyTrash (list only the trashed snapshots, for a 'trash can' view)

might work?

or else, `Regular` for non-trash snapshots? `Trash` might also be nicer
than `Trashed`.



From f.gruenbichler at proxmox.com  Mon May 12 12:09:31 2025
From: f.gruenbichler at proxmox.com (=?UTF-8?Q?Fabian_Gr=C3=BCnbichler?=)
Date: Mon, 12 May 2025 12:09:31 +0200 (CEST)
Subject: [pbs-devel] [RFC v2 proxmox-backup 06/21] api: tape: check
 trash marker when trying to write snapshot
In-Reply-To: <3f0e6a0f-424a-494b-b114-e059b01e8e73@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-7-c.ebner@proxmox.com>
 <1746790719.mae7zgxkos.astroid@yuna.none>
 <b26b95c3-2722-4458-a050-2c64ba9f6aae@proxmox.com>
 <1747042630.pyemyxspmp.astroid@yuna.none>
 <3f0e6a0f-424a-494b-b114-e059b01e8e73@proxmox.com>
Message-ID: <1751045582.13725.1747044571580@webmail.proxmox.com>


> Christian Ebner <c.ebner at proxmox.com> hat am 12.05.2025 11:55 CEST geschrieben:
> 
>  
> On 5/12/25 11:38, Fabian Gr?nbichler wrote:
> > On May 12, 2025 11:19 am, Christian Ebner wrote:
> >> On 5/9/25 14:27, Fabian Gr?nbichler wrote:
> >>> On May 8, 2025 3:05 pm, Christian Ebner wrote:
> >>>> Since snapshots might be marked as trash, the snapshot directory
> >>>> can still be present until cleaned up by garbage collection.
> >>>>
> >>>> Therefore, check for the presence of the trash marker after acquiring
> >>>> the locked snapshot reader and skip over marked ones.
> >>>>
> >>>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> >>>> ---
> >>>>    src/api2/tape/backup.rs | 8 +++++++-
> >>>>    1 file changed, 7 insertions(+), 1 deletion(-)
> >>>>
> >>>> diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
> >>>> index 923cb7834..17c8bc605 100644
> >>>> --- a/src/api2/tape/backup.rs
> >>>> +++ b/src/api2/tape/backup.rs
> >>>> @@ -574,7 +574,13 @@ fn backup_snapshot(
> >>>>        info!("backup snapshot {snapshot_path:?}");
> >>>>    
> >>>>        let snapshot_reader = match snapshot.locked_reader() {
> >>>> -        Ok(reader) => reader,
> >>>> +        Ok(reader) => {
> >>>> +            if snapshot.is_trashed() {
> >>>> +                info!("snapshot {snapshot_path:?} trashed, skipping");
> >>>
> >>> not sure why we log this, but don't log this in other places?
> >>
> >> The intention was to keep the pre-existing logging as for vanished
> >> snapshots (since pruned in the mean time), but make the 2 cases
> >> distinguishable.
> >>
> >> So I think that either the logging should be dropped for both cases, or
> >> this should be logged as is. Opinions?
> > 
> > iff we make trashing the default at some point, this would become very
> > noisy.. the vanished logging is different, since it "just" triggers on
> > the time window between listing snapshots and attempting to read from
> > them..
> 
> On second thought, the logging here would act just the same as for the 
> vanished case, as the snapshot list generate at the start of the job is 
> pre-filtered already, only considering not trashed snapshots...

I guess then we can keep it..



From c.ebner at proxmox.com  Mon May 12 12:35:35 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 12 May 2025 12:35:35 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 10/21] datastore: mark
 namespace as trash instead of deleting it
In-Reply-To: <1747042727.5oy8el2pyl.astroid@yuna.none>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
 <20250508130555.494782-11-c.ebner@proxmox.com>
 <1746793154.56dzxckyhs.astroid@yuna.none>
 <8c214e1c-9878-41e6-a988-706eab6601a1@proxmox.com>
 <1747042727.5oy8el2pyl.astroid@yuna.none>
Message-ID: <6144ff70-d18c-4526-a693-575bd6f0a5d9@proxmox.com>

On 5/12/25 11:46, Fabian Gr?nbichler wrote:
> On May 12, 2025 9:47 am, Christian Ebner wrote:
>> On 5/9/25 14:27, Fabian Gr?nbichler wrote:
>>> On May 8, 2025 3:05 pm, Christian Ebner wrote:
>>>> As for backup snapshots and groups, mark the namespace as trash
>>>> instead of removing it and the contents right away, if the trash
>>>> should not be bypassed.
>>>>
>>>> Actual removal of the hirarchical folder structure has to be taken
>>>> care of by the garbage collection.
>>>>
>>>> In order to avoid races during removal, first mark the namespaces as
>>>> trash pending, mark the snapshots and groups as trash and only after
>>>> rename the pending marker file to the trash marker file. By this,
>>>> concurrent backups can remove the trash pending marker to avoid the
>>>> namespace being trashed.
>>>>
>>>> On re-creation of a trashed namespace remove the marker file on itself
>>>> and any parent component from deepest to shallowest. As trashing a full
>>>> namespace can also set the trash pending state for recursive namespace
>>>> cleanup, remove encounters of that marker file as well to avoid the
>>>> namespace or its parent being trashed.
>>>
>>> this is fairly involved since we don't have locks on namespaces..
>>>
>>> should we have them for creation/removal/trashing/untrashing?
>>
>> That is what I did try to avoid at all cost with the two-marker
>> approach, as locking the namespace might be rather invasive. But if that
>> does not work out as intended, I see no other way as to add exclusive
>> locking for namespaces as well, yes.
>>
>>>
>>> I assume those are fairly rare occurrences, I haven't yet analyzed the
>>> interactions here to see whether the two-marker approach is actually
>>> race-free..
>>>
>>> OTOH, do we really need to (be able to) trash namespaces?
>>
>> Yes, I think we do need that as well since the datastore's hierarchy
>> should remain in place, and the namespace iterator requires a way to
>> distinguish between a namespace which has been trashed/deleted and a
>> namespace which has not, but might contain trashed items. Otherwise a
>> user requesting to forget a namespace, still sees the (empty as only
>> trashed contents) namespace tree after the operation. Which would be
>> rather unexpected?
> 
> we could also require emptying the trash as part of forgetting a
> namespace? we already have the `delete_groups` option and only remove a
> non-empty namespace is set, we could re-use that or add a new
> `empty_trash` option next to it, if we want double-opt-in ;)
> 
> else, we'd also need to support trashing whole datastores if
> we follow this line of thinking..
> 
> like I said, I don't think forgetting namespaces is something that is
> done very often, as namespaces are normally fairly static.. and if I
> want to remove a namespace, I probably also want to remove all its
> contents (trash or regular ;)).

Ah okay, requiring to clear trashed contents as well when deleting a 
namespace seems the better approach indeed. Will adapt the code 
accordingly.



From h.laimer at proxmox.com  Mon May 12 14:59:33 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 12 May 2025 14:59:33 +0200
Subject: [pbs-devel] [PATCH proxmox-backup] api: also update datastore cache
 on api process
Message-ID: <20250512125933.156192-1-h.laimer@proxmox.com>

Until now we only told the proxy through the command socket that it
should check if it has to update its cache. We never did that for the
cache the api process holds, this wasn't really a problem because we
never actually used any chunks store refs that would have been cached in
the first place, still, if we should ever have the api process do
anything that involves a `datastore_lookup` it will also be in the
cache on the api process.

This also updates the process-local cache whenever we tell the proxy
process to update its cache. And since this is the same logic in a few
places I moved it into a new helper function that does both,
- tell the proxy to update its cache
- and, update its own

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 src/api2/admin/datastore.rs  | 30 ++++++++++++++++++------------
 src/api2/config/datastore.rs | 32 +++++---------------------------
 2 files changed, 23 insertions(+), 39 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index a3ba82e21..02a53a932 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -164,6 +164,23 @@ fn get_all_snapshot_files(
     Ok((manifest, files))
 }
 
+/// Triggers a cache update (if needed) on both the api and proxy process
+pub async fn update_datastore_cache(name: &str) -> Result<(), Error> {
+    if let Ok(proxy_pid) = proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
+    {
+        let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
+        let _ = proxmox_daemon::command_socket::send_raw(
+            sock,
+            &format!(
+                "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
+                name
+            ),
+        )
+        .await;
+    };
+    DataStore::update_datastore_cache(name)
+}
+
 #[api(
     input: {
         properties: {
@@ -2752,18 +2769,7 @@ pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<V
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
     let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
 
-    if let Ok(proxy_pid) = proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
-    {
-        let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
-        let _ = proxmox_daemon::command_socket::send_raw(
-            sock,
-            &format!(
-                "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
-                &store
-            ),
-        )
-        .await;
-    }
+    let _ = update_datastore_cache(&store).await;
 
     let upid = WorkerTask::new_thread(
         "unmount-device",
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index b133be707..8233ec1a1 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -21,8 +21,8 @@ use pbs_config::BackupLockGuard;
 use pbs_datastore::chunk_store::ChunkStore;
 
 use crate::api2::admin::{
-    datastore::do_mount_device, prune::list_prune_jobs, sync::list_config_sync_jobs,
-    verify::list_verification_jobs,
+    datastore::do_mount_device, datastore::update_datastore_cache, prune::list_prune_jobs,
+    sync::list_config_sync_jobs, verify::list_verification_jobs,
 };
 use crate::api2::config::prune::{delete_prune_job, do_create_prune_job, has_prune_job};
 use crate::api2::config::sync::delete_sync_job;
@@ -533,19 +533,7 @@ pub fn update_datastore(
     // tell the proxy it might have to clear a cache entry
     if maintenance_mode_changed {
         tokio::spawn(async move {
-            if let Ok(proxy_pid) =
-                proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
-            {
-                let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
-                let _ = proxmox_daemon::command_socket::send_raw(
-                    sock,
-                    &format!(
-                        "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
-                        &name
-                    ),
-                )
-                .await;
-            }
+            let _ = update_datastore_cache(&name).await;
         });
     }
 
@@ -647,18 +635,8 @@ pub async fn delete_datastore(
 
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
     let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
-    if let Ok(proxy_pid) = proxmox_rest_server::read_pid(pbs_buildcfg::PROXMOX_BACKUP_PROXY_PID_FN)
-    {
-        let sock = proxmox_daemon::command_socket::path_from_pid(proxy_pid);
-        let _ = proxmox_daemon::command_socket::send_raw(
-            sock,
-            &format!(
-                "{{\"command\":\"update-datastore-cache\",\"args\":\"{}\"}}\n",
-                name.clone()
-            ),
-        )
-        .await;
-    };
+
+    let _ = update_datastore_cache(&name).await;
 
     let upid = WorkerTask::new_thread(
         "delete-datastore",
-- 
2.39.5




From s.hanreich at proxmox.com  Tue May 13 12:14:53 2025
From: s.hanreich at proxmox.com (Stefan Hanreich)
Date: Tue, 13 May 2025 12:14:53 +0200
Subject: [pbs-devel] [PATCH proxmox{, -api-types,
 -ve-rs} 0/6] Move perl deserializers from proxmox-login to
 proxmox-serde
Message-ID: <20250513101459.122641-1-s.hanreich@proxmox.com>

There are other places where we need to deal with data coming from Perl as well,
so move those helpers to a more appropriate place, where they can be re-used
across multiple crates, without having to add a dependency to proxmox-login.

proxmox-serde needs a bump, all other crates from this series depend on the
changes there.

proxmox:

Stefan Hanreich (3):
  serde: add parsing helpers for perl
  login: move parse module to proxmox-serde
  client: move to proxmox_serde perl helpers

 proxmox-client/Cargo.toml                               | 2 ++
 proxmox-client/src/lib.rs                               | 4 ++--
 proxmox-login/Cargo.toml                                | 1 +
 proxmox-login/src/api.rs                                | 2 +-
 proxmox-login/src/lib.rs                                | 2 --
 proxmox-serde/Cargo.toml                                | 3 +++
 proxmox-serde/src/lib.rs                                | 3 +++
 proxmox-login/src/parse.rs => proxmox-serde/src/perl.rs | 0
 8 files changed, 12 insertions(+), 5 deletions(-)
 rename proxmox-login/src/parse.rs => proxmox-serde/src/perl.rs (100%)


proxmox-api-types:

Stefan Hanreich (2):
  generator: use proxmox_serde for perl helpers
  regenerate

 Cargo.toml                                 |   2 +-
 pve-api-types/Cargo.toml                   |   2 +-
 pve-api-types/generator-lib/Schema2Rust.pm |  26 +-
 pve-api-types/src/generated/types.rs       | 798 ++++++++++-----------
 4 files changed, 414 insertions(+), 414 deletions(-)


proxmox-ve-rs:

Stefan Hanreich (1):
  config: use proxmox_serde perl helpers

 proxmox-ve-config/Cargo.toml              |  1 +
 proxmox-ve-config/src/firewall/bridge.rs  |  3 +-
 proxmox-ve-config/src/firewall/cluster.rs |  6 +-
 proxmox-ve-config/src/firewall/guest.rs   | 14 ++--
 proxmox-ve-config/src/firewall/host.rs    | 26 ++++----
 proxmox-ve-config/src/firewall/parse.rs   | 80 -----------------------
 6 files changed, 24 insertions(+), 106 deletions(-)


Summary over all repositories:
  18 files changed, 450 insertions(+), 525 deletions(-)

-- 
Generated by git-murpp 0.8.0


From s.hanreich at proxmox.com  Tue May 13 12:14:56 2025
From: s.hanreich at proxmox.com (Stefan Hanreich)
Date: Tue, 13 May 2025 12:14:56 +0200
Subject: [pbs-devel] [PATCH proxmox 3/3] client: move to proxmox_serde perl
 helpers
In-Reply-To: <20250513101459.122641-1-s.hanreich@proxmox.com>
References: <20250513101459.122641-1-s.hanreich@proxmox.com>
Message-ID: <20250513101459.122641-4-s.hanreich@proxmox.com>

The perl helpers have been moved to proxmox_serde from proxmox_login,
so fix all occurences using the proxmox_login crate.

Signed-off-by: Stefan Hanreich <s.hanreich at proxmox.com>
---
 proxmox-client/Cargo.toml | 2 ++
 proxmox-client/src/lib.rs | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/proxmox-client/Cargo.toml b/proxmox-client/Cargo.toml
index c2682e77..b15c9faa 100644
--- a/proxmox-client/Cargo.toml
+++ b/proxmox-client/Cargo.toml
@@ -27,6 +27,8 @@ proxmox-login = { workspace = true, features = [ "http" ] }
 proxmox-http = { workspace = true, optional = true, features = [ "client" ] }
 hyper = { workspace = true, optional = true }
 
+proxmox-serde = { workspace = true, features = [ "perl" ] }
+
 [dev-dependencies]
 serde_plain.workspace = true
 
diff --git a/proxmox-client/src/lib.rs b/proxmox-client/src/lib.rs
index e802f4ce..f1df1e1d 100644
--- a/proxmox-client/src/lib.rs
+++ b/proxmox-client/src/lib.rs
@@ -173,10 +173,10 @@ pub struct ApiResponseData<T> {
 
 #[derive(serde::Deserialize)]
 struct RawApiResponse<T> {
-    #[serde(default, deserialize_with = "proxmox_login::parse::deserialize_u16")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_u16")]
     status: Option<u16>,
     message: Option<String>,
-    #[serde(default, deserialize_with = "proxmox_login::parse::deserialize_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     success: Option<bool>,
     data: Option<T>,
 
-- 
2.39.5



From s.hanreich at proxmox.com  Tue May 13 12:14:57 2025
From: s.hanreich at proxmox.com (Stefan Hanreich)
Date: Tue, 13 May 2025 12:14:57 +0200
Subject: [pbs-devel] [PATCH proxmox-api-types 1/2] generator: use
 proxmox_serde for perl helpers
In-Reply-To: <20250513101459.122641-1-s.hanreich@proxmox.com>
References: <20250513101459.122641-1-s.hanreich@proxmox.com>
Message-ID: <20250513101459.122641-5-s.hanreich@proxmox.com>

The helpers for parsing perl values have been moved to proxmox_serde,
update all references to proxmox_login. No functional changes.

Signed-off-by: Stefan Hanreich <s.hanreich at proxmox.com>
---
 Cargo.toml                                 |  2 +-
 pve-api-types/Cargo.toml                   |  2 +-
 pve-api-types/generator-lib/Schema2Rust.pm | 26 +++++++++++-----------
 3 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/Cargo.toml b/Cargo.toml
index 1bbdd01..1e119d3 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -22,7 +22,7 @@ serde_plain = "1"
 serde_json = "1"
 
 proxmox-api-macro = "1.3"
-proxmox-login = "0.2"
+proxmox-serde = "0.1.2"
 proxmox-schema = "4"
 
 proxmox-client = "0.5"
diff --git a/pve-api-types/Cargo.toml b/pve-api-types/Cargo.toml
index 73cd3ef..e388cac 100644
--- a/pve-api-types/Cargo.toml
+++ b/pve-api-types/Cargo.toml
@@ -18,7 +18,7 @@ serde_json.workspace = true
 serde_plain.workspace = true
 #
 proxmox-api-macro.workspace = true
-proxmox-login.workspace = true
+proxmox-serde = { workspace = true, features = [ "perl" ] }
 proxmox-schema = { workspace = true, features = [ "api-types", "api-macro" ] }
 
 # For the client feature:
diff --git a/pve-api-types/generator-lib/Schema2Rust.pm b/pve-api-types/generator-lib/Schema2Rust.pm
index 009cf13..99a8fd6 100644
--- a/pve-api-types/generator-lib/Schema2Rust.pm
+++ b/pve-api-types/generator-lib/Schema2Rust.pm
@@ -1127,18 +1127,18 @@ my sub array_type : prototype($$$) {
 }
 
 my %serde_num = (
-    usize => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_usize")]',
-    isize => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_isize")]',
-    u8 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_u8")]',
-    u16 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_u16")]',
-    u32 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_u32")]',
-    u64 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_u64")]',
-    i8 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_i8")]',
-    i16 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_i16")]',
-    i32 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_i32")]',
-    i64 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_i64")]',
-    f32 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_f32")]',
-    f64 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_f64")]',
+    usize => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_usize")]',
+    isize => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_isize")]',
+    u8 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_u8")]',
+    u16 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_u16")]',
+    u32 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_u32")]',
+    u64 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_u64")]',
+    i8 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_i8")]',
+    i16 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_i16")]',
+    i32 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_i32")]',
+    i64 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_i64")]',
+    f32 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_f32")]',
+    f64 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_f64")]',
 );
 
 sub handle_def : prototype($$$) {
@@ -1169,7 +1169,7 @@ sub handle_def : prototype($$$) {
     } elsif ($type eq 'boolean') {
         $def->{type} = 'bool';
         push $def->{attrs}->@*,
-            "#[serde(deserialize_with = \"proxmox_login::parse::deserialize_bool\")]";
+            "#[serde(deserialize_with = \"proxmox_serde::perl::deserialize_bool\")]";
         $def->{api}->{default} = bool(delete $schema->{default});
     } elsif ($type eq 'number') {
         $def->{api}->{default} = delete $schema->{default};
-- 
2.39.5



From s.hanreich at proxmox.com  Tue May 13 12:14:55 2025
From: s.hanreich at proxmox.com (Stefan Hanreich)
Date: Tue, 13 May 2025 12:14:55 +0200
Subject: [pbs-devel] [PATCH proxmox 2/3] login: move parse module to
 proxmox-serde
In-Reply-To: <20250513101459.122641-1-s.hanreich@proxmox.com>
References: <20250513101459.122641-1-s.hanreich@proxmox.com>
Message-ID: <20250513101459.122641-3-s.hanreich@proxmox.com>

Remove the parse module, that has been moved to proxmox-serde. Fix all
usages of the parse module in the process and add the new dependency.
No functional changes intended.

Signed-off-by: Stefan Hanreich <s.hanreich at proxmox.com>
---
 proxmox-login/Cargo.toml   |   1 +
 proxmox-login/src/api.rs   |   2 +-
 proxmox-login/src/lib.rs   |   2 -
 proxmox-login/src/parse.rs | 373 -------------------------------------
 4 files changed, 2 insertions(+), 376 deletions(-)
 delete mode 100644 proxmox-login/src/parse.rs

diff --git a/proxmox-login/Cargo.toml b/proxmox-login/Cargo.toml
index 50dfe2c8..2dc28d52 100644
--- a/proxmox-login/Cargo.toml
+++ b/proxmox-login/Cargo.toml
@@ -16,6 +16,7 @@ base64.workspace = true
 percent-encoding.workspace = true
 serde = { workspace = true, features = [ "derive" ] }
 serde_json.workspace = true
+proxmox-serde = { workspace = true, features = [ "perl" ] }
 
 # For webauthn types
 webauthn-rs = { workspace = true, optional = true }
diff --git a/proxmox-login/src/api.rs b/proxmox-login/src/api.rs
index b7107312..6023485c 100644
--- a/proxmox-login/src/api.rs
+++ b/proxmox-login/src/api.rs
@@ -11,7 +11,7 @@ pub struct CreateTicket {
     /// With webauthn the format of half-authenticated tickts changed. New
     /// clients should pass 1 here and not worry about the old format. The old
     /// format is deprecated and will be retired with PVE-8.0
-    #[serde(deserialize_with = "crate::parse::deserialize_bool")]
+    #[serde(deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     #[serde(default, skip_serializing_if = "Option::is_none")]
     #[serde(rename = "new-format")]
     pub new_format: Option<bool>,
diff --git a/proxmox-login/src/lib.rs b/proxmox-login/src/lib.rs
index e97ece7b..4482f2e4 100644
--- a/proxmox-login/src/lib.rs
+++ b/proxmox-login/src/lib.rs
@@ -5,8 +5,6 @@
 
 use serde::{Deserialize, Serialize};
 
-pub mod parse;
-
 pub mod api;
 pub mod error;
 pub mod tfa;
diff --git a/proxmox-login/src/parse.rs b/proxmox-login/src/parse.rs
deleted file mode 100644
index 8efa86c9..00000000
--- a/proxmox-login/src/parse.rs
+++ /dev/null
@@ -1,373 +0,0 @@
-//! Some parsing helpers for the PVE API, mainly to deal with perl's untypedness.
-
-use std::fmt;
-
-use serde::de::Unexpected;
-
-// Boolean:
-
-pub trait FromBool: Sized + Default {
-    fn from_bool(value: bool) -> Self;
-}
-
-impl FromBool for bool {
-    fn from_bool(value: bool) -> Self {
-        value
-    }
-}
-
-impl FromBool for Option<bool> {
-    fn from_bool(value: bool) -> Self {
-        Some(value)
-    }
-}
-
-pub fn deserialize_bool<'de, D, T>(deserializer: D) -> Result<T, D::Error>
-where
-    D: serde::Deserializer<'de>,
-    T: FromBool,
-{
-    deserializer.deserialize_any(BoolVisitor::<T>::new())
-}
-
-struct BoolVisitor<T>(std::marker::PhantomData<T>);
-
-impl<T> BoolVisitor<T> {
-    fn new() -> Self {
-        Self(std::marker::PhantomData)
-    }
-}
-
-impl<'de, T: FromBool> serde::de::DeserializeSeed<'de> for BoolVisitor<T> {
-    type Value = T;
-
-    fn deserialize<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
-    where
-        D: serde::Deserializer<'de>,
-    {
-        deserialize_bool(deserializer)
-    }
-}
-
-impl<'de, T> serde::de::Visitor<'de> for BoolVisitor<T>
-where
-    T: FromBool,
-{
-    type Value = T;
-
-    fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
-        f.write_str("a boolean-ish...")
-    }
-
-    fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
-    where
-        D: serde::Deserializer<'de>,
-    {
-        deserializer.deserialize_any(self)
-    }
-
-    fn visit_none<E>(self) -> Result<Self::Value, E> {
-        Ok(Default::default())
-    }
-
-    fn visit_bool<E: serde::de::Error>(self, value: bool) -> Result<Self::Value, E> {
-        Ok(Self::Value::from_bool(value))
-    }
-
-    fn visit_i128<E: serde::de::Error>(self, value: i128) -> Result<Self::Value, E> {
-        Ok(Self::Value::from_bool(value != 0))
-    }
-
-    fn visit_i64<E: serde::de::Error>(self, value: i64) -> Result<Self::Value, E> {
-        Ok(Self::Value::from_bool(value != 0))
-    }
-
-    fn visit_u64<E: serde::de::Error>(self, value: u64) -> Result<Self::Value, E> {
-        Ok(Self::Value::from_bool(value != 0))
-    }
-
-    fn visit_u128<E: serde::de::Error>(self, value: u128) -> Result<Self::Value, E> {
-        Ok(Self::Value::from_bool(value != 0))
-    }
-
-    fn visit_str<E: serde::de::Error>(self, value: &str) -> Result<Self::Value, E> {
-        let value = if value.eq_ignore_ascii_case("true")
-            || value.eq_ignore_ascii_case("yes")
-            || value.eq_ignore_ascii_case("on")
-            || value == "1"
-        {
-            true
-        } else if value.eq_ignore_ascii_case("false")
-            || value.eq_ignore_ascii_case("no")
-            || value.eq_ignore_ascii_case("off")
-            || value == "0"
-        {
-            false
-        } else {
-            return Err(E::invalid_value(
-                serde::de::Unexpected::Str(value),
-                &"a boolean-like value",
-            ));
-        };
-        Ok(Self::Value::from_bool(value))
-    }
-}
-
-// integer helpers:
-
-macro_rules! integer_helper {
-    ($ty:ident, $deserialize_name:ident, $trait: ident, $from_name:ident, $visitor:ident) => {
-        #[doc(hidden)]
-        pub trait $trait: Sized + Default {
-            fn $from_name(value: $ty) -> Self;
-        }
-
-        impl $trait for $ty {
-            fn $from_name(value: $ty) -> Self {
-                value
-            }
-        }
-
-        impl $trait for Option<$ty> {
-            fn $from_name(value: $ty) -> Self {
-                Some(value)
-            }
-        }
-
-        pub fn $deserialize_name<'de, D, T>(deserializer: D) -> Result<T, D::Error>
-        where
-            D: serde::Deserializer<'de>,
-            T: $trait,
-        {
-            deserializer.deserialize_any($visitor::<T>::new())
-        }
-
-        struct $visitor<T>(std::marker::PhantomData<T>);
-
-        impl<T> $visitor<T> {
-            fn new() -> Self {
-                Self(std::marker::PhantomData)
-            }
-        }
-
-        impl<'de, T: $trait> serde::de::DeserializeSeed<'de> for $visitor<T> {
-            type Value = T;
-
-            fn deserialize<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
-            where
-                D: serde::Deserializer<'de>,
-            {
-                $deserialize_name(deserializer)
-            }
-        }
-
-        impl<'de, T> serde::de::Visitor<'de> for $visitor<T>
-        where
-            T: $trait,
-        {
-            type Value = T;
-
-            fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
-                f.write_str(concat!("a ", stringify!($ty), "-ish..."))
-            }
-
-            fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
-            where
-                D: serde::Deserializer<'de>,
-            {
-                deserializer.deserialize_any(self)
-            }
-
-            fn visit_none<E>(self) -> Result<Self::Value, E> {
-                Ok(Default::default())
-            }
-
-            fn visit_i128<E: serde::de::Error>(self, value: i128) -> Result<Self::Value, E> {
-                $ty::try_from(value)
-                    .map_err(|_| E::invalid_value(Unexpected::Other("i128"), &self))
-                    .map(Self::Value::$from_name)
-            }
-
-            fn visit_i64<E: serde::de::Error>(self, value: i64) -> Result<Self::Value, E> {
-                $ty::try_from(value)
-                    .map_err(|_| E::invalid_value(Unexpected::Signed(value), &self))
-                    .map(Self::Value::$from_name)
-            }
-
-            fn visit_u64<E: serde::de::Error>(self, value: u64) -> Result<Self::Value, E> {
-                $ty::try_from(value)
-                    .map_err(|_| E::invalid_value(Unexpected::Unsigned(value), &self))
-                    .map(Self::Value::$from_name)
-            }
-
-            fn visit_u128<E: serde::de::Error>(self, value: u128) -> Result<Self::Value, E> {
-                $ty::try_from(value)
-                    .map_err(|_| E::invalid_value(Unexpected::Other("u128"), &self))
-                    .map(Self::Value::$from_name)
-            }
-
-            fn visit_str<E: serde::de::Error>(self, value: &str) -> Result<Self::Value, E> {
-                let value = value
-                    .parse()
-                    .map_err(|_| E::invalid_value(Unexpected::Str(value), &self))?;
-                self.visit_i64(value)
-            }
-        }
-    };
-}
-
-integer_helper!(
-    isize,
-    deserialize_isize,
-    FromIsize,
-    from_isize,
-    IsizeVisitor
-);
-
-integer_helper!(
-    usize,
-    deserialize_usize,
-    FromUsize,
-    from_usize,
-    UsizeVisitor
-);
-
-integer_helper!(u8, deserialize_u8, FromU8, from_u8, U8Visitor);
-integer_helper!(u16, deserialize_u16, FromU16, from_u16, U16Visitor);
-integer_helper!(u32, deserialize_u32, FromU32, from_u32, U32Visitor);
-integer_helper!(u64, deserialize_u64, FromU64, from_u64, U64Visitor);
-integer_helper!(i8, deserialize_i8, FromI8, from_i8, I8Visitor);
-integer_helper!(i16, deserialize_i16, FromI16, from_i16, I16Visitor);
-integer_helper!(i32, deserialize_i32, FromI32, from_i32, I32Visitor);
-integer_helper!(i64, deserialize_i64, FromI64, from_i64, I64Visitor);
-
-// float helpers:
-
-macro_rules! float_helper {
-    ($ty:ident, $deserialize_name:ident, $visitor:ident) => {
-        pub fn $deserialize_name<'de, D, T>(deserializer: D) -> Result<T, D::Error>
-        where
-            D: serde::Deserializer<'de>,
-            T: FromF64,
-        {
-            deserializer.deserialize_any($visitor::<T>::new())
-        }
-
-        struct $visitor<T>(std::marker::PhantomData<T>);
-
-        impl<T> $visitor<T> {
-            fn new() -> Self {
-                Self(std::marker::PhantomData)
-            }
-        }
-
-        impl<'de, T: FromF64> serde::de::DeserializeSeed<'de> for $visitor<T> {
-            type Value = T;
-
-            fn deserialize<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
-            where
-                D: serde::Deserializer<'de>,
-            {
-                $deserialize_name(deserializer)
-            }
-        }
-
-        impl<'de, T> serde::de::Visitor<'de> for $visitor<T>
-        where
-            T: FromF64,
-        {
-            type Value = T;
-
-            fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
-                f.write_str(concat!("a ", stringify!($ty), "-ish..."))
-            }
-
-            fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
-            where
-                D: serde::Deserializer<'de>,
-            {
-                deserializer.deserialize_any(self)
-            }
-
-            fn visit_none<E>(self) -> Result<Self::Value, E> {
-                Ok(Default::default())
-            }
-
-            fn visit_f64<E: serde::de::Error>(self, value: f64) -> Result<Self::Value, E> {
-                Ok(T::from_f64(value))
-            }
-
-            fn visit_i128<E: serde::de::Error>(self, value: i128) -> Result<Self::Value, E> {
-                let conv = value as f64;
-                if conv as i128 == value {
-                    Ok(T::from_f64(conv))
-                } else {
-                    Err(E::invalid_value(Unexpected::Other("i128"), &self))
-                }
-            }
-
-            fn visit_i64<E: serde::de::Error>(self, value: i64) -> Result<Self::Value, E> {
-                let conv = value as f64;
-                if conv as i64 == value {
-                    Ok(T::from_f64(conv))
-                } else {
-                    Err(E::invalid_value(Unexpected::Signed(value), &self))
-                }
-            }
-
-            fn visit_u128<E: serde::de::Error>(self, value: u128) -> Result<Self::Value, E> {
-                let conv = value as f64;
-                if conv as u128 == value {
-                    Ok(T::from_f64(conv))
-                } else {
-                    Err(E::invalid_value(Unexpected::Other("u128"), &self))
-                }
-            }
-
-            fn visit_u64<E: serde::de::Error>(self, value: u64) -> Result<Self::Value, E> {
-                let conv = value as f64;
-                if conv as u64 == value {
-                    Ok(T::from_f64(conv))
-                } else {
-                    Err(E::invalid_value(Unexpected::Unsigned(value), &self))
-                }
-            }
-
-            fn visit_str<E: serde::de::Error>(self, value: &str) -> Result<Self::Value, E> {
-                let value = value
-                    .parse()
-                    .map_err(|_| E::invalid_value(Unexpected::Str(value), &self))?;
-                self.visit_f64(value)
-            }
-        }
-    };
-}
-
-#[doc(hidden)]
-pub trait FromF64: Sized + Default {
-    fn from_f64(value: f64) -> Self;
-}
-
-impl FromF64 for f32 {
-    #[inline(always)]
-    fn from_f64(f: f64) -> f32 {
-        f as f32
-    }
-}
-
-impl FromF64 for f64 {
-    #[inline(always)]
-    fn from_f64(f: f64) -> f64 {
-        f
-    }
-}
-
-impl<T: FromF64> FromF64 for Option<T> {
-    #[inline(always)]
-    fn from_f64(f: f64) -> Option<T> {
-        Some(T::from_f64(f))
-    }
-}
-
-float_helper!(f32, deserialize_f32, F32Visitor);
-float_helper!(f64, deserialize_f64, F64Visitor);
-- 
2.39.5



From s.hanreich at proxmox.com  Tue May 13 12:14:54 2025
From: s.hanreich at proxmox.com (Stefan Hanreich)
Date: Tue, 13 May 2025 12:14:54 +0200
Subject: [pbs-devel] [PATCH proxmox 1/3] serde: add parsing helpers for perl
In-Reply-To: <20250513101459.122641-1-s.hanreich@proxmox.com>
References: <20250513101459.122641-1-s.hanreich@proxmox.com>
Message-ID: <20250513101459.122641-2-s.hanreich@proxmox.com>

Those have been moved from the proxmox-login crate. This crate seems
like a more natural place for hosting helpers to parse data coming
from perl via serde.

Signed-off-by: Stefan Hanreich <s.hanreich at proxmox.com>
---
 proxmox-serde/Cargo.toml  |   3 +
 proxmox-serde/src/lib.rs  |   3 +
 proxmox-serde/src/perl.rs | 373 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 379 insertions(+)
 create mode 100644 proxmox-serde/src/perl.rs

diff --git a/proxmox-serde/Cargo.toml b/proxmox-serde/Cargo.toml
index aa77099a..b1edb60d 100644
--- a/proxmox-serde/Cargo.toml
+++ b/proxmox-serde/Cargo.toml
@@ -20,3 +20,6 @@ proxmox-time.workspace = true
 
 [dev-dependencies]
 serde_json.workspace = true
+
+[features]
+perl = []
diff --git a/proxmox-serde/src/lib.rs b/proxmox-serde/src/lib.rs
index c1628349..8a3d8794 100644
--- a/proxmox-serde/src/lib.rs
+++ b/proxmox-serde/src/lib.rs
@@ -8,6 +8,9 @@ pub mod serde_macros;
 #[cfg(feature = "serde_json")]
 pub mod json;
 
+#[cfg(feature = "perl")]
+pub mod perl;
+
 /// Serialize Unix epoch (i64) as RFC3339.
 ///
 /// Usage example:
diff --git a/proxmox-serde/src/perl.rs b/proxmox-serde/src/perl.rs
new file mode 100644
index 00000000..8efa86c9
--- /dev/null
+++ b/proxmox-serde/src/perl.rs
@@ -0,0 +1,373 @@
+//! Some parsing helpers for the PVE API, mainly to deal with perl's untypedness.
+
+use std::fmt;
+
+use serde::de::Unexpected;
+
+// Boolean:
+
+pub trait FromBool: Sized + Default {
+    fn from_bool(value: bool) -> Self;
+}
+
+impl FromBool for bool {
+    fn from_bool(value: bool) -> Self {
+        value
+    }
+}
+
+impl FromBool for Option<bool> {
+    fn from_bool(value: bool) -> Self {
+        Some(value)
+    }
+}
+
+pub fn deserialize_bool<'de, D, T>(deserializer: D) -> Result<T, D::Error>
+where
+    D: serde::Deserializer<'de>,
+    T: FromBool,
+{
+    deserializer.deserialize_any(BoolVisitor::<T>::new())
+}
+
+struct BoolVisitor<T>(std::marker::PhantomData<T>);
+
+impl<T> BoolVisitor<T> {
+    fn new() -> Self {
+        Self(std::marker::PhantomData)
+    }
+}
+
+impl<'de, T: FromBool> serde::de::DeserializeSeed<'de> for BoolVisitor<T> {
+    type Value = T;
+
+    fn deserialize<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        deserialize_bool(deserializer)
+    }
+}
+
+impl<'de, T> serde::de::Visitor<'de> for BoolVisitor<T>
+where
+    T: FromBool,
+{
+    type Value = T;
+
+    fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        f.write_str("a boolean-ish...")
+    }
+
+    fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
+    where
+        D: serde::Deserializer<'de>,
+    {
+        deserializer.deserialize_any(self)
+    }
+
+    fn visit_none<E>(self) -> Result<Self::Value, E> {
+        Ok(Default::default())
+    }
+
+    fn visit_bool<E: serde::de::Error>(self, value: bool) -> Result<Self::Value, E> {
+        Ok(Self::Value::from_bool(value))
+    }
+
+    fn visit_i128<E: serde::de::Error>(self, value: i128) -> Result<Self::Value, E> {
+        Ok(Self::Value::from_bool(value != 0))
+    }
+
+    fn visit_i64<E: serde::de::Error>(self, value: i64) -> Result<Self::Value, E> {
+        Ok(Self::Value::from_bool(value != 0))
+    }
+
+    fn visit_u64<E: serde::de::Error>(self, value: u64) -> Result<Self::Value, E> {
+        Ok(Self::Value::from_bool(value != 0))
+    }
+
+    fn visit_u128<E: serde::de::Error>(self, value: u128) -> Result<Self::Value, E> {
+        Ok(Self::Value::from_bool(value != 0))
+    }
+
+    fn visit_str<E: serde::de::Error>(self, value: &str) -> Result<Self::Value, E> {
+        let value = if value.eq_ignore_ascii_case("true")
+            || value.eq_ignore_ascii_case("yes")
+            || value.eq_ignore_ascii_case("on")
+            || value == "1"
+        {
+            true
+        } else if value.eq_ignore_ascii_case("false")
+            || value.eq_ignore_ascii_case("no")
+            || value.eq_ignore_ascii_case("off")
+            || value == "0"
+        {
+            false
+        } else {
+            return Err(E::invalid_value(
+                serde::de::Unexpected::Str(value),
+                &"a boolean-like value",
+            ));
+        };
+        Ok(Self::Value::from_bool(value))
+    }
+}
+
+// integer helpers:
+
+macro_rules! integer_helper {
+    ($ty:ident, $deserialize_name:ident, $trait: ident, $from_name:ident, $visitor:ident) => {
+        #[doc(hidden)]
+        pub trait $trait: Sized + Default {
+            fn $from_name(value: $ty) -> Self;
+        }
+
+        impl $trait for $ty {
+            fn $from_name(value: $ty) -> Self {
+                value
+            }
+        }
+
+        impl $trait for Option<$ty> {
+            fn $from_name(value: $ty) -> Self {
+                Some(value)
+            }
+        }
+
+        pub fn $deserialize_name<'de, D, T>(deserializer: D) -> Result<T, D::Error>
+        where
+            D: serde::Deserializer<'de>,
+            T: $trait,
+        {
+            deserializer.deserialize_any($visitor::<T>::new())
+        }
+
+        struct $visitor<T>(std::marker::PhantomData<T>);
+
+        impl<T> $visitor<T> {
+            fn new() -> Self {
+                Self(std::marker::PhantomData)
+            }
+        }
+
+        impl<'de, T: $trait> serde::de::DeserializeSeed<'de> for $visitor<T> {
+            type Value = T;
+
+            fn deserialize<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
+            where
+                D: serde::Deserializer<'de>,
+            {
+                $deserialize_name(deserializer)
+            }
+        }
+
+        impl<'de, T> serde::de::Visitor<'de> for $visitor<T>
+        where
+            T: $trait,
+        {
+            type Value = T;
+
+            fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
+                f.write_str(concat!("a ", stringify!($ty), "-ish..."))
+            }
+
+            fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
+            where
+                D: serde::Deserializer<'de>,
+            {
+                deserializer.deserialize_any(self)
+            }
+
+            fn visit_none<E>(self) -> Result<Self::Value, E> {
+                Ok(Default::default())
+            }
+
+            fn visit_i128<E: serde::de::Error>(self, value: i128) -> Result<Self::Value, E> {
+                $ty::try_from(value)
+                    .map_err(|_| E::invalid_value(Unexpected::Other("i128"), &self))
+                    .map(Self::Value::$from_name)
+            }
+
+            fn visit_i64<E: serde::de::Error>(self, value: i64) -> Result<Self::Value, E> {
+                $ty::try_from(value)
+                    .map_err(|_| E::invalid_value(Unexpected::Signed(value), &self))
+                    .map(Self::Value::$from_name)
+            }
+
+            fn visit_u64<E: serde::de::Error>(self, value: u64) -> Result<Self::Value, E> {
+                $ty::try_from(value)
+                    .map_err(|_| E::invalid_value(Unexpected::Unsigned(value), &self))
+                    .map(Self::Value::$from_name)
+            }
+
+            fn visit_u128<E: serde::de::Error>(self, value: u128) -> Result<Self::Value, E> {
+                $ty::try_from(value)
+                    .map_err(|_| E::invalid_value(Unexpected::Other("u128"), &self))
+                    .map(Self::Value::$from_name)
+            }
+
+            fn visit_str<E: serde::de::Error>(self, value: &str) -> Result<Self::Value, E> {
+                let value = value
+                    .parse()
+                    .map_err(|_| E::invalid_value(Unexpected::Str(value), &self))?;
+                self.visit_i64(value)
+            }
+        }
+    };
+}
+
+integer_helper!(
+    isize,
+    deserialize_isize,
+    FromIsize,
+    from_isize,
+    IsizeVisitor
+);
+
+integer_helper!(
+    usize,
+    deserialize_usize,
+    FromUsize,
+    from_usize,
+    UsizeVisitor
+);
+
+integer_helper!(u8, deserialize_u8, FromU8, from_u8, U8Visitor);
+integer_helper!(u16, deserialize_u16, FromU16, from_u16, U16Visitor);
+integer_helper!(u32, deserialize_u32, FromU32, from_u32, U32Visitor);
+integer_helper!(u64, deserialize_u64, FromU64, from_u64, U64Visitor);
+integer_helper!(i8, deserialize_i8, FromI8, from_i8, I8Visitor);
+integer_helper!(i16, deserialize_i16, FromI16, from_i16, I16Visitor);
+integer_helper!(i32, deserialize_i32, FromI32, from_i32, I32Visitor);
+integer_helper!(i64, deserialize_i64, FromI64, from_i64, I64Visitor);
+
+// float helpers:
+
+macro_rules! float_helper {
+    ($ty:ident, $deserialize_name:ident, $visitor:ident) => {
+        pub fn $deserialize_name<'de, D, T>(deserializer: D) -> Result<T, D::Error>
+        where
+            D: serde::Deserializer<'de>,
+            T: FromF64,
+        {
+            deserializer.deserialize_any($visitor::<T>::new())
+        }
+
+        struct $visitor<T>(std::marker::PhantomData<T>);
+
+        impl<T> $visitor<T> {
+            fn new() -> Self {
+                Self(std::marker::PhantomData)
+            }
+        }
+
+        impl<'de, T: FromF64> serde::de::DeserializeSeed<'de> for $visitor<T> {
+            type Value = T;
+
+            fn deserialize<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
+            where
+                D: serde::Deserializer<'de>,
+            {
+                $deserialize_name(deserializer)
+            }
+        }
+
+        impl<'de, T> serde::de::Visitor<'de> for $visitor<T>
+        where
+            T: FromF64,
+        {
+            type Value = T;
+
+            fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
+                f.write_str(concat!("a ", stringify!($ty), "-ish..."))
+            }
+
+            fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
+            where
+                D: serde::Deserializer<'de>,
+            {
+                deserializer.deserialize_any(self)
+            }
+
+            fn visit_none<E>(self) -> Result<Self::Value, E> {
+                Ok(Default::default())
+            }
+
+            fn visit_f64<E: serde::de::Error>(self, value: f64) -> Result<Self::Value, E> {
+                Ok(T::from_f64(value))
+            }
+
+            fn visit_i128<E: serde::de::Error>(self, value: i128) -> Result<Self::Value, E> {
+                let conv = value as f64;
+                if conv as i128 == value {
+                    Ok(T::from_f64(conv))
+                } else {
+                    Err(E::invalid_value(Unexpected::Other("i128"), &self))
+                }
+            }
+
+            fn visit_i64<E: serde::de::Error>(self, value: i64) -> Result<Self::Value, E> {
+                let conv = value as f64;
+                if conv as i64 == value {
+                    Ok(T::from_f64(conv))
+                } else {
+                    Err(E::invalid_value(Unexpected::Signed(value), &self))
+                }
+            }
+
+            fn visit_u128<E: serde::de::Error>(self, value: u128) -> Result<Self::Value, E> {
+                let conv = value as f64;
+                if conv as u128 == value {
+                    Ok(T::from_f64(conv))
+                } else {
+                    Err(E::invalid_value(Unexpected::Other("u128"), &self))
+                }
+            }
+
+            fn visit_u64<E: serde::de::Error>(self, value: u64) -> Result<Self::Value, E> {
+                let conv = value as f64;
+                if conv as u64 == value {
+                    Ok(T::from_f64(conv))
+                } else {
+                    Err(E::invalid_value(Unexpected::Unsigned(value), &self))
+                }
+            }
+
+            fn visit_str<E: serde::de::Error>(self, value: &str) -> Result<Self::Value, E> {
+                let value = value
+                    .parse()
+                    .map_err(|_| E::invalid_value(Unexpected::Str(value), &self))?;
+                self.visit_f64(value)
+            }
+        }
+    };
+}
+
+#[doc(hidden)]
+pub trait FromF64: Sized + Default {
+    fn from_f64(value: f64) -> Self;
+}
+
+impl FromF64 for f32 {
+    #[inline(always)]
+    fn from_f64(f: f64) -> f32 {
+        f as f32
+    }
+}
+
+impl FromF64 for f64 {
+    #[inline(always)]
+    fn from_f64(f: f64) -> f64 {
+        f
+    }
+}
+
+impl<T: FromF64> FromF64 for Option<T> {
+    #[inline(always)]
+    fn from_f64(f: f64) -> Option<T> {
+        Some(T::from_f64(f))
+    }
+}
+
+float_helper!(f32, deserialize_f32, F32Visitor);
+float_helper!(f64, deserialize_f64, F64Visitor);
-- 
2.39.5



From s.hanreich at proxmox.com  Tue May 13 12:14:59 2025
From: s.hanreich at proxmox.com (Stefan Hanreich)
Date: Tue, 13 May 2025 12:14:59 +0200
Subject: [pbs-devel] [PATCH proxmox-ve-rs 1/1] config: use proxmox_serde
 perl helpers
In-Reply-To: <20250513101459.122641-1-s.hanreich@proxmox.com>
References: <20250513101459.122641-1-s.hanreich@proxmox.com>
Message-ID: <20250513101459.122641-7-s.hanreich@proxmox.com>

proxmox_serde provides helpers for parsing optional numbers / booleans
coming from perl, so move to using them instead of implementing our
own versions here. No functional changes intended.

Signed-off-by: Stefan Hanreich <s.hanreich at proxmox.com>
---
 proxmox-ve-config/Cargo.toml              |  1 +
 proxmox-ve-config/src/firewall/bridge.rs  |  3 +-
 proxmox-ve-config/src/firewall/cluster.rs |  6 +-
 proxmox-ve-config/src/firewall/guest.rs   | 14 ++--
 proxmox-ve-config/src/firewall/host.rs    | 26 ++++----
 proxmox-ve-config/src/firewall/parse.rs   | 80 -----------------------
 6 files changed, 24 insertions(+), 106 deletions(-)

diff --git a/proxmox-ve-config/Cargo.toml b/proxmox-ve-config/Cargo.toml
index 6c7a47e..e998b45 100644
--- a/proxmox-ve-config/Cargo.toml
+++ b/proxmox-ve-config/Cargo.toml
@@ -16,6 +16,7 @@ serde = { version = "1", features = [ "derive" ] }
 serde_json = "1"
 serde_plain = "1"
 serde_with = "3"
+proxmox-serde = { version = "0.1.2", features = [ "perl" ]}
 
 proxmox-schema = "4"
 proxmox-sys = "0.6.4"
diff --git a/proxmox-ve-config/src/firewall/bridge.rs b/proxmox-ve-config/src/firewall/bridge.rs
index 4acb6fa..6dea60e 100644
--- a/proxmox-ve-config/src/firewall/bridge.rs
+++ b/proxmox-ve-config/src/firewall/bridge.rs
@@ -3,7 +3,6 @@ use std::io;
 use anyhow::Error;
 use serde::Deserialize;
 
-use crate::firewall::parse::serde_option_bool;
 use crate::firewall::types::log::LogLevel;
 use crate::firewall::types::rule::{Direction, Verdict};
 
@@ -55,7 +54,7 @@ impl Config {
 #[derive(Debug, Default, Deserialize)]
 #[cfg_attr(test, derive(Eq, PartialEq))]
 pub struct Options {
-    #[serde(default, with = "serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     enable: Option<bool>,
 
     policy_forward: Option<Verdict>,
diff --git a/proxmox-ve-config/src/firewall/cluster.rs b/proxmox-ve-config/src/firewall/cluster.rs
index ce3dd53..a775cd9 100644
--- a/proxmox-ve-config/src/firewall/cluster.rs
+++ b/proxmox-ve-config/src/firewall/cluster.rs
@@ -10,7 +10,7 @@ use crate::firewall::types::log::LogRateLimit;
 use crate::firewall::types::rule::{Direction, Verdict};
 use crate::firewall::types::{Alias, Group, Rule};
 
-use crate::firewall::parse::{serde_option_bool, serde_option_log_ratelimit};
+use crate::firewall::parse::serde_option_log_ratelimit;
 
 #[derive(Debug, Default)]
 pub struct Config {
@@ -118,10 +118,10 @@ impl Config {
 #[derive(Debug, Default, Deserialize)]
 #[cfg_attr(test, derive(Eq, PartialEq))]
 pub struct Options {
-    #[serde(default, with = "serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     enable: Option<bool>,
 
-    #[serde(default, with = "serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     ebtables: Option<bool>,
 
     #[serde(default, with = "serde_option_log_ratelimit")]
diff --git a/proxmox-ve-config/src/firewall/guest.rs b/proxmox-ve-config/src/firewall/guest.rs
index 23eaa4e..4428a75 100644
--- a/proxmox-ve-config/src/firewall/guest.rs
+++ b/proxmox-ve-config/src/firewall/guest.rs
@@ -13,8 +13,6 @@ use crate::firewall::types::Ipset;
 use anyhow::{bail, Error};
 use serde::Deserialize;
 
-use crate::firewall::parse::serde_option_bool;
-
 /// default return value for [`Config::is_enabled()`]
 pub const GUEST_ENABLED_DEFAULT: bool = false;
 /// default return value for [`Config::allow_ndp()`]
@@ -37,25 +35,25 @@ pub const GUEST_POLICY_FORWARD_DEFAULT: Verdict = Verdict::Accept;
 #[derive(Debug, Default, Deserialize)]
 #[cfg_attr(test, derive(Eq, PartialEq))]
 pub struct Options {
-    #[serde(default, with = "serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     dhcp: Option<bool>,
 
-    #[serde(default, with = "serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     enable: Option<bool>,
 
-    #[serde(default, with = "serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     ipfilter: Option<bool>,
 
-    #[serde(default, with = "serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     ndp: Option<bool>,
 
-    #[serde(default, with = "serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     radv: Option<bool>,
 
     log_level_in: Option<LogLevel>,
     log_level_out: Option<LogLevel>,
 
-    #[serde(default, with = "serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     macfilter: Option<bool>,
 
     #[serde(rename = "policy_in")]
diff --git a/proxmox-ve-config/src/firewall/host.rs b/proxmox-ve-config/src/firewall/host.rs
index 394896c..f7b02f9 100644
--- a/proxmox-ve-config/src/firewall/host.rs
+++ b/proxmox-ve-config/src/firewall/host.rs
@@ -36,49 +36,49 @@ pub const HOST_LOG_INVALID_CONNTRACK: bool = false;
 #[derive(Debug, Default, Deserialize)]
 #[cfg_attr(test, derive(Eq, PartialEq))]
 pub struct Options {
-    #[serde(default, with = "parse::serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     enable: Option<bool>,
 
-    #[serde(default, with = "parse::serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     nftables: Option<bool>,
 
     log_level_in: Option<LogLevel>,
     log_level_out: Option<LogLevel>,
     log_level_forward: Option<LogLevel>,
 
-    #[serde(default, with = "parse::serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     log_nf_conntrack: Option<bool>,
-    #[serde(default, with = "parse::serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     ndp: Option<bool>,
 
-    #[serde(default, with = "parse::serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     nf_conntrack_allow_invalid: Option<bool>,
 
     // is Option<Vec<>> for easier deserialization
     #[serde(default, with = "parse::serde_option_conntrack_helpers")]
     nf_conntrack_helpers: Option<Vec<String>>,
 
-    #[serde(default, with = "parse::serde_option_number")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_i64")]
     nf_conntrack_max: Option<i64>,
-    #[serde(default, with = "parse::serde_option_number")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_i64")]
     nf_conntrack_tcp_timeout_established: Option<i64>,
-    #[serde(default, with = "parse::serde_option_number")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_i64")]
     nf_conntrack_tcp_timeout_syn_recv: Option<i64>,
 
-    #[serde(default, with = "parse::serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     nosmurfs: Option<bool>,
 
-    #[serde(default, with = "parse::serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     protection_synflood: Option<bool>,
-    #[serde(default, with = "parse::serde_option_number")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_i64")]
     protection_synflood_burst: Option<i64>,
-    #[serde(default, with = "parse::serde_option_number")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_i64")]
     protection_synflood_rate: Option<i64>,
 
     smurf_log_level: Option<LogLevel>,
     tcp_flags_log_level: Option<LogLevel>,
 
-    #[serde(default, with = "parse::serde_option_bool")]
+    #[serde(default, deserialize_with = "proxmox_serde::perl::deserialize_bool")]
     tcpflags: Option<bool>,
 }
 
diff --git a/proxmox-ve-config/src/firewall/parse.rs b/proxmox-ve-config/src/firewall/parse.rs
index 8cf4757..7fd5c84 100644
--- a/proxmox-ve-config/src/firewall/parse.rs
+++ b/proxmox-ve-config/src/firewall/parse.rs
@@ -148,86 +148,6 @@ pub fn parse_named_section_tail<'a>(
     })
 }
 
-// parses a number from a string OR number
-pub mod serde_option_number {
-    use std::fmt;
-
-    use serde::de::{Deserializer, Error, Visitor};
-
-    pub fn deserialize<'de, D: Deserializer<'de>>(
-        deserializer: D,
-    ) -> Result<Option<i64>, D::Error> {
-        struct V;
-
-        impl<'de> Visitor<'de> for V {
-            type Value = Option<i64>;
-
-            fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
-                f.write_str("a numerical value")
-            }
-
-            fn visit_str<E: Error>(self, v: &str) -> Result<Self::Value, E> {
-                v.parse().map_err(E::custom).map(Some)
-            }
-
-            fn visit_none<E: Error>(self) -> Result<Self::Value, E> {
-                Ok(None)
-            }
-
-            fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
-            where
-                D: Deserializer<'de>,
-            {
-                deserializer.deserialize_any(self)
-            }
-        }
-
-        deserializer.deserialize_any(V)
-    }
-}
-
-// parses a bool from a string OR bool
-pub mod serde_option_bool {
-    use std::fmt;
-
-    use serde::de::{Deserializer, Error, Visitor};
-
-    pub fn deserialize<'de, D: Deserializer<'de>>(
-        deserializer: D,
-    ) -> Result<Option<bool>, D::Error> {
-        struct V;
-
-        impl<'de> Visitor<'de> for V {
-            type Value = Option<bool>;
-
-            fn expecting(&self, f: &mut fmt::Formatter) -> fmt::Result {
-                f.write_str("a boolean-like value")
-            }
-
-            fn visit_bool<E: Error>(self, v: bool) -> Result<Self::Value, E> {
-                Ok(Some(v))
-            }
-
-            fn visit_str<E: Error>(self, v: &str) -> Result<Self::Value, E> {
-                super::parse_bool(v).map_err(E::custom).map(Some)
-            }
-
-            fn visit_none<E: Error>(self) -> Result<Self::Value, E> {
-                Ok(None)
-            }
-
-            fn visit_some<D>(self, deserializer: D) -> Result<Self::Value, D::Error>
-            where
-                D: Deserializer<'de>,
-            {
-                deserializer.deserialize_any(self)
-            }
-        }
-
-        deserializer.deserialize_any(V)
-    }
-}
-
 // parses a comma_separated list of strings
 pub mod serde_option_conntrack_helpers {
     use std::fmt;
-- 
2.39.5



From s.hanreich at proxmox.com  Tue May 13 13:44:11 2025
From: s.hanreich at proxmox.com (Stefan Hanreich)
Date: Tue, 13 May 2025 13:44:11 +0200
Subject: [pbs-devel] [PATCH proxmox-api-types 1/2] generator: use
 proxmox_serde for perl helpers
In-Reply-To: <20250513101459.122641-5-s.hanreich@proxmox.com>
References: <20250513101459.122641-1-s.hanreich@proxmox.com>
 <20250513101459.122641-5-s.hanreich@proxmox.com>
Message-ID: <83eed3d4-43b8-44e7-a562-4c2e50891c0e@proxmox.com>

Since the patch #2 didn't go through due to size:

pve-api-types would need to be re-generated after this patch.

On 5/13/25 12:14, Stefan Hanreich wrote:
> The helpers for parsing perl values have been moved to proxmox_serde,
> update all references to proxmox_login. No functional changes.
> 
> Signed-off-by: Stefan Hanreich <s.hanreich at proxmox.com>
> ---
>  Cargo.toml                                 |  2 +-
>  pve-api-types/Cargo.toml                   |  2 +-
>  pve-api-types/generator-lib/Schema2Rust.pm | 26 +++++++++++-----------
>  3 files changed, 15 insertions(+), 15 deletions(-)
> 
> diff --git a/Cargo.toml b/Cargo.toml
> index 1bbdd01..1e119d3 100644
> --- a/Cargo.toml
> +++ b/Cargo.toml
> @@ -22,7 +22,7 @@ serde_plain = "1"
>  serde_json = "1"
>  
>  proxmox-api-macro = "1.3"
> -proxmox-login = "0.2"
> +proxmox-serde = "0.1.2"
>  proxmox-schema = "4"
>  
>  proxmox-client = "0.5"
> diff --git a/pve-api-types/Cargo.toml b/pve-api-types/Cargo.toml
> index 73cd3ef..e388cac 100644
> --- a/pve-api-types/Cargo.toml
> +++ b/pve-api-types/Cargo.toml
> @@ -18,7 +18,7 @@ serde_json.workspace = true
>  serde_plain.workspace = true
>  #
>  proxmox-api-macro.workspace = true
> -proxmox-login.workspace = true
> +proxmox-serde = { workspace = true, features = [ "perl" ] }
>  proxmox-schema = { workspace = true, features = [ "api-types", "api-macro" ] }
>  
>  # For the client feature:
> diff --git a/pve-api-types/generator-lib/Schema2Rust.pm b/pve-api-types/generator-lib/Schema2Rust.pm
> index 009cf13..99a8fd6 100644
> --- a/pve-api-types/generator-lib/Schema2Rust.pm
> +++ b/pve-api-types/generator-lib/Schema2Rust.pm
> @@ -1127,18 +1127,18 @@ my sub array_type : prototype($$$) {
>  }
>  
>  my %serde_num = (
> -    usize => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_usize")]',
> -    isize => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_isize")]',
> -    u8 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_u8")]',
> -    u16 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_u16")]',
> -    u32 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_u32")]',
> -    u64 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_u64")]',
> -    i8 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_i8")]',
> -    i16 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_i16")]',
> -    i32 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_i32")]',
> -    i64 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_i64")]',
> -    f32 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_f32")]',
> -    f64 => '#[serde(deserialize_with = "proxmox_login::parse::deserialize_f64")]',
> +    usize => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_usize")]',
> +    isize => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_isize")]',
> +    u8 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_u8")]',
> +    u16 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_u16")]',
> +    u32 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_u32")]',
> +    u64 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_u64")]',
> +    i8 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_i8")]',
> +    i16 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_i16")]',
> +    i32 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_i32")]',
> +    i64 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_i64")]',
> +    f32 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_f32")]',
> +    f64 => '#[serde(deserialize_with = "proxmox_serde::perl::deserialize_f64")]',
>  );
>  
>  sub handle_def : prototype($$$) {
> @@ -1169,7 +1169,7 @@ sub handle_def : prototype($$$) {
>      } elsif ($type eq 'boolean') {
>          $def->{type} = 'bool';
>          push $def->{attrs}->@*,
> -            "#[serde(deserialize_with = \"proxmox_login::parse::deserialize_bool\")]";
> +            "#[serde(deserialize_with = \"proxmox_serde::perl::deserialize_bool\")]";
>          $def->{api}->{default} = bool(delete $schema->{default});
>      } elsif ($type eq 'number') {
>          $def->{api}->{default} = delete $schema->{default};




From c.ebner at proxmox.com  Tue May 13 15:52:29 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:29 +0200
Subject: [pbs-devel] [PATCH v3 proxmox 2/20] pbs api types: datastore: add
 trash marker to snapshot list item
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-3-c.ebner@proxmox.com>

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-api-types/src/datastore.rs | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index d50d6f2b..ef5700d8 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -1311,6 +1311,9 @@ pub struct SnapshotListItem {
     /// Protection from prunes
     #[serde(default)]
     pub protected: bool,
+    /// Snapshot is marked as trash, only present if marked as trash
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub trash: Option<bool>,
 }
 
 #[api(
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:31 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:31 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 04/20] datastore: mark groups
 as trash on destroy
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-5-c.ebner@proxmox.com>

In order to implement the trash can functionality, mark all the
snapshots of the group and the group itself as trash instead of
deleting them right away. Cleanup of the group is deferred to the
garbage collection.

The group is marked as trash in order to skip over it in listings
without trashed items and to force an owner check when re-creation is
requested by a backup task to this group.

Snapshots already marked as trash within the same backup group will
be cleared as well when the group is requested to be destroyed with
skip trash.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 19 ++++++++++++++++---
 pbs-datastore/src/datastore.rs   |  4 ++--
 2 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index 76bcd15f5..9ce4cb0f8 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -215,7 +215,7 @@ impl BackupGroup {
     ///
     /// Returns `BackupGroupDeleteStats`, containing the number of deleted snapshots
     /// and number of protected snaphsots, which therefore were not removed.
-    pub fn destroy(&self) -> Result<BackupGroupDeleteStats, Error> {
+    pub fn destroy(&self, skip_trash: bool) -> Result<BackupGroupDeleteStats, Error> {
         let _guard = self
             .lock()
             .with_context(|| format!("while destroying group '{self:?}'"))?;
@@ -229,14 +229,20 @@ impl BackupGroup {
                 delete_stats.increment_protected_snapshots();
                 continue;
             }
-            snap.destroy(false, false)?;
+            snap.destroy(false, skip_trash)?;
             delete_stats.increment_removed_snapshots();
         }
 
         // Note: make sure the old locking mechanism isn't used as `remove_dir_all` is not safe in
         // that case
         if delete_stats.all_removed() && !*OLD_LOCKING {
-            self.remove_group_dir()?;
+            if skip_trash {
+                self.remove_group_dir()?;
+            } else {
+                let path = self.full_group_path().join(TRASH_MARKER_FILENAME);
+                let _trash_file =
+                    std::fs::File::create(path).context("failed to set trash file")?;
+            }
             delete_stats.increment_removed_groups();
         }
 
@@ -245,6 +251,13 @@ impl BackupGroup {
 
     /// Helper function, assumes that no more snapshots are present in the group.
     fn remove_group_dir(&self) -> Result<(), Error> {
+        let trash_path = self.full_group_path().join(TRASH_MARKER_FILENAME);
+        if let Err(err) = std::fs::remove_file(&trash_path) {
+            if err.kind() != std::io::ErrorKind::NotFound {
+                bail!("removing the trash file '{trash_path:?}' failed - {err}")
+            }
+        }
+
         let owner_path = self.store.owner_path(&self.ns, &self.group);
 
         std::fs::remove_file(&owner_path).map_err(|err| {
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 6df26e825..e546bc532 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -581,7 +581,7 @@ impl DataStore {
         let mut stats = BackupGroupDeleteStats::default();
 
         for group in self.iter_backup_groups(ns.to_owned())? {
-            let delete_stats = group?.destroy()?;
+            let delete_stats = group?.destroy(true)?;
             stats.add(&delete_stats);
             removed_all_groups = removed_all_groups && delete_stats.all_removed();
         }
@@ -674,7 +674,7 @@ impl DataStore {
     ) -> Result<BackupGroupDeleteStats, Error> {
         let backup_group = self.backup_group(ns.clone(), backup_group.clone());
 
-        backup_group.destroy()
+        backup_group.destroy(true)
     }
 
     /// Remove a backup directory including all content
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:27 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:27 +0200
Subject: [pbs-devel] [PATCH v3 proxmox proxmox-backup 00/20] implement trash
 bin functionality
Message-ID: <20250513135247.644260-1-c.ebner@proxmox.com>

This patch series implements a trash bin functionality, marking
backup snapshots and groups as trashed on prune and forget instead of
deleting them immediately. Cleanup is deferred to the garbage
collection job, allowing to recover the trashed items if removed by
accident.

Items are marked as trash by creating a `.trashed` marker file within
the respective snapshot or group directory. If group directories
contain the marker file, the whole group is considered trash. New
backups to a trashed group will remove the group trash marker file,
but only if the group is owned by the user creating the backup.
Backup creation will fail otherwise, in order to guarantee ownership.
If an backup is created to a pre-existing but trashed backup
directory, the backup directories contents are cleared and the backup
can proceed as if no such snapshot existed.

Most notably changes since the previous RFC version [0] (thanks Fabian
for feedback and discussion):
- Do not allow to mark namespaces as trash. Instead, in order to
  remove a namespace all contents (including trashed items) must be
  deleted. Drop all related patches as there is no longer the need to
  operate and filter on namespaces directly.
- Use exclusive snapshot/group locks in order to guarantee
  consistency during concurrent operations.
- Only clear backup snapshot directories in case of a new backup to the
  same backup directory. For groups, only allow to create a new backup
  to a trashed group if the ownership matches, but do not clear
  pre-existing trashed contents.
- Use pre-existing helpers for snapshot/group cleanup during garbage
  collection.
- Allow to clear only the trashed contents for a namespace or group via
  the API/WebUI, not removing all contents.
- Allow to filter for all trash states while listing snapshots,
  introducing a dedicated api type for this.
- Fix several smaller UI issues

As most of the patches changed significantly and many got dropped while
new ones added, no per-patch changes are noted.

Note: Patches 5 and onwards require patches 1 and 2 to the PBS api types
for compilation.

[0] https://lore.proxmox.com/pbs-devel/20250508130555.494782-1-c.ebner at proxmox.com/T/

proxmox:

Christian Ebner (2):
  pbs api types: add type for snapshot list filtering based on trash
    state
  pbs api types: datastore: add trash marker to snapshot list item

 pbs-api-types/src/datastore.rs | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

proxmox-backup:

Christian Ebner (18):
  datastore/api: mark snapshots as trash on destroy
  datastore: mark groups as trash on destroy
  datastore: add helpers to check if snapshot/group is trash
  datastore: allow filtering of backups by their trash state
  api: datastore: add trash state filtering for snapshot listing
  datastore: ignore trashed snapshots for last successful backup
  sync: ignore trashed snapshots/groups when reading from local source
  api: tape: check trash marker when trying to write snapshot
  datastore: clear trashed snapshot dir if re-creation requested
  datastore: recover backup group from trash for new backups
  datastore: garbage collection: clean-up trashed snapshots and groups
  client: expose skip trash flags for cli commands
  api: admin: implement endpoints to recover trashed contents
  api: admin: move backup group list generation into helper
  api: admin: add endpoint to clear trashed items from group
  ui: add recover for trashed items tab to datastore panel
  ui: drop 'permanent' in group/snapshot forget, default is to trash
  ui: mention trash items will be cleared on namespace deletion

 pbs-datastore/src/backup_info.rs      | 152 +++--
 pbs-datastore/src/datastore.rs        |  90 ++-
 proxmox-backup-client/src/group.rs    |  14 +-
 proxmox-backup-client/src/snapshot.rs |  16 +-
 src/api2/admin/datastore.rs           | 319 ++++++++--
 src/api2/backup/environment.rs        |   1 +
 src/api2/tape/backup.rs               |  12 +-
 src/backup/verify.rs                  |   6 +-
 src/server/prune_job.rs               |  10 +-
 src/server/pull.rs                    |  20 +-
 src/server/sync.rs                    |   2 +
 tests/prune.rs                        |   1 +
 www/Makefile                          |   1 +
 www/Utils.js                          |   1 +
 www/datastore/Content.js              |   4 +-
 www/datastore/Panel.js                |   8 +
 www/datastore/RecoverTrashed.js       | 805 ++++++++++++++++++++++++++
 www/window/NamespaceEdit.js           |   2 +-
 18 files changed, 1359 insertions(+), 105 deletions(-)
 create mode 100644 www/datastore/RecoverTrashed.js

-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:34 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:34 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 07/20] api: datastore: add
 trash state filtering for snapshot listing
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-8-c.ebner@proxmox.com>

Allows to conditionally include, exclude or show trashed backup
snapshots only, the latter being used when displaying the contents of
the trash for given datastore.

Default to exclude snapshots marked as trash from the list.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index a59e39abe..d371a46b0 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -466,6 +466,10 @@ pub async fn delete_snapshot(
                 optional: true,
                 schema: BACKUP_ID_SCHEMA,
             },
+            "trash-state": {
+                type: TrashStateFilter,
+                optional: true,
+            },
         },
     },
     returns: pbs_api_types::ADMIN_DATASTORE_LIST_SNAPSHOTS_RETURN_TYPE,
@@ -481,14 +485,16 @@ pub async fn list_snapshots(
     ns: Option<BackupNamespace>,
     backup_type: Option<BackupType>,
     backup_id: Option<String>,
+    trash_state: Option<TrashStateFilter>,
     _param: Value,
     _info: &ApiMethod,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Vec<SnapshotListItem>, Error> {
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let trash_state = trash_state.unwrap_or_default();
 
     tokio::task::spawn_blocking(move || unsafe {
-        list_snapshots_blocking(store, ns, backup_type, backup_id, auth_id)
+        list_snapshots_blocking(store, ns, backup_type, backup_id, auth_id, trash_state)
     })
     .await
     .map_err(|err| format_err!("failed to await blocking task: {err}"))?
@@ -501,6 +507,7 @@ unsafe fn list_snapshots_blocking(
     backup_type: Option<BackupType>,
     backup_id: Option<String>,
     auth_id: Authid,
+    trash_state: TrashStateFilter,
 ) -> Result<Vec<SnapshotListItem>, Error> {
     let ns = ns.unwrap_or_default();
 
@@ -627,7 +634,7 @@ unsafe fn list_snapshots_blocking(
             return Ok(snapshots);
         }
 
-        let group_backups = group.list_backups(TrashStateFilter::ExcludeTrash)?;
+        let group_backups = group.list_backups(trash_state)?;
 
         snapshots.extend(
             group_backups
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:35 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:35 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 08/20] datastore: ignore
 trashed snapshots for last successful backup
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-9-c.ebner@proxmox.com>

Exclude trashed snapshots from looking up the last successful backup
of the group, as trashed items are marked for deletion by the next
garbage collection run and must be considered as if not present
anymore.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index 53acc5baf..ca0c4bc55 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -175,8 +175,13 @@ impl BackupGroup {
                     return Ok(());
                 }
 
-                let mut manifest_path = PathBuf::from(backup_time);
-                manifest_path.push(MANIFEST_BLOB_NAME.as_ref());
+                let path = PathBuf::from(backup_time);
+                let trash_marker_path = path.join(TRASH_MARKER_FILENAME);
+                if trash_marker_path.exists() {
+                    return Ok(());
+                }
+
+                let manifest_path = path.join(MANIFEST_BLOB_NAME.as_ref());
 
                 use nix::fcntl::{openat, OFlag};
                 match openat(
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:37 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:37 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 10/20] api: tape: check trash
 marker when trying to write snapshot
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-11-c.ebner@proxmox.com>

Since snapshots might be marked as trash, the snapshot directory
can still be present until cleaned up by garbage collection.

Therefore, check for the presence of the trash marker after acquiring
the locked snapshot reader and skip over marked ones.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/tape/backup.rs | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
index 158905990..0adac96af 100644
--- a/src/api2/tape/backup.rs
+++ b/src/api2/tape/backup.rs
@@ -574,7 +574,13 @@ fn backup_snapshot(
     info!("backup snapshot {snapshot_path:?}");
 
     let snapshot_reader = match snapshot.locked_reader() {
-        Ok(reader) => reader,
+        Ok(reader) => {
+            if snapshot.is_trash() {
+                info!("snapshot {snapshot_path:?} marked as trash, skipping");
+                return Ok(SnapshotBackupResult::Ignored);
+            }
+            reader
+        }
         Err(err) => {
             if !snapshot.full_path().exists() {
                 // we got an error and the dir does not exist,
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:40 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:40 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 13/20] datastore: garbage
 collection: clean-up trashed snapshots and groups
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-14-c.ebner@proxmox.com>

Cleanup trashed items during phase 1 of garbage collection. If
encountered, index files located within trashed snapshots are still
touched.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs |  2 +-
 pbs-datastore/src/datastore.rs   | 28 ++++++++++++++++++++++++++++
 2 files changed, 29 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index ca0c4bc55..f334600c7 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -277,7 +277,7 @@ impl BackupGroup {
     }
 
     /// Helper function, assumes that no more snapshots are present in the group.
-    fn remove_group_dir(&self) -> Result<(), Error> {
+    pub(crate) fn remove_group_dir(&self) -> Result<(), Error> {
         let trash_path = self.full_group_path().join(TRASH_MARKER_FILENAME);
         if let Err(err) = std::fs::remove_file(&trash_path) {
             if err.kind() != std::io::ErrorKind::NotFound {
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 1bc096420..fd6eaadbb 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1254,10 +1254,38 @@ impl DataStore {
                             }
                             processed_index_files += 1;
                         }
+
+                        // Only try to lock trashed snapshots and continue if that is not possible,
+                        // as then most likely this is in the process of being untrashed.
+                        // Check trash state before and after locking to avoid otherwise possible
+                        // races.
+                        if snapshot.backup_dir.is_trash() {
+                            if let Ok(_lock) = snapshot.backup_dir.lock() {
+                                if snapshot.backup_dir.is_trash() {
+                                    snapshot.backup_dir.destroy(true, true)?;
+                                }
+                            } else {
+                                let path = snapshot.backup_dir.full_path();
+                                warn!("failed to lock trashed backup snapshot {path:?}, ignore");
+                            }
+                        }
                     }
 
                     break;
                 }
+                if group.is_trash() {
+                    if let Ok(_lock) = group.lock() {
+                        if group.is_trash() {
+                            if let Err(err) = group.remove_group_dir() {
+                                let path = group.full_group_path();
+                                warn!("failed to remove trashed backup group {path:?} - {err}");
+                            }
+                        } else {
+                            let path = group.full_group_path();
+                            warn!("failed to lock trashed backup group {path:?}");
+                        }
+                    }
+                }
             }
         }
 
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:43 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:43 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 16/20] api: admin: move backup
 group list generation into helper
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-17-c.ebner@proxmox.com>

Move the logic to generate a backup group list given backup type or
backup_id into a helper function. This allows to reuse the same logic
for generating the list of groups for which to clear trashed items.

No functional change intended.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs | 53 ++++++++++++++++++++++---------------
 1 file changed, 32 insertions(+), 21 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 3ea5b19f1..bc2d51612 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -507,6 +507,37 @@ pub async fn list_snapshots(
     .map_err(|err| format_err!("failed to await blocking task: {err}"))?
 }
 
+fn groups_by_type_or_id(
+    datastore: Arc<DataStore>,
+    ns: &BackupNamespace,
+    backup_type: Option<BackupType>,
+    backup_id: Option<String>,
+) -> Result<Vec<BackupGroup>, Error> {
+    // FIXME: filter also owner before collecting, for doing that nicely the owner should move into
+    // backup group and provide an error free (Err -> None) accessor
+    match (backup_type, backup_id) {
+        (Some(backup_type), Some(backup_id)) => Ok(vec![datastore.backup_group_from_parts(
+            ns.clone(),
+            backup_type,
+            backup_id,
+        )]),
+        // FIXME: Recursion
+        (Some(backup_type), None) => Ok(datastore
+            .iter_backup_type_ok(ns.clone(), backup_type)?
+            .collect()),
+        // FIXME: Recursion
+        (None, Some(backup_id)) => Ok(BackupType::iter()
+            .filter_map(|backup_type| {
+                let group =
+                    datastore.backup_group_from_parts(ns.clone(), backup_type, backup_id.clone());
+                group.exists().then_some(group)
+            })
+            .collect()),
+        // FIXME: Recursion
+        (None, None) => datastore.list_backup_groups(ns.clone()),
+    }
+}
+
 /// This must not run in a main worker thread as it potentially does tons of I/O.
 unsafe fn list_snapshots_blocking(
     store: String,
@@ -528,27 +559,7 @@ unsafe fn list_snapshots_blocking(
 
     let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
 
-    // FIXME: filter also owner before collecting, for doing that nicely the owner should move into
-    // backup group and provide an error free (Err -> None) accessor
-    let groups = match (backup_type, backup_id) {
-        (Some(backup_type), Some(backup_id)) => {
-            vec![datastore.backup_group_from_parts(ns.clone(), backup_type, backup_id)]
-        }
-        // FIXME: Recursion
-        (Some(backup_type), None) => datastore
-            .iter_backup_type_ok(ns.clone(), backup_type)?
-            .collect(),
-        // FIXME: Recursion
-        (None, Some(backup_id)) => BackupType::iter()
-            .filter_map(|backup_type| {
-                let group =
-                    datastore.backup_group_from_parts(ns.clone(), backup_type, backup_id.clone());
-                group.exists().then_some(group)
-            })
-            .collect(),
-        // FIXME: Recursion
-        (None, None) => datastore.list_backup_groups(ns.clone())?,
-    };
+    let groups = groups_by_type_or_id(datastore, &ns, backup_type, backup_id)?;
 
     let info_to_snapshot_list_item = |group: &BackupGroup, owner, info: BackupInfo| {
         let backup = pbs_api_types::BackupDir {
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:46 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:46 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 19/20] ui: drop 'permanent' in
 group/snapshot forget, default is to trash
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-20-c.ebner@proxmox.com>

Soften the message as the snapshots and groups will not be deleted
immediately anymore, but rather moved to the trash, from where they
still can be restored until the next garbage collection run.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/datastore/Content.js | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/www/datastore/Content.js b/www/datastore/Content.js
index fffd8c160..a6e28a773 100644
--- a/www/datastore/Content.js
+++ b/www/datastore/Content.js
@@ -1029,9 +1029,9 @@ Ext.define('PBS.DataStoreContent', {
 			if (data.ty === 'ns') {
 			    tip = gettext("Remove namespace '{0}'");
 			} else if (data.ty === 'dir') {
-			    tip = gettext("Permanently forget snapshot '{0}'");
+			    tip = gettext("Forget snapshot '{0}'");
 			} else if (data.ty === 'group') {
-			    tip = gettext("Permanently forget group '{0}'");
+			    tip = gettext("Forget group '{0}'");
 			}
 			return Ext.String.format(tip, v);
 		    },
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:47 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:47 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 20/20] ui: mention trash items
 will be cleared on namespace deletion
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-21-c.ebner@proxmox.com>

Include the information that removing backup groups will include
tashed items as well.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/window/NamespaceEdit.js | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/www/window/NamespaceEdit.js b/www/window/NamespaceEdit.js
index a9a440bbf..0f950e0d5 100644
--- a/www/window/NamespaceEdit.js
+++ b/www/window/NamespaceEdit.js
@@ -70,7 +70,7 @@ Ext.define('PBS.window.NamespaceDelete', {
 	    xtype: 'proxmoxcheckbox',
 	    name: 'delete-groups',
 	    reference: 'rmGroups',
-	    boxLabel: gettext('Delete all Backup Groups'),
+	    boxLabel: gettext('Delete all Backup Groups (including trash items)'),
 	    value: false,
 	    listeners: {
 		change: function(field, value) {
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:42 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:42 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 15/20] api: admin: implement
 endpoints to recover trashed contents
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-16-c.ebner@proxmox.com>

Implements the api endpoints to recover trashed contents contained
within backup groups or individual snapshots.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs | 143 +++++++++++++++++++++++++++++++++++-
 1 file changed, 142 insertions(+), 1 deletion(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 3f68edf24..3ea5b19f1 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -51,7 +51,7 @@ use pbs_api_types::{
 };
 use pbs_client::pxar::{create_tar, create_zip};
 use pbs_config::CachedUserInfo;
-use pbs_datastore::backup_info::BackupInfo;
+use pbs_datastore::backup_info::{BackupInfo, TRASH_MARKER_FILENAME};
 use pbs_datastore::cached_chunk_reader::CachedChunkReader;
 use pbs_datastore::catalog::{ArchiveEntry, CatalogReader};
 use pbs_datastore::data_blob::DataBlob;
@@ -2724,6 +2724,139 @@ pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<V
     Ok(json!(upid))
 }
 
+#[api(
+    input: {
+        properties: {
+            store: { schema: DATASTORE_SCHEMA },
+            group: {
+                type: pbs_api_types::BackupGroup,
+                flatten: true,
+            },
+            ns: {
+                type: BackupNamespace,
+                optional: true,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Anybody,
+        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
+            or DATASTORE_BACKUP and being the owner of the group",
+    },
+)]
+/// Recover trashed contents of a backup group.
+pub fn recover_group(
+    store: String,
+    group: pbs_api_types::BackupGroup,
+    ns: Option<BackupNamespace>,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let ns = ns.unwrap_or_default();
+    let datastore = check_privs_and_load_store(
+        &store,
+        &ns,
+        &auth_id,
+        PRIV_DATASTORE_MODIFY,
+        PRIV_DATASTORE_BACKUP,
+        Some(Operation::Write),
+        &group,
+    )?;
+
+    let backup_group = datastore.backup_group(ns, group);
+    do_recover_group(&backup_group)?;
+
+    Ok(())
+}
+
+fn do_recover_group(backup_group: &BackupGroup) -> Result<(), Error> {
+    let _exclusive_lock = backup_group
+        .lock()
+        .with_context(|| "while recovering group {backup_group:?}")?;
+    let trashed_snapshots = backup_group.list_backups(TrashStateFilter::OnlyTrash)?;
+    for snapshot in trashed_snapshots {
+        do_recover_snapshot(&snapshot.backup_dir)?;
+    }
+
+    let group_trash_path = backup_group.full_group_path().join(TRASH_MARKER_FILENAME);
+    if let Err(err) = std::fs::remove_file(&group_trash_path) {
+        if err.kind() != std::io::ErrorKind::NotFound {
+            bail!("failed to remove group trash file {group_trash_path:?} - {err}");
+        }
+    }
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            store: { schema: DATASTORE_SCHEMA },
+            backup_dir: {
+                type: pbs_api_types::BackupDir,
+                flatten: true,
+            },
+            ns: {
+                type: BackupNamespace,
+                optional: true,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Anybody,
+        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
+            or DATASTORE_BACKUP and being the owner of the group",
+    },
+)]
+/// Recover trashed contents of a backup snapshot.
+pub fn recover_snapshot(
+    store: String,
+    backup_dir: pbs_api_types::BackupDir,
+    ns: Option<BackupNamespace>,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let ns = ns.unwrap_or_default();
+    let datastore = check_privs_and_load_store(
+        &store,
+        &ns,
+        &auth_id,
+        PRIV_DATASTORE_MODIFY,
+        PRIV_DATASTORE_BACKUP,
+        Some(Operation::Write),
+        &backup_dir.group,
+    )?;
+
+    let backup_group = datastore.backup_group(ns.clone(), backup_dir.group.clone());
+    let snapshot = datastore.backup_dir(ns, backup_dir)?;
+    let _exclusive_group_lock = backup_group
+        .lock()
+        .with_context(|| "while recovering snapshot {snapshot_dir:?}")?;
+
+    do_recover_snapshot(&snapshot)?;
+
+    let group_trash_path = backup_group.full_group_path().join(TRASH_MARKER_FILENAME);
+    if let Err(err) = std::fs::remove_file(&group_trash_path) {
+        if err.kind() != std::io::ErrorKind::NotFound {
+            bail!("failed to remove group trash file {group_trash_path:?} - {err}");
+        }
+    }
+
+    Ok(())
+}
+
+fn do_recover_snapshot(snapshot_dir: &BackupDir) -> Result<(), Error> {
+    let _exclusive_lock = snapshot_dir
+        .lock()
+        .with_context(|| "while recovering snapshot {snapshot_dir:?}")?;
+    let trash_path = snapshot_dir.full_path().join(TRASH_MARKER_FILENAME);
+    if let Err(err) = std::fs::remove_file(&trash_path) {
+        if err.kind() != std::io::ErrorKind::NotFound {
+            bail!("failed to remove trash file {trash_path:?} - {err}");
+        }
+    }
+    Ok(())
+}
+
 #[sortable]
 const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
     (
@@ -2789,6 +2922,14 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
         "pxar-file-download",
         &Router::new().download(&API_METHOD_PXAR_FILE_DOWNLOAD),
     ),
+    (
+        "recover-group",
+        &Router::new().put(&API_METHOD_RECOVER_GROUP),
+    ),
+    (
+        "recover-snapshot",
+        &Router::new().put(&API_METHOD_RECOVER_SNAPSHOT),
+    ),
     ("rrd", &Router::new().get(&API_METHOD_GET_RRD_STATS)),
     (
         "snapshots",
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:41 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:41 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 14/20] client: expose skip
 trash flags for cli commands
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-15-c.ebner@proxmox.com>

Allows to explicitly set/clear the `skip-trash` flag when pruning
groups or snapshots via the client cli command.

Set defaults for `skip-trash` to false in order to use the trash.

Further, never add the flag to the api call parameters in the client
if not explicitly set, in order to keep backwards compatibility to
older PBS instances.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs        |  6 ++++--
 proxmox-backup-client/src/group.rs    | 14 +++++++++++++-
 proxmox-backup-client/src/snapshot.rs | 16 ++++++++++++----
 src/api2/admin/datastore.rs           | 11 +++++++++--
 src/api2/backup/environment.rs        |  1 +
 src/server/prune_job.rs               |  4 +++-
 src/server/pull.rs                    | 12 +++++++-----
 7 files changed, 49 insertions(+), 15 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index fd6eaadbb..574d6ec26 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -671,10 +671,11 @@ impl DataStore {
         self: &Arc<Self>,
         ns: &BackupNamespace,
         backup_group: &pbs_api_types::BackupGroup,
+        skip_trash: bool,
     ) -> Result<BackupGroupDeleteStats, Error> {
         let backup_group = self.backup_group(ns.clone(), backup_group.clone());
 
-        backup_group.destroy(true)
+        backup_group.destroy(skip_trash)
     }
 
     /// Remove a backup directory including all content
@@ -683,10 +684,11 @@ impl DataStore {
         ns: &BackupNamespace,
         backup_dir: &pbs_api_types::BackupDir,
         force: bool,
+        skip_trash: bool,
     ) -> Result<(), Error> {
         let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
 
-        backup_dir.destroy(force, true)
+        backup_dir.destroy(force, skip_trash)
     }
 
     /// Returns the time of the last successful backup
diff --git a/proxmox-backup-client/src/group.rs b/proxmox-backup-client/src/group.rs
index 67f26e261..42f8e1e61 100644
--- a/proxmox-backup-client/src/group.rs
+++ b/proxmox-backup-client/src/group.rs
@@ -37,11 +37,20 @@ pub fn group_mgmt_cli() -> CliCommandMap {
                 type: BackupNamespace,
                 optional: true,
             },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                description: "Immediately remove the group, not marking contents as trash.",
+            },
         }
     }
 )]
 /// Forget (remove) backup snapshots.
-async fn forget_group(group: String, mut param: Value) -> Result<(), Error> {
+async fn forget_group(
+    group: String,
+    skip_trash: Option<bool>,
+    mut param: Value,
+) -> Result<(), Error> {
     let backup_group: BackupGroup = group.parse()?;
     let repo = remove_repository_from_value(&mut param)?;
     let client = connect(&repo)?;
@@ -63,6 +72,9 @@ async fn forget_group(group: String, mut param: Value) -> Result<(), Error> {
     )?;
     if confirmation.is_yes() {
         let path = format!("api2/json/admin/datastore/{}/groups", repo.store());
+        if let Some(skip_trash) = skip_trash {
+            api_param["skip-trash"] = Value::from(skip_trash);
+        }
         if let Err(err) = client.delete(&path, Some(api_param)).await {
             // "ENOENT: No such file or directory" is part of the error returned when the group
             // has not been found. The full error contains the full datastore path and we would
diff --git a/proxmox-backup-client/src/snapshot.rs b/proxmox-backup-client/src/snapshot.rs
index f195c23b7..a9b46726a 100644
--- a/proxmox-backup-client/src/snapshot.rs
+++ b/proxmox-backup-client/src/snapshot.rs
@@ -173,11 +173,16 @@ async fn list_snapshot_files(param: Value) -> Result<Value, Error> {
                 type: String,
                 description: "Snapshot path.",
             },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                description: "Immediately remove the snapshot, not marking it as trash.",
+            },
         }
     }
 )]
 /// Forget (remove) backup snapshots.
-async fn forget_snapshots(param: Value) -> Result<(), Error> {
+async fn forget_snapshots(skip_trash: Option<bool>, param: Value) -> Result<(), Error> {
     let repo = extract_repository_from_value(&param)?;
 
     let backup_ns = optional_ns_param(&param)?;
@@ -188,9 +193,12 @@ async fn forget_snapshots(param: Value) -> Result<(), Error> {
 
     let path = format!("api2/json/admin/datastore/{}/snapshots", repo.store());
 
-    client
-        .delete(&path, Some(snapshot_args(&backup_ns, &snapshot)?))
-        .await?;
+    let mut args = snapshot_args(&backup_ns, &snapshot)?;
+    if let Some(skip_trash) = skip_trash {
+        args["skip-trash"] = Value::from(skip_trash);
+    }
+
+    client.delete(&path, Some(args)).await?;
 
     record_repository(&repo);
 
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index d371a46b0..3f68edf24 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -277,7 +277,13 @@ pub fn list_groups(
                 optional: true,
                 default: true,
                 description: "Return error when group cannot be deleted because of protected snapshots",
-            }
+            },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                default: false,
+                description: "Immediately remove the group including all snapshots, not marking it as trash.",
+            },
         },
     },
     returns: {
@@ -295,6 +301,7 @@ pub async fn delete_group(
     ns: Option<BackupNamespace>,
     error_on_protected: bool,
     group: pbs_api_types::BackupGroup,
+    skip_trash: bool,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<BackupGroupDeleteStats, Error> {
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
@@ -312,7 +319,7 @@ pub async fn delete_group(
             &group,
         )?;
 
-        let delete_stats = datastore.remove_backup_group(&ns, &group)?;
+        let delete_stats = datastore.remove_backup_group(&ns, &group, skip_trash)?;
 
         let error_msg = if datastore.old_locking() {
             "could not remove empty groups directories due to old locking mechanism.\n\
diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 3d541b461..b5619eb8c 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -730,6 +730,7 @@ impl BackupEnvironment {
             self.backup_dir.backup_ns(),
             self.backup_dir.as_ref(),
             true,
+            true,
         )?;
 
         Ok(())
diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
index 24359efc7..3b6e168d3 100644
--- a/src/server/prune_job.rs
+++ b/src/server/prune_job.rs
@@ -75,7 +75,9 @@ pub fn prune_datastore(
                 info.backup_dir.backup_time_string()
             );
             if !keep && !dry_run {
-                if let Err(err) = datastore.remove_backup_dir(ns, info.backup_dir.as_ref(), false) {
+                if let Err(err) =
+                    datastore.remove_backup_dir(ns, info.backup_dir.as_ref(), false, true)
+                {
                     let path = info.backup_dir.relative_path();
                     warn!("failed to remove dir {path:?}: {err}");
                 }
diff --git a/src/server/pull.rs b/src/server/pull.rs
index 7aeb2bd56..623419884 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -503,6 +503,7 @@ async fn pull_snapshot_from<'a>(
                     snapshot.backup_ns(),
                     snapshot.as_ref(),
                     true,
+                    true,
                 ) {
                     info!("cleanup error - {cleanup_err}");
                 }
@@ -677,7 +678,7 @@ async fn pull_group(
             params
                 .target
                 .store
-                .remove_backup_dir(&target_ns, snapshot.as_ref(), false)?;
+                .remove_backup_dir(&target_ns, snapshot.as_ref(), false, true)?;
             sync_stats.add(SyncStats::from(RemovedVanishedStats {
                 snapshots: 1,
                 groups: 0,
@@ -992,10 +993,11 @@ pub(crate) async fn pull_ns(
                     continue;
                 }
                 info!("delete vanished group '{local_group}'");
-                let delete_stats_result = params
-                    .target
-                    .store
-                    .remove_backup_group(&target_ns, local_group);
+                let delete_stats_result =
+                    params
+                        .target
+                        .store
+                        .remove_backup_group(&target_ns, local_group, false);
 
                 match delete_stats_result {
                     Ok(stats) => {
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:28 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:28 +0200
Subject: [pbs-devel] [PATCH v3 proxmox 1/20] pbs api types: add type for
 snapshot list filtering based on trash state
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-2-c.ebner@proxmox.com>

Adds a dedicated enum to allow filtering snapshots based on their
trash state. Allows to include, exclude or only show trashed
snapshots when listing.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-api-types/src/datastore.rs | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 5bd953ac..d50d6f2b 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -1772,6 +1772,20 @@ impl BackupGroupDeleteStats {
     }
 }
 
+#[api()]
+#[derive(Serialize, Deserialize, Copy, Clone, Default, Debug, PartialOrd, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// Filter snapshots based on their trash state.
+pub enum TrashStateFilter {
+    #[default]
+    /// Exclude snapshots which are marked as trash.
+    ExcludeTrash,
+    /// Include snapshots which are marked as trash.
+    IncludeTrash,
+    /// Only include snapshots which are marked as trash.
+    OnlyTrash,
+}
+
 #[derive(Clone, PartialEq, Eq)]
 /// Allowed variants of backup archives to be contained in a snapshot's manifest
 pub enum ArchiveType {
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:32 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:32 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 05/20] datastore: add helpers
 to check if snapshot/group is trash
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-6-c.ebner@proxmox.com>

The convenience helper method check for the trash marker file being
present within the snapshot/group directory. From the absence of the
marker file it may not be inferred that the snapshot/group is present
as regular item however.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index 9ce4cb0f8..ec6d97790 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -99,6 +99,13 @@ impl BackupGroup {
         self.full_group_path().exists()
     }
 
+    /// Check if the group is currently marked as trash by checking the presence of the trash
+    /// marker file in the group's directory
+    pub fn is_trash(&self) -> bool {
+        let path = self.full_group_path().join(TRASH_MARKER_FILENAME);
+        path.exists()
+    }
+
     pub fn list_backups(&self) -> Result<Vec<BackupInfo>, Error> {
         let mut list = vec![];
 
@@ -480,6 +487,13 @@ impl BackupDir {
         path.exists()
     }
 
+    /// Check if the snapshot is currently marked as trash by checking the presence of the trash
+    /// marker file in the snapshot's directory
+    pub fn is_trash(&self) -> bool {
+        let path = self.full_path().join(TRASH_MARKER_FILENAME);
+        path.exists()
+    }
+
     pub fn backup_time_to_string(backup_time: i64) -> Result<String, Error> {
         // fixme: can this fail? (avoid unwrap)
         proxmox_time::epoch_to_rfc3339_utc(backup_time)
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:30 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:30 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 03/20] datastore/api: mark
 snapshots as trash on destroy
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-4-c.ebner@proxmox.com>

In order to implement the trash functionality, mark snapshots as
trash instead of removing them by default. However, provide a
`skip-trash` flag to opt-out and destroy the snapshot including it's
contents immediately.

Trashed snapshots are marked by creating a `.trashed` marker file
inside the snapshot folder. Actual removal of the snapshot will be
deferred to the garbage collection task.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 66 ++++++++++++++++++--------------
 pbs-datastore/src/datastore.rs   |  2 +-
 src/api2/admin/datastore.rs      | 18 ++++++++-
 3 files changed, 55 insertions(+), 31 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index d4732fdd9..76bcd15f5 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -21,6 +21,7 @@ use crate::manifest::{BackupManifest, MANIFEST_LOCK_NAME};
 use crate::{DataBlob, DataStore};
 
 pub const DATASTORE_LOCKS_DIR: &str = "/run/proxmox-backup/locks";
+pub const TRASH_MARKER_FILENAME: &str = ".trashed";
 
 // TODO: Remove with PBS 5
 // Note: The `expect()` call here will only happen if we can neither confirm nor deny the existence
@@ -228,7 +229,7 @@ impl BackupGroup {
                 delete_stats.increment_protected_snapshots();
                 continue;
             }
-            snap.destroy(false)?;
+            snap.destroy(false, false)?;
             delete_stats.increment_removed_snapshots();
         }
 
@@ -575,7 +576,8 @@ impl BackupDir {
     /// Destroy the whole snapshot, bails if it's protected
     ///
     /// Setting `force` to true skips locking and thus ignores if the backup is currently in use.
-    pub fn destroy(&self, force: bool) -> Result<(), Error> {
+    /// Setting `skip_trash` to true will remove the snapshot instead of marking it as trash.
+    pub fn destroy(&self, force: bool, skip_trash: bool) -> Result<(), Error> {
         let (_guard, _manifest_guard);
         if !force {
             _guard = self
@@ -588,37 +590,45 @@ impl BackupDir {
             bail!("cannot remove protected snapshot"); // use special error type?
         }
 
-        let full_path = self.full_path();
-        log::info!("removing backup snapshot {:?}", full_path);
-        std::fs::remove_dir_all(&full_path).map_err(|err| {
-            format_err!("removing backup snapshot {:?} failed - {}", full_path, err,)
-        })?;
+        let mut full_path = self.full_path();
+        log::info!("removing backup snapshot {full_path:?}");
+        if skip_trash {
+            std::fs::remove_dir_all(&full_path).map_err(|err| {
+                format_err!("removing backup snapshot {full_path:?} failed - {err}")
+            })?;
+        } else {
+            full_path.push(TRASH_MARKER_FILENAME);
+            let _trash_file =
+                std::fs::File::create(full_path).context("failed to set trash file")?;
+        }
 
         // remove no longer needed lock files
         let _ = std::fs::remove_file(self.manifest_lock_path()); // ignore errors
         let _ = std::fs::remove_file(self.lock_path()); // ignore errors
 
-        let group = BackupGroup::from(self);
-        let guard = group.lock().with_context(|| {
-            format!("while checking if group '{group:?}' is empty during snapshot destruction")
-        });
-
-        // Only remove the group if all of the following is true:
-        //
-        // - we can lock it: if we can't lock the group, it is still in use (either by another
-        //   backup process or a parent caller (who needs to take care that empty groups are
-        //   removed themselves).
-        // - it is now empty: if the group isn't empty, removing it will fail (to avoid removing
-        //   backups that might still be used).
-        // - the new locking mechanism is used: if the old mechanism is used, a group removal here
-        //   could lead to a race condition.
-        //
-        // Do not error out, as we have already removed the snapshot, there is nothing a user could
-        // do to rectify the situation.
-        if guard.is_ok() && group.list_backups()?.is_empty() && !*OLD_LOCKING {
-            group.remove_group_dir()?;
-        } else if let Err(err) = guard {
-            log::debug!("{err:#}");
+        if skip_trash {
+            let group = BackupGroup::from(self);
+            let guard = group.lock().with_context(|| {
+                format!("while checking if group '{group:?}' is empty during snapshot destruction")
+            });
+
+            // Only remove the group if all of the following is true:
+            //
+            // - we can lock it: if we can't lock the group, it is still in use (either by another
+            //   backup process or a parent caller (who needs to take care that empty groups are
+            //   removed themselves).
+            // - it is now empty: if the group isn't empty, removing it will fail (to avoid removing
+            //   backups that might still be used).
+            // - the new locking mechanism is used: if the old mechanism is used, a group removal here
+            //   could lead to a race condition.
+            //
+            // Do not error out, as we have already removed the snapshot, there is nothing a user could
+            // do to rectify the situation.
+            if guard.is_ok() && group.list_backups()?.is_empty() && !*OLD_LOCKING {
+                group.remove_group_dir()?;
+            } else if let Err(err) = guard {
+                log::debug!("{err:#}");
+            }
         }
 
         Ok(())
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index cbf78ecb6..6df26e825 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -686,7 +686,7 @@ impl DataStore {
     ) -> Result<(), Error> {
         let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
 
-        backup_dir.destroy(force)
+        backup_dir.destroy(force, true)
     }
 
     /// Returns the time of the last successful backup
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 392494488..aa9202ed5 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -402,6 +402,12 @@ pub async fn list_snapshot_files(
                 type: pbs_api_types::BackupDir,
                 flatten: true,
             },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                default: false,
+                description: "Immediately remove the snapshot, not marking it as trash.",
+            },
         },
     },
     access: {
@@ -415,6 +421,7 @@ pub async fn delete_snapshot(
     store: String,
     ns: Option<BackupNamespace>,
     backup_dir: pbs_api_types::BackupDir,
+    skip_trash: bool,
     _info: &ApiMethod,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Value, Error> {
@@ -435,7 +442,7 @@ pub async fn delete_snapshot(
 
         let snapshot = datastore.backup_dir(ns, backup_dir)?;
 
-        snapshot.destroy(false)?;
+        snapshot.destroy(false, skip_trash)?;
 
         Ok(Value::Null)
     })
@@ -979,6 +986,12 @@ pub fn verify(
                 optional: true,
                 description: "Spins up an asynchronous task that does the work.",
             },
+            "skip-trash": {
+                type: bool,
+                optional: true,
+                default: false,
+                description: "Immediately remove the group including all snapshots, not marking it as trash.",
+            },
         },
     },
     returns: pbs_api_types::ADMIN_DATASTORE_PRUNE_RETURN_TYPE,
@@ -995,6 +1008,7 @@ pub fn prune(
     keep_options: KeepOptions,
     store: String,
     ns: Option<BackupNamespace>,
+    skip_trash: bool,
     param: Value,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Value, Error> {
@@ -1098,7 +1112,7 @@ pub fn prune(
             });
 
             if !keep {
-                if let Err(err) = backup_dir.destroy(false) {
+                if let Err(err) = backup_dir.destroy(false, skip_trash) {
                     warn!(
                         "failed to remove dir {:?}: {}",
                         backup_dir.relative_path(),
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:36 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:36 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 09/20] sync: ignore trashed
 snapshots/groups when reading from local source
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-10-c.ebner@proxmox.com>

Trashed snapshots and backup groups should never be synced, so filter
them out when listing the contents to be synced.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/server/sync.rs | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/server/sync.rs b/src/server/sync.rs
index 09814ef0c..a32c914ff 100644
--- a/src/server/sync.rs
+++ b/src/server/sync.rs
@@ -447,6 +447,7 @@ impl SyncSource for LocalSource {
             Some(owner),
         )?
         .filter_map(Result::ok)
+        .filter(|backup_group| !backup_group.is_trash())
         .map(|backup_group| backup_group.group().clone())
         .collect::<Vec<pbs_api_types::BackupGroup>>())
     }
@@ -461,6 +462,7 @@ impl SyncSource for LocalSource {
             .backup_group(namespace.clone(), group.clone())
             .iter_snapshots()?
             .filter_map(Result::ok)
+            .filter(|snapshot| !snapshot.is_trash())
             .map(|snapshot| snapshot.dir().to_owned())
             .collect::<Vec<BackupDir>>())
     }
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:38 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:38 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 11/20] datastore: clear
 trashed snapshot dir if re-creation requested
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-12-c.ebner@proxmox.com>

If a previously trashed snapshot has been requested for re-creation
(e.g. by a sync job in push direction), drop the contents of the
currently trashed snapshot.
The snapshot directory itself is already locked at that point, either
by the old locking mechanism acting on the directory itself or by the
new locking mechanism. Therefore, concurrent operations can be
excluded.

For the call site this acts as if the snapshot directory has been
newly created.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index dc4059789..6f99ff572 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -826,8 +826,9 @@ impl DataStore {
     ) -> Result<(PathBuf, bool, BackupLockGuard), Error> {
         let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
         let relative_path = backup_dir.relative_path();
+        let full_path = backup_dir.full_path();
 
-        match std::fs::create_dir(backup_dir.full_path()) {
+        match std::fs::create_dir(&full_path) {
             Ok(_) => {
                 let guard = backup_dir.lock().with_context(|| {
                     format!("while creating new locked snapshot '{backup_dir:?}'")
@@ -838,6 +839,28 @@ impl DataStore {
                 let guard = backup_dir
                     .lock()
                     .with_context(|| format!("while creating locked snapshot '{backup_dir:?}'"))?;
+
+                if backup_dir.is_trash() {
+                    info!("clear trashed backup snapshot {full_path:?}");
+                    let dir_entries = std::fs::read_dir(&full_path).context(
+                        "failed to read directory contents during cleanup of trashed snapshot",
+                    )?;
+                    for entry in dir_entries {
+                        let entry = entry.context(
+                            "failed to read directory entry during clenup of trashed snapshot",
+                        )?;
+                        // Only expect regular file entries
+                        std::fs::remove_file(entry.path()).context(
+                            "failed to remove directory entry during clenup of trashed snapshot",
+                        )?;
+                    }
+
+                    // Group already untrashed by `create_locked_backup_group`, no further action
+                    // required.
+
+                    return Ok((relative_path, true, guard));
+                }
+
                 Ok((relative_path, false, guard))
             }
             Err(e) => Err(e.into()),
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:39 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:39 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 12/20] datastore: recover
 backup group from trash for new backups
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-13-c.ebner@proxmox.com>

A whole backup group might have been marked as trashed, including all
of the contained snapshots.

A new backup to the trashed group will only be allowed if owner and
user match, restoring the group. Otherwise, fail and do not allow
backups until the group is either removed from trash or permanently
cleared.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 6f99ff572..1bc096420 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -28,7 +28,7 @@ use pbs_api_types::{
 };
 use pbs_config::BackupLockGuard;
 
-use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, OLD_LOCKING};
+use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, OLD_LOCKING, TRASH_MARKER_FILENAME};
 use crate::chunk_store::ChunkStore;
 use crate::dynamic_index::{DynamicIndexReader, DynamicIndexWriter};
 use crate::fixed_index::{FixedIndexReader, FixedIndexWriter};
@@ -809,6 +809,16 @@ impl DataStore {
                 let guard = backup_group.lock().with_context(|| {
                     format!("while creating locked backup group '{backup_group:?}'")
                 })?;
+                if backup_group.is_trash() {
+                    let owner = self.get_owner(ns, backup_group.group())?;
+                    check_backup_owner(&owner, auth_id)?;
+                    info!("remove trash marker for backup group {full_path:?}");
+                    let trash_path = full_path.join(TRASH_MARKER_FILENAME);
+                    std::fs::remove_file(trash_path)
+                        .context("failed to remove trash marker file")?;
+                    return Ok((owner, guard));
+                }
+
                 let owner = self.get_owner(ns, backup_group.group())?; // just to be sure
                 Ok((owner, guard))
             }
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:33 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:33 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 06/20] datastore: allow
 filtering of backups by their trash state
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-7-c.ebner@proxmox.com>

Extends the BackupGroup::list_backups method by an enum parameter to
filter backup snapshots based on their trash state.

This allows to reuse the same logic for listing snapshots including
trashed, excluding trashed or only trashed snapshots.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 34 +++++++++++++++++++++++++++-----
 pbs-datastore/src/datastore.rs   |  4 ++--
 src/api2/admin/datastore.rs      | 15 ++++++++------
 src/api2/tape/backup.rs          |  4 ++--
 src/backup/verify.rs             |  6 +++---
 src/server/prune_job.rs          |  6 +++---
 src/server/pull.rs               |  8 ++++----
 tests/prune.rs                   |  1 +
 8 files changed, 53 insertions(+), 25 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index ec6d97790..53acc5baf 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -12,8 +12,8 @@ use proxmox_sys::fs::{lock_dir_noblock, lock_dir_noblock_shared, replace_file, C
 use proxmox_systemd::escape_unit;
 
 use pbs_api_types::{
-    Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, GroupFilter, VerifyState,
-    BACKUP_DATE_REGEX, BACKUP_FILE_REGEX, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME,
+    Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, GroupFilter, TrashStateFilter,
+    VerifyState, BACKUP_DATE_REGEX, BACKUP_FILE_REGEX, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME,
 };
 use pbs_config::{open_backup_lockfile, BackupLockGuard};
 
@@ -106,7 +106,7 @@ impl BackupGroup {
         path.exists()
     }
 
-    pub fn list_backups(&self) -> Result<Vec<BackupInfo>, Error> {
+    pub fn list_backups(&self, filter: TrashStateFilter) -> Result<Vec<BackupInfo>, Error> {
         let mut list = vec![];
 
         let path = self.full_group_path();
@@ -124,11 +124,26 @@ impl BackupGroup {
                 let files = list_backup_files(l2_fd, backup_time)?;
 
                 let protected = backup_dir.is_protected();
+                let trash = backup_dir.is_trash();
+                match filter {
+                    TrashStateFilter::IncludeTrash => (),
+                    TrashStateFilter::OnlyTrash => {
+                        if !trash {
+                            return Ok(());
+                        }
+                    }
+                    TrashStateFilter::ExcludeTrash => {
+                        if trash {
+                            return Ok(());
+                        }
+                    }
+                }
 
                 list.push(BackupInfo {
                     backup_dir,
                     files,
                     protected,
+                    trash,
                 });
 
                 Ok(())
@@ -139,7 +154,7 @@ impl BackupGroup {
 
     /// Finds the latest backup inside a backup group
     pub fn last_backup(&self, only_finished: bool) -> Result<Option<BackupInfo>, Error> {
-        let backups = self.list_backups()?;
+        let backups = self.list_backups(TrashStateFilter::ExcludeTrash)?;
         Ok(backups
             .into_iter()
             .filter(|item| !only_finished || item.is_finished())
@@ -651,7 +666,12 @@ impl BackupDir {
             //
             // Do not error out, as we have already removed the snapshot, there is nothing a user could
             // do to rectify the situation.
-            if guard.is_ok() && group.list_backups()?.is_empty() && !*OLD_LOCKING {
+            if guard.is_ok()
+                && group
+                    .list_backups(TrashStateFilter::ExcludeTrash)?
+                    .is_empty()
+                && !*OLD_LOCKING
+            {
                 group.remove_group_dir()?;
             } else if let Err(err) = guard {
                 log::debug!("{err:#}");
@@ -801,6 +821,8 @@ pub struct BackupInfo {
     pub files: Vec<String>,
     /// Protection Status
     pub protected: bool,
+    /// Trash state
+    pub trash: bool,
 }
 
 impl BackupInfo {
@@ -809,11 +831,13 @@ impl BackupInfo {
 
         let files = list_backup_files(libc::AT_FDCWD, &path)?;
         let protected = backup_dir.is_protected();
+        let trash = backup_dir.is_trash();
 
         Ok(BackupInfo {
             backup_dir,
             files,
             protected,
+            trash,
         })
     }
 
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index e546bc532..dc4059789 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -24,7 +24,7 @@ use proxmox_worker_task::WorkerTaskContext;
 use pbs_api_types::{
     ArchiveType, Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, ChunkOrder,
     DataStoreConfig, DatastoreFSyncLevel, DatastoreTuning, GarbageCollectionStatus,
-    MaintenanceMode, MaintenanceType, Operation, UPID,
+    MaintenanceMode, MaintenanceType, Operation, TrashStateFilter, UPID,
 };
 use pbs_config::BackupLockGuard;
 
@@ -1158,7 +1158,7 @@ impl DataStore {
                         _ => bail!("exhausted retries and unexpected counter overrun"),
                     };
 
-                    let mut snapshots = match group.list_backups() {
+                    let mut snapshots = match group.list_backups(TrashStateFilter::IncludeTrash) {
                         Ok(snapshots) => snapshots,
                         Err(err) => {
                             if group.exists() {
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index aa9202ed5..a59e39abe 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -42,8 +42,8 @@ use pbs_api_types::{
     DataStoreConfig, DataStoreListItem, DataStoreMountStatus, DataStoreStatus,
     GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
     MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
-    BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
-    BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
+    TrashStateFilter, BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA,
+    BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
     IGNORE_VERIFIED_BACKUPS_SCHEMA, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
     PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
     PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, PRIV_SYS_MODIFY, UPID, UPID_SCHEMA,
@@ -223,7 +223,7 @@ pub fn list_groups(
                 return Ok(group_info);
             }
 
-            let snapshots = match group.list_backups() {
+            let snapshots = match group.list_backups(TrashStateFilter::ExcludeTrash) {
                 Ok(snapshots) => snapshots,
                 Err(_) => return Ok(group_info),
             };
@@ -542,6 +542,7 @@ unsafe fn list_snapshots_blocking(
             time: info.backup_dir.backup_time(),
         };
         let protected = info.protected;
+        let trash = if info.trash { Some(info.trash) } else { None };
 
         match get_all_snapshot_files(&info) {
             Ok((manifest, files)) => {
@@ -578,6 +579,7 @@ unsafe fn list_snapshots_blocking(
                     size,
                     owner,
                     protected,
+                    trash,
                 }
             }
             Err(err) => {
@@ -601,6 +603,7 @@ unsafe fn list_snapshots_blocking(
                     size: None,
                     owner,
                     protected,
+                    trash,
                 }
             }
         }
@@ -624,7 +627,7 @@ unsafe fn list_snapshots_blocking(
             return Ok(snapshots);
         }
 
-        let group_backups = group.list_backups()?;
+        let group_backups = group.list_backups(TrashStateFilter::ExcludeTrash)?;
 
         snapshots.extend(
             group_backups
@@ -657,7 +660,7 @@ async fn get_snapshots_count(
                 Ok(group) => group,
                 Err(_) => return Ok(counts), // TODO: add this as error counts?
             };
-            let snapshot_count = group.list_backups()?.len() as u64;
+            let snapshot_count = group.list_backups(TrashStateFilter::ExcludeTrash)?.len() as u64;
 
             // only include groups with snapshots, counting/displaying empty groups can confuse
             if snapshot_count > 0 {
@@ -1042,7 +1045,7 @@ pub fn prune(
     }
     let mut prune_result: Vec<PruneResult> = Vec::new();
 
-    let list = group.list_backups()?;
+    let list = group.list_backups(TrashStateFilter::ExcludeTrash)?;
 
     let mut prune_info = compute_prune_info(list, &keep_options)?;
 
diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
index 31293a9a9..158905990 100644
--- a/src/api2/tape/backup.rs
+++ b/src/api2/tape/backup.rs
@@ -12,7 +12,7 @@ use proxmox_worker_task::WorkerTaskContext;
 
 use pbs_api_types::{
     print_ns_and_snapshot, print_store_and_ns, Authid, MediaPoolConfig, Operation,
-    TapeBackupJobConfig, TapeBackupJobSetup, TapeBackupJobStatus, JOB_ID_SCHEMA,
+    TapeBackupJobConfig, TapeBackupJobSetup, TapeBackupJobStatus, TrashStateFilter, JOB_ID_SCHEMA,
     PRIV_DATASTORE_READ, PRIV_TAPE_AUDIT, PRIV_TAPE_WRITE, UPID_SCHEMA,
 };
 
@@ -433,7 +433,7 @@ fn backup_worker(
         progress.done_snapshots = 0;
         progress.group_snapshots = 0;
 
-        let snapshot_list = group.list_backups()?;
+        let snapshot_list = group.list_backups(TrashStateFilter::ExcludeTrash)?;
 
         // filter out unfinished backups
         let mut snapshot_list: Vec<_> = snapshot_list
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 3d2cba8ac..bf7affe09 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -11,8 +11,8 @@ use proxmox_worker_task::WorkerTaskContext;
 
 use pbs_api_types::{
     print_ns_and_snapshot, print_store_and_ns, ArchiveType, Authid, BackupNamespace, BackupType,
-    CryptMode, SnapshotVerifyState, VerifyState, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_VERIFY,
-    UPID,
+    CryptMode, SnapshotVerifyState, TrashStateFilter, VerifyState, PRIV_DATASTORE_BACKUP,
+    PRIV_DATASTORE_VERIFY, UPID,
 };
 use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo};
 use pbs_datastore::index::IndexFile;
@@ -411,7 +411,7 @@ pub fn verify_backup_group(
     filter: Option<&dyn Fn(&BackupManifest) -> bool>,
 ) -> Result<Vec<String>, Error> {
     let mut errors = Vec::new();
-    let mut list = match group.list_backups() {
+    let mut list = match group.list_backups(TrashStateFilter::ExcludeTrash) {
         Ok(list) => list,
         Err(err) => {
             info!(
diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
index 1c86647a0..24359efc7 100644
--- a/src/server/prune_job.rs
+++ b/src/server/prune_job.rs
@@ -4,8 +4,8 @@ use anyhow::Error;
 use tracing::{info, warn};
 
 use pbs_api_types::{
-    print_store_and_ns, Authid, KeepOptions, Operation, PruneJobOptions, MAX_NAMESPACE_DEPTH,
-    PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
+    print_store_and_ns, Authid, KeepOptions, Operation, PruneJobOptions, TrashStateFilter,
+    MAX_NAMESPACE_DEPTH, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
 };
 use pbs_datastore::prune::compute_prune_info;
 use pbs_datastore::DataStore;
@@ -54,7 +54,7 @@ pub fn prune_datastore(
     )? {
         let group = group?;
         let ns = group.backup_ns();
-        let list = group.list_backups()?;
+        let list = group.list_backups(TrashStateFilter::ExcludeTrash)?;
 
         let mut prune_info = compute_prune_info(list, &prune_options.keep)?;
         prune_info.reverse(); // delete older snapshots first
diff --git a/src/server/pull.rs b/src/server/pull.rs
index b1724c142..7aeb2bd56 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -12,9 +12,9 @@ use tracing::info;
 
 use pbs_api_types::{
     print_store_and_ns, ArchiveType, Authid, BackupArchiveName, BackupDir, BackupGroup,
-    BackupNamespace, GroupFilter, Operation, RateLimitConfig, Remote, VerifyState,
-    CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, PRIV_DATASTORE_AUDIT,
-    PRIV_DATASTORE_BACKUP,
+    BackupNamespace, GroupFilter, Operation, RateLimitConfig, Remote, TrashStateFilter,
+    VerifyState, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH,
+    PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP,
 };
 use pbs_client::BackupRepository;
 use pbs_config::CachedUserInfo;
@@ -660,7 +660,7 @@ async fn pull_group(
             .target
             .store
             .backup_group(target_ns.clone(), group.clone());
-        let local_list = group.list_backups()?;
+        let local_list = group.list_backups(TrashStateFilter::ExcludeTrash)?;
         for info in local_list {
             let snapshot = info.backup_dir;
             if source_snapshots.contains(&snapshot.backup_time()) {
diff --git a/tests/prune.rs b/tests/prune.rs
index b11449ca0..02e9bc200 100644
--- a/tests/prune.rs
+++ b/tests/prune.rs
@@ -40,6 +40,7 @@ fn create_info(snapshot: &str, partial: bool) -> BackupInfo {
         backup_dir,
         files,
         protected: false,
+        trash: false,
     }
 }
 
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:44 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:44 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 17/20] api: admin: add
 endpoint to clear trashed items from group
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-18-c.ebner@proxmox.com>

Allows to remove only the trashed snapshot items of a backup group,
including the backup group itself if all the contents have been
cleared. Instead of using the backup group delete stats to determine
whether the group directory should be cleaned up or not, use a local
variable instead, as the removed trash is otherwise not correctly
accounted for.

This allows to manually clear trashed groups from the UI for
convenience.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 14 ++++++-
 pbs-datastore/src/datastore.rs   | 17 +++++++-
 src/api2/admin/datastore.rs      | 70 ++++++++++++++++++++++++++++++++
 3 files changed, 97 insertions(+), 4 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index f334600c7..85d856888 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -242,7 +242,11 @@ impl BackupGroup {
     ///
     /// Returns `BackupGroupDeleteStats`, containing the number of deleted snapshots
     /// and number of protected snaphsots, which therefore were not removed.
-    pub fn destroy(&self, skip_trash: bool) -> Result<BackupGroupDeleteStats, Error> {
+    pub fn destroy(
+        &self,
+        skip_trash: bool,
+        trash_only: bool,
+    ) -> Result<BackupGroupDeleteStats, Error> {
         let _guard = self
             .lock()
             .with_context(|| format!("while destroying group '{self:?}'"))?;
@@ -250,10 +254,16 @@ impl BackupGroup {
 
         log::info!("removing backup group {:?}", path);
         let mut delete_stats = BackupGroupDeleteStats::default();
+        let mut cleanup_group_dir = true;
         for snap in self.iter_snapshots()? {
             let snap = snap?;
             if snap.is_protected() {
                 delete_stats.increment_protected_snapshots();
+                cleanup_group_dir = false;
+                continue;
+            }
+            if trash_only && !snap.is_trash() {
+                cleanup_group_dir = false;
                 continue;
             }
             snap.destroy(false, skip_trash)?;
@@ -262,7 +272,7 @@ impl BackupGroup {
 
         // Note: make sure the old locking mechanism isn't used as `remove_dir_all` is not safe in
         // that case
-        if delete_stats.all_removed() && !*OLD_LOCKING {
+        if cleanup_group_dir && !*OLD_LOCKING {
             if skip_trash {
                 self.remove_group_dir()?;
             } else {
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 574d6ec26..fde0096bf 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -581,7 +581,7 @@ impl DataStore {
         let mut stats = BackupGroupDeleteStats::default();
 
         for group in self.iter_backup_groups(ns.to_owned())? {
-            let delete_stats = group?.destroy(true)?;
+            let delete_stats = group?.destroy(true, false)?;
             stats.add(&delete_stats);
             removed_all_groups = removed_all_groups && delete_stats.all_removed();
         }
@@ -675,7 +675,20 @@ impl DataStore {
     ) -> Result<BackupGroupDeleteStats, Error> {
         let backup_group = self.backup_group(ns.clone(), backup_group.clone());
 
-        backup_group.destroy(skip_trash)
+        backup_group.destroy(skip_trash, false)
+    }
+
+    /// Remove snapshots marked as trash from a backup group, including the group if it is empty
+    /// afterwards.
+    ///
+    /// Returns `BackupGroupDeleteStats`, containing the number of deleted snapshots.
+    pub fn clear_backup_group(
+        self: &Arc<Self>,
+        ns: &BackupNamespace,
+        backup_group: &pbs_api_types::BackupGroup,
+    ) -> Result<BackupGroupDeleteStats, Error> {
+        let backup_group = self.backup_group(ns.clone(), backup_group.clone());
+        backup_group.destroy(true, true)
     }
 
     /// Remove a backup directory including all content
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index bc2d51612..f97aeb5cb 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -2868,6 +2868,72 @@ fn do_recover_snapshot(snapshot_dir: &BackupDir) -> Result<(), Error> {
     Ok(())
 }
 
+#[api(
+    input: {
+        properties: {
+            store: { schema: DATASTORE_SCHEMA },
+            ns: {
+                type: BackupNamespace,
+                optional: true,
+            },
+            "backup-type": {
+                optional: true,
+                type: BackupType,
+            },
+            "backup-id": {
+                optional: true,
+                schema: BACKUP_ID_SCHEMA,
+            },
+        },
+    },
+    returns: {
+        type: BackupGroupDeleteStats,
+    },
+    access: {
+        permission: &Permission::Anybody,
+        description: "Requires on /datastore/{store}[/{namespace}] either DATASTORE_MODIFY for any \
+            or DATASTORE_PRUNE and being the owner of the group",
+    },
+)]
+/// Clear trash items in a namespace or backup group including the group itself it is marked as trash.
+pub async fn clear_trash(
+    store: String,
+    ns: Option<BackupNamespace>,
+    backup_type: Option<BackupType>,
+    backup_id: Option<String>,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<BackupGroupDeleteStats, Error> {
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+
+    tokio::task::spawn_blocking(move || {
+        let ns = ns.unwrap_or_default();
+        let limited = check_ns_privs_full(
+            &store,
+            &ns,
+            &auth_id,
+            PRIV_DATASTORE_MODIFY,
+            PRIV_DATASTORE_PRUNE,
+        )?;
+        let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
+
+        let groups = groups_by_type_or_id(datastore.clone(), &ns, backup_type, backup_id)?;
+        let mut delete_stats = BackupGroupDeleteStats::default();
+        for group in groups {
+            if limited {
+                let owner = datastore.get_owner(&ns, group.group())?;
+                if check_backup_owner(&owner, &auth_id).is_err() {
+                    continue;
+                }
+            }
+            let stats = datastore.clear_backup_group(&ns, group.group())?;
+            delete_stats.add(&stats);
+        }
+
+        Ok(delete_stats)
+    })
+    .await?
+}
+
 #[sortable]
 const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
     (
@@ -2879,6 +2945,10 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
         "change-owner",
         &Router::new().post(&API_METHOD_SET_BACKUP_OWNER),
     ),
+    (
+        "clear-trash",
+        &Router::new().delete(&API_METHOD_CLEAR_TRASH),
+    ),
     (
         "download",
         &Router::new().download(&API_METHOD_DOWNLOAD_FILE),
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:52:45 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:52:45 +0200
Subject: [pbs-devel] [PATCH v3 proxmox-backup 18/20] ui: add recover for
 trashed items tab to datastore panel
In-Reply-To: <20250513135247.644260-1-c.ebner@proxmox.com>
References: <20250513135247.644260-1-c.ebner@proxmox.com>
Message-ID: <20250513135247.644260-19-c.ebner@proxmox.com>

Display a dedicated recover trashed tab which allows to inspect and
recover trashed items.

This is based on the pre-existing contents tab but  drops any actions
which make no sense for the given context, such as editing of group
ownership, notes, verification, ecc.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/Makefile                    |   1 +
 www/Utils.js                    |   1 +
 www/datastore/Panel.js          |   8 +
 www/datastore/RecoverTrashed.js | 805 ++++++++++++++++++++++++++++++++
 4 files changed, 815 insertions(+)
 create mode 100644 www/datastore/RecoverTrashed.js

diff --git a/www/Makefile b/www/Makefile
index 44c5fa133..aa8955460 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -115,6 +115,7 @@ JSSRC=							\
 	datastore/Panel.js				\
 	datastore/DataStoreListSummary.js		\
 	datastore/DataStoreList.js			\
+	datastore/RecoverTrashed.js			\
 	ServerStatus.js					\
 	ServerAdministration.js				\
 	NodeNotes.js				        \
diff --git a/www/Utils.js b/www/Utils.js
index 9dcde6941..37523deb0 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -404,6 +404,7 @@ Ext.define('PBS.Utils', {
 	    backup: (type, id) => PBS.Utils.render_datastore_worker_id(id, gettext('Backup')),
 	    'barcode-label-media': [gettext('Drive'), gettext('Barcode-Label Media')],
 	    'catalog-media': [gettext('Drive'), gettext('Catalog Media')],
+	    "clear-trash": [gettext('Trash'), gettext('Clear Trashed Items of Group')],
 	    'delete-datastore': [gettext('Datastore'), gettext('Remove Datastore')],
 	    'delete-namespace': [gettext('Namespace'), gettext('Remove Namespace')],
 	    dircreate: [gettext('Directory Storage'), gettext('Create')],
diff --git a/www/datastore/Panel.js b/www/datastore/Panel.js
index ad9fc10fe..386b62284 100644
--- a/www/datastore/Panel.js
+++ b/www/datastore/Panel.js
@@ -99,6 +99,14 @@ Ext.define('PBS.DataStorePanel', {
 		datastore: '{datastore}',
 	    },
 	},
+	{
+	    xtype: 'pbsDataStoreRecoverTrashed',
+	    itemId: 'trashed',
+	    iconCls: 'fa fa-rotate-left',
+	    cbind: {
+		datastore: '{datastore}',
+	    },
+	},
     ],
 
     initComponent: function() {
diff --git a/www/datastore/RecoverTrashed.js b/www/datastore/RecoverTrashed.js
new file mode 100644
index 000000000..b730f6fd5
--- /dev/null
+++ b/www/datastore/RecoverTrashed.js
@@ -0,0 +1,805 @@
+Ext.define('PBS.DataStoreRecoverTrashed', {
+    extend: 'Ext.tree.Panel',
+    alias: 'widget.pbsDataStoreRecoverTrashed',
+    mixins: ['Proxmox.Mixin.CBind'],
+
+    rootVisible: false,
+
+    title: gettext('Recover Trashed'),
+
+    controller: {
+	xclass: 'Ext.app.ViewController',
+
+	init: function(view) {
+	    if (!view.datastore) {
+		throw "no datastore specified";
+	    }
+
+	    this.store = Ext.create('Ext.data.Store', {
+		model: 'pbs-data-store-snapshots',
+		groupField: 'backup-group',
+	    });
+	    this.store.on('load', this.onLoad, this);
+
+	    view.getStore().setSorters([
+		'sortWeight',
+		'text',
+		'backup-time',
+	    ]);
+	},
+
+	control: {
+	    '#': { // view
+		rowdblclick: 'rowDoubleClicked',
+	    },
+	    'pbsNamespaceSelector': {
+		change: 'nsChange',
+	    },
+	},
+
+	rowDoubleClicked: function(table, rec, el, rowId, ev) {
+	    if (rec?.data?.ty === 'ns' && !rec.data.root) {
+		this.nsChange(null, rec.data.ns);
+	    }
+	},
+
+	nsChange: function(field, value) {
+	    let view = this.getView();
+	    if (field === null) {
+		field = view.down('pbsNamespaceSelector');
+		field.setValue(value);
+		return;
+	    }
+	    view.namespace = value;
+	    this.reload();
+	},
+
+	reload: function() {
+	    let view = this.getView();
+
+	    if (!view.store || !this.store) {
+		console.warn('cannot reload, no store(s)');
+		return;
+	    }
+
+	    let url = `/api2/json/admin/datastore/${view.datastore}/snapshots?trash-state=only-trash`;
+	    if (view.namespace && view.namespace !== '') {
+		url += `&ns=${encodeURIComponent(view.namespace)}`;
+	    }
+	    this.store.setProxy({
+		type: 'proxmox',
+		timeout: 5*60*1000,
+		url: url,
+	    });
+
+	    this.store.load();
+	},
+
+	getRecordGroups: function(records) {
+	    let groups = {};
+
+	    for (const item of records) {
+		var btype = item.data["backup-type"];
+		let group = btype + "/" + item.data["backup-id"];
+
+		if (groups[group] !== undefined) {
+		    continue;
+		}
+
+		var cls = PBS.Utils.get_type_icon_cls(btype);
+		if (cls === "") {
+		    console.warn(`got unknown backup-type '${btype}'`);
+		    continue;
+		}
+
+		groups[group] = {
+		    text: group,
+		    leaf: false,
+		    iconCls: "fa " + cls,
+		    expanded: false,
+		    backup_type: item.data["backup-type"],
+		    backup_id: item.data["backup-id"],
+		    children: [],
+		};
+	    }
+
+	    return groups;
+	},
+
+	loadNamespaceFromSameLevel: async function() {
+	    let view = this.getView();
+	    try {
+		let url = `/api2/extjs/admin/datastore/${view.datastore}/namespace?max-depth=1`;
+		if (view.namespace && view.namespace !== '') {
+		    url += `&parent=${encodeURIComponent(view.namespace)}`;
+		}
+		let { result: { data: ns } } = await Proxmox.Async.api2({ url });
+		return ns;
+	    } catch (err) {
+		console.debug(err);
+	    }
+	    return [];
+	},
+
+	onLoad: async function(store, records, success, operation) {
+	    let me = this;
+	    let view = this.getView();
+
+	    let namespaces = await me.loadNamespaceFromSameLevel();
+
+	    if (!success) {
+		if (namespaces.length === 0) {
+		    let error = Proxmox.Utils.getResponseErrorMessage(operation.getError());
+		    Proxmox.Utils.setErrorMask(view.down('treeview'), error);
+		    return;
+		} else {
+		    records = [];
+		}
+	    } else {
+		Proxmox.Utils.setErrorMask(view.down('treeview'));
+	    }
+
+	    let groups = this.getRecordGroups(records);
+
+	    let selected;
+	    let expanded = {};
+
+	    view.getSelection().some(function(item) {
+		let id = item.data.text;
+		if (item.data.leaf) {
+		    id = item.parentNode.data.text + id;
+		}
+		selected = id;
+		return true;
+	    });
+
+	    view.getRootNode().cascadeBy({
+		before: item => {
+		    if (item.isExpanded() && !item.data.leaf) {
+			let id = item.data.text;
+			expanded[id] = true;
+			return true;
+		    }
+		    return false;
+		},
+		after: Ext.emptyFn,
+	    });
+
+	    for (const item of records) {
+		let group = item.data["backup-type"] + "/" + item.data["backup-id"];
+		let children = groups[group].children;
+
+		let data = item.data;
+
+		data.text = group + '/' + PBS.Utils.render_datetime_utc(data["backup-time"]);
+		data.leaf = false;
+		data.cls = 'no-leaf-icons';
+		data.matchesFilter = true;
+		data.ty = 'dir';
+
+		data.expanded = !!expanded[data.text];
+
+		data.children = [];
+		for (const file of data.files) {
+		    file.text = file.filename;
+		    file['crypt-mode'] = PBS.Utils.cryptmap.indexOf(file['crypt-mode']);
+		    file.fingerprint = data.fingerprint;
+		    file.leaf = true;
+		    file.matchesFilter = true;
+		    file.ty = 'file';
+
+		    data.children.push(file);
+		}
+
+		children.push(data);
+	    }
+
+	    let children = [];
+	    for (const [name, group] of Object.entries(groups)) {
+		let last_backup = 0;
+		let crypt = {
+		    none: 0,
+		    mixed: 0,
+		    'sign-only': 0,
+		    encrypt: 0,
+		};
+		for (let item of group.children) {
+		    crypt[PBS.Utils.cryptmap[item['crypt-mode']]]++;
+		    if (item["backup-time"] > last_backup && item.size !== null) {
+			last_backup = item["backup-time"];
+			group["backup-time"] = last_backup;
+			group["last-comment"] = item.comment;
+			group.files = item.files;
+			group.size = item.size;
+			group.owner = item.owner;
+		    }
+		}
+		group.count = group.children.length;
+		group.matchesFilter = true;
+		crypt.count = group.count;
+		group['crypt-mode'] = PBS.Utils.calculateCryptMode(crypt);
+		group.expanded = !!expanded[name];
+		group.sortWeight = 0;
+		group.ty = 'group';
+		children.push(group);
+	    }
+
+	    for (const item of namespaces) {
+		if (item.ns === view.namespace || (!view.namespace && item.ns === '')) {
+		    continue;
+		}
+		children.push({
+		    text: item.ns,
+		    iconCls: 'fa fa-object-group',
+		    expanded: true,
+		    expandable: false,
+		    ns: (view.namespaces ?? '') !== '' ? `/${item.ns}` : item.ns,
+		    ty: 'ns',
+		    sortWeight: 10,
+		    leaf: true,
+		});
+	    }
+
+	    let isRootNS = !view.namespace || view.namespace === '';
+	    let rootText = isRootNS
+		? gettext('Root Namespace')
+		: Ext.String.format(gettext("Namespace '{0}'"), view.namespace);
+
+	    let topNodes = [];
+	    if (!isRootNS) {
+		let parentNS = view.namespace.split('/').slice(0, -1).join('/');
+		topNodes.push({
+		    text: `.. (${parentNS === '' ? gettext('Root') : parentNS})`,
+		    iconCls: 'fa fa-level-up',
+		    ty: 'ns',
+		    ns: parentNS,
+		    sortWeight: -10,
+		    leaf: true,
+		});
+	    }
+	    topNodes.push({
+		text: rootText,
+		iconCls: "fa fa-" + (isRootNS ? 'database' : 'object-group'),
+		expanded: true,
+		expandable: false,
+		sortWeight: -5,
+		root: true, // fake root
+		isRootNS,
+		ty: 'ns',
+		children: children,
+	    });
+
+	    view.setRootNode({
+		expanded: true,
+		children: topNodes,
+	    });
+
+	    if (!children.length) {
+		view.setEmptyText(Ext.String.format(
+		    gettext('No accessible snapshots found in namespace {0}'),
+		    view.namespace && view.namespace !== '' ? `'${view.namespace}'`: gettext('Root'),
+		));
+	    }
+
+	    if (selected !== undefined) {
+		let selection = view.getRootNode().findChildBy(function(item) {
+		    let id = item.data.text;
+		    if (item.data.leaf) {
+			id = item.parentNode.data.text + id;
+		    }
+		    return selected === id;
+		}, undefined, true);
+		if (selection) {
+		    view.setSelection(selection);
+		    view.getView().focusRow(selection);
+		}
+	    }
+
+	    Proxmox.Utils.setErrorMask(view, false);
+	    if (view.getStore().getFilters().length > 0) {
+		let searchBox = me.lookup("searchbox");
+		let searchvalue = searchBox.getValue();
+		me.search(searchBox, searchvalue);
+	    }
+	},
+
+	recoverGroup: function(data) {
+	    let me = this;
+	    let view = me.getView();
+
+	    let params = {
+		"backup-type": data.backup_type,
+		"backup-id": data.backup_id,
+	    };
+	    if (view.namespace && view.namespace !== '') {
+		params.ns = view.namespace;
+	    }
+
+	    Ext.Msg.show({
+		title: gettext('Confirm'),
+		icon: Ext.Msg.WARNING,
+		message: Ext.String.format(gettext('Are you sure you want to recover group {0}'), `'${data.text}'`),
+		buttons: Ext.Msg.YESNO,
+		defaultFocus: 'yes',
+		callback: function(btn) {
+		    if (btn !== 'yes') {
+		        return;
+		    }
+
+		    Proxmox.Utils.API2Request({
+			url: `/admin/datastore/${view.datastore}/recover-group`,
+			params,
+			method: 'PUT',
+			waitMsgTarget: view,
+			failure: function(response, opts) {
+			    Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+			},
+			callback: me.reload.bind(me),
+		    });
+		},
+	    });
+	},
+
+	forgetGroup: function(data) {
+	    let me = this;
+	    let view = me.getView();
+
+	    let params = {
+		"backup-type": data.backup_type,
+		"backup-id": data.backup_id,
+	    };
+	    if (view.namespace && view.namespace !== '') {
+		params.ns = view.namespace;
+	    }
+
+	    Ext.create('Proxmox.window.SafeDestroy', {
+		url: `/admin/datastore/${view.datastore}/clear-trash`,
+		params,
+		item: {
+		    id: data.text,
+		},
+		autoShow: true,
+		taskName: 'clear-trash',
+		listeners: {
+		    destroy: () => me.reload(),
+		},
+	    });
+	},
+
+	recoverSnapshot: function(data) {
+	    let me = this;
+	    let view = me.getView();
+
+	    Ext.Msg.show({
+		title: gettext('Confirm'),
+		icon: Ext.Msg.WARNING,
+		message: Ext.String.format(gettext('Are you sure you want to recover snapshot {0}'), `'${data.text}'`),
+		buttons: Ext.Msg.YESNO,
+		defaultFocus: 'yes',
+		callback: function(btn) {
+		    if (btn !== 'yes') {
+		        return;
+		    }
+		    let params = {
+			"backup-type": data["backup-type"],
+			"backup-id": data["backup-id"],
+			"backup-time": (data['backup-time'].getTime()/1000).toFixed(0),
+		    };
+		    if (view.namespace && view.namespace !== '') {
+			params.ns = view.namespace;
+		    }
+
+		    //TODO adapt to recover api endpoint
+		    Proxmox.Utils.API2Request({
+			url: `/admin/datastore/${view.datastore}/recover-snapshot`,
+			params,
+			method: 'PUT',
+			waitMsgTarget: view,
+			failure: function(response, opts) {
+			    Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+			},
+			callback: me.reload.bind(me),
+		    });
+		},
+	    });
+	},
+
+	forgetSnapshot: function(data) {
+	    let me = this;
+	    let view = me.getView();
+
+	    Ext.Msg.show({
+		title: gettext('Confirm'),
+		icon: Ext.Msg.WARNING,
+		message: Ext.String.format(gettext('Are you sure you want to remove snapshot {0}'), `'${data.text}'`),
+		buttons: Ext.Msg.YESNO,
+		defaultFocus: 'no',
+		callback: function(btn) {
+		    if (btn !== 'yes') {
+		        return;
+		    }
+		    let params = {
+			"backup-type": data["backup-type"],
+			"backup-id": data["backup-id"],
+			"backup-time": (data['backup-time'].getTime()/1000).toFixed(0),
+			"skip-trash": true,
+		    };
+		    if (view.namespace && view.namespace !== '') {
+			params.ns = view.namespace;
+		    }
+
+		    Proxmox.Utils.API2Request({
+			url: `/admin/datastore/${view.datastore}/snapshots`,
+			params,
+			method: 'DELETE',
+			waitMsgTarget: view,
+			failure: function(response, opts) {
+			    Ext.Msg.alert(gettext('Error'), response.htmlStatus);
+			},
+			callback: me.reload.bind(me),
+		    });
+		},
+	    });
+	},
+
+	onRecover: function(table, rI, cI, item, e, { data }) {
+	    let me = this;
+	    let view = this.getView();
+	    if ((data.ty !== 'group' && data.ty !== 'dir') || !view.datastore) {
+		return;
+	    }
+
+	    if (data.ty === 'dir') {
+		me.recoverSnapshot(data);
+	    } else {
+		me.recoverGroup(data);
+	    }
+	},
+
+	onForget: function(table, rI, cI, item, e, { data }) {
+	    let me = this;
+	    let view = this.getView();
+	    if ((data.ty !== 'group' && data.ty !== 'dir') || !view.datastore) {
+		return;
+	    }
+
+	    if (data.ty === 'dir') {
+		me.forgetSnapshot(data);
+	    } else {
+		me.forgetGroup(data);
+	    }
+	},
+
+	// opens a namespace browser
+	openBrowser: function(tv, rI, Ci, item, e, rec) {
+	    let me = this;
+	    if (rec.data.ty === 'ns') {
+		me.nsChange(null, rec.data.ns);
+	    }
+	},
+
+	filter: function(item, value) {
+	    if (item.data.text.indexOf(value) !== -1) {
+		return true;
+	    }
+
+	    if (item.data.owner && item.data.owner.indexOf(value) !== -1) {
+		return true;
+	    }
+
+	    return false;
+	},
+
+	search: function(tf, value) {
+	    let me = this;
+	    let view = me.getView();
+	    let store = view.getStore();
+	    if (!value && value !== 0) {
+		store.clearFilter();
+		// only collapse the children below our toplevel namespace "root"
+		store.getRoot().lastChild.collapseChildren(true);
+		tf.triggers.clear.setVisible(false);
+		return;
+	    }
+	    tf.triggers.clear.setVisible(true);
+	    if (value.length < 2) return;
+	    Proxmox.Utils.setErrorMask(view, true);
+	    // we do it a little bit later for the error mask to work
+	    setTimeout(function() {
+		store.clearFilter();
+		store.getRoot().collapseChildren(true);
+
+		store.beginUpdate();
+		store.getRoot().cascadeBy({
+		    before: function(item) {
+			if (me.filter(item, value)) {
+			    item.set('matchesFilter', true);
+			    if (item.parentNode && item.parentNode.id !== 'root') {
+				item.parentNode.childmatches = true;
+			    }
+			    return false;
+			}
+			return true;
+		    },
+		    after: function(item) {
+			if (me.filter(item, value) || item.id === 'root' || item.childmatches) {
+			    item.set('matchesFilter', true);
+			    if (item.parentNode && item.parentNode.id !== 'root') {
+				item.parentNode.childmatches = true;
+			    }
+			    if (item.childmatches) {
+				item.expand();
+			    }
+			} else {
+			    item.set('matchesFilter', false);
+			}
+			delete item.childmatches;
+		    },
+		});
+		store.endUpdate();
+
+		store.filter((item) => !!item.get('matchesFilter'));
+		Proxmox.Utils.setErrorMask(view, false);
+	    }, 10);
+	},
+    },
+
+    listeners: {
+	activate: function() {
+	    let me = this;
+	    me.getController().reload();
+	},
+	itemcontextmenu: function(panel, record, item, index, event) {
+	    event.stopEvent();
+	    let title;
+	    let view = panel.up('pbsDataStoreRecoverTrashed');
+	    let controller = view.getController();
+	    let createControllerCallback = function(name) {
+		return function() {
+		    controller[name](view, undefined, undefined, undefined, undefined, record);
+		};
+	    };
+	    if (record.data.ty === 'group') {
+		title = gettext('Group');
+	    } else if (record.data.ty === 'dir') {
+		title = gettext('Snapshot');
+	    }
+	    if (title) {
+		let menu = Ext.create('PBS.datastore.RecoverTrashedContextMenu', {
+		    title: title,
+		    onRecover: createControllerCallback('onRecover'),
+		    onForget: createControllerCallback('onForget'),
+		});
+		menu.showAt(event.getXY());
+	    }
+	},
+    },
+
+    columns: [
+	{
+	    xtype: 'treecolumn',
+	    header: gettext("Backup Group"),
+	    dataIndex: 'text',
+	    renderer: (value, meta, record) => {
+		if (record.data.protected) {
+		    return `${value} (${gettext('protected')})`;
+		}
+		return value;
+	    },
+	    flex: 1,
+	},
+	{
+	    text: gettext('Comment'),
+	    dataIndex: 'comment',
+	    flex: 1,
+	    renderer: (v, meta, record) => {
+		let data = record.data;
+		if (!data || data.leaf || data.root) {
+		    return '';
+		}
+
+		let additionalClasses = "";
+		if (!v) {
+		    if (!data.expanded) {
+			v = data['last-comment'] ?? '';
+			additionalClasses = 'pmx-opacity-75';
+		    } else {
+			v = '';
+		    }
+		}
+		v = Ext.String.htmlEncode(v);
+		return `<span class="snapshot-comment-column ${additionalClasses}">${v}</span>`;
+	    },
+	},
+	{
+	    header: gettext('Actions'),
+	    xtype: 'actioncolumn',
+	    dataIndex: 'text',
+	    width: 100,
+	    items: [
+		{
+		    handler: 'onRecover',
+		    getTip: (v, m, { data }) => {
+			let tip = '{0}';
+			if (data.ty === 'dir') {
+			    tip = gettext("Recover trashed snapshot '{0}'");
+			} else if (data.ty === 'group') {
+			    tip = gettext("Recover trashed items of group '{0}'");
+			}
+			return Ext.String.format(tip, v);
+		    },
+		    getClass: (v, m, { data }) =>
+			data.ty === 'group' || data.ty === 'dir'
+		        ? 'fa fa-rotate-left'
+		        : 'pmx-hidden',
+		    isActionDisabled: (v, r, c, i, { data }) => false,
+		},
+		'->',
+		{
+		    handler: 'onForget',
+		    getTip: (v, m, { data }) => {
+			let tip = '{0}';
+			if (data.ty === 'dir') {
+			    tip = gettext("Permanently forget trashed snapshot '{0}'");
+			} else if (data.ty === 'group') {
+			    tip = gettext("Permanently forget trashed items of group '{0}'");
+			}
+			return Ext.String.format(tip, v);
+		    },
+		    getClass: (v, m, { data }) =>
+		       data.ty === 'group' || data.ty === 'dir'
+		        ? 'fa critical fa-trash-o'
+		        : 'pmx-hidden',
+		    isActionDisabled: (v, r, c, i, { data }) => false,
+		},
+		{
+		    handler: 'openBrowser',
+		    tooltip: gettext('Browse'),
+		    getClass: (v, m, { data }) =>
+			data.ty === 'ns' && !data.root
+			? 'fa fa-folder-open-o'
+			: 'pmx-hidden',
+		    isActionDisabled: (v, r, c, i, { data }) => data.ty !== 'ns',
+		},
+	    ],
+	},
+	{
+	    xtype: 'datecolumn',
+	    header: gettext('Backup Time'),
+	    sortable: true,
+	    dataIndex: 'backup-time',
+	    format: 'Y-m-d H:i:s',
+	    width: 150,
+	},
+	{
+	    header: gettext("Size"),
+	    sortable: true,
+	    dataIndex: 'size',
+	    renderer: (v, meta, { data }) => {
+		if ((data.text === 'client.log.blob' && v === undefined) || (data.ty !== 'dir' && data.ty !== 'file')) {
+		    return '';
+		}
+		if (v === undefined || v === null) {
+		    meta.tdCls = "x-grid-row-loading";
+		    return '';
+		}
+		return Proxmox.Utils.format_size(v);
+	    },
+	},
+	{
+	    xtype: 'numbercolumn',
+	    format: '0',
+	    header: gettext("Count"),
+	    sortable: true,
+	    width: 75,
+	    align: 'right',
+	    dataIndex: 'count',
+	},
+	{
+	    header: gettext("Owner"),
+	    sortable: true,
+	    dataIndex: 'owner',
+	},
+	{
+	    header: gettext('Encrypted'),
+	    dataIndex: 'crypt-mode',
+	    renderer: (v, meta, record) => {
+		if (record.data.size === undefined || record.data.size === null) {
+		    return '';
+		}
+		if (v === -1) {
+		    return '';
+		}
+		let iconCls = PBS.Utils.cryptIconCls[v] || '';
+		let iconTxt = "";
+		if (iconCls) {
+		    iconTxt = `<i class="fa fa-fw fa-${iconCls}"></i> `;
+		}
+		let tip;
+		if (v !== PBS.Utils.cryptmap.indexOf('none') && record.data.fingerprint !== undefined) {
+		    tip = "Key: " + PBS.Utils.renderKeyID(record.data.fingerprint);
+		}
+		let txt = (iconTxt + PBS.Utils.cryptText[v]) || Proxmox.Utils.unknownText;
+		if (record.data.ty === 'group' || tip === undefined) {
+		    return txt;
+		} else {
+		    return `<span data-qtip="${tip}">${txt}</span>`;
+		}
+	    },
+	},
+    ],
+
+    tbar: [
+	{
+	    text: gettext('Reload'),
+	    iconCls: 'fa fa-refresh',
+	    handler: 'reload',
+	},
+	'->',
+	{
+	    xtype: 'tbtext',
+	    html: gettext('Namespace') + ':',
+	},
+	{
+	    xtype: 'pbsNamespaceSelector',
+	    width: 200,
+	    cbind: {
+		datastore: '{datastore}',
+	    },
+	},
+	'-',
+	{
+	    xtype: 'tbtext',
+	    html: gettext('Search'),
+	},
+	{
+	    xtype: 'textfield',
+	    reference: 'searchbox',
+	    emptyText: gettext('group, date or owner'),
+	    triggers: {
+		clear: {
+		    cls: 'pmx-clear-trigger',
+		    weight: -1,
+		    hidden: true,
+		    handler: function() {
+			this.triggers.clear.setVisible(false);
+			this.setValue('');
+		    },
+		},
+	    },
+	    listeners: {
+		change: {
+		    fn: 'search',
+		    buffer: 500,
+		},
+	    },
+	},
+    ],
+});
+
+Ext.define('PBS.datastore.RecoverTrashedContextMenu', {
+    extend: 'Ext.menu.Menu',
+    mixins: ['Proxmox.Mixin.CBind'],
+
+    onRecover: undefined,
+    onForget: undefined,
+
+    items: [
+	{
+	    text: gettext('Recover'),
+	    iconCls: 'fa fa-rotate-left',
+	    handler: function() { this.up('menu').onRecover(); },
+	    cbind: {
+		hidden: '{!onRecover}',
+	    },
+	},
+	{
+	    text: gettext('Remove'),
+	    iconCls: 'fa critical fa-trash-o',
+	    handler: function() { this.up('menu').onForget(); },
+	    cbind: {
+		hidden: '{!onForget}',
+	    },
+	},
+    ],
+});
-- 
2.39.5




From c.ebner at proxmox.com  Tue May 13 15:54:21 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Tue, 13 May 2025 15:54:21 +0200
Subject: [pbs-devel] superseded: [RFC v2 proxmox-backup 00/21] implement
 trash bin functionality
In-Reply-To: <20250508130555.494782-1-c.ebner@proxmox.com>
References: <20250508130555.494782-1-c.ebner@proxmox.com>
Message-ID: <b94a8c50-b5cf-48dc-8b6a-25d62195c441@proxmox.com>

superseded-by version 3:
https://lore.proxmox.com/pbs-devel/20250513135247.644260-1-c.ebner at proxmox.com/T/



From h.laimer at proxmox.com  Thu May 15 13:33:41 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Thu, 15 May 2025 13:33:41 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 4/7] api: admin: trigger sync
 jobs only on datastore mount
In-Reply-To: <618be27b-49fe-4bd5-b980-211ef7d93efc@proxmox.com>
References: <20250116064543.74619-1-h.laimer@proxmox.com>
 <20250116064543.74619-5-h.laimer@proxmox.com>
 <618be27b-49fe-4bd5-b980-211ef7d93efc@proxmox.com>
Message-ID: <33b47879-fadf-46f6-94ee-135560cda8e5@proxmox.com>



On 2/4/25 15:34, Christian Ebner wrote:
> This patch should rather be placed as preparatory patch before patch 3 
> and changes to `do_sync_jobs` directly included in the previous patch.
> 

Maybe, but these changes only really make sense in this context. If done
before it would be something like
```
           move |_worker| async move {
-            do_mount_device(datastore.clone())?;
+            if !do_mount_device(datastore.clone())? {
+                return Ok(());
+            }
	  }
```
I feel like having this change in the context of where it is needed does
add some value. Especially when one tries to find out later why this was
needed. Having code under the `return Ok(())` is pretty clear in terms
of what it is we try to skip here in some cases. Without I could imagine
being like "th was that good for?", then having to go through the commit
message. IMOH, having the extra context here is better than a possibly
better order. But no hard feelings though, if you'd really like this one
before I'll move it for v2.

> On 1/16/25 07:45, Hannes Laimer wrote:
>> Ensure sync jobs are triggered only when the datastore is actually
>> mounted. If the datastore is already mounted, we don't fail,
>> but sync jobs should not be re-triggered unnecessarily. This change
>> prevents redundant sync job execution.
>>
>> Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
>> ---
>> ? src/api2/admin/datastore.rs | 10 ++++++----
>> ? 1 file changed, 6 insertions(+), 4 deletions(-)
>>
>> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
>> index 21b58391d..e29ff9b99 100644
>> --- a/src/api2/admin/datastore.rs
>> +++ b/src/api2/admin/datastore.rs
>> @@ -2445,14 +2445,14 @@ fn setup_mounted_device(datastore: 
>> &DataStoreConfig, tmp_mount_path: &str) -> Re
>> ? /// The reason for the randomized device mounting paths is to avoid 
>> two tasks trying to mount to
>> ? /// the same path, this is *very* unlikely since the device is only 
>> mounted really shortly, but
>> ? /// technically possible.
>> -pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), 
>> Error> {
>> +pub fn do_mount_device(datastore: DataStoreConfig) -> Result<bool, 
>> Error> {
>> ????? if let Some(uuid) = datastore.backing_device.as_ref() {
>> ????????? if pbs_datastore::get_datastore_mount_status(&datastore) == 
>> Some(true) {
>> ????????????? info!(
>> ????????????????? "device is already mounted at '{}'",
>> ????????????????? datastore.absolute_path()
>> ????????????? );
>> -??????????? return Ok(());
>> +??????????? return Ok(false);
>> ????????? }
>> ????????? let tmp_mount_path = format!(
>> ????????????? "{}/{:x}",
>> @@ -2497,7 +2497,7 @@ pub fn do_mount_device(datastore: 
>> DataStoreConfig) -> Result<(), Error> {
>> ????????????? datastore.name
>> ????????? )
>> ????? }
>> -??? Ok(())
>> +??? Ok(true)
>> ? }
>> ? async fn do_sync_jobs(
>> @@ -2582,7 +2582,9 @@ pub fn mount(store: String, rpcenv: &mut dyn 
>> RpcEnvironment) -> Result<Value, Er
>> ????????? auth_id.to_string(),
>> ????????? to_stdout,
>> ????????? move |_worker| async move {
>> -??????????? do_mount_device(datastore.clone())?;
>> +??????????? if !do_mount_device(datastore.clone())? {
>> +??????????????? return Ok(());
>> +??????????? }
>> ????????????? let Ok((sync_config, _digest)) = 
>> pbs_config::sync::config() else {
>> ????????????????? warn!("unable to read sync job config, won't run any 
>> sync jobs");
>> ????????????????? return Ok(());
> 




From c.ebner at proxmox.com  Thu May 15 13:45:57 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 15 May 2025 13:45:57 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 4/7] api: admin: trigger sync
 jobs only on datastore mount
In-Reply-To: <33b47879-fadf-46f6-94ee-135560cda8e5@proxmox.com>
References: <20250116064543.74619-1-h.laimer@proxmox.com>
 <20250116064543.74619-5-h.laimer@proxmox.com>
 <618be27b-49fe-4bd5-b980-211ef7d93efc@proxmox.com>
 <33b47879-fadf-46f6-94ee-135560cda8e5@proxmox.com>
Message-ID: <73106bc1-5ebf-4ae9-8811-f077effaedf8@proxmox.com>

On 5/15/25 13:33, Hannes Laimer wrote:
> 
> 
> On 2/4/25 15:34, Christian Ebner wrote:
>> This patch should rather be placed as preparatory patch before patch 3 
>> and changes to `do_sync_jobs` directly included in the previous patch.
>>
> 
> Maybe, but these changes only really make sense in this context. If done
> before it would be something like
> ```
>  ????????? move |_worker| async move {
> -??????????? do_mount_device(datastore.clone())?;
> +??????????? if !do_mount_device(datastore.clone())? {
> +??????????????? return Ok(());
> +??????????? }
>  ????? }
> ```
> I feel like having this change in the context of where it is needed does
> add some value. Especially when one tries to find out later why this was
> needed. Having code under the `return Ok(())` is pretty clear in terms
> of what it is we try to skip here in some cases. Without I could imagine
> being like "th was that good for?", then having to go through the commit
> message. IMOH, having the extra context here is better than a possibly
> better order. But no hard feelings though, if you'd really like this one
> before I'll move it for v2.

No, fine by me to keep as is given your reasoning!



From h.laimer at proxmox.com  Thu May 15 14:41:33 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Thu, 15 May 2025 14:41:33 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 3/8] api: config: sync: update
 run-on-mount correctly
In-Reply-To: <20250515124138.55436-1-h.laimer@proxmox.com>
References: <20250515124138.55436-1-h.laimer@proxmox.com>
Message-ID: <20250515124138.55436-4-h.laimer@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 src/api2/config/sync.rs | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/src/api2/config/sync.rs b/src/api2/config/sync.rs
index 6194d8653..358409b54 100644
--- a/src/api2/config/sync.rs
+++ b/src/api2/config/sync.rs
@@ -339,6 +339,8 @@ pub enum DeletableProperty {
     EncryptedOnly,
     /// Delete the verified_only property,
     VerifiedOnly,
+    /// Delete the run_on_mount property,
+    RunOnMount,
     /// Delete the sync_direction property,
     SyncDirection,
 }
@@ -458,6 +460,9 @@ pub fn update_sync_job(
                 DeletableProperty::VerifiedOnly => {
                     data.verified_only = None;
                 }
+                DeletableProperty::RunOnMount => {
+                    data.run_on_mount = None;
+                }
                 DeletableProperty::SyncDirection => {
                     data.sync_direction = None;
                 }
@@ -507,6 +512,9 @@ pub fn update_sync_job(
     if let Some(verified_only) = update.verified_only {
         data.verified_only = Some(verified_only);
     }
+    if let Some(run_on_mount) = update.run_on_mount {
+        data.run_on_mount = Some(run_on_mount);
+    }
     if let Some(sync_direction) = update.sync_direction {
         data.sync_direction = Some(sync_direction);
     }
@@ -683,6 +691,7 @@ acl:1:/remote/remote1/remotestore1:write at pbs:RemoteSyncOperator
         transfer_last: None,
         encrypted_only: None,
         verified_only: None,
+        run_on_mount: None,
         sync_direction: None, // use default
     };
 
-- 
2.39.5




From h.laimer at proxmox.com  Thu May 15 14:41:37 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Thu, 15 May 2025 14:41:37 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 7/8] ui: add run-on-mount
 checkbox to SyncJob form
In-Reply-To: <20250515124138.55436-1-h.laimer@proxmox.com>
References: <20250515124138.55436-1-h.laimer@proxmox.com>
Message-ID: <20250515124138.55436-8-h.laimer@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 www/window/SyncJobEdit.js | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/www/window/SyncJobEdit.js b/www/window/SyncJobEdit.js
index 4cef9a1d1..73a489f1d 100644
--- a/www/window/SyncJobEdit.js
+++ b/www/window/SyncJobEdit.js
@@ -195,7 +195,7 @@ Ext.define('PBS.window.SyncJobEdit', {
 			xtype: 'pbsCalendarEvent',
 			name: 'schedule',
 			fieldLabel: gettext('Sync Schedule'),
-			emptyText: gettext('none (disabled)'),
+			emptyText: gettext('none'),
 			cbind: {
 			    deleteEmpty: '{!isCreate}',
 			    value: '{scheduleValue}',
@@ -451,6 +451,17 @@ Ext.define('PBS.window.SyncJobEdit', {
 			uncheckedValue: false,
 			value: false,
 		    },
+		    {
+			xtype: 'proxmoxcheckbox',
+			name: 'run-on-mount',
+			fieldLabel: gettext('Run when mounted'),
+			autoEl: {
+			    tag: 'div',
+			    'data-qtip': gettext('Run this job when a relevant removable datastore is mounted.'),
+			},
+			uncheckedValue: false,
+			value: false,
+		    },
 		],
 	    },
 	    {
-- 
2.39.5




From h.laimer at proxmox.com  Thu May 15 14:41:32 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Thu, 15 May 2025 14:41:32 +0200
Subject: [pbs-devel] [PATCH proxmox v2 2/8] pbs-api-types: add run-on-mount
 flag to SyncJobConfig
In-Reply-To: <20250515124138.55436-1-h.laimer@proxmox.com>
References: <20250515124138.55436-1-h.laimer@proxmox.com>
Message-ID: <20250515124138.55436-3-h.laimer@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 pbs-api-types/src/jobs.rs | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/pbs-api-types/src/jobs.rs b/pbs-api-types/src/jobs.rs
index 6ef13dc2..3eb61cde 100644
--- a/pbs-api-types/src/jobs.rs
+++ b/pbs-api-types/src/jobs.rs
@@ -536,6 +536,8 @@ pub const SYNC_ENCRYPTED_ONLY_SCHEMA: Schema =
     BooleanSchema::new("Only synchronize encrypted backup snapshots, exclude others.").schema();
 pub const SYNC_VERIFIED_ONLY_SCHEMA: Schema =
     BooleanSchema::new("Only synchronize verified backup snapshots, exclude others.").schema();
+pub const RUN_SYNC_ON_MOUNT_SCHEMA: Schema =
+    BooleanSchema::new("Run this job when a relevant datastore is mounted.").schema();
 
 #[api(
     properties: {
@@ -603,6 +605,10 @@ pub const SYNC_VERIFIED_ONLY_SCHEMA: Schema =
             schema: SYNC_VERIFIED_ONLY_SCHEMA,
             optional: true,
         },
+        "run-on-mount": {
+            schema: RUN_SYNC_ON_MOUNT_SCHEMA,
+            optional: true,
+        },
         "sync-direction": {
             type: SyncDirection,
             optional: true,
@@ -647,6 +653,8 @@ pub struct SyncJobConfig {
     #[serde(skip_serializing_if = "Option::is_none")]
     pub verified_only: Option<bool>,
     #[serde(skip_serializing_if = "Option::is_none")]
+    pub run_on_mount: Option<bool>,
+    #[serde(skip_serializing_if = "Option::is_none")]
     pub sync_direction: Option<SyncDirection>,
 }
 
-- 
2.39.5




From h.laimer at proxmox.com  Thu May 15 14:41:38 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Thu, 15 May 2025 14:41:38 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 8/8] ui: add task title for
 triggering sync jobs
In-Reply-To: <20250515124138.55436-1-h.laimer@proxmox.com>
References: <20250515124138.55436-1-h.laimer@proxmox.com>
Message-ID: <20250515124138.55436-9-h.laimer@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 www/Utils.js | 1 +
 1 file changed, 1 insertion(+)

diff --git a/www/Utils.js b/www/Utils.js
index 9dcde6941..eb4943010 100644
--- a/www/Utils.js
+++ b/www/Utils.js
@@ -422,6 +422,7 @@ Ext.define('PBS.Utils', {
 	    prunejob: (type, id) => PBS.Utils.render_prune_job_worker_id(id, gettext('Prune Job')),
 	    reader: (type, id) => PBS.Utils.render_datastore_worker_id(id, gettext('Read Objects')),
 	    'rewind-media': [gettext('Drive'), gettext('Rewind Media')],
+	    'mount-sync-jobs': [gettext('Datastore'), gettext('trigger sync jobs')],
 	    sync: ['Datastore', gettext('Remote Sync')],
 	    syncjob: [gettext('Sync Job'), gettext('Remote Sync')],
 	    'tape-backup': (type, id) => PBS.Utils.render_tape_backup_id(id, gettext('Tape Backup')),
-- 
2.39.5




From h.laimer at proxmox.com  Thu May 15 14:41:35 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Thu, 15 May 2025 14:41:35 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 5/8] api: admin: trigger sync
 jobs only on datastore mount
In-Reply-To: <20250515124138.55436-1-h.laimer@proxmox.com>
References: <20250515124138.55436-1-h.laimer@proxmox.com>
Message-ID: <20250515124138.55436-6-h.laimer@proxmox.com>

Ensure sync jobs are triggered only when the datastore is actually
mounted. If the datastore is already mounted, we don't fail,
but sync jobs should not be re-triggered unnecessarily. This change
prevents redundant sync job execution.

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 src/api2/admin/datastore.rs | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 8463adb6a..a3ba82e21 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -2455,14 +2455,14 @@ fn setup_mounted_device(datastore: &DataStoreConfig, tmp_mount_path: &str) -> Re
 /// The reason for the randomized device mounting paths is to avoid two tasks trying to mount to
 /// the same path, this is *very* unlikely since the device is only mounted really shortly, but
 /// technically possible.
-pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
+pub fn do_mount_device(datastore: DataStoreConfig) -> Result<bool, Error> {
     if let Some(uuid) = datastore.backing_device.as_ref() {
         if pbs_datastore::get_datastore_mount_status(&datastore) == Some(true) {
             info!(
                 "device is already mounted at '{}'",
                 datastore.absolute_path()
             );
-            return Ok(());
+            return Ok(false);
         }
         let tmp_mount_path = format!(
             "{}/{:x}",
@@ -2507,7 +2507,7 @@ pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
             datastore.name
         )
     }
-    Ok(())
+    Ok(true)
 }
 
 async fn do_sync_jobs(
@@ -2592,7 +2592,9 @@ pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Er
         auth_id.to_string(),
         to_stdout,
         move |_worker| async move {
-            do_mount_device(datastore.clone())?;
+            if !do_mount_device(datastore.clone())? {
+                return Ok(());
+            }
             let Ok((sync_config, _digest)) = pbs_config::sync::config() else {
                 warn!("unable to read sync job config, won't run any sync jobs");
                 return Ok(());
-- 
2.39.5




From h.laimer at proxmox.com  Thu May 15 14:41:36 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Thu, 15 May 2025 14:41:36 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 6/8] bin: manager: run
 uuid_mount/mount tasks on the proxy
In-Reply-To: <20250515124138.55436-1-h.laimer@proxmox.com>
References: <20250515124138.55436-1-h.laimer@proxmox.com>
Message-ID: <20250515124138.55436-7-h.laimer@proxmox.com>

Use the API instead of running uuid_mount/mount directly in the CLI binary.
This ensures that all triggered tasks are handled by the proxy process.

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 src/bin/proxmox_backup_manager/datastore.rs | 41 ++++++++++++++++-----
 1 file changed, 31 insertions(+), 10 deletions(-)

diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index 1922a55a2..3fbb5fe5b 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -49,6 +49,10 @@ fn list_datastores(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
             store: {
                 schema: DATASTORE_SCHEMA,
             },
+            "output-format": {
+                schema: OUTPUT_FORMAT,
+                optional: true,
+            },
             digest: {
                 optional: true,
                 schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
@@ -57,16 +61,23 @@ fn list_datastores(param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<Valu
     },
 )]
 /// Mount a removable datastore.
-async fn mount_datastore(mut param: Value, rpcenv: &mut dyn RpcEnvironment) -> Result<(), Error> {
-    param["node"] = "localhost".into();
+async fn mount_datastore(
+    store: String,
+    mut param: Value,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let output_format = extract_output_format(&mut param);
 
-    let info = &api2::admin::datastore::API_METHOD_MOUNT;
-    let result = match info.handler {
-        ApiHandler::Sync(handler) => (handler)(param, info, rpcenv)?,
-        _ => unreachable!(),
-    };
+    let client = connect_to_localhost()?;
+    let result = client
+        .post(
+            format!("api2/json/admin/datastore/{store}/mount").as_str(),
+            None,
+        )
+        .await?;
+
+    view_task_result(&client, result, &output_format).await?;
 
-    crate::wait_for_local_worker(result.as_str().unwrap()).await?;
     Ok(())
 }
 
@@ -260,7 +271,8 @@ async fn update_datastore(name: String, mut param: Value) -> Result<(), Error> {
     },
 )]
 /// Try mounting a removable datastore given the UUID.
-async fn uuid_mount(param: Value, _rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+async fn uuid_mount(mut param: Value, _rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+    let output_format = extract_output_format(&mut param);
     let uuid = param["uuid"]
         .as_str()
         .ok_or_else(|| format_err!("uuid has to be specified"))?;
@@ -282,7 +294,16 @@ async fn uuid_mount(param: Value, _rpcenv: &mut dyn RpcEnvironment) -> Result<Va
     }
 
     if let Some(store) = matching_stores.first() {
-        api2::admin::datastore::do_mount_device(store.clone())?;
+        let client = connect_to_localhost()?;
+        let result = client
+            .post(
+                format!("api2/json/admin/datastore/{}/mount", store.name).as_str(),
+                None,
+            )
+            .await?;
+
+        view_task_result(&client, result, &output_format).await?;
+        return Ok(Value::Null);
     }
 
     // we don't want to fail for UUIDs that are not associated with datastores, as that produces
-- 
2.39.5




From h.laimer at proxmox.com  Thu May 15 14:41:34 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Thu, 15 May 2025 14:41:34 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 4/8] api: admin: run
 configured sync jobs when a datastore is mounted
In-Reply-To: <20250515124138.55436-1-h.laimer@proxmox.com>
References: <20250515124138.55436-1-h.laimer@proxmox.com>
Message-ID: <20250515124138.55436-5-h.laimer@proxmox.com>

When a datastore is mounted, spawn a new task to run all sync jobs
marked with `run-on-mount`. These jobs run sequentially and include
any job for which the mounted datastore is:

- The source or target in a local push/pull job
- The source in a push job to a remote datastore
- The target in a pull job from a remote datastore

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 src/api2/admin/datastore.rs | 91 +++++++++++++++++++++++++++++++++++--
 src/api2/admin/sync.rs      |  2 +-
 src/server/sync.rs          |  7 +--
 3 files changed, 91 insertions(+), 9 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 392494488..8463adb6a 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -42,8 +42,8 @@ use pbs_api_types::{
     DataStoreConfig, DataStoreListItem, DataStoreMountStatus, DataStoreStatus,
     GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
     MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
-    BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
-    BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
+    SyncJobConfig, BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA,
+    BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
     IGNORE_VERIFIED_BACKUPS_SCHEMA, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
     PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
     PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, PRIV_SYS_MODIFY, UPID, UPID_SCHEMA,
@@ -2510,6 +2510,51 @@ pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
     Ok(())
 }
 
+async fn do_sync_jobs(
+    jobs_to_run: Vec<SyncJobConfig>,
+    worker: Arc<WorkerTask>,
+) -> Result<(), Error> {
+    let count = jobs_to_run.len();
+    info!(
+        "will run {} sync jobs: {}",
+        count,
+        jobs_to_run
+            .iter()
+            .map(|j| j.id.clone())
+            .collect::<Vec<String>>()
+            .join(", ")
+    );
+
+    for (i, job_config) in jobs_to_run.into_iter().enumerate() {
+        if worker.abort_requested() {
+            bail!("aborted due to user request");
+        }
+        let job_id = job_config.id.clone();
+        let Ok(job) = Job::new("syncjob", &job_id) else {
+            continue;
+        };
+        let auth_id = Authid::root_auth_id().clone();
+        info!("[{}/{count}] starting '{job_id}'...", i + 1);
+        match crate::server::do_sync_job(
+            job,
+            job_config,
+            &auth_id,
+            Some("mount".to_string()),
+            false,
+        ) {
+            Ok((_upid, handle)) => {
+                tokio::select! {
+                    sync_done = handle.fuse() => if let Err(err) = sync_done { warn!("could not wait for job to finish: {err}"); },
+                    _abort = worker.abort_future() => bail!("aborted due to user request"),
+                };
+            }
+            Err(err) => warn!("unable to start sync job {job_id} - {err}"),
+        }
+    }
+
+    Ok(())
+}
+
 #[api(
     protected: true,
     input: {
@@ -2541,12 +2586,48 @@ pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Er
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
     let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
 
-    let upid = WorkerTask::new_thread(
+    let upid = WorkerTask::spawn(
         "mount-device",
-        Some(store),
+        Some(store.clone()),
         auth_id.to_string(),
         to_stdout,
-        move |_worker| do_mount_device(datastore),
+        move |_worker| async move {
+            do_mount_device(datastore.clone())?;
+            let Ok((sync_config, _digest)) = pbs_config::sync::config() else {
+                warn!("unable to read sync job config, won't run any sync jobs");
+                return Ok(());
+            };
+            let Ok(list) = sync_config.convert_to_typed_array("sync") else {
+                warn!("unable to parse sync job config, won't run any sync jobs");
+                return Ok(());
+            };
+            let jobs_to_run: Vec<SyncJobConfig> = list
+                .into_iter()
+                .filter(|job: &SyncJobConfig| {
+                    // add job iff (running on mount is enabled and) any of these apply
+                    //   - the jobs is local and we are source or target
+                    //   - we are the source of a push to a remote
+                    //   - we are the target of a pull from a remote
+                    //
+                    // `job.store == datastore.name` iff we are the target for pull from remote or we
+                    // are the source for push to remote, therefore we don't have to check for the
+                    // direction of the job.
+                    job.run_on_mount.unwrap_or(false)
+                        && (job.remote.is_none() && job.remote_store == datastore.name
+                            || job.store == datastore.name)
+                })
+                .collect();
+            if !jobs_to_run.is_empty() {
+                let _ = WorkerTask::spawn(
+                    "mount-sync-jobs",
+                    Some(store),
+                    auth_id.to_string(),
+                    false,
+                    move |worker| async move { do_sync_jobs(jobs_to_run, worker).await },
+                );
+            }
+            Ok(())
+        },
     )?;
 
     Ok(json!(upid))
diff --git a/src/api2/admin/sync.rs b/src/api2/admin/sync.rs
index 6722ebea0..01dea5126 100644
--- a/src/api2/admin/sync.rs
+++ b/src/api2/admin/sync.rs
@@ -161,7 +161,7 @@ pub fn run_sync_job(
 
     let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
 
-    let upid_str = do_sync_job(job, sync_job, &auth_id, None, to_stdout)?;
+    let (upid_str, _) = do_sync_job(job, sync_job, &auth_id, None, to_stdout)?;
 
     Ok(upid_str)
 }
diff --git a/src/server/sync.rs b/src/server/sync.rs
index 09814ef0c..c45a8975e 100644
--- a/src/server/sync.rs
+++ b/src/server/sync.rs
@@ -12,6 +12,7 @@ use futures::{future::FutureExt, select};
 use hyper::http::StatusCode;
 use pbs_config::BackupLockGuard;
 use serde_json::json;
+use tokio::task::JoinHandle;
 use tracing::{info, warn};
 
 use proxmox_human_byte::HumanByte;
@@ -598,7 +599,7 @@ pub fn do_sync_job(
     auth_id: &Authid,
     schedule: Option<String>,
     to_stdout: bool,
-) -> Result<String, Error> {
+) -> Result<(String, JoinHandle<()>), Error> {
     let job_id = format!(
         "{}:{}:{}:{}:{}",
         sync_job.remote.as_deref().unwrap_or("-"),
@@ -614,7 +615,7 @@ pub fn do_sync_job(
         bail!("can't sync to same datastore");
     }
 
-    let upid_str = WorkerTask::spawn(
+    let (upid_str, handle) = WorkerTask::spawn_with_handle(
         &worker_type,
         Some(job_id.clone()),
         auth_id.to_string(),
@@ -730,7 +731,7 @@ pub fn do_sync_job(
         },
     )?;
 
-    Ok(upid_str)
+    Ok((upid_str, handle))
 }
 
 pub(super) fn ignore_not_verified_or_encrypted(
-- 
2.39.5




From h.laimer at proxmox.com  Thu May 15 14:41:31 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Thu, 15 May 2025 14:41:31 +0200
Subject: [pbs-devel] [PATCH proxmox v2 1/8] rest-server: add function that
 returns a join handle for spawn
In-Reply-To: <20250515124138.55436-1-h.laimer@proxmox.com>
References: <20250515124138.55436-1-h.laimer@proxmox.com>
Message-ID: <20250515124138.55436-2-h.laimer@proxmox.com>

We need this handle when we want to start multiple jobs sequentially
from a 'manage' task. With the handle we can avoid polling for the
task status with its upid.

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 proxmox-rest-server/src/worker_task.rs | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/proxmox-rest-server/src/worker_task.rs b/proxmox-rest-server/src/worker_task.rs
index a3a65add..459728c3 100644
--- a/proxmox-rest-server/src/worker_task.rs
+++ b/proxmox-rest-server/src/worker_task.rs
@@ -14,6 +14,7 @@ use serde::{Deserialize, Serialize};
 use serde_json::{json, Value};
 use tokio::signal::unix::SignalKind;
 use tokio::sync::{oneshot, watch};
+use tokio::task::JoinHandle;
 use tracing::{error, info, warn};
 
 use proxmox_daemon::command_socket::CommandSocket;
@@ -936,6 +937,25 @@ impl WorkerTask {
         to_stdout: bool,
         f: F,
     ) -> Result<String, Error>
+    where
+        F: Send + 'static + FnOnce(Arc<WorkerTask>) -> T,
+        T: Send + 'static + Future<Output = Result<(), Error>>,
+    {
+        let (upid_str, _) = Self::spawn_with_handle(worker_type, worker_id, auth_id, to_stdout, f)?;
+        Ok(upid_str)
+    }
+
+    /// Spawn a new worker task with log context like `spawn`, but
+    /// return the tasks join handle in addition to its upid.
+    ///
+    /// Allows for a management tasks to handle multiple worker tasks.
+    pub fn spawn_with_handle<F, T>(
+        worker_type: &str,
+        worker_id: Option<String>,
+        auth_id: String,
+        to_stdout: bool,
+        f: F,
+    ) -> Result<(String, JoinHandle<()>), Error>
     where
         F: Send + 'static + FnOnce(Arc<WorkerTask>) -> T,
         T: Send + 'static + Future<Output = Result<(), Error>>,
@@ -944,12 +964,12 @@ impl WorkerTask {
         let upid_str = worker.upid.to_string();
         let f = f(worker.clone());
 
-        tokio::spawn(LogContext::new(logger).scope(async move {
+        let handle = tokio::spawn(LogContext::new(logger).scope(async move {
             let result = f.await;
             worker.log_result(&result);
         }));
 
-        Ok(upid_str)
+        Ok((upid_str, handle))
     }
 
     /// Create a new worker thread.
-- 
2.39.5




From h.laimer at proxmox.com  Thu May 15 14:41:30 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Thu, 15 May 2025 14:41:30 +0200
Subject: [pbs-devel] [PATCH proxmox/proxmox-backup v2 0/8] trigger sync jobs
 on mount
Message-ID: <20250515124138.55436-1-h.laimer@proxmox.com>

Sync jobs now have a run-on-mount flag, that, if set, runs the job whenever
a relevant removable datastore is mounted. 

This depends on [1], without it the api process does not drop the file
handle on the `.lock` file which leads to the datastore being not
unmountable after sync jobs were triggered. (now thinking about it,
might have made sense to include it in this series directly, but it also
does make sense on its own)

v2, thanks @Chris:
- rebased onto master
- improve some docstrings
- move/fix config flag
- drop not-needed changes for the manager binary
- ui: move checkbox to advenced section + don't clear schedule field
- fix test
- actually check the configured flag when deciding if a job should
  run...

[1] https://lore.proxmox.com/pbs-devel/20250512125933.156192-1-h.laimer at proxmox.com/T/#u


proxmox:
Hannes Laimer (2):
  rest-server: add function that returns a join handle for spawn
  pbs-api-types: add run-on-mount flag to SyncJobConfig

 pbs-api-types/src/jobs.rs              |  8 ++++++++
 proxmox-rest-server/src/worker_task.rs | 24 ++++++++++++++++++++++--
 2 files changed, 30 insertions(+), 2 deletions(-)

proxmox-backup:
Hannes Laimer (6):
  api: config: sync: update run-on-mount correctly
  api: admin: run configured sync jobs when a datastore is mounted
  api: admin: trigger sync jobs only on datastore mount
  bin: manager: run uuid_mount/mount tasks on the proxy
  ui: add run-on-mount checkbox to SyncJob form
  ui: add task title for triggering sync jobs

 src/api2/admin/datastore.rs                 | 97 +++++++++++++++++++--
 src/api2/admin/sync.rs                      |  2 +-
 src/api2/config/sync.rs                     |  9 ++
 src/bin/proxmox_backup_manager/datastore.rs | 41 ++++++---
 src/server/sync.rs                          |  7 +-
 www/Utils.js                                |  1 +
 www/window/SyncJobEdit.js                   | 13 ++-
 7 files changed, 148 insertions(+), 22 deletions(-)

-- 
2.39.5




From h.laimer at proxmox.com  Thu May 15 14:43:18 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Thu, 15 May 2025 14:43:18 +0200
Subject: [pbs-devel] [PATCH proxmox/proxmox-backup 0/7] trigger sync
 jobs on mount
In-Reply-To: <20250116064543.74619-1-h.laimer@proxmox.com>
References: <20250116064543.74619-1-h.laimer@proxmox.com>
Message-ID: <aaf07d01-c3d4-4757-bc43-06c8e8669f6f@proxmox.com>

superseded by: 
https://lore.proxmox.com/pbs-devel/20250515124138.55436-1-h.laimer at proxmox.com/T/#t

On 1/16/25 07:45, Hannes Laimer wrote:
> Sync jobs now have a run-on-mount flag, that, if set, runs the job whenever
> a relevant removable datastore is mounted.
> 
> 
> * proxmox:
> Hannes Laimer (1):
>    rest-server: add function that returns a join handle for spawn
> 
>   proxmox-rest-server/src/worker_task.rs | 21 +++++++++++++++++++--
>   1 file changed, 19 insertions(+), 2 deletions(-)
>   
> 
> * proxmox-backup:
> Hannes Laimer (6):
>    api types: add run-on-mount flag to SyncJobConfig
>    api: admin: run configured sync jobs when a datastore is mounted
>    api: admin: trigger sync jobs only on datastore mount
>    bin: manager: run uuid_mount/mount tasks on the proxy
>    ui: add run-on-mount checkbox to SyncJob form
>    ui: add task title for triggering sync jobs
> 
>   pbs-api-types/src/jobs.rs                   |  3 +
>   src/api2/admin/datastore.rs                 | 96 +++++++++++++++++++--
>   src/api2/admin/sync.rs                      |  2 +-
>   src/bin/proxmox_backup_manager/datastore.rs | 42 +++++----
>   src/server/sync.rs                          |  7 +-
>   www/Utils.js                                |  1 +
>   www/window/SyncJobEdit.js                   | 23 ++++-
>   7 files changed, 147 insertions(+), 27 deletions(-)
> 




From c.ebner at proxmox.com  Fri May 16 09:44:46 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 16 May 2025 09:44:46 +0200
Subject: [pbs-devel] [PATCH proxmox-backup] docs: tuning: list default and
 maximum values for `gc-cache-capacity`
Message-ID: <20250516074446.20660-1-c.ebner@proxmox.com>

Explicitly mention that the value sets the available cache slots and
not only mention the value being set to 0 disables the cache, but
rather give also the default and maximum values.

Reported in the community forum:
https://forum.proxmox.com/threads/164869/post-771224

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 docs/storage.rst | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/docs/storage.rst b/docs/storage.rst
index 2e648b8dc..4a8d8255e 100644
--- a/docs/storage.rst
+++ b/docs/storage.rst
@@ -456,7 +456,9 @@ There are some tuning related options for the datastore that are more advanced:
   the access time has already been updated during phase 1 of garbage collection.
   This avoids multiple updates and increases GC runtime performance. Higher
   values can reduce GC runtime at the cost of increase memory usage, setting the
-  value to 0 disables caching.
+  value to 0 disables caching. The given value sets the number of available
+  cache slots, 1048576 (= 1024 * 1024) being the default, 8388608 (= 8192 *
+  1024) the maximum value.
 
 If you want to set multiple tuning options simultaneously, you can separate them
 with a comma, like this:
-- 
2.39.5




From c.ebner at proxmox.com  Fri May 16 10:58:35 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 16 May 2025 10:58:35 +0200
Subject: [pbs-devel] [PATCH proxmox 1/1] pbs api types: extend garbage
 collection status by cache stats
In-Reply-To: <20250516085836.82494-1-c.ebner@proxmox.com>
References: <20250516085836.82494-1-c.ebner@proxmox.com>
Message-ID: <20250516085836.82494-2-c.ebner@proxmox.com>

Add the number of cache hits and cache misses encountered during
phase 1 of garbage collection in order to display this information
in the garbage collection task log summary.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-api-types/src/datastore.rs | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 5bd953ac..4fb1eb80 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -1459,6 +1459,10 @@ pub struct GarbageCollectionStatus {
     pub removed_bad: usize,
     /// Number of chunks still marked as .bad after garbage collection.
     pub still_bad: usize,
+    /// Number of atime update cache hits
+    pub cache_hits: usize,
+    /// Number of atime update cache misses
+    pub cache_misses: usize,
 }
 
 #[api(
-- 
2.39.5




From c.ebner at proxmox.com  Fri May 16 10:58:34 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 16 May 2025 10:58:34 +0200
Subject: [pbs-devel] [PATCH proxmox proxmox-backup 0/2] add GC chunk cache
 stats
Message-ID: <20250516085836.82494-1-c.ebner@proxmox.com>

In an effort to give some more insights into phase one of garbage
collection, add counters for hits and misses of the chunk cache to
the garbage collection status and output the obtained values in the
garbage collection task log.

This will allow to easier investigate possible issues and adapt the
tuning parameters based on the output.

proxmox:

Christian Ebner (1):
  pbs api types: extend garbage collection status by cache stats

 pbs-api-types/src/datastore.rs | 4 ++++
 1 file changed, 4 insertions(+)

proxmox-backup:

Christian Ebner (1):
  garbage collection: track chunk cache stats and show in task log

 pbs-datastore/src/datastore.rs | 6 ++++++
 1 file changed, 6 insertions(+)

-- 
2.39.5




From c.ebner at proxmox.com  Fri May 16 10:58:36 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 16 May 2025 10:58:36 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 2/2] garbage collection: track
 chunk cache stats and show in task log
In-Reply-To: <20250516085836.82494-1-c.ebner@proxmox.com>
References: <20250516085836.82494-1-c.ebner@proxmox.com>
Message-ID: <20250516085836.82494-3-c.ebner@proxmox.com>

Count the chunk cache hits and misses and display the resulting
values in the garbage collection task log summary.

This allows to investigate possible issues and tune cache capacity,
also by being able to compare to other values in the summary such
as the on disk chunk count.

Exemplary output
```
2025-05-16T10:41:29+02:00: Chunk cache: hits 38118, misses 10017
2025-05-16T10:41:29+02:00: Removed garbage: 3.384 GiB
2025-05-16T10:41:29+02:00: Removed chunks: 2835
2025-05-16T10:41:29+02:00: Original data usage: 155.165 GiB
2025-05-16T10:41:29+02:00: On-Disk usage: 7.312 GiB (4.71%)
2025-05-16T10:41:29+02:00: On-Disk chunks: 10017
2025-05-16T10:41:29+02:00: Deduplication factor: 21.22
2025-05-16T10:41:29+02:00: Average chunk size: 765.438 KiB
```

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index cbf78ecb6..479b76cd8 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1086,8 +1086,10 @@ impl DataStore {
 
             // Avoid multiple expensive atime updates by utimensat
             if chunk_lru_cache.insert(*digest, ()) {
+                status.cache_hits += 1;
                 continue;
             }
+            status.cache_misses += 1;
 
             if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
                 let hex = hex::encode(digest);
@@ -1349,6 +1351,10 @@ impl DataStore {
                 worker,
             )?;
 
+            info!(
+                "Chunk cache: hits {}, misses {}",
+                gc_status.cache_hits, gc_status.cache_misses,
+            );
             info!(
                 "Removed garbage: {}",
                 HumanByte::from(gc_status.removed_bytes),
-- 
2.39.5




From l.wagner at proxmox.com  Fri May 16 11:54:08 2025
From: l.wagner at proxmox.com (Lukas Wagner)
Date: Fri, 16 May 2025 11:54:08 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 2/2] garbage collection:
 track chunk cache stats and show in task log
In-Reply-To: <20250516085836.82494-3-c.ebner@proxmox.com>
References: <20250516085836.82494-1-c.ebner@proxmox.com>
 <20250516085836.82494-3-c.ebner@proxmox.com>
Message-ID: <be14a6f0-b959-4837-9215-c0849c564bf8@proxmox.com>



On  2025-05-16 10:58, Christian Ebner wrote:
> Count the chunk cache hits and misses and display the resulting
> values in the garbage collection task log summary.
> 
> This allows to investigate possible issues and tune cache capacity,
> also by being able to compare to other values in the summary such
> as the on disk chunk count.
> 
> Exemplary output
> ```
> 2025-05-16T10:41:29+02:00: Chunk cache: hits 38118, misses 10017

I think it would be nice to also print the hit ratio :) For the
example here it should be 79%, if my math is correct.


-- 
- Lukas




From c.ebner at proxmox.com  Fri May 16 12:29:27 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 16 May 2025 12:29:27 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 2/2] garbage collection:
 track chunk cache stats and show in task log
In-Reply-To: <be14a6f0-b959-4837-9215-c0849c564bf8@proxmox.com>
References: <20250516085836.82494-1-c.ebner@proxmox.com>
 <20250516085836.82494-3-c.ebner@proxmox.com>
 <be14a6f0-b959-4837-9215-c0849c564bf8@proxmox.com>
Message-ID: <a8dae229-2677-4e63-aab3-d550558a4b8a@proxmox.com>

On 5/16/25 11:54, Lukas Wagner wrote:
> 
> 
> On  2025-05-16 10:58, Christian Ebner wrote:
>> Count the chunk cache hits and misses and display the resulting
>> values in the garbage collection task log summary.
>>
>> This allows to investigate possible issues and tune cache capacity,
>> also by being able to compare to other values in the summary such
>> as the on disk chunk count.
>>
>> Exemplary output
>> ```
>> 2025-05-16T10:41:29+02:00: Chunk cache: hits 38118, misses 10017
> 
> I think it would be nice to also print the hit ratio :) For the
> example here it should be 79%, if my math is correct.

Sure, can add that as well as it is indeed more straight forward and 
intuitive!



From c.ebner at proxmox.com  Fri May 16 13:54:25 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 16 May 2025 13:54:25 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 1/5] api: admin: refactor
 generation of backup dir for snapshot listing
In-Reply-To: <20250516115429.208563-1-c.ebner@proxmox.com>
References: <20250516115429.208563-1-c.ebner@proxmox.com>
Message-ID: <20250516115429.208563-2-c.ebner@proxmox.com>

Instead of recreating the `pbs_api_types::BackupDir` from information
via the backup group, use the `BackupDir` as stored in `BackupInfo`.

As a side effect this allows to drop the backup group as parameter to
the closure generating the snapshot list items from the backup info
objects.

No functional changes intended.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs | 13 +++++--------
 1 file changed, 5 insertions(+), 8 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 392494488..55cb3a0b3 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -62,8 +62,8 @@ use pbs_datastore::index::IndexFile;
 use pbs_datastore::manifest::BackupManifest;
 use pbs_datastore::prune::compute_prune_info;
 use pbs_datastore::{
-    check_backup_owner, ensure_datastore_is_mounted, task_tracking, BackupDir, BackupGroup,
-    DataStore, LocalChunkReader, StoreProgress,
+    check_backup_owner, ensure_datastore_is_mounted, task_tracking, BackupDir, DataStore,
+    LocalChunkReader, StoreProgress,
 };
 use pbs_tools::json::required_string_param;
 use proxmox_rest_server::{formatter, WorkerTask};
@@ -529,11 +529,8 @@ unsafe fn list_snapshots_blocking(
         (None, None) => datastore.list_backup_groups(ns.clone())?,
     };
 
-    let info_to_snapshot_list_item = |group: &BackupGroup, owner, info: BackupInfo| {
-        let backup = pbs_api_types::BackupDir {
-            group: group.into(),
-            time: info.backup_dir.backup_time(),
-        };
+    let info_to_snapshot_list_item = |owner, info: BackupInfo| {
+        let backup = info.backup_dir.dir().to_owned();
         let protected = info.protected;
 
         match get_all_snapshot_files(&info) {
@@ -622,7 +619,7 @@ unsafe fn list_snapshots_blocking(
         snapshots.extend(
             group_backups
                 .into_iter()
-                .map(|info| info_to_snapshot_list_item(group, Some(owner.clone()), info)),
+                .map(|info| info_to_snapshot_list_item(Some(owner.clone()), info)),
         );
 
         Ok(snapshots)
-- 
2.39.5




From c.ebner at proxmox.com  Fri May 16 13:54:24 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 16 May 2025 13:54:24 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 0/5] prefilter source list by
 verify state for sync jobs
Message-ID: <20250516115429.208563-1-c.ebner@proxmox.com>

Since the introduction of the `verified-only` and `encrypted-only`
optional parameters for sync jobs, it is possible to exclude snapshot
if their are not encrypted or verified from the sync.

The current implementation, filtering the snapshots based on the
manifest information only directly before syncing said snapshot, leads
to the somewhat unexpected behaviour that in combination with the
transfer-last parameter set, the transfer-last filtering is performed
before the encryption/verificaton state filtering.

This patch series improves the current behaviour by pre-filtering the
snapshots already when generating the list of to be synced items.
This allows to apply the transfer-last filtering only after the
verified-only and encrypted-only filters, resulting in a more natural
behaviour.

Since this breaks with the current filter order, the breaking changes
should be considered for version 4 only.

Christian Ebner (5):
  api: admin: refactor generation of backup dir for snapshot listing
  api: factor out helper converting backup info to snapshot list item
  sync: source: list snapshot items instead of backup directories
  pull: refactor source snapshot list filtering logic
  sync: conditionally pre-filter source list by verified or encrypted
    state

 src/api2/admin/datastore.rs | 142 +++---------------------------------
 src/server/pull.rs          |  49 ++++++++-----
 src/server/push.rs          |  44 +++++++----
 src/server/sync.rs          |  73 +++++++++++++-----
 src/tools/mod.rs            | 129 ++++++++++++++++++++++++++++++++
 5 files changed, 252 insertions(+), 185 deletions(-)

-- 
2.39.5




From c.ebner at proxmox.com  Fri May 16 13:54:27 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 16 May 2025 13:54:27 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 3/5] sync: source: list snapshot
 items instead of backup directories
In-Reply-To: <20250516115429.208563-1-c.ebner@proxmox.com>
References: <20250516115429.208563-1-c.ebner@proxmox.com>
Message-ID: <20250516115429.208563-4-c.ebner@proxmox.com>

The snapshot list items contain further information such as the
source verify and encryption state, so that allows filtering based on
these information after list generation already. Since this
information is already returned by the api calls in case of remote
sources and obtained by switching to `BackupGroup::list_backups`
instead of `BackupGroup::iter_snapshots` for local sources, return
the whole snapshot list item information instead of reducing it to
the subset containing the backup directory only when reading from
source.

Adapts the trait accordingly and renames `list_backup_dirs` to the
now better fitting `list_backup_snapshots`.

No functional changes intended.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/server/pull.rs | 16 +++++++++-------
 src/server/push.rs | 17 ++++++++++-------
 src/server/sync.rs | 39 ++++++++++++++++++++++-----------------
 3 files changed, 41 insertions(+), 31 deletions(-)

diff --git a/src/server/pull.rs b/src/server/pull.rs
index b1724c142..832ee1b85 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -12,9 +12,9 @@ use tracing::info;
 
 use pbs_api_types::{
     print_store_and_ns, ArchiveType, Authid, BackupArchiveName, BackupDir, BackupGroup,
-    BackupNamespace, GroupFilter, Operation, RateLimitConfig, Remote, VerifyState,
-    CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, PRIV_DATASTORE_AUDIT,
-    PRIV_DATASTORE_BACKUP,
+    BackupNamespace, GroupFilter, Operation, RateLimitConfig, Remote, SnapshotListItem,
+    VerifyState, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH,
+    PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP,
 };
 use pbs_client::BackupRepository;
 use pbs_config::CachedUserInfo;
@@ -541,11 +541,11 @@ async fn pull_group(
     let mut already_synced_skip_info = SkipInfo::new(SkipReason::AlreadySynced);
     let mut transfer_last_skip_info = SkipInfo::new(SkipReason::TransferLast);
 
-    let mut raw_list: Vec<BackupDir> = params
+    let mut raw_list: Vec<SnapshotListItem> = params
         .source
-        .list_backup_dirs(source_namespace, group)
+        .list_backup_snapshots(source_namespace, group)
         .await?;
-    raw_list.sort_unstable_by(|a, b| a.time.cmp(&b.time));
+    raw_list.sort_unstable_by(|a, b| a.backup.time.cmp(&b.backup.time));
 
     let total_amount = raw_list.len();
 
@@ -568,7 +568,9 @@ async fn pull_group(
     let list: Vec<(BackupDir, bool)> = raw_list
         .into_iter()
         .enumerate()
-        .filter_map(|(pos, dir)| {
+        .filter_map(|(pos, item)| {
+            let dir = item.backup;
+
             source_snapshots.insert(dir.time);
             // If resync_corrupt is set, check if the corresponding local snapshot failed to
             // verification
diff --git a/src/server/push.rs b/src/server/push.rs
index e71012ed8..fdfed1b5a 100644
--- a/src/server/push.rs
+++ b/src/server/push.rs
@@ -672,8 +672,11 @@ pub(crate) async fn push_group(
     let mut already_synced_skip_info = SkipInfo::new(SkipReason::AlreadySynced);
     let mut transfer_last_skip_info = SkipInfo::new(SkipReason::TransferLast);
 
-    let mut snapshots: Vec<BackupDir> = params.source.list_backup_dirs(namespace, group).await?;
-    snapshots.sort_unstable_by(|a, b| a.time.cmp(&b.time));
+    let mut snapshots: Vec<SnapshotListItem> = params
+        .source
+        .list_backup_snapshots(namespace, group)
+        .await?;
+    snapshots.sort_unstable_by(|a, b| a.backup.time.cmp(&b.backup.time));
 
     if snapshots.is_empty() {
         info!("Group '{group}' contains no snapshots to sync to remote");
@@ -698,19 +701,19 @@ pub(crate) async fn push_group(
     let snapshots: Vec<BackupDir> = snapshots
         .into_iter()
         .enumerate()
-        .filter(|&(pos, ref snapshot)| {
+        .filter_map(|(pos, item)| {
+            let snapshot = item.backup;
             source_snapshots.insert(snapshot.time);
             if last_snapshot_time >= snapshot.time {
                 already_synced_skip_info.update(snapshot.time);
-                return false;
+                return None;
             }
             if pos < cutoff {
                 transfer_last_skip_info.update(snapshot.time);
-                return false;
+                return None;
             }
-            true
+            Some(snapshot)
         })
-        .map(|(_, dir)| dir)
         .collect();
 
     if already_synced_skip_info.count > 0 {
diff --git a/src/server/sync.rs b/src/server/sync.rs
index 09814ef0c..155a0cb4b 100644
--- a/src/server/sync.rs
+++ b/src/server/sync.rs
@@ -32,6 +32,7 @@ use crate::backup::ListAccessibleBackupGroups;
 use crate::server::jobstate::Job;
 use crate::server::pull::{pull_store, PullParameters};
 use crate::server::push::{push_store, PushParameters};
+use crate::tools::backup_info_to_snapshot_list_item;
 
 #[derive(Default)]
 pub(crate) struct RemovedVanishedStats {
@@ -244,11 +245,11 @@ pub(crate) trait SyncSource: Send + Sync {
     ) -> Result<Vec<BackupGroup>, Error>;
 
     /// Lists backup directories for a specific group within a specific namespace from the source.
-    async fn list_backup_dirs(
+    async fn list_backup_snapshots(
         &self,
         namespace: &BackupNamespace,
         group: &BackupGroup,
-    ) -> Result<Vec<BackupDir>, Error>;
+    ) -> Result<Vec<SnapshotListItem>, Error>;
     fn get_ns(&self) -> BackupNamespace;
     fn get_store(&self) -> &str;
 
@@ -357,11 +358,11 @@ impl SyncSource for RemoteSource {
         )
     }
 
-    async fn list_backup_dirs(
+    async fn list_backup_snapshots(
         &self,
         namespace: &BackupNamespace,
         group: &BackupGroup,
-    ) -> Result<Vec<BackupDir>, Error> {
+    ) -> Result<Vec<SnapshotListItem>, Error> {
         let path = format!("api2/json/admin/datastore/{}/snapshots", self.repo.store());
 
         let mut args = json!({
@@ -380,16 +381,15 @@ impl SyncSource for RemoteSource {
         Ok(snapshot_list
             .into_iter()
             .filter_map(|item: SnapshotListItem| {
-                let snapshot = item.backup;
                 // in-progress backups can't be synced
                 if item.size.is_none() {
-                    info!("skipping snapshot {snapshot} - in-progress backup");
+                    info!("skipping snapshot {} - in-progress backup", item.backup);
                     return None;
                 }
 
-                Some(snapshot)
+                Some(item)
             })
-            .collect::<Vec<BackupDir>>())
+            .collect::<Vec<SnapshotListItem>>())
     }
 
     fn get_ns(&self) -> BackupNamespace {
@@ -451,18 +451,23 @@ impl SyncSource for LocalSource {
         .collect::<Vec<pbs_api_types::BackupGroup>>())
     }
 
-    async fn list_backup_dirs(
+    async fn list_backup_snapshots(
         &self,
         namespace: &BackupNamespace,
         group: &BackupGroup,
-    ) -> Result<Vec<BackupDir>, Error> {
-        Ok(self
-            .store
-            .backup_group(namespace.clone(), group.clone())
-            .iter_snapshots()?
-            .filter_map(Result::ok)
-            .map(|snapshot| snapshot.dir().to_owned())
-            .collect::<Vec<BackupDir>>())
+    ) -> Result<Vec<SnapshotListItem>, Error> {
+        let backup_group = self.store.backup_group(namespace.clone(), group.clone());
+        Ok(backup_group
+            .list_backups()?
+            .iter()
+            .filter_map(|info| {
+                let owner = match backup_group.get_owner() {
+                    Ok(owner) => owner,
+                    Err(_) => return None,
+                };
+                Some(backup_info_to_snapshot_list_item(info, &owner))
+            })
+            .collect::<Vec<SnapshotListItem>>())
     }
 
     fn get_ns(&self) -> BackupNamespace {
-- 
2.39.5




From c.ebner at proxmox.com  Fri May 16 13:54:28 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 16 May 2025 13:54:28 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 4/5] pull: refactor source
 snapshot list filtering logic
In-Reply-To: <20250516115429.208563-1-c.ebner@proxmox.com>
References: <20250516115429.208563-1-c.ebner@proxmox.com>
Message-ID: <20250516115429.208563-5-c.ebner@proxmox.com>

In preparation for filter based on the snapshot's verify and
encryption state before applying the transfer last cutoff.

This will allow to correctly use the transfer last filter on the
correspondingly prefiltered items only.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/server/pull.rs | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)

diff --git a/src/server/pull.rs b/src/server/pull.rs
index 832ee1b85..53de74f19 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -547,13 +547,6 @@ async fn pull_group(
         .await?;
     raw_list.sort_unstable_by(|a, b| a.backup.time.cmp(&b.backup.time));
 
-    let total_amount = raw_list.len();
-
-    let cutoff = params
-        .transfer_last
-        .map(|count| total_amount.saturating_sub(count))
-        .unwrap_or_default();
-
     let target_ns = source_namespace.map_prefix(&params.source.get_ns(), &params.target.ns)?;
 
     let mut source_snapshots = HashSet::new();
@@ -567,8 +560,7 @@ async fn pull_group(
     // Also stores if the snapshot is corrupt (verification job failed)
     let list: Vec<(BackupDir, bool)> = raw_list
         .into_iter()
-        .enumerate()
-        .filter_map(|(pos, item)| {
+        .filter_map(|item| {
             let dir = item.backup;
 
             source_snapshots.insert(dir.time);
@@ -600,6 +592,20 @@ async fn pull_group(
                     }
                 }
             }
+            Some((dir, false))
+        })
+        .collect();
+
+    let total_amount = list.len();
+    let cutoff = params
+        .transfer_last
+        .map(|count| total_amount.saturating_sub(count))
+        .unwrap_or_default();
+
+    let list: Vec<(BackupDir, bool)> = list
+        .into_iter()
+        .enumerate()
+        .filter_map(|(pos, (dir, verified))| {
             // Note: the snapshot represented by `last_sync_time` might be missing its backup log
             // or post-backup verification state if those were not yet available during the last
             // sync run, always resync it
@@ -611,7 +617,7 @@ async fn pull_group(
                 transfer_last_skip_info.update(dir.time);
                 return None;
             }
-            Some((dir, false))
+            Some((dir, verified))
         })
         .collect();
 
-- 
2.39.5




From c.ebner at proxmox.com  Fri May 16 13:54:29 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 16 May 2025 13:54:29 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 5/5] sync: conditionally
 pre-filter source list by verified or encrypted state
In-Reply-To: <20250516115429.208563-1-c.ebner@proxmox.com>
References: <20250516115429.208563-1-c.ebner@proxmox.com>
Message-ID: <20250516115429.208563-6-c.ebner@proxmox.com>

Commit 40ccd1ac ("fix #6072: server: sync encrypted or verified
snapshots only") introduced flags for sync jobs to include encrypted
or verified snapshots only. The chosen approach to check the snapshot
state right before sync only does however lead to possibly unexpected
results if the jobs parameters also include a transfer last setting,
as it is applied beforehand.

In order to improve this behaviour, already pre-filter the snapshot
source list if either the `verified-only` or the `encrypted-only`
flags are set.

The state has to be re-checked once again for consistency reasons,
when the snapshot is read locked and about to be synced. Snapshots
are not included if the state would match after the list generation
and will be excluded if the state no longer matches.

This is a breaking change in the filter logic.

Reported-by: Stefan Hanreich <s.hanreich at proxmox.com>
Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/server/pull.rs | 11 ++++++++---
 src/server/push.rs | 31 +++++++++++++++++++++----------
 src/server/sync.rs | 34 +++++++++++++++++++++++++++++++---
 3 files changed, 60 insertions(+), 16 deletions(-)

diff --git a/src/server/pull.rs b/src/server/pull.rs
index 53de74f19..49f1d7a66 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -28,8 +28,9 @@ use pbs_datastore::{check_backup_owner, DataStore, StoreProgress};
 use pbs_tools::sha::sha256;
 
 use super::sync::{
-    check_namespace_depth_limit, ignore_not_verified_or_encrypted, LocalSource, RemoteSource,
-    RemovedVanishedStats, SkipInfo, SkipReason, SyncSource, SyncSourceReader, SyncStats,
+    check_namespace_depth_limit, exclude_not_verified_or_encrypted,
+    ignore_not_verified_or_encrypted, LocalSource, RemoteSource, RemovedVanishedStats, SkipInfo,
+    SkipReason, SyncSource, SyncSourceReader, SyncStats,
 };
 use crate::backup::{check_ns_modification_privs, check_ns_privs};
 use crate::tools::parallel_handler::ParallelHandler;
@@ -561,8 +562,12 @@ async fn pull_group(
     let list: Vec<(BackupDir, bool)> = raw_list
         .into_iter()
         .filter_map(|item| {
-            let dir = item.backup;
+            if exclude_not_verified_or_encrypted(&item, params.verified_only, params.encrypted_only)
+            {
+                return None;
+            }
 
+            let dir = item.backup;
             source_snapshots.insert(dir.time);
             // If resync_corrupt is set, check if the corresponding local snapshot failed to
             // verification
diff --git a/src/server/push.rs b/src/server/push.rs
index fdfed1b5a..d9a7aea8a 100644
--- a/src/server/push.rs
+++ b/src/server/push.rs
@@ -26,8 +26,9 @@ use pbs_datastore::read_chunk::AsyncReadChunk;
 use pbs_datastore::{DataStore, StoreProgress};
 
 use super::sync::{
-    check_namespace_depth_limit, ignore_not_verified_or_encrypted, LocalSource,
-    RemovedVanishedStats, SkipInfo, SkipReason, SyncSource, SyncStats,
+    check_namespace_depth_limit, exclude_not_verified_or_encrypted,
+    ignore_not_verified_or_encrypted, LocalSource, RemovedVanishedStats, SkipInfo, SkipReason,
+    SyncSource, SyncStats,
 };
 use crate::api2::config::remote;
 
@@ -682,12 +683,6 @@ pub(crate) async fn push_group(
         info!("Group '{group}' contains no snapshots to sync to remote");
     }
 
-    let total_snapshots = snapshots.len();
-    let cutoff = params
-        .transfer_last
-        .map(|count| total_snapshots.saturating_sub(count))
-        .unwrap_or_default();
-
     let target_namespace = params.map_to_target(namespace)?;
     let mut target_snapshots = fetch_target_snapshots(params, &target_namespace, group).await?;
     target_snapshots.sort_unstable_by_key(|a| a.backup.time);
@@ -698,11 +693,27 @@ pub(crate) async fn push_group(
         .unwrap_or(i64::MIN);
 
     let mut source_snapshots = HashSet::new();
+    let snapshots: Vec<BackupDir> = snapshots
+        .into_iter()
+        .filter_map(|item| {
+            if exclude_not_verified_or_encrypted(&item, params.verified_only, params.encrypted_only)
+            {
+                return None;
+            }
+            Some(item.backup)
+        })
+        .collect();
+
+    let total_snapshots = snapshots.len();
+    let cutoff = params
+        .transfer_last
+        .map(|count| total_snapshots.saturating_sub(count))
+        .unwrap_or_default();
+
     let snapshots: Vec<BackupDir> = snapshots
         .into_iter()
         .enumerate()
-        .filter_map(|(pos, item)| {
-            let snapshot = item.backup;
+        .filter_map(|(pos, snapshot)| {
             source_snapshots.insert(snapshot.time);
             if last_snapshot_time >= snapshot.time {
                 already_synced_skip_info.update(snapshot.time);
diff --git a/src/server/sync.rs b/src/server/sync.rs
index 155a0cb4b..8c0569672 100644
--- a/src/server/sync.rs
+++ b/src/server/sync.rs
@@ -20,8 +20,8 @@ use proxmox_router::HttpError;
 
 use pbs_api_types::{
     Authid, BackupDir, BackupGroup, BackupNamespace, CryptMode, GroupListItem, SnapshotListItem,
-    SyncDirection, SyncJobConfig, VerifyState, CLIENT_LOG_BLOB_NAME, MAX_NAMESPACE_DEPTH,
-    PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_READ,
+    SyncDirection, SyncJobConfig, VerifyState, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME,
+    MAX_NAMESPACE_DEPTH, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_READ,
 };
 use pbs_client::{BackupReader, BackupRepository, HttpClient, RemoteChunkReader};
 use pbs_datastore::data_blob::DataBlob;
@@ -761,10 +761,38 @@ pub(super) fn ignore_not_verified_or_encrypted(
             .iter()
             .all(|file| file.chunk_crypt_mode() == CryptMode::Encrypt)
         {
-            info!("Snapshot {snapshot} not encrypted but encrypted-only set, snapshot skipped");
             return true;
         }
     }
 
     false
 }
+
+pub(super) fn exclude_not_verified_or_encrypted(
+    item: &SnapshotListItem,
+    verified_only: bool,
+    encrypted_only: bool,
+) -> bool {
+    if verified_only {
+        match &item.verification {
+            Some(state) if state.state == VerifyState::Ok => (),
+            _ => return true,
+        }
+    }
+
+    if encrypted_only
+        && !item.files.iter().all(|content| {
+            if content.filename == MANIFEST_BLOB_NAME.as_ref() {
+                content.crypt_mode == Some(CryptMode::SignOnly)
+            } else if content.filename == CLIENT_LOG_BLOB_NAME.as_ref() {
+                true
+            } else {
+                content.crypt_mode == Some(CryptMode::Encrypt)
+            }
+        })
+    {
+        return true;
+    }
+
+    false
+}
-- 
2.39.5




From c.ebner at proxmox.com  Fri May 16 13:54:26 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 16 May 2025 13:54:26 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 2/5] api: factor out helper
 converting backup info to snapshot list item
In-Reply-To: <20250516115429.208563-1-c.ebner@proxmox.com>
References: <20250516115429.208563-1-c.ebner@proxmox.com>
Message-ID: <20250516115429.208563-3-c.ebner@proxmox.com>

Move the logic currently provided by a closure to it's dedicated
named helper function.

Further, in order to reuse the same code for the generation of the
snapshot list items to be used by the sync job source readers, move
the helper and related helper functions to the tools module.

No functional changes intended.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs | 135 +++---------------------------------
 src/tools/mod.rs            | 129 ++++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+), 127 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 55cb3a0b3..9b3ffb8e0 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -1,6 +1,5 @@
 //! Datastore Management
 
-use std::collections::HashSet;
 use std::ffi::OsStr;
 use std::ops::Deref;
 use std::os::unix::ffi::OsStrExt;
@@ -41,13 +40,12 @@ use pbs_api_types::{
     BackupContent, BackupGroupDeleteStats, BackupNamespace, BackupType, Counts, CryptMode,
     DataStoreConfig, DataStoreListItem, DataStoreMountStatus, DataStoreStatus,
     GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
-    MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
-    BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
-    BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
-    IGNORE_VERIFIED_BACKUPS_SCHEMA, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
-    PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
-    PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, PRIV_SYS_MODIFY, UPID, UPID_SCHEMA,
-    VERIFICATION_OUTDATED_AFTER_SCHEMA,
+    MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, BACKUP_ARCHIVE_NAME_SCHEMA,
+    BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA,
+    CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA,
+    MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP,
+    PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY,
+    PRIV_SYS_MODIFY, UPID, UPID_SCHEMA, VERIFICATION_OUTDATED_AFTER_SCHEMA,
 };
 use pbs_client::pxar::{create_tar, create_zip};
 use pbs_config::CachedUserInfo;
@@ -74,8 +72,8 @@ use crate::backup::{
     check_ns_privs_full, verify_all_backups, verify_backup_dir, verify_backup_group, verify_filter,
     ListAccessibleBackupGroups, NS_PRIVS_OK,
 };
-
 use crate::server::jobstate::{compute_schedule_status, Job, JobState};
+use crate::tools::{backup_info_to_snapshot_list_item, get_all_snapshot_files, read_backup_index};
 
 const GROUP_NOTES_FILE_NAME: &str = "notes";
 
@@ -114,56 +112,6 @@ fn check_privs_and_load_store(
     Ok(datastore)
 }
 
-fn read_backup_index(
-    backup_dir: &BackupDir,
-) -> Result<(BackupManifest, Vec<BackupContent>), Error> {
-    let (manifest, index_size) = backup_dir.load_manifest()?;
-
-    let mut result = Vec::new();
-    for item in manifest.files() {
-        result.push(BackupContent {
-            filename: item.filename.clone(),
-            crypt_mode: Some(item.crypt_mode),
-            size: Some(item.size),
-        });
-    }
-
-    result.push(BackupContent {
-        filename: MANIFEST_BLOB_NAME.to_string(),
-        crypt_mode: match manifest.signature {
-            Some(_) => Some(CryptMode::SignOnly),
-            None => Some(CryptMode::None),
-        },
-        size: Some(index_size),
-    });
-
-    Ok((manifest, result))
-}
-
-fn get_all_snapshot_files(
-    info: &BackupInfo,
-) -> Result<(BackupManifest, Vec<BackupContent>), Error> {
-    let (manifest, mut files) = read_backup_index(&info.backup_dir)?;
-
-    let file_set = files.iter().fold(HashSet::new(), |mut acc, item| {
-        acc.insert(item.filename.clone());
-        acc
-    });
-
-    for file in &info.files {
-        if file_set.contains(file) {
-            continue;
-        }
-        files.push(BackupContent {
-            filename: file.to_string(),
-            size: None,
-            crypt_mode: None,
-        });
-    }
-
-    Ok((manifest, files))
-}
-
 #[api(
     input: {
         properties: {
@@ -529,73 +477,6 @@ unsafe fn list_snapshots_blocking(
         (None, None) => datastore.list_backup_groups(ns.clone())?,
     };
 
-    let info_to_snapshot_list_item = |owner, info: BackupInfo| {
-        let backup = info.backup_dir.dir().to_owned();
-        let protected = info.protected;
-
-        match get_all_snapshot_files(&info) {
-            Ok((manifest, files)) => {
-                // extract the first line from notes
-                let comment: Option<String> = manifest.unprotected["notes"]
-                    .as_str()
-                    .and_then(|notes| notes.lines().next())
-                    .map(String::from);
-
-                let fingerprint = match manifest.fingerprint() {
-                    Ok(fp) => fp,
-                    Err(err) => {
-                        eprintln!("error parsing fingerprint: '{}'", err);
-                        None
-                    }
-                };
-
-                let verification: Option<SnapshotVerifyState> = match manifest.verify_state() {
-                    Ok(verify) => verify,
-                    Err(err) => {
-                        eprintln!("error parsing verification state : '{}'", err);
-                        None
-                    }
-                };
-
-                let size = Some(files.iter().map(|x| x.size.unwrap_or(0)).sum());
-
-                SnapshotListItem {
-                    backup,
-                    comment,
-                    verification,
-                    fingerprint,
-                    files,
-                    size,
-                    owner,
-                    protected,
-                }
-            }
-            Err(err) => {
-                eprintln!("error during snapshot file listing: '{}'", err);
-                let files = info
-                    .files
-                    .into_iter()
-                    .map(|filename| BackupContent {
-                        filename,
-                        size: None,
-                        crypt_mode: None,
-                    })
-                    .collect();
-
-                SnapshotListItem {
-                    backup,
-                    comment: None,
-                    verification: None,
-                    fingerprint: None,
-                    files,
-                    size: None,
-                    owner,
-                    protected,
-                }
-            }
-        }
-    };
-
     groups.iter().try_fold(Vec::new(), |mut snapshots, group| {
         let owner = match group.get_owner() {
             Ok(auth_id) => auth_id,
@@ -619,7 +500,7 @@ unsafe fn list_snapshots_blocking(
         snapshots.extend(
             group_backups
                 .into_iter()
-                .map(|info| info_to_snapshot_list_item(Some(owner.clone()), info)),
+                .map(|info| backup_info_to_snapshot_list_item(&info, &owner)),
         );
 
         Ok(snapshots)
diff --git a/src/tools/mod.rs b/src/tools/mod.rs
index 322894dd7..6556effe3 100644
--- a/src/tools/mod.rs
+++ b/src/tools/mod.rs
@@ -3,9 +3,16 @@
 //! This is a collection of small and useful tools.
 
 use anyhow::{bail, Error};
+use std::collections::HashSet;
 
+use pbs_api_types::{
+    Authid, BackupContent, CryptMode, SnapshotListItem, SnapshotVerifyState, MANIFEST_BLOB_NAME,
+};
 use proxmox_http::{client::Client, HttpOptions, ProxyConfig};
 
+use pbs_datastore::backup_info::{BackupDir, BackupInfo};
+use pbs_datastore::manifest::BackupManifest;
+
 pub mod config;
 pub mod disks;
 pub mod fs;
@@ -61,3 +68,125 @@ pub fn setup_safe_path_env() {
         std::env::remove_var(name);
     }
 }
+
+pub(crate) fn read_backup_index(
+    backup_dir: &BackupDir,
+) -> Result<(BackupManifest, Vec<BackupContent>), Error> {
+    let (manifest, index_size) = backup_dir.load_manifest()?;
+
+    let mut result = Vec::new();
+    for item in manifest.files() {
+        result.push(BackupContent {
+            filename: item.filename.clone(),
+            crypt_mode: Some(item.crypt_mode),
+            size: Some(item.size),
+        });
+    }
+
+    result.push(BackupContent {
+        filename: MANIFEST_BLOB_NAME.to_string(),
+        crypt_mode: match manifest.signature {
+            Some(_) => Some(CryptMode::SignOnly),
+            None => Some(CryptMode::None),
+        },
+        size: Some(index_size),
+    });
+
+    Ok((manifest, result))
+}
+
+pub(crate) fn get_all_snapshot_files(
+    info: &BackupInfo,
+) -> Result<(BackupManifest, Vec<BackupContent>), Error> {
+    let (manifest, mut files) = read_backup_index(&info.backup_dir)?;
+
+    let file_set = files.iter().fold(HashSet::new(), |mut acc, item| {
+        acc.insert(item.filename.clone());
+        acc
+    });
+
+    for file in &info.files {
+        if file_set.contains(file) {
+            continue;
+        }
+        files.push(BackupContent {
+            filename: file.to_string(),
+            size: None,
+            crypt_mode: None,
+        });
+    }
+
+    Ok((manifest, files))
+}
+
+/// Helper to transform `BackupInfo` to `SnapshotListItem` with given owner.
+pub(crate) fn backup_info_to_snapshot_list_item(
+    info: &BackupInfo,
+    owner: &Authid,
+) -> SnapshotListItem {
+    let backup = info.backup_dir.dir().to_owned();
+    let protected = info.protected;
+    let owner = Some(owner.to_owned());
+
+    match get_all_snapshot_files(info) {
+        Ok((manifest, files)) => {
+            // extract the first line from notes
+            let comment: Option<String> = manifest.unprotected["notes"]
+                .as_str()
+                .and_then(|notes| notes.lines().next())
+                .map(String::from);
+
+            let fingerprint = match manifest.fingerprint() {
+                Ok(fp) => fp,
+                Err(err) => {
+                    eprintln!("error parsing fingerprint: '{}'", err);
+                    None
+                }
+            };
+
+            let verification: Option<SnapshotVerifyState> = match manifest.verify_state() {
+                Ok(verify) => verify,
+                Err(err) => {
+                    eprintln!("error parsing verification state : '{err}'");
+                    None
+                }
+            };
+
+            let size = Some(files.iter().map(|x| x.size.unwrap_or(0)).sum());
+
+            SnapshotListItem {
+                backup,
+                comment,
+                verification,
+                fingerprint,
+                files,
+                size,
+                owner,
+                protected,
+            }
+        }
+        Err(err) => {
+            eprintln!("error during snapshot file listing: '{err}'");
+            let files = info
+                .files
+                .iter()
+                .map(|filename| BackupContent {
+                    filename: filename.to_owned(),
+                    size: None,
+                    crypt_mode: None,
+                })
+                .collect();
+
+            SnapshotListItem {
+                backup,
+                comment: None,
+                verification: None,
+                fingerprint: None,
+                files,
+                size: None,
+                owner,
+                protected,
+            }
+        }
+    }
+}
-- 
2.39.5




From s.sterz at proxmox.com  Fri May 16 15:11:22 2025
From: s.sterz at proxmox.com (Shannon Sterz)
Date: Fri, 16 May 2025 15:11:22 +0200
Subject: [pbs-devel] [PATCH proxmox] auth-api: remove ticket info in old
 create ticket endpoint
Message-ID: <20250516131122.276231-1-s.sterz@proxmox.com>

this should make the endpoint behave closer to how it behaved before
the HttpOnly changes. the `ticket_info` field is superfluous anyway,
as the response also includes the proper ticket in this case already.

Signed-off-by: Shannon Sterz <s.sterz at proxmox.com>
---

this came to light when Dietmar tried to use the new proxmox-yew-comp
client for authenticating against PBS. the client would default to the
new HttpOnly cookie there as it say the response contained the
ticket_info field. however, that is wrong, the old api endpoint should
not have returned this additional info, so remove it here again.

 proxmox-auth-api/src/api/access.rs | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/proxmox-auth-api/src/api/access.rs b/proxmox-auth-api/src/api/access.rs
index 396935f5..95f36f53 100644
--- a/proxmox-auth-api/src/api/access.rs
+++ b/proxmox-auth-api/src/api/access.rs
@@ -60,7 +60,13 @@ pub async fn create_ticket(
         .downcast_ref::<RestEnvironment>()
         .ok_or_else(|| format_err!("detected wrong RpcEnvironment type"))?;

-    handle_ticket_creation(create_params, env).await
+    handle_ticket_creation(create_params, env)
+        .await
+        // remove the superfluous ticket_info to not confuse clients
+        .map(|mut info| {
+            info.ticket_info = None;
+            info
+        })
 }

 pub const API_METHOD_LOGOUT: ApiMethod = ApiMethod::new(
--
2.39.5




From c.ebner at proxmox.com  Mon May 19 07:55:18 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 07:55:18 +0200
Subject: [pbs-devel] [PATCH v2 proxmox-backup 4/4] garbage collection: track
 chunk cache stats and show in task log
In-Reply-To: <20250519055518.3747-1-c.ebner@proxmox.com>
References: <20250519055518.3747-1-c.ebner@proxmox.com>
Message-ID: <20250519055518.3747-5-c.ebner@proxmox.com>

Count the chunk cache hits and misses and display the resulting
values and the hit ratio in the garbage collection task log summary.

This allows to investigate possible issues and tune cache capacity,
also by being able to compare to other values in the summary such
as the on disk chunk count.

Exemplary output
```
2025-05-16T22:31:53+02:00: Chunk cache: hits 15817, misses 873 (hit ratio 94.77%)
2025-05-16T22:31:53+02:00: Removed garbage: 0 B
2025-05-16T22:31:53+02:00: Removed chunks: 0
2025-05-16T22:31:53+02:00: Original data usage: 64.961 GiB
2025-05-16T22:31:53+02:00: On-Disk usage: 1.037 GiB (1.60%)
2025-05-16T22:31:53+02:00: On-Disk chunks: 874
2025-05-16T22:31:53+02:00: Deduplication factor: 62.66
2025-05-16T22:31:53+02:00: Average chunk size: 1.215 MiB
```

Sidenote: the discrepancy between cache miss counter and on-disk
chunk count in the output shown above can be attributed to the all
zero chunk, inserted during the atime update check at the start of
garbage collection, however not being referenced by any index file in
this examplary case.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
changes since version 1:
- add cache hit ratio to output

 pbs-datastore/src/datastore.rs | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index dbff84bf3..fcfa7e694 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1087,8 +1087,10 @@ impl DataStore {
             // Avoid multiple expensive atime updates by utimensat
             if let Some(chunk_lru_cache) = chunk_lru_cache {
                 if chunk_lru_cache.insert(*digest, ()) {
+                    status.cache_hits += 1;
                     continue;
                 }
+                status.cache_misses += 1;
             }
 
             if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
@@ -1355,6 +1357,15 @@ impl DataStore {
                 worker,
             )?;
 
+            let total_cache_counts = gc_status.cache_hits + gc_status.cache_misses;
+            if total_cache_counts > 0 {
+                let cache_hit_ratio =
+                    (gc_status.cache_hits as f64 * 100.) / total_cache_counts as f64;
+                info!(
+                    "Chunk cache: hits {}, misses {} (hit ratio {cache_hit_ratio:.2}%)",
+                    gc_status.cache_hits, gc_status.cache_misses,
+                );
+            }
             info!(
                 "Removed garbage: {}",
                 HumanByte::from(gc_status.removed_bytes),
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 07:55:16 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 07:55:16 +0200
Subject: [pbs-devel] [PATCH v2 proxmox-backup 2/4] tools: lru cache:
 document limitations for cache capacity
In-Reply-To: <20250519055518.3747-1-c.ebner@proxmox.com>
References: <20250519055518.3747-1-c.ebner@proxmox.com>
Message-ID: <20250519055518.3747-3-c.ebner@proxmox.com>

Since commit 1e7639bf ("fixup minimum lru capacity") the minimum
cache capacity is forced to be 1 to bypass edge cases for it being 0.

Explicitly mention this in the doc comment, as this behavior can be
unexpected.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
changes since version 1:
- not present in previous version

 pbs-tools/src/lru_cache.rs | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/pbs-tools/src/lru_cache.rs b/pbs-tools/src/lru_cache.rs
index 9e0112647..94757bbf7 100644
--- a/pbs-tools/src/lru_cache.rs
+++ b/pbs-tools/src/lru_cache.rs
@@ -121,6 +121,8 @@ impl<K, V> LruCache<K, V> {
 
 impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
     /// Create LRU cache instance which holds up to `capacity` nodes at once.
+    ///
+    /// Forces a minimum `capacity` of 1 in case of the given value being 0.
     pub fn new(capacity: usize) -> Self {
         let capacity = capacity.max(1);
         Self {
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 07:55:14 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 07:55:14 +0200
Subject: [pbs-devel] [PATCH v2 proxmox-backup 0/4] add GC cache stats and
 fix disabled state
Message-ID: <20250519055518.3747-1-c.ebner@proxmox.com>

Allows better fine-tuning of the garbage collection cache capacity by
providing the hit and miss count, as well as the hit ratio as output
to the garbage collection task log.

Further, fixes an issue with the cache not being disabled in case the
cache capacity was explicitly set to 0, by bypassing it altogether for
that case.

Changes since version 1 (thanks Lukas for feedback):
- Also display the cache hit ratio
- Fix the cache not being disabled when the capacity is set to 0,
  discovered while investigating the hit ratio for different
  capacities.

proxmox:

Christian Ebner (1):
  pbs api types: extend garbage collection status by cache stats

 pbs-api-types/src/datastore.rs | 4 ++++
 1 file changed, 4 insertions(+)

proxmox-backup:

Christian Ebner (3):
  tools: lru cache: document limitations for cache capacity
  garbage collection: bypass cache if gc-cache-capacity is 0
  garbage collection: track chunk cache stats and show in task log

 pbs-datastore/src/datastore.rs | 25 +++++++++++++++++++++----
 pbs-tools/src/lru_cache.rs     |  2 ++
 2 files changed, 23 insertions(+), 4 deletions(-)

-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 07:55:15 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 07:55:15 +0200
Subject: [pbs-devel] [PATCH v2 proxmox 1/4] pbs api types: extend garbage
 collection status by cache stats
In-Reply-To: <20250519055518.3747-1-c.ebner@proxmox.com>
References: <20250519055518.3747-1-c.ebner@proxmox.com>
Message-ID: <20250519055518.3747-2-c.ebner@proxmox.com>

Add the number of cache hits and cache misses encountered during
phase 1 of garbage collection in order to display this information
in the garbage collection task log summary.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
changes since version 1:
- no changes

 pbs-api-types/src/datastore.rs | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 5bd953ac..4fb1eb80 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -1459,6 +1459,10 @@ pub struct GarbageCollectionStatus {
     pub removed_bad: usize,
     /// Number of chunks still marked as .bad after garbage collection.
     pub still_bad: usize,
+    /// Number of atime update cache hits
+    pub cache_hits: usize,
+    /// Number of atime update cache misses
+    pub cache_misses: usize,
 }
 
 #[api(
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 07:55:17 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 07:55:17 +0200
Subject: [pbs-devel] [PATCH v2 proxmox-backup 3/4] garbage collection:
 bypass cache if gc-cache-capacity is 0
In-Reply-To: <20250519055518.3747-1-c.ebner@proxmox.com>
References: <20250519055518.3747-1-c.ebner@proxmox.com>
Message-ID: <20250519055518.3747-4-c.ebner@proxmox.com>

Since commit 1e7639bf ("fixup minimum lru capacity") the LRU cache
capacity is set to a minimum value of 1 to avoid issues with the edge
case of 0 capacity.

In commit f1a711c8 ("garbage collection: set phase1 LRU cache
capacity by tuning option") this was not taken into account, allowing
to set values in the range [0, 8*1024*1024] via the datastores tuning
parameters.

Bypass the cache by making it optional and do not use it if the cache
capacity is set to 0, which implies it being disabled.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
changes since version 1:
- not present in previous version

 pbs-datastore/src/datastore.rs | 14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index cbf78ecb6..dbff84bf3 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1072,7 +1072,7 @@ impl DataStore {
         &self,
         index: Box<dyn IndexFile>,
         file_name: &Path, // only used for error reporting
-        chunk_lru_cache: &mut LruCache<[u8; 32], ()>,
+        chunk_lru_cache: &mut Option<LruCache<[u8; 32], ()>>,
         status: &mut GarbageCollectionStatus,
         worker: &dyn WorkerTaskContext,
     ) -> Result<(), Error> {
@@ -1085,8 +1085,10 @@ impl DataStore {
             let digest = index.index_digest(pos).unwrap();
 
             // Avoid multiple expensive atime updates by utimensat
-            if chunk_lru_cache.insert(*digest, ()) {
-                continue;
+            if let Some(chunk_lru_cache) = chunk_lru_cache {
+                if chunk_lru_cache.insert(*digest, ()) {
+                    continue;
+                }
             }
 
             if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
@@ -1130,7 +1132,11 @@ impl DataStore {
         let mut unprocessed_index_list = self.list_index_files()?;
         let mut index_count = unprocessed_index_list.len();
 
-        let mut chunk_lru_cache = LruCache::new(cache_capacity);
+        let mut chunk_lru_cache = if cache_capacity > 0 {
+            Some(LruCache::new(cache_capacity))
+        } else {
+            None
+        };
         let mut processed_index_files = 0;
         let mut last_percentage: usize = 0;
 
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 07:56:35 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 07:56:35 +0200
Subject: [pbs-devel] superseded: [PATCH proxmox proxmox-backup 0/2] add GC
 chunk cache stats
In-Reply-To: <20250516085836.82494-1-c.ebner@proxmox.com>
References: <20250516085836.82494-1-c.ebner@proxmox.com>
Message-ID: <918f746b-93e0-4265-9fb0-18a485201fbf@proxmox.com>

superseded-by version 2:
https://lore.proxmox.com/pbs-devel/20250519055518.3747-1-c.ebner at proxmox.com/T/



From c.ebner at proxmox.com  Mon May 19 13:46:04 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:04 +0200
Subject: [pbs-devel] [RFC proxmox-backup 03/39] fmt: fix minor formatting
 issues
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-4-c.ebner@proxmox.com>

These are currently not shown by a `cargo fmt --check`.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/mod.rs | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/src/api2/backup/mod.rs b/src/api2/backup/mod.rs
index 629df933e..567bca3ef 100644
--- a/src/api2/backup/mod.rs
+++ b/src/api2/backup/mod.rs
@@ -166,7 +166,7 @@ fn upgrade_to_backup_protocol(
                     Ok(None) => {
                         // no verify state found, treat as valid
                         Some(info)
-                    },
+                    }
                     Err(err) => {
                         warn!("error parsing the snapshot manifest: {err:#}");
                         Some(info)
@@ -236,7 +236,8 @@ fn upgrade_to_backup_protocol(
                     .and_then(move |conn| {
                         env2.debug("protocol upgrade done");
 
-                        let mut http = hyper::server::conn::http2::Builder::new(ExecInheritLogContext);
+                        let mut http =
+                            hyper::server::conn::http2::Builder::new(ExecInheritLogContext);
                         // increase window size: todo - find optiomal size
                         let window_size = 32 * 1024 * 1024; // max = (1 << 31) - 2
                         http.initial_stream_window_size(window_size);
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:02 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:02 +0200
Subject: [pbs-devel] [RFC proxmox 1/2] pbs-api-types: add types for S3
 client configs and secrets
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-2-c.ebner@proxmox.com>

Adds the new config types `S3ClientConfig` and `S3ClientSecret` to
configure datastore backends using an S3 compatible object store.

Secrets are stored as different config to never be returned on api
calls, only allowing to set/update the values.

Use a different name (`secrets_id`) for the unique identifier in case
of the secrets type, although the same id should be used for storing
and lookup. By this, clashing of property names when using flattened
types as api parameters is avoided.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-api-types/src/lib.rs |   3 +
 pbs-api-types/src/s3.rs  | 138 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 141 insertions(+)
 create mode 100644 pbs-api-types/src/s3.rs

diff --git a/pbs-api-types/src/lib.rs b/pbs-api-types/src/lib.rs
index 99ec7961..7a5ea11d 100644
--- a/pbs-api-types/src/lib.rs
+++ b/pbs-api-types/src/lib.rs
@@ -147,6 +147,9 @@ pub use remote::*;
 mod pathpatterns;
 pub use pathpatterns::*;
 
+mod s3;
+pub use s3::*;
+
 mod tape;
 pub use tape::*;
 
diff --git a/pbs-api-types/src/s3.rs b/pbs-api-types/src/s3.rs
new file mode 100644
index 00000000..40c502ba
--- /dev/null
+++ b/pbs-api-types/src/s3.rs
@@ -0,0 +1,138 @@
+use anyhow::bail;
+use serde::{Deserialize, Serialize};
+
+use proxmox_schema::api_types::{
+    CERT_FINGERPRINT_SHA256_SCHEMA, DNS_NAME_OR_IP_SCHEMA, SAFE_ID_FORMAT,
+};
+use proxmox_schema::{api, const_regex, ApiStringFormat, Schema, StringSchema, Updater};
+
+#[rustfmt::skip]
+pub const S3_BUCKET_NAME_REGEX_STR: &str = r"^[a-z0-9]([a-z0-9\-]*[a-z0-9])?$";
+
+const_regex! {
+    /// Regex to match S3 bucket names.
+    ///
+    /// Be as strict as possible following the rules as described here:
+    /// https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html#general-purpose-bucket-names
+    pub S3_BUCKET_NAME_REGEX = r"^[a-z0-9]([a-z0-9\-]*[a-z0-9])?$";
+    /// Regex to match S3 regions.
+    pub S3_REGION_REGEX = r"^[a-z]{2}\-[a-z]{4,}\-[0-9]$";
+}
+
+pub const S3_REGION_FORMAT: ApiStringFormat = ApiStringFormat::Pattern(&S3_REGION_REGEX);
+
+pub const S3_CLIENT_ID_SCHEMA: Schema =
+    StringSchema::new("Unique ID to identify s3 client config.")
+        .format(&SAFE_ID_FORMAT)
+        .min_length(3)
+        .max_length(32)
+        .schema();
+
+pub const S3_REGION_SCHEMA: Schema = StringSchema::new("Region to access S3 object store.")
+    .format(&S3_REGION_FORMAT)
+    .min_length(3)
+    .max_length(32)
+    .schema();
+
+pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3 object store.")
+    .format(&ApiStringFormat::VerifyFn(|bucket_name| {
+        if !(S3_BUCKET_NAME_REGEX.regex_obj)().is_match(bucket_name) {
+            bail!("Bucket name does not match the regex pattern");
+        }
+
+        // Exclude pre- and postfixes described here:
+        // https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html#general-purpose-bucket-names
+        let forbidden_prefixes = ["xn--", "sthree-", "amzn-s3-demo-"];
+        for prefix in forbidden_prefixes {
+            if bucket_name.starts_with(prefix) {
+                bail!("Bucket name cannot start with '{prefix}'");
+            }
+        }
+
+        let forbidden_postfixes = ["--ol-s3", ".mrap", "--x-s3"];
+        for postfix in forbidden_postfixes {
+            if bucket_name.ends_with(postfix) {
+                bail!("Bucket name cannot end with '{postfix}'");
+            }
+        }
+
+        Ok(())
+    }))
+    .min_length(3)
+    .max_length(63)
+    .schema();
+
+#[api(
+    properties: {
+        id: {
+            schema: S3_CLIENT_ID_SCHEMA,
+        },
+        host: {
+            schema: DNS_NAME_OR_IP_SCHEMA,
+        },
+        bucket: {
+            schema: S3_BUCKET_NAME_SCHEMA,
+        },
+        port: {
+            type: u16,
+            description: "Port to access S3 object store.",
+            optional: true,
+        },
+        region: {
+            schema: S3_REGION_SCHEMA,
+            optional: true,
+        },
+        fingerprint: {
+            schema: CERT_FINGERPRINT_SHA256_SCHEMA,
+            optional: true,
+        },
+        "access-key": {
+            type: String,
+            description: "Access key for S3 object store.",
+        },
+    }
+)]
+#[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// S3 client configuration properties.
+pub struct S3ClientConfig {
+    #[updater(skip)]
+    pub id: String,
+    pub host: String,
+    pub bucket: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub port: Option<u16>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub region: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub fingerprint: Option<String>,
+    pub access_key: String,
+}
+
+impl S3ClientConfig {
+    pub fn acl_path(&self) -> Vec<&str> {
+        // Needs permissions on root path
+        Vec::new()
+    }
+}
+
+#[api(
+    properties: {
+        "secrets-id": {
+            type: String,
+            description: "Unique ID to identify s3 client secret config.",
+        },
+        "secret-key": {
+            type: String,
+            description: "Secret key for S3 object store.",
+        },
+    }
+)]
+#[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// S3 client secrets configuration properties.
+pub struct S3ClientSecretsConfig {
+    #[updater(skip)]
+    pub secrets_id: String,
+    pub secret_key: String,
+}
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:06 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:06 +0200
Subject: [pbs-devel] [RFC proxmox-backup 05/39] s3 client: add crate for AWS
 S3 compatible object store client
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-6-c.ebner@proxmox.com>

Adds the client to connect to an AWS S3 compatible object store API.
Force the use of an TLS encrypted connection as the communication
with the object store will contain sensitive information.
For self-signed certificates, check the fingerprint against the one
configured. This follows along the lines of the PBS client, used to
connect to the PBS server API.

The `S3Client` stores the client state and has to be configured upon
instantiation by providing `S3ClientOptions`.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 Cargo.toml                  |   3 +
 pbs-s3-client/Cargo.toml    |  16 +++++
 pbs-s3-client/src/client.rs | 131 ++++++++++++++++++++++++++++++++++++
 pbs-s3-client/src/lib.rs    |   2 +
 4 files changed, 152 insertions(+)
 create mode 100644 pbs-s3-client/Cargo.toml
 create mode 100644 pbs-s3-client/src/client.rs
 create mode 100644 pbs-s3-client/src/lib.rs

diff --git a/Cargo.toml b/Cargo.toml
index 6de6a6527..c2b0029ac 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -36,6 +36,7 @@ members = [
     "pbs-fuse-loop",
     "pbs-key-config",
     "pbs-pxar-fuse",
+    "pbs-s3-client",
     "pbs-tape",
     "pbs-tools",
 
@@ -105,6 +106,7 @@ pbs-datastore = { path = "pbs-datastore" }
 pbs-fuse-loop = { path = "pbs-fuse-loop" }
 pbs-key-config = { path = "pbs-key-config" }
 pbs-pxar-fuse = { path = "pbs-pxar-fuse" }
+pbs-s3-client = { path = "pbs-s3-client" }
 pbs-tape = { path = "pbs-tape" }
 pbs-tools = { path = "pbs-tools" }
 
@@ -245,6 +247,7 @@ pbs-client.workspace = true
 pbs-config.workspace = true
 pbs-datastore.workspace = true
 pbs-key-config.workspace = true
+pbs-s3-client.workspace = true
 pbs-tape.workspace = true
 pbs-tools.workspace = true
 proxmox-rrd.workspace = true
diff --git a/pbs-s3-client/Cargo.toml b/pbs-s3-client/Cargo.toml
new file mode 100644
index 000000000..1999c3323
--- /dev/null
+++ b/pbs-s3-client/Cargo.toml
@@ -0,0 +1,16 @@
+[package]
+name = "pbs-s3-client"
+version = "0.1.0"
+authors.workspace = true
+edition.workspace = true
+description = "low level client for AWS S3 compatible object stores"
+rust-version.workspace = true
+
+[dependencies]
+anyhow.workspace = true
+hex = { workspace = true, features = [ "serde" ] }
+hyper.workspace = true
+openssl.workspace = true
+tracing.workspace = true
+
+proxmox-http.workspace = true
diff --git a/pbs-s3-client/src/client.rs b/pbs-s3-client/src/client.rs
new file mode 100644
index 000000000..e001cc7b0
--- /dev/null
+++ b/pbs-s3-client/src/client.rs
@@ -0,0 +1,131 @@
+use std::sync::{Arc, Mutex};
+use std::time::Duration;
+
+use anyhow::{bail, format_err, Context, Error};
+use hyper::client::{Client, HttpConnector};
+use hyper::http::uri::Authority;
+use hyper::Body;
+use openssl::hash::MessageDigest;
+use openssl::ssl::{SslConnector, SslMethod, SslVerifyMode};
+use openssl::x509::X509StoreContextRef;
+use tracing::error;
+
+use proxmox_http::client::HttpsConnector;
+
+const S3_HTTP_CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
+const S3_TCP_KEEPALIVE_TIME: u32 = 120;
+
+/// Configuration options for client
+pub struct S3ClientOptions {
+    pub host: String,
+    pub port: Option<u16>,
+    pub bucket: String,
+    pub secret_key: String,
+    pub access_key: String,
+    pub region: String,
+    pub fingerprint: Option<String>,
+}
+
+/// S3 client for object stores compatible with the AWS S3 API
+pub struct S3Client {
+    client: Client<HttpsConnector>,
+    options: S3ClientOptions,
+    authority: Authority,
+}
+
+impl S3Client {
+    pub fn new(options: S3ClientOptions) -> Result<Self, Error> {
+        let expected_fingerprint = options.fingerprint.clone();
+        let verified_fingerprint = Arc::new(Mutex::new(None));
+        let trust_openssl_valid = Arc::new(Mutex::new(true));
+        let mut ssl_connector_builder = SslConnector::builder(SslMethod::tls())?;
+        ssl_connector_builder.set_verify_callback(
+            SslVerifyMode::PEER,
+            move |openssl_valid, context| match Self::verify_certificate_fingerprint(
+                openssl_valid,
+                context,
+                expected_fingerprint.clone(),
+                trust_openssl_valid.clone(),
+            ) {
+                Ok(None) => true,
+                Ok(Some(fingerprint)) => {
+                    *verified_fingerprint.lock().unwrap() = Some(fingerprint);
+                    true
+                }
+                Err(err) => {
+                    error!("certificate validation failed {err:#}");
+                    false
+                }
+            },
+        );
+
+        let mut http_connector = HttpConnector::new();
+        // want communication to object store backend api to always use https
+        http_connector.enforce_http(false);
+        http_connector.set_connect_timeout(Some(S3_HTTP_CONNECT_TIMEOUT));
+        let https_connector = HttpsConnector::with_connector(
+            http_connector,
+            ssl_connector_builder.build(),
+            S3_TCP_KEEPALIVE_TIME,
+        );
+        let client = Client::builder().build::<_, Body>(https_connector);
+        let authority = if let Some(port) = options.port {
+            format!("{}:{port}", options.host)
+        } else {
+            options.host.clone()
+        };
+        let authority = Authority::try_from(authority)?;
+
+        Ok(Self {
+            client,
+            options,
+            authority,
+        })
+    }
+
+    fn verify_certificate_fingerprint(
+        openssl_valid: bool,
+        context: &mut X509StoreContextRef,
+        expected_fingerprint: Option<String>,
+        trust_openssl: Arc<Mutex<bool>>,
+    ) -> Result<Option<String>, Error> {
+        let mut trust_openssl_valid = trust_openssl.lock().unwrap();
+
+        // only rely on openssl prevalidation if was not forced earlier
+        if openssl_valid && *trust_openssl_valid {
+            return Ok(None);
+        }
+
+        let certificate = match context.current_cert() {
+            Some(certificate) => certificate,
+            None => bail!("context lacks current certificate."),
+        };
+
+        if context.error_depth() > 0 {
+            *trust_openssl_valid = false;
+            return Ok(None);
+        }
+
+        let certificate_digest = certificate
+            .digest(MessageDigest::sha256())
+            .context("failed to calculate certificate digest")?;
+        let certificate_fingerprint = hex::encode(certificate_digest);
+        let certificate_fingerprint = certificate_fingerprint
+            .as_bytes()
+            .chunks(2)
+            .map(|v| std::str::from_utf8(v).unwrap())
+            .collect::<Vec<&str>>()
+            .join(":");
+
+        if let Some(expected_fingerprint) = expected_fingerprint {
+            let expected_fingerprint = expected_fingerprint.to_lowercase();
+            if expected_fingerprint == certificate_fingerprint {
+                return Ok(Some(certificate_fingerprint));
+            }
+        }
+
+        Err(format_err!(
+            "unexpected certificate fingerprint {certificate_fingerprint}"
+        ))
+    }
+}
diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
new file mode 100644
index 000000000..533ceab8e
--- /dev/null
+++ b/pbs-s3-client/src/lib.rs
@@ -0,0 +1,2 @@
+mod client;
+pub use client::{S3Client, S3ClientOptions};
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:09 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:09 +0200
Subject: [pbs-devel] [RFC proxmox-backup 08/39] s3 client: add helper for
 last modified timestamp parsing
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-9-c.ebner@proxmox.com>

Adds a helper to parse modified timestamps as encountered in s3 list
objects v2 and copy object api calls. Further, allow to convert a
timestamp to a Duration since unix epoch in order for easy comparison
between timestamps during phase 2 of garbage collection.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 Cargo.toml               |   1 +
 pbs-s3-client/Cargo.toml |   2 +
 pbs-s3-client/src/lib.rs | 118 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 121 insertions(+)

diff --git a/Cargo.toml b/Cargo.toml
index c2b0029ac..3f51b356c 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -144,6 +144,7 @@ regex = "1.5.5"
 rustyline = "9"
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
+serde_plain = "1.0"
 siphasher = "0.3"
 syslog = "6"
 tar = "0.4"
diff --git a/pbs-s3-client/Cargo.toml b/pbs-s3-client/Cargo.toml
index 11189ea50..9ee546200 100644
--- a/pbs-s3-client/Cargo.toml
+++ b/pbs-s3-client/Cargo.toml
@@ -11,6 +11,8 @@ anyhow.workspace = true
 hex = { workspace = true, features = [ "serde" ] }
 hyper.workspace = true
 openssl.workspace = true
+serde.workspace = true
+serde_plain.workspace = true
 tracing.workspace = true
 url.workspace = true
 
diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
index a4081df15..308db64d8 100644
--- a/pbs-s3-client/src/lib.rs
+++ b/pbs-s3-client/src/lib.rs
@@ -3,3 +3,121 @@ mod client;
 pub use client::{S3Client, S3ClientOptions};
 mod object_key;
 pub use object_key::{S3ObjectKey, S3_CONTENT_PREFIX};
+
+use std::time::Duration;
+
+use anyhow::{bail, Error};
+
+#[derive(Debug)]
+pub struct LastModifiedTimestamp {
+    epoch: i64,
+    milliseconds: u64,
+}
+
+impl LastModifiedTimestamp {
+    pub fn to_duration(&self) -> Result<Duration, Error> {
+        let secs = u64::try_from(self.epoch)?;
+        let mut duration = Duration::from_secs(secs);
+        duration += Duration::from_millis(self.milliseconds);
+        Ok(duration)
+    }
+}
+
+impl std::str::FromStr for LastModifiedTimestamp {
+    type Err = Error;
+
+    fn from_str(timestamp: &str) -> Result<Self, Self::Err> {
+        let input = timestamp.as_bytes();
+
+        let expect = |pos: usize, c: u8| {
+            if input[pos] != c {
+                bail!("unexpected char at pos {pos}");
+            }
+            Ok(())
+        };
+
+        let digit = |pos: usize| -> Result<i32, Error> {
+            let digit = input[pos] as i32;
+            if !(48..=57).contains(&digit) {
+                bail!("unexpected char at pos {pos}");
+            }
+            Ok(digit - 48)
+        };
+
+        fn check_max(i: i32, max: i32) -> Result<i32, Error> {
+            if i > max {
+                bail!("value too large ({i} > {max})");
+            }
+            Ok(i)
+        }
+
+        if input.len() < 20 || input.len() > 25 {
+            bail!("timestamp of unexpected length");
+        }
+
+        if b'.' != input[19] {
+            bail!("unexpected milliseconds separator");
+        }
+        let tz = input[23];
+
+        match tz {
+            b'Z' => {
+                if input.len() != 24 {
+                    bail!("unexpected length in UTC timestamp");
+                }
+            }
+            b'+' | b'-' => {
+                if input.len() != 29 {
+                    bail!("unexpected length in timestamp");
+                }
+            }
+            _ => bail!("unexpected timezone indicator"),
+        }
+
+        let mut tm = proxmox_time::TmEditor::new(true);
+
+        tm.set_year(digit(0)? * 1000 + digit(1)? * 100 + digit(2)? * 10 + digit(3)?)?;
+        expect(4, b'-')?;
+        tm.set_mon(check_max(digit(5)? * 10 + digit(6)?, 12)?)?;
+        expect(7, b'-')?;
+        tm.set_mday(check_max(digit(8)? * 10 + digit(9)?, 31)?)?;
+
+        expect(10, b'T')?;
+
+        tm.set_hour(check_max(digit(11)? * 10 + digit(12)?, 23)?)?;
+        expect(13, b':')?;
+        tm.set_min(check_max(digit(14)? * 10 + digit(15)?, 59)?)?;
+        expect(16, b':')?;
+        tm.set_sec(check_max(digit(17)? * 10 + digit(18)?, 60)?)?;
+        expect(19, b'.')?;
+        let milliseconds: u64 = String::from_utf8(input[20..23].to_vec())?.parse()?;
+
+        let epoch = tm.into_epoch()?;
+
+        if tz == b'Z' {
+            return Ok(Self {
+                epoch,
+                milliseconds,
+            });
+        }
+
+        let hours = check_max(digit(20)? * 10 + digit(21)?, 23)?;
+        expect(22, b':')?;
+        let mins = check_max(digit(23)? * 10 + digit(24)?, 59)?;
+
+        let offset = (hours * 3600 + mins * 60) as i64;
+
+        let epoch = match tz {
+            b'+' => epoch - offset,
+            b'-' => epoch + offset,
+            _ => unreachable!(), // already checked above
+        };
+
+        Ok(Self {
+            epoch,
+            milliseconds,
+        })
+    }
+}
+
+serde_plain::derive_deserialize_from_fromstr!(LastModifiedTimestamp, "last modified timestamp");
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:12 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:12 +0200
Subject: [pbs-devel] [RFC proxmox-backup 11/39] config: introduce s3 object
 store client configuration
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-12-c.ebner@proxmox.com>

Adds the client configuration for s3 object store as dedicated
configuration files, with secrets being stored separately from the
regular configuration and excluded from api responses for security
reasons.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-config/src/lib.rs |  1 +
 pbs-config/src/s3.rs  | 73 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 74 insertions(+)
 create mode 100644 pbs-config/src/s3.rs

diff --git a/pbs-config/src/lib.rs b/pbs-config/src/lib.rs
index 9c4d77c24..d03c079ab 100644
--- a/pbs-config/src/lib.rs
+++ b/pbs-config/src/lib.rs
@@ -10,6 +10,7 @@ pub mod network;
 pub mod notifications;
 pub mod prune;
 pub mod remote;
+pub mod s3;
 pub mod sync;
 pub mod tape_job;
 pub mod token_shadow;
diff --git a/pbs-config/src/s3.rs b/pbs-config/src/s3.rs
new file mode 100644
index 000000000..5496b12bb
--- /dev/null
+++ b/pbs-config/src/s3.rs
@@ -0,0 +1,73 @@
+use std::sync::LazyLock;
+
+use anyhow::Error;
+
+use proxmox_schema::*;
+use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
+
+use pbs_api_types::{S3ClientConfig, S3ClientSecretsConfig, JOB_ID_SCHEMA};
+
+use crate::{open_backup_lockfile, replace_backup_config, BackupLockGuard};
+
+pub static CONFIG: LazyLock<SectionConfig> = LazyLock::new(init);
+
+fn init() -> SectionConfig {
+    let obj_schema = match S3ClientConfig::API_SCHEMA {
+        Schema::Object(ref obj_schema) => obj_schema,
+        _ => unreachable!(),
+    };
+    let secrets_obj_schema = match S3ClientSecretsConfig::API_SCHEMA {
+        Schema::Object(ref obj_schema) => obj_schema,
+        _ => unreachable!(),
+    };
+
+    let plugin =
+        SectionConfigPlugin::new("s3client".to_string(), Some(String::from("id")), obj_schema);
+    let secrets_plugin = SectionConfigPlugin::new(
+        "s3secrets".to_string(),
+        Some(String::from("secrets-id")),
+        secrets_obj_schema,
+    );
+    let mut config = SectionConfig::new(&JOB_ID_SCHEMA);
+    config.register_plugin(plugin);
+    config.register_plugin(secrets_plugin);
+
+    config
+}
+
+pub const S3_CFG_FILENAME: &str = "/etc/proxmox-backup/s3.cfg";
+pub const S3_SECRETS_CFG_FILENAME: &str = "/etc/proxmox-backup/s3-secrets.cfg";
+pub const S3_CFG_LOCKFILE: &str = "/etc/proxmox-backup/.s3.lck";
+
+/// Get exclusive lock
+pub fn lock_config() -> Result<BackupLockGuard, Error> {
+    open_backup_lockfile(S3_CFG_LOCKFILE, None, true)
+}
+
+pub fn config() -> Result<(SectionConfigData, [u8; 32]), Error> {
+    parse_config(S3_CFG_FILENAME)
+}
+
+pub fn secrets_config() -> Result<(SectionConfigData, [u8; 32]), Error> {
+    parse_config(S3_SECRETS_CFG_FILENAME)
+}
+
+pub fn save_config(config: &SectionConfigData, secrets: &SectionConfigData) -> Result<(), Error> {
+    let raw = CONFIG.write(S3_CFG_FILENAME, config)?;
+    replace_backup_config(S3_CFG_FILENAME, raw.as_bytes())?;
+
+    let secrets_raw = CONFIG.write(S3_SECRETS_CFG_FILENAME, secrets)?;
+    // Secrets are stored with `backup` permissions to allow reading from
+    // not protected api endpoints as well.
+    replace_backup_config(S3_SECRETS_CFG_FILENAME, secrets_raw.as_bytes())?;
+
+    Ok(())
+}
+
+fn parse_config(path: &str) -> Result<(SectionConfigData, [u8; 32]), Error> {
+    let content = proxmox_sys::fs::file_read_optional_string(path)?;
+    let content = content.unwrap_or_default();
+    let digest = openssl::sha::sha256(content.as_bytes());
+    let data = CONFIG.parse(path, &content)?;
+    Ok((data, digest))
+}
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:07 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:07 +0200
Subject: [pbs-devel] [RFC proxmox-backup 06/39] s3 client: implement AWS
 signature v4 request authentication
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-7-c.ebner@proxmox.com>

The S3 API authenticates client requests by checking the
authentication signature provided by the requests `Authorization`
header. The latest AWS signature v4 signature is required for the
newest AWS regions [0] and most widely adapted [1-4], so rely soly on
that, not implementing older versions.

Adds helper methods to sign client requests, this includes encoding
and normalization of the headers, digest calculation of the request body
(if any) and signature generation.

[0] https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html
[1] https://docs.ceph.com/en/reef/radosgw/s3/authentication/#aws-signature-v4
[2] https://cloud.google.com/storage/docs/interoperability
[3] https://docs.wasabi.com/v1/docs/how-do-i-use-aws-signature-version-4-with-wasabi
[4] https://min.io/product/s3-compatibility

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-s3-client/Cargo.toml         |   2 +
 pbs-s3-client/src/aws_sign_v4.rs | 140 +++++++++++++++++++++++++++++++
 pbs-s3-client/src/lib.rs         |   1 +
 3 files changed, 143 insertions(+)
 create mode 100644 pbs-s3-client/src/aws_sign_v4.rs

diff --git a/pbs-s3-client/Cargo.toml b/pbs-s3-client/Cargo.toml
index 1999c3323..11189ea50 100644
--- a/pbs-s3-client/Cargo.toml
+++ b/pbs-s3-client/Cargo.toml
@@ -12,5 +12,7 @@ hex = { workspace = true, features = [ "serde" ] }
 hyper.workspace = true
 openssl.workspace = true
 tracing.workspace = true
+url.workspace = true
 
 proxmox-http.workspace = true
+proxmox-time.workspace = true
diff --git a/pbs-s3-client/src/aws_sign_v4.rs b/pbs-s3-client/src/aws_sign_v4.rs
new file mode 100644
index 000000000..8a538e868
--- /dev/null
+++ b/pbs-s3-client/src/aws_sign_v4.rs
@@ -0,0 +1,140 @@
+//! Helpers for request authentication using AWS signature version 4
+
+use anyhow::Error;
+use hyper::Body;
+use hyper::Request;
+use openssl::hash::MessageDigest;
+use openssl::pkey::{PKey, Private};
+use openssl::sha::sha256;
+use openssl::sign::Signer;
+use url::Url;
+
+use super::client::S3ClientOptions;
+
+pub(crate) const AWS_SIGN_V4_DATETIME_FORMAT: &str = "%Y%m%dT%H%M%SZ";
+
+const AWS_SIGN_V4_DATE_FORMAT: &str = "%Y%m%d";
+const AWS_SIGN_V4_SERVICE_S3: &str = "s3";
+const AWS_SIGN_V4_REQUEST_POSTFIX: &str = "aws4_request";
+
+/// Generate signature for S3 request authentication using AWS signature version 4.
+/// See: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html
+pub(crate) fn aws_sign_v4_signature(
+    request: &Request<Body>,
+    options: &S3ClientOptions,
+    epoch: i64,
+    payload_digest: &str,
+) -> Result<String, Error> {
+    // Include all headers in signature calculation since the reference docs note:
+    // "For the purpose of calculating an authorization signature, only the 'host' and any 'x-amz-*'
+    // headers are required. however, in order to prevent data tampering, you should consider
+    // including all the headers in the signature calculation."
+    // See https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
+    let mut canonical_headers = Vec::new();
+    let mut signed_headers = Vec::new();
+    for (key, value) in request.headers() {
+        canonical_headers.push(format!(
+            "{}:{}",
+            // Header name has to be lower case, key.as_str() does guarantee that, see
+            // https://docs.rs/http/0.2.0/http/header/struct.HeaderName.html
+            key.as_str(),
+            // No need to trim since `HeaderValue` only allows visible UTF8 chars, see
+            // https://docs.rs/http/0.2.0/http/header/struct.HeaderValue.html
+            value.to_str()?,
+        ));
+        signed_headers.push(key.as_str());
+    }
+    canonical_headers.sort();
+    signed_headers.sort();
+    let signed_headers_string = signed_headers.join(";");
+
+    let mut canonical_queries = Url::parse(&request.uri().to_string())?
+        .query_pairs()
+        .map(|(key, value)| {
+            format!(
+                "{}={}",
+                aws_sign_v4_uri_encode(&key, false),
+                aws_sign_v4_uri_encode(&value, false),
+            )
+        })
+        .collect::<Vec<String>>();
+    canonical_queries.sort();
+
+    let canonical_request = format!(
+        "{}\n{}\n{}\n{}\n\n{}\n{}",
+        request.method().as_str(),
+        request.uri().path(),
+        canonical_queries.join("&"),
+        canonical_headers.join("\n"),
+        signed_headers_string,
+        payload_digest,
+    );
+
+    let date = proxmox_time::strftime_utc(AWS_SIGN_V4_DATE_FORMAT, epoch)?;
+    let datetime = proxmox_time::strftime_utc(AWS_SIGN_V4_DATETIME_FORMAT, epoch)?;
+
+    let credential_scope = format!(
+        "{date}/{}/{AWS_SIGN_V4_SERVICE_S3}/{AWS_SIGN_V4_REQUEST_POSTFIX}",
+        options.region,
+    );
+    let canonical_request_hash = hex::encode(sha256(canonical_request.as_bytes()));
+    let string_to_sign =
+        format!("AWS4-HMAC-SHA256\n{datetime}\n{credential_scope}\n{canonical_request_hash}");
+
+    let date_sign_key = PKey::hmac(format!("AWS4{}", options.secret_key).as_bytes())?;
+    let date_tag = hmac_sha256(&date_sign_key, date.as_bytes())?;
+
+    let region_sign_key = PKey::hmac(&date_tag)?;
+    let region_tag = hmac_sha256(&region_sign_key, options.region.as_bytes())?;
+
+    let service_sign_key = PKey::hmac(&region_tag)?;
+    let service_tag = hmac_sha256(&service_sign_key, AWS_SIGN_V4_SERVICE_S3.as_bytes())?;
+
+    let signing_key = PKey::hmac(&service_tag)?;
+    let signing_tag = hmac_sha256(&signing_key, AWS_SIGN_V4_REQUEST_POSTFIX.as_bytes())?;
+
+    let signature_key = PKey::hmac(&signing_tag)?;
+    let signature = hmac_sha256(&signature_key, string_to_sign.as_bytes())?;
+    let signature = hex::encode(&signature);
+
+    Ok(format!(
+        "AWS4-HMAC-SHA256 Credential={}/{credential_scope},SignedHeaders={signed_headers_string},Signature={signature}",
+        options.access_key,
+    ))
+}
+// Custom `uri_encode` implementation as recommended by AWS docs, since possible implementation
+// incompatibility with uri encoding libraries.
+// See: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
+pub(crate) fn aws_sign_v4_uri_encode(input: &str, is_object_key_name: bool) -> String {
+    // Assume up to  2 bytes per char max in output
+    let mut accumulator = String::with_capacity(2 * input.len());
+
+    input.chars().for_each(|char| {
+        match char {
+            // Unreserved characters, do not uri encode these bytes
+            'A'..='Z' | 'a'..='z' | '0'..='9' | '-' | '.' | '_' | '~' => accumulator.push(char),
+            // Space character is reserved, must be encoded as '%20', not '+'
+            ' ' => accumulator.push_str("%20"),
+            // Encode the forward slash character, '/', everywhere except in the object key name
+            '/' if !is_object_key_name => accumulator.push_str("%2F"),
+            '/' if is_object_key_name => accumulator.push(char),
+            // URI encoded byte is formed by a '%' and the two-digit hexadecimal value of the byte
+            // Letters in the hexadecimal value must be uppercase
+            _ => {
+                for byte in char.to_string().as_bytes() {
+                    accumulator.push_str(&format!("%{byte:02X}"));
+                }
+            }
+        }
+    });
+
+    accumulator
+}
+
+// Helper for hmac sha256 calculation
+fn hmac_sha256(key: &PKey<Private>, data: &[u8]) -> Result<Vec<u8>, Error> {
+    let mut signer = Signer::new(MessageDigest::sha256(), key)?;
+    signer.update(data)?;
+    let hmac = signer.sign_to_vec()?;
+    Ok(hmac)
+}
diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
index 533ceab8e..5a60b92ec 100644
--- a/pbs-s3-client/src/lib.rs
+++ b/pbs-s3-client/src/lib.rs
@@ -1,2 +1,3 @@
+mod aws_sign_v4;
 mod client;
 pub use client::{S3Client, S3ClientOptions};
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:17 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:17 +0200
Subject: [pbs-devel] [RFC proxmox-backup 16/39] api: backup: conditionally
 upload chunks to S3 object store backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-17-c.ebner@proxmox.com>

Upload fixed and dynamic sized chunks to either the filesystem or
the S3 object store, depending on the configured backend.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/upload_chunk.rs | 45 ++++++++++++++++++++++-----------
 1 file changed, 30 insertions(+), 15 deletions(-)

diff --git a/src/api2/backup/upload_chunk.rs b/src/api2/backup/upload_chunk.rs
index 20259660a..59f9ca558 100644
--- a/src/api2/backup/upload_chunk.rs
+++ b/src/api2/backup/upload_chunk.rs
@@ -15,7 +15,8 @@ use proxmox_sortable_macro::sortable;
 
 use pbs_api_types::{BACKUP_ARCHIVE_NAME_SCHEMA, CHUNK_DIGEST_SCHEMA};
 use pbs_datastore::file_formats::{DataBlobHeader, EncryptedDataBlobHeader};
-use pbs_datastore::{DataBlob, DataStore};
+use pbs_datastore::{DataBlob, DataStore, DatastoreBackend};
+use pbs_s3_client::PutObjectResponse;
 use pbs_tools::json::{required_integer_param, required_string_param};
 
 use super::environment::*;
@@ -153,16 +154,10 @@ fn upload_fixed_chunk(
 ) -> ApiResponseFuture {
     async move {
         let wid = required_integer_param(&param, "wid")? as usize;
-        let size = required_integer_param(&param, "size")? as u32;
-        let encoded_size = required_integer_param(&param, "encoded-size")? as u32;
-
-        let digest_str = required_string_param(&param, "digest")?;
-        let digest = <[u8; 32]>::from_hex(digest_str)?;
-
         let env: &BackupEnvironment = rpcenv.as_ref();
 
         let (digest, size, compressed_size, is_duplicate) =
-            UploadChunk::new(req_body, env.datastore.clone(), digest, size, encoded_size).await?;
+            upload_to_backend(req_body, param, env.datastore.clone(), &env.backend).await?;
 
         env.register_fixed_chunk(wid, digest, size, compressed_size, is_duplicate)?;
         let digest_str = hex::encode(digest);
@@ -222,16 +217,10 @@ fn upload_dynamic_chunk(
 ) -> ApiResponseFuture {
     async move {
         let wid = required_integer_param(&param, "wid")? as usize;
-        let size = required_integer_param(&param, "size")? as u32;
-        let encoded_size = required_integer_param(&param, "encoded-size")? as u32;
-
-        let digest_str = required_string_param(&param, "digest")?;
-        let digest = <[u8; 32]>::from_hex(digest_str)?;
-
         let env: &BackupEnvironment = rpcenv.as_ref();
 
         let (digest, size, compressed_size, is_duplicate) =
-            UploadChunk::new(req_body, env.datastore.clone(), digest, size, encoded_size).await?;
+            upload_to_backend(req_body, param, env.datastore.clone(), &env.backend).await?;
 
         env.register_dynamic_chunk(wid, digest, size, compressed_size, is_duplicate)?;
         let digest_str = hex::encode(digest);
@@ -243,6 +232,32 @@ fn upload_dynamic_chunk(
     .boxed()
 }
 
+async fn upload_to_backend(
+    req_body: Body,
+    param: Value,
+    datastore: Arc<DataStore>,
+    backend: &DatastoreBackend,
+) -> Result<([u8; 32], u32, u32, bool), Error> {
+    let size = required_integer_param(&param, "size")? as u32;
+    let encoded_size = required_integer_param(&param, "encoded-size")? as u32;
+    let digest_str = required_string_param(&param, "digest")?;
+    let digest = <[u8; 32]>::from_hex(digest_str)?;
+
+    match backend {
+        DatastoreBackend::Filesystem => {
+            UploadChunk::new(req_body, datastore, digest, size, encoded_size).await
+        }
+        DatastoreBackend::S3(s3_client) => {
+            let is_duplicate = match s3_client.put_object(digest.into(), req_body).await? {
+                PutObjectResponse::PreconditionFailed => true,
+                PutObjectResponse::NeedsRetry => bail!("concurrent operation, reupload required"),
+                PutObjectResponse::Success(_content) => false,
+            };
+            Ok((digest, size, encoded_size, is_duplicate))
+        }
+    }
+}
+
 pub const API_METHOD_UPLOAD_SPEEDTEST: ApiMethod = ApiMethod::new(
     &ApiHandler::AsyncHttp(&upload_speedtest),
     &ObjectSchema::new("Test upload speed.", &[]),
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:19 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:19 +0200
Subject: [pbs-devel] [RFC proxmox-backup 18/39] api: backup: conditionally
 upload indices to S3 object store backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-19-c.ebner@proxmox.com>

If the datastore is backed by an S3 compatible object store, upload
the dynamic or fixed index files to the object store after closing
them. The local index files are kept in the local caching datastore
to allow for fast and efficient content lookups, avoiding expensive
(as in monetary cost and IO latency) requests.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/environment.rs | 65 ++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 393a8351d..72e369bcf 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -2,6 +2,7 @@ use anyhow::{bail, format_err, Context, Error};
 use pbs_config::BackupLockGuard;
 
 use std::collections::HashMap;
+use std::io::Read;
 use std::sync::{Arc, Mutex};
 use tracing::info;
 
@@ -479,6 +480,38 @@ impl BackupEnvironment {
             );
         }
 
+        // For S3 backends, upload the index file to the object store after closing
+        if let DatastoreBackend::S3(s3_client) = &self.backend {
+            let mut object_key = self.backup_dir.relative_path();
+            object_key.push(&data.name);
+            let object_key = object_key
+                .as_os_str()
+                .to_str()
+                .ok_or_else(|| format_err!("invalid file name"))?;
+
+            let mut full_path = self.datastore.base_path();
+            full_path.push(object_key);
+            let mut file = std::fs::File::open(&full_path)?;
+            let mut buffer = Vec::new();
+            file.read_to_end(&mut buffer)?;
+            let data = Body::from(buffer);
+            match proxmox_async::runtime::block_on(s3_client.put_object(object_key.into(), data))? {
+                PutObjectResponse::PreconditionFailed => {
+                    self.log(format!(
+                        "Upload of dynamic index failed, object key {object_key} already present"
+                    ));
+                    bail!("Upload of dynamic index failed");
+                }
+                PutObjectResponse::NeedsRetry => {
+                    self.log("Upload of dynamic index failed, reupload required");
+                    bail!("concurrent operation, reupload required");
+                }
+                PutObjectResponse::Success(_content) => {
+                    self.log(format!("Uploaded index file to object store: {object_key}"))
+                }
+            }
+        }
+
         self.log_upload_stat(
             &data.name,
             &csum,
@@ -553,6 +586,38 @@ impl BackupEnvironment {
             );
         }
 
+        // For S3 backends, upload the index file to the object store after closing
+        if let DatastoreBackend::S3(s3_client) = &self.backend {
+            let mut object_key = self.backup_dir.relative_path();
+            object_key.push(&data.name);
+            let object_key = object_key
+                .as_os_str()
+                .to_str()
+                .ok_or_else(|| format_err!("invalid file name"))?;
+
+            let mut full_path = self.datastore.base_path();
+            full_path.push(object_key);
+            let mut file = std::fs::File::open(&full_path)?;
+            let mut buffer = Vec::new();
+            file.read_to_end(&mut buffer)?;
+            let data = Body::from(buffer);
+            match proxmox_async::runtime::block_on(s3_client.put_object(object_key.into(), data))? {
+                PutObjectResponse::PreconditionFailed => {
+                    self.log(format!(
+                        "Upload of fixed index failed, object {object_key} already present."
+                    ));
+                    bail!("upload of fixed index failed");
+                }
+                PutObjectResponse::NeedsRetry => {
+                    self.log("Upload of fixed index failed, reupload required.");
+                    bail!("concurrent operation, reupload required");
+                }
+                PutObjectResponse::Success(_content) => {
+                    self.log(format!("Uploaded index file to object store: {object_key}"))
+                }
+            }
+        }
+
         self.log_upload_stat(
             &data.name,
             &expected_csum,
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:30 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:30 +0200
Subject: [pbs-devel] [RFC proxmox-backup 29/39] ui: expose the S3 client
 view in the navigation tree
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-30-c.ebner@proxmox.com>

Add a `S3 Clients` item to the navigation tree to allow accessing the
S3 client configuration view and edit windows.

Adds the required source files to the Makefile.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/Makefile          | 2 ++
 www/NavigationTree.js | 6 ++++++
 2 files changed, 8 insertions(+)

diff --git a/www/Makefile b/www/Makefile
index 44c5fa133..ca4683941 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -61,6 +61,7 @@ JSSRC=							\
 	config/RemoteView.js				\
 	config/TrafficControlView.js			\
 	config/ACLView.js				\
+	config/S3BucketView.js				\
 	config/SyncView.js				\
 	config/VerifyView.js				\
 	config/PruneView.js				\
@@ -85,6 +86,7 @@ JSSRC=							\
 	window/PruneJobEdit.js				\
 	window/GCJobEdit.js				\
 	window/UserEdit.js				\
+	window/S3BucketEdit.js				\
 	window/Settings.js				\
 	window/TokenEdit.js				\
 	window/VerifyJobEdit.js				\
diff --git a/www/NavigationTree.js b/www/NavigationTree.js
index f10b0cd63..c79797d79 100644
--- a/www/NavigationTree.js
+++ b/www/NavigationTree.js
@@ -80,6 +80,12 @@ Ext.define('PBS.store.NavigationStore', {
 			path: 'pbsSubscription',
 			leaf: true,
 		    },
+		    {
+			text: gettext('S3 Buckets'),
+			iconCls: 'fa fa-trash',
+			path: 'pbsS3BucketView',
+			leaf: true,
+		    },
 		],
 	    },
 	    {
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:03 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:03 +0200
Subject: [pbs-devel] [RFC proxmox 2/2] pbs-api-types: extend datastore
 config by backend config enum
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-3-c.ebner@proxmox.com>

Allows to configure a backend config variant for a datastore on
creation. The current default `Filesystem` backend variant is
introduced to be compatible with existing storages. A new S3 backend
variant allows to create datastores backed by an S3 compatible object
store instead.

For S3 backends, the id of the corresponding S3 client configuration
is storered. A valid datastore backend configuration for S3 therefore
contains:
```
    ...
    backend s3=<S3_CONFIG_ID>
    ...
```

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-api-types/src/datastore.rs | 58 +++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 5bd953ac..2b983cb2 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -336,7 +336,11 @@ pub const DATASTORE_TUNING_STRING_SCHEMA: Schema = StringSchema::new("Datastore
             optional: true,
             format: &proxmox_schema::api_types::UUID_FORMAT,
             type: String,
-        }
+        },
+        backend: {
+            schema: DATASTORE_BACKEND_SCHEMA,
+            optional: true,
+        },
     }
 )]
 #[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
@@ -389,8 +393,59 @@ pub struct DataStoreConfig {
     #[updater(skip)]
     #[serde(skip_serializing_if = "Option::is_none")]
     pub backing_device: Option<String>,
+
+    /// Backend to be used by datastore
+    #[updater(skip)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub backend: Option<String>,
+}
+
+pub const DATASTORE_BACKEND_SCHEMA: Schema = StringSchema::new("Backend config to be used for datastore.")
+    .format(&ApiStringFormat::VerifyFn(verify_datastore_backend))
+    .type_text("<filesystem|s3=S3_CONFIG_ID>")
+    .schema();
+
+fn verify_datastore_backend(input: &str) -> Result<(), Error> {
+    DatastoreBackendConfig::from_str(input).map(|_| ())
+}
+
+#[derive(Clone, Default)]
+/// Available backend configurations for datastores.
+pub enum DatastoreBackendConfig {
+    #[default]
+    Filesystem,
+    S3(String),
 }
 
+impl std::str::FromStr for DatastoreBackendConfig {
+    type Err = Error;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        if s == "filesystem" {
+            return Ok(Self::Filesystem);
+        }
+        match s.split_once('=') {
+            Some(("s3", value)) => {
+                let s3_config_id = value.parse()?;
+                Ok(Self::S3(s3_config_id))
+            }
+            _ => bail!("invalid datastore backend configuration"),
+        }
+    }
+}
+
+impl std::fmt::Display for DatastoreBackendConfig {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::Filesystem => write!(f, "filesystem"),
+            Self::S3(s3_config_id) => write!(f, "s3:{s3_config_id}"),
+        }
+    }
+}
+
+proxmox_serde::forward_serialize_to_display!(DatastoreBackendConfig);
+proxmox_serde::forward_deserialize_to_from_str!(DatastoreBackendConfig);
+
 #[api]
 #[derive(Serialize, Deserialize, Updater, Clone, PartialEq, Default)]
 #[serde(rename_all = "kebab-case")]
@@ -424,6 +479,7 @@ impl DataStoreConfig {
             tuning: None,
             maintenance_mode: None,
             backing_device: None,
+            backend: None,
         }
     }
 
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:23 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:23 +0200
Subject: [pbs-devel] [RFC proxmox-backup 22/39] verify worker: add datastore
 backed to verify worker
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-23-c.ebner@proxmox.com>

In order to fetch chunks from an S3 compatible object store,
instantiate and store the s3 client in the verify worker by storing
the datastore's backend. This allows to reuse the same instance for
the whole verification task.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs    |  2 +-
 src/api2/backup/environment.rs |  2 +-
 src/backup/verify.rs           | 14 ++++++++++----
 src/server/verify_job.rs       |  2 +-
 4 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 7dc881ade..7b7f79b22 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -893,7 +893,7 @@ pub fn verify(
         auth_id.to_string(),
         to_stdout,
         move |worker| {
-            let verify_worker = VerifyWorker::new(worker.clone(), datastore);
+            let verify_worker = VerifyWorker::new(worker.clone(), datastore)?;
             let failed_dirs = if let Some(backup_dir) = backup_dir {
                 let mut res = Vec::new();
                 if !verify_worker.verify_backup_dir(
diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 685b78e89..384e8a73f 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -796,7 +796,7 @@ impl BackupEnvironment {
             move |worker| {
                 worker.log_message("Automatically verifying newly added snapshot");
 
-                let verify_worker = VerifyWorker::new(worker.clone(), datastore);
+                let verify_worker = VerifyWorker::new(worker.clone(), datastore)?;
                 if !verify_worker.verify_backup_dir_with_lock(
                     &backup_dir,
                     worker.upid().clone(),
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 0b954ae23..a01ddcca3 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -17,7 +17,7 @@ use pbs_api_types::{
 use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo};
 use pbs_datastore::index::IndexFile;
 use pbs_datastore::manifest::{BackupManifest, FileInfo};
-use pbs_datastore::{DataBlob, DataStore, StoreProgress};
+use pbs_datastore::{DataBlob, DataStore, DatastoreBackend, StoreProgress};
 
 use crate::tools::parallel_handler::ParallelHandler;
 
@@ -30,19 +30,25 @@ pub struct VerifyWorker {
     datastore: Arc<DataStore>,
     verified_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
     corrupt_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
+    backend: DatastoreBackend,
 }
 
 impl VerifyWorker {
     /// Creates a new VerifyWorker for a given task worker and datastore.
-    pub fn new(worker: Arc<dyn WorkerTaskContext>, datastore: Arc<DataStore>) -> Self {
-        Self {
+    pub fn new(
+        worker: Arc<dyn WorkerTaskContext>,
+        datastore: Arc<DataStore>,
+    ) -> Result<Self, Error> {
+        let backend = datastore.backend()?;
+        Ok(Self {
             worker,
             datastore,
             // start with 16k chunks == up to 64G data
             verified_chunks: Arc::new(Mutex::new(HashSet::with_capacity(16 * 1024))),
             // start with 64 chunks since we assume there are few corrupt ones
             corrupt_chunks: Arc::new(Mutex::new(HashSet::with_capacity(64))),
-        }
+            backend,
+        })
     }
 
     fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
diff --git a/src/server/verify_job.rs b/src/server/verify_job.rs
index 95a7b2a9b..c8792174b 100644
--- a/src/server/verify_job.rs
+++ b/src/server/verify_job.rs
@@ -41,7 +41,7 @@ pub fn do_verification_job(
                 None => Default::default(),
             };
 
-            let verify_worker = VerifyWorker::new(worker.clone(), datastore);
+            let verify_worker = VerifyWorker::new(worker.clone(), datastore)?;
             let result = verify_worker.verify_all_backups(
                 worker.upid(),
                 ns,
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:22 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:22 +0200
Subject: [pbs-devel] [RFC proxmox-backup 21/39] datastore: local chunk
 reader: read chunks based on backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-22-c.ebner@proxmox.com>

Get and store the datastore's backend on local chunk reader
instantiantion and fetch chunks based on the variant from either the
filesystem or the s3 object store.

By storing the backend variant, the s3 client is instantiated only
once and reused until the local chunk reader instance is dropped.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/Cargo.toml                |  2 ++
 pbs-datastore/src/local_chunk_reader.rs | 37 +++++++++++++++++++++----
 2 files changed, 33 insertions(+), 6 deletions(-)

diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index 3ee06c9bb..323f5e270 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -13,6 +13,7 @@ crc32fast.workspace = true
 endian_trait.workspace = true
 futures.workspace = true
 hex = { workspace = true, features = [ "serde" ] }
+hyper.workspace = true
 libc.workspace = true
 log.workspace = true
 nix.workspace = true
@@ -28,6 +29,7 @@ zstd-safe.workspace = true
 pathpatterns.workspace = true
 pxar.workspace = true
 
+proxmox-async.workspace = true
 proxmox-borrow.workspace = true
 proxmox-human-byte.workspace = true
 proxmox-io.workspace = true
diff --git a/pbs-datastore/src/local_chunk_reader.rs b/pbs-datastore/src/local_chunk_reader.rs
index 05a70c068..a363059a1 100644
--- a/pbs-datastore/src/local_chunk_reader.rs
+++ b/pbs-datastore/src/local_chunk_reader.rs
@@ -3,17 +3,21 @@ use std::pin::Pin;
 use std::sync::Arc;
 
 use anyhow::{bail, Error};
+use hyper::body::HttpBody;
 
 use pbs_api_types::CryptMode;
+use pbs_s3_client::S3Client;
 use pbs_tools::crypt_config::CryptConfig;
 
 use crate::data_blob::DataBlob;
+use crate::datastore::DatastoreBackend;
 use crate::read_chunk::{AsyncReadChunk, ReadChunk};
 use crate::DataStore;
 
 #[derive(Clone)]
 pub struct LocalChunkReader {
     store: Arc<DataStore>,
+    backend: DatastoreBackend,
     crypt_config: Option<Arc<CryptConfig>>,
     crypt_mode: CryptMode,
 }
@@ -24,8 +28,11 @@ impl LocalChunkReader {
         crypt_config: Option<Arc<CryptConfig>>,
         crypt_mode: CryptMode,
     ) -> Self {
+        // TODO: Error handling!
+        let backend = store.backend().unwrap();
         Self {
             store,
+            backend,
             crypt_config,
             crypt_mode,
         }
@@ -47,10 +54,25 @@ impl LocalChunkReader {
     }
 }
 
+async fn fetch(s3_client: Arc<S3Client>, digest: &[u8; 32]) -> Result<DataBlob, Error> {
+    if let Some(response) = s3_client.get_object(digest.into()).await? {
+        let bytes = response.content.collect().await?.to_bytes();
+        DataBlob::from_raw(bytes.to_vec())
+    } else {
+        bail!("no object with digest {}", hex::encode(digest));
+    }
+}
+
 impl ReadChunk for LocalChunkReader {
     fn read_raw_chunk(&self, digest: &[u8; 32]) -> Result<DataBlob, Error> {
-        let chunk = self.store.load_chunk(digest)?;
+        let chunk = match &self.backend {
+            DatastoreBackend::Filesystem => self.store.load_chunk(digest)?,
+            DatastoreBackend::S3(s3_client) => {
+                proxmox_async::runtime::block_on(fetch(s3_client.clone(), digest))?
+            }
+        };
         self.ensure_crypt_mode(chunk.crypt_mode()?)?;
+
         Ok(chunk)
     }
 
@@ -69,11 +91,14 @@ impl AsyncReadChunk for LocalChunkReader {
         digest: &'a [u8; 32],
     ) -> Pin<Box<dyn Future<Output = Result<DataBlob, Error>> + Send + 'a>> {
         Box::pin(async move {
-            let (path, _) = self.store.chunk_path(digest);
-
-            let raw_data = tokio::fs::read(&path).await?;
-
-            let chunk = DataBlob::load_from_reader(&mut &raw_data[..])?;
+            let chunk = match &self.backend {
+                DatastoreBackend::Filesystem => {
+                    let (path, _) = self.store.chunk_path(digest);
+                    let raw_data = tokio::fs::read(&path).await?;
+                    DataBlob::load_from_reader(&mut &raw_data[..])?
+                }
+                DatastoreBackend::S3(s3_client) => fetch(s3_client.clone(), digest).await?,
+            };
             self.ensure_crypt_mode(chunk.crypt_mode()?)?;
 
             Ok(chunk)
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:13 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:13 +0200
Subject: [pbs-devel] [RFC proxmox-backup 12/39] api: config: implement
 endpoints to manipulate and list s3 configs
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-13-c.ebner@proxmox.com>

Allows to create, list, modify and delete configurations for s3
clients via the api.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/config/mod.rs |   2 +
 src/api2/config/s3.rs  | 349 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 351 insertions(+)
 create mode 100644 src/api2/config/s3.rs

diff --git a/src/api2/config/mod.rs b/src/api2/config/mod.rs
index 15dc5db92..1cd9ead76 100644
--- a/src/api2/config/mod.rs
+++ b/src/api2/config/mod.rs
@@ -14,6 +14,7 @@ pub mod metrics;
 pub mod notifications;
 pub mod prune;
 pub mod remote;
+pub mod s3;
 pub mod sync;
 pub mod tape_backup_job;
 pub mod tape_encryption_keys;
@@ -32,6 +33,7 @@ const SUBDIRS: SubdirMap = &sorted!([
     ("notifications", &notifications::ROUTER),
     ("prune", &prune::ROUTER),
     ("remote", &remote::ROUTER),
+    ("s3", &s3::ROUTER),
     ("sync", &sync::ROUTER),
     ("tape-backup-job", &tape_backup_job::ROUTER),
     ("tape-encryption-keys", &tape_encryption_keys::ROUTER),
diff --git a/src/api2/config/s3.rs b/src/api2/config/s3.rs
new file mode 100644
index 000000000..11cf16411
--- /dev/null
+++ b/src/api2/config/s3.rs
@@ -0,0 +1,349 @@
+use ::serde::{Deserialize, Serialize};
+use anyhow::Error;
+use hex::FromHex;
+use serde_json::Value;
+
+use proxmox_router::{http_bail, Permission, Router, RpcEnvironment};
+use proxmox_schema::{api, param_bail};
+
+use pbs_api_types::{
+    Authid, S3ClientConfig, S3ClientConfigUpdater, S3ClientSecretsConfig,
+    S3ClientSecretsConfigUpdater, JOB_ID_SCHEMA, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_MODIFY,
+    PROXMOX_CONFIG_DIGEST_SCHEMA,
+};
+use pbs_config::s3;
+
+use pbs_config::CachedUserInfo;
+
+#[api(
+    input: {
+        properties: {},
+    },
+    returns: {
+        description: "List configured s3 clients.",
+        type: Array,
+        items: { type: S3ClientConfig },
+    },
+    access: {
+        permission: &Permission::Anybody,
+        description: "Requires Datastore.Audit or Datastore.Modify on datastore.",
+    },
+)]
+/// List all s3 client configurations.
+pub fn list_s3_client_config(
+    _param: Value,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<S3ClientConfig>, Error> {
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let user_info = CachedUserInfo::new()?;
+    let required_privs = PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_MODIFY;
+
+    let (config, digest) = s3::config()?;
+    let list = config.convert_to_typed_array("s3client")?;
+    let list = list
+        .into_iter()
+        .filter(|s3_client_config: &S3ClientConfig| {
+            let privs = user_info.lookup_privs(&auth_id, &s3_client_config.acl_path());
+            privs & required_privs != 00
+        })
+        .collect();
+
+    let (_secrets, secrets_digest) = s3::secrets_config()?;
+    let digest = digest_with_secrets(&digest, &secrets_digest);
+    rpcenv["digest"] = hex::encode(digest).into();
+
+    Ok(list)
+}
+
+#[api(
+    protected: true,
+    input: {
+        properties: {
+            config: {
+                type: S3ClientConfig,
+                flatten: true,
+            },
+            secrets: {
+                type: S3ClientSecretsConfig,
+                flatten: true,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Anybody,
+        description: "Requires Datastore.Modify on datastore.",
+    },
+)]
+/// Create a new s3 client configuration.
+pub fn create_s3_client_config(
+    config: S3ClientConfig,
+    secrets: S3ClientSecretsConfig,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    // Asssure both, config and secrets are referenced by the same `id`
+    if config.id != secrets.secrets_id {
+        param_bail!(
+            "id",
+            "config and secrets must use the same id ({} != {})",
+            config.id,
+            secrets.secrets_id
+        );
+    }
+
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let user_info = CachedUserInfo::new()?;
+    user_info.check_privs(&auth_id, &config.acl_path(), PRIV_DATASTORE_MODIFY, false)?;
+
+    let _lock = s3::lock_config()?;
+    let (mut section_config, _digest) = s3::config()?;
+    if section_config.sections.contains_key(&config.id) {
+        param_bail!("id", "s3 client config '{}' already exists.", config.id);
+    }
+
+    let (mut section_secrets, _secrets_digest) = s3::secrets_config()?;
+    if section_secrets.sections.contains_key(&config.id) {
+        param_bail!("id", "s3 secrets config '{}' already exists.", config.id);
+    }
+
+    section_config.set_data(&config.id, "s3client", &config)?;
+    section_secrets.set_data(&config.id, "s3secrets", &secrets)?;
+    s3::save_config(&section_config, &section_secrets)?;
+
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            id: {
+                schema: JOB_ID_SCHEMA,
+            },
+        },
+    },
+    returns: { type: S3ClientConfig },
+    access: {
+        permission: &Permission::Anybody,
+        description: "Requires Datastore.Audit or Datastore.Modify on datastore.",
+    },
+)]
+/// Read an s3 client configuration.
+pub fn read_s3_client_config(
+    id: String,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<S3ClientConfig, Error> {
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let user_info = CachedUserInfo::new()?;
+
+    let (config, digest) = s3::config()?;
+    let s3_client_config: S3ClientConfig = config.lookup("s3client", &id)?;
+
+    let required_privs = PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_MODIFY;
+    user_info.check_privs(&auth_id, &s3_client_config.acl_path(), required_privs, true)?;
+
+    let (_secrets, secrets_digest) = s3::secrets_config()?;
+    let digest = digest_with_secrets(&digest, &secrets_digest);
+    rpcenv["digest"] = hex::encode(digest).into();
+
+    Ok(s3_client_config)
+}
+
+#[api()]
+#[derive(Serialize, Deserialize)]
+#[serde(rename_all = "kebab-case")]
+/// Deletable property name
+pub enum DeletableProperty {
+    /// Delete the port property.
+    Port,
+    /// Delete the region property.
+    Region,
+    /// Delete the fingerprint property.
+    Fingerprint,
+}
+
+#[api(
+    protected: true,
+    input: {
+        properties: {
+            id: {
+                schema: JOB_ID_SCHEMA,
+            },
+            update: {
+                type: S3ClientConfigUpdater,
+                flatten: true,
+            },
+            "update-secrets": {
+                type: S3ClientSecretsConfigUpdater,
+                flatten: true,
+            },
+            delete: {
+                description: "List of properties to delete.",
+                type: Array,
+                optional: true,
+                items: {
+                    type: DeletableProperty,
+                }
+            },
+            digest: {
+                optional: true,
+                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Anybody,
+        description: "Requires Datastore.Verify on job's datastore.",
+    },
+)]
+/// Update an s3 client configuration.
+#[allow(clippy::too_many_arguments)]
+pub fn update_s3_client_config(
+    id: String,
+    update: S3ClientConfigUpdater,
+    update_secrets: S3ClientSecretsConfigUpdater,
+    delete: Option<Vec<DeletableProperty>>,
+    digest: Option<String>,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let user_info = CachedUserInfo::new()?;
+
+    let _lock = s3::lock_config()?;
+    let (mut config, expected_digest) = s3::config()?;
+    let (mut secrets, secrets_digest) = s3::secrets_config()?;
+    let expected_digest = digest_with_secrets(&expected_digest, &secrets_digest);
+
+    // Secrets are not included in digest concurrent changes therefore not detected.
+    if let Some(ref digest) = digest {
+        let digest = <[u8; 32]>::from_hex(digest)?;
+        crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
+    }
+
+    let mut data: S3ClientConfig = config.lookup("s3client", &id)?;
+    user_info.check_privs(&auth_id, &data.acl_path(), PRIV_DATASTORE_MODIFY, true)?;
+
+    if let Some(delete) = delete {
+        for delete_prop in delete {
+            match delete_prop {
+                DeletableProperty::Port => {
+                    data.port = None;
+                }
+                DeletableProperty::Region => {
+                    data.region = None;
+                }
+                DeletableProperty::Fingerprint => {
+                    data.fingerprint = None;
+                }
+            }
+        }
+    }
+
+    if let Some(host) = update.host {
+        data.host = host;
+    }
+    if let Some(bucket) = update.bucket {
+        data.bucket = bucket;
+    }
+    if let Some(port) = update.port {
+        data.port = Some(port);
+    }
+    if let Some(region) = update.region {
+        data.region = Some(region);
+    }
+    if let Some(access_key) = update.access_key {
+        data.access_key = access_key;
+    }
+    if let Some(fingerprint) = update.fingerprint {
+        data.fingerprint = Some(fingerprint);
+    }
+
+    let mut secrets_data: S3ClientSecretsConfig = secrets.lookup("s3secrets", &id)?;
+    if let Some(secret_key) = update_secrets.secret_key {
+        secrets_data.secret_key = secret_key;
+    }
+
+    config.set_data(&id, "s3client", &data)?;
+    secrets.set_data(&id, "s3secrets", &secrets_data)?;
+    s3::save_config(&config, &secrets)?;
+
+    Ok(())
+}
+
+#[api(
+    protected: true,
+    input: {
+        properties: {
+            id: {
+                schema: JOB_ID_SCHEMA,
+            },
+            digest: {
+                optional: true,
+                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Anybody,
+        description: "Requires Datastore.Modify on job's datastore.",
+    },
+)]
+/// Remove an s3 client configuration.
+pub fn delete_s3_client_config(
+    id: String,
+    digest: Option<String>,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
+    let user_info = CachedUserInfo::new()?;
+
+    let _lock = s3::lock_config()?;
+    let (mut config, expected_digest) = s3::config()?;
+    let s3_client_config: S3ClientConfig = config.lookup("s3client", &id)?;
+    user_info.check_privs(
+        &auth_id,
+        &s3_client_config.acl_path(),
+        PRIV_DATASTORE_MODIFY,
+        true,
+    )?;
+
+    let (mut secrets, secrets_digest) = s3::secrets_config()?;
+    let expected_digest = digest_with_secrets(&expected_digest, &secrets_digest);
+
+    if let Some(ref digest) = digest {
+        let digest = <[u8; 32]>::from_hex(digest)?;
+        crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
+    }
+
+    match (config.sections.remove(&id), secrets.sections.remove(&id)) {
+        (Some(_), Some(_)) => {}
+        (None, None) => http_bail!(
+            NOT_FOUND,
+            "s3 client config and secrets '{id}' do not exist."
+        ),
+        (Some(_), None) => http_bail!(
+            NOT_FOUND,
+            "removed s3 client config, but no secrets for '{id}' found."
+        ),
+        (None, Some(_)) => http_bail!(
+            NOT_FOUND,
+            "removed s3 client secrets, but no config for '{id}' found."
+        ),
+    }
+    s3::save_config(&config, &secrets)
+}
+
+// Calculate the digest based on the digest of config and secrets to detect changes for both
+fn digest_with_secrets(digest: &[u8; 32], secrets_digest: &[u8; 32]) -> [u8; 32] {
+    let mut digest = digest.to_vec();
+    digest.append(&mut secrets_digest.to_vec());
+    openssl::sha::sha256(&digest)
+}
+
+const ITEM_ROUTER: Router = Router::new()
+    .get(&API_METHOD_READ_S3_CLIENT_CONFIG)
+    .put(&API_METHOD_UPDATE_S3_CLIENT_CONFIG)
+    .delete(&API_METHOD_DELETE_S3_CLIENT_CONFIG);
+
+pub const ROUTER: Router = Router::new()
+    .get(&API_METHOD_LIST_S3_CLIENT_CONFIG)
+    .post(&API_METHOD_CREATE_S3_CLIENT_CONFIG)
+    .match_all("id", &ITEM_ROUTER);
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:31 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:31 +0200
Subject: [pbs-devel] [RFC proxmox-backup 30/39] ui: add s3 bucket selector
 and allow to set s3 backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-31-c.ebner@proxmox.com>

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/Makefile                 |  1 +
 www/form/S3BucketSelector.js | 40 ++++++++++++++++++++++++++++++++++++
 www/window/DataStoreEdit.js  | 35 +++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+)
 create mode 100644 www/form/S3BucketSelector.js

diff --git a/www/Makefile b/www/Makefile
index ca4683941..41deeee00 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -42,6 +42,7 @@ JSSRC=							\
 	Schema.js					\
 	form/TokenSelector.js				\
 	form/AuthidSelector.js				\
+	form/S3BucketSelector.js			\
 	form/RemoteSelector.js				\
 	form/RemoteTargetSelector.js   			\
 	form/DataStoreSelector.js			\
diff --git a/www/form/S3BucketSelector.js b/www/form/S3BucketSelector.js
new file mode 100644
index 000000000..c9905feb9
--- /dev/null
+++ b/www/form/S3BucketSelector.js
@@ -0,0 +1,40 @@
+Ext.define('PBS.form.S3BucketSelector', {
+    extend: 'Proxmox.form.ComboGrid',
+    alias: 'widget.pbsS3BucketSelector',
+
+    allowBlank: false,
+    autoSelect: false,
+    valueField: 'id',
+    displayField: 'id',
+
+    store: {
+	model: 'pmx-s3bucket',
+	autoLoad: true,
+	sorters: 'id',
+    },
+
+    listConfig: {
+	columns: [
+	    {
+		header: gettext('S3 Bucket ID'),
+		sortable: true,
+		dataIndex: 'id',
+		renderer: Ext.String.htmlEncode,
+		flex: 1,
+	    },
+	    {
+		header: gettext('Bucket'),
+		sortable: true,
+		dataIndex: 'bucket',
+		renderer: Ext.String.htmlEncode,
+		flex: 1,
+	    },
+	    {
+		header: gettext('Host'),
+		sortable: true,
+		dataIndex: 'host',
+		flex: 1,
+	    },
+	],
+    },
+});
diff --git a/www/window/DataStoreEdit.js b/www/window/DataStoreEdit.js
index 4a0b8d819..dffd2b2e0 100644
--- a/www/window/DataStoreEdit.js
+++ b/www/window/DataStoreEdit.js
@@ -101,6 +101,7 @@ Ext.define('PBS.DataStoreEdit', {
 		columnB: [
 		    {
 			xtype: 'checkbox',
+			name: 'removable-datastore',
 			boxLabel: gettext('Removable datastore'),
 			submitValue: false,
 			listeners: {
@@ -135,6 +136,37 @@ Ext.define('PBS.DataStoreEdit', {
 			fieldLabel: gettext('Reuse existing datastore'),
 		    },
 		],
+		advancedColumn2: [
+		    {
+			xtype: 'checkbox',
+			boxLabel: gettext('With S3 object store'),
+			submitValue: false,
+			listeners: {
+			    change: function(checkbox, withS3Backend) {
+				let inputPanel = checkbox.up('inputpanel');
+
+				let bucketSelector = inputPanel.down('[name=backend]');
+				bucketSelector.setDisabled(!withS3Backend);
+				bucketSelector.allowBlank = !withS3Backend;
+				bucketSelector.setValue('');
+
+				let removableDatastore = inputPanel.down('[name=removable-datastore]');
+				removableDatastore.setDisabled(withS3Backend);
+				removableDatastore.allowBlank = withS3Backend;
+				removableDatastore.setValue('');
+			    },
+			},
+		    },
+		    {
+			xtype: 'pbsS3BucketSelector',
+			name: 'backend',
+			fieldLabel: gettext('S3 Bucket ID'),
+			disabled: true,
+			cbind: {
+			    editable: '{isCreate}',
+			},
+		    },
+		],
 
 		onGetValues: function(values) {
 		    let me = this;
@@ -143,6 +175,9 @@ Ext.define('PBS.DataStoreEdit', {
 			// New datastores default to using the notification system
 			values['notification-mode'] = 'notification-system';
 		    }
+		    if (values.backend) {
+			values.backend = PBS.Utils.printPropertyString({ 's3': values.backend });
+		    }
 		    return values;
 		},
 	    },
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:36 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:36 +0200
Subject: [pbs-devel] [RFC proxmox-backup 35/39] api: backup: use local
 datastore cache on S3 backend chunk upload
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-36-c.ebner@proxmox.com>

Take advantage of the local datastore cache to avoid re-uploading of
already known chunks. This not only helps improve the backup/upload
speeds, but also avoids additionally costs by reducing the number of
requests and transferred payload data to the S3 object store api.

If the cache is present, lookup if it contains the chunk, skipping
upload altogether if it is. Otherwise, upload the chunk into memory,
upload it to the S3 object store api and insert it into the local
datastore cache.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/upload_chunk.rs | 47 ++++++++++++++++++++++++++++++---
 1 file changed, 43 insertions(+), 4 deletions(-)

diff --git a/src/api2/backup/upload_chunk.rs b/src/api2/backup/upload_chunk.rs
index 59f9ca558..1d82936e6 100644
--- a/src/api2/backup/upload_chunk.rs
+++ b/src/api2/backup/upload_chunk.rs
@@ -248,10 +248,49 @@ async fn upload_to_backend(
             UploadChunk::new(req_body, datastore, digest, size, encoded_size).await
         }
         DatastoreBackend::S3(s3_client) => {
-            let is_duplicate = match s3_client.put_object(digest.into(), req_body).await? {
-                PutObjectResponse::PreconditionFailed => true,
-                PutObjectResponse::NeedsRetry => bail!("concurrent operation, reupload required"),
-                PutObjectResponse::Success(_content) => false,
+            if datastore.cache_contains(&digest) {
+                return Ok((digest, size, encoded_size, true));
+            }
+            // TODO: Avoid this altoghether? put_object already loads the whole
+            // chunk into memory and does also hashing and crc32sum calculation
+            // for s3 request.
+            //
+            // Load chunk data into memory, need to write it twice,
+            // to S3 object store and local cache store.
+            let data = req_body
+                .map_err(Error::from)
+                .try_fold(Vec::new(), |mut acc, chunk| {
+                    acc.extend_from_slice(&chunk);
+                    future::ok::<_, Error>(acc)
+                })
+                .await?;
+
+            if encoded_size != data.len() as u32 {
+                bail!(
+                    "got blob with unexpected length ({encoded_size} != {})",
+                    data.len()
+                );
+            }
+
+            let upload_body = hyper::Body::from(data.clone());
+            let upload = s3_client.put_object(digest.into(), upload_body);
+            let cache_insert = tokio::task::spawn_blocking(move || {
+                let chunk = DataBlob::from_raw(data)?;
+                datastore.cache_insert(&digest, &chunk)
+            });
+            let is_duplicate = match futures::join!(upload, cache_insert) {
+                (Ok(upload_response), Ok(Ok(()))) => match upload_response {
+                    PutObjectResponse::PreconditionFailed => true,
+                    PutObjectResponse::NeedsRetry => {
+                        bail!("concurrent operation, reupload required")
+                    }
+                    PutObjectResponse::Success(_content) => false,
+                },
+                (Ok(_), Ok(Err(err))) => return Err(err.context("chunk cache insert failed")),
+                (Ok(_), Err(err)) => {
+                    return Err(Error::from(err).context("chunk cache insert task failed"))
+                }
+                (Err(err), _) => return Err(err.context("chunk upload failed")),
             };
             Ok((digest, size, encoded_size, is_duplicate))
         }
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:37 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:37 +0200
Subject: [pbs-devel] [RFC proxmox-backup 36/39] api: reader: use local
 datastore cache on S3 backend chunk fetching
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-37-c.ebner@proxmox.com>

Take advantage of the local datastore filesystem cache for datastores
backed by an s3 object store in order to reduce number of requests
and latency, and increase throughput.

Also, reducing the number of requests is cost beneficial for S3 object
stores charging for fetching of objects.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/reader/mod.rs | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/src/api2/reader/mod.rs b/src/api2/reader/mod.rs
index 3417f49be..edf7a738b 100644
--- a/src/api2/reader/mod.rs
+++ b/src/api2/reader/mod.rs
@@ -323,7 +323,29 @@ fn download_chunk(
 
         let body = match &env.backend {
             DatastoreBackend::Filesystem => load_from_filesystem(env, &digest)?,
-            DatastoreBackend::S3(s3_client) => fetch_from_object_store(s3_client, &digest).await?,
+            DatastoreBackend::S3(s3_client) => {
+                match env.datastore.cache() {
+                    None => fetch_from_object_store(s3_client, &digest).await?,
+                    Some(cache) => {
+                        //TODO: Avoid creating a new s3 client with new connection,
+                        let mut cacher = env
+                            .datastore
+                            .cacher()?
+                            .ok_or(format_err!("no cacher for datastore"))?;
+                        // Download from object store, insert to local cache store and read from
+                        // file. Can this be optimized?
+                        let chunk =
+                            cache
+                                .access(&digest, &mut cacher)
+                                .await?
+                                .ok_or(format_err!(
+                                    "unable to access chunk with digest {}",
+                                    hex::encode(digest)
+                                ))?;
+                        Body::from(chunk.raw_data().to_owned())
+                    }
+                }
+            }
         };
 
         // fixme: set other headers ?
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:38 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:38 +0200
Subject: [pbs-devel] [RFC proxmox-backup 37/39] api: backup: add no-cache
 flag to bypass local datastore cache
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-38-c.ebner@proxmox.com>

Adds the `no-cache` flag so the client can request to bypass the
local datastore cache for chunk uploads. This is mainly intended for
debugging and benchmarking, but can be used in cases the caching is
known to be ineffective (no possible deduplication).

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 examples/upload-speed.rs               |  1 +
 pbs-client/src/backup_writer.rs        |  4 +++-
 proxmox-backup-client/src/benchmark.rs |  1 +
 proxmox-backup-client/src/main.rs      |  8 +++++++
 src/api2/backup/environment.rs         |  3 +++
 src/api2/backup/mod.rs                 |  3 +++
 src/api2/backup/upload_chunk.rs        | 32 ++++++++++++++++++++++----
 src/server/push.rs                     |  1 +
 8 files changed, 48 insertions(+), 5 deletions(-)

diff --git a/examples/upload-speed.rs b/examples/upload-speed.rs
index e4b570ec5..8a6594a47 100644
--- a/examples/upload-speed.rs
+++ b/examples/upload-speed.rs
@@ -25,6 +25,7 @@ async fn upload_speed() -> Result<f64, Error> {
         &(BackupType::Host, "speedtest".to_string(), backup_time).into(),
         false,
         true,
+        false,
     )
     .await?;
 
diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs
index 325425069..a91880720 100644
--- a/pbs-client/src/backup_writer.rs
+++ b/pbs-client/src/backup_writer.rs
@@ -82,6 +82,7 @@ impl BackupWriter {
         backup: &BackupDir,
         debug: bool,
         benchmark: bool,
+        no_cache: bool,
     ) -> Result<Arc<BackupWriter>, Error> {
         let mut param = json!({
             "backup-type": backup.ty(),
@@ -89,7 +90,8 @@ impl BackupWriter {
             "backup-time": backup.time,
             "store": datastore,
             "debug": debug,
-            "benchmark": benchmark
+            "benchmark": benchmark,
+            "no-cache": no_cache,
         });
 
         if !ns.is_root() {
diff --git a/proxmox-backup-client/src/benchmark.rs b/proxmox-backup-client/src/benchmark.rs
index a6f24d745..ed21c7a91 100644
--- a/proxmox-backup-client/src/benchmark.rs
+++ b/proxmox-backup-client/src/benchmark.rs
@@ -236,6 +236,7 @@ async fn test_upload_speed(
         &(BackupType::Host, "benchmark".to_string(), backup_time).into(),
         false,
         true,
+        true,
     )
     .await?;
 
diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
index 44f4f5db5..83fc9309a 100644
--- a/proxmox-backup-client/src/main.rs
+++ b/proxmox-backup-client/src/main.rs
@@ -742,6 +742,12 @@ fn spawn_catalog_upload(
                 optional: true,
                 default: false,
             },
+            "no-cache": {
+                type: Boolean,
+                description: "Bypass local datastore cache for network storages.",
+                optional: true,
+                default: false,
+            },
         }
     }
  )]
@@ -754,6 +760,7 @@ async fn create_backup(
     change_detection_mode: Option<BackupDetectionMode>,
     dry_run: bool,
     skip_e2big_xattr: bool,
+    no_cache: bool,
     limit: ClientRateLimitConfig,
     _info: &ApiMethod,
     _rpcenv: &mut dyn RpcEnvironment,
@@ -960,6 +967,7 @@ async fn create_backup(
         &snapshot,
         true,
         false,
+        no_cache,
     )
     .await?;
 
diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 384e8a73f..874f0c44d 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -112,6 +112,7 @@ pub struct BackupEnvironment {
     result_attributes: Value,
     auth_id: Authid,
     pub debug: bool,
+    pub no_cache: bool,
     pub formatter: &'static dyn OutputFormatter,
     pub worker: Arc<WorkerTask>,
     pub datastore: Arc<DataStore>,
@@ -128,6 +129,7 @@ impl BackupEnvironment {
         worker: Arc<WorkerTask>,
         datastore: Arc<DataStore>,
         backup_dir: BackupDir,
+        no_cache: bool,
     ) -> Result<Self, Error> {
         let state = SharedBackupState {
             finished: false,
@@ -148,6 +150,7 @@ impl BackupEnvironment {
             worker,
             datastore,
             debug: tracing::enabled!(tracing::Level::DEBUG),
+            no_cache,
             formatter: JSON_FORMATTER,
             backup_dir,
             last_backup: None,
diff --git a/src/api2/backup/mod.rs b/src/api2/backup/mod.rs
index 2c6afca41..0913d4264 100644
--- a/src/api2/backup/mod.rs
+++ b/src/api2/backup/mod.rs
@@ -51,6 +51,7 @@ pub const API_METHOD_UPGRADE_BACKUP: ApiMethod = ApiMethod::new(
             ("backup-time", false, &BACKUP_TIME_SCHEMA),
             ("debug", true, &BooleanSchema::new("Enable verbose debug logging.").schema()),
             ("benchmark", true, &BooleanSchema::new("Job is a benchmark (do not keep data).").schema()),
+            ("no-cache", true, &BooleanSchema::new("Disable local datastore cache for network storages").schema()),
         ]),
     )
 ).access(
@@ -77,6 +78,7 @@ fn upgrade_to_backup_protocol(
     async move {
         let debug = param["debug"].as_bool().unwrap_or(false);
         let benchmark = param["benchmark"].as_bool().unwrap_or(false);
+        let no_cache = param["no-cache"].as_bool().unwrap_or(false);
 
         let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
 
@@ -212,6 +214,7 @@ fn upgrade_to_backup_protocol(
                     worker.clone(),
                     datastore,
                     backup_dir,
+                    no_cache,
                 )?;
 
                 env.debug = debug;
diff --git a/src/api2/backup/upload_chunk.rs b/src/api2/backup/upload_chunk.rs
index 1d82936e6..3b236238f 100644
--- a/src/api2/backup/upload_chunk.rs
+++ b/src/api2/backup/upload_chunk.rs
@@ -156,8 +156,14 @@ fn upload_fixed_chunk(
         let wid = required_integer_param(&param, "wid")? as usize;
         let env: &BackupEnvironment = rpcenv.as_ref();
 
-        let (digest, size, compressed_size, is_duplicate) =
-            upload_to_backend(req_body, param, env.datastore.clone(), &env.backend).await?;
+        let (digest, size, compressed_size, is_duplicate) = upload_to_backend(
+            req_body,
+            param,
+            env.datastore.clone(),
+            &env.backend,
+            env.no_cache,
+        )
+        .await?;
 
         env.register_fixed_chunk(wid, digest, size, compressed_size, is_duplicate)?;
         let digest_str = hex::encode(digest);
@@ -219,8 +225,14 @@ fn upload_dynamic_chunk(
         let wid = required_integer_param(&param, "wid")? as usize;
         let env: &BackupEnvironment = rpcenv.as_ref();
 
-        let (digest, size, compressed_size, is_duplicate) =
-            upload_to_backend(req_body, param, env.datastore.clone(), &env.backend).await?;
+        let (digest, size, compressed_size, is_duplicate) = upload_to_backend(
+            req_body,
+            param,
+            env.datastore.clone(),
+            &env.backend,
+            env.no_cache,
+        )
+        .await?;
 
         env.register_dynamic_chunk(wid, digest, size, compressed_size, is_duplicate)?;
         let digest_str = hex::encode(digest);
@@ -237,6 +249,7 @@ async fn upload_to_backend(
     param: Value,
     datastore: Arc<DataStore>,
     backend: &DatastoreBackend,
+    no_cache: bool,
 ) -> Result<([u8; 32], u32, u32, bool), Error> {
     let size = required_integer_param(&param, "size")? as u32;
     let encoded_size = required_integer_param(&param, "encoded-size")? as u32;
@@ -248,6 +261,17 @@ async fn upload_to_backend(
             UploadChunk::new(req_body, datastore, digest, size, encoded_size).await
         }
         DatastoreBackend::S3(s3_client) => {
+            if no_cache {
+                let is_duplicate = match s3_client.put_object(digest.into(), req_body).await? {
+                    PutObjectResponse::PreconditionFailed => true,
+                    PutObjectResponse::NeedsRetry => {
+                        bail!("concurrent operation, reupload required")
+                    }
+                    PutObjectResponse::Success(_content) => false,
+                };
+                return Ok((digest, size, encoded_size, is_duplicate));
+            }
+
             if datastore.cache_contains(&digest) {
                 return Ok((digest, size, encoded_size, true));
             }
diff --git a/src/server/push.rs b/src/server/push.rs
index e71012ed8..6a31d2abe 100644
--- a/src/server/push.rs
+++ b/src/server/push.rs
@@ -828,6 +828,7 @@ pub(crate) async fn push_snapshot(
         snapshot,
         false,
         false,
+        false,
     )
     .await?;
 
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:27 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:27 +0200
Subject: [pbs-devel] [RFC proxmox-backup 26/39] datastore: implement garbage
 collection for s3 backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-27-c.ebner@proxmox.com>

Implements the garbage collection for datastore's backed by an s3
object store.
Take advantage of the local datastore by placing marker files in the
chunk store during phase 1 of the garbage collection, updating their
atime if already present. By this expensive api calls can be avoided
to update the object metadata (only possible via a copy object
operation).

The phase 2 is implemented by fetching a list of all the chunks via
the ListObjectsV2 api call, filtered by the chunk folder prefix.
This operation has to be performed in patches of 1000 objects, given
by the api's response limits.
For each object key, lookup the marker file and decide based on the
marker existence and it's atime if the chunk object needs to be
removed. Deletion happens via the delete objects operation, allowing
to delete multiple chunks by a single request.

This allows to efficiently lookup chunks which are not in use
anymore while being performant and cost effective.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 200 ++++++++++++++++++++++++++++-----
 1 file changed, 175 insertions(+), 25 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 4fc6fe9a5..68d3ac6e2 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -4,7 +4,7 @@ use std::os::unix::ffi::OsStrExt;
 use std::os::unix::io::AsRawFd;
 use std::path::{Path, PathBuf};
 use std::sync::{Arc, LazyLock, Mutex};
-use std::time::Duration;
+use std::time::{Duration, SystemTime};
 
 use anyhow::{bail, format_err, Context, Error};
 use nix::unistd::{unlinkat, UnlinkatFlags};
@@ -1145,6 +1145,7 @@ impl DataStore {
         chunk_lru_cache: &mut LruCache<[u8; 32], ()>,
         status: &mut GarbageCollectionStatus,
         worker: &dyn WorkerTaskContext,
+        s3_client: Option<Arc<S3Client>>,
     ) -> Result<(), Error> {
         status.index_file_count += 1;
         status.index_data_bytes += index.index_bytes();
@@ -1159,21 +1160,41 @@ impl DataStore {
                 continue;
             }
 
-            if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
-                let hex = hex::encode(digest);
-                warn!(
-                    "warning: unable to access non-existent chunk {hex}, required by {file_name:?}"
-                );
-
-                // touch any corresponding .bad files to keep them around, meaning if a chunk is
-                // rewritten correctly they will be removed automatically, as well as if no index
-                // file requires the chunk anymore (won't get to this loop then)
-                for i in 0..=9 {
-                    let bad_ext = format!("{}.bad", i);
-                    let mut bad_path = PathBuf::new();
-                    bad_path.push(self.chunk_path(digest).0);
-                    bad_path.set_extension(bad_ext);
-                    self.inner.chunk_store.cond_touch_path(&bad_path, false)?;
+            match s3_client {
+                None => {
+                    // Filesystem backend
+                    if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
+                        let hex = hex::encode(digest);
+                        warn!(
+                            "warning: unable to access non-existent chunk {hex}, required by {file_name:?}"
+                        );
+
+                        // touch any corresponding .bad files to keep them around, meaning if a chunk is
+                        // rewritten correctly they will be removed automatically, as well as if no index
+                        // file requires the chunk anymore (won't get to this loop then)
+                        for i in 0..=9 {
+                            let bad_ext = format!("{}.bad", i);
+                            let mut bad_path = PathBuf::new();
+                            bad_path.push(self.chunk_path(digest).0);
+                            bad_path.set_extension(bad_ext);
+                            self.inner.chunk_store.cond_touch_path(&bad_path, false)?;
+                        }
+                    }
+                }
+                Some(ref _s3_client) => {
+                    // Update atime on local cache marker files.
+                    if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
+                        let (chunk_path, _digest) = self.chunk_path(digest);
+                        // Insert empty file as marker to tell GC phase2 that this is
+                        // a chunk still in-use, so to keep in the S3 object store.
+                        std::fs::File::options()
+                            .write(true)
+                            .create_new(true)
+                            .open(chunk_path)
+                            .with_context(|| {
+                                format!("failed to create marker for chunk {}", hex::encode(digest))
+                            })?;
+                    }
                 }
             }
         }
@@ -1185,6 +1206,7 @@ impl DataStore {
         status: &mut GarbageCollectionStatus,
         worker: &dyn WorkerTaskContext,
         cache_capacity: usize,
+        s3_client: Option<Arc<S3Client>>,
     ) -> Result<(), Error> {
         // Iterate twice over the datastore to fetch index files, even if this comes with an
         // additional runtime cost:
@@ -1274,6 +1296,7 @@ impl DataStore {
                                 &mut chunk_lru_cache,
                                 status,
                                 worker,
+                                s3_client.as_ref().cloned(),
                             )?;
 
                             if !unprocessed_index_list.remove(&path) {
@@ -1308,7 +1331,14 @@ impl DataStore {
                     continue;
                 }
             };
-            self.index_mark_used_chunks(index, &path, &mut chunk_lru_cache, status, worker)?;
+            self.index_mark_used_chunks(
+                index,
+                &path,
+                &mut chunk_lru_cache,
+                status,
+                worker,
+                s3_client.as_ref().cloned(),
+            )?;
             warn!("Marked chunks for unexpected index file at '{path:?}'");
         }
         if strange_paths_count > 0 {
@@ -1406,18 +1436,138 @@ impl DataStore {
                 1024 * 1024
             };
 
-            info!("Start GC phase1 (mark used chunks)");
+            let s3_client = match self.backend()? {
+                DatastoreBackend::Filesystem => None,
+                DatastoreBackend::S3(s3_client) => {
+                    proxmox_async::runtime::block_on(s3_client.head_bucket())
+                        .context("failed to reach bucket")?;
+                    Some(s3_client)
+                }
+            };
 
-            self.mark_used_chunks(&mut gc_status, worker, gc_cache_capacity)
-                .context("marking used chunks failed")?;
+            info!("Start GC phase1 (mark used chunks)");
 
-            info!("Start GC phase2 (sweep unused chunks)");
-            self.inner.chunk_store.sweep_unused_chunks(
-                oldest_writer,
-                min_atime,
+            self.mark_used_chunks(
                 &mut gc_status,
                 worker,
-            )?;
+                gc_cache_capacity,
+                s3_client.as_ref().cloned(),
+            )
+            .context("marking used chunks failed")?;
+
+            info!("Start GC phase2 (sweep unused chunks)");
+
+            if let Some(ref s3_client) = s3_client {
+                let mut chunk_count = 0;
+                let prefix = Some(".chunks/");
+                // Operates in batches of 1000 objects max per request
+                let mut list_bucket_result =
+                    proxmox_async::runtime::block_on(s3_client.list_objects_v2(prefix, None, None))?;
+
+                let mut delete_list = Vec::with_capacity(1000);
+                loop {
+                    for content in list_bucket_result.contents {
+                        // Check object is actually a chunk
+                        let digest = match Path::new(&content.key).file_name() {
+                            Some(file_name) => file_name,
+                            // should never be the case as objects will have a filename
+                            None => continue,
+                        };
+                        let bytes = digest.as_bytes();
+                        if bytes.len() != 64 && bytes.len() != 64 + ".0.bad".len() {
+                            continue;
+                        }
+                        if !bytes.iter().take(64).all(u8::is_ascii_hexdigit) {
+                            continue;
+                        }
+
+                        let bad = bytes.ends_with(b".bad");
+
+                        // Check local markers (created or atime updated during phase1) and
+                        // keep or delete chunk based on that.
+
+                        let mut chunk_path = self.base_path();
+                        chunk_path.push(&content.key);
+                        let atime = match std::fs::metadata(chunk_path) {
+                            Ok(stat) => stat.accessed()?,
+                            Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
+                                // File not found, delete by setting atime to unix epoch
+                                info!("Not found, mark for deletion: {}", content.key);
+                                SystemTime::UNIX_EPOCH
+                            }
+                            Err(err) => return Err(err.into()),
+                        };
+                        let atime = atime.duration_since(SystemTime::UNIX_EPOCH)?.as_secs() as i64;
+
+                        chunk_count += 1;
+
+                        if atime < min_atime {
+                            delete_list.push(content.key);
+                            if bad {
+                                gc_status.removed_bad += 1;
+                            } else {
+                                gc_status.removed_chunks += 1;
+                            }
+                            gc_status.removed_bytes += content.size;
+                        } else if atime < oldest_writer {
+                            if bad {
+                                gc_status.still_bad += 1;
+                            } else {
+                                gc_status.pending_chunks += 1;
+                            }
+                            gc_status.pending_bytes += content.size;
+                        } else {
+                            if !bad {
+                                gc_status.disk_chunks += 1;
+                            }
+                            gc_status.disk_bytes += content.size;
+                        }
+                    }
+
+                    if !delete_list.is_empty() {
+                        //TODO: error handling
+                        let _delete_objects_result = proxmox_async::runtime::block_on(
+                            s3_client.delete_objects(&delete_list),
+                        )?;
+                        delete_list.clear();
+                    }
+
+                    // Process next batch of chunks if there is more
+                    if list_bucket_result.is_truncated {
+                        list_bucket_result =
+                            proxmox_async::runtime::block_on(s3_client.list_objects_v2(
+                                prefix,
+                                None,
+                                list_bucket_result.next_continuation_token.as_deref(),
+                            ))?;
+                        continue;
+                    }
+
+                    break;
+                }
+                info!("processed {chunk_count} total chunks");
+
+                // Phase 2 GC of Filesystem backed storage is phase 3 for S3 backed GC
+                info!("Start GC phase3 (sweep unused chunk markers)");
+
+                let mut tmp_gc_status = GarbageCollectionStatus {
+                    upid: Some(upid.to_string()),
+                    ..Default::default()
+                };
+                self.inner.chunk_store.sweep_unused_chunks(
+                    oldest_writer,
+                    min_atime,
+                    &mut tmp_gc_status,
+                    worker,
+                )?;
+            } else {
+                self.inner.chunk_store.sweep_unused_chunks(
+                    oldest_writer,
+                    min_atime,
+                    &mut gc_status,
+                    worker,
+                )?;
+            }
 
             info!(
                 "Removed garbage: {}",
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:01 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:01 +0200
Subject: [pbs-devel] [RFC proxmox proxmox-backup 00/39] S3 storage backend
 for datastores
Message-ID: <20250519114640.303640-1-c.ebner@proxmox.com>

Disclaimer: These patches are in a development state and are not
intended for production use.

This patch series aims to add S3 compatible object stores as storage
backend for PBS datastores. A PBS local cache store using the regular
datastore layout is used for faster operation, bypassing requests to
the S3 api when possible. Further, the local cache store allows to
keep frequently used chunks and is used to avoid expensive metadata
updates on the object store, e.g. by using local marker file during
garbage collection.

Backups are created by upload chunks to the corresponding S3 bucket,
while keeping the index files in the local cache store, on backup
finish, the snapshot metadata are persisted to the S3 storage backend.

Snapshot restores read chunks preferably from the local cache store,
downloading and insterting them if not present from the S3 object
store.

Listing and snapsoht metadata operation currently rely soly on the
local cache store, with the intention to provide a mechanism to
re-sync and merge with object stored on the S3 backend if requested.

Sending this patch series as RFC to get some initial feedback, mostly
on the S3 client implementation part and the corresponding
configuration integration with PBS, which is already in an advanced
stage and warants initial review and real world testing.

Datastore operations on the S3 backend are still work in progress,
but feedback on that is appreciated very much as well.

Among the open points still being worked on are:
- Locking mechanism and consistency between local cache and S3 store.
- Sync and merge of namespace, group snapshot and index files when
  required or requested.
- Advanced packing mechanism for chunks to significantly reduce the
  number of api requests and therefore be more cost effective.
- Reduction of in-memory copies for chunks/blobs and recalculation of
  checksums.

Testing:
For testing, an S3 compatible object store provided via Ceph RADOS
gateway can be used by the following setup. This was performed on a
pre-existing Ceph Reef 18.2 cluster.

Install radosgw on all the nodes:
```
apt install radosgw
```

On one node, generate client keyring:
```
ceph-authtool --create-keyring /etc/pve/priv/ceph.client.radosgw.keyring
```

For each node, generate key and add it to the keyring (adapt name
accordingly):
```
ceph-authtool /etc/pve/priv/ceph.client.radosgw.keyring -n client.radosgw.pve-c0-n1 --gen-key
```

Setup capabilities for client keys:
```
ceph-authtool -n client.radosgw.pve-c0-n1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/pve/priv/ceph.client.radosgw.keyring
```

Add the keys (repeat for each) to the cluster:
```
ceph -k /etc/pve/priv/ceph.client.admin.keyring auth add client.radosgw.pve-c0-n1 -i /etc/pve/priv/ceph.client.radosgw.keyring
```

For each client, add a config based on the one below to /etc/ceph/ceph.conf
```
[client.radosgw.pve-c0-n1]
        host = pve-c0-n1
        keyring = /etc/pve/priv/ceph.client.radosgw.keyring
        log file = /var/log/ceph/client.radosgw.$host.log
        rgw_dns_name = s3.pve-c0-n1.local
```

Restart the service for each node, e.g.
```
systemctl daemon-reload
systemctl restart radosgw.service
```

Setup a new user, generating access key and secret key shown in
output:
```
radosgw-admin user create --uid=testuser --display-name="TestUser" --email=your at mail.com
```

Since the configuration and keyring are located on the pmxcfs, add
the following override so the gateway service is only started after
pve-cluster by adding to
`/etc/systemd/system/radosgw.service.d/override.conf`:
```
[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/radosgw
Description=LSB: radosgw RESTful rados gateway
After=pve-cluster.service
Wants=pve-cluster.service
```

A custom certificate must be added since the client forces tls by
extending the config with a path to a custom generated certificate and
key:
```
[client.radosgw.pve-c0-n1]
	host = pve-c0-n1
	keyring = /etc/pve/priv/ceph.client.radosgw.keyring
	logfile = /var/log/ceph/client.radsgw.$host.log
	rgw_dns_name = s3.pve-c0-n1.local
	rgw_frontends = "beast ssl_port=7480 ssl_certificate=/etc/pve/ceph/server-cert.pem ssl_private_key=/etc/pve/ceph/server-key.pem"
```

A new bucket can finally be created using the `s3cmd` cli tool after
initial configuration.

proxmox:

Christian Ebner (2):
  pbs-api-types: add types for S3 client configs and secrets
  pbs-api-types: extend datastore config by backend config enum

 pbs-api-types/src/datastore.rs |  58 +++++++++++++-
 pbs-api-types/src/lib.rs       |   3 +
 pbs-api-types/src/s3.rs        | 138 +++++++++++++++++++++++++++++++++
 3 files changed, 198 insertions(+), 1 deletion(-)
 create mode 100644 pbs-api-types/src/s3.rs

proxmox-backup:

Christian Ebner (37):
  fmt: fix minor formatting issues
  verify: refactor verify related functions to be methods of worker
  s3 client: add crate for AWS S3 compatible object store client
  s3 client: implement AWS signature v4 request authentication
  s3 client: add dedicated type for s3 object keys
  s3 client: add helper for last modified timestamp parsing
  s3 client: add helper to parse http date headers
  s3 client: implement methods to operate on s3 objects in bucket
  config: introduce s3 object store client configuration
  api: config: implement endpoints to manipulate and list s3 configs
  api: datastore: check S3 backend bucket access on datastore create
  datastore: allow to get the backend for a datastore
  api: backup: store datastore backend in runtime environment
  api: backup: conditionally upload chunks to S3 object store backend
  api: backup: conditionally upload blobs to S3 object store backend
  api: backup: conditionally upload indices to S3 object store backend
  api: backup: conditionally upload manifest to S3 object store backend
  api: reader: fetch chunks based on datastore backend
  datastore: local chunk reader: read chunks based on backend
  verify worker: add datastore backed to verify worker
  verify: implement chunk verification for stores with s3 backend
  api: remove snapshot from S3 backend on snapshot delete
  datastore: prune groups/snapshots from S3 object store backend
  datastore: implement garbage collection for s3 backend
  ui: add S3 client edit window for configuration create/edit
  ui: add S3 client view for configuration
  ui: expose the S3 client view in the navigation tree
  ui: add s3 bucket selector and allow to set s3 backend
  api/bin: add endpoint and command to test s3 backend for datastore
  tools: lru cache: add removed callback for evicted nodes
  tools: async lru cache: implement insert, remove and contains methods
  datastore: add local datastore cache for network attached storages
  api: backup: use local datastore cache on S3 backend chunk upload
  api: reader: use local datastore cache on S3 backend chunk fetching
  api: backup: add no-cache flag to bypass local datastore cache
  datastore: get and set owner for S3 store backend
  datastore: create namespace marker in S3 backend

 Cargo.toml                                    |   6 +
 examples/upload-speed.rs                      |   1 +
 pbs-client/src/backup_writer.rs               |   4 +-
 pbs-config/src/lib.rs                         |   1 +
 pbs-config/src/s3.rs                          |  73 ++
 pbs-datastore/Cargo.toml                      |   3 +
 pbs-datastore/src/cached_chunk_reader.rs      |   6 +-
 pbs-datastore/src/datastore.rs                | 387 +++++++-
 pbs-datastore/src/dynamic_index.rs            |   1 +
 pbs-datastore/src/lib.rs                      |   4 +
 pbs-datastore/src/local_chunk_reader.rs       |  37 +-
 .../src/local_datastore_lru_cache.rs          | 116 +++
 pbs-s3-client/Cargo.toml                      |  28 +
 pbs-s3-client/src/aws_sign_v4.rs              | 140 +++
 pbs-s3-client/src/client.rs                   | 501 ++++++++++
 pbs-s3-client/src/lib.rs                      | 220 +++++
 pbs-s3-client/src/object_key.rs               |  64 ++
 pbs-s3-client/src/response_reader.rs          | 324 +++++++
 pbs-tools/src/async_lru_cache.rs              |  46 +-
 pbs-tools/src/lru_cache.rs                    |  42 +-
 proxmox-backup-client/src/benchmark.rs        |   1 +
 proxmox-backup-client/src/main.rs             |   8 +
 src/api2/admin/datastore.rs                   | 129 ++-
 src/api2/backup/environment.rs                | 145 ++-
 src/api2/backup/mod.rs                        | 107 +--
 src/api2/backup/upload_chunk.rs               | 112 ++-
 src/api2/config/datastore.rs                  |  41 +-
 src/api2/config/mod.rs                        |   2 +
 src/api2/config/s3.rs                         | 349 +++++++
 src/api2/reader/environment.rs                |  12 +-
 src/api2/reader/mod.rs                        |  60 +-
 src/backup/verify.rs                          | 879 +++++++++---------
 src/bin/proxmox_backup_manager/datastore.rs   |  24 +
 src/server/push.rs                            |   1 +
 src/server/verify_job.rs                      |  12 +-
 www/Makefile                                  |   3 +
 www/NavigationTree.js                         |   6 +
 www/config/S3BucketView.js                    | 144 +++
 www/form/S3BucketSelector.js                  |  40 +
 www/window/DataStoreEdit.js                   |  35 +
 www/window/S3BucketEdit.js                    | 125 +++
 41 files changed, 3618 insertions(+), 621 deletions(-)
 create mode 100644 pbs-config/src/s3.rs
 create mode 100644 pbs-datastore/src/local_datastore_lru_cache.rs
 create mode 100644 pbs-s3-client/Cargo.toml
 create mode 100644 pbs-s3-client/src/aws_sign_v4.rs
 create mode 100644 pbs-s3-client/src/client.rs
 create mode 100644 pbs-s3-client/src/lib.rs
 create mode 100644 pbs-s3-client/src/object_key.rs
 create mode 100644 pbs-s3-client/src/response_reader.rs
 create mode 100644 src/api2/config/s3.rs
 create mode 100644 www/config/S3BucketView.js
 create mode 100644 www/form/S3BucketSelector.js
 create mode 100644 www/window/S3BucketEdit.js

-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:08 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:08 +0200
Subject: [pbs-devel] [RFC proxmox-backup 07/39] s3 client: add dedicated
 type for s3 object keys
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-8-c.ebner@proxmox.com>

S3 objects are uniquely identified within a bucket by their object
key [0].

Implements conversion and utility traits to easily convert and encode
a string or a chunk digest as corresponding object key for the S3
storage backend. Adds type checking for s3 client operations requiring
an object key.

[0] https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-s3-client/src/lib.rs        |  2 ++
 pbs-s3-client/src/object_key.rs | 64 +++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+)
 create mode 100644 pbs-s3-client/src/object_key.rs

diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
index 5a60b92ec..a4081df15 100644
--- a/pbs-s3-client/src/lib.rs
+++ b/pbs-s3-client/src/lib.rs
@@ -1,3 +1,5 @@
 mod aws_sign_v4;
 mod client;
 pub use client::{S3Client, S3ClientOptions};
+mod object_key;
+pub use object_key::{S3ObjectKey, S3_CONTENT_PREFIX};
diff --git a/pbs-s3-client/src/object_key.rs b/pbs-s3-client/src/object_key.rs
new file mode 100644
index 000000000..362c0cd55
--- /dev/null
+++ b/pbs-s3-client/src/object_key.rs
@@ -0,0 +1,64 @@
+use crate::aws_sign_v4::aws_sign_v4_uri_encode;
+
+pub const S3_CONTENT_PREFIX: &str = ".content";
+
+#[derive(Clone)]
+pub struct S3ObjectKey {
+    object_key: String,
+}
+
+// All regular keys (non-digests) get prefixed by a `/.contents`, so that
+// content listing without all the chunks can be done by that prefix.
+impl core::convert::From<&str> for S3ObjectKey {
+    fn from(object_key: &str) -> Self {
+        let object_key = object_key.strip_prefix("/").unwrap_or(object_key);
+        let object_key = format!(
+            "/{S3_CONTENT_PREFIX}/{object_key}",
+            object_key = aws_sign_v4_uri_encode(object_key, true)
+        );
+
+        Self { object_key }
+    }
+}
+
+impl core::convert::From<&[u8; 32]> for S3ObjectKey {
+    fn from(digest: &[u8; 32]) -> Self {
+        // Use the same layout as on regular PBS datastores, including the 4 hex digit prefix
+        let object_key = hex::encode(digest);
+        let prefix = &object_key[..4];
+        Self {
+            object_key: format!("/.chunks/{prefix}/{object_key}"),
+        }
+    }
+}
+
+impl core::convert::From<[u8; 32]> for S3ObjectKey {
+    fn from(digest: [u8; 32]) -> Self {
+        Self::from(&digest)
+    }
+}
+
+impl std::ops::Deref for S3ObjectKey {
+    type Target = str;
+
+    fn deref(&self) -> &Self::Target {
+        &self.object_key
+    }
+}
+
+impl std::fmt::Display for S3ObjectKey {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", self.object_key)
+    }
+}
+
+impl S3ObjectKey {
+    /// Generate source key for copy object operations given the source bucket.
+    pub fn to_copy_source_key(&self, source_bucket: &str) -> Self {
+        Self {
+            // object key already contains the required separator slash in-between source bucket
+            // and source object key.
+            object_key: format!("{source_bucket}{}", self.object_key),
+        }
+    }
+}
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:10 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:10 +0200
Subject: [pbs-devel] [RFC proxmox-backup 09/39] s3 client: add helper to
 parse http date headers
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-10-c.ebner@proxmox.com>

Add a helper to parse the preferred date/time format for http `Date`
headers as specified in RFC 2616 [0], which is a fixed-length subset
of the format specified in RFC 1123 [1], itself being a followup to
RFC 822 [2]. Does not implement the format as described in the
obsolete RFC 850 [3].

This allows to parse the `Date` and `Last-Modified` headers of S3 API
responses.

[0] https://datatracker.ietf.org/doc/html/rfc2616#section-3.3
[1] https://datatracker.ietf.org/doc/html/rfc1123#section-5.2.14
[2] https://datatracker.ietf.org/doc/html/rfc822#section-5
[3] https://datatracker.ietf.org/doc/html/rfc850

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-s3-client/src/lib.rs | 97 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 96 insertions(+), 1 deletion(-)

diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
index 308db64d8..00fa26455 100644
--- a/pbs-s3-client/src/lib.rs
+++ b/pbs-s3-client/src/lib.rs
@@ -6,7 +6,12 @@ pub use object_key::{S3ObjectKey, S3_CONTENT_PREFIX};
 
 use std::time::Duration;
 
-use anyhow::{bail, Error};
+use anyhow::{bail, Context, Error};
+
+const VALID_DAYS_OF_WEEK: [&str; 7] = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"];
+const VALID_MONTHS: [&str; 12] = [
+    "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec",
+];
 
 #[derive(Debug)]
 pub struct LastModifiedTimestamp {
@@ -121,3 +126,93 @@ impl std::str::FromStr for LastModifiedTimestamp {
 }
 
 serde_plain::derive_deserialize_from_fromstr!(LastModifiedTimestamp, "last modified timestamp");
+
+/// Preferred date format specified by RFC2616, given as fixed-length
+/// subset of RFC1123, which itself is a followup to RFC822.
+///
+/// https://datatracker.ietf.org/doc/html/rfc2616#section-3.3
+/// https://datatracker.ietf.org/doc/html/rfc1123#section-5.2.14
+/// https://datatracker.ietf.org/doc/html/rfc822#section-5
+#[derive(Debug)]
+pub struct HttpDate {
+    epoch: i64,
+}
+
+impl HttpDate {
+    pub fn to_duration(&self) -> Result<Duration, Error> {
+        let seconds = u64::try_from(self.epoch)?;
+        Ok(Duration::from_secs(seconds))
+    }
+}
+
+impl std::str::FromStr for HttpDate {
+    type Err = Error;
+
+    fn from_str(timestamp: &str) -> Result<Self, Self::Err> {
+        let input = timestamp.as_bytes();
+        if input.len() != 29 {
+            bail!("unexpected length: got {}", input.len());
+        }
+
+        let expect = |pos: usize, c: u8| {
+            if input[pos] != c {
+                bail!("unexpected char at pos {pos}");
+            }
+            Ok(())
+        };
+
+        let digit = |pos: usize| -> Result<i32, Error> {
+            let digit = input[pos] as i32;
+            if !(48..=57).contains(&digit) {
+                bail!("unexpected char at pos {pos}");
+            }
+            Ok(digit - 48)
+        };
+
+        fn check_max(i: i32, max: i32) -> Result<i32, Error> {
+            if i > max {
+                bail!("value too large ({i} > {max})");
+            }
+            Ok(i)
+        }
+
+        let mut tm = proxmox_time::TmEditor::new(true);
+
+        if !VALID_DAYS_OF_WEEK
+            .iter()
+            .any(|valid| valid.as_bytes() == &input[0..3])
+        {
+            bail!("unexpected day of week, got {:?}", &input[0..3]);
+        }
+
+        expect(3, b',').context("unexpected separator after day of week")?;
+        expect(4, b' ').context("missing space after day of week separator")?;
+        tm.set_mday(check_max(digit(5)? * 10 + digit(6)?, 31)?)?;
+        expect(7, b' ').context("unexpected separator after day")?;
+        if let Some(month) = VALID_MONTHS
+            .iter()
+            .position(|month| month.as_bytes() == &input[8..11])
+        {
+            // valid conversion to i32, position stems from fixed size array of 12 months.
+            tm.set_mon(check_max(month as i32 + 1, 12)?)?;
+        } else {
+            bail!("invalid month");
+        }
+        expect(11, b' ').context("unexpected separator after month")?;
+        tm.set_year(digit(12)? * 1000 + digit(13)? * 100 + digit(14)? * 10 + digit(15)?)?;
+        expect(16, b' ').context("unexpected separator after year")?;
+        tm.set_hour(check_max(digit(17)? * 10 + digit(18)?, 23)?)?;
+        expect(19, b':').context("unexpected separator after hour")?;
+        tm.set_min(check_max(digit(20)? * 10 + digit(21)?, 59)?)?;
+        expect(22, b':').context("unexpected separator after minute")?;
+        tm.set_sec(check_max(digit(23)? * 10 + digit(24)?, 60)?)?;
+        expect(25, b' ').context("unexpected separator after second")?;
+        if !input.ends_with(b"GMT") {
+            bail!("unexpected timezone");
+        }
+
+        let epoch = tm.into_epoch()?;
+
+        Ok(Self { epoch })
+    }
+}
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:14 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:14 +0200
Subject: [pbs-devel] [RFC proxmox-backup 13/39] api: datastore: check S3
 backend bucket access on datastore create
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-14-c.ebner@proxmox.com>

Check if the configured S3 object store backend can be reached and
the provided secrets have the permissions to access the bucket.

Perform the check before creating the chunk store, so it is not left
behind if the bucket cannot be reached.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/config/datastore.rs | 41 ++++++++++++++++++++++++++++++++----
 1 file changed, 37 insertions(+), 4 deletions(-)

diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index b133be707..19b08b7e4 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -3,6 +3,7 @@ use std::path::{Path, PathBuf};
 use ::serde::{Deserialize, Serialize};
 use anyhow::{bail, Context, Error};
 use hex::FromHex;
+use pbs_s3_client::{S3Client, S3ClientOptions};
 use serde_json::Value;
 use tracing::{info, warn};
 
@@ -12,10 +13,10 @@ use proxmox_section_config::SectionConfigData;
 use proxmox_uuid::Uuid;
 
 use pbs_api_types::{
-    Authid, DataStoreConfig, DataStoreConfigUpdater, DatastoreNotify, DatastoreTuning, KeepOptions,
-    MaintenanceMode, PruneJobConfig, PruneJobOptions, DATASTORE_SCHEMA, PRIV_DATASTORE_ALLOCATE,
-    PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_MODIFY, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA,
-    UPID_SCHEMA,
+    Authid, DataStoreConfig, DataStoreConfigUpdater, DatastoreBackendConfig, DatastoreNotify,
+    DatastoreTuning, KeepOptions, MaintenanceMode, PruneJobConfig, PruneJobOptions, S3ClientConfig,
+    S3ClientSecretsConfig, DATASTORE_SCHEMA, PRIV_DATASTORE_ALLOCATE, PRIV_DATASTORE_AUDIT,
+    PRIV_DATASTORE_MODIFY, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA, UPID_SCHEMA,
 };
 use pbs_config::BackupLockGuard;
 use pbs_datastore::chunk_store::ChunkStore;
@@ -116,6 +117,38 @@ pub(crate) fn do_create_datastore(
             .parse_property_string(datastore.tuning.as_deref().unwrap_or(""))?,
     )?;
 
+    if let Some(ref backend_config) = datastore.backend {
+        let backend_config: DatastoreBackendConfig = backend_config.parse()?;
+        match backend_config {
+            DatastoreBackendConfig::Filesystem => (),
+            DatastoreBackendConfig::S3(ref s3_client_id) => {
+                let (config, _config_digest) =
+                    pbs_config::s3::config().context("failed to get s3 config")?;
+                let (secrets, _secrets_digest) =
+                    pbs_config::s3::secrets_config().context("failed to get s3 secrets")?;
+                let config: S3ClientConfig = config
+                    .lookup("s3client", s3_client_id)
+                    .with_context(|| format!("no '{s3_client_id}' in config"))?;
+                let secrets: S3ClientSecretsConfig = secrets
+                    .lookup("s3secrets", s3_client_id)
+                    .with_context(|| format!("no '{s3_client_id}' in secrets"))?;
+                let options = S3ClientOptions {
+                    host: config.host,
+                    port: config.port,
+                    bucket: config.bucket,
+                    region: config.region.unwrap_or("us-west-1".to_string()),
+                    fingerprint: config.fingerprint,
+                    access_key: config.access_key,
+                    secret_key: secrets.secret_key,
+                };
+                let s3_client = S3Client::new(options).context("failed to create s3 client")?;
+                // Fine to block since this runs in worker task
+                proxmox_async::runtime::block_on(s3_client.head_bucket())
+                    .context("failed to access bucket")?;
+            }
+        }
+    }
+
     let unmount_guard = if datastore.backing_device.is_some() {
         do_mount_device(datastore.clone())?;
         UnmountGuard::new(Some(path.clone()))
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:18 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:18 +0200
Subject: [pbs-devel] [RFC proxmox-backup 17/39] api: backup: conditionally
 upload blobs to S3 object store backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-18-c.ebner@proxmox.com>

Upload blobs to both, the local datastore cache and the S3 object
store if s3 is configured as backend.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/environment.rs | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 8919b919a..393a8351d 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -581,6 +581,31 @@ impl BackupEnvironment {
         let blob = DataBlob::load_from_reader(&mut &data[..])?;
 
         let raw_data = blob.raw_data();
+        if let DatastoreBackend::S3(s3_client) = &self.backend {
+            let data = Body::from(raw_data.to_vec());
+            let mut object_key = self.backup_dir.relative_path();
+            object_key.push(file_name);
+            let object_key = object_key
+                .as_os_str()
+                .to_str()
+                .ok_or_else(|| format_err!("invalid path"))?;
+            match proxmox_async::runtime::block_on(s3_client.put_object(object_key.into(), data))? {
+                PutObjectResponse::PreconditionFailed => {
+                    self.log(format!(
+                        "Upload of blob failed, object {object_key} already present."
+                    ));
+                    bail!("upload of blob failed");
+                }
+                PutObjectResponse::NeedsRetry => {
+                    self.log("Upload of blob failed, reupload required.");
+                    bail!("concurrent operation, reupload required");
+                }
+                PutObjectResponse::Success(_content) => {
+                    self.log(format!("Uploaded blob to object store: {object_key}"))
+                }
+            }
+        }
+
         replace_file(&path, raw_data, CreateOptions::new(), false)?;
 
         self.log(format!(
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:15 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:15 +0200
Subject: [pbs-devel] [RFC proxmox-backup 14/39] datastore: allow to get the
 backend for a datastore
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-15-c.ebner@proxmox.com>

Implements an enum with variants Filesystem and S3 to distinguish
between available backends. Filesystem will be used as default, if no
backend is configured in the datastores configuration. If the
datastore has an s3 backend configured, the backend method will
instantiate and s3 client and return it with the S3 variant.

This allows to instantiate the client once, keeping and reusing the
same open connection to the api for the lifetime of task or job, e.g.
in the backup writer/readers runtime environment.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/Cargo.toml       |  1 +
 pbs-datastore/src/datastore.rs | 46 ++++++++++++++++++++++++++++++++--
 pbs-datastore/src/lib.rs       |  1 +
 3 files changed, 46 insertions(+), 2 deletions(-)

diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index 7623adc28..3ee06c9bb 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -44,4 +44,5 @@ pbs-api-types.workspace = true
 pbs-buildcfg.workspace = true
 pbs-config.workspace = true
 pbs-key-config.workspace = true
+pbs-s3-client.workspace = true
 pbs-tools.workspace = true
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index cbf78ecb6..42d27d249 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -8,6 +8,7 @@ use std::time::Duration;
 
 use anyhow::{bail, format_err, Context, Error};
 use nix::unistd::{unlinkat, UnlinkatFlags};
+use pbs_s3_client::{S3Client, S3ClientOptions};
 use pbs_tools::lru_cache::LruCache;
 use tracing::{info, warn};
 
@@ -23,8 +24,9 @@ use proxmox_worker_task::WorkerTaskContext;
 
 use pbs_api_types::{
     ArchiveType, Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, ChunkOrder,
-    DataStoreConfig, DatastoreFSyncLevel, DatastoreTuning, GarbageCollectionStatus,
-    MaintenanceMode, MaintenanceType, Operation, UPID,
+    DataStoreConfig, DatastoreBackendConfig, DatastoreFSyncLevel, DatastoreTuning,
+    GarbageCollectionStatus, MaintenanceMode, MaintenanceType, Operation, S3ClientConfig,
+    S3ClientSecretsConfig, UPID,
 };
 use pbs_config::BackupLockGuard;
 
@@ -125,6 +127,7 @@ pub struct DataStoreImpl {
     chunk_order: ChunkOrder,
     last_digest: Option<[u8; 32]>,
     sync_level: DatastoreFSyncLevel,
+    backend_config: DatastoreBackendConfig,
 }
 
 impl DataStoreImpl {
@@ -139,6 +142,7 @@ impl DataStoreImpl {
             chunk_order: Default::default(),
             last_digest: None,
             sync_level: Default::default(),
+            backend_config: Default::default(),
         })
     }
 }
@@ -194,6 +198,12 @@ impl Drop for DataStore {
     }
 }
 
+#[derive(Clone)]
+pub enum DatastoreBackend {
+    Filesystem,
+    S3(Arc<S3Client>),
+}
+
 impl DataStore {
     // This one just panics on everything
     #[doc(hidden)]
@@ -204,6 +214,32 @@ impl DataStore {
         })
     }
 
+    /// Get the backend for this datastore based on it's configuration
+    pub fn backend(&self) -> Result<DatastoreBackend, Error> {
+        let backend_type = match self.inner.backend_config {
+            DatastoreBackendConfig::Filesystem => DatastoreBackend::Filesystem,
+            DatastoreBackendConfig::S3(ref s3_client_id) => {
+                let (config, _config_digest) = pbs_config::s3::config()?;
+                let (secrets, _secrets_digest) = pbs_config::s3::secrets_config()?;
+                let config: S3ClientConfig = config.lookup("s3client", s3_client_id)?;
+                let secrets: S3ClientSecretsConfig = secrets.lookup("s3secrets", s3_client_id)?;
+                let options = S3ClientOptions {
+                    host: config.host,
+                    port: config.port,
+                    bucket: config.bucket,
+                    region: config.region.unwrap_or("us-west-1".to_string()),
+                    fingerprint: config.fingerprint,
+                    access_key: config.access_key,
+                    secret_key: secrets.secret_key,
+                };
+                let s3_client = S3Client::new(options)?;
+                DatastoreBackend::S3(Arc::new(s3_client))
+            }
+        };
+
+        Ok(backend_type)
+    }
+
     pub fn lookup_datastore(
         name: &str,
         operation: Option<Operation>,
@@ -381,6 +417,11 @@ impl DataStore {
                 .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
         )?;
 
+        let backend_config = match config.backend {
+            Some(config) => config.parse()?,
+            None => Default::default(),
+        };
+
         Ok(DataStoreImpl {
             chunk_store,
             gc_mutex: Mutex::new(()),
@@ -389,6 +430,7 @@ impl DataStore {
             chunk_order: tuning.chunk_order.unwrap_or_default(),
             last_digest,
             sync_level: tuning.sync_level.unwrap_or_default(),
+            backend_config,
         })
     }
 
diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index 5014b6c09..e6f65575b 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -203,6 +203,7 @@ pub use store_progress::StoreProgress;
 mod datastore;
 pub use datastore::{
     check_backup_owner, ensure_datastore_is_mounted, get_datastore_mount_status, DataStore,
+    DatastoreBackend,
 };
 
 mod hierarchy;
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:20 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:20 +0200
Subject: [pbs-devel] [RFC proxmox-backup 19/39] api: backup: conditionally
 upload manifest to S3 object store backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-20-c.ebner@proxmox.com>

Upload the manifest to the S3 object store backend after it has been
finished in the backup api call handler, if s3 is configured as
backend. Keep also the locally cached version for fast and efficient
listing of contents without the need to perform expensive (as in
monetary cost and IO latency) requests.

The datastore's metadata contents will be synced from the S3 backend
during datastore opening.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/environment.rs | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 72e369bcf..685b78e89 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -12,7 +12,7 @@ use serde_json::{json, Value};
 use proxmox_router::{RpcEnvironment, RpcEnvironmentType};
 use proxmox_sys::fs::{replace_file, CreateOptions};
 
-use pbs_api_types::Authid;
+use pbs_api_types::{Authid, MANIFEST_BLOB_NAME};
 use pbs_datastore::backup_info::{BackupDir, BackupInfo};
 use pbs_datastore::dynamic_index::DynamicIndexWriter;
 use pbs_datastore::fixed_index::FixedIndexWriter;
@@ -719,6 +719,37 @@ impl BackupEnvironment {
             }
         }
 
+        if let DatastoreBackend::S3(s3_client) = &self.backend {
+            // Upload manifest to S3 object store
+            let mut object_key = self.backup_dir.relative_path();
+            object_key.push(MANIFEST_BLOB_NAME.as_ref());
+            let mut path = self.datastore.base_path();
+            path.push(&object_key);
+            let mut manifest = std::fs::File::open(&path)?;
+            let mut buffer = Vec::new();
+            manifest.read_to_end(&mut buffer)?;
+            let data = Body::from(buffer);
+            let object_key = object_key
+                .as_os_str()
+                .to_str()
+                .ok_or_else(|| format_err!("invalid path"))?;
+            match proxmox_async::runtime::block_on(s3_client.put_object(object_key.into(), data))? {
+                PutObjectResponse::PreconditionFailed => {
+                    self.log(format!(
+                        "Upload of manifest failed, object {object_key} already present."
+                    ));
+                    bail!("upload of manifest failed");
+                }
+                PutObjectResponse::NeedsRetry => {
+                    self.log("Upload of manifest failed, reupload required.");
+                    bail!("concurrent operation, reupload required");
+                }
+                PutObjectResponse::Success(_content) => {
+                    self.log(format!("Uploaded manifest to object store: {object_key}"))
+                }
+            }
+        }
+
         self.datastore.try_ensure_sync_level()?;
 
         // marks the backup as successful
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:16 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:16 +0200
Subject: [pbs-devel] [RFC proxmox-backup 15/39] api: backup: store datastore
 backend in runtime environment
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-16-c.ebner@proxmox.com>

Get and store the datastore's backend during creation of the backup
runtime environment and upload the chunks to the local filesystem or
s3 object store based on the backend variant.

By storing the backend variant in the environment the s3 client is
instantiated only once and reused for all api calls in the same
backup http/2 connection.

Refactor the upgrade method by moving all logic into the async block,
such that the now possible error on backup environment creation gets
propagated to the thread spawn call side.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/environment.rs | 12 +++--
 src/api2/backup/mod.rs         | 99 +++++++++++++++++-----------------
 2 files changed, 57 insertions(+), 54 deletions(-)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 6cd29f512..8919b919a 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -15,7 +15,8 @@ use pbs_api_types::Authid;
 use pbs_datastore::backup_info::{BackupDir, BackupInfo};
 use pbs_datastore::dynamic_index::DynamicIndexWriter;
 use pbs_datastore::fixed_index::FixedIndexWriter;
-use pbs_datastore::{DataBlob, DataStore};
+use pbs_datastore::{DataBlob, DataStore, DatastoreBackend};
+use pbs_s3_client::PutObjectResponse;
 use proxmox_rest_server::{formatter::*, WorkerTask};
 
 use crate::backup::VerifyWorker;
@@ -115,6 +116,7 @@ pub struct BackupEnvironment {
     pub datastore: Arc<DataStore>,
     pub backup_dir: BackupDir,
     pub last_backup: Option<BackupInfo>,
+    pub backend: DatastoreBackend,
     state: Arc<Mutex<SharedBackupState>>,
 }
 
@@ -125,7 +127,7 @@ impl BackupEnvironment {
         worker: Arc<WorkerTask>,
         datastore: Arc<DataStore>,
         backup_dir: BackupDir,
-    ) -> Self {
+    ) -> Result<Self, Error> {
         let state = SharedBackupState {
             finished: false,
             uid_counter: 0,
@@ -137,7 +139,8 @@ impl BackupEnvironment {
             backup_stat: UploadStatistic::new(),
         };
 
-        Self {
+        let backend = datastore.backend()?;
+        Ok(Self {
             result_attributes: json!({}),
             env_type,
             auth_id,
@@ -147,8 +150,9 @@ impl BackupEnvironment {
             formatter: JSON_FORMATTER,
             backup_dir,
             last_backup: None,
+            backend,
             state: Arc::new(Mutex::new(state)),
-        }
+        })
     }
 
     /// Register a Chunk with associated length.
diff --git a/src/api2/backup/mod.rs b/src/api2/backup/mod.rs
index 567bca3ef..2c6afca41 100644
--- a/src/api2/backup/mod.rs
+++ b/src/api2/backup/mod.rs
@@ -185,7 +185,8 @@ fn upgrade_to_backup_protocol(
             }
 
             // lock last snapshot to prevent forgetting/pruning it during backup
-            let guard = last.backup_dir
+            let guard = last
+                .backup_dir
                 .lock_shared()
                 .with_context(|| format!("while locking last snapshot during backup '{last:?}'"))?;
             Some(guard)
@@ -204,14 +205,14 @@ fn upgrade_to_backup_protocol(
             Some(worker_id),
             auth_id.to_string(),
             true,
-            move |worker| {
+            move |worker| async move {
                 let mut env = BackupEnvironment::new(
                     env_type,
                     auth_id,
                     worker.clone(),
                     datastore,
                     backup_dir,
-                );
+                )?;
 
                 env.debug = debug;
                 env.last_backup = last_backup;
@@ -264,55 +265,53 @@ fn upgrade_to_backup_protocol(
                     });
                 let mut abort_future = abort_future.map(|_| Err(format_err!("task aborted")));
 
-                async move {
-                    // keep flock until task ends
-                    let _group_guard = _group_guard;
-                    let snap_guard = snap_guard;
-                    let _last_guard = _last_guard;
-
-                    let res = select! {
-                        req = req_fut => req,
-                        abrt = abort_future => abrt,
-                    };
-                    if benchmark {
-                        env.log("benchmark finished successfully");
-                        proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
-                        return Ok(());
+                // keep flock until task ends
+                let _group_guard = _group_guard;
+                let snap_guard = snap_guard;
+                let _last_guard = _last_guard;
+
+                let res = select! {
+                    req = req_fut => req,
+                    abrt = abort_future => abrt,
+                };
+                if benchmark {
+                    env.log("benchmark finished successfully");
+                    proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
+                    return Ok(());
+                }
+
+                let verify = |env: BackupEnvironment| {
+                    if let Err(err) = env.verify_after_complete(snap_guard) {
+                        env.log(format!(
+                            "backup finished, but starting the requested verify task failed: {}",
+                            err
+                        ));
                     }
+                };
 
-                    let verify = |env: BackupEnvironment| {
-                        if let Err(err) = env.verify_after_complete(snap_guard) {
-                            env.log(format!(
-                                "backup finished, but starting the requested verify task failed: {}",
-                                err
-                            ));
-                        }
-                    };
-
-                    match (res, env.ensure_finished()) {
-                        (Ok(_), Ok(())) => {
-                            env.log("backup finished successfully");
-                            verify(env);
-                            Ok(())
-                        }
-                        (Err(err), Ok(())) => {
-                            // ignore errors after finish
-                            env.log(format!("backup had errors but finished: {}", err));
-                            verify(env);
-                            Ok(())
-                        }
-                        (Ok(_), Err(err)) => {
-                            env.log(format!("backup ended and finish failed: {}", err));
-                            env.log("removing unfinished backup");
-                            proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
-                            Err(err)
-                        }
-                        (Err(err), Err(_)) => {
-                            env.log(format!("backup failed: {}", err));
-                            env.log("removing failed backup");
-                            proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
-                            Err(err)
-                        }
+                match (res, env.ensure_finished()) {
+                    (Ok(_), Ok(())) => {
+                        env.log("backup finished successfully");
+                        verify(env);
+                        Ok(())
+                    }
+                    (Err(err), Ok(())) => {
+                        // ignore errors after finish
+                        env.log(format!("backup had errors but finished: {}", err));
+                        verify(env);
+                        Ok(())
+                    }
+                    (Ok(_), Err(err)) => {
+                        env.log(format!("backup ended and finish failed: {}", err));
+                        env.log("removing unfinished backup");
+                        proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
+                        Err(err)
+                    }
+                    (Err(err), Err(_)) => {
+                        env.log(format!("backup failed: {}", err));
+                        env.log("removing failed backup");
+                        proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
+                        Err(err)
                     }
                 }
             },
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:21 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:21 +0200
Subject: [pbs-devel] [RFC proxmox-backup 20/39] api: reader: fetch chunks
 based on datastore backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-21-c.ebner@proxmox.com>

Read the chunk based on the datastores backend, reading from local
filesystem or fetching from S3 object store.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/reader/environment.rs | 12 +++++++----
 src/api2/reader/mod.rs         | 38 ++++++++++++++++++++++------------
 2 files changed, 33 insertions(+), 17 deletions(-)

diff --git a/src/api2/reader/environment.rs b/src/api2/reader/environment.rs
index 3b2f06f43..8924352b0 100644
--- a/src/api2/reader/environment.rs
+++ b/src/api2/reader/environment.rs
@@ -1,13 +1,14 @@
 use std::collections::HashSet;
 use std::sync::{Arc, RwLock};
 
+use anyhow::Error;
 use serde_json::{json, Value};
 
 use proxmox_router::{RpcEnvironment, RpcEnvironmentType};
 
 use pbs_api_types::Authid;
 use pbs_datastore::backup_info::BackupDir;
-use pbs_datastore::DataStore;
+use pbs_datastore::{DataStore, DatastoreBackend};
 use proxmox_rest_server::formatter::*;
 use proxmox_rest_server::WorkerTask;
 use tracing::info;
@@ -23,6 +24,7 @@ pub struct ReaderEnvironment {
     pub worker: Arc<WorkerTask>,
     pub datastore: Arc<DataStore>,
     pub backup_dir: BackupDir,
+    pub backend: DatastoreBackend,
     allowed_chunks: Arc<RwLock<HashSet<[u8; 32]>>>,
 }
 
@@ -33,8 +35,9 @@ impl ReaderEnvironment {
         worker: Arc<WorkerTask>,
         datastore: Arc<DataStore>,
         backup_dir: BackupDir,
-    ) -> Self {
-        Self {
+    ) -> Result<Self, Error> {
+        let backend = datastore.backend()?;
+        Ok(Self {
             result_attributes: json!({}),
             env_type,
             auth_id,
@@ -43,8 +46,9 @@ impl ReaderEnvironment {
             debug: tracing::enabled!(tracing::Level::DEBUG),
             formatter: JSON_FORMATTER,
             backup_dir,
+            backend,
             allowed_chunks: Arc::new(RwLock::new(HashSet::new())),
-        }
+        })
     }
 
     pub fn log<S: AsRef<str>>(&self, msg: S) {
diff --git a/src/api2/reader/mod.rs b/src/api2/reader/mod.rs
index cc791299c..3417f49be 100644
--- a/src/api2/reader/mod.rs
+++ b/src/api2/reader/mod.rs
@@ -24,7 +24,8 @@ use pbs_api_types::{
 };
 use pbs_config::CachedUserInfo;
 use pbs_datastore::index::IndexFile;
-use pbs_datastore::{DataStore, PROXMOX_BACKUP_READER_PROTOCOL_ID_V1};
+use pbs_datastore::{DataStore, DatastoreBackend, PROXMOX_BACKUP_READER_PROTOCOL_ID_V1};
+use pbs_s3_client::S3Client;
 use pbs_tools::json::required_string_param;
 
 use crate::api2::backup::optional_ns_param;
@@ -159,7 +160,7 @@ fn upgrade_to_backup_reader_protocol(
                     worker.clone(),
                     datastore,
                     backup_dir,
-                );
+                )?;
 
                 env.debug = debug;
 
@@ -320,17 +321,10 @@ fn download_chunk(
             ));
         }
 
-        let (path, _) = env.datastore.chunk_path(&digest);
-        let path2 = path.clone();
-
-        env.debug(format!("download chunk {:?}", path));
-
-        let data =
-            proxmox_async::runtime::block_in_place(|| std::fs::read(path)).map_err(move |err| {
-                http_err!(BAD_REQUEST, "reading file {:?} failed: {}", path2, err)
-            })?;
-
-        let body = Body::from(data);
+        let body = match &env.backend {
+            DatastoreBackend::Filesystem => load_from_filesystem(env, &digest)?,
+            DatastoreBackend::S3(s3_client) => fetch_from_object_store(s3_client, &digest).await?,
+        };
 
         // fixme: set other headers ?
         Ok(Response::builder()
@@ -342,6 +336,24 @@ fn download_chunk(
     .boxed()
 }
 
+async fn fetch_from_object_store(s3_client: &S3Client, digest: &[u8; 32]) -> Result<Body, Error> {
+    if let Some(response) = s3_client.get_object(digest.into()).await? {
+        return Ok(response.content);
+    }
+    bail!("cannot find chunk with digest {}", hex::encode(digest));
+}
+
+fn load_from_filesystem(env: &ReaderEnvironment, digest: &[u8; 32]) -> Result<Body, Error> {
+    let (path, _) = env.datastore.chunk_path(digest);
+    let path2 = path.clone();
+
+    env.debug(format!("download chunk {path:?}"));
+
+    let data = proxmox_async::runtime::block_in_place(|| std::fs::read(path))
+        .map_err(move |err| http_err!(BAD_REQUEST, "reading file {path2:?} failed: {err}"))?;
+    Ok(Body::from(data))
+}
+
 /* this is too slow
 fn download_chunk_old(
     _parts: Parts,
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:25 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:25 +0200
Subject: [pbs-devel] [RFC proxmox-backup 24/39] api: remove snapshot from S3
 backend on snapshot delete
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-25-c.ebner@proxmox.com>

Prune the snapshot from the S3 object store followed by removing the
contents from the local cache store.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 7b7f79b22..45204369a 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -63,8 +63,9 @@ use pbs_datastore::manifest::BackupManifest;
 use pbs_datastore::prune::compute_prune_info;
 use pbs_datastore::{
     check_backup_owner, ensure_datastore_is_mounted, task_tracking, BackupDir, BackupGroup,
-    DataStore, LocalChunkReader, StoreProgress,
+    DataStore, DatastoreBackend, LocalChunkReader, StoreProgress,
 };
+use pbs_s3_client::S3_CONTENT_PREFIX;
 use pbs_tools::json::required_string_param;
 use proxmox_rest_server::{formatter, WorkerTask};
 
@@ -432,6 +433,20 @@ pub async fn delete_snapshot(
 
         let snapshot = datastore.backup_dir(ns, backup_dir)?;
 
+        // TODO: How to handle locking for consistency?
+        if let DatastoreBackend::S3(s3_client) = datastore.backend()? {
+            let path = snapshot.relative_path();
+            let snapshot_prefix = path
+                .to_str()
+                .ok_or_else(|| format_err!("invalid snapshot path prefix"))?;
+            let prefix = format!("{S3_CONTENT_PREFIX}/{snapshot_prefix}");
+            let delete_object_result =
+                proxmox_async::runtime::block_on(s3_client.delete_objects_by_prefix(&prefix))?;
+            if delete_object_result.error.is_some() {
+                bail!("deleting objects failed");
+            }
+        }
+
         snapshot.destroy(false)?;
 
         Ok(Value::Null)
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:26 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:26 +0200
Subject: [pbs-devel] [RFC proxmox-backup 25/39] datastore: prune
 groups/snapshots from S3 object store backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-26-c.ebner@proxmox.com>

When pruning a backup group or a backup snapshot for a datastore with
S3 object store backend, remove the associated objects by removing
them based on the prefix.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 42d27d249..4fc6fe9a5 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -29,6 +29,7 @@ use pbs_api_types::{
     S3ClientSecretsConfig, UPID,
 };
 use pbs_config::BackupLockGuard;
+use pbs_s3_client::S3_CONTENT_PREFIX;
 
 use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, OLD_LOCKING};
 use crate::chunk_store::ChunkStore;
@@ -716,6 +717,20 @@ impl DataStore {
     ) -> Result<BackupGroupDeleteStats, Error> {
         let backup_group = self.backup_group(ns.clone(), backup_group.clone());
 
+        // TODO: Handle protected snapshots and consistency
+        if let DatastoreBackend::S3(s3_client) = self.backend()? {
+            let path = backup_group.relative_group_path();
+            let group_prefix = path
+                .to_str()
+                .ok_or_else(|| format_err!("invalid group path prefix"))?;
+            let prefix = format!("{S3_CONTENT_PREFIX}/{group_prefix}");
+            let delete_object_result =
+                proxmox_async::runtime::block_on(s3_client.delete_objects_by_prefix(&prefix))?;
+            if delete_object_result.error.is_some() {
+                bail!("deleting objects failed");
+            }
+        }
+
         backup_group.destroy()
     }
 
@@ -728,6 +743,19 @@ impl DataStore {
     ) -> Result<(), Error> {
         let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
 
+        if let DatastoreBackend::S3(s3_client) = self.backend()? {
+            let path = backup_dir.relative_path();
+            let snapshot_prefix = path
+                .to_str()
+                .ok_or_else(|| format_err!("invalid snapshot path prefix"))?;
+            let prefix = format!("{S3_CONTENT_PREFIX}/{snapshot_prefix}");
+            let delete_object_result =
+                proxmox_async::runtime::block_on(s3_client.delete_objects_by_prefix(&prefix))?;
+            if delete_object_result.error.is_some() {
+                bail!("deleting objects failed");
+            }
+        }
+
         backup_dir.destroy(force)
     }
 
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:24 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:24 +0200
Subject: [pbs-devel] [RFC proxmox-backup 23/39] verify: implement chunk
 verification for stores with s3 backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-24-c.ebner@proxmox.com>

For datastores backed by an S3 compatible object store, rather than
reading the chunks to be verified from the local filesystem, fetch
them via the s3 client from the configured bucket.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/backup/verify.rs | 59 +++++++++++++++++++++++++++++++++++---------
 1 file changed, 47 insertions(+), 12 deletions(-)

diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index a01ddcca3..2c28c6af5 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -5,6 +5,7 @@ use std::sync::{Arc, Mutex};
 use std::time::Instant;
 
 use anyhow::{bail, Error};
+use hyper::body::HttpBody;
 use tracing::{error, info, warn};
 
 use proxmox_worker_task::WorkerTaskContext;
@@ -189,18 +190,52 @@ impl VerifyWorker {
                 continue; // already verified or marked corrupt
             }
 
-            match self.datastore.load_chunk(&info.digest) {
-                Err(err) => {
-                    self.corrupt_chunks.lock().unwrap().insert(info.digest);
-                    error!("can't verify chunk, load failed - {err}");
-                    errors.fetch_add(1, Ordering::SeqCst);
-                    Self::rename_corrupted_chunk(self.datastore.clone(), &info.digest);
-                }
-                Ok(chunk) => {
-                    let size = info.size();
-                    read_bytes += chunk.raw_size();
-                    decoder_pool.send((chunk, info.digest, size))?;
-                    decoded_bytes += size;
+            match &self.backend {
+                DatastoreBackend::Filesystem => match self.datastore.load_chunk(&info.digest) {
+                    Err(err) => {
+                        self.corrupt_chunks.lock().unwrap().insert(info.digest);
+                        error!("can't verify chunk, load failed - {err}");
+                        errors.fetch_add(1, Ordering::SeqCst);
+                        Self::rename_corrupted_chunk(self.datastore.clone(), &info.digest);
+                    }
+                    Ok(chunk) => {
+                        let size = info.size();
+                        read_bytes += chunk.raw_size();
+                        decoder_pool.send((chunk, info.digest, size))?;
+                        decoded_bytes += size;
+                    }
+                },
+                DatastoreBackend::S3(s3_client) => {
+                    //TODO: How to avoid all these requests? Does the AWS api offer other means
+                    // to verify the contents/integrity of objects?
+                    match proxmox_async::runtime::block_on(s3_client.get_object(info.digest.into()))
+                    {
+                        Ok(Some(response)) => {
+                            let bytes =
+                                proxmox_async::runtime::block_on(response.content.collect())?
+                                    .to_bytes();
+                            let chunk = DataBlob::from_raw(bytes.to_vec())?;
+                            let size = info.size();
+                            read_bytes += chunk.raw_size();
+                            decoder_pool.send((chunk, info.digest, size))?;
+                            decoded_bytes += size;
+                        }
+                        Ok(None) => {
+                            self.corrupt_chunks.lock().unwrap().insert(info.digest);
+                            error!(
+                                "can't verify missing chunk with digest {}",
+                                hex::encode(info.digest)
+                            );
+                            errors.fetch_add(1, Ordering::SeqCst);
+                        }
+                        Err(err) => {
+                            self.corrupt_chunks.lock().unwrap().insert(info.digest);
+                            error!("can't verify chunk, load failed - {err}");
+                            errors.fetch_add(1, Ordering::SeqCst);
+                            //TODO: How to handle corrupt chunks for S3 store?
+                            //Self::rename_corrupted_chunk(self.datastore.clone(), &info.digest);
+                        }
+                    }
                 }
             }
         }
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:28 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:28 +0200
Subject: [pbs-devel] [RFC proxmox-backup 27/39] ui: add S3 client edit
 window for configuration create/edit
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-28-c.ebner@proxmox.com>

Adds an edit window for creating or editing S3 client configurations.
Loosely based on the same edit window for the remote configuration.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs |   5 +-
 www/window/S3BucketEdit.js     | 125 +++++++++++++++++++++++++++++++++
 2 files changed, 128 insertions(+), 2 deletions(-)
 create mode 100644 www/window/S3BucketEdit.js

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 68d3ac6e2..a15f82b5b 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1461,8 +1461,9 @@ impl DataStore {
                 let mut chunk_count = 0;
                 let prefix = Some(".chunks/");
                 // Operates in batches of 1000 objects max per request
-                let mut list_bucket_result =
-                    proxmox_async::runtime::block_on(s3_client.list_objects_v2(prefix, None, None))?;
+                let mut list_bucket_result = proxmox_async::runtime::block_on(
+                    s3_client.list_objects_v2(prefix, None, None),
+                )?;
 
                 let mut delete_list = Vec::with_capacity(1000);
                 loop {
diff --git a/www/window/S3BucketEdit.js b/www/window/S3BucketEdit.js
new file mode 100644
index 000000000..1491ddbe5
--- /dev/null
+++ b/www/window/S3BucketEdit.js
@@ -0,0 +1,125 @@
+Ext.define('PBS.window.S3BucketEdit', {
+    extend: 'Proxmox.window.Edit',
+    alias: 'widget.pbsS3BucketEdit',
+    mixins: ['Proxmox.Mixin.CBind'],
+
+    onlineHelp: 'backup_s3bucket',
+
+    isAdd: true,
+
+    subject: gettext('S3 Bucket'),
+
+    fieldDefaults: { labelWidth: 120 },
+
+    cbindData: function(initialConfig) {
+	let me = this;
+
+	let baseurl = '/api2/extjs/config/s3';
+	let id = initialConfig.id;
+
+	me.isCreate = !id;
+	me.url = id ? `${baseurl}/${id}` : baseurl;
+	me.method = id ? 'PUT' : 'POST';
+	me.autoLoad = !!id;
+	return {
+	    passwordEmptyText: me.isCreate ? '' : gettext('Unchanged'),
+	};
+    },
+
+    items: {
+	xtype: 'inputpanel',
+	column1: [
+	    {
+		xtype: 'pmxDisplayEditField',
+		name: 'id',
+		fieldLabel: gettext('Unique Identifier'),
+		renderer: Ext.htmlEncode,
+		allowBlank: false,
+		minLength: 4,
+		cbind: {
+		    editable: '{isCreate}',
+		},
+	    },
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'host',
+		fieldLabel: gettext('Host'),
+		allowBlank: false,
+		emptyText: gettext('FQDN or IP-address'),
+	    },
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'port',
+		fieldLabel: gettext('Port'),
+		emptyText: gettext("default"),
+		cbind: {
+		    deleteEmpty: '{!isCreate}',
+		},
+	    },
+	],
+
+	column2: [
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'bucket',
+		fieldLabel: gettext('Bucket'),
+		allowBlank: false,
+	    },
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'region',
+		fieldLabel: gettext('Region'),
+		emptyText: gettext("default"),
+	    },
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'access-key',
+		fieldLabel: gettext('Access Key'),
+		cbind: {
+		    emptyText: '{passwordEmptyText}',
+		    allowBlank: '{!isCreate}',
+		},
+	    },
+	    {
+		xtype: 'textfield',
+		name: 'secret-key',
+		inputType: 'password',
+		fieldLabel: gettext('Secret Key'),
+		cbind: {
+		    emptyText: '{passwordEmptyText}',
+		    allowBlank: '{!isCreate}',
+		},
+	    },
+	],
+
+	columnB: [
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'fingerprint',
+		fieldLabel: gettext('Fingerprint'),
+		emptyText: gettext("Server certificate's SHA-256 fingerprint, required for self-signed certificates"),
+		cbind: {
+		    deleteEmpty: '{!isCreate}',
+		},
+	    },
+	],
+    },
+
+    getValues: function() {
+	let me = this;
+	let values = me.callParent(arguments);
+
+	if (me.isCreate) {
+	    /// Secrets are stored into separate config, but set the same id for both configs
+	    values['secrets-id'] = values.id;
+	}
+	if (values['access-key'] === '') {
+	    delete values['access-key']
+	}
+	if (values['secret-key'] === '') {
+	    delete values['secret-key']
+	}
+
+	return values;
+    },
+});
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:29 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:29 +0200
Subject: [pbs-devel] [RFC proxmox-backup 28/39] ui: add S3 client view for
 configuration
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-29-c.ebner@proxmox.com>

Adds the view to configure S3 clients in the Configuration section of
the UI.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/config/S3BucketView.js | 144 +++++++++++++++++++++++++++++++++++++
 1 file changed, 144 insertions(+)
 create mode 100644 www/config/S3BucketView.js

diff --git a/www/config/S3BucketView.js b/www/config/S3BucketView.js
new file mode 100644
index 000000000..85ac6c49c
--- /dev/null
+++ b/www/config/S3BucketView.js
@@ -0,0 +1,144 @@
+Ext.define('pmx-s3bucket', {
+    extend: 'Ext.data.Model',
+    fields: ['id', 'host', 'bucket', 'port', 'access-key', 'secret-key', 'region', 'fingerprint'],
+    idProperty: 'id',
+    proxy: {
+	type: 'proxmox',
+	url: '/api2/json/config/s3',
+    },
+});
+
+Ext.define('PBS.config.S3BucketView', {
+    extend: 'Ext.grid.GridPanel',
+    alias: 'widget.pbsS3BucketView',
+
+    title: gettext('S3 Buckets'),
+
+    stateful: true,
+    stateId: 'grid-s3buckets',
+    tools: [PBS.Utils.get_help_tool("backup-s3-bucket")],
+
+    controller: {
+	xclass: 'Ext.app.ViewController',
+
+	addS3Bucket: function() {
+	    let me = this;
+            Ext.create('PBS.window.S3BucketEdit', {
+		listeners: {
+		    destroy: function() {
+			me.reload();
+		    },
+		},
+            }).show();
+	},
+
+	editS3Bucket: function() {
+	    let me = this;
+	    let view = me.getView();
+	    let selection = view.getSelection();
+	    if (selection.length < 1) return;
+
+            Ext.create('PBS.window.S3BucketEdit', {
+                id: selection[0].data.id,
+		listeners: {
+		    destroy: function() {
+			me.reload();
+		    },
+		},
+            }).show();
+	},
+
+	reload: function() { this.getView().getStore().rstore.load(); },
+
+	init: function(view) {
+	    Proxmox.Utils.monStoreErrors(view, view.getStore().rstore);
+	},
+    },
+
+    listeners: {
+	activate: 'reload',
+	itemdblclick: 'editS3Bucket',
+    },
+
+    store: {
+	type: 'diff',
+	autoDestroy: true,
+	autoDestroyRstore: true,
+	sorters: 'id',
+	rstore: {
+	    type: 'update',
+	    storeid: 'pmx-s3bucket',
+	    model: 'pmx-s3bucket',
+	    autoStart: true,
+	    interval: 5000,
+	},
+    },
+
+    tbar: [
+	{
+	    xtype: 'proxmoxButton',
+	    text: gettext('Add'),
+	    handler: 'addS3Bucket',
+	    selModel: false,
+	},
+	{
+	    xtype: 'proxmoxButton',
+	    text: gettext('Edit'),
+	    handler: 'editS3Bucket',
+	    disabled: true,
+	},
+	{
+	    xtype: 'proxmoxStdRemoveButton',
+	    baseurl: '/config/s3',
+	    callback: 'reload',
+	},
+    ],
+
+    viewConfig: {
+	trackOver: false,
+    },
+
+    columns: [
+	{
+	    dataIndex: 'id',
+	    header: gettext('Unique Identifier'),
+	    renderer: Ext.String.htmlEncode,
+	    sortable: true,
+	    width: 200,
+	},
+	{
+	    dataIndex: 'bucket',
+	    header: gettext('Bucket'),
+	    renderer: Ext.String.htmlEncode,
+	    sortable: true,
+	    width: 200,
+	},
+	{
+	    dataIndex: 'host',
+	    header: gettext('Host'),
+	    sortable: true,
+	    width: 200,
+	},
+	{
+	    dataIndex: 'port',
+	    header: gettext('Port'),
+	    renderer: Ext.String.htmlEncode,
+	    sortable: true,
+	    width: 100,
+	},
+	{
+	    dataIndex: 'region',
+	    header: gettext('Region'),
+	    renderer: Ext.String.htmlEncode,
+	    sortable: true,
+	    width: 100,
+	},
+	{
+	    dataIndex: 'fingerprint',
+	    header: gettext('Fingerprint'),
+	    renderer: Ext.String.htmlEncode,
+	    sortable: false,
+	    flex: 1,
+	},
+    ],
+});
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:40 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:40 +0200
Subject: [pbs-devel] [RFC proxmox-backup 39/39] datastore: create namespace
 marker in S3 backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-40-c.ebner@proxmox.com>

The S3 object store only allows to store objects, referenced by their
key. For backup namespaces datastores however use directories, so
they cannot be represented as one to one mapping.

Instead, create an empty marker file for each namespace and operate
based on that.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 9c8b7de03..a45b21413 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -635,6 +635,25 @@ impl DataStore {
         // construct ns before mkdir to enforce max-depth and name validity
         let ns = BackupNamespace::from_parent_ns(parent, name)?;
 
+        if let DatastoreBackend::S3(s3_client) = self.backend()? {
+            let mut marker = ns.path();
+            marker.push(".namespace");
+            let namespace_marker = marker
+                .to_str()
+                .ok_or_else(|| format_err!("unexpected namespace path"))?;
+
+            let response = proxmox_async::runtime::block_on(
+                s3_client.put_object(namespace_marker.into(), hyper::body::Body::empty()),
+            )?;
+            match response {
+                PutObjectResponse::NeedsRetry => bail!("failed to create namespace, needs retry"),
+                PutObjectResponse::PreconditionFailed => {
+                    bail!("failed to create namespace, precondition failed")
+                }
+                PutObjectResponse::Success(_) => (),
+            }
+        }
+
         let mut ns_full_path = self.base_path();
         ns_full_path.push(ns.path());
 
@@ -645,6 +664,19 @@ impl DataStore {
 
     /// Returns if the given namespace exists on the datastore
     pub fn namespace_exists(&self, ns: &BackupNamespace) -> bool {
+        if let DatastoreBackend::S3(s3_client) = self.backend().unwrap() {
+            let mut marker = ns.path();
+            marker.push(".namespace");
+            let namespace_marker = marker
+                .to_str()
+                .ok_or_else(|| format_err!("unexpected namespace path"))
+                .unwrap();
+
+            let response =
+                proxmox_async::runtime::block_on(s3_client.head_object(namespace_marker.into()))
+                    .unwrap();
+            return response.is_some();
+        }
         let mut path = self.base_path();
         path.push(ns.path());
         path.exists()
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:39 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:39 +0200
Subject: [pbs-devel] [RFC proxmox-backup 38/39] datastore: get and set owner
 for S3 store backend
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-39-c.ebner@proxmox.com>

Read or write the ownership information from/to the corresponding
object in the S3 object store. Keep that information available if
the bucket is reused as datastore.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 34 +++++++++++++++++++++++++++++++++-
 1 file changed, 33 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 19d4ca02c..9c8b7de03 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -8,7 +8,7 @@ use std::time::{Duration, SystemTime};
 
 use anyhow::{bail, format_err, Context, Error};
 use nix::unistd::{unlinkat, UnlinkatFlags};
-use pbs_s3_client::{S3Client, S3ClientOptions};
+use pbs_s3_client::{PutObjectResponse, S3Client, S3ClientOptions};
 use pbs_tools::lru_cache::LruCache;
 use tracing::{info, warn};
 
@@ -840,6 +840,22 @@ impl DataStore {
         backup_group: &pbs_api_types::BackupGroup,
     ) -> Result<Authid, Error> {
         let full_path = self.owner_path(ns, backup_group);
+
+        if let DatastoreBackend::S3(s3_client) = self.backend()? {
+            let object_key = full_path
+                .to_str()
+                .ok_or_else(|| format_err!("unexpected owner path"))?;
+            let response =
+                proxmox_async::runtime::block_on(s3_client.get_object(object_key.into()))?
+                    .ok_or_else(|| format_err!("fetching owner failed"))?;
+            let content =
+                proxmox_async::runtime::block_on(hyper::body::HttpBody::collect(response.content))?;
+            let owner = String::from_utf8(content.to_bytes().trim_ascii_end().to_vec())?;
+            return owner
+                .parse()
+                .map_err(|err| format_err!("parsing owner for {backup_group} failed: {err}"));
+        }
+
         let owner = proxmox_sys::fs::file_read_firstline(full_path)?;
         owner
             .trim_end() // remove trailing newline
@@ -868,6 +884,22 @@ impl DataStore {
     ) -> Result<(), Error> {
         let path = self.owner_path(ns, backup_group);
 
+        if let DatastoreBackend::S3(s3_client) = self.backend()? {
+            let object_key = path
+                .to_str()
+                .ok_or_else(|| format_err!("unexpected owner path"))?;
+            let data = hyper::body::Body::from(format!("{auth_id}\n"));
+            let response =
+                proxmox_async::runtime::block_on(s3_client.put_object(object_key.into(), data))?;
+            match response {
+                PutObjectResponse::NeedsRetry => bail!("failed to set owner, needs retry"),
+                PutObjectResponse::PreconditionFailed => {
+                    bail!("failed to set owner, precondition failed")
+                }
+                PutObjectResponse::Success(_) => return Ok(()),
+            }
+        }
+
         let mut open_options = std::fs::OpenOptions::new();
         open_options.write(true);
 
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:05 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:05 +0200
Subject: [pbs-devel] [RFC proxmox-backup 04/39] verify: refactor verify
 related functions to be methods of worker
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-5-c.ebner@proxmox.com>

Instead of passing the VerifyWorker state as reference to the various
verification related functions, implement them as methods or
associated functions of the VerifyWorker. This does not only make
their correlation more clear, but it also reduces the number of
function call parameters and improves readability.

No functional changes intended.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs    |  28 +-
 src/api2/backup/environment.rs |   7 +-
 src/backup/verify.rs           | 830 ++++++++++++++++-----------------
 src/server/verify_job.rs       |  12 +-
 4 files changed, 423 insertions(+), 454 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 392494488..7dc881ade 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -70,10 +70,7 @@ use proxmox_rest_server::{formatter, WorkerTask};
 
 use crate::api2::backup::optional_ns_param;
 use crate::api2::node::rrd::create_value_from_rrd;
-use crate::backup::{
-    check_ns_privs_full, verify_all_backups, verify_backup_dir, verify_backup_group, verify_filter,
-    ListAccessibleBackupGroups, NS_PRIVS_OK,
-};
+use crate::backup::{check_ns_privs_full, ListAccessibleBackupGroups, VerifyWorker, NS_PRIVS_OK};
 
 use crate::server::jobstate::{compute_schedule_status, Job, JobState};
 
@@ -896,14 +893,15 @@ pub fn verify(
         auth_id.to_string(),
         to_stdout,
         move |worker| {
-            let verify_worker = crate::backup::VerifyWorker::new(worker.clone(), datastore);
+            let verify_worker = VerifyWorker::new(worker.clone(), datastore);
             let failed_dirs = if let Some(backup_dir) = backup_dir {
                 let mut res = Vec::new();
-                if !verify_backup_dir(
-                    &verify_worker,
+                if !verify_worker.verify_backup_dir(
                     &backup_dir,
                     worker.upid().clone(),
-                    Some(&move |manifest| verify_filter(ignore_verified, outdated_after, manifest)),
+                    Some(&move |manifest| {
+                        VerifyWorker::verify_filter(ignore_verified, outdated_after, manifest)
+                    }),
                 )? {
                     res.push(print_ns_and_snapshot(
                         backup_dir.backup_ns(),
@@ -912,12 +910,13 @@ pub fn verify(
                 }
                 res
             } else if let Some(backup_group) = backup_group {
-                verify_backup_group(
-                    &verify_worker,
+                verify_worker.verify_backup_group(
                     &backup_group,
                     &mut StoreProgress::new(1),
                     worker.upid(),
-                    Some(&move |manifest| verify_filter(ignore_verified, outdated_after, manifest)),
+                    Some(&move |manifest| {
+                        VerifyWorker::verify_filter(ignore_verified, outdated_after, manifest)
+                    }),
                 )?
             } else {
                 let owner = if owner_check_required {
@@ -926,13 +925,14 @@ pub fn verify(
                     None
                 };
 
-                verify_all_backups(
-                    &verify_worker,
+                verify_worker.verify_all_backups(
                     worker.upid(),
                     ns,
                     max_depth,
                     owner,
-                    Some(&move |manifest| verify_filter(ignore_verified, outdated_after, manifest)),
+                    Some(&move |manifest| {
+                        VerifyWorker::verify_filter(ignore_verified, outdated_after, manifest)
+                    }),
                 )?
             };
             if !failed_dirs.is_empty() {
diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 3d541b461..6cd29f512 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -18,7 +18,7 @@ use pbs_datastore::fixed_index::FixedIndexWriter;
 use pbs_datastore::{DataBlob, DataStore};
 use proxmox_rest_server::{formatter::*, WorkerTask};
 
-use crate::backup::verify_backup_dir_with_lock;
+use crate::backup::VerifyWorker;
 
 use hyper::{Body, Response};
 
@@ -671,9 +671,8 @@ impl BackupEnvironment {
             move |worker| {
                 worker.log_message("Automatically verifying newly added snapshot");
 
-                let verify_worker = crate::backup::VerifyWorker::new(worker.clone(), datastore);
-                if !verify_backup_dir_with_lock(
-                    &verify_worker,
+                let verify_worker = VerifyWorker::new(worker.clone(), datastore);
+                if !verify_worker.verify_backup_dir_with_lock(
                     &backup_dir,
                     worker.upid().clone(),
                     None,
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 3d2cba8ac..0b954ae23 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -44,517 +44,491 @@ impl VerifyWorker {
             corrupt_chunks: Arc::new(Mutex::new(HashSet::with_capacity(64))),
         }
     }
-}
-
-fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
-    let blob = backup_dir.load_blob(&info.filename)?;
 
-    let raw_size = blob.raw_size();
-    if raw_size != info.size {
-        bail!("wrong size ({} != {})", info.size, raw_size);
-    }
-
-    let csum = openssl::sha::sha256(blob.raw_data());
-    if csum != info.csum {
-        bail!("wrong index checksum");
-    }
+    fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
+        let blob = backup_dir.load_blob(&info.filename)?;
 
-    match blob.crypt_mode()? {
-        CryptMode::Encrypt => Ok(()),
-        CryptMode::None => {
-            // digest already verified above
-            blob.decode(None, None)?;
-            Ok(())
+        let raw_size = blob.raw_size();
+        if raw_size != info.size {
+            bail!("wrong size ({} != {})", info.size, raw_size);
         }
-        CryptMode::SignOnly => bail!("Invalid CryptMode for blob"),
-    }
-}
-
-fn rename_corrupted_chunk(datastore: Arc<DataStore>, digest: &[u8; 32]) {
-    let (path, digest_str) = datastore.chunk_path(digest);
 
-    let mut counter = 0;
-    let mut new_path = path.clone();
-    loop {
-        new_path.set_file_name(format!("{}.{}.bad", digest_str, counter));
-        if new_path.exists() && counter < 9 {
-            counter += 1;
-        } else {
-            break;
+        let csum = openssl::sha::sha256(blob.raw_data());
+        if csum != info.csum {
+            bail!("wrong index checksum");
         }
-    }
 
-    match std::fs::rename(&path, &new_path) {
-        Ok(_) => {
-            info!("corrupted chunk renamed to {:?}", &new_path);
-        }
-        Err(err) => {
-            match err.kind() {
-                std::io::ErrorKind::NotFound => { /* ignored */ }
-                _ => info!("could not rename corrupted chunk {:?} - {err}", &path),
+        match blob.crypt_mode()? {
+            CryptMode::Encrypt => Ok(()),
+            CryptMode::None => {
+                // digest already verified above
+                blob.decode(None, None)?;
+                Ok(())
             }
+            CryptMode::SignOnly => bail!("Invalid CryptMode for blob"),
         }
-    };
-}
+    }
 
-fn verify_index_chunks(
-    verify_worker: &VerifyWorker,
-    index: Box<dyn IndexFile + Send>,
-    crypt_mode: CryptMode,
-) -> Result<(), Error> {
-    let errors = Arc::new(AtomicUsize::new(0));
+    fn rename_corrupted_chunk(datastore: Arc<DataStore>, digest: &[u8; 32]) {
+        let (path, digest_str) = datastore.chunk_path(digest);
 
-    let start_time = Instant::now();
+        let mut counter = 0;
+        let mut new_path = path.clone();
+        loop {
+            new_path.set_file_name(format!("{}.{}.bad", digest_str, counter));
+            if new_path.exists() && counter < 9 {
+                counter += 1;
+            } else {
+                break;
+            }
+        }
 
-    let mut read_bytes = 0;
-    let mut decoded_bytes = 0;
+        match std::fs::rename(&path, &new_path) {
+            Ok(_) => {
+                info!("corrupted chunk renamed to {:?}", &new_path);
+            }
+            Err(err) => {
+                match err.kind() {
+                    std::io::ErrorKind::NotFound => { /* ignored */ }
+                    _ => info!("could not rename corrupted chunk {:?} - {err}", &path),
+                }
+            }
+        };
+    }
 
-    let datastore2 = Arc::clone(&verify_worker.datastore);
-    let corrupt_chunks2 = Arc::clone(&verify_worker.corrupt_chunks);
-    let verified_chunks2 = Arc::clone(&verify_worker.verified_chunks);
-    let errors2 = Arc::clone(&errors);
+    fn verify_index_chunks(
+        &self,
+        index: Box<dyn IndexFile + Send>,
+        crypt_mode: CryptMode,
+    ) -> Result<(), Error> {
+        let errors = Arc::new(AtomicUsize::new(0));
+
+        let start_time = Instant::now();
+
+        let mut read_bytes = 0;
+        let mut decoded_bytes = 0;
+
+        let datastore2 = Arc::clone(&self.datastore);
+        let corrupt_chunks2 = Arc::clone(&self.corrupt_chunks);
+        let verified_chunks2 = Arc::clone(&self.verified_chunks);
+        let errors2 = Arc::clone(&errors);
+
+        let decoder_pool = ParallelHandler::new(
+            "verify chunk decoder",
+            4,
+            move |(chunk, digest, size): (DataBlob, [u8; 32], u64)| {
+                let chunk_crypt_mode = match chunk.crypt_mode() {
+                    Err(err) => {
+                        corrupt_chunks2.lock().unwrap().insert(digest);
+                        info!("can't verify chunk, unknown CryptMode - {err}");
+                        errors2.fetch_add(1, Ordering::SeqCst);
+                        return Ok(());
+                    }
+                    Ok(mode) => mode,
+                };
+
+                if chunk_crypt_mode != crypt_mode {
+                    info!(
+                    "chunk CryptMode {chunk_crypt_mode:?} does not match index CryptMode {crypt_mode:?}"
+                );
+                    errors2.fetch_add(1, Ordering::SeqCst);
+                }
 
-    let decoder_pool = ParallelHandler::new(
-        "verify chunk decoder",
-        4,
-        move |(chunk, digest, size): (DataBlob, [u8; 32], u64)| {
-            let chunk_crypt_mode = match chunk.crypt_mode() {
-                Err(err) => {
+                if let Err(err) = chunk.verify_unencrypted(size as usize, &digest) {
                     corrupt_chunks2.lock().unwrap().insert(digest);
-                    info!("can't verify chunk, unknown CryptMode - {err}");
+                    info!("{err}");
                     errors2.fetch_add(1, Ordering::SeqCst);
-                    return Ok(());
+                    Self::rename_corrupted_chunk(datastore2.clone(), &digest);
+                } else {
+                    verified_chunks2.lock().unwrap().insert(digest);
                 }
-                Ok(mode) => mode,
-            };
 
-            if chunk_crypt_mode != crypt_mode {
-                info!(
-                    "chunk CryptMode {chunk_crypt_mode:?} does not match index CryptMode {crypt_mode:?}"
-                );
-                errors2.fetch_add(1, Ordering::SeqCst);
-            }
+                Ok(())
+            },
+        );
 
-            if let Err(err) = chunk.verify_unencrypted(size as usize, &digest) {
-                corrupt_chunks2.lock().unwrap().insert(digest);
-                info!("{err}");
-                errors2.fetch_add(1, Ordering::SeqCst);
-                rename_corrupted_chunk(datastore2.clone(), &digest);
+        let skip_chunk = |digest: &[u8; 32]| -> bool {
+            if self.verified_chunks.lock().unwrap().contains(digest) {
+                true
+            } else if self.corrupt_chunks.lock().unwrap().contains(digest) {
+                let digest_str = hex::encode(digest);
+                info!("chunk {digest_str} was marked as corrupt");
+                errors.fetch_add(1, Ordering::SeqCst);
+                true
             } else {
-                verified_chunks2.lock().unwrap().insert(digest);
+                false
             }
+        };
 
+        let check_abort = |pos: usize| -> Result<(), Error> {
+            if pos & 1023 == 0 {
+                self.worker.check_abort()?;
+                self.worker.fail_on_shutdown()?;
+            }
             Ok(())
-        },
-    );
-
-    let skip_chunk = |digest: &[u8; 32]| -> bool {
-        if verify_worker
-            .verified_chunks
-            .lock()
-            .unwrap()
-            .contains(digest)
-        {
-            true
-        } else if verify_worker
-            .corrupt_chunks
-            .lock()
-            .unwrap()
-            .contains(digest)
-        {
-            let digest_str = hex::encode(digest);
-            info!("chunk {digest_str} was marked as corrupt");
-            errors.fetch_add(1, Ordering::SeqCst);
-            true
-        } else {
-            false
-        }
-    };
-
-    let check_abort = |pos: usize| -> Result<(), Error> {
-        if pos & 1023 == 0 {
-            verify_worker.worker.check_abort()?;
-            verify_worker.worker.fail_on_shutdown()?;
-        }
-        Ok(())
-    };
+        };
 
-    let chunk_list =
-        verify_worker
+        let chunk_list = self
             .datastore
             .get_chunks_in_order(&*index, skip_chunk, check_abort)?;
 
-    for (pos, _) in chunk_list {
-        verify_worker.worker.check_abort()?;
-        verify_worker.worker.fail_on_shutdown()?;
+        for (pos, _) in chunk_list {
+            self.worker.check_abort()?;
+            self.worker.fail_on_shutdown()?;
 
-        let info = index.chunk_info(pos).unwrap();
+            let info = index.chunk_info(pos).unwrap();
 
-        // we must always recheck this here, the parallel worker below alter it!
-        if skip_chunk(&info.digest) {
-            continue; // already verified or marked corrupt
-        }
-
-        match verify_worker.datastore.load_chunk(&info.digest) {
-            Err(err) => {
-                verify_worker
-                    .corrupt_chunks
-                    .lock()
-                    .unwrap()
-                    .insert(info.digest);
-                error!("can't verify chunk, load failed - {err}");
-                errors.fetch_add(1, Ordering::SeqCst);
-                rename_corrupted_chunk(verify_worker.datastore.clone(), &info.digest);
+            // we must always recheck this here, the parallel worker below alter it!
+            if skip_chunk(&info.digest) {
+                continue; // already verified or marked corrupt
             }
-            Ok(chunk) => {
-                let size = info.size();
-                read_bytes += chunk.raw_size();
-                decoder_pool.send((chunk, info.digest, size))?;
-                decoded_bytes += size;
+
+            match self.datastore.load_chunk(&info.digest) {
+                Err(err) => {
+                    self.corrupt_chunks.lock().unwrap().insert(info.digest);
+                    error!("can't verify chunk, load failed - {err}");
+                    errors.fetch_add(1, Ordering::SeqCst);
+                    Self::rename_corrupted_chunk(self.datastore.clone(), &info.digest);
+                }
+                Ok(chunk) => {
+                    let size = info.size();
+                    read_bytes += chunk.raw_size();
+                    decoder_pool.send((chunk, info.digest, size))?;
+                    decoded_bytes += size;
+                }
             }
         }
-    }
 
-    decoder_pool.complete()?;
+        decoder_pool.complete()?;
+
+        let elapsed = start_time.elapsed().as_secs_f64();
 
-    let elapsed = start_time.elapsed().as_secs_f64();
+        let read_bytes_mib = (read_bytes as f64) / (1024.0 * 1024.0);
+        let decoded_bytes_mib = (decoded_bytes as f64) / (1024.0 * 1024.0);
 
-    let read_bytes_mib = (read_bytes as f64) / (1024.0 * 1024.0);
-    let decoded_bytes_mib = (decoded_bytes as f64) / (1024.0 * 1024.0);
+        let read_speed = read_bytes_mib / elapsed;
+        let decode_speed = decoded_bytes_mib / elapsed;
 
-    let read_speed = read_bytes_mib / elapsed;
-    let decode_speed = decoded_bytes_mib / elapsed;
+        let error_count = errors.load(Ordering::SeqCst);
 
-    let error_count = errors.load(Ordering::SeqCst);
+        info!(
+            "  verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors)"
+        );
 
-    info!(
-        "  verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors)"
-    );
+        if errors.load(Ordering::SeqCst) > 0 {
+            bail!("chunks could not be verified");
+        }
 
-    if errors.load(Ordering::SeqCst) > 0 {
-        bail!("chunks could not be verified");
+        Ok(())
     }
 
-    Ok(())
-}
+    fn verify_fixed_index(&self, backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
+        let mut path = backup_dir.relative_path();
+        path.push(&info.filename);
 
-fn verify_fixed_index(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
-    info: &FileInfo,
-) -> Result<(), Error> {
-    let mut path = backup_dir.relative_path();
-    path.push(&info.filename);
+        let index = self.datastore.open_fixed_reader(&path)?;
 
-    let index = verify_worker.datastore.open_fixed_reader(&path)?;
+        let (csum, size) = index.compute_csum();
+        if size != info.size {
+            bail!("wrong size ({} != {})", info.size, size);
+        }
 
-    let (csum, size) = index.compute_csum();
-    if size != info.size {
-        bail!("wrong size ({} != {})", info.size, size);
-    }
+        if csum != info.csum {
+            bail!("wrong index checksum");
+        }
 
-    if csum != info.csum {
-        bail!("wrong index checksum");
+        self.verify_index_chunks(Box::new(index), info.chunk_crypt_mode())
     }
 
-    verify_index_chunks(verify_worker, Box::new(index), info.chunk_crypt_mode())
-}
-
-fn verify_dynamic_index(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
-    info: &FileInfo,
-) -> Result<(), Error> {
-    let mut path = backup_dir.relative_path();
-    path.push(&info.filename);
+    fn verify_dynamic_index(&self, backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
+        let mut path = backup_dir.relative_path();
+        path.push(&info.filename);
 
-    let index = verify_worker.datastore.open_dynamic_reader(&path)?;
+        let index = self.datastore.open_dynamic_reader(&path)?;
 
-    let (csum, size) = index.compute_csum();
-    if size != info.size {
-        bail!("wrong size ({} != {})", info.size, size);
-    }
-
-    if csum != info.csum {
-        bail!("wrong index checksum");
-    }
+        let (csum, size) = index.compute_csum();
+        if size != info.size {
+            bail!("wrong size ({} != {})", info.size, size);
+        }
 
-    verify_index_chunks(verify_worker, Box::new(index), info.chunk_crypt_mode())
-}
+        if csum != info.csum {
+            bail!("wrong index checksum");
+        }
 
-/// Verify a single backup snapshot
-///
-/// This checks all archives inside a backup snapshot.
-/// Errors are logged to the worker log.
-///
-/// Returns
-/// - Ok(true) if verify is successful
-/// - Ok(false) if there were verification errors
-/// - Err(_) if task was aborted
-pub fn verify_backup_dir(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
-    upid: UPID,
-    filter: Option<&dyn Fn(&BackupManifest) -> bool>,
-) -> Result<bool, Error> {
-    if !backup_dir.full_path().exists() {
-        info!(
-            "SKIPPED: verify {}:{} - snapshot does not exist (anymore).",
-            verify_worker.datastore.name(),
-            backup_dir.dir(),
-        );
-        return Ok(true);
+        self.verify_index_chunks(Box::new(index), info.chunk_crypt_mode())
     }
 
-    let snap_lock = backup_dir.lock_shared();
-
-    match snap_lock {
-        Ok(snap_lock) => {
-            verify_backup_dir_with_lock(verify_worker, backup_dir, upid, filter, snap_lock)
-        }
-        Err(err) => {
+    /// Verify a single backup snapshot
+    ///
+    /// This checks all archives inside a backup snapshot.
+    /// Errors are logged to the worker log.
+    ///
+    /// Returns
+    /// - Ok(true) if verify is successful
+    /// - Ok(false) if there were verification errors
+    /// - Err(_) if task was aborted
+    pub fn verify_backup_dir(
+        &self,
+        backup_dir: &BackupDir,
+        upid: UPID,
+        filter: Option<&dyn Fn(&BackupManifest) -> bool>,
+    ) -> Result<bool, Error> {
+        if !backup_dir.full_path().exists() {
             info!(
-                "SKIPPED: verify {}:{} - could not acquire snapshot lock: {}",
-                verify_worker.datastore.name(),
+                "SKIPPED: verify {}:{} - snapshot does not exist (anymore).",
+                self.datastore.name(),
                 backup_dir.dir(),
-                err,
             );
-            Ok(true)
+            return Ok(true);
         }
-    }
-}
 
-/// See verify_backup_dir
-pub fn verify_backup_dir_with_lock(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
-    upid: UPID,
-    filter: Option<&dyn Fn(&BackupManifest) -> bool>,
-    _snap_lock: BackupLockGuard,
-) -> Result<bool, Error> {
-    let datastore_name = verify_worker.datastore.name();
-    let backup_dir_name = backup_dir.dir();
-
-    let manifest = match backup_dir.load_manifest() {
-        Ok((manifest, _)) => manifest,
-        Err(err) => {
-            info!("verify {datastore_name}:{backup_dir_name} - manifest load error: {err}");
-            return Ok(false);
-        }
-    };
+        let snap_lock = backup_dir.lock_shared();
 
-    if let Some(filter) = filter {
-        if !filter(&manifest) {
-            info!("SKIPPED: verify {datastore_name}:{backup_dir_name} (recently verified)");
-            return Ok(true);
+        match snap_lock {
+            Ok(snap_lock) => self.verify_backup_dir_with_lock(backup_dir, upid, filter, snap_lock),
+            Err(err) => {
+                info!(
+                    "SKIPPED: verify {}:{} - could not acquire snapshot lock: {}",
+                    self.datastore.name(),
+                    backup_dir.dir(),
+                    err,
+                );
+                Ok(true)
+            }
         }
     }
 
-    info!("verify {datastore_name}:{backup_dir_name}");
-
-    let mut error_count = 0;
+    /// See verify_backup_dir
+    pub fn verify_backup_dir_with_lock(
+        &self,
+        backup_dir: &BackupDir,
+        upid: UPID,
+        filter: Option<&dyn Fn(&BackupManifest) -> bool>,
+        _snap_lock: BackupLockGuard,
+    ) -> Result<bool, Error> {
+        let datastore_name = self.datastore.name();
+        let backup_dir_name = backup_dir.dir();
+
+        let manifest = match backup_dir.load_manifest() {
+            Ok((manifest, _)) => manifest,
+            Err(err) => {
+                info!("verify {datastore_name}:{backup_dir_name} - manifest load error: {err}");
+                return Ok(false);
+            }
+        };
 
-    let mut verify_result = VerifyState::Ok;
-    for info in manifest.files() {
-        let result = proxmox_lang::try_block!({
-            info!("  check {}", info.filename);
-            match ArchiveType::from_path(&info.filename)? {
-                ArchiveType::FixedIndex => verify_fixed_index(verify_worker, backup_dir, info),
-                ArchiveType::DynamicIndex => verify_dynamic_index(verify_worker, backup_dir, info),
-                ArchiveType::Blob => verify_blob(backup_dir, info),
+        if let Some(filter) = filter {
+            if !filter(&manifest) {
+                info!("SKIPPED: verify {datastore_name}:{backup_dir_name} (recently verified)");
+                return Ok(true);
             }
-        });
+        }
 
-        verify_worker.worker.check_abort()?;
-        verify_worker.worker.fail_on_shutdown()?;
+        info!("verify {datastore_name}:{backup_dir_name}");
 
-        if let Err(err) = result {
-            info!(
-                "verify {datastore_name}:{backup_dir_name}/{file_name} failed: {err}",
-                file_name = info.filename,
-            );
-            error_count += 1;
-            verify_result = VerifyState::Failed;
-        }
-    }
+        let mut error_count = 0;
 
-    let verify_state = SnapshotVerifyState {
-        state: verify_result,
-        upid,
-    };
-
-    if let Err(err) = {
-        let verify_state = serde_json::to_value(verify_state)?;
-        backup_dir.update_manifest(|manifest| {
-            manifest.unprotected["verify_state"] = verify_state;
-        })
-    } {
-        info!("verify {datastore_name}:{backup_dir_name} - manifest update error: {err}");
-        return Ok(false);
-    }
+        let mut verify_result = VerifyState::Ok;
+        for info in manifest.files() {
+            let result = proxmox_lang::try_block!({
+                info!("  check {}", info.filename);
+                match ArchiveType::from_path(&info.filename)? {
+                    ArchiveType::FixedIndex => self.verify_fixed_index(backup_dir, info),
+                    ArchiveType::DynamicIndex => self.verify_dynamic_index(backup_dir, info),
+                    ArchiveType::Blob => Self::verify_blob(backup_dir, info),
+                }
+            });
 
-    Ok(error_count == 0)
-}
+            self.worker.check_abort()?;
+            self.worker.fail_on_shutdown()?;
 
-/// Verify all backups inside a backup group
-///
-/// Errors are logged to the worker log.
-///
-/// Returns
-/// - Ok((count, failed_dirs)) where failed_dirs had verification errors
-/// - Err(_) if task was aborted
-pub fn verify_backup_group(
-    verify_worker: &VerifyWorker,
-    group: &BackupGroup,
-    progress: &mut StoreProgress,
-    upid: &UPID,
-    filter: Option<&dyn Fn(&BackupManifest) -> bool>,
-) -> Result<Vec<String>, Error> {
-    let mut errors = Vec::new();
-    let mut list = match group.list_backups() {
-        Ok(list) => list,
-        Err(err) => {
-            info!(
-                "verify {}, group {} - unable to list backups: {}",
-                print_store_and_ns(verify_worker.datastore.name(), group.backup_ns()),
-                group.group(),
-                err,
-            );
-            return Ok(errors);
-        }
-    };
-
-    let snapshot_count = list.len();
-    info!(
-        "verify group {}:{} ({} snapshots)",
-        verify_worker.datastore.name(),
-        group.group(),
-        snapshot_count
-    );
-
-    progress.group_snapshots = snapshot_count as u64;
-
-    BackupInfo::sort_list(&mut list, false); // newest first
-    for (pos, info) in list.into_iter().enumerate() {
-        if !verify_backup_dir(verify_worker, &info.backup_dir, upid.clone(), filter)? {
-            errors.push(print_ns_and_snapshot(
-                info.backup_dir.backup_ns(),
-                info.backup_dir.as_ref(),
-            ));
+            if let Err(err) = result {
+                info!(
+                    "verify {datastore_name}:{backup_dir_name}/{file_name} failed: {err}",
+                    file_name = info.filename,
+                );
+                error_count += 1;
+                verify_result = VerifyState::Failed;
+            }
         }
-        progress.done_snapshots = pos as u64 + 1;
-        info!("percentage done: {progress}");
-    }
-    Ok(errors)
-}
 
-/// Verify all (owned) backups inside a datastore
-///
-/// Errors are logged to the worker log.
-///
-/// Returns
-/// - Ok(failed_dirs) where failed_dirs had verification errors
-/// - Err(_) if task was aborted
-pub fn verify_all_backups(
-    verify_worker: &VerifyWorker,
-    upid: &UPID,
-    ns: BackupNamespace,
-    max_depth: Option<usize>,
-    owner: Option<&Authid>,
-    filter: Option<&dyn Fn(&BackupManifest) -> bool>,
-) -> Result<Vec<String>, Error> {
-    let mut errors = Vec::new();
-
-    info!("verify datastore {}", verify_worker.datastore.name());
-
-    let owner_filtered = if let Some(owner) = &owner {
-        info!("limiting to backups owned by {owner}");
-        true
-    } else {
-        false
-    };
-
-    // FIXME: This should probably simply enable recursion (or the call have a recursion parameter)
-    let store = &verify_worker.datastore;
-    let max_depth = max_depth.unwrap_or(pbs_api_types::MAX_NAMESPACE_DEPTH);
-
-    let mut list = match ListAccessibleBackupGroups::new_with_privs(
-        store,
-        ns.clone(),
-        max_depth,
-        Some(PRIV_DATASTORE_VERIFY),
-        Some(PRIV_DATASTORE_BACKUP),
-        owner,
-    ) {
-        Ok(list) => list
-            .filter_map(|group| match group {
-                Ok(group) => Some(group),
-                Err(err) if owner_filtered => {
-                    // intentionally not in task log, the user might not see this group!
-                    println!("error on iterating groups in ns '{ns}' - {err}");
-                    None
-                }
-                Err(err) => {
-                    // we don't filter by owner, but we want to log the error
-                    info!("error on iterating groups in ns '{ns}' - {err}");
-                    errors.push(err.to_string());
-                    None
-                }
-            })
-            .filter(|group| {
-                !(group.backup_type() == BackupType::Host && group.backup_id() == "benchmark")
+        let verify_state = SnapshotVerifyState {
+            state: verify_result,
+            upid,
+        };
+
+        if let Err(err) = {
+            let verify_state = serde_json::to_value(verify_state)?;
+            backup_dir.update_manifest(|manifest| {
+                manifest.unprotected["verify_state"] = verify_state;
             })
-            .collect::<Vec<BackupGroup>>(),
-        Err(err) => {
-            info!("unable to list backups: {err}");
-            return Ok(errors);
+        } {
+            info!("verify {datastore_name}:{backup_dir_name} - manifest update error: {err}");
+            return Ok(false);
         }
-    };
 
-    list.sort_unstable_by(|a, b| a.group().cmp(b.group()));
+        Ok(error_count == 0)
+    }
 
-    let group_count = list.len();
-    info!("found {group_count} groups");
+    /// Verify all backups inside a backup group
+    ///
+    /// Errors are logged to the worker log.
+    ///
+    /// Returns
+    /// - Ok((count, failed_dirs)) where failed_dirs had verification errors
+    /// - Err(_) if task was aborted
+    pub fn verify_backup_group(
+        &self,
+        group: &BackupGroup,
+        progress: &mut StoreProgress,
+        upid: &UPID,
+        filter: Option<&dyn Fn(&BackupManifest) -> bool>,
+    ) -> Result<Vec<String>, Error> {
+        let mut errors = Vec::new();
+        let mut list = match group.list_backups() {
+            Ok(list) => list,
+            Err(err) => {
+                info!(
+                    "verify {}, group {} - unable to list backups: {}",
+                    print_store_and_ns(self.datastore.name(), group.backup_ns()),
+                    group.group(),
+                    err,
+                );
+                return Ok(errors);
+            }
+        };
 
-    let mut progress = StoreProgress::new(group_count as u64);
+        let snapshot_count = list.len();
+        info!(
+            "verify group {}:{} ({} snapshots)",
+            self.datastore.name(),
+            group.group(),
+            snapshot_count
+        );
 
-    for (pos, group) in list.into_iter().enumerate() {
-        progress.done_groups = pos as u64;
-        progress.done_snapshots = 0;
-        progress.group_snapshots = 0;
+        progress.group_snapshots = snapshot_count as u64;
 
-        let mut group_errors =
-            verify_backup_group(verify_worker, &group, &mut progress, upid, filter)?;
-        errors.append(&mut group_errors);
+        BackupInfo::sort_list(&mut list, false); // newest first
+        for (pos, info) in list.into_iter().enumerate() {
+            if !self.verify_backup_dir(&info.backup_dir, upid.clone(), filter)? {
+                errors.push(print_ns_and_snapshot(
+                    info.backup_dir.backup_ns(),
+                    info.backup_dir.as_ref(),
+                ));
+            }
+            progress.done_snapshots = pos as u64 + 1;
+            info!("percentage done: {progress}");
+        }
+        Ok(errors)
     }
 
-    Ok(errors)
-}
+    /// Verify all (owned) backups inside a datastore
+    ///
+    /// Errors are logged to the worker log.
+    ///
+    /// Returns
+    /// - Ok(failed_dirs) where failed_dirs had verification errors
+    /// - Err(_) if task was aborted
+    pub fn verify_all_backups(
+        &self,
+        upid: &UPID,
+        ns: BackupNamespace,
+        max_depth: Option<usize>,
+        owner: Option<&Authid>,
+        filter: Option<&dyn Fn(&BackupManifest) -> bool>,
+    ) -> Result<Vec<String>, Error> {
+        let mut errors = Vec::new();
+
+        info!("verify datastore {}", self.datastore.name());
+
+        let owner_filtered = if let Some(owner) = &owner {
+            info!("limiting to backups owned by {owner}");
+            true
+        } else {
+            false
+        };
+
+        // FIXME: This should probably simply enable recursion (or the call have a recursion parameter)
+        let store = &self.datastore;
+        let max_depth = max_depth.unwrap_or(pbs_api_types::MAX_NAMESPACE_DEPTH);
+
+        let mut list = match ListAccessibleBackupGroups::new_with_privs(
+            store,
+            ns.clone(),
+            max_depth,
+            Some(PRIV_DATASTORE_VERIFY),
+            Some(PRIV_DATASTORE_BACKUP),
+            owner,
+        ) {
+            Ok(list) => list
+                .filter_map(|group| match group {
+                    Ok(group) => Some(group),
+                    Err(err) if owner_filtered => {
+                        // intentionally not in task log, the user might not see this group!
+                        println!("error on iterating groups in ns '{ns}' - {err}");
+                        None
+                    }
+                    Err(err) => {
+                        // we don't filter by owner, but we want to log the error
+                        info!("error on iterating groups in ns '{ns}' - {err}");
+                        errors.push(err.to_string());
+                        None
+                    }
+                })
+                .filter(|group| {
+                    !(group.backup_type() == BackupType::Host && group.backup_id() == "benchmark")
+                })
+                .collect::<Vec<BackupGroup>>(),
+            Err(err) => {
+                info!("unable to list backups: {err}");
+                return Ok(errors);
+            }
+        };
+
+        list.sort_unstable_by(|a, b| a.group().cmp(b.group()));
+
+        let group_count = list.len();
+        info!("found {group_count} groups");
 
-/// Filter out any snapshot from being (re-)verified where this fn returns false.
-pub fn verify_filter(
-    ignore_verified_snapshots: bool,
-    outdated_after: Option<i64>,
-    manifest: &BackupManifest,
-) -> bool {
-    if !ignore_verified_snapshots {
-        return true;
+        let mut progress = StoreProgress::new(group_count as u64);
+
+        for (pos, group) in list.into_iter().enumerate() {
+            progress.done_groups = pos as u64;
+            progress.done_snapshots = 0;
+            progress.group_snapshots = 0;
+
+            let mut group_errors = self.verify_backup_group(&group, &mut progress, upid, filter)?;
+            errors.append(&mut group_errors);
+        }
+
+        Ok(errors)
     }
 
-    match manifest.verify_state() {
-        Err(err) => {
-            warn!("error reading manifest: {err:#}");
-            true
+    /// Filter out any snapshot from being (re-)verified where this fn returns false.
+    pub fn verify_filter(
+        ignore_verified_snapshots: bool,
+        outdated_after: Option<i64>,
+        manifest: &BackupManifest,
+    ) -> bool {
+        if !ignore_verified_snapshots {
+            return true;
         }
-        Ok(None) => true, // no last verification, always include
-        Ok(Some(last_verify)) => {
-            match outdated_after {
-                None => false, // never re-verify if ignored and no max age
-                Some(max_age) => {
-                    let now = proxmox_time::epoch_i64();
-                    let days_since_last_verify = (now - last_verify.upid.starttime) / 86400;
-
-                    days_since_last_verify > max_age
+
+        match manifest.verify_state() {
+            Err(err) => {
+                warn!("error reading manifest: {err:#}");
+                true
+            }
+            Ok(None) => true, // no last verification, always include
+            Ok(Some(last_verify)) => {
+                match outdated_after {
+                    None => false, // never re-verify if ignored and no max age
+                    Some(max_age) => {
+                        let now = proxmox_time::epoch_i64();
+                        let days_since_last_verify = (now - last_verify.upid.starttime) / 86400;
+
+                        days_since_last_verify > max_age
+                    }
                 }
             }
         }
diff --git a/src/server/verify_job.rs b/src/server/verify_job.rs
index a15a257da..95a7b2a9b 100644
--- a/src/server/verify_job.rs
+++ b/src/server/verify_job.rs
@@ -5,10 +5,7 @@ use pbs_api_types::{Authid, Operation, VerificationJobConfig};
 use pbs_datastore::DataStore;
 use proxmox_rest_server::WorkerTask;
 
-use crate::{
-    backup::{verify_all_backups, verify_filter},
-    server::jobstate::Job,
-};
+use crate::{backup::VerifyWorker, server::jobstate::Job};
 
 /// Runs a verification job.
 pub fn do_verification_job(
@@ -44,15 +41,14 @@ pub fn do_verification_job(
                 None => Default::default(),
             };
 
-            let verify_worker = crate::backup::VerifyWorker::new(worker.clone(), datastore);
-            let result = verify_all_backups(
-                &verify_worker,
+            let verify_worker = VerifyWorker::new(worker.clone(), datastore);
+            let result = verify_worker.verify_all_backups(
                 worker.upid(),
                 ns,
                 verification_job.max_depth,
                 None,
                 Some(&move |manifest| {
-                    verify_filter(ignore_verified_snapshots, outdated_after, manifest)
+                    VerifyWorker::verify_filter(ignore_verified_snapshots, outdated_after, manifest)
                 }),
             );
             let job_result = match result {
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:32 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:32 +0200
Subject: [pbs-devel] [RFC proxmox-backup 31/39] api/bin: add endpoint and
 command to test s3 backend for datastore
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-32-c.ebner@proxmox.com>

Adds a dedicated endpoint and a proxmox-backup-manager command to test
access to the S3 backend for a datastore configured as such.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs                 | 84 +++++++++++++++++++--
 src/bin/proxmox_backup_manager/datastore.rs | 24 ++++++
 2 files changed, 100 insertions(+), 8 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 45204369a..1e6b10f51 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -40,14 +40,14 @@ use pbs_api_types::{
     print_ns_and_snapshot, print_store_and_ns, ArchiveType, Authid, BackupArchiveName,
     BackupContent, BackupGroupDeleteStats, BackupNamespace, BackupType, Counts, CryptMode,
     DataStoreConfig, DataStoreListItem, DataStoreMountStatus, DataStoreStatus,
-    GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
-    MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
-    BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
-    BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
-    IGNORE_VERIFIED_BACKUPS_SCHEMA, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
-    PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
-    PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, PRIV_SYS_MODIFY, UPID, UPID_SCHEMA,
-    VERIFICATION_OUTDATED_AFTER_SCHEMA,
+    DatastoreBackendConfig, GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus,
+    KeepOptions, MaintenanceMode, MaintenanceType, Operation, PruneJobOptions, S3ClientConfig,
+    S3ClientSecretsConfig, SnapshotListItem, SnapshotVerifyState, BACKUP_ARCHIVE_NAME_SCHEMA,
+    BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA,
+    CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA, IGNORE_VERIFIED_BACKUPS_SCHEMA,
+    MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA, PRIV_DATASTORE_AUDIT,
+    PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE, PRIV_DATASTORE_READ,
+    PRIV_DATASTORE_VERIFY, PRIV_SYS_MODIFY, UPID, UPID_SCHEMA, VERIFICATION_OUTDATED_AFTER_SCHEMA,
 };
 use pbs_client::pxar::{create_tar, create_zip};
 use pbs_config::CachedUserInfo;
@@ -2708,6 +2708,70 @@ pub async fn unmount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<V
     Ok(json!(upid))
 }
 
+#[api(
+    input: {
+        properties: {
+            store: {
+                schema: DATASTORE_SCHEMA,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&["datastore", "{store}"], PRIV_DATASTORE_MODIFY, false),
+    },
+)]
+/// Check s3 backend for given datastore
+pub async fn s3_backend_check(
+    store: String,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Value, Error> {
+    let (section_config, _digest) = pbs_config::datastore::config()?;
+    let datastore: DataStoreConfig = section_config.lookup("datastore", &store)?;
+    let backend = datastore.backend.unwrap_or_default();
+
+    let client_id = match backend.parse()? {
+        DatastoreBackendConfig::S3(client_id) => client_id,
+        _ => bail!("datastore not of s3 backend type"),
+    };
+
+    let (config, _digest) = pbs_config::s3::config()?;
+    let config: S3ClientConfig = config.lookup("s3client", &client_id)?;
+    let (secrets, _secrets_digest) = pbs_config::s3::secrets_config()?;
+    let secrets: S3ClientSecretsConfig = secrets.lookup("s3secrets", &client_id)?;
+
+    let options = pbs_s3_client::S3ClientOptions {
+        host: config.host,
+        port: config.port,
+        bucket: config.bucket,
+        region: config.region.unwrap_or_default(),
+        fingerprint: config.fingerprint,
+        access_key: config.access_key,
+        secret_key: secrets.secret_key,
+    };
+    let client = pbs_s3_client::S3Client::new(options)?;
+
+    let object_path = "test.txt";
+    let object_data = "testtest".as_bytes().to_vec();
+
+    info!("HeadBucket: {:?}", client.head_bucket().await?);
+    info!(
+        "PutObject: {:?}",
+        client
+            .put_object(object_path.into(), hyper::Body::from(object_data))
+            .await?
+    );
+    info!(
+        "HeadObject: {:?}",
+        client.head_object(object_path.into()).await?
+    );
+    info!(
+        "GetObject: {:?}",
+        client.get_object(object_path.into()).await?
+    );
+
+    Ok(Value::Null)
+}
+
 #[sortable]
 const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
     (
@@ -2774,6 +2838,10 @@ const DATASTORE_INFO_SUBDIRS: SubdirMap = &[
         &Router::new().download(&API_METHOD_PXAR_FILE_DOWNLOAD),
     ),
     ("rrd", &Router::new().get(&API_METHOD_GET_RRD_STATS)),
+    (
+        "s3-backend-check",
+        &Router::new().get(&API_METHOD_S3_BACKEND_CHECK),
+    ),
     (
         "snapshots",
         &Router::new()
diff --git a/src/bin/proxmox_backup_manager/datastore.rs b/src/bin/proxmox_backup_manager/datastore.rs
index 1922a55a2..342284933 100644
--- a/src/bin/proxmox_backup_manager/datastore.rs
+++ b/src/bin/proxmox_backup_manager/datastore.rs
@@ -290,6 +290,24 @@ async fn uuid_mount(param: Value, _rpcenv: &mut dyn RpcEnvironment) -> Result<Va
     Ok(Value::Null)
 }
 
+#[api(
+    input: {
+        properties: {
+            name: {
+                schema: DATASTORE_SCHEMA,
+            },
+        },
+    },
+)]
+/// Check s3 backend for given datastore
+async fn s3_backend_check(name: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+    let result = api2::admin::datastore::s3_backend_check(name, rpcenv).await;
+
+    println!("Got: {result:#?}");
+
+    Ok(Value::Null)
+}
+
 pub fn datastore_commands() -> CommandLineInterface {
     let cmd_def = CliCommandMap::new()
         .insert("list", CliCommand::new(&API_METHOD_LIST_DATASTORES))
@@ -344,6 +362,12 @@ pub fn datastore_commands() -> CommandLineInterface {
             CliCommand::new(&API_METHOD_DELETE_DATASTORE)
                 .arg_param(&["name"])
                 .completion_cb("name", pbs_config::datastore::complete_datastore_name),
+        )
+        .insert(
+            "s3-backend-check",
+            CliCommand::new(&API_METHOD_S3_BACKEND_CHECK)
+                .arg_param(&["name"])
+                .completion_cb("name", pbs_config::datastore::complete_datastore_name),
         );
 
     cmd_def.into()
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:34 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:34 +0200
Subject: [pbs-devel] [RFC proxmox-backup 33/39] tools: async lru cache:
 implement insert, remove and contains methods
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-34-c.ebner@proxmox.com>

Add methods to insert new cache entries without using the cacher,
remove cache entries given their key and check if the cache contains
a key, marking it the most recently used one if it does.

These methods will be used to implement the local datastore cache
which stores the values (chunks) on the filesystem rather than
keeping track of them by storing them in-memory in the cache. The lru
cache will only be used to allow for fast lookup and keep track of
the lookup order.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-tools/src/async_lru_cache.rs | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/pbs-tools/src/async_lru_cache.rs b/pbs-tools/src/async_lru_cache.rs
index 141114933..3a975de32 100644
--- a/pbs-tools/src/async_lru_cache.rs
+++ b/pbs-tools/src/async_lru_cache.rs
@@ -87,6 +87,29 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V: Clone + Send + 'static> AsyncL
 
         result
     }
+
+    /// Insert an item as the most recently used one into the cache, calling the removed callback
+    /// on the evicted cache item, if any.
+    pub fn insert<F>(&self, key: K, value: V, removed: F) -> Result<(), Error>
+    where
+        F: Fn(K) -> Result<(), Error>,
+    {
+        let mut maps = self.maps.lock().unwrap();
+        maps.0.insert(key, value.clone(), removed)?;
+        Ok(())
+    }
+
+    /// Check if the item exists and if so, mark it as the most recently uses one.
+    pub fn contains(&self, key: K) -> bool {
+        let mut maps = self.maps.lock().unwrap();
+        maps.0.get_mut(key).is_some()
+    }
+
+    /// Remove the item from the cache.
+    pub fn remove(&self, key: K) {
+        let mut maps = self.maps.lock().unwrap();
+        maps.0.remove(key);
+    }
 }
 
 #[cfg(test)]
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:35 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:35 +0200
Subject: [pbs-devel] [RFC proxmox-backup 34/39] datastore: add local
 datastore cache for network attached storages
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-35-c.ebner@proxmox.com>

Use a local datastore as cache using LRU cache replacement policy for
operations on a datastore backed by a network, e.g. by an S3 object
store backend. The goal is to reduce number of requests to the
backend and thereby save costs (monetary as well as time).

The cacher allows to fetch cache items on cache misses via the access
method.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs                |  46 ++++++-
 pbs-datastore/src/lib.rs                      |   3 +
 .../src/local_datastore_lru_cache.rs          | 116 ++++++++++++++++++
 3 files changed, 164 insertions(+), 1 deletion(-)
 create mode 100644 pbs-datastore/src/local_datastore_lru_cache.rs

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 22ad566ca..19d4ca02c 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -37,8 +37,9 @@ use crate::dynamic_index::{DynamicIndexReader, DynamicIndexWriter};
 use crate::fixed_index::{FixedIndexReader, FixedIndexWriter};
 use crate::hierarchy::{ListGroups, ListGroupsType, ListNamespaces, ListNamespacesRecursive};
 use crate::index::IndexFile;
+use crate::local_datastore_lru_cache::S3Cacher;
 use crate::task_tracking::{self, update_active_operations};
-use crate::DataBlob;
+use crate::{DataBlob, LocalDatastoreLruCache};
 
 static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
     LazyLock::new(|| Mutex::new(HashMap::new()));
@@ -129,6 +130,7 @@ pub struct DataStoreImpl {
     last_digest: Option<[u8; 32]>,
     sync_level: DatastoreFSyncLevel,
     backend_config: DatastoreBackendConfig,
+    lru_store_caching: Option<LocalDatastoreLruCache>,
 }
 
 impl DataStoreImpl {
@@ -144,6 +146,7 @@ impl DataStoreImpl {
             last_digest: None,
             sync_level: Default::default(),
             backend_config: Default::default(),
+            lru_store_caching: None,
         })
     }
 }
@@ -241,6 +244,37 @@ impl DataStore {
         Ok(backend_type)
     }
 
+    pub fn cache(&self) -> Option<&LocalDatastoreLruCache> {
+        self.inner.lru_store_caching.as_ref()
+    }
+
+    /// Check if the digest is present in the local datastore cache.
+    /// Always returns false if there is no cache configured for this datastore.
+    pub fn cache_contains(&self, digest: &[u8; 32]) -> bool {
+        if let Some(cache) = self.inner.lru_store_caching.as_ref() {
+            return cache.contains(digest);
+        }
+        false
+    }
+
+    /// Insert digest as most recently used on in the cache.
+    /// Returns with success if there is no cache configured for this datastore.
+    pub fn cache_insert(&self, digest: &[u8; 32], chunk: &DataBlob) -> Result<(), Error> {
+        if let Some(cache) = self.inner.lru_store_caching.as_ref() {
+            return cache.insert(digest, chunk);
+        }
+        Ok(())
+    }
+
+    pub fn cacher(&self) -> Result<Option<S3Cacher>, Error> {
+        self.backend().map(|backend| match backend {
+            DatastoreBackend::S3(s3_client) => {
+                Some(S3Cacher::new(s3_client, self.inner.chunk_store.clone()))
+            }
+            DatastoreBackend::Filesystem => None,
+        })
+    }
+
     pub fn lookup_datastore(
         name: &str,
         operation: Option<Operation>,
@@ -423,6 +457,15 @@ impl DataStore {
             None => Default::default(),
         };
 
+        const LOCAL_DATASTORE_CACHE_SIZE: usize = 10_000_000;
+        let lru_store_caching = if let DatastoreBackendConfig::S3(_) = backend_config {
+            let cache =
+                LocalDatastoreLruCache::new(LOCAL_DATASTORE_CACHE_SIZE, chunk_store.clone());
+            Some(cache)
+        } else {
+            None
+        };
+
         Ok(DataStoreImpl {
             chunk_store,
             gc_mutex: Mutex::new(()),
@@ -432,6 +475,7 @@ impl DataStore {
             last_digest,
             sync_level: tuning.sync_level.unwrap_or_default(),
             backend_config,
+            lru_store_caching,
         })
     }
 
diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index e6f65575b..f1ad3d4c2 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -216,3 +216,6 @@ pub use snapshot_reader::SnapshotReader;
 
 mod local_chunk_reader;
 pub use local_chunk_reader::LocalChunkReader;
+
+mod local_datastore_lru_cache;
+pub use local_datastore_lru_cache::LocalDatastoreLruCache;
diff --git a/pbs-datastore/src/local_datastore_lru_cache.rs b/pbs-datastore/src/local_datastore_lru_cache.rs
new file mode 100644
index 000000000..c711c5208
--- /dev/null
+++ b/pbs-datastore/src/local_datastore_lru_cache.rs
@@ -0,0 +1,116 @@
+//! Use a local datastore as cache for operations on a datastore attached via
+//! a network layer (e.g. via the S3 backend).
+
+use std::future::Future;
+use std::sync::Arc;
+
+use anyhow::{bail, Error};
+use hyper::body::HttpBody;
+
+use pbs_s3_client::S3Client;
+use pbs_tools::async_lru_cache::{AsyncCacher, AsyncLruCache};
+
+use crate::ChunkStore;
+use crate::DataBlob;
+
+#[derive(Clone)]
+pub struct S3Cacher {
+    client: Arc<S3Client>,
+    store: Arc<ChunkStore>,
+}
+
+impl AsyncCacher<[u8; 32], ()> for S3Cacher {
+    fn fetch(
+        &self,
+        key: [u8; 32],
+    ) -> Box<dyn Future<Output = Result<Option<()>, Error>> + Send + 'static> {
+        let client = self.client.clone();
+        let store = self.store.clone();
+        Box::new(async move {
+            match client.get_object(key.into()).await? {
+                None => bail!("could not fetch object with key {}", hex::encode(key)),
+                Some(response) => {
+                    let bytes = response.content.collect().await?.to_bytes();
+                    let chunk = DataBlob::from_raw(bytes.to_vec())?;
+                    store.insert_chunk(&chunk, &key)?;
+                    Ok(Some(()))
+                }
+            }
+        })
+    }
+}
+
+impl S3Cacher {
+    pub fn new(client: Arc<S3Client>, store: Arc<ChunkStore>) -> Self {
+        Self { client, store }
+    }
+}
+
+/// LRU cache using local datastore for caching chunks
+///
+/// Uses a LRU cache, but without storing the values in-memory but rather
+/// on the filesystem
+pub struct LocalDatastoreLruCache {
+    cache: AsyncLruCache<[u8; 32], ()>,
+    store: Arc<ChunkStore>,
+}
+
+impl LocalDatastoreLruCache {
+    pub fn new(capacity: usize, store: Arc<ChunkStore>) -> Self {
+        Self {
+            cache: AsyncLruCache::new(capacity),
+            store,
+        }
+    }
+
+    /// Insert a new chunk into the local datastore cache.
+    ///
+    /// Fails if the chunk cannot be inserted successfully.
+    pub fn insert(&self, digest: &[u8; 32], chunk: &DataBlob) -> Result<(), Error> {
+        self.store.insert_chunk(chunk, digest)?;
+        self.cache.insert(*digest, (), |digest| {
+            let (path, _digest_str) = self.store.chunk_path(&digest);
+            // Truncate to free up space but keep the inode around, since that
+            // is used as marker for chunks in use by garbage collection.
+            nix::unistd::truncate(&path, 0).map_err(Error::from)
+        })
+    }
+
+    /// Remove a chunk from the local datastore cache.
+    ///
+    /// Fails if the chunk cannot be deleted successfully.
+    pub fn remove(&self, digest: &[u8; 32]) -> Result<(), Error> {
+        self.cache.remove(*digest);
+        let (path, _digest_str) = self.store.chunk_path(digest);
+        std::fs::remove_file(path).map_err(Error::from)
+    }
+
+    pub async fn access(
+        &self,
+        digest: &[u8; 32],
+        cacher: &mut S3Cacher,
+    ) -> Result<Option<DataBlob>, Error> {
+        if self
+            .cache
+            .access(*digest, cacher, |digest| {
+                let (path, _digest_str) = self.store.chunk_path(&digest);
+                // Truncate to free up space but keep the inode around, since that
+                // is used as marker for chunks in use by garbage collection.
+                nix::unistd::truncate(&path, 0).map_err(Error::from)
+            })
+            .await?
+            .is_some()
+        {
+            let (path, _digest_str) = self.store.chunk_path(digest);
+            let mut file = std::fs::File::open(&path)?;
+            let chunk = DataBlob::load_from_reader(&mut file)?;
+            Ok(Some(chunk))
+        } else {
+            Ok(None)
+        }
+    }
+
+    pub fn contains(&self, digest: &[u8; 32]) -> bool {
+        self.cache.contains(*digest)
+    }
+}
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:33 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:33 +0200
Subject: [pbs-devel] [RFC proxmox-backup 32/39] tools: lru cache: add
 removed callback for evicted nodes
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-33-c.ebner@proxmox.com>

Add a callback function to be executed on evicted cache nodes. The
callback gets the key of the removed node, allowing to externally act
based on that value.

Since the callback might fail, extend the current LRU cache api to
return an error on insert, covering the error for the `removed`
callback.

Async lru cache, callsites and tests are adapted to include the
additional callback parameter accordingly.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/cached_chunk_reader.rs |  6 +++-
 pbs-datastore/src/datastore.rs           |  2 +-
 pbs-datastore/src/dynamic_index.rs       |  1 +
 pbs-tools/src/async_lru_cache.rs         | 23 +++++++++----
 pbs-tools/src/lru_cache.rs               | 42 +++++++++++++++---------
 5 files changed, 50 insertions(+), 24 deletions(-)

diff --git a/pbs-datastore/src/cached_chunk_reader.rs b/pbs-datastore/src/cached_chunk_reader.rs
index be7f2a1e2..95ac23a54 100644
--- a/pbs-datastore/src/cached_chunk_reader.rs
+++ b/pbs-datastore/src/cached_chunk_reader.rs
@@ -81,7 +81,11 @@ impl<I: IndexFile, R: AsyncReadChunk + Send + Sync + 'static> CachedChunkReader<
                 let info = self.index.chunk_info(chunk.0).unwrap();
 
                 // will never be None, see AsyncChunkCacher
-                let data = self.cache.access(info.digest, &self.cacher).await?.unwrap();
+                let data = self
+                    .cache
+                    .access(info.digest, &self.cacher, |_| Ok(()))
+                    .await?
+                    .unwrap();
 
                 let want_bytes = ((info.range.end - cur_offset) as usize).min(size - read);
                 let slice = &mut buf[read..(read + want_bytes)];
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index a15f82b5b..22ad566ca 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1156,7 +1156,7 @@ impl DataStore {
             let digest = index.index_digest(pos).unwrap();
 
             // Avoid multiple expensive atime updates by utimensat
-            if chunk_lru_cache.insert(*digest, ()) {
+            if chunk_lru_cache.insert(*digest, (), |_| Ok(()))? {
                 continue;
             }
 
diff --git a/pbs-datastore/src/dynamic_index.rs b/pbs-datastore/src/dynamic_index.rs
index 8e9cb1163..e9d28c7de 100644
--- a/pbs-datastore/src/dynamic_index.rs
+++ b/pbs-datastore/src/dynamic_index.rs
@@ -599,6 +599,7 @@ impl<S: ReadChunk> BufferedDynamicReader<S> {
                     store: &mut self.store,
                     index: &self.index,
                 },
+                |_| Ok(()),
             )?
             .ok_or_else(|| format_err!("chunk not found by cacher"))?;
 
diff --git a/pbs-tools/src/async_lru_cache.rs b/pbs-tools/src/async_lru_cache.rs
index c43b87717..141114933 100644
--- a/pbs-tools/src/async_lru_cache.rs
+++ b/pbs-tools/src/async_lru_cache.rs
@@ -42,7 +42,16 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V: Clone + Send + 'static> AsyncL
     /// Access an item either via the cache or by calling cacher.fetch. A return value of Ok(None)
     /// means the item requested has no representation, Err(_) means a call to fetch() failed,
     /// regardless of whether it was initiated by this call or a previous one.
-    pub async fn access(&self, key: K, cacher: &dyn AsyncCacher<K, V>) -> Result<Option<V>, Error> {
+    /// Calls the removed callback on  the evicted item, if any.
+    pub async fn access<F>(
+        &self,
+        key: K,
+        cacher: &dyn AsyncCacher<K, V>,
+        removed: F,
+    ) -> Result<Option<V>, Error>
+    where
+        F: Fn(K) -> Result<(), Error>,
+    {
         let (owner, result_fut) = {
             // check if already requested
             let mut maps = self.maps.lock().unwrap();
@@ -71,7 +80,7 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V: Clone + Send + 'static> AsyncL
             // this call was the one initiating the request, put into LRU and remove from map
             let mut maps = self.maps.lock().unwrap();
             if let Ok(Some(ref value)) = result {
-                maps.0.insert(key, value.clone());
+                maps.0.insert(key, value.clone(), removed)?;
             }
             maps.1.remove(&key);
         }
@@ -106,15 +115,15 @@ mod test {
             let cache: AsyncLruCache<i32, String> = AsyncLruCache::new(2);
 
             assert_eq!(
-                cache.access(10, &cacher).await.unwrap(),
+                cache.access(10, &cacher, |_| Ok(())).await.unwrap(),
                 Some("x10".to_string())
             );
             assert_eq!(
-                cache.access(20, &cacher).await.unwrap(),
+                cache.access(20, &cacher, |_| Ok(())).await.unwrap(),
                 Some("x20".to_string())
             );
             assert_eq!(
-                cache.access(30, &cacher).await.unwrap(),
+                cache.access(30, &cacher, |_| Ok(())).await.unwrap(),
                 Some("x30".to_string())
             );
 
@@ -123,14 +132,14 @@ mod test {
                 tokio::spawn(async move {
                     let cacher = TestAsyncCacher { prefix: "y" };
                     assert_eq!(
-                        c.access(40, &cacher).await.unwrap(),
+                        c.access(40, &cacher, |_| Ok(())).await.unwrap(),
                         Some("y40".to_string())
                     );
                 });
             }
 
             assert_eq!(
-                cache.access(20, &cacher).await.unwrap(),
+                cache.access(20, &cacher, |_| Ok(())).await.unwrap(),
                 Some("x20".to_string())
             );
         });
diff --git a/pbs-tools/src/lru_cache.rs b/pbs-tools/src/lru_cache.rs
index 9e0112647..53b84ec41 100644
--- a/pbs-tools/src/lru_cache.rs
+++ b/pbs-tools/src/lru_cache.rs
@@ -60,10 +60,10 @@ impl<K, V> CacheNode<K, V> {
 /// assert_eq!(cache.get_mut(1), None);
 /// assert_eq!(cache.len(), 0);
 ///
-/// cache.insert(1, 1);
-/// cache.insert(2, 2);
-/// cache.insert(3, 3);
-/// cache.insert(4, 4);
+/// cache.insert(1, 1, |_| Ok(()));
+/// cache.insert(2, 2, |_| Ok(()));
+/// cache.insert(3, 3, |_| Ok(()));
+/// cache.insert(4, 4, |_| Ok(()));
 /// assert_eq!(cache.len(), 3);
 ///
 /// assert_eq!(cache.get_mut(1), None);
@@ -77,9 +77,9 @@ impl<K, V> CacheNode<K, V> {
 /// assert_eq!(cache.len(), 0);
 /// assert_eq!(cache.get_mut(2), None);
 /// // access will fill in missing cache entry by fetching from LruCacher
-/// assert_eq!(cache.access(2, &mut LruCacher {}).unwrap(), Some(&mut 2));
+/// assert_eq!(cache.access(2, &mut LruCacher {}, |_| Ok(())).unwrap(), Some(&mut 2));
 ///
-/// cache.insert(1, 1);
+/// cache.insert(1, 1, |_| Ok(()));
 /// assert_eq!(cache.get_mut(1), Some(&mut 1));
 ///
 /// cache.clear();
@@ -133,7 +133,10 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
 
     /// Insert or update an entry identified by `key` with the given `value`.
     /// This entry is placed as the most recently used node at the head.
-    pub fn insert(&mut self, key: K, value: V) -> bool {
+    pub fn insert<F>(&mut self, key: K, value: V, removed: F) -> Result<bool, anyhow::Error>
+    where
+        F: Fn(K) -> Result<(), anyhow::Error>,
+    {
         match self.map.entry(key) {
             Entry::Occupied(mut o) => {
                 // Node present, update value
@@ -142,7 +145,7 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
                 let mut node = unsafe { Box::from_raw(node_ptr) };
                 node.value = value;
                 let _node_ptr = Box::into_raw(node);
-                true
+                Ok(true)
             }
             Entry::Vacant(v) => {
                 // Node not present, insert a new one
@@ -158,9 +161,11 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
                 // avoid borrow conflict. This means there are temporarily
                 // self.capacity + 1 cache nodes.
                 if self.map.len() > self.capacity {
-                    self.pop_tail();
+                    if let Some(removed_node) = self.pop_tail() {
+                        removed(removed_node)?;
+                    }
                 }
-                false
+                Ok(false)
             }
         }
     }
@@ -174,11 +179,12 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
     }
 
     /// Remove the least recently used node from the cache.
-    fn pop_tail(&mut self) {
+    fn pop_tail(&mut self) -> Option<K> {
         if let Some(old_tail) = self.list.pop_tail() {
             // Remove HashMap entry for old tail
-            self.map.remove(&old_tail.key);
+            return self.map.remove(&old_tail.key).map(|_| old_tail.key);
         }
+        None
     }
 
     /// Get a mutable reference to the value identified by `key`.
@@ -206,11 +212,15 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
     /// value.
     /// If fetch returns a value, it is inserted as the most recently used entry
     /// in the cache.
-    pub fn access<'a>(
+    pub fn access<'a, F>(
         &'a mut self,
         key: K,
         cacher: &mut dyn Cacher<K, V>,
-    ) -> Result<Option<&'a mut V>, anyhow::Error> {
+        removed: F,
+    ) -> Result<Option<&'a mut V>, anyhow::Error>
+    where
+        F: Fn(K) -> Result<(), anyhow::Error>,
+    {
         match self.map.entry(key) {
             Entry::Occupied(mut o) => {
                 // Cache hit, birng node to front of list
@@ -234,7 +244,9 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
                         // avoid borrow conflict. This means there are temporarily
                         // self.capacity + 1 cache nodes.
                         if self.map.len() > self.capacity {
-                            self.pop_tail();
+                            if let Some(removed_node) = self.pop_tail() {
+                                removed(removed_node)?;
+                            }
                         }
                     }
                 }
-- 
2.39.5




From c.ebner at proxmox.com  Mon May 19 13:46:11 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Mon, 19 May 2025 13:46:11 +0200
Subject: [pbs-devel] [RFC proxmox-backup 10/39] s3 client: implement methods
 to operate on s3 objects in bucket
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <20250519114640.303640-11-c.ebner@proxmox.com>

Adds the basic implementation of the client to use s3 object stores
as backend for PBS datastores.

This implements the basic client actions on a bucket and objects
stored within given bucket.

This is not feature complete and intended to be extended on a
per-demand fashion rather than implementing the whole client at once.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 Cargo.toml                           |   2 +
 pbs-s3-client/Cargo.toml             |   8 +
 pbs-s3-client/src/client.rs          | 370 +++++++++++++++++++++++++++
 pbs-s3-client/src/lib.rs             |   2 +
 pbs-s3-client/src/response_reader.rs | 324 +++++++++++++++++++++++
 5 files changed, 706 insertions(+)
 create mode 100644 pbs-s3-client/src/response_reader.rs

diff --git a/Cargo.toml b/Cargo.toml
index 3f51b356c..229ba1692 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -140,11 +140,13 @@ once_cell = "1.3.1"
 openssl = "0.10.40"
 percent-encoding = "2.1"
 pin-project-lite = "0.2"
+quick-xml = "0.26"
 regex = "1.5.5"
 rustyline = "9"
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
 serde_plain = "1.0"
+serde-xml-rs = "0.5"
 siphasher = "0.3"
 syslog = "6"
 tar = "0.4"
diff --git a/pbs-s3-client/Cargo.toml b/pbs-s3-client/Cargo.toml
index 9ee546200..6fd0f7cca 100644
--- a/pbs-s3-client/Cargo.toml
+++ b/pbs-s3-client/Cargo.toml
@@ -8,11 +8,19 @@ rust-version.workspace = true
 
 [dependencies]
 anyhow.workspace = true
+base64.workspace = true
+bytes.workspace = true
+crc32fast.workspace = true
+futures.workspace = true
 hex = { workspace = true, features = [ "serde" ] }
 hyper.workspace = true
 openssl.workspace = true
+quick-xml  = { workspace = true, features = ["async-tokio"] }
 serde.workspace = true
 serde_plain.workspace = true
+serde-xml-rs.workspace = true
+tokio = { workspace = true, features = [] }
+tokio-util = { workspace = true, features = ["compat"] }
 tracing.workspace = true
 url.workspace = true
 
diff --git a/pbs-s3-client/src/client.rs b/pbs-s3-client/src/client.rs
index e001cc7b0..972578e66 100644
--- a/pbs-s3-client/src/client.rs
+++ b/pbs-s3-client/src/client.rs
@@ -1,17 +1,36 @@
+use std::collections::HashMap;
+use std::io::Cursor;
 use std::sync::{Arc, Mutex};
 use std::time::Duration;
 
 use anyhow::{bail, format_err, Context, Error};
+use bytes::{Bytes, BytesMut};
+use hyper::body::HttpBody;
 use hyper::client::{Client, HttpConnector};
+use hyper::http::method::Method;
 use hyper::http::uri::Authority;
+use hyper::http::StatusCode;
+use hyper::http::{header, HeaderValue, Uri};
 use hyper::Body;
+use hyper::{Request, Response};
 use openssl::hash::MessageDigest;
+use openssl::sha::Sha256;
 use openssl::ssl::{SslConnector, SslMethod, SslVerifyMode};
 use openssl::x509::X509StoreContextRef;
+use quick_xml::events::BytesText;
+use quick_xml::writer::Writer;
 use tracing::error;
 
 use proxmox_http::client::HttpsConnector;
 
+use crate::aws_sign_v4::aws_sign_v4_signature;
+use crate::aws_sign_v4::AWS_SIGN_V4_DATETIME_FORMAT;
+use crate::object_key::S3ObjectKey;
+use crate::response_reader::{
+    CopyObjectResponse, DeleteObjectsResponse, GetObjectResponse, HeadObjectResponse,
+    ListObjectsV2Response, PutObjectResponse, ResponseReader,
+};
+
 const S3_HTTP_CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
 const S3_TCP_KEEPALIVE_TIME: u32 = 120;
 
@@ -128,4 +147,355 @@ impl S3Client {
             "unexpected certificate fingerprint {certificate_fingerprint}"
         ))
     }
+
+    async fn prepare(&self, mut request: Request<Body>) -> Result<Request<Body>, Error> {
+        let host_header = request
+            .uri()
+            .authority()
+            .ok_or_else(|| format_err!("request missing authority"))?
+            .to_string();
+
+        // Calculate the crc32 sum of the whole body, while the DataBlob of a chunk store does skip
+        // over the DataBlob header, so that must be considered when using this to check for
+        // changes/interity.
+        let mut crc32sum = crc32fast::Hasher::new();
+        // Content verification for aws s3 signature
+        let mut hasher = Sha256::new();
+        // Load payload into memory, needed as the hash and checksum have to be calculated a-priori
+        let buffer: Bytes = {
+            let body = request.body_mut();
+            let mut buf = BytesMut::with_capacity(body.size_hint().lower() as usize);
+            while let Some(chunk) = body.data().await {
+                let chunk = chunk?;
+                hasher.update(&chunk);
+                crc32sum.update(&chunk);
+                buf.extend_from_slice(&chunk);
+            }
+            buf.freeze()
+        };
+        let payload_digest = hex::encode(hasher.finish());
+        let payload_crc32sum = base64::encode(crc32sum.finalize().to_be_bytes());
+        let payload_len = buffer.len();
+        *request.body_mut() = Body::from(buffer);
+
+        let epoch = proxmox_time::epoch_i64();
+        let datetime = proxmox_time::strftime_utc(AWS_SIGN_V4_DATETIME_FORMAT, epoch)?;
+
+        request
+            .headers_mut()
+            .insert("x-amz-date", HeaderValue::from_str(&datetime)?);
+        request
+            .headers_mut()
+            .insert("host", HeaderValue::from_str(&host_header)?);
+        request.headers_mut().insert(
+            "x-amz-content-sha256",
+            HeaderValue::from_str(&payload_digest)?,
+        );
+        if payload_len > 0 {
+            request.headers_mut().insert(
+                header::CONTENT_LENGTH,
+                HeaderValue::from_str(&payload_len.to_string())?,
+            );
+        }
+        if !payload_crc32sum.is_empty() {
+            request.headers_mut().insert(
+                "x-amz-checksum-crc32",
+                HeaderValue::from_str(&payload_crc32sum)?,
+            );
+        }
+
+        let signature = aws_sign_v4_signature(&request, &self.options, epoch, &payload_digest)?;
+
+        request
+            .headers_mut()
+            .insert(header::AUTHORIZATION, HeaderValue::from_str(&signature)?);
+
+        Ok(request)
+    }
+
+    pub async fn send(&self, request: Request<Body>) -> Result<Response<Body>, Error> {
+        let request = self.prepare(request).await?;
+        let response = tokio::time::timeout(S3_HTTP_CONNECT_TIMEOUT, self.client.request(request))
+            .await
+            .context("request timeout")??;
+        Ok(response)
+    }
+
+    /// Check if bucket exists and got permissions to access it.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html
+    pub async fn head_bucket(&self) -> Result<(), Error> {
+        let request = Request::builder()
+            .method(Method::HEAD)
+            .uri(self.uri_builder("/")?)
+            .body(Body::empty())?;
+        let response = self.send(request).await?;
+        let (parts, _body) = response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::BAD_REQUEST | StatusCode::FORBIDDEN | StatusCode::NOT_FOUND => {
+                bail!("bucket does not exist or no permission to access it")
+            }
+            status_code => bail!("unexpected status code {status_code}"),
+        }
+
+        Ok(())
+    }
+
+    /// Fetch metadata from an object without returning the object itself.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html
+    pub async fn head_object(
+        &self,
+        object_key: S3ObjectKey,
+    ) -> Result<Option<HeadObjectResponse>, Error> {
+        let request = Request::builder()
+            .method(Method::HEAD)
+            .uri(self.uri_builder(&object_key)?)
+            .body(Body::empty())?;
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.head_object_response().await
+    }
+
+    /// Fetch an object from object store.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
+    pub async fn get_object(
+        &self,
+        object_key: S3ObjectKey,
+    ) -> Result<Option<GetObjectResponse>, Error> {
+        let request = Request::builder()
+            .method(Method::GET)
+            .uri(self.uri_builder(&object_key)?)
+            .body(Body::empty())?;
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.get_object_response().await
+    }
+
+    /// Returns some or all (up to 1,000) of the objects in a bucket with each request.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html
+    pub async fn list_objects_v2(
+        &self,
+        prefix: Option<&str>,
+        max_keys: Option<u64>,
+        continuation_token: Option<&str>,
+    ) -> Result<ListObjectsV2Response, Error> {
+        let mut path_and_query = String::from("/?list-type=2");
+        if let Some(prefix) = prefix {
+            path_and_query.push_str("&prefix=");
+            path_and_query.push_str(prefix);
+        }
+        if let Some(max_keys) = max_keys {
+            path_and_query.push_str("&max-keys=");
+            path_and_query.push_str(&max_keys.to_string());
+        }
+        if let Some(token) = continuation_token {
+            path_and_query.push_str("&continuation-token=");
+            path_and_query.push_str(token);
+        }
+        let request = Request::builder()
+            .method(Method::GET)
+            .uri(self.uri_builder(&path_and_query)?)
+            .body(Body::empty())?;
+
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.list_objects_v2_response().await
+    }
+
+    /// Add a new object to a bucket.
+    ///
+    /// Do not reupload if an object with matching key already exists in the bucket.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
+    pub async fn put_object(
+        &self,
+        object_key: S3ObjectKey,
+        object_data: Body,
+    ) -> Result<PutObjectResponse, Error> {
+        // Assure data integrity after upload by providing a trailing checksum header. This value
+        // can also be used to compare a local chunk content against an object stored in S3.
+        // https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html
+        let request = Request::builder()
+            .method(Method::PUT)
+            .uri(self.uri_builder(&object_key)?)
+            .header(header::CONTENT_TYPE, "binary/octet")
+            // Never overwrite pre-existing objects with the same key.
+            //.header(header::IF_NONE_MATCH, "*")
+            .body(object_data)?;
+
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.put_object_response().await
+    }
+
+    /// Sets the supplied tag-set to an object that already exists in a bucket. A tag is a key-value pair.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html
+    pub async fn put_object_tagging(
+        &self,
+        object_key: S3ObjectKey,
+        tagset: &HashMap<String, String>,
+    ) -> Result<bool, Error> {
+        let mut writer = Writer::new(Cursor::new(Vec::new()));
+        writer
+            .create_element("Tagging")
+            .with_attribute(("xmlns", "http://s3.amazonaws.com/doc/2006-03-01/"))
+            .write_inner_content(|writer| {
+                writer
+                    .create_element("TagSet")
+                    .write_inner_content(|writer| {
+                        for (key, value) in tagset.iter() {
+                            writer.create_element("Tag").write_inner_content(|writer| {
+                                writer
+                                    .create_element("Key")
+                                    .write_text_content(BytesText::new(key))?;
+                                writer
+                                    .create_element("Value")
+                                    .write_text_content(BytesText::new(value))?;
+                                Ok(())
+                            })?;
+                        }
+                        Ok(())
+                    })?;
+                Ok(())
+            })?;
+
+        let body: Body = writer.into_inner().into_inner().into();
+        let request = Request::builder()
+            .method(Method::PUT)
+            .uri(self.uri_builder(&format!("{object_key}?tagging"))?)
+            .body(body)?;
+
+        let response = self.send(request).await?;
+        Ok(response.status().is_success())
+    }
+
+    /// Sets the supplied tag to an object that already exists in a bucket. A tag is a key-value pair.
+    /// Optimized version of the `put_object_tagging` to only set a single tag.
+    pub async fn put_object_tag(
+        &self,
+        object_key: S3ObjectKey,
+        tag_key: &str,
+        tag_value: &str,
+    ) -> Result<bool, Error> {
+        let body: Body = format!(
+            r#"<Tagging xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
+                <TagSet>
+                    <Tag>
+                        <Key>{tag_key}</Key>
+                        <Value>{tag_value}</Value>
+                    </Tag>
+                </TagSet>
+            </Tagging>"#
+        )
+        .into();
+
+        let request = Request::builder()
+            .method(Method::PUT)
+            .uri(self.uri_builder(&format!("{object_key}?tagging"))?)
+            .body(body)?;
+
+        let response = self.send(request).await?;
+        //TODO: Response and error handling!
+        Ok(response.status().is_success())
+    }
+
+    /// Creates a copy of an object that is already stored in Amazon S3.
+    /// Uses the `x-amz-metadata-directive` set to `REPLACE`, therefore resulting in updated metadata.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
+    pub async fn copy_object(
+        &self,
+        destination_key: S3ObjectKey,
+        source_bucket: &str,
+        source_key: S3ObjectKey,
+    ) -> Result<CopyObjectResponse, Error> {
+        let copy_source = source_key.to_copy_source_key(source_bucket);
+        let request = Request::builder()
+            .method(Method::PUT)
+            .uri(self.uri_builder(&destination_key)?)
+            .header("x-amz-copy-source", HeaderValue::from_str(&copy_source)?)
+            .header(
+                "x-amz-metadata-directive",
+                HeaderValue::from_str("REPLACE")?,
+            )
+            .body(Body::empty())?;
+
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.copy_object_response().await
+    }
+
+    /// Helper to update the metadata for an object by copying it to itself. This will not cause
+    /// any additional costs other than the request cost itself.
+    ///
+    /// Note: This will actually create a new object for buckets with versioning enabled.
+    /// Return with error if that is the case, detected by checking the presence of the
+    /// `x-amz-version-id` header in the response.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
+    pub async fn update_object_metadata(
+        &self,
+        object_key: S3ObjectKey,
+    ) -> Result<CopyObjectResponse, Error> {
+        let response = self
+            .copy_object(object_key.clone(), &self.options.bucket, object_key)
+            .await?;
+        if response.x_amz_version_id.is_some() {
+            // Return an error if the response contains an `x-amz-version-id`, indicating that the
+            // bucket has versioning enabled, as that will bloat the bucket size and therefore cost.
+            bail!("Failed to update object metadata as versioning is enabled");
+        }
+        Ok(response)
+    }
+
+    /// Delete multiple objects from a bucket using a single HTTP request.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
+    pub async fn delete_objects(
+        &self,
+        object_keys: &[String],
+    ) -> Result<DeleteObjectsResponse, Error> {
+        let mut body = String::from(r#"<Delete xmlns="http://s3.amazonaws.com/doc/2006-03-01/">"#);
+        for object_key in object_keys {
+            let object = format!("<Object><Key>{object_key}</Key></Object>");
+            body.push_str(&object);
+        }
+        body.push_str("</Delete>");
+        let request = Request::builder()
+            .method(Method::POST)
+            .uri(self.uri_builder("/?delete")?)
+            .body(Body::from(body))?;
+
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.delete_objects_response().await
+    }
+
+    /// Delete objects by given key prefix.
+    /// Requires at least 2 api calls.
+    pub async fn delete_objects_by_prefix(
+        &self,
+        prefix: &str,
+    ) -> Result<DeleteObjectsResponse, Error> {
+        // S3 API does not provide a convenient way to delete objects by key prefix.
+        // List all objects with given group prefix and delete all objects found, so this
+        // requires at least 2 API calls.
+        // TODO: fix for more than 1000 response items given by api limit.
+        let list_objects_result = self.list_objects_v2(Some(prefix), None, None).await?;
+        let objects_to_delete: Vec<String> = list_objects_result
+            .contents
+            .into_iter()
+            .map(|item| item.key)
+            .collect();
+        self.delete_objects(&objects_to_delete).await
+    }
+
+    #[inline(always)]
+    /// Helper to generate [`Uri`] instance with common properties based on given path and query
+    /// string
+    fn uri_builder(&self, path_and_query: &str) -> Result<Uri, Error> {
+        Uri::builder()
+            .scheme("https")
+            .authority(self.authority.clone())
+            .path_and_query(path_and_query)
+            .build()
+            .context("failed to build uri")
+    }
 }
diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
index 00fa26455..7cc0ea841 100644
--- a/pbs-s3-client/src/lib.rs
+++ b/pbs-s3-client/src/lib.rs
@@ -3,6 +3,8 @@ mod client;
 pub use client::{S3Client, S3ClientOptions};
 mod object_key;
 pub use object_key::{S3ObjectKey, S3_CONTENT_PREFIX};
+mod response_reader;
+pub use response_reader::PutObjectResponse;
 
 use std::time::Duration;
 
diff --git a/pbs-s3-client/src/response_reader.rs b/pbs-s3-client/src/response_reader.rs
new file mode 100644
index 000000000..ed82d77b9
--- /dev/null
+++ b/pbs-s3-client/src/response_reader.rs
@@ -0,0 +1,324 @@
+use std::str::FromStr;
+
+use anyhow::{anyhow, bail, Context, Error};
+use hyper::body::HttpBody;
+use hyper::header::HeaderName;
+use hyper::http::header;
+use hyper::http::StatusCode;
+use hyper::{Body, HeaderMap, Response};
+use serde::Deserialize;
+
+use crate::{HttpDate, LastModifiedTimestamp};
+
+pub(crate) struct ResponseReader {
+    response: Response<Body>,
+}
+
+#[derive(Debug)]
+pub struct ListObjectsV2Response {
+    pub date: HttpDate,
+    pub name: String,
+    pub prefix: String,
+    pub key_count: u64,
+    pub max_keys: u64,
+    pub is_truncated: bool,
+    pub continuation_token: Option<String>,
+    pub next_continuation_token: Option<String>,
+    pub contents: Vec<ListObjectsV2Contents>,
+}
+
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+struct ListObjectsV2ResponseBody {
+    pub name: String,
+    pub prefix: String,
+    pub key_count: u64,
+    pub max_keys: u64,
+    pub is_truncated: bool,
+    pub continuation_token: Option<String>,
+    pub next_continuation_token: Option<String>,
+    pub contents: Option<Vec<ListObjectsV2Contents>>,
+}
+
+impl ListObjectsV2ResponseBody {
+    fn with_date(self, date: HttpDate) -> ListObjectsV2Response {
+        ListObjectsV2Response {
+            date,
+            name: self.name,
+            prefix: self.prefix,
+            key_count: self.key_count,
+            max_keys: self.max_keys,
+            is_truncated: self.is_truncated,
+            continuation_token: self.continuation_token,
+            next_continuation_token: self.next_continuation_token,
+            contents: self.contents.unwrap_or_else(|| Vec::new()),
+        }
+    }
+}
+
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+pub struct ListObjectsV2Contents {
+    pub key: String,
+    pub last_modified: LastModifiedTimestamp,
+    pub e_tag: String,
+    pub size: u64,
+    pub storage_class: String,
+}
+
+#[derive(Debug)]
+/// Subset of the head object response (headers only, there is no body)
+/// See https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html#API_HeadObject_ResponseSyntax
+pub struct HeadObjectResponse {
+    pub content_length: u64,
+    pub content_type: String,
+    pub date: HttpDate,
+    pub e_tag: String,
+    pub last_modified: HttpDate,
+}
+
+#[derive(Debug)]
+/// Subset of the get object response
+/// https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_ResponseSyntax
+pub struct GetObjectResponse {
+    pub content_length: u64,
+    pub content_type: String,
+    pub date: HttpDate,
+    pub e_tag: String,
+    pub last_modified: HttpDate,
+    pub content: Body,
+}
+
+#[derive(Debug)]
+pub struct CopyObjectResponse {
+    pub copy_object_result: CopyObjectResult,
+    pub x_amz_version_id: Option<String>,
+}
+
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+pub struct CopyObjectResult {
+    pub e_tag: String,
+    pub last_modified: LastModifiedTimestamp,
+}
+
+/// Subset of the put object response
+/// https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_ResponseSyntax
+#[derive(Debug)]
+pub enum PutObjectResponse {
+    NeedsRetry,
+    PreconditionFailed,
+    Success(String),
+}
+
+/// Subset of the delete objects response
+/// https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html#API_DeleteObjects_ResponseElements
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+pub struct DeleteObjectsResponse {
+    pub deleted: Option<Vec<DeletedObject>>,
+    pub error: Option<Vec<DeleteObjectError>>,
+}
+
+/// https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeletedObject.html
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+pub struct DeletedObject {
+    pub delete_marker: Option<bool>,
+    pub delete_marker_version_id: Option<String>,
+    pub key: Option<String>,
+    pub version_id: Option<String>,
+}
+
+/// https://docs.aws.amazon.com/AmazonS3/latest/API/API_Error.html
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+pub struct DeleteObjectError {
+    pub code: Option<String>,
+    pub key: Option<String>,
+    pub message: Option<String>,
+    pub version_id: Option<String>,
+}
+
+impl ResponseReader {
+    pub(crate) fn new(response: Response<Body>) -> Self {
+        Self { response }
+    }
+
+    pub(crate) async fn list_objects_v2_response(self) -> Result<ListObjectsV2Response, Error> {
+        let (parts, body) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::NOT_FOUND => bail!("bucket does not exist"),
+            status_code => bail!("unexpected status code {status_code}"),
+        }
+
+        let body = body.collect().await?.to_bytes();
+        let body = String::from_utf8(body.to_vec())?;
+
+        let date: HttpDate = Self::parse_header(header::DATE, &parts.headers)?;
+
+        let response: ListObjectsV2ResponseBody =
+            serde_xml_rs::from_str(&body).context("failed to parse response body")?;
+
+        Ok(response.with_date(date))
+    }
+
+    pub(crate) async fn head_object_response(self) -> Result<Option<HeadObjectResponse>, Error> {
+        let (parts, body) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::NOT_FOUND => return Ok(None),
+            status_code => bail!("unexpected status code {status_code}"),
+        }
+        let body = body.collect().await?.to_bytes();
+        if !body.is_empty() {
+            bail!("got unexpected non-empty response body");
+        }
+
+        let content_length: u64 = Self::parse_header(header::CONTENT_LENGTH, &parts.headers)?;
+        let content_type = Self::parse_header(header::CONTENT_TYPE, &parts.headers)?;
+        let e_tag = Self::parse_header(header::ETAG, &parts.headers)?;
+        let date = Self::parse_header(header::DATE, &parts.headers)?;
+        let last_modified = Self::parse_header(header::LAST_MODIFIED, &parts.headers)?;
+
+        Ok(Some(HeadObjectResponse {
+            content_length,
+            content_type,
+            date,
+            e_tag,
+            last_modified,
+        }))
+    }
+
+    pub(crate) async fn get_object_response(self) -> Result<Option<GetObjectResponse>, Error> {
+        let (parts, content) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::NOT_FOUND => return Ok(None),
+            StatusCode::FORBIDDEN => bail!("object is archived and inaccessible until restored"),
+            status_code => bail!("unexpected status code {status_code}"),
+        }
+
+        let content_length: u64 = Self::parse_header(header::CONTENT_LENGTH, &parts.headers)?;
+        let content_type = Self::parse_header(header::CONTENT_TYPE, &parts.headers)?;
+        let e_tag = Self::parse_header(header::ETAG, &parts.headers)?;
+        let date = Self::parse_header(header::DATE, &parts.headers)?;
+        let last_modified = Self::parse_header(header::LAST_MODIFIED, &parts.headers)?;
+
+        Ok(Some(GetObjectResponse {
+            content_length,
+            content_type,
+            date,
+            e_tag,
+            last_modified,
+            content,
+        }))
+    }
+
+    pub(crate) async fn copy_object_response(self) -> Result<CopyObjectResponse, Error> {
+        let (parts, content) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::NOT_FOUND => bail!("object not found"),
+            StatusCode::FORBIDDEN => bail!("the source object is not in the active tier"),
+            status_code => bail!("unexpected status code {status_code}"),
+        }
+
+        let body = content.collect().await?.to_bytes();
+        let body = String::from_utf8(body.to_vec())?;
+
+        let x_amz_version_id = match parts.headers.get("x-amz-version-id") {
+            Some(version_id) => Some(
+                version_id
+                    .to_str()
+                    .context("failed to parse version id header")?
+                    .to_owned(),
+            ),
+            None => None,
+        };
+
+        let copy_object_result: CopyObjectResult =
+            serde_xml_rs::from_str(&body).context("failed to parse response body")?;
+
+        Ok(CopyObjectResponse {
+            copy_object_result,
+            x_amz_version_id,
+        })
+    }
+
+    pub(crate) async fn put_object_response(self) -> Result<PutObjectResponse, Error> {
+        let (parts, body) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            // If-None-Match precondition failed, an object with same key already present.
+            // FIXME: Should this be dropped in favor of re-uploading and rely on the local
+            // cache to detect duplicates to increase data safety guarantees?
+            StatusCode::PRECONDITION_FAILED => return Ok(PutObjectResponse::PreconditionFailed),
+            StatusCode::CONFLICT => return Ok(PutObjectResponse::NeedsRetry),
+            StatusCode::BAD_REQUEST => bail!("invalid request: {body:?}"),
+            status_code => bail!("unexpected status code {status_code}"),
+        };
+
+        let body = body.collect().await?.to_bytes();
+        if !body.is_empty() {
+            bail!("got unexpected non-empty response body");
+        }
+
+        let e_tag = Self::parse_header(header::ETAG, &parts.headers)?;
+
+        Ok(PutObjectResponse::Success(e_tag))
+    }
+
+    pub(crate) async fn delete_objects_response(self) -> Result<DeleteObjectsResponse, Error> {
+        let (parts, body) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::BAD_REQUEST => bail!("invalid request: {body:?}"),
+            status_code => bail!("unexpected status code {status_code}"),
+        };
+
+        let body = body.collect().await?.to_bytes();
+        let body = String::from_utf8(body.to_vec())?;
+
+        let delete_objects_response: DeleteObjectsResponse =
+            serde_xml_rs::from_str(&body).context("failed to parse response body")?;
+
+        Ok(delete_objects_response)
+    }
+
+    fn parse_header<T: FromStr>(name: HeaderName, headers: &HeaderMap) -> Result<T, Error>
+    where
+        <T as FromStr>::Err: Send + Sync + 'static,
+        Result<T, <T as FromStr>::Err>: Context<T, <T as FromStr>::Err>,
+    {
+        let header_value = headers
+            .get(&name)
+            .ok_or_else(|| anyhow!("missing header '{name}'"))?;
+        let header_str = header_value
+            .to_str()
+            .with_context(|| format!("non UTF-8 header '{name}'"))?;
+        let value = header_str
+            .parse()
+            .with_context(|| format!("failed to parse header '{name}'"))?;
+        Ok(value)
+    }
+
+    fn parse_x_amz_checksum_crc32_header(headers: &HeaderMap) -> Result<u32, Error> {
+        let x_amz_checksum_crc32 = headers
+            .get("x-amz-checksum-crc32")
+            .ok_or_else(|| anyhow!("missing header 'x-amz-checksum-crc32'"))?;
+        let x_amz_checksum_crc32 = base64::decode(x_amz_checksum_crc32.to_str()?)?;
+        let x_amz_checksum_crc32: [u8; 4] = x_amz_checksum_crc32
+            .try_into()
+            .map_err(|_e| anyhow!("failed to convert x-amz-checksum-crc32 header"))?;
+        let x_amz_checksum_crc32 = u32::from_be_bytes(x_amz_checksum_crc32);
+        Ok(x_amz_checksum_crc32)
+    }
+}
-- 
2.39.5




From dietmar at proxmox.com  Tue May 20 10:11:21 2025
From: dietmar at proxmox.com (Dietmar Maurer)
Date: Tue, 20 May 2025 10:11:21 +0200 (CEST)
Subject: [pbs-devel] applied: [PATCH proxmox] auth-api: remove ticket info
 in old create ticket endpoint
In-Reply-To: <20250516131122.276231-1-s.sterz@proxmox.com>
References: <20250516131122.276231-1-s.sterz@proxmox.com>
Message-ID: <943047991.16724.1747728681298@webmail.proxmox.com>

applied



From s.sterz at proxmox.com  Tue May 20 10:55:48 2025
From: s.sterz at proxmox.com (Shannon Sterz)
Date: Tue, 20 May 2025 10:55:48 +0200
Subject: [pbs-devel] [PATCH proxmox] login: use `ticket` if both it and
 `ticket_info` are provided
Message-ID: <20250520085549.56525-1-s.sterz@proxmox.com>

previously the precense of `ticket_info` was assumed to indicate the
HTTPOnly authentication flow. the `ticket` field was ignore in that
case, because the client has no way of validating a ticket anyway.

this commit changes the behaviour to assume that the server is not
trying to "trick us" and that the presence of a `ticket` field
indicates that this value should be used for authentication. if the
`ticket_info` field is also present, the field will be ignored.

Signed-off-by: Shannon Sterz <s.sterz at proxmox.com>
---

this is basically a different approach to fixing an issue that occured
when a server also returned a `ticket_info` field but was still using
the non-HTTPOnly auth flow [1]. while this shouldn't be strictly
necessary, it shouldn't hurt either.

[1]: https://lore.proxmox.com/all/20250516131122.276231-1-s.sterz at proxmox.com/

 proxmox-login/src/lib.rs | 26 ++++++++++++++------------
 1 file changed, 14 insertions(+), 12 deletions(-)

diff --git a/proxmox-login/src/lib.rs b/proxmox-login/src/lib.rs
index e97ece7b..710434d6 100644
--- a/proxmox-login/src/lib.rs
+++ b/proxmox-login/src/lib.rs
@@ -200,22 +200,24 @@ impl Login {
             ));
         }

-        // `ticket_info` is set when the server sets the ticket via an HttpOnly cookie. this also
-        // means we do not have access to the cookie itself which happens for example in a browser.
-        // assume that the cookie is handled properly by the context (browser) and don't worry
-        // about handling it ourselves.
-        if let Some(ref ticket) = response.ticket_info {
-            let ticket = ticket.parse()?;
-            return Ok(TicketResult::HttpOnly(
-                self.authentication_for(ticket, response)?,
-            ));
-        }
-
         // old authentication flow where we needed to handle the ticket ourselves even in the
         // browser etc.
         let ticket: TicketResponse = match response.ticket {
             Some(ref ticket) => ticket.parse()?,
-            None => return Err("no ticket information in response".into()),
+            None => {
+                // `ticket_info` is set when the server sets the ticket via a HttpOnly cookie. this
+                // also means we do not have access to the cookie itself which happens for example
+                // in a browser. assume that the cookie is handled properly by the context
+                // (browser) and don't worry about handling it ourselves.
+                if let Some(ref ticket) = response.ticket_info {
+                    let ticket = ticket.parse()?;
+                    return Ok(TicketResult::HttpOnly(
+                        self.authentication_for(ticket, response)?,
+                    ));
+                }
+
+                return Err("no ticket information in response".into());
+            }
         };

         Ok(match ticket {
--
2.39.5




From d.csapak at proxmox.com  Wed May 21 10:45:22 2025
From: d.csapak at proxmox.com (Dominik Csapak)
Date: Wed, 21 May 2025 10:45:22 +0200
Subject: [pbs-devel] [PATCH proxmox-websocket-tunnel 1/2] update base64
 dependency
In-Reply-To: <20250521084524.829496-1-d.csapak@proxmox.com>
References: <20250521084524.829496-1-d.csapak@proxmox.com>
Message-ID: <20250521084524.829496-4-d.csapak@proxmox.com>

to make it compile again

Signed-off-by: Dominik Csapak <d.csapak at proxmox.com>
---
 Cargo.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Cargo.toml b/Cargo.toml
index de5e45b..008bf4c 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -10,7 +10,7 @@ exclude = ["debian"]
 
 [dependencies]
 anyhow = "1.0"
-base64 = "0.21"
+base64 = "0.22"
 futures = "0.3"
 futures-util = "0.3"
 hex = "0.4"
-- 
2.39.5




From d.csapak at proxmox.com  Wed May 21 10:45:23 2025
From: d.csapak at proxmox.com (Dominik Csapak)
Date: Wed, 21 May 2025 10:45:23 +0200
Subject: [pbs-devel] [PATCH proxmox-websocket-tunnel 2/2] use proxmox-http's
 openssl callback
In-Reply-To: <20250521084524.829496-1-d.csapak@proxmox.com>
References: <20250521084524.829496-1-d.csapak@proxmox.com>
Message-ID: <20250521084524.829496-5-d.csapak@proxmox.com>

no functional change intended, since the callback there should implement
the same behavior.

With this, we can drop the dependency on itertools.

Signed-off-by: Dominik Csapak <d.csapak at proxmox.com>
---
 Cargo.toml  |  3 +--
 src/main.rs | 66 ++++++++++++++++++++++-------------------------------
 2 files changed, 28 insertions(+), 41 deletions(-)

diff --git a/Cargo.toml b/Cargo.toml
index 008bf4c..0f8f375 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -15,7 +15,6 @@ futures = "0.3"
 futures-util = "0.3"
 hex = "0.4"
 hyper = "0.14"
-itertools = "0.10"
 openssl = "0.10"
 percent-encoding = "2"
 serde = { version = "1.0", features = ["derive"] }
@@ -24,5 +23,5 @@ tokio = { version = "1", features = ["io-std", "io-util", "macros", "rt-multi-th
 tokio-stream = { version = "0.1", features = ["io-util"] }
 tokio-util = "0.7"
 
-proxmox-http = { version = "0.9", features = ["websocket", "client"] }
+proxmox-http = { version = "0.9", features = ["websocket", "client", "tls"] }
 proxmox-sys = "0.6"
diff --git a/src/main.rs b/src/main.rs
index 53ac48d..0ab2943 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -24,6 +24,7 @@ use tokio_stream::StreamExt;
 
 use proxmox_http::client::HttpsConnector;
 use proxmox_http::websocket::{OpCode, WebSocket, WebSocketReader, WebSocketWriter};
+use proxmox_http::SslVerifyError;
 
 #[derive(Serialize, Deserialize, Debug)]
 #[serde(rename_all = "kebab-case")]
@@ -140,48 +141,35 @@ impl CtrlTunnel {
         }
 
         let mut ssl_connector_builder = SslConnector::builder(SslMethod::tls())?;
-        if let Some(expected) = fingerprint {
+        if fingerprint.is_some() {
             ssl_connector_builder.set_verify_callback(
                 openssl::ssl::SslVerifyMode::PEER,
-                move |_valid, ctx| {
-                    let cert = match ctx.current_cert() {
-                        Some(cert) => cert,
-                        None => {
-                            // should not happen
-                            eprintln!("SSL context lacks current certificate.");
-                            return false;
-                        }
-                    };
-
-                    // skip CA certificates, we only care about the peer cert
-                    let depth = ctx.error_depth();
-                    if depth != 0 {
-                        return true;
-                    }
-
-                    use itertools::Itertools;
-                    let fp = match cert.digest(openssl::hash::MessageDigest::sha256()) {
-                        Ok(fp) => fp,
-                        Err(err) => {
-                            // should not happen
-                            eprintln!("failed to calculate certificate FP - {}", err);
-                            return false;
+                move |valid, ctx| match proxmox_http::openssl_verify_callback(
+                    valid,
+                    ctx,
+                    fingerprint.as_deref(),
+                ) {
+                    Ok(()) => true,
+                    Err(err) => {
+                        match err {
+                            SslVerifyError::NoCertificate => {
+                                eprintln!("SSL context lacks current certificate");
+                            }
+                            SslVerifyError::InvalidFingerprint(err) => {
+                                eprintln!("failed to calculate certificate FP - {err}")
+                            }
+                            SslVerifyError::FingerprintMismatch {
+                                fingerprint,
+                                expected,
+                            } => {
+                                eprintln!(
+                                    "certificate fingerprint does not match expected fingerprint!"
+                                );
+                                eprintln!("expected:    {expected}");
+                                eprintln!("encountered: {fingerprint}");
+                            }
+                            SslVerifyError::UntrustedCertificate { .. } => {}
                         }
-                    };
-                    let fp_string = hex::encode(fp);
-                    let fp_string = fp_string
-                        .as_bytes()
-                        .chunks(2)
-                        .map(|v| unsafe { std::str::from_utf8_unchecked(v) })
-                        .join(":");
-
-                    let expected = expected.to_lowercase();
-                    if expected == fp_string {
-                        true
-                    } else {
-                        eprintln!("certificate fingerprint does not match expected fingerprint!");
-                        eprintln!("expected:    {}", expected);
-                        eprintln!("encountered: {}", fp_string);
                         false
                     }
                 },
-- 
2.39.5




From d.csapak at proxmox.com  Wed May 21 10:45:19 2025
From: d.csapak at proxmox.com (Dominik Csapak)
Date: Wed, 21 May 2025 10:45:19 +0200
Subject: [pbs-devel] [PATCH proxmox{, -websocket-tunnel,
 -backup} 0/5] unify openssl callback logic
Message-ID: <20250521084524.829496-1-d.csapak@proxmox.com>

There are currently 3 slightly different implementations of the openssl
verify callback in place. They differ in how an explicit fingerprint
would be checked:

* pbs-client: if verification was on, a valid certificate would trump a
  wrong epxlicit fingerprint
* proxmox-websocket-tunnel: if an explicit fingerprint was given, it was
  checked, regardless of the openssl result
* proxmox-client: the openssl validity had priority as in pbs-client,
  but the fingerprint was not checked against the leaf certificate, but
  agains all certificates in the chain (which would lead to false
  negatives). Note that this is currently only used in PDM

This series aims to unify the general behavior, but design the interface
to be flexible enought to accomodate the different call sites needs.

I included the change of features for crates, but they have to be bumped
before hand of course and the version must be changed in Cargo.toml.

(if I should send that differently, please do tell how it should be done)

Since that is technically a breaking change for PBS, we should only
change that for the next major release.

Also, since it rather deep in the stack for PBS (remotes sync, etc.) and
PVE (remote migration) IMHO this is a series that should be tested very
well.

Further work could be to unify this behavior for our perl clients too,
but it seemed out of scope for this series. (notably the PVE::APIClient
and the client used in the SDN code)

I tried to implement some tests, but due to the openssl interface this
seems to be not really possible, except if we'd start a server + client
in the tests (which seems overkill). But if anyone has an idea how we
could test this code (and i mean not only it's interface, but the
openssl connection behavior), I'd be glad.

I sent the patch to the pbs-devel list, but actually it affects PBS, PVE
and PDM.

patch 1/2 of the websocket-tunnel, is not really related, but was
necessary to build.

proxmox:

Dominik Csapak (2):
  http: factor out openssl verification callback
  client: use proxmox-http's openssl verification callback

 Cargo.toml                   |  1 +
 proxmox-client/Cargo.toml    |  2 +-
 proxmox-client/src/client.rs | 48 ++++---------------
 proxmox-http/Cargo.toml      |  7 +++
 proxmox-http/src/lib.rs      |  5 ++
 proxmox-http/src/tls.rs      | 89 ++++++++++++++++++++++++++++++++++++
 proxmox-openid/Cargo.toml    |  2 +-
 7 files changed, 112 insertions(+), 42 deletions(-)
 create mode 100644 proxmox-http/src/tls.rs


proxmox-websocket-tunnel:

Dominik Csapak (2):
  update base64 dependency
  use proxmox-http's openssl callback

 Cargo.toml  |  5 ++--
 src/main.rs | 66 ++++++++++++++++++++++-------------------------------
 2 files changed, 29 insertions(+), 42 deletions(-)


proxmox-backup:

Dominik Csapak (1):
  pbs-client: use proxmox-https openssl callback

 Cargo.toml                    |   2 +-
 pbs-client/src/http_client.rs | 151 ++++++++++++++--------------------
 2 files changed, 62 insertions(+), 91 deletions(-)


Summary over all repositories:
  11 files changed, 203 insertions(+), 175 deletions(-)

-- 
Generated by git-murpp 0.8.1



From d.csapak at proxmox.com  Wed May 21 10:45:21 2025
From: d.csapak at proxmox.com (Dominik Csapak)
Date: Wed, 21 May 2025 10:45:21 +0200
Subject: [pbs-devel] [PATCH proxmox 2/2] client: use proxmox-http's openssl
 verification callback
In-Reply-To: <20250521084524.829496-1-d.csapak@proxmox.com>
References: <20250521084524.829496-1-d.csapak@proxmox.com>
Message-ID: <20250521084524.829496-3-d.csapak@proxmox.com>

This changes the validation logic by always checking the fingerprint of
the leaf certificate, ignoring the openssl verification if a fingerprint
is configured. This now aligns with our perl implementation and the one
for proxmox-websocket-tunnel.

Before, a valid certificate chain would have precedence over an explicit
fingerprint.

Additionally, fingerprints will be correctly checked only against the
leaf certificate instead of every part of the chain for which the
callback is called.

Signed-off-by: Dominik Csapak <d.csapak at proxmox.com>
---
 proxmox-client/Cargo.toml    |  2 +-
 proxmox-client/src/client.rs | 48 ++++++------------------------------
 2 files changed, 9 insertions(+), 41 deletions(-)

diff --git a/proxmox-client/Cargo.toml b/proxmox-client/Cargo.toml
index c2682e77..477e9393 100644
--- a/proxmox-client/Cargo.toml
+++ b/proxmox-client/Cargo.toml
@@ -24,7 +24,7 @@ openssl = { workspace = true, optional = true }
 
 proxmox-login = { workspace = true, features = [ "http" ] }
 
-proxmox-http = { workspace = true, optional = true, features = [ "client" ] }
+proxmox-http = { workspace = true, optional = true, features = [ "client", "tls" ] }
 hyper = { workspace = true, optional = true }
 
 [dev-dependencies]
diff --git a/proxmox-client/src/client.rs b/proxmox-client/src/client.rs
index 6f1c9ef1..ce0c4841 100644
--- a/proxmox-client/src/client.rs
+++ b/proxmox-client/src/client.rs
@@ -9,9 +9,9 @@ use http::uri::PathAndQuery;
 use http::Method;
 use http::{StatusCode, Uri};
 use hyper::body::{Body, HttpBody};
-use openssl::hash::MessageDigest;
 use openssl::ssl::{SslConnector, SslMethod, SslVerifyMode};
 use openssl::x509::{self, X509};
+use proxmox_http::get_fingerprint_from_u8;
 use proxmox_login::Ticket;
 use serde::Serialize;
 
@@ -109,10 +109,14 @@ impl Client {
             TlsOptions::Insecure => connector.set_verify(SslVerifyMode::NONE),
             TlsOptions::Fingerprint(expected_fingerprint) => {
                 connector.set_verify_callback(SslVerifyMode::PEER, move |valid, chain| {
-                    if valid {
-                        return true;
+                    let fp = get_fingerprint_from_u8(&expected_fingerprint);
+                    match proxmox_http::openssl_verify_callback(valid, chain, Some(&fp)) {
+                        Ok(()) => true,
+                        Err(err) => {
+                            log::error!("{err}");
+                            false
+                        }
                     }
-                    verify_fingerprint(chain, &expected_fingerprint)
                 });
             }
             TlsOptions::Callback(cb) => {
@@ -543,42 +547,6 @@ impl HttpApiClient for Client {
     }
 }
 
-fn verify_fingerprint(chain: &x509::X509StoreContextRef, expected_fingerprint: &[u8]) -> bool {
-    let Some(cert) = chain.current_cert() else {
-        log::error!("no certificate in chain?");
-        return false;
-    };
-
-    let fp = match cert.digest(MessageDigest::sha256()) {
-        Err(err) => {
-            log::error!("error calculating certificate fingerprint: {err}");
-            return false;
-        }
-        Ok(fp) => fp,
-    };
-
-    if expected_fingerprint != fp.as_ref() {
-        log::error!("bad fingerprint: {}", fp_string(&fp));
-        log::error!("expected fingerprint: {}", fp_string(expected_fingerprint));
-        return false;
-    }
-
-    true
-}
-
-fn fp_string(fp: &[u8]) -> String {
-    use std::fmt::Write as _;
-
-    let mut out = String::new();
-    for b in fp {
-        if !out.is_empty() {
-            out.push(':');
-        }
-        let _ = write!(out, "{b:02x}");
-    }
-    out
-}
-
 impl Error {
     pub(crate) fn internal<E>(context: &'static str, err: E) -> Self
     where
-- 
2.39.5




From d.csapak at proxmox.com  Wed May 21 10:45:20 2025
From: d.csapak at proxmox.com (Dominik Csapak)
Date: Wed, 21 May 2025 10:45:20 +0200
Subject: [pbs-devel] [PATCH proxmox 1/2] http: factor out openssl
 verification callback
In-Reply-To: <20250521084524.829496-1-d.csapak@proxmox.com>
References: <20250521084524.829496-1-d.csapak@proxmox.com>
Message-ID: <20250521084524.829496-2-d.csapak@proxmox.com>

with the 'tls' feature offers a callback method that can be used within
openssl's `set_verify_callback` with a given expected fingerprint.

The logic is inspired by our perl and proxmox-websocket-tunnel
verification logic:

Use openssl's verification if no fingerprint is pinned. If a fingerprint
is given, ignore openssl's verification and check if the leafs
certificate is a match.

This introduces a custom error type for this, since we need to handle
errors differently for different users, e.g. pbs-client wants to be able
to use a fingerprint cache and let the user accept it in interactive cli
sessions. For this we want the 'thiserror' crate, so move it to the
workspace Cargo.toml and depend from there. (also change this for
proxmox-openid)

One thing to note here is that  the APPLICATION_VERIFICATION error of
openssl is used to mark the case where an untrusted root or intermediate
certificate is trusted from the callback. When that happens, openssl
might return true for the following certificates (if nothing else is
wrong aside from a missing trust anchor), so the error is checked for
this special value to determine if the openssl validation can be
trusted.

Signed-off-by: Dominik Csapak <d.csapak at proxmox.com>
---
 Cargo.toml                |  1 +
 proxmox-http/Cargo.toml   |  7 +++
 proxmox-http/src/lib.rs   |  5 +++
 proxmox-http/src/tls.rs   | 89 +++++++++++++++++++++++++++++++++++++++
 proxmox-openid/Cargo.toml |  2 +-
 5 files changed, 103 insertions(+), 1 deletion(-)
 create mode 100644 proxmox-http/src/tls.rs

diff --git a/Cargo.toml b/Cargo.toml
index 95ade7d7..8aecc6ba 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -107,6 +107,7 @@ serde_json = "1.0"
 serde_plain = "1.0"
 syn = { version = "2", features = [ "full", "visit-mut" ] }
 tar = "0.4"
+thiserror = "1"
 tokio = "1.6"
 tokio-openssl = "0.6.1"
 tokio-stream = "0.1.0"
diff --git a/proxmox-http/Cargo.toml b/proxmox-http/Cargo.toml
index afa5981d..fb2eb26d 100644
--- a/proxmox-http/Cargo.toml
+++ b/proxmox-http/Cargo.toml
@@ -15,11 +15,13 @@ rust-version.workspace = true
 anyhow.workspace = true
 base64 = { workspace = true, optional = true }
 futures = { workspace = true, optional = true }
+hex = { workspace = true, optional = true }
 http = { workspace = true, optional = true }
 hyper = { workspace = true, optional = true }
 native-tls = { workspace = true, optional = true }
 openssl =  { version = "0.10", optional = true }
 serde_json = { workspace = true, optional = true }
+thiserror = { workspace = true, optional = true }
 tokio = { workspace = true, features = [], optional = true }
 tokio-openssl = { workspace = true, optional = true }
 tower-service.workspace = true
@@ -79,3 +81,8 @@ websocket = [
     "tokio?/io-util",
     "tokio?/sync",
 ]
+tls = [
+    "dep:hex",
+    "dep:openssl",
+    "dep:thiserror",
+]
diff --git a/proxmox-http/src/lib.rs b/proxmox-http/src/lib.rs
index 4770aaf4..a676acf9 100644
--- a/proxmox-http/src/lib.rs
+++ b/proxmox-http/src/lib.rs
@@ -35,3 +35,8 @@ pub use rate_limiter::{RateLimit, RateLimiter, RateLimiterVec, ShareableRateLimi
 mod rate_limited_stream;
 #[cfg(feature = "rate-limited-stream")]
 pub use rate_limited_stream::RateLimitedStream;
+
+#[cfg(feature = "tls")]
+mod tls;
+#[cfg(feature = "tls")]
+pub use tls::*;
diff --git a/proxmox-http/src/tls.rs b/proxmox-http/src/tls.rs
new file mode 100644
index 00000000..6da03cd9
--- /dev/null
+++ b/proxmox-http/src/tls.rs
@@ -0,0 +1,89 @@
+use openssl::x509::{X509StoreContextRef, X509VerifyResult};
+
+///
+/// Error type returned by failed [`openssl_verify_callback`].
+///
+#[derive(Debug, thiserror::Error)]
+pub enum SslVerifyError {
+    /// Occurs if no certificate is found in the current part of the chain. Should never happen!
+    #[error("SSL context lacks current certificate")]
+    NoCertificate,
+
+    /// Cannot calculate fingerprint from connection
+    #[error("failed to calculate fingerprint - {0}")]
+    InvalidFingerprint(openssl::error::ErrorStack),
+
+    /// Fingerprint match error
+    #[error("found fingerprint ({fingerprint}) does not match expected fingerprint ({expected})")]
+    FingerprintMismatch {
+        fingerprint: String,
+        expected: String,
+    },
+
+    /// Untrusted certificate with fingerprint information
+    #[error("certificate validation failed")]
+    UntrustedCertificate { fingerprint: String },
+}
+
+/// Intended as an openssl verification callback.
+///
+/// The following things are checked:
+///
+/// * If no fingerprint is given, return the openssl verification result
+/// * If a fingerprint is given, do:
+///     * Ignore all non-leaf certificates/
+pub fn openssl_verify_callback(
+    openssl_valid: bool,
+    ctx: &mut X509StoreContextRef,
+    expected_fp: Option<&str>,
+) -> Result<(), SslVerifyError> {
+    let trust_openssl = ctx.error() != X509VerifyResult::APPLICATION_VERIFICATION;
+    if expected_fp.is_none() && openssl_valid && trust_openssl {
+        return Ok(());
+    }
+
+    let cert = match ctx.current_cert() {
+        Some(cert) => cert,
+        None => {
+            return Err(SslVerifyError::NoCertificate);
+        }
+    };
+
+    if ctx.error_depth() > 0 {
+        // openssl was not valid, but we want to continue, so save that we don't trust openssl
+        ctx.set_error(X509VerifyResult::APPLICATION_VERIFICATION);
+        return Ok(());
+    }
+
+    let digest = cert
+        .digest(openssl::hash::MessageDigest::sha256())
+        .map_err(SslVerifyError::InvalidFingerprint)?;
+    let fingerprint = get_fingerprint_from_u8(&digest);
+
+    if let Some(expected_fp) = expected_fp {
+        if expected_fp.to_lowercase() == fingerprint.to_lowercase() {
+            ctx.set_error(X509VerifyResult::OK);
+            Ok(())
+        } else {
+            Err(SslVerifyError::FingerprintMismatch {
+                fingerprint,
+                expected: expected_fp.to_string(),
+            })
+        }
+    } else {
+        Err(SslVerifyError::UntrustedCertificate { fingerprint })
+    }
+}
+
+/// Returns the fingerprint from a byte slice ([`&[u8]`]) in the form `00:11:22:...`
+pub fn get_fingerprint_from_u8(fp: &[u8]) -> String {
+    let fp_string = hex::encode(fp);
+    let fp_string = fp_string
+        .as_bytes()
+        .chunks(2)
+        .map(|v| unsafe { std::str::from_utf8_unchecked(v) })
+        .collect::<Vec<_>>()
+        .join(":");
+
+    fp_string
+}
diff --git a/proxmox-openid/Cargo.toml b/proxmox-openid/Cargo.toml
index a5249214..4223d10c 100644
--- a/proxmox-openid/Cargo.toml
+++ b/proxmox-openid/Cargo.toml
@@ -18,7 +18,7 @@ http.workspace = true
 nix.workspace = true
 serde = { workspace = true, features = ["derive"] }
 serde_json.workspace = true
-thiserror = "1"
+thiserror.workspace = true
 native-tls.workspace = true
 
 openidconnect = { version = "2.4", default-features = false, features = ["accept-rfc3339-timestamps"] }
-- 
2.39.5




From d.csapak at proxmox.com  Wed May 21 10:45:24 2025
From: d.csapak at proxmox.com (Dominik Csapak)
Date: Wed, 21 May 2025 10:45:24 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 1/1] pbs-client: use
 proxmox-https openssl callback
In-Reply-To: <20250521084524.829496-1-d.csapak@proxmox.com>
References: <20250521084524.829496-1-d.csapak@proxmox.com>
Message-ID: <20250521084524.829496-6-d.csapak@proxmox.com>

instead of implementing it here. This changes the behavior when giving a
fingerprint explicitly when the certificate chain is trusted by openssl.
Previously this would be accepted due to openssls checks, regardless if
the given fingerprint would match or not.

With this patch, a given fingerprint has higher priority than openssls
validation.

Signed-off-by: Dominik Csapak <d.csapak at proxmox.com>
---
 Cargo.toml                    |   2 +-
 pbs-client/src/http_client.rs | 151 ++++++++++++++--------------------
 2 files changed, 62 insertions(+), 91 deletions(-)

diff --git a/Cargo.toml b/Cargo.toml
index 6de6a6527..c6076c9a6 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -62,7 +62,7 @@ proxmox-compression = "0.2"
 proxmox-config-digest = "0.1.0"
 proxmox-daemon = "0.1.0"
 proxmox-fuse = "0.1.3"
-proxmox-http = { version = "0.9.5", features = [ "client", "http-helpers", "websocket" ] } # see below
+proxmox-http = { version = "0.9.5", features = [ "client", "http-helpers", "websocket", "tls" ] } # see below
 proxmox-human-byte = "0.1"
 proxmox-io = "1.0.1" # tools and client use "tokio" feature
 proxmox-lang = "1.1"
diff --git a/pbs-client/src/http_client.rs b/pbs-client/src/http_client.rs
index c95def07b..4356af210 100644
--- a/pbs-client/src/http_client.rs
+++ b/pbs-client/src/http_client.rs
@@ -11,10 +11,7 @@ use hyper::http::header::HeaderValue;
 use hyper::http::Uri;
 use hyper::http::{Request, Response};
 use hyper::{body::HttpBody, Body};
-use openssl::{
-    ssl::{SslConnector, SslMethod},
-    x509::X509StoreContextRef,
-};
+use openssl::ssl::{SslConnector, SslMethod};
 use percent_encoding::percent_encode;
 use serde_json::{json, Value};
 use xdg::BaseDirectories;
@@ -26,7 +23,7 @@ use proxmox_sys::linux::tty;
 use proxmox_async::broadcast_future::BroadcastFuture;
 use proxmox_http::client::HttpsConnector;
 use proxmox_http::uri::{build_authority, json_object_to_query};
-use proxmox_http::{ProxyConfig, RateLimiter};
+use proxmox_http::{openssl_verify_callback, ProxyConfig, RateLimiter, SslVerifyError};
 use proxmox_log::{error, info, warn};
 
 use pbs_api_types::percent_encoding::DEFAULT_ENCODE_SET;
@@ -403,30 +400,42 @@ impl HttpClient {
             let interactive = options.interactive;
             let fingerprint_cache = options.fingerprint_cache;
             let prefix = options.prefix.clone();
-            let trust_openssl_valid = Arc::new(Mutex::new(true));
             ssl_connector_builder.set_verify_callback(
                 openssl::ssl::SslVerifyMode::PEER,
-                move |valid, ctx| match Self::verify_callback(
+                move |valid, ctx| match openssl_verify_callback(
                     valid,
                     ctx,
-                    expected_fingerprint.as_ref(),
-                    interactive,
-                    Arc::clone(&trust_openssl_valid),
+                    expected_fingerprint.as_deref(),
                 ) {
-                    Ok(None) => true,
-                    Ok(Some(fingerprint)) => {
-                        if fingerprint_cache && prefix.is_some() {
-                            if let Err(err) =
-                                store_fingerprint(prefix.as_ref().unwrap(), &server, &fingerprint)
-                            {
-                                error!("{}", err);
+                    Ok(()) => true,
+                    Err(err) => {
+                        match err {
+                            SslVerifyError::NoCertificate => error!(
+                                "certificate validation failed - context lacks current certificate"
+                            ),
+                            SslVerifyError::InvalidFingerprint(error_stack) => {
+                                error!("certificate validation failed - failed to calculate FP - {error_stack}")
+                            },
+                            SslVerifyError::UntrustedCertificate { fingerprint } => {
+                                if interactive && std::io::stdin().is_terminal() {
+                                    match Self::interactive_fp_check(prefix.as_deref(), &server, verified_fingerprint.clone(), fingerprint_cache, fingerprint) {
+                                        Ok(()) => return true,
+                                        Err(err) => error!("certificate validation failed - {err}"),
+                                    }
+                                }
                             }
+                            SslVerifyError::FingerprintMismatch { fingerprint, expected } => {
+                                warn!("WARNING: certificate fingerprint does not match expected fingerprint!");
+                                warn!("expected:    {expected}");
+
+                                if interactive && std::io::stdin().is_terminal() {
+                                    match Self::interactive_fp_check(prefix.as_deref(), &server, verified_fingerprint.clone(), fingerprint_cache, fingerprint) {
+                                        Ok(()) => return true,
+                                        Err(err) => error!("certificate validation failed - {err}"),
+                                    }
+                                }
+                            },
                         }
-                        *verified_fingerprint.lock().unwrap() = Some(fingerprint);
-                        true
-                    }
-                    Err(err) => {
-                        error!("certificate validation failed - {}", err);
                         false
                     }
                 },
@@ -631,79 +640,41 @@ impl HttpClient {
         bail!("no password input mechanism available");
     }
 
-    fn verify_callback(
-        openssl_valid: bool,
-        ctx: &mut X509StoreContextRef,
-        expected_fingerprint: Option<&String>,
-        interactive: bool,
-        trust_openssl: Arc<Mutex<bool>>,
-    ) -> Result<Option<String>, Error> {
-        let mut trust_openssl_valid = trust_openssl.lock().unwrap();
-
-        // we can only rely on openssl's prevalidation if we haven't forced it earlier
-        if openssl_valid && *trust_openssl_valid {
-            return Ok(None);
-        }
-
-        let cert = match ctx.current_cert() {
-            Some(cert) => cert,
-            None => bail!("context lacks current certificate."),
-        };
-
-        // force trust in case of a chain, but set flag to no longer trust prevalidation by openssl
-        if ctx.error_depth() > 0 {
-            *trust_openssl_valid = false;
-            return Ok(None);
-        }
-
-        // leaf certificate - if we end up here, we have to verify the fingerprint!
-        let fp = match cert.digest(openssl::hash::MessageDigest::sha256()) {
-            Ok(fp) => fp,
-            Err(err) => bail!("failed to calculate certificate FP - {}", err), // should not happen
-        };
-        let fp_string = hex::encode(fp);
-        let fp_string = fp_string
-            .as_bytes()
-            .chunks(2)
-            .map(|v| std::str::from_utf8(v).unwrap())
-            .collect::<Vec<&str>>()
-            .join(":");
-
-        if let Some(expected_fingerprint) = expected_fingerprint {
-            let expected_fingerprint = expected_fingerprint.to_lowercase();
-            if expected_fingerprint == fp_string {
-                return Ok(Some(fp_string));
-            } else {
-                warn!("WARNING: certificate fingerprint does not match expected fingerprint!");
-                warn!("expected:    {}", expected_fingerprint);
-            }
-        }
-
-        // If we're on a TTY, query the user
-        if interactive && std::io::stdin().is_terminal() {
-            info!("fingerprint: {}", fp_string);
-            loop {
-                eprint!("Are you sure you want to continue connecting? (y/n): ");
-                let _ = std::io::stdout().flush();
-                use std::io::{BufRead, BufReader};
-                let mut line = String::new();
-                match BufReader::new(std::io::stdin()).read_line(&mut line) {
-                    Ok(_) => {
-                        let trimmed = line.trim();
-                        if trimmed == "y" || trimmed == "Y" {
-                            return Ok(Some(fp_string));
-                        } else if trimmed == "n" || trimmed == "N" {
-                            bail!("Certificate fingerprint was not confirmed.");
-                        } else {
-                            continue;
+    fn interactive_fp_check(
+        prefix: Option<&str>,
+        server: &str,
+        verified_fingerprint: Arc<Mutex<Option<String>>>,
+        fingerprint_cache: bool,
+        fingerprint: String,
+    ) -> Result<(), Error> {
+        info!("fingerprint: {fingerprint}");
+        loop {
+            eprint!("Are you sure you want to continue connecting? (y/n): ");
+            let _ = std::io::stdout().flush();
+            use std::io::{BufRead, BufReader};
+            let mut line = String::new();
+            match BufReader::new(std::io::stdin()).read_line(&mut line) {
+                Ok(_) => {
+                    let trimmed = line.trim();
+                    if trimmed == "y" || trimmed == "Y" {
+                        if fingerprint_cache && prefix.is_some() {
+                            if let Err(err) =
+                                store_fingerprint(prefix.unwrap(), server, &fingerprint)
+                            {
+                                error!("{}", err);
+                            }
                         }
+                        *verified_fingerprint.lock().unwrap() = Some(fingerprint);
+                        return Ok(());
+                    } else if trimmed == "n" || trimmed == "N" {
+                        bail!("Certificate fingerprint was not confirmed.");
+                    } else {
+                        continue;
                     }
-                    Err(err) => bail!("Certificate fingerprint was not confirmed - {}.", err),
                 }
+                Err(err) => bail!("Certificate fingerprint was not confirmed - {}.", err),
             }
         }
-
-        bail!("Certificate fingerprint was not confirmed.");
     }
 
     pub async fn request(&self, mut req: Request<Body>) -> Result<Value, Error> {
-- 
2.39.5




From l.wagner at proxmox.com  Wed May 21 16:23:05 2025
From: l.wagner at proxmox.com (Lukas Wagner)
Date: Wed, 21 May 2025 16:23:05 +0200
Subject: [pbs-devel] [PATCH proxmox{, -widget-toolkit,
 -backup} 0/4] notifications: add support for nested matchers
Message-ID: <20250521142309.264719-1-l.wagner@proxmox.com>

This series adds support for nested notification matchers. A new match rule is
added, 'eval-matcher', which allows to evaluate other matchers and use their
result in the current matcher.

Any matcher that shall be used as a nested matcher must have the (new) `nested`
flag set to true. These matchers are only evaluated if they are referenced
by another matcher. Any target configured for a nested matcher is ignored,
only the 'top-level'/'outermost' matcher decides which targets to notify.

This patch series includes:
  - patches for proxmox-notify, which add the new config entries for matchers,
    API methods and nested matcher evaluation
  - patches for proxmox-backup, which contain documentation for the new
    feature
  - patches for proxmox-widget-toolkit, containing the GUI changes needed
    for this feature.

Small config example:

matcher: top-level
  target mail-to-root
  mode any
  eval-matcher a
  eval-matcher b

matcher: a
  nested true
  mode all
  match-field exact:datastore=store
  match-field exact:type=gc
  match-severity error

matcher: b
  nested true
  mode all
  match-field exact:datastore=store
  match-field exact:type=prune
  match-severity error

The GUI remains functional if changes of proxmox-widget-toolkit are rolled out
before the proxmox-notify/proxmox-backup ones, so we should get away without
any sort of versioned break between widget-toolkit and proxmox-backup. Matchers
can be added/modified/deleted without problems, only if one tries to add a
nested-matcher rule or declare a matcher as a nested matcher an error is
displayed. Everything else should work.

Patches for PVE will follow once the core elements for this series have been
reviewed.

Note:
  The patches for proxmox-notify are based on the `stable-bookworm` branch.

## Patch series revisions:
v1: you are here

proxmox:

Lukas Wagner (2):
  notify: matcher: allow to evaluate other matchers via `eval-matcher`
  notify: matcher: API: update methods for nested matcher support

 proxmox-notify/src/api/matcher.rs | 229 +++++++++++++++++++++-
 proxmox-notify/src/api/mod.rs     |   4 +
 proxmox-notify/src/lib.rs         |  20 +-
 proxmox-notify/src/matcher.rs     | 313 ++++++++++++++++++++++++++++--
 4 files changed, 532 insertions(+), 34 deletions(-)


proxmox-widget-toolkit:

Lukas Wagner (1):
  notifications: matchers: add nested matcher support

 src/data/model/NotificationConfig.js  |   8 +-
 src/panel/NotificationConfigView.js   |   6 +
 src/window/NotificationMatcherEdit.js | 220 +++++++++++++++++++++++++-
 3 files changed, 232 insertions(+), 2 deletions(-)


proxmox-backup:

Lukas Wagner (1):
  docs: notifications: add section about nested matchers

 docs/notifications.rst | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)


Summary over all repositories:
  8 files changed, 787 insertions(+), 36 deletions(-)

-- 
Generated by git-murpp 0.8.1



From l.wagner at proxmox.com  Wed May 21 16:23:09 2025
From: l.wagner at proxmox.com (Lukas Wagner)
Date: Wed, 21 May 2025 16:23:09 +0200
Subject: [pbs-devel] [PATCH proxmox-backup 1/1] docs: notifications: add
 section about nested matchers
In-Reply-To: <20250521142309.264719-1-l.wagner@proxmox.com>
References: <20250521142309.264719-1-l.wagner@proxmox.com>
Message-ID: <20250521142309.264719-5-l.wagner@proxmox.com>

Signed-off-by: Lukas Wagner <l.wagner at proxmox.com>
---
 docs/notifications.rst | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/docs/notifications.rst b/docs/notifications.rst
index 5669dab0..5b990394 100644
--- a/docs/notifications.rst
+++ b/docs/notifications.rst
@@ -258,6 +258,29 @@ Examples:
 The following severities are in use:
 ``info``, ``notice``, ``warning``, ``error``, ``unknown``.
 
+Nested Matcher Rules
+^^^^^^^^^^^^^^^^^^^^
+
+By using ``eval-matcher <other-matcher-name>`` rules, a matcher can evaluate
+other matchers that were marked as a nested matcher. This allows to build
+arbitrarily complex matching rules.
+
+Any matcher that shall be used as a nested matcher must have the ``nested``
+flag set. Nested matchers do not have any associated notification targets. They
+are only evaluated if they are referenced by another matcher's ``eval-matcher``
+rule. If a nested matcher is disabled, it will not be evaluated even if
+referenced by another matcher.
+
+A nested matcher itself can evaluate other nested matchers. Direct (a matcher
+evaluates itself) and indirect (two or more matchers evaluate each other)
+recursion is not allowed.
+
+Examples:
+
+* ``eval-matcher sub-1``: Evaluates the ``sub-1`` matcher and uses its result
+  to compute the result of the current matcher.
+
+
 .. _notification_events:
 
 Notification Events
-- 
2.39.5




From l.wagner at proxmox.com  Wed May 21 16:23:07 2025
From: l.wagner at proxmox.com (Lukas Wagner)
Date: Wed, 21 May 2025 16:23:07 +0200
Subject: [pbs-devel] [PATCH proxmox 2/2] notify: matcher: API: update
 methods for nested matcher support
In-Reply-To: <20250521142309.264719-1-l.wagner@proxmox.com>
References: <20250521142309.264719-1-l.wagner@proxmox.com>
Message-ID: <20250521142309.264719-3-l.wagner@proxmox.com>

This commit adds support for setting the two new configuration keys for
matchers, `eval-matcher` and `nested`. Some checks are added to prevent
invalid configs which would lead to an error when a matcher is evaluated
(e.g. recursion).

Signed-off-by: Lukas Wagner <l.wagner at proxmox.com>
---
 proxmox-notify/src/api/matcher.rs | 229 ++++++++++++++++++++++++++++--
 proxmox-notify/src/api/mod.rs     |   4 +
 proxmox-notify/src/matcher.rs     |   4 +
 3 files changed, 228 insertions(+), 9 deletions(-)

diff --git a/proxmox-notify/src/api/matcher.rs b/proxmox-notify/src/api/matcher.rs
index f5605acb..4ab837c5 100644
--- a/proxmox-notify/src/api/matcher.rs
+++ b/proxmox-notify/src/api/matcher.rs
@@ -1,10 +1,13 @@
+use std::collections::HashMap;
+use std::ops::Deref;
+
 use proxmox_http_error::HttpError;
 
 use crate::api::http_err;
 use crate::matcher::{
     DeleteableMatcherProperty, MatcherConfig, MatcherConfigUpdater, MATCHER_TYPENAME,
 };
-use crate::Config;
+use crate::{http_bail, Config};
 
 /// Get a list of all matchers
 ///
@@ -39,6 +42,7 @@ pub fn get_matcher(config: &Config, name: &str) -> Result<MatcherConfig, HttpErr
 pub fn add_matcher(config: &mut Config, matcher_config: MatcherConfig) -> Result<(), HttpError> {
     super::ensure_unique(config, &matcher_config.name)?;
     super::ensure_endpoints_exist(config, &matcher_config.target)?;
+    ensure_no_recursion(config, &matcher_config)?;
 
     config
         .config
@@ -83,43 +87,73 @@ pub fn update_matcher(
                 DeleteableMatcherProperty::InvertMatch => matcher.invert_match = None,
                 DeleteableMatcherProperty::Comment => matcher.comment = None,
                 DeleteableMatcherProperty::Disable => matcher.disable = None,
+                DeleteableMatcherProperty::EvalMatcher => matcher.eval_matcher.clear(),
+                DeleteableMatcherProperty::Nested => matcher.nested = None,
             }
         }
     }
 
-    if let Some(match_severity) = matcher_updater.match_severity {
+    let MatcherConfigUpdater {
+        match_severity,
+        match_field,
+        match_calendar,
+        mode,
+        invert_match,
+        nested,
+        eval_matcher,
+        comment,
+        disable,
+        target,
+    } = matcher_updater;
+
+    if let Some(match_severity) = match_severity {
         matcher.match_severity = match_severity;
     }
 
-    if let Some(match_field) = matcher_updater.match_field {
+    if let Some(match_field) = match_field {
         matcher.match_field = match_field;
     }
 
-    if let Some(match_calendar) = matcher_updater.match_calendar {
+    if let Some(match_calendar) = match_calendar {
         matcher.match_calendar = match_calendar;
     }
 
-    if let Some(mode) = matcher_updater.mode {
+    if let Some(mode) = mode {
         matcher.mode = Some(mode);
     }
 
-    if let Some(invert_match) = matcher_updater.invert_match {
+    if let Some(invert_match) = invert_match {
         matcher.invert_match = Some(invert_match);
     }
 
-    if let Some(comment) = matcher_updater.comment {
+    if let Some(comment) = comment {
         matcher.comment = Some(comment);
     }
 
-    if let Some(disable) = matcher_updater.disable {
+    if let Some(disable) = disable {
         matcher.disable = Some(disable);
     }
 
-    if let Some(target) = matcher_updater.target {
+    if let Some(nested) = nested {
+        matcher.nested = Some(nested);
+    }
+
+    if let Some(eval_matcher) = eval_matcher {
+        ensure_matchers_exist(config, eval_matcher.iter().map(Deref::deref))?;
+        matcher.eval_matcher = eval_matcher;
+    }
+
+    if let Some(target) = target {
         super::ensure_endpoints_exist(config, target.as_slice())?;
         matcher.target = target;
     }
 
+    ensure_no_recursion(config, &matcher)?;
+
+    if !matcher.nested.unwrap_or_default() {
+        ensure_not_referenced_by_other_matcher(config, &matcher.name)?;
+    }
+
     config
         .config
         .set_data(name, MATCHER_TYPENAME, &matcher)
@@ -142,12 +176,92 @@ pub fn update_matcher(
 pub fn delete_matcher(config: &mut Config, name: &str) -> Result<(), HttpError> {
     // Check if the matcher exists
     let _ = get_matcher(config, name)?;
+    super::ensure_safe_to_delete(config, name)?;
 
     config.config.sections.remove(name);
 
     Ok(())
 }
 
+/// Ensure that a matcher, identified by it's name, is not referenced by another matcher's
+/// `eval-matcher` statement.
+fn ensure_not_referenced_by_other_matcher(config: &Config, name: &str) -> Result<(), HttpError> {
+    let referrers = super::get_referrers(config, name)?;
+
+    if !referrers.is_empty() {
+        let used_by = referrers.into_iter().collect::<Vec<_>>().join(", ");
+        http_bail!(
+            BAD_REQUEST,
+            "can't remove 'nested' flag, matcher is referenced by: {used_by}"
+        );
+    }
+
+    Ok(())
+}
+
+/// Ensure that adding the matcher in `new_entry` to `config` would not create
+/// any direct or indirect recursion.
+fn ensure_no_recursion(config: &Config, new_entry: &MatcherConfig) -> Result<(), HttpError> {
+    let all_matchers = get_matchers(config)?;
+
+    let mut map: HashMap<String, &MatcherConfig> =
+        HashMap::from_iter(all_matchers.iter().map(|i| (i.name.clone(), i)));
+    map.insert(new_entry.name.clone(), new_entry);
+
+    for matcher in map.values() {
+        let mut trace = Vec::new();
+        traverse(matcher, &map, &mut trace)?;
+    }
+
+    Ok(())
+}
+
+/// Traverse matcher graph and check for any recursion.
+fn traverse<'a>(
+    matcher: &'a MatcherConfig,
+    matchers: &'a HashMap<String, &MatcherConfig>,
+    trace: &mut Vec<&'a String>,
+) -> Result<(), HttpError> {
+    // Recursion safety: Either we have a DAG and will terminate after visiting all
+    // nodes in the graph, or we quit early because we detected a loop by having the same
+    // matcher name twice in the trace.
+
+    if trace.contains(&&matcher.name) {
+        let mut trace_str = String::new();
+        for item in trace {
+            trace_str.push_str(item);
+            trace_str.push_str(" ? ");
+        }
+
+        trace_str.push_str(&matcher.name);
+        http_bail!(BAD_REQUEST, "matcher recursion detected: {trace_str}");
+    }
+
+    trace.push(&matcher.name);
+
+    for next_name in &matcher.eval_matcher {
+        if let Some(next) = matchers.get(next_name) {
+            traverse(next, matchers, trace)?;
+        }
+    }
+
+    trace.pop();
+
+    Ok(())
+}
+
+/// Ensure that `config` contains all matchers in `matchers`.
+fn ensure_matchers_exist<'a>(
+    config: &'a Config,
+    matchers: impl Iterator<Item = &'a str>,
+) -> Result<(), HttpError> {
+    for name in matchers {
+        get_matcher(config, name)?;
+    }
+
+    Ok(())
+}
+
 #[cfg(all(test, feature = "sendmail"))]
 mod tests {
     use super::*;
@@ -259,4 +373,101 @@ matcher: matcher2
 
         Ok(())
     }
+
+    #[test]
+    fn test_matcher_delete_referenced_matcher_fails() {
+        let mut config = Config::new(
+            "
+matcher: matcher1
+    eval-matcher matcher2
+
+matcher: matcher2
+    nested true
+",
+            "",
+        )
+        .unwrap();
+
+        assert!(delete_matcher(&mut config, "matcher2").is_err());
+    }
+
+    #[test]
+    fn test_matcher_update_would_create_indirect_recursion() {
+        let mut config = Config::new(
+            "
+matcher: matcher1
+    nested true
+    eval-matcher matcher2
+
+matcher: matcher2
+    nested true
+",
+            "",
+        )
+        .unwrap();
+
+        assert!(update_matcher(
+            &mut config,
+            "matcher2",
+            MatcherConfigUpdater {
+                eval_matcher: Some(vec!["matcher1".into()]),
+                ..Default::default()
+            },
+            None,
+            None,
+        )
+        .is_err());
+    }
+
+    #[test]
+    fn test_matcher_update_would_create_direct_recursion() {
+        let mut config = Config::new(
+            "
+matcher: matcher1
+    nested true
+",
+            "",
+        )
+        .unwrap();
+
+        assert!(update_matcher(
+            &mut config,
+            "matcher1",
+            MatcherConfigUpdater {
+                eval_matcher: Some(vec!["matcher1".into()]),
+                ..Default::default()
+            },
+            None,
+            None,
+        )
+        .is_err());
+    }
+
+    #[test]
+    fn test_remove_nested_for_referenced_matcher() {
+        let mut config = Config::new(
+            "
+matcher: matcher1
+    nested true
+    eval-matcher matcher2
+
+matcher: matcher2
+    nested true
+",
+            "",
+        )
+        .unwrap();
+
+        assert!(update_matcher(
+            &mut config,
+            "matcher2",
+            MatcherConfigUpdater {
+                nested: Some(false),
+                ..Default::default()
+            },
+            None,
+            None,
+        )
+        .is_err());
+    }
 }
diff --git a/proxmox-notify/src/api/mod.rs b/proxmox-notify/src/api/mod.rs
index 7f823bc7..ee9bb2b9 100644
--- a/proxmox-notify/src/api/mod.rs
+++ b/proxmox-notify/src/api/mod.rs
@@ -202,6 +202,10 @@ fn get_referrers(config: &Config, entity: &str) -> Result<HashSet<String>, HttpE
         if matcher.target.iter().any(|target| target == entity) {
             referrers.insert(matcher.name.clone());
         }
+
+        if matcher.eval_matcher.iter().any(|target| target == entity) {
+            referrers.insert(matcher.name.clone());
+        }
     }
 
     Ok(referrers)
diff --git a/proxmox-notify/src/matcher.rs b/proxmox-notify/src/matcher.rs
index 3cc0189a..40867ab9 100644
--- a/proxmox-notify/src/matcher.rs
+++ b/proxmox-notify/src/matcher.rs
@@ -516,6 +516,10 @@ pub enum DeleteableMatcherProperty {
     Mode,
     /// Delete `target`
     Target,
+    /// Delete `nested`.
+    Nested,
+    /// Delete `eval-matcher`.
+    EvalMatcher,
 }
 
 pub fn check_matches<'a>(
-- 
2.39.5




From l.wagner at proxmox.com  Wed May 21 16:23:08 2025
From: l.wagner at proxmox.com (Lukas Wagner)
Date: Wed, 21 May 2025 16:23:08 +0200
Subject: [pbs-devel] [PATCH proxmox-widget-toolkit 1/1] notifications:
 matchers: add nested matcher support
In-Reply-To: <20250521142309.264719-1-l.wagner@proxmox.com>
References: <20250521142309.264719-1-l.wagner@proxmox.com>
Message-ID: <20250521142309.264719-4-l.wagner@proxmox.com>

Adds a new checkbox to declare a matcher is 'nested', as well as a
new node type in the rule tree 'Evaluate nested matcher'. The latter
allows one to select other matcher which is declared as 'nested'.

Signed-off-by: Lukas Wagner <l.wagner at proxmox.com>
---
 src/data/model/NotificationConfig.js  |   8 +-
 src/panel/NotificationConfigView.js   |   6 +
 src/window/NotificationMatcherEdit.js | 220 +++++++++++++++++++++++++-
 3 files changed, 232 insertions(+), 2 deletions(-)

diff --git a/src/data/model/NotificationConfig.js b/src/data/model/NotificationConfig.js
index 03cf317..e1f7058 100644
--- a/src/data/model/NotificationConfig.js
+++ b/src/data/model/NotificationConfig.js
@@ -9,7 +9,13 @@ Ext.define('proxmox-notification-endpoints', {
 
 Ext.define('proxmox-notification-matchers', {
     extend: 'Ext.data.Model',
-    fields: ['name', 'comment', 'disable', 'origin'],
+    fields: [
+        'name',
+        'comment',
+        'disable',
+        'origin',
+        'nested',
+    ],
     proxy: {
         type: 'proxmox',
     },
diff --git a/src/panel/NotificationConfigView.js b/src/panel/NotificationConfigView.js
index 9505eb7..994ff9d 100644
--- a/src/panel/NotificationConfigView.js
+++ b/src/panel/NotificationConfigView.js
@@ -326,6 +326,12 @@ Ext.define('Proxmox.panel.NotificationMatcherView', {
 	    renderer: (disable) => Proxmox.Utils.renderEnabledIcon(!disable),
 	    align: 'center',
 	},
+	{
+	    dataIndex: 'nested',
+	    text: gettext('Nested'),
+	    renderer: (nested) => Proxmox.Utils.renderEnabledIcon(nested),
+	    align: 'center',
+	},
 	{
 	    dataIndex: 'name',
 	    text: gettext('Matcher Name'),
diff --git a/src/window/NotificationMatcherEdit.js b/src/window/NotificationMatcherEdit.js
index 83c09ea..06597d5 100644
--- a/src/window/NotificationMatcherEdit.js
+++ b/src/window/NotificationMatcherEdit.js
@@ -21,6 +21,21 @@ Ext.define('Proxmox.panel.NotificationMatcherGeneralPanel', {
 	    allowBlank: false,
 	    checked: true,
 	},
+	{
+	    xtype: 'proxmoxcheckbox',
+	    name: 'nested',
+	    fieldLabel: gettext('Nested matcher'),
+	    allowBlank: false,
+	    checked: false,
+	    bind: '{isNestedMatcher}',
+	    autoEl: {
+		tag: 'div',
+		'data-qtip': gettext('Nested matchers can be used by other matchers to create sophisticated matching rules.'),
+	    },
+	    cbind: {
+		deleteEmpty: '{!isCreate}',
+	    },
+	},
 	{
 	    xtype: 'proxmoxtextfield',
 	    name: 'comment',
@@ -64,9 +79,24 @@ Ext.define('Proxmox.panel.NotificationMatcherTargetPanel', {
 	{
 	    xtype: 'pmxNotificationTargetSelector',
 	    name: 'target',
-	    allowBlank: false,
+	    allowBlank: true,
 	},
     ],
+
+    onGetValues: function(values) {
+	let me = this;
+
+	if (Ext.isArray(values.target)) {
+	    if (values.target.length === 0) {
+		delete values.target;
+		if (!me.isCreate) {
+		    Proxmox.Utils.assemble_field_data(values, { 'delete': 'target' });
+		}
+	    }
+	}
+
+	return values;
+    },
 });
 
 Ext.define('Proxmox.window.NotificationMatcherEdit', {
@@ -81,6 +111,12 @@ Ext.define('Proxmox.window.NotificationMatcherEdit', {
 
     width: 800,
 
+    viewModel: {
+	data: {
+	    isNestedMatcher: false,
+	},
+    },
+
     initComponent: function() {
 	let me = this;
 
@@ -130,6 +166,9 @@ Ext.define('Proxmox.window.NotificationMatcherEdit', {
 			    xtype: 'pmxNotificationMatcherTargetPanel',
 			    isCreate: me.isCreate,
 			    baseUrl: me.baseUrl,
+			    bind: {
+				disabled: '{isNestedMatcher}',
+			    },
 			},
 		    ],
 		},
@@ -357,6 +396,11 @@ Ext.define('Proxmox.panel.NotificationRulesEditPanel', {
 				value: '',
 			    };
 			    break;
+			case 'eval-matcher':
+			    data = {
+				matcher: '',
+			    };
+			    break;
 		    }
 
 		    let node = {
@@ -424,6 +468,7 @@ Ext.define('Proxmox.panel.NotificationRulesEditPanel', {
 	    xtype: 'pmxNotificationMatchRuleSettings',
 	    cbind: {
 		baseUrl: '{baseUrl}',
+		name: '{name}',
 	    },
 	},
 
@@ -445,6 +490,7 @@ Ext.define('Proxmox.panel.NotificationRulesEditPanel', {
 	deleteArrayIfEmtpy('match-field');
 	deleteArrayIfEmtpy('match-severity');
 	deleteArrayIfEmtpy('match-calendar');
+	deleteArrayIfEmtpy('eval-matcher');
 
 	return values;
     },
@@ -506,6 +552,14 @@ Ext.define('Proxmox.panel.NotificationMatchRuleTree', {
 		iconCls = 'fa fa-filter';
 
 		break;
+	    case 'eval-matcher': {
+		let v = data.matcher;
+		text = Ext.String.format(gettext("Evaluate nested matcher: {0}"), v);
+		iconCls = 'fa fa-filter';
+		if (!v) {
+		    iconCls += ' internal-error';
+		}
+	    } break;
 	}
 
 	return [text, iconCls];
@@ -632,12 +686,29 @@ Ext.define('Proxmox.panel.NotificationMatchRuleTree', {
 	    deleteEmpty: !me.isCreate,
 	});
 
+	let realEvalMatcher = Ext.create({
+	    xtype: 'hiddenfield',
+	    name: 'eval-matcher',
+	    setValue: function(value) {
+		this.value = value;
+		this.checkChange();
+	    },
+	    getValue: function() {
+		return this.value;
+	    },
+	    getSubmitValue: function() {
+		let value = this.value;
+		return value;
+	    },
+	});
+
 	let storeChanged = function(store) {
 	    store.suspendEvent('datachanged');
 
 	    let matchFieldStmts = [];
 	    let matchSeverityStmts = [];
 	    let matchCalendarStmts = [];
+	    let evalMatcherStmts = [];
 	    let modeStmt = 'all';
 	    let invertMatchStmt = false;
 
@@ -663,6 +734,9 @@ Ext.define('Proxmox.panel.NotificationMatchRuleTree', {
 			modeStmt = data.value;
 			invertMatchStmt = data.invert;
 			break;
+		    case 'eval-matcher':
+			evalMatcherStmts.push(data.matcher);
+			break;
 		}
 
 		let [text, iconCls] = me.getNodeTextAndIcon(type, data);
@@ -692,6 +766,10 @@ Ext.define('Proxmox.panel.NotificationMatchRuleTree', {
 	    realMatchSeverity.setValue(matchSeverityStmts);
 	    realMatchSeverity.resumeEvent('change');
 
+	    realEvalMatcher.suspendEvent('change');
+	    realEvalMatcher.setValue(evalMatcherStmts);
+	    realEvalMatcher.resumeEvent('change');
+
 	    store.resumeEvent('datachanged');
 	};
 
@@ -800,6 +878,30 @@ Ext.define('Proxmox.panel.NotificationMatchRuleTree', {
 	    });
 	});
 
+	realEvalMatcher.addListener('change', function(field, value) {
+	    let parseEvalMatcher = function(matcher) {
+		return {
+		    type: 'eval-matcher',
+		    data: {
+			matcher: matcher,
+		    },
+		    leaf: true,
+		};
+	    };
+
+	    for (let node of treeStore.queryBy(
+		record => record.get('type') === 'eval-matcher').getRange()) {
+		node.remove(true);
+	    }
+
+	    let records = value.map(parseEvalMatcher);
+	    let rootNode = treeStore.getRootNode();
+
+	    for (let record of records) {
+		rootNode.appendChild(record);
+	    }
+	});
+
 	treeStore.addListener('datachanged', storeChanged);
 
 	let treePanel = Ext.create({
@@ -844,6 +946,7 @@ Ext.define('Proxmox.panel.NotificationMatchRuleTree', {
 		realMatchSeverity,
 		realInvertMatch,
 		realMatchCalendar,
+		realEvalMatcher,
 		treePanel,
 		{
 		    xtype: 'button',
@@ -913,6 +1016,7 @@ Ext.define('Proxmox.panel.NotificationMatchRuleSettings', {
 		['match-field', gettext('Match Field')],
 		['match-severity', gettext('Match Severity')],
 		['match-calendar', gettext('Match Calendar')],
+		['eval-matcher', gettext('Evaluate nested matcher')],
 	    ],
 	},
 	{
@@ -927,6 +1031,13 @@ Ext.define('Proxmox.panel.NotificationMatchRuleSettings', {
 	{
 	    xtype: 'pmxNotificationMatchCalendarSettings',
 	},
+	{
+	    xtype: 'pmxNotificationEvalMatcherSettings',
+	    cbind: {
+		baseUrl: '{baseUrl}',
+		name: '{name}',
+	    },
+	},
     ],
 });
 
@@ -1372,3 +1483,110 @@ Ext.define('Proxmox.panel.MatchFieldSettings', {
 	me.callParent();
     },
 });
+
+Ext.define('Proxmox.panel.EvalMatcherSettings', {
+    extend: 'Ext.panel.Panel',
+    xtype: 'pmxNotificationEvalMatcherSettings',
+    border: false,
+    layout: 'anchor',
+    // Hide initially to avoid glitches when opening the window
+    hidden: true,
+    bind: {
+	hidden: '{!typeIsEvalMatcher}',
+    },
+    viewModel: {
+	// parent is set in `initComponents`
+	formulas: {
+	    typeIsEvalMatcher: {
+		bind: {
+		    bindTo: '{selectedRecord}',
+		    deep: true,
+		},
+		get: function(record) {
+		    return record?.get('type') === 'eval-matcher';
+		},
+	    },
+	    evalMatcherValue: {
+		bind: {
+		    bindTo: '{selectedRecord}',
+		    deep: true,
+		},
+		set: function(value) {
+		    let record = this.get('selectedRecord');
+		    let currentData = record.get('data');
+		    record.set({
+			data: {
+			    ...currentData,
+			    matcher: value,
+			},
+		    });
+		},
+		get: function(record) {
+		    return record?.get('data')?.matcher;
+		},
+	    },
+	},
+    },
+
+    initComponent: function() {
+	let me = this;
+
+	let valueStore = Ext.create('Ext.data.Store', {
+	    model: 'proxmox-notification-matchers',
+	    autoLoad: true,
+	    proxy: {
+		type: 'proxmox',
+		url: `/api2/json/${me.baseUrl}/matchers`,
+	    },
+	    filters: [
+		{
+		    property: 'nested',
+		    value: true,
+		},
+		(item) => item.get('name') !== me.name,
+	    ],
+	});
+
+	Ext.apply(me.viewModel, {
+	    parent: me.up('pmxNotificationMatchRulesEditPanel').getViewModel(),
+	});
+	Ext.apply(me, {
+	    items: [
+		{
+		    fieldLabel: gettext('Matcher'),
+		    xtype: 'proxmoxComboGrid',
+		    autoSelect: false,
+		    editable: false,
+		    isFormField: false,
+		    submitValue: false,
+		    allowBlank: false,
+		    showClearTrigger: true,
+		    field: 'name',
+		    store: valueStore,
+		    valueField: 'name',
+		    displayField: 'name',
+		    notFoundIsValid: false,
+		    multiSelect: false,
+		    bind: {
+			value: '{evalMatcherValue}',
+		    },
+		    listConfig: {
+			columns: [
+			    {
+				header: gettext('Name'),
+				dataIndex: 'name',
+				flex: 1,
+			    },
+			    {
+				header: gettext('Comment'),
+				dataIndex: 'comment',
+				flex: 2,
+			    },
+			],
+		    },
+		},
+	    ],
+	});
+	me.callParent();
+    },
+});
-- 
2.39.5




From l.wagner at proxmox.com  Wed May 21 16:23:06 2025
From: l.wagner at proxmox.com (Lukas Wagner)
Date: Wed, 21 May 2025 16:23:06 +0200
Subject: [pbs-devel] [PATCH proxmox 1/2] notify: matcher: allow to evaluate
 other matchers via `eval-matcher`
In-Reply-To: <20250521142309.264719-1-l.wagner@proxmox.com>
References: <20250521142309.264719-1-l.wagner@proxmox.com>
Message-ID: <20250521142309.264719-2-l.wagner@proxmox.com>

This commit adds support for nested matchers, allowing to build
arbitrary matching 'formulas'.
A matcher can now have one or more `eval-matcher` directives. These
take the name of another matcher as a parameter. When the
matcher is evaluated, we first check any referenced nested matcher and
use their results to compute the final verdict for the matcher.

If one wants to use a matcher via `eval-matcher`, the matcher
must be marked as nested. Any nested matcher is only evaluated when
another matcher references it. Any configured targets for a nested
matcher are ignored, only the 'top-most' matcher's targets are
notified if the notification matches.

Direct/indirect recursion is not allowed (A -> A, A -> B -> A)
an raises an error.

A simple cache is introduced to make sure that we don't have to evaluate
any nested matcher more than once.

Simple config example:

matcher: top-level
  target mail-to-root
  mode any
  eval-matcher a
  eval-matcher b

matcher: a
  nested true
  mode all
  match-field exact:datastore=store
  match-field exact:type=gc
  match-severity error

matcher: b
  nested true
  mode all
  match-field exact:datastore=store
  match-field exact:type=prune
  match-severity error

Signed-off-by: Lukas Wagner <l.wagner at proxmox.com>
---
 proxmox-notify/src/lib.rs     |  20 ++-
 proxmox-notify/src/matcher.rs | 309 ++++++++++++++++++++++++++++++++--
 2 files changed, 304 insertions(+), 25 deletions(-)

diff --git a/proxmox-notify/src/lib.rs b/proxmox-notify/src/lib.rs
index 12e59474..bb5bb605 100644
--- a/proxmox-notify/src/lib.rs
+++ b/proxmox-notify/src/lib.rs
@@ -376,7 +376,7 @@ impl Config {
 #[derive(Default)]
 pub struct Bus {
     endpoints: HashMap<String, Box<dyn Endpoint>>,
-    matchers: Vec<MatcherConfig>,
+    matchers: HashMap<String, MatcherConfig>,
 }
 
 #[allow(unused_macros)]
@@ -514,10 +514,14 @@ impl Bus {
             );
         }
 
-        let matchers = config
-            .config
-            .convert_to_typed_array(MATCHER_TYPENAME)
-            .map_err(|err| Error::ConfigDeserialization(err.into()))?;
+        let matchers = HashMap::from_iter(
+            config
+                .config
+                .convert_to_typed_array(MATCHER_TYPENAME)
+                .map_err(|err| Error::ConfigDeserialization(err.into()))?
+                .into_iter()
+                .map(|e: MatcherConfig| (e.name.clone(), e)),
+        );
 
         Ok(Bus {
             endpoints,
@@ -531,8 +535,8 @@ impl Bus {
     }
 
     #[cfg(test)]
-    pub fn add_matcher(&mut self, filter: MatcherConfig) {
-        self.matchers.push(filter)
+    pub fn add_matcher(&mut self, matcher: MatcherConfig) {
+        self.matchers.insert(matcher.name.clone(), matcher);
     }
 
     /// Send a notification. Notification matchers will determine which targets will receive
@@ -540,7 +544,7 @@ impl Bus {
     ///
     /// Any errors will not be returned but only logged.
     pub fn send(&self, notification: &Notification) {
-        let targets = matcher::check_matches(self.matchers.as_slice(), notification);
+        let targets = matcher::check_matches(&self.matchers, notification);
 
         for target in targets {
             if let Some(endpoint) = self.endpoints.get(target) {
diff --git a/proxmox-notify/src/matcher.rs b/proxmox-notify/src/matcher.rs
index 083c2dbd..3cc0189a 100644
--- a/proxmox-notify/src/matcher.rs
+++ b/proxmox-notify/src/matcher.rs
@@ -1,4 +1,4 @@
-use std::collections::HashSet;
+use std::collections::{HashMap, HashSet};
 use std::fmt;
 use std::fmt::Debug;
 use std::str::FromStr;
@@ -98,6 +98,13 @@ pub const MATCH_FIELD_ENTRY_SCHEMA: Schema = StringSchema::new("Match metadata f
             },
             optional: true,
         },
+        "eval-matcher": {
+            type: Array,
+            items: {
+                schema: ENTITY_NAME_SCHEMA,
+            },
+            optional: true,
+        },
         "target": {
             type: Array,
             items: {
@@ -106,7 +113,7 @@ pub const MATCH_FIELD_ENTRY_SCHEMA: Schema = StringSchema::new("Match metadata f
             optional: true,
         },
     })]
-#[derive(Debug, Serialize, Deserialize, Updater, Default)]
+#[derive(Clone, Debug, Serialize, Deserialize, Updater, Default)]
 #[serde(rename_all = "kebab-case")]
 /// Config for Sendmail notification endpoints
 pub struct MatcherConfig {
@@ -136,6 +143,11 @@ pub struct MatcherConfig {
     #[serde(skip_serializing_if = "Option::is_none")]
     pub invert_match: Option<bool>,
 
+    /// List of nested matchers to evaluate.
+    #[serde(default, skip_serializing_if = "Vec::is_empty")]
+    #[updater(serde(skip_serializing_if = "Option::is_none"))]
+    pub eval_matcher: Vec<String>,
+
     /// Targets to notify.
     #[serde(default, skip_serializing_if = "Vec::is_empty")]
     #[updater(serde(skip_serializing_if = "Option::is_none"))]
@@ -149,6 +161,13 @@ pub struct MatcherConfig {
     #[serde(skip_serializing_if = "Option::is_none")]
     pub disable: Option<bool>,
 
+    /// Determine if this matcher can be used as a nested matcher.
+    /// Nested matchers are only evaluated if they are referenced by
+    /// another matcher. Any configured targets of a nested matcher
+    /// are ignored.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub nested: Option<bool>,
+
     /// Origin of this config entry.
     #[serde(skip_serializing_if = "Option::is_none")]
     #[updater(skip)]
@@ -282,7 +301,30 @@ impl FromStr for FieldMatcher {
 }
 
 impl MatcherConfig {
-    pub fn matches(&self, notification: &Notification) -> Result<Option<&[String]>, Error> {
+    pub fn matches(
+        &self,
+        notification: &Notification,
+        all_matchers: &HashMap<String, MatcherConfig>,
+        cache: &mut HashMap<String, bool>,
+    ) -> Result<Option<&[String]>, Error> {
+        let mut trace = HashSet::new();
+        trace.insert(self.name.clone());
+        let result = self.matches_impl(notification, all_matchers, &mut trace, cache)?;
+
+        if result {
+            Ok(Some(&self.target))
+        } else {
+            Ok(None)
+        }
+    }
+
+    fn matches_impl(
+        &self,
+        notification: &Notification,
+        all_matchers: &HashMap<String, MatcherConfig>,
+        trace: &mut HashSet<String>,
+        cache: &mut HashMap<String, bool>,
+    ) -> Result<bool, Error> {
         let mode = self.mode.unwrap_or_default();
 
         let mut is_match = mode.neutral_element();
@@ -311,13 +353,52 @@ impl MatcherConfig {
             );
         }
 
-        let invert_match = self.invert_match.unwrap_or_default();
+        for matcher_name in &self.eval_matcher {
+            no_matchers = false;
+            match all_matchers.get(matcher_name) {
+                Some(matcher) if matcher.nested.unwrap_or_default() => {
+                    if trace.contains(matcher_name) {
+                        return Err(Error::FilterFailed(
+                            "recursive sub-matcher definition".into(),
+                        ));
+                    }
 
-        Ok(if is_match != invert_match || no_matchers {
-            Some(&self.target)
-        } else {
-            None
-        })
+                    if matcher.disable.unwrap_or_default() {
+                        continue;
+                    }
+
+                    trace.insert(matcher.name.clone());
+
+                    if let Some(cached_result) = cache.get(&matcher.name) {
+                        is_match = mode.apply(is_match, *cached_result);
+                    } else {
+                        is_match = mode.apply(
+                            is_match,
+                            matcher.matches_impl(notification, all_matchers, trace, cache)?,
+                        );
+                    }
+
+                    trace.remove(matcher_name);
+                }
+                Some(_) => {
+                    return Err(Error::FilterFailed(
+                        "referenced matcher is not declared as sub-matcher".into(),
+                    ));
+                }
+                None => {
+                    return Err(Error::FilterFailed(
+                        "referenced sub-matcher does not exist".into(),
+                    ));
+                }
+            }
+        }
+
+        let invert_match = self.invert_match.unwrap_or_default();
+        is_match = is_match != invert_match || no_matchers;
+
+        cache.insert(self.name.clone(), is_match);
+
+        Ok(is_match)
     }
 
     /// Check if given `MatchDirectives` match a notification.
@@ -438,24 +519,31 @@ pub enum DeleteableMatcherProperty {
 }
 
 pub fn check_matches<'a>(
-    matchers: &'a [MatcherConfig],
+    matchers: &'a HashMap<String, MatcherConfig>,
     notification: &Notification,
 ) -> HashSet<&'a str> {
     let mut targets = HashSet::new();
+    let mut cache = HashMap::new();
 
-    for matcher in matchers {
-        if matcher.disable.unwrap_or_default() {
-            // Skip this matcher if it is disabled
-            info!("skipping disabled matcher '{name}'", name = matcher.name);
+    for (name, matcher) in matchers {
+        if matcher.nested.unwrap_or_default() {
+            // Matchers which are declared are only evaluated if a
+            // top-level, non-nested matcher references it.
             continue;
         }
 
-        match matcher.matches(notification) {
+        if matcher.disable.unwrap_or_default() {
+            // Skip this matcher if it is disabled
+            info!("skipping disabled matcher '{name}'");
+            continue;
+        }
+
+        match matcher.matches(notification, matchers, &mut cache) {
             Ok(t) => {
                 let t = t.unwrap_or_default();
                 targets.extend(t.iter().map(|s| s.as_str()));
             }
-            Err(err) => error!("matcher '{matcher}' failed: {err}", matcher = matcher.name),
+            Err(err) => error!("matcher '{name}' failed: {err}"),
         }
     }
 
@@ -505,6 +593,7 @@ mod tests {
         assert!("regex:'3=b.*".parse::<FieldMatcher>().is_err());
         assert!("invalid:'bar=b.*".parse::<FieldMatcher>().is_err());
     }
+
     #[test]
     fn test_severities() {
         let notification =
@@ -526,7 +615,193 @@ mod tests {
                 ..Default::default()
             };
 
-            assert!(config.matches(&notification).unwrap().is_some())
+            let mut all_matchers = HashMap::new();
+            all_matchers.insert("default".to_string(), config.clone());
+
+            assert!(config
+                .matches(&notification, &all_matchers, &mut HashMap::new())
+                .unwrap()
+                .is_some())
         }
     }
+
+    #[test]
+    fn test_submatcher() {
+        let mut all_matchers = HashMap::new();
+
+        let config = MatcherConfig {
+            name: "sub-1".to_string(),
+            nested: Some(true),
+            mode: Some(MatchModeOperator::All),
+            match_severity: vec!["error".parse().unwrap()],
+            ..Default::default()
+        };
+        all_matchers.insert(config.name.clone(), config.clone());
+
+        let config = MatcherConfig {
+            name: "sub-2".to_string(),
+            nested: Some(true),
+            mode: Some(MatchModeOperator::All),
+            match_field: vec!["exact:datastore=backups".parse().unwrap()],
+            ..Default::default()
+        };
+        all_matchers.insert(config.name.clone(), config.clone());
+
+        let config = MatcherConfig {
+            name: "top".to_string(),
+            eval_matcher: vec!["sub-1".into(), "sub-2".into()],
+            mode: Some(MatchModeOperator::Any),
+            ..Default::default()
+        };
+        all_matchers.insert(config.name.clone(), config.clone());
+
+        let notification = Notification::from_template(
+            Severity::Notice,
+            "test",
+            Value::Null,
+            HashMap::from_iter([("datastore".into(), "backups".into())]),
+        );
+
+        assert!(config
+            .matches(&notification, &all_matchers, &mut HashMap::new())
+            .unwrap()
+            .is_some());
+
+        let notification = Notification::from_template(
+            Severity::Error,
+            "test",
+            Value::Null,
+            HashMap::from_iter([("datastore".into(), "other".into())]),
+        );
+
+        assert!(config
+            .matches(&notification, &all_matchers, &mut HashMap::new())
+            .unwrap()
+            .is_some());
+
+        assert!(config
+            .matches(&notification, &all_matchers, &mut HashMap::new())
+            .unwrap()
+            .is_some());
+
+        let notification = Notification::from_template(
+            Severity::Warning,
+            "test",
+            Value::Null,
+            HashMap::from_iter([("datastore".into(), "other".into())]),
+        );
+
+        assert!(config
+            .matches(&notification, &all_matchers, &mut HashMap::new())
+            .unwrap()
+            .is_none());
+    }
+
+    #[test]
+    fn test_submatcher_recursion_direct() {
+        let mut all_matchers = HashMap::new();
+
+        let config = MatcherConfig {
+            name: "sub-1".to_string(),
+            nested: Some(true),
+            eval_matcher: vec!["sub-1".into()],
+            ..Default::default()
+        };
+        all_matchers.insert(config.name.clone(), config.clone());
+
+        let config = MatcherConfig {
+            name: "top".to_string(),
+            eval_matcher: vec!["sub-1".into()],
+            mode: Some(MatchModeOperator::Any),
+            ..Default::default()
+        };
+        all_matchers.insert(config.name.clone(), config.clone());
+
+        let notification =
+            Notification::from_template(Severity::Notice, "test", Value::Null, HashMap::new());
+
+        assert!(config
+            .matches(&notification, &all_matchers, &mut HashMap::new())
+            .is_err());
+    }
+
+    #[test]
+    fn test_submatcher_recursion_indirect() {
+        let mut all_matchers = HashMap::new();
+
+        let config = MatcherConfig {
+            name: "sub-1".to_string(),
+            nested: Some(true),
+            eval_matcher: vec!["sub-2".into()],
+            ..Default::default()
+        };
+        all_matchers.insert(config.name.clone(), config.clone());
+
+        let config = MatcherConfig {
+            name: "sub-2".to_string(),
+            nested: Some(true),
+            eval_matcher: vec!["sub-1".into()],
+            ..Default::default()
+        };
+        all_matchers.insert(config.name.clone(), config.clone());
+
+        let config = MatcherConfig {
+            name: "top".to_string(),
+            eval_matcher: vec!["sub-1".into()],
+            ..Default::default()
+        };
+        all_matchers.insert(config.name.clone(), config.clone());
+
+        let notification =
+            Notification::from_template(Severity::Notice, "test", Value::Null, HashMap::new());
+
+        assert!(config
+            .matches(&notification, &all_matchers, &mut HashMap::new())
+            .is_err());
+    }
+
+    #[test]
+    fn test_submatcher_does_not_exist() {
+        let mut all_matchers = HashMap::new();
+
+        let config = MatcherConfig {
+            name: "top".to_string(),
+            eval_matcher: vec!["doesntexist".into()],
+            ..Default::default()
+        };
+        all_matchers.insert(config.name.clone(), config.clone());
+
+        let notification =
+            Notification::from_template(Severity::Notice, "test", Value::Null, HashMap::new());
+
+        assert!(config
+            .matches(&notification, &all_matchers, &mut HashMap::new())
+            .is_err());
+    }
+
+    #[test]
+    fn test_submatcher_does_not_declared_as_submatcher() {
+        let mut all_matchers = HashMap::new();
+
+        let config = MatcherConfig {
+            name: "sub-1".to_string(),
+            nested: Some(true),
+            ..Default::default()
+        };
+        all_matchers.insert(config.name.clone(), config.clone());
+
+        let config = MatcherConfig {
+            name: "top".to_string(),
+            eval_matcher: vec!["doesntexist".into()],
+            ..Default::default()
+        };
+        all_matchers.insert(config.name.clone(), config.clone());
+
+        let notification =
+            Notification::from_template(Severity::Notice, "test", Value::Null, HashMap::new());
+
+        assert!(config
+            .matches(&notification, &all_matchers, &mut HashMap::new())
+            .is_err());
+    }
 }
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 22 08:21:40 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 22 May 2025 08:21:40 +0200
Subject: [pbs-devel] [PATCH proxmox-backup] backup info: avoid additional
 stat syscall for protected check
Message-ID: <20250522062140.51956-1-c.ebner@proxmox.com>

`BackupDir::is_protected` is the general helper method to check the
protected state for a snapshot. This checks for the presence of the
protected marker file, which is performed by stating the file and
requires traversing the full path.

When generating the backup list for a backup group, the snapshot
directory contents are however scanned nevertheless. Take advantage
of this by extending the regex used to filter contents by scandir to
include also the protected marker filename and set the state based on
the presence/absence, thereby avoiding the additional stat syscall
altogether.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---

This was encountered while investigating possible causes for slow GC
phase1 since PBS version 3.4 on some particular storages as reported
in the community forum [0]. While an improvement, this is very
unlikely the issue at hand.

[0] https://forum.proxmox.com/threads/166214/

 pbs-datastore/Cargo.toml         |  2 ++
 pbs-datastore/src/backup_info.rs | 29 ++++++++++++++++++++++++++---
 2 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index 7623adc28..d7fe71b34 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -9,6 +9,7 @@ rust-version.workspace = true
 [dependencies]
 anyhow.workspace = true
 base64.workspace = true
+const_format.workspace = true
 crc32fast.workspace = true
 endian_trait.workspace = true
 futures.workspace = true
@@ -17,6 +18,7 @@ libc.workspace = true
 log.workspace = true
 nix.workspace = true
 openssl.workspace = true
+regex.workspace = true
 serde.workspace = true
 serde_json.workspace = true
 tokio = { workspace = true, features = [] }
diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index d4732fdd9..dbd0905c6 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -7,6 +7,7 @@ use std::sync::{Arc, LazyLock};
 use std::time::Duration;
 
 use anyhow::{bail, format_err, Context, Error};
+use const_format::concatcp;
 
 use proxmox_sys::fs::{lock_dir_noblock, lock_dir_noblock_shared, replace_file, CreateOptions};
 use proxmox_systemd::escape_unit;
@@ -21,6 +22,11 @@ use crate::manifest::{BackupManifest, MANIFEST_LOCK_NAME};
 use crate::{DataBlob, DataStore};
 
 pub const DATASTORE_LOCKS_DIR: &str = "/run/proxmox-backup/locks";
+const PROTECTED_MARKER_FILENAME: &str = ".protected";
+
+proxmox_schema::const_regex! {
+    pub BACKUP_FILES_AND_PROTECTED_REGEX = concatcp!(r"^(.*\.([fd]idx|blob)|\", PROTECTED_MARKER_FILENAME, ")$");
+}
 
 // TODO: Remove with PBS 5
 // Note: The `expect()` call here will only happen if we can neither confirm nor deny the existence
@@ -113,9 +119,26 @@ impl BackupGroup {
                 }
 
                 let backup_dir = self.backup_dir_with_rfc3339(backup_time)?;
-                let files = list_backup_files(l2_fd, backup_time)?;
+                let mut protected = false;
+                let mut files = Vec::new();
 
-                let protected = backup_dir.is_protected();
+                proxmox_sys::fs::scandir(
+                    l2_fd,
+                    backup_time,
+                    &BACKUP_FILES_AND_PROTECTED_REGEX,
+                    |_, filename, file_type| {
+                        if file_type != nix::dir::Type::File {
+                            return Ok(());
+                        }
+                        // avoids more expensive check via `BackupDir::is_protected`
+                        if filename == ".protected" {
+                            protected = true;
+                        } else {
+                            files.push(filename.to_owned());
+                        }
+                        Ok(())
+                    },
+                )?;
 
                 list.push(BackupInfo {
                     backup_dir,
@@ -457,7 +480,7 @@ impl BackupDir {
 
     pub fn protected_file(&self) -> PathBuf {
         let mut path = self.full_path();
-        path.push(".protected");
+        path.push(PROTECTED_MARKER_FILENAME);
         path
     }
 
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 22 14:51:28 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 22 May 2025 14:51:28 +0200
Subject: [pbs-devel] [PATCH proxmox-backup] datastore: pass relative path to
 group type iterator
Message-ID: <20250522125128.428673-1-c.ebner@proxmox.com>

`ListGroupsType::new_at` creates a new iterator over all groups of
give backup type with provided parent file descriptor.

The parent directory file descriptor is passed to the `read_subdir`
call, which itself uses it to open the type directory via `openat`.
This call does however ignore the passed file handle if the given
path is absolute [0], which is always the case for the type path
generated via `DataStore::type_path`.

Fix this by passing only the type name as relative path to the
`read_subdir` call, use the absolute path only for
`ListGroupType::new`.

This helps avoiding re-traversing the absolute path in the
`ListGroups` iterator, and since it is then the only callside for
`ListGroupsType::new_at`, inline the instantiation.

[0] https://linux.die.net/man/2/openat

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---

Similar to [1], this was encountered while inspecting strace outputs
in order to investigate the garbage collection performance issues as
reported in the community forum [2].

[1] https://lore.proxmox.com/pbs-devel/20250522062140.51956-1-c.ebner at proxmox.com/T/
[2] https://forum.proxmox.com/threads/166214/

 pbs-datastore/src/hierarchy.rs | 31 +++++++++++--------------------
 1 file changed, 11 insertions(+), 20 deletions(-)

diff --git a/pbs-datastore/src/hierarchy.rs b/pbs-datastore/src/hierarchy.rs
index e0bf84419..6c1287c3e 100644
--- a/pbs-datastore/src/hierarchy.rs
+++ b/pbs-datastore/src/hierarchy.rs
@@ -1,4 +1,3 @@
-use std::os::unix::io::RawFd;
 use std::path::PathBuf;
 use std::str::FromStr;
 use std::sync::Arc;
@@ -73,17 +72,8 @@ pub struct ListGroupsType {
 
 impl ListGroupsType {
     pub fn new(store: Arc<DataStore>, ns: BackupNamespace, ty: BackupType) -> Result<Self, Error> {
-        Self::new_at(libc::AT_FDCWD, store, ns, ty)
-    }
-
-    fn new_at(
-        fd: RawFd,
-        store: Arc<DataStore>,
-        ns: BackupNamespace,
-        ty: BackupType,
-    ) -> Result<Self, Error> {
         Ok(Self {
-            dir: proxmox_sys::fs::read_subdir(fd, &store.type_path(&ns, ty))?,
+            dir: proxmox_sys::fs::read_subdir(libc::AT_FDCWD, &store.type_path(&ns, ty))?,
             store,
             ns,
             ty,
@@ -197,15 +187,16 @@ impl Iterator for ListGroups {
                     if let Ok(group_type) = BackupType::from_str(name) {
                         // found a backup group type, descend into it to scan all IDs in it
                         // by switching to the id-state branch
-                        match ListGroupsType::new_at(
-                            entry.parent_fd(),
-                            Arc::clone(&self.store),
-                            self.ns.clone(),
-                            group_type,
-                        ) {
-                            Ok(ty) => self.id_state = Some(ty),
-                            Err(err) => return Some(Err(err)),
-                        }
+                        let dir = match proxmox_sys::fs::read_subdir(entry.parent_fd(), name) {
+                            Ok(dir) => dir,
+                            Err(err) => return Some(Err(err.into())),
+                        };
+                        self.id_state = Some(ListGroupsType {
+                            dir,
+                            store: Arc::clone(&self.store),
+                            ns: self.ns.clone(),
+                            ty: group_type,
+                        });
                     }
                 }
             }
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:34 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:34 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 01/12] chunkstore: add CanRead
 and CanWrite trait
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-2-h.laimer@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 pbs-datastore/src/chunk_store.rs | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/chunk_store.rs b/pbs-datastore/src/chunk_store.rs
index 29a3d477..9a77bef2 100644
--- a/pbs-datastore/src/chunk_store.rs
+++ b/pbs-datastore/src/chunk_store.rs
@@ -21,14 +21,36 @@ use crate::file_formats::{
 };
 use crate::DataBlob;
 
+mod private {
+    pub trait Sealed: Clone + Copy {}
+    impl Sealed for super::Read {}
+    impl Sealed for super::Write {}
+    impl Sealed for super::Lookup {}
+}
+
+pub trait CanRead: private::Sealed {}
+pub trait CanWrite: CanRead + private::Sealed {}
+
+#[derive(Clone, Copy, Debug)]
+pub struct Read;
+#[derive(Clone, Copy, Debug)]
+pub struct Write;
+#[derive(Clone, Copy, Debug)]
+pub struct Lookup;
+
+impl CanRead for Read {}
+impl CanRead for Write {}
+impl CanWrite for Write {}
+
 /// File system based chunk store
-pub struct ChunkStore {
+pub struct ChunkStore<T> {
     name: String, // used for error reporting
     pub(crate) base: PathBuf,
     chunk_dir: PathBuf,
     mutex: Mutex<()>,
     locker: Option<Arc<Mutex<ProcessLocker>>>,
     sync_level: DatastoreFSyncLevel,
+    _marker: std::marker::PhantomData<T>,
 }
 
 // TODO: what about sysctl setting vm.vfs_cache_pressure (0 - 100) ?
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:36 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:36 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 03/12] datastore: add generics
 and new lookup functions
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-4-h.laimer@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 80 +++++++++++++++++++++++++++++-----
 1 file changed, 68 insertions(+), 12 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index cbf78ecb..6936875e 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -8,6 +8,7 @@ use std::time::Duration;
 
 use anyhow::{bail, format_err, Context, Error};
 use nix::unistd::{unlinkat, UnlinkatFlags};
+use pbs_config::BackupLockGuard;
 use pbs_tools::lru_cache::LruCache;
 use tracing::{info, warn};
 
@@ -29,7 +30,7 @@ use pbs_api_types::{
 use pbs_config::BackupLockGuard;
 
 use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, OLD_LOCKING};
-use crate::chunk_store::ChunkStore;
+use crate::chunk_store::{CanRead, CanWrite, ChunkStore, Lookup as L, Read as R, Write as W};
 use crate::dynamic_index::{DynamicIndexReader, DynamicIndexWriter};
 use crate::fixed_index::{FixedIndexReader, FixedIndexWriter};
 use crate::hierarchy::{ListGroups, ListGroupsType, ListNamespaces, ListNamespacesRecursive};
@@ -37,7 +38,12 @@ use crate::index::IndexFile;
 use crate::task_tracking::{self, update_active_operations};
 use crate::DataBlob;
 
-static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
+type DataStoreCache<T> = HashMap<String, Arc<DataStoreImpl<T>>>;
+
+static DATASTORE_MAP_READ: LazyLock<Mutex<DataStoreCache<R>>> =
+    LazyLock::new(|| Mutex::new(HashMap::new()));
+
+static DATASTORE_MAP_WRITE: LazyLock<Mutex<DataStoreCache<W>>> =
     LazyLock::new(|| Mutex::new(HashMap::new()));
 
 /// checks if auth_id is owner, or, if owner is a token, if
@@ -117,8 +123,8 @@ pub fn ensure_datastore_is_mounted(config: &DataStoreConfig) -> Result<(), Error
 ///
 /// A Datastore can store severals backups, and provides the
 /// management interface for backup.
-pub struct DataStoreImpl {
-    chunk_store: Arc<ChunkStore>,
+pub struct DataStoreImpl<T> {
+    chunk_store: Arc<ChunkStore<T>>,
     gc_mutex: Mutex<()>,
     last_gc_status: Mutex<GarbageCollectionStatus>,
     verify_new: bool,
@@ -127,12 +133,12 @@ pub struct DataStoreImpl {
     sync_level: DatastoreFSyncLevel,
 }
 
-impl DataStoreImpl {
+impl<T> DataStoreImpl<T> {
     // This one just panics on everything
     #[doc(hidden)]
-    pub(crate) unsafe fn new_test() -> Arc<Self> {
+    pub(crate) fn new_test() -> Arc<Self> {
         Arc::new(Self {
-            chunk_store: Arc::new(unsafe { ChunkStore::panic_store() }),
+            chunk_store: Arc::new(ChunkStore::dummy_store()),
             gc_mutex: Mutex::new(()),
             last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
             verify_new: false,
@@ -143,12 +149,12 @@ impl DataStoreImpl {
     }
 }
 
-pub struct DataStore {
-    inner: Arc<DataStoreImpl>,
+pub struct DataStore<T> {
+    inner: Arc<DataStoreImpl<T>>,
     operation: Option<Operation>,
 }
 
-impl Clone for DataStore {
+impl<T> Clone for DataStore<T> {
     fn clone(&self) -> Self {
         let mut new_operation = self.operation;
         if let Some(operation) = self.operation {
@@ -165,7 +171,7 @@ impl Clone for DataStore {
     }
 }
 
-impl Drop for DataStore {
+impl<T> Drop for DataStore<T> {
     fn drop(&mut self) {
         if let Some(operation) = self.operation {
             let mut last_task = false;
@@ -188,12 +194,62 @@ impl Drop for DataStore {
                     });
 
             if remove_from_cache {
-                DATASTORE_MAP.lock().unwrap().remove(self.name());
+                DATASTORE_MAP_READ.lock().unwrap().remove(self.name());
+                DATASTORE_MAP_WRITE.lock().unwrap().remove(self.name());
             }
         }
     }
 }
 
+impl DataStore<L> {
+    pub fn lookup_datastore(name: &str) -> Result<Arc<Self>, Error> {
+        let (config, digest, _lock) = Self::read_config(name)?;
+        let chunk_store = Arc::new(ChunkStore::open_lookup(name, config.absolute_path())?);
+        let tuning: DatastoreTuning = serde_json::from_value(
+            DatastoreTuning::API_SCHEMA
+                .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
+        )?;
+        let store = DataStoreImpl {
+            chunk_store,
+            gc_mutex: Mutex::new(()),
+            last_gc_status: Mutex::new(GarbageCollectionStatus::default()),
+            verify_new: config.verify_new.unwrap_or(false),
+            chunk_order: tuning.chunk_order.unwrap_or_default(),
+            last_digest: Some(digest),
+            sync_level: tuning.sync_level.unwrap_or_default(),
+        };
+
+        Ok(Arc::new(Self {
+            inner: Arc::new(store),
+            operation: Some(Operation::Lookup),
+        }))
+    }
+}
+
+impl DataStore<R> {
+    pub fn lookup_datastore_read(name: &str) -> Result<Arc<Self>, Error> {
+        let mut datastore_cache = DATASTORE_MAP_READ.lock().unwrap();
+        let cache_entry = datastore_cache.get(name);
+        let store = Self::open_datastore(name, Some(Operation::Read), cache_entry.cloned())?;
+        if cache_entry.is_none() {
+            datastore_cache.insert(name.to_string(), store.inner.clone());
+        }
+        Ok(store)
+    }
+}
+
+impl DataStore<W> {
+    pub fn lookup_datastore_write(name: &str) -> Result<Arc<Self>, Error> {
+        let mut datastore_cache = DATASTORE_MAP_WRITE.lock().unwrap();
+        let cache_entry = datastore_cache.get(name);
+        let store = Self::open_datastore(name, Some(Operation::Write), cache_entry.cloned())?;
+        if cache_entry.is_none() {
+            datastore_cache.insert(name.to_string(), store.inner.clone());
+        }
+        Ok(store)
+    }
+}
+
 impl DataStore {
     // This one just panics on everything
     #[doc(hidden)]
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:33 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:33 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 00/12] introduce typestate for
 datastore/chunkstore
Message-ID: <20250526141445.228717-1-h.laimer@proxmox.com>

This patch series introduces two traits, CanRead and CanWrite, to define whether
a datastore reference is readable, writable, or neither. Functions that read
or write are now implemented in `impl<T: CanRead>` or `impl<T: CanWrite>` blocks, ensuring
that they are only available to references that are supposed to read/write.

Motivation:
Currently, we track the number of read/write references of a datastore but we don't
track Lookup operations as they don't read or write, they still need a chunkstore, so
eventhough they don't neccessarily directly do IO, they hold an open file handle.
This is a problem for things like unmounting, currently lookup operations are only really
short, so you'd need really unlucky timing to actually run into problems, but still,
if a datastore is in "offline" maintenance mode, we shouldn't open filehandles on it.

By encoding state in the type:
1. We can assign non-readable/writable references for lookup operations.
2. The compiler ensures correct usage of references. Since it is easy to miss
    what might happen a few function calls down the line, having the compiler
    yell at you for easily missed things like this, is a really good thing
    I think.

Changes:
* Added CanRead and CanWrite traits.
* Separated functions into impl<T: CanRead> or impl<T: CanWrite>.
* Introduced three new datastore lookup functions that return concrete types implementing
   CanRead, CanWrite, or neither.
* Renamed lookup_datastore() to open_datastore() and made it private.

The main downside is needing separate datastore caches for read and write references due to
concrete type requirements in the cache HashMap.

Almost all changes are either adding generics or moving functions into the appropriate
trait implementations. The logic itself is only touched three times
- once in datastore_lookup()
- once check_privs_and_load_store() in /api/admin/datastore, this function now only checks 
   the privs, the datastore opening happens in the endpoint function directly. 
-(new in v2) and the checking of if a gc is currently running is now done without the need for a datastore reference
   instead we just try to get the gc lock directly from the cached write reference(only if one even exists)
   of the datastore in question. This was only used once by the job scheduler, now we just call a function that
   checks the relevant cache entries instead of actually getting the whole store reference.

changes since v1:
- seal trait implementations
- re-structure patches
- changed how checking if gc is running is done
- "rebased" onto master, was actually mostly rewritten, given the age and type of changes it just wouldn't really
   apply all that well anymore...
- we used Operation::Read for verification, turns out verification does also rename currupted chunks, only noticed because
   the compiler yelled at me :). Not necessarily changed from v1, but didn't mention it there.

--
Since I didn't add new comp times for v1, @Wolfgang suggested to maybe monomorphise some
functions manually to potentially reduce the impact on comp time/binary sizes. But given the 
minimal differences on comp time and binary sizes, I don't think that would be worth the
effort.

Binary sizes were unchanged(`ls -lah`).

Compile times:
        | dbg  | release
--------|------|---------
master  | 52s  | 92s
series  | 53s  | 94s

individual measurements:
* master -> dbg: 52s,52s,53s release: 92s,93s,92s
* series -> dbg: 53s,53s,53s release: 94s,96s,95s

Hannes Laimer (12):
  chunkstore: add CanRead and CanWrite trait
  chunkstore: separate functions into impl block
  datastore: add generics and new lookup functions
  datastore: separate functions into impl block
  backup_info: add generics and separate functions into impl blocks
  pbs-datastore: add generics and separate functions into impl blocks
  api: backup: env: add generics and separate functions into impl block
  api/backup/bin/server/tape: add missing generics
  examples/tests: add missing generics
  api: admin: pull datastore loading out of check_privs helper
  datastore: move `fn gc_running` out of DataStoreImpl
  api/server: replace datastore_lookup with new, state-typed datastore
    returning functions

 pbs-datastore/examples/ls-snapshots.rs      |    4 +-
 pbs-datastore/src/backup_info.rs            |  579 ++++----
 pbs-datastore/src/chunk_store.rs            |  329 +++--
 pbs-datastore/src/datastore.rs              | 1342 ++++++++++---------
 pbs-datastore/src/dynamic_index.rs          |   22 +-
 pbs-datastore/src/fixed_index.rs            |   50 +-
 pbs-datastore/src/hierarchy.rs              |   92 +-
 pbs-datastore/src/lib.rs                    |    3 +-
 pbs-datastore/src/local_chunk_reader.rs     |   13 +-
 pbs-datastore/src/prune.rs                  |   19 +-
 pbs-datastore/src/snapshot_reader.rs        |   31 +-
 src/api2/admin/datastore.rs                 |  161 +--
 src/api2/admin/namespace.rs                 |   10 +-
 src/api2/backup/environment.rs              |  337 ++---
 src/api2/backup/mod.rs                      |   29 +-
 src/api2/backup/upload_chunk.rs             |   19 +-
 src/api2/config/datastore.rs                |    5 +-
 src/api2/reader/environment.rs              |   30 +-
 src/api2/reader/mod.rs                      |   13 +-
 src/api2/status/mod.rs                      |    8 +-
 src/api2/tape/backup.rs                     |   21 +-
 src/api2/tape/drive.rs                      |    3 +-
 src/api2/tape/restore.rs                    |   83 +-
 src/backup/hierarchy.rs                     |   23 +-
 src/backup/verify.rs                        |   53 +-
 src/bin/proxmox-backup-proxy.rs             |   26 +-
 src/server/gc_job.rs                        |    7 +-
 src/server/prune_job.rs                     |    9 +-
 src/server/pull.rs                          |   32 +-
 src/server/push.rs                          |    7 +-
 src/server/sync.rs                          |   13 +-
 src/server/verify_job.rs                    |    4 +-
 src/tape/file_formats/snapshot_archive.rs   |    5 +-
 src/tape/pool_writer/mod.rs                 |   11 +-
 src/tape/pool_writer/new_chunks_iterator.rs |    7 +-
 tests/prune.rs                              |    8 +-
 36 files changed, 1794 insertions(+), 1614 deletions(-)

-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:35 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:35 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 02/12] chunkstore: separate
 functions into impl block
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-3-h.laimer@proxmox.com>

... based on whether they are reading/writing.

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 pbs-datastore/src/chunk_store.rs | 305 +++++++++++++++++--------------
 1 file changed, 169 insertions(+), 136 deletions(-)

diff --git a/pbs-datastore/src/chunk_store.rs b/pbs-datastore/src/chunk_store.rs
index 9a77bef2..e998c798 100644
--- a/pbs-datastore/src/chunk_store.rs
+++ b/pbs-datastore/src/chunk_store.rs
@@ -88,30 +88,29 @@ fn digest_to_prefix(digest: &[u8]) -> PathBuf {
     path.into()
 }
 
-impl ChunkStore {
-    #[doc(hidden)]
-    pub unsafe fn panic_store() -> Self {
-        Self {
-            name: String::new(),
-            base: PathBuf::new(),
-            chunk_dir: PathBuf::new(),
-            mutex: Mutex::new(()),
-            locker: None,
-            sync_level: Default::default(),
-        }
-    }
+impl ChunkStore<Lookup> {
+    pub fn open_lookup<P: Into<PathBuf>>(name: &str, base: P) -> Result<Self, Error> {
+        let base: PathBuf = base.into();
 
-    fn chunk_dir<P: AsRef<Path>>(path: P) -> PathBuf {
-        let mut chunk_dir: PathBuf = PathBuf::from(path.as_ref());
-        chunk_dir.push(".chunks");
+        if !base.is_absolute() {
+            bail!("expected absolute path - got {:?}", base);
+        }
 
-        chunk_dir
-    }
+        let chunk_dir = Self::chunk_dir(&base);
 
-    pub fn base(&self) -> &Path {
-        &self.base
+        Ok(Self {
+            name: name.to_owned(),
+            base,
+            chunk_dir,
+            locker: None,
+            mutex: Mutex::new(()),
+            sync_level: DatastoreFSyncLevel::None,
+            _marker: std::marker::PhantomData,
+        })
     }
+}
 
+impl ChunkStore<Write> {
     pub fn create<P>(
         name: &str,
         path: P,
@@ -174,13 +173,9 @@ impl ChunkStore {
 
         Self::open(name, base, sync_level)
     }
+}
 
-    fn lockfile_path<P: Into<PathBuf>>(base: P) -> PathBuf {
-        let mut lockfile_path: PathBuf = base.into();
-        lockfile_path.push(".lock");
-        lockfile_path
-    }
-
+impl<T: CanRead> ChunkStore<T> {
     /// Check if the chunkstore path is absolute and that we can
     /// access it. Returns the absolute '.chunks' path on success.
     fn chunk_dir_accessible(base: &Path) -> Result<PathBuf, Error> {
@@ -209,7 +204,7 @@ impl ChunkStore {
     ) -> Result<Self, Error> {
         let base: PathBuf = base.into();
 
-        let chunk_dir = ChunkStore::chunk_dir_accessible(&base)?;
+        let chunk_dir = Self::chunk_dir_accessible(&base)?;
 
         let lockfile_path = Self::lockfile_path(&base);
 
@@ -222,59 +217,10 @@ impl ChunkStore {
             locker: Some(locker),
             mutex: Mutex::new(()),
             sync_level,
+            _marker: std::marker::PhantomData,
         })
     }
 
-    pub fn touch_chunk(&self, digest: &[u8; 32]) -> Result<(), Error> {
-        // unwrap: only `None` in unit tests
-        assert!(self.locker.is_some());
-
-        self.cond_touch_chunk(digest, true)?;
-        Ok(())
-    }
-
-    pub fn cond_touch_chunk(&self, digest: &[u8; 32], assert_exists: bool) -> Result<bool, Error> {
-        // unwrap: only `None` in unit tests
-        assert!(self.locker.is_some());
-
-        let (chunk_path, _digest_str) = self.chunk_path(digest);
-        self.cond_touch_path(&chunk_path, assert_exists)
-    }
-
-    pub fn cond_touch_path(&self, path: &Path, assert_exists: bool) -> Result<bool, Error> {
-        // unwrap: only `None` in unit tests
-        assert!(self.locker.is_some());
-
-        let times: [libc::timespec; 2] = [
-            // access time -> update to now
-            libc::timespec {
-                tv_sec: 0,
-                tv_nsec: libc::UTIME_NOW,
-            },
-            // modification time -> keep as is
-            libc::timespec {
-                tv_sec: 0,
-                tv_nsec: libc::UTIME_OMIT,
-            },
-        ];
-
-        use nix::NixPath;
-
-        let res = path.with_nix_path(|cstr| unsafe {
-            let tmp = libc::utimensat(-1, cstr.as_ptr(), &times[0], libc::AT_SYMLINK_NOFOLLOW);
-            nix::errno::Errno::result(tmp)
-        })?;
-
-        if let Err(err) = res {
-            if !assert_exists && err == nix::errno::Errno::ENOENT {
-                return Ok(false);
-            }
-            bail!("update atime failed for chunk/file {path:?} - {err}");
-        }
-
-        Ok(true)
-    }
-
     pub fn get_chunk_iterator(
         &self,
     ) -> Result<
@@ -370,10 +316,116 @@ impl ChunkStore {
         .fuse())
     }
 
+    /// Checks permissions and owner of passed path.
+    fn check_permissions<P: AsRef<Path>>(path: P, file_mode: u32) -> Result<(), Error> {
+        match nix::sys::stat::stat(path.as_ref()) {
+            Ok(stat) => {
+                if stat.st_uid != u32::from(pbs_config::backup_user()?.uid)
+                    || stat.st_gid != u32::from(pbs_config::backup_group()?.gid)
+                    || stat.st_mode & 0o777 != file_mode
+                {
+                    bail!(
+                              "unable to open existing chunk store path {:?} - permissions or owner not correct",
+                              path.as_ref(),
+                          );
+                }
+            }
+            Err(err) => {
+                bail!(
+                    "unable to open existing chunk store path {:?} - {err}",
+                    path.as_ref(),
+                );
+            }
+        }
+        Ok(())
+    }
+
+    /// Verify vital files in datastore. Checks the owner and permissions of: the chunkstore, it's
+    /// subdirectories and the lock file.
+    pub fn verify_chunkstore<P: AsRef<Path>>(path: P) -> Result<(), Error> {
+        // Check datastore root path perm/owner
+        Self::check_permissions(path.as_ref(), 0o755)?;
+
+        let chunk_dir = Self::chunk_dir(path.as_ref());
+        // Check datastore .chunks path perm/owner
+        Self::check_permissions(&chunk_dir, 0o750)?;
+
+        // Check all .chunks subdirectories
+        for i in 0..64 * 1024 {
+            let mut l1path = chunk_dir.clone();
+            l1path.push(format!("{:04x}", i));
+            Self::check_permissions(&l1path, 0o750)?;
+        }
+
+        // Check .lock file
+        let lockfile_path = Self::lockfile_path(path.as_ref());
+        Self::check_permissions(lockfile_path, 0o644)?;
+        Ok(())
+    }
+
+    pub fn try_shared_lock(&self) -> Result<ProcessLockSharedGuard, Error> {
+        // unwrap: only `None` in unit tests
+        ProcessLocker::try_shared_lock(self.locker.clone().unwrap())
+    }
     pub fn oldest_writer(&self) -> Option<i64> {
         // unwrap: only `None` in unit tests
         ProcessLocker::oldest_shared_lock(self.locker.clone().unwrap())
     }
+}
+
+impl<T: CanWrite> ChunkStore<T> {
+    pub fn touch_chunk(&self, digest: &[u8; 32]) -> Result<(), Error> {
+        // unwrap: only `None` in unit tests
+        assert!(self.locker.is_some());
+
+        self.cond_touch_chunk(digest, true)?;
+        Ok(())
+    }
+
+    pub fn cond_touch_chunk(&self, digest: &[u8; 32], assert_exists: bool) -> Result<bool, Error> {
+        // unwrap: only `None` in unit tests
+        assert!(self.locker.is_some());
+
+        let (chunk_path, _digest_str) = self.chunk_path(digest);
+        self.cond_touch_path(&chunk_path, assert_exists)
+    }
+
+    pub fn cond_touch_path(&self, path: &Path, assert_exists: bool) -> Result<bool, Error> {
+        // unwrap: only `None` in unit tests
+        assert!(self.locker.is_some());
+
+        const UTIME_NOW: i64 = (1 << 30) - 1;
+        const UTIME_OMIT: i64 = (1 << 30) - 2;
+
+        let times: [libc::timespec; 2] = [
+            // access time -> update to now
+            libc::timespec {
+                tv_sec: 0,
+                tv_nsec: UTIME_NOW,
+            },
+            // modification time -> keep as is
+            libc::timespec {
+                tv_sec: 0,
+                tv_nsec: UTIME_OMIT,
+            },
+        ];
+
+        use nix::NixPath;
+
+        let res = path.with_nix_path(|cstr| unsafe {
+            let tmp = libc::utimensat(-1, cstr.as_ptr(), &times[0], libc::AT_SYMLINK_NOFOLLOW);
+            nix::errno::Errno::result(tmp)
+        })?;
+
+        if let Err(err) = res {
+            if !assert_exists && err == nix::errno::Errno::ENOENT {
+                return Ok(false);
+            }
+            bail!("update atime failed for chunk/file {path:?} - {err}");
+        }
+
+        Ok(true)
+    }
 
     pub fn sweep_unused_chunks(
         &self,
@@ -611,6 +663,43 @@ impl ChunkStore {
         Ok((false, encoded_size))
     }
 
+    pub fn try_exclusive_lock(&self) -> Result<ProcessLockExclusiveGuard, Error> {
+        // unwrap: only `None` in unit tests
+        ProcessLocker::try_exclusive_lock(self.locker.clone().unwrap())
+    }
+}
+
+impl<T> ChunkStore<T> {
+    #[doc(hidden)]
+    pub fn dummy_store() -> Self {
+        Self {
+            name: String::new(),
+            base: PathBuf::new(),
+            chunk_dir: PathBuf::new(),
+            mutex: Mutex::new(()),
+            locker: None,
+            sync_level: Default::default(),
+            _marker: std::marker::PhantomData,
+        }
+    }
+
+    fn chunk_dir<P: AsRef<Path>>(path: P) -> PathBuf {
+        let mut chunk_dir: PathBuf = PathBuf::from(path.as_ref());
+        chunk_dir.push(".chunks");
+
+        chunk_dir
+    }
+
+    pub fn base(&self) -> &Path {
+        &self.base
+    }
+
+    fn lockfile_path<P: Into<PathBuf>>(base: P) -> PathBuf {
+        let mut lockfile_path: PathBuf = base.into();
+        lockfile_path.push(".lock");
+        lockfile_path
+    }
+
     pub fn chunk_path(&self, digest: &[u8; 32]) -> (PathBuf, String) {
         // unwrap: only `None` in unit tests
         assert!(self.locker.is_some());
@@ -642,63 +731,6 @@ impl ChunkStore {
 
         self.base.clone()
     }
-
-    pub fn try_shared_lock(&self) -> Result<ProcessLockSharedGuard, Error> {
-        // unwrap: only `None` in unit tests
-        ProcessLocker::try_shared_lock(self.locker.clone().unwrap())
-    }
-
-    pub fn try_exclusive_lock(&self) -> Result<ProcessLockExclusiveGuard, Error> {
-        // unwrap: only `None` in unit tests
-        ProcessLocker::try_exclusive_lock(self.locker.clone().unwrap())
-    }
-
-    /// Checks permissions and owner of passed path.
-    fn check_permissions<T: AsRef<Path>>(path: T, file_mode: u32) -> Result<(), Error> {
-        match nix::sys::stat::stat(path.as_ref()) {
-            Ok(stat) => {
-                if stat.st_uid != u32::from(pbs_config::backup_user()?.uid)
-                    || stat.st_gid != u32::from(pbs_config::backup_group()?.gid)
-                    || stat.st_mode & 0o777 != file_mode
-                {
-                    bail!(
-                            "unable to open existing chunk store path {:?} - permissions or owner not correct",
-                            path.as_ref(),
-                        );
-                }
-            }
-            Err(err) => {
-                bail!(
-                    "unable to open existing chunk store path {:?} - {err}",
-                    path.as_ref(),
-                );
-            }
-        }
-        Ok(())
-    }
-
-    /// Verify vital files in datastore. Checks the owner and permissions of: the chunkstore, it's
-    /// subdirectories and the lock file.
-    pub fn verify_chunkstore<T: AsRef<Path>>(path: T) -> Result<(), Error> {
-        // Check datastore root path perm/owner
-        ChunkStore::check_permissions(path.as_ref(), 0o755)?;
-
-        let chunk_dir = Self::chunk_dir(path.as_ref());
-        // Check datastore .chunks path perm/owner
-        ChunkStore::check_permissions(&chunk_dir, 0o750)?;
-
-        // Check all .chunks subdirectories
-        for i in 0..64 * 1024 {
-            let mut l1path = chunk_dir.clone();
-            l1path.push(format!("{:04x}", i));
-            ChunkStore::check_permissions(&l1path, 0o750)?;
-        }
-
-        // Check .lock file
-        let lockfile_path = Self::lockfile_path(path.as_ref());
-        ChunkStore::check_permissions(lockfile_path, 0o644)?;
-        Ok(())
-    }
 }
 
 #[test]
@@ -708,13 +740,14 @@ fn test_chunk_store1() {
 
     if let Err(_e) = std::fs::remove_dir_all(".testdir") { /* ignore */ }
 
-    let chunk_store = ChunkStore::open("test", &path, DatastoreFSyncLevel::None);
+    let chunk_store: Result<ChunkStore<Read>, _> =
+        ChunkStore::open("test", &path, DatastoreFSyncLevel::None);
     assert!(chunk_store.is_err());
 
     let user = nix::unistd::User::from_uid(nix::unistd::Uid::current())
         .unwrap()
         .unwrap();
-    let chunk_store =
+    let chunk_store: ChunkStore<Write> =
         ChunkStore::create("test", &path, user.uid, user.gid, DatastoreFSyncLevel::None).unwrap();
 
     let (chunk, digest) = crate::data_blob::DataChunkBuilder::new(&[0u8, 1u8])
@@ -727,7 +760,7 @@ fn test_chunk_store1() {
     let (exists, _) = chunk_store.insert_chunk(&chunk, &digest).unwrap();
     assert!(exists);
 
-    let chunk_store =
+    let chunk_store: Result<ChunkStore<Write>, _> =
         ChunkStore::create("test", &path, user.uid, user.gid, DatastoreFSyncLevel::None);
     assert!(chunk_store.is_err());
 
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:44 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:44 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 11/12] datastore: move `fn
 gc_running` out of DataStoreImpl
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-12-h.laimer@proxmox.com>

Like this we can avoid having to get a `CanWrite` datastore reference
just to check if we can obtain the gc lock, as lookup references
(neither `CanRead` nor `CanWrite`) will not come from the cache that
would contain the relevant locks.

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 pbs-datastore/src/datastore.rs  | 10 ++++++----
 pbs-datastore/src/lib.rs        |  3 ++-
 src/bin/proxmox-backup-proxy.rs | 15 ++-------------
 3 files changed, 10 insertions(+), 18 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index cb2d2172..20ad73c5 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -118,6 +118,12 @@ pub fn ensure_datastore_is_mounted(config: &DataStoreConfig) -> Result<(), Error
     }
 }
 
+pub fn is_garbage_collection_running(name: &str) -> bool {
+    let datastore_cache = DATASTORE_MAP_WRITE.lock().unwrap();
+    let cache_entry = datastore_cache.get(name);
+    cache_entry.is_some_and(|s| s.gc_mutex.try_lock().is_err())
+}
+
 /// Datastore Management
 ///
 /// A Datastore can store severals backups, and provides the
@@ -752,10 +758,6 @@ impl<T: CanRead> DataStore<T> {
         self.inner.last_gc_status.lock().unwrap().clone()
     }
 
-    pub fn garbage_collection_running(&self) -> bool {
-        self.inner.gc_mutex.try_lock().is_err()
-    }
-
     pub fn try_shared_chunk_store_lock(&self) -> Result<ProcessLockSharedGuard, Error> {
         self.inner.chunk_store.try_shared_lock()
     }
diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index 5014b6c0..857ee78e 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -202,7 +202,8 @@ pub use store_progress::StoreProgress;
 
 mod datastore;
 pub use datastore::{
-    check_backup_owner, ensure_datastore_is_mounted, get_datastore_mount_status, DataStore,
+    check_backup_owner, ensure_datastore_is_mounted, get_datastore_mount_status,
+    is_garbage_collection_running, DataStore,
 };
 
 mod hierarchy;
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index bda2f17b..643a2dbd 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -482,19 +482,8 @@ async fn schedule_datastore_garbage_collection() {
             }
         };
 
-        {
-            // limit datastore scope due to Op::Lookup
-            let datastore = match DataStore::lookup_datastore(&store, Some(Operation::Lookup)) {
-                Ok(datastore) => datastore,
-                Err(err) => {
-                    eprintln!("lookup_datastore failed - {err}");
-                    continue;
-                }
-            };
-
-            if datastore.garbage_collection_running() {
-                continue;
-            }
+        if is_garbage_collection_running(&store) {
+            continue;
         }
 
         let worker_type = "garbage_collection";
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:40 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:40 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 07/12] api: backup: env: add
 generics and separate functions into impl block
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-8-h.laimer@proxmox.com>

... based on whether they read or write.

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 src/api2/backup/environment.rs | 337 +++++++++++++++++----------------
 1 file changed, 174 insertions(+), 163 deletions(-)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 3d541b46..a1620fb9 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -13,6 +13,7 @@ use proxmox_sys::fs::{replace_file, CreateOptions};
 
 use pbs_api_types::Authid;
 use pbs_datastore::backup_info::{BackupDir, BackupInfo};
+use pbs_datastore::chunk_store::CanWrite;
 use pbs_datastore::dynamic_index::DynamicIndexWriter;
 use pbs_datastore::fixed_index::FixedIndexWriter;
 use pbs_datastore::{DataBlob, DataStore};
@@ -54,17 +55,17 @@ impl std::ops::Add for UploadStatistic {
     }
 }
 
-struct DynamicWriterState {
+struct DynamicWriterState<T> {
     name: String,
-    index: DynamicIndexWriter,
+    index: DynamicIndexWriter<T>,
     offset: u64,
     chunk_count: u64,
     upload_stat: UploadStatistic,
 }
 
-struct FixedWriterState {
+struct FixedWriterState<T> {
     name: String,
-    index: FixedIndexWriter,
+    index: FixedIndexWriter<T>,
     size: usize,
     chunk_size: u32,
     chunk_count: u64,
@@ -76,18 +77,18 @@ struct FixedWriterState {
 // key=digest, value=length
 type KnownChunksMap = HashMap<[u8; 32], u32>;
 
-struct SharedBackupState {
+struct SharedBackupState<T> {
     finished: bool,
     uid_counter: usize,
     file_counter: usize, // successfully uploaded files
-    dynamic_writers: HashMap<usize, DynamicWriterState>,
-    fixed_writers: HashMap<usize, FixedWriterState>,
+    dynamic_writers: HashMap<usize, DynamicWriterState<T>>,
+    fixed_writers: HashMap<usize, FixedWriterState<T>>,
     known_chunks: KnownChunksMap,
     backup_size: u64, // sums up size of all files
     backup_stat: UploadStatistic,
 }
 
-impl SharedBackupState {
+impl<T> SharedBackupState<T> {
     // Raise error if finished flag is set
     fn ensure_unfinished(&self) -> Result<(), Error> {
         if self.finished {
@@ -105,26 +106,32 @@ impl SharedBackupState {
 
 /// `RpcEnvironment` implementation for backup service
 #[derive(Clone)]
-pub struct BackupEnvironment {
+pub struct BackupEnvironment<T> {
     env_type: RpcEnvironmentType,
     result_attributes: Value,
     auth_id: Authid,
     pub debug: bool,
     pub formatter: &'static dyn OutputFormatter,
     pub worker: Arc<WorkerTask>,
-    pub datastore: Arc<DataStore>,
-    pub backup_dir: BackupDir,
-    pub last_backup: Option<BackupInfo>,
-    state: Arc<Mutex<SharedBackupState>>,
+    pub datastore: Arc<DataStore<T>>,
+    pub backup_dir: BackupDir<T>,
+    pub last_backup: Option<BackupInfo<T>>,
+    state: Arc<Mutex<SharedBackupState<T>>>,
 }
 
-impl BackupEnvironment {
+impl<T: Send + Sync + 'static> BackupEnvironment<T> {
+    pub fn format_response(&self, result: Result<Value, Error>) -> Response<Body> {
+        self.formatter.format_result(result, self)
+    }
+}
+
+impl<T> BackupEnvironment<T> {
     pub fn new(
         env_type: RpcEnvironmentType,
         auth_id: Authid,
         worker: Arc<WorkerTask>,
-        datastore: Arc<DataStore>,
-        backup_dir: BackupDir,
+        datastore: Arc<DataStore<T>>,
+        backup_dir: BackupDir<T>,
     ) -> Self {
         let state = SharedBackupState {
             finished: false,
@@ -260,10 +267,148 @@ impl BackupEnvironment {
         state.known_chunks.get(digest).copied()
     }
 
+    fn log_upload_stat(
+        &self,
+        archive_name: &str,
+        csum: &[u8; 32],
+        uuid: &[u8; 16],
+        size: u64,
+        chunk_count: u64,
+        upload_stat: &UploadStatistic,
+    ) {
+        self.log(format!("Upload statistics for '{}'", archive_name));
+        self.log(format!("UUID: {}", hex::encode(uuid)));
+        self.log(format!("Checksum: {}", hex::encode(csum)));
+        self.log(format!("Size: {}", size));
+        self.log(format!("Chunk count: {}", chunk_count));
+
+        if size == 0 || chunk_count == 0 {
+            return;
+        }
+
+        self.log(format!(
+            "Upload size: {} ({}%)",
+            upload_stat.size,
+            (upload_stat.size * 100) / size
+        ));
+
+        // account for zero chunk, which might be uploaded but never used
+        let client_side_duplicates = if chunk_count < upload_stat.count {
+            0
+        } else {
+            chunk_count - upload_stat.count
+        };
+
+        let server_side_duplicates = upload_stat.duplicates;
+
+        if (client_side_duplicates + server_side_duplicates) > 0 {
+            let per = (client_side_duplicates + server_side_duplicates) * 100 / chunk_count;
+            self.log(format!(
+                "Duplicates: {}+{} ({}%)",
+                client_side_duplicates, server_side_duplicates, per
+            ));
+        }
+
+        if upload_stat.size > 0 {
+            self.log(format!(
+                "Compression: {}%",
+                (upload_stat.compressed_size * 100) / upload_stat.size
+            ));
+        }
+    }
+
+    pub fn log<S: AsRef<str>>(&self, msg: S) {
+        info!("{}", msg.as_ref());
+    }
+
+    pub fn debug<S: AsRef<str>>(&self, msg: S) {
+        if self.debug {
+            // This is kinda weird, we would like to use tracing::debug! here and automatically
+            // filter it, but self.debug is set from the client-side and the logs are printed on
+            // client and server side. This means that if the client sets the log level to debug,
+            // both server and client need to have 'debug' logs printed.
+            self.log(msg);
+        }
+    }
+
+    /// Raise error if finished flag is not set
+    pub fn ensure_finished(&self) -> Result<(), Error> {
+        let state = self.state.lock().unwrap();
+        if !state.finished {
+            bail!("backup ended but finished flag is not set.");
+        }
+        Ok(())
+    }
+
+    /// Return true if the finished flag is set
+    pub fn finished(&self) -> bool {
+        let state = self.state.lock().unwrap();
+        state.finished
+    }
+}
+
+impl<T: CanWrite + Send + Sync + std::panic::RefUnwindSafe + 'static> BackupEnvironment<T> {
+    /// If verify-new is set on the datastore, this will run a new verify task
+    /// for the backup. If not, this will return and also drop the passed lock
+    /// immediately.
+    pub fn verify_after_complete(&self, excl_snap_lock: BackupLockGuard) -> Result<(), Error> {
+        self.ensure_finished()?;
+
+        if !self.datastore.verify_new() {
+            // no verify requested, do nothing
+            return Ok(());
+        }
+
+        // Downgrade to shared lock, the backup itself is finished
+        drop(excl_snap_lock);
+        let snap_lock = self.backup_dir.lock_shared().with_context(|| {
+            format!(
+                "while trying to verify snapshot '{:?}' after completion",
+                self.backup_dir
+            )
+        })?;
+        let worker_id = format!(
+            "{}:{}/{}/{:08X}",
+            self.datastore.name(),
+            self.backup_dir.backup_type(),
+            self.backup_dir.backup_id(),
+            self.backup_dir.backup_time()
+        );
+
+        let datastore = self.datastore.clone();
+        let backup_dir = self.backup_dir.clone();
+
+        WorkerTask::new_thread(
+            "verify",
+            Some(worker_id),
+            self.auth_id.to_string(),
+            false,
+            move |worker| {
+                worker.log_message("Automatically verifying newly added snapshot");
+
+                let verify_worker = crate::backup::VerifyWorker::new(worker.clone(), datastore);
+                if !verify_backup_dir_with_lock(
+                    &verify_worker,
+                    &backup_dir,
+                    worker.upid().clone(),
+                    None,
+                    snap_lock,
+                )? {
+                    bail!("verification failed - please check the log for details");
+                }
+
+                Ok(())
+            },
+        )
+        .map(|_| ())
+    }
+}
+
+impl<T: CanWrite> BackupEnvironment<T> {
     /// Store the writer with an unique ID
     pub fn register_dynamic_writer(
         &self,
-        index: DynamicIndexWriter,
+        index: DynamicIndexWriter<T>,
         name: String,
     ) -> Result<usize, Error> {
         let mut state = self.state.lock().unwrap();
@@ -289,7 +434,7 @@ impl BackupEnvironment {
     /// Store the writer with an unique ID
     pub fn register_fixed_writer(
         &self,
-        index: FixedIndexWriter,
+        index: FixedIndexWriter<T>,
         name: String,
         size: usize,
         chunk_size: u32,
@@ -379,56 +524,6 @@ impl BackupEnvironment {
         Ok(())
     }
 
-    fn log_upload_stat(
-        &self,
-        archive_name: &str,
-        csum: &[u8; 32],
-        uuid: &[u8; 16],
-        size: u64,
-        chunk_count: u64,
-        upload_stat: &UploadStatistic,
-    ) {
-        self.log(format!("Upload statistics for '{}'", archive_name));
-        self.log(format!("UUID: {}", hex::encode(uuid)));
-        self.log(format!("Checksum: {}", hex::encode(csum)));
-        self.log(format!("Size: {}", size));
-        self.log(format!("Chunk count: {}", chunk_count));
-
-        if size == 0 || chunk_count == 0 {
-            return;
-        }
-
-        self.log(format!(
-            "Upload size: {} ({}%)",
-            upload_stat.size,
-            (upload_stat.size * 100) / size
-        ));
-
-        // account for zero chunk, which might be uploaded but never used
-        let client_side_duplicates = if chunk_count < upload_stat.count {
-            0
-        } else {
-            chunk_count - upload_stat.count
-        };
-
-        let server_side_duplicates = upload_stat.duplicates;
-
-        if (client_side_duplicates + server_side_duplicates) > 0 {
-            let per = (client_side_duplicates + server_side_duplicates) * 100 / chunk_count;
-            self.log(format!(
-                "Duplicates: {}+{} ({}%)",
-                client_side_duplicates, server_side_duplicates, per
-            ));
-        }
-
-        if upload_stat.size > 0 {
-            self.log(format!(
-                "Compression: {}%",
-                (upload_stat.compressed_size * 100) / upload_stat.size
-            ));
-        }
-    }
-
     /// Close dynamic writer
     pub fn dynamic_writer_close(
         &self,
@@ -633,94 +728,6 @@ impl BackupEnvironment {
         Ok(())
     }
 
-    /// If verify-new is set on the datastore, this will run a new verify task
-    /// for the backup. If not, this will return and also drop the passed lock
-    /// immediately.
-    pub fn verify_after_complete(&self, excl_snap_lock: BackupLockGuard) -> Result<(), Error> {
-        self.ensure_finished()?;
-
-        if !self.datastore.verify_new() {
-            // no verify requested, do nothing
-            return Ok(());
-        }
-
-        // Downgrade to shared lock, the backup itself is finished
-        drop(excl_snap_lock);
-        let snap_lock = self.backup_dir.lock_shared().with_context(|| {
-            format!(
-                "while trying to verify snapshot '{:?}' after completion",
-                self.backup_dir
-            )
-        })?;
-        let worker_id = format!(
-            "{}:{}/{}/{:08X}",
-            self.datastore.name(),
-            self.backup_dir.backup_type(),
-            self.backup_dir.backup_id(),
-            self.backup_dir.backup_time()
-        );
-
-        let datastore = self.datastore.clone();
-        let backup_dir = self.backup_dir.clone();
-
-        WorkerTask::new_thread(
-            "verify",
-            Some(worker_id),
-            self.auth_id.to_string(),
-            false,
-            move |worker| {
-                worker.log_message("Automatically verifying newly added snapshot");
-
-                let verify_worker = crate::backup::VerifyWorker::new(worker.clone(), datastore);
-                if !verify_backup_dir_with_lock(
-                    &verify_worker,
-                    &backup_dir,
-                    worker.upid().clone(),
-                    None,
-                    snap_lock,
-                )? {
-                    bail!("verification failed - please check the log for details");
-                }
-
-                Ok(())
-            },
-        )
-        .map(|_| ())
-    }
-
-    pub fn log<S: AsRef<str>>(&self, msg: S) {
-        info!("{}", msg.as_ref());
-    }
-
-    pub fn debug<S: AsRef<str>>(&self, msg: S) {
-        if self.debug {
-            // This is kinda weird, we would like to use tracing::debug! here and automatically
-            // filter it, but self.debug is set from the client-side and the logs are printed on
-            // client and server side. This means that if the client sets the log level to debug,
-            // both server and client need to have 'debug' logs printed.
-            self.log(msg);
-        }
-    }
-
-    pub fn format_response(&self, result: Result<Value, Error>) -> Response<Body> {
-        self.formatter.format_result(result, self)
-    }
-
-    /// Raise error if finished flag is not set
-    pub fn ensure_finished(&self) -> Result<(), Error> {
-        let state = self.state.lock().unwrap();
-        if !state.finished {
-            bail!("backup ended but finished flag is not set.");
-        }
-        Ok(())
-    }
-
-    /// Return true if the finished flag is set
-    pub fn finished(&self) -> bool {
-        let state = self.state.lock().unwrap();
-        state.finished
-    }
-
     /// Remove complete backup
     pub fn remove_backup(&self) -> Result<(), Error> {
         let mut state = self.state.lock().unwrap();
@@ -736,7 +743,7 @@ impl BackupEnvironment {
     }
 }
 
-impl RpcEnvironment for BackupEnvironment {
+impl<T: Send + Sync + 'static> RpcEnvironment for BackupEnvironment<T> {
     fn result_attrib_mut(&mut self) -> &mut Value {
         &mut self.result_attributes
     }
@@ -758,14 +765,18 @@ impl RpcEnvironment for BackupEnvironment {
     }
 }
 
-impl AsRef<BackupEnvironment> for dyn RpcEnvironment {
-    fn as_ref(&self) -> &BackupEnvironment {
-        self.as_any().downcast_ref::<BackupEnvironment>().unwrap()
+impl<T: 'static> AsRef<BackupEnvironment<T>> for dyn RpcEnvironment {
+    fn as_ref(&self) -> &BackupEnvironment<T> {
+        self.as_any()
+            .downcast_ref::<BackupEnvironment<T>>()
+            .unwrap()
     }
 }
 
-impl AsRef<BackupEnvironment> for Box<dyn RpcEnvironment> {
-    fn as_ref(&self) -> &BackupEnvironment {
-        self.as_any().downcast_ref::<BackupEnvironment>().unwrap()
+impl<T: 'static> AsRef<BackupEnvironment<T>> for Box<dyn RpcEnvironment> {
+    fn as_ref(&self) -> &BackupEnvironment<T> {
+        self.as_any()
+            .downcast_ref::<BackupEnvironment<T>>()
+            .unwrap()
     }
 }
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:38 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:38 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 05/12] backup_info: add
 generics and separate functions into impl blocks
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-6-h.laimer@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 583 ++++++++++++++++---------------
 pbs-datastore/src/datastore.rs   |  26 +-
 2 files changed, 313 insertions(+), 296 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index d4732fdd..f3ca283c 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -18,7 +18,10 @@ use pbs_api_types::{
 use pbs_config::{open_backup_lockfile, BackupLockGuard};
 
 use crate::manifest::{BackupManifest, MANIFEST_LOCK_NAME};
-use crate::{DataBlob, DataStore};
+use crate::{
+    chunk_store::{CanRead, CanWrite},
+    DataBlob, DataStore,
+};
 
 pub const DATASTORE_LOCKS_DIR: &str = "/run/proxmox-backup/locks";
 
@@ -34,14 +37,14 @@ pub(crate) static OLD_LOCKING: LazyLock<bool> = LazyLock::new(|| {
 
 /// BackupGroup is a directory containing a list of BackupDir
 #[derive(Clone)]
-pub struct BackupGroup {
-    store: Arc<DataStore>,
+pub struct BackupGroup<T> {
+    store: Arc<DataStore<T>>,
 
     ns: BackupNamespace,
     group: pbs_api_types::BackupGroup,
 }
 
-impl fmt::Debug for BackupGroup {
+impl<T> fmt::Debug for BackupGroup<T> {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         f.debug_struct("BackupGroup")
             .field("store", &self.store.name())
@@ -51,45 +54,12 @@ impl fmt::Debug for BackupGroup {
     }
 }
 
-impl BackupGroup {
-    pub(crate) fn new(
-        store: Arc<DataStore>,
-        ns: BackupNamespace,
-        group: pbs_api_types::BackupGroup,
-    ) -> Self {
-        Self { store, ns, group }
-    }
-
-    /// Access the underlying [`BackupGroup`](pbs_api_types::BackupGroup).
-    #[inline]
-    pub fn group(&self) -> &pbs_api_types::BackupGroup {
-        &self.group
-    }
-
-    #[inline]
-    pub fn backup_ns(&self) -> &BackupNamespace {
-        &self.ns
-    }
-
-    #[inline]
-    pub fn backup_type(&self) -> BackupType {
-        self.group.ty
-    }
-
-    #[inline]
-    pub fn backup_id(&self) -> &str {
-        &self.group.id
-    }
-
-    pub fn full_group_path(&self) -> PathBuf {
-        self.store.group_path(&self.ns, &self.group)
-    }
-
-    pub fn relative_group_path(&self) -> PathBuf {
-        let mut path = self.ns.path();
-        path.push(self.group.ty.as_str());
-        path.push(&self.group.id);
-        path
+impl<T: CanRead> BackupGroup<T> {
+    /// Returns the backup owner.
+    ///
+    /// The backup owner is the entity who first created the backup group.
+    pub fn get_owner(&self) -> Result<Authid, Error> {
+        self.store.get_owner(&self.ns, self.as_ref())
     }
 
     /// Simple check whether a group exists. This does not check whether there are any snapshots,
@@ -98,7 +68,7 @@ impl BackupGroup {
         self.full_group_path().exists()
     }
 
-    pub fn list_backups(&self) -> Result<Vec<BackupInfo>, Error> {
+    pub fn list_backups(&self) -> Result<Vec<BackupInfo<T>>, Error> {
         let mut list = vec![];
 
         let path = self.full_group_path();
@@ -130,7 +100,7 @@ impl BackupGroup {
     }
 
     /// Finds the latest backup inside a backup group
-    pub fn last_backup(&self, only_finished: bool) -> Result<Option<BackupInfo>, Error> {
+    pub fn last_backup(&self, only_finished: bool) -> Result<Option<BackupInfo<T>>, Error> {
         let backups = self.list_backups()?;
         Ok(backups
             .into_iter()
@@ -190,24 +160,13 @@ impl BackupGroup {
 
         Ok(last)
     }
+}
 
-    pub fn matches(&self, filter: &GroupFilter) -> bool {
-        self.group.matches(filter)
-    }
-
-    pub fn backup_dir(&self, time: i64) -> Result<BackupDir, Error> {
-        BackupDir::with_group(self.clone(), time)
-    }
-
-    pub fn backup_dir_with_rfc3339<T: Into<String>>(
-        &self,
-        time_string: T,
-    ) -> Result<BackupDir, Error> {
-        BackupDir::with_rfc3339(self.clone(), time_string.into())
-    }
-
-    pub fn iter_snapshots(&self) -> Result<crate::ListSnapshots, Error> {
-        crate::ListSnapshots::new(self.clone())
+impl<T: CanWrite> BackupGroup<T> {
+    /// Set the backup owner.
+    pub fn set_owner(&self, auth_id: &Authid, force: bool) -> Result<(), Error> {
+        self.store
+            .set_owner(&self.ns, self.as_ref(), auth_id, force)
     }
 
     /// Destroy the group inclusive all its backup snapshots (BackupDir's)
@@ -260,32 +219,6 @@ impl BackupGroup {
         Ok(())
     }
 
-    /// Returns the backup owner.
-    ///
-    /// The backup owner is the entity who first created the backup group.
-    pub fn get_owner(&self) -> Result<Authid, Error> {
-        self.store.get_owner(&self.ns, self.as_ref())
-    }
-
-    /// Set the backup owner.
-    pub fn set_owner(&self, auth_id: &Authid, force: bool) -> Result<(), Error> {
-        self.store
-            .set_owner(&self.ns, self.as_ref(), auth_id, force)
-    }
-
-    /// Returns a file name for locking a group.
-    ///
-    /// The lock file will be located in:
-    /// `${DATASTORE_LOCKS_DIR}/${datastore name}/${lock_file_path_helper(rpath)}`
-    /// where `rpath` is the relative path of the group.
-    fn lock_path(&self) -> PathBuf {
-        let path = Path::new(DATASTORE_LOCKS_DIR).join(self.store.name());
-
-        let rpath = Path::new(self.group.ty.as_str()).join(&self.group.id);
-
-        path.join(lock_file_path_helper(&self.ns, rpath))
-    }
-
     /// Locks a group exclusively.
     pub fn lock(&self) -> Result<BackupLockGuard, Error> {
         if *OLD_LOCKING {
@@ -304,34 +237,108 @@ impl BackupGroup {
     }
 }
 
-impl AsRef<pbs_api_types::BackupNamespace> for BackupGroup {
+impl<T: Clone> BackupGroup<T> {
+    pub(crate) fn new(
+        store: Arc<DataStore<T>>,
+        ns: BackupNamespace,
+        group: pbs_api_types::BackupGroup,
+    ) -> Self {
+        Self { store, ns, group }
+    }
+
+    /// Access the underlying [`BackupGroup`](pbs_api_types::BackupGroup).
+    #[inline]
+    pub fn group(&self) -> &pbs_api_types::BackupGroup {
+        &self.group
+    }
+
+    #[inline]
+    pub fn backup_ns(&self) -> &BackupNamespace {
+        &self.ns
+    }
+
+    #[inline]
+    pub fn backup_type(&self) -> BackupType {
+        self.group.ty
+    }
+
+    #[inline]
+    pub fn backup_id(&self) -> &str {
+        &self.group.id
+    }
+
+    pub fn full_group_path(&self) -> PathBuf {
+        self.store.group_path(&self.ns, &self.group)
+    }
+
+    pub fn relative_group_path(&self) -> PathBuf {
+        let mut path = self.ns.path();
+        path.push(self.group.ty.as_str());
+        path.push(&self.group.id);
+        path
+    }
+
+    pub fn matches(&self, filter: &GroupFilter) -> bool {
+        self.group.matches(filter)
+    }
+
+    pub fn backup_dir(&self, time: i64) -> Result<BackupDir<T>, Error> {
+        BackupDir::with_group(self.clone(), time)
+    }
+
+    pub fn backup_dir_with_rfc3339<D: Into<String>>(
+        &self,
+        time_string: D,
+    ) -> Result<BackupDir<T>, Error> {
+        BackupDir::with_rfc3339(self.clone(), time_string.into())
+    }
+
+    pub fn iter_snapshots(&self) -> Result<crate::ListSnapshots, Error> {
+        crate::ListSnapshots::new(self.clone())
+    }
+
+    /// Returns a file name for locking a group.
+    ///
+    /// The lock file will be located in:
+    /// `${DATASTORE_LOCKS_DIR}/${datastore name}/${lock_file_path_helper(rpath)}`
+    /// where `rpath` is the relative path of the group.
+    fn lock_path(&self) -> PathBuf {
+        let path = Path::new(DATASTORE_LOCKS_DIR).join(self.store.name());
+
+        let rpath = Path::new(self.group.ty.as_str()).join(&self.group.id);
+
+        path.join(lock_file_path_helper(&self.ns, rpath))
+    }
+}
+
+impl<T> AsRef<pbs_api_types::BackupNamespace> for BackupGroup<T> {
     #[inline]
     fn as_ref(&self) -> &pbs_api_types::BackupNamespace {
         &self.ns
     }
 }
 
-impl AsRef<pbs_api_types::BackupGroup> for BackupGroup {
+impl<T> AsRef<pbs_api_types::BackupGroup> for BackupGroup<T> {
     #[inline]
     fn as_ref(&self) -> &pbs_api_types::BackupGroup {
         &self.group
     }
 }
 
-impl From<&BackupGroup> for pbs_api_types::BackupGroup {
-    fn from(group: &BackupGroup) -> pbs_api_types::BackupGroup {
+impl<T> From<&BackupGroup<T>> for pbs_api_types::BackupGroup {
+    fn from(group: &BackupGroup<T>) -> pbs_api_types::BackupGroup {
         group.group.clone()
     }
 }
 
-impl From<BackupGroup> for pbs_api_types::BackupGroup {
-    fn from(group: BackupGroup) -> pbs_api_types::BackupGroup {
+impl<T> From<BackupGroup<T>> for pbs_api_types::BackupGroup {
+    fn from(group: BackupGroup<T>) -> pbs_api_types::BackupGroup {
         group.group
     }
 }
 
-impl From<BackupDir> for BackupGroup {
-    fn from(dir: BackupDir) -> BackupGroup {
+impl<T> From<BackupDir<T>> for BackupGroup<T> {
+    fn from(dir: BackupDir<T>) -> BackupGroup<T> {
         BackupGroup {
             store: dir.store,
             ns: dir.ns,
@@ -340,8 +347,8 @@ impl From<BackupDir> for BackupGroup {
     }
 }
 
-impl From<&BackupDir> for BackupGroup {
-    fn from(dir: &BackupDir) -> BackupGroup {
+impl<T> From<&BackupDir<T>> for BackupGroup<T> {
+    fn from(dir: &BackupDir<T>) -> BackupGroup<T> {
         BackupGroup {
             store: Arc::clone(&dir.store),
             ns: dir.ns.clone(),
@@ -354,15 +361,15 @@ impl From<&BackupDir> for BackupGroup {
 ///
 /// We also call this a backup snaphost.
 #[derive(Clone)]
-pub struct BackupDir {
-    store: Arc<DataStore>,
+pub struct BackupDir<T> {
+    store: Arc<DataStore<T>>,
     ns: BackupNamespace,
     dir: pbs_api_types::BackupDir,
     // backup_time as rfc3339
     backup_time_string: String,
 }
 
-impl fmt::Debug for BackupDir {
+impl<T> fmt::Debug for BackupDir<T> {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         f.debug_struct("BackupDir")
             .field("store", &self.store.name())
@@ -373,102 +380,12 @@ impl fmt::Debug for BackupDir {
     }
 }
 
-impl BackupDir {
-    /// Temporarily used for tests.
-    #[doc(hidden)]
-    pub fn new_test(dir: pbs_api_types::BackupDir) -> Self {
-        Self {
-            store: unsafe { DataStore::new_test() },
-            backup_time_string: Self::backup_time_to_string(dir.time).unwrap(),
-            ns: BackupNamespace::root(),
-            dir,
-        }
-    }
-
-    pub(crate) fn with_group(group: BackupGroup, backup_time: i64) -> Result<Self, Error> {
-        let backup_time_string = Self::backup_time_to_string(backup_time)?;
-        Ok(Self {
-            store: group.store,
-            ns: group.ns,
-            dir: (group.group, backup_time).into(),
-            backup_time_string,
-        })
-    }
-
-    pub(crate) fn with_rfc3339(
-        group: BackupGroup,
-        backup_time_string: String,
-    ) -> Result<Self, Error> {
-        let backup_time = proxmox_time::parse_rfc3339(&backup_time_string)?;
-        Ok(Self {
-            store: group.store,
-            ns: group.ns,
-            dir: (group.group, backup_time).into(),
-            backup_time_string,
-        })
-    }
-
-    #[inline]
-    pub fn backup_ns(&self) -> &BackupNamespace {
-        &self.ns
-    }
-
-    #[inline]
-    pub fn backup_type(&self) -> BackupType {
-        self.dir.group.ty
-    }
-
-    #[inline]
-    pub fn backup_id(&self) -> &str {
-        &self.dir.group.id
-    }
-
-    #[inline]
-    pub fn backup_time(&self) -> i64 {
-        self.dir.time
-    }
-
-    pub fn backup_time_string(&self) -> &str {
-        &self.backup_time_string
-    }
-
-    pub fn dir(&self) -> &pbs_api_types::BackupDir {
-        &self.dir
-    }
-
-    pub fn group(&self) -> &pbs_api_types::BackupGroup {
-        &self.dir.group
-    }
-
-    pub fn relative_path(&self) -> PathBuf {
-        let mut path = self.ns.path();
-        path.push(self.dir.group.ty.as_str());
-        path.push(&self.dir.group.id);
-        path.push(&self.backup_time_string);
-        path
-    }
-
-    /// Returns the absolute path for backup_dir, using the cached formatted time string.
-    pub fn full_path(&self) -> PathBuf {
-        let mut path = self.store.base_path();
-        path.push(self.relative_path());
-        path
-    }
-
-    pub fn protected_file(&self) -> PathBuf {
-        let mut path = self.full_path();
-        path.push(".protected");
-        path
-    }
-
-    pub fn is_protected(&self) -> bool {
-        let path = self.protected_file();
-        path.exists()
-    }
-
-    pub fn backup_time_to_string(backup_time: i64) -> Result<String, Error> {
-        // fixme: can this fail? (avoid unwrap)
-        proxmox_time::epoch_to_rfc3339_utc(backup_time)
+impl<T: CanRead> BackupDir<T> {
+    /// Returns the backup owner.
+    ///
+    /// The backup owner is the entity who first created the backup group.
+    pub fn get_owner(&self) -> Result<Authid, Error> {
+        self.store.get_owner(&self.ns, self.as_ref())
     }
 
     /// load a `DataBlob` from this snapshot's backup dir.
@@ -483,22 +400,38 @@ impl BackupDir {
         .map_err(|err| format_err!("unable to load blob '{:?}' - {}", path, err))
     }
 
-    /// Returns the filename to lock a manifest
-    ///
-    /// Also creates the basedir. The lockfile is located in
-    /// `${DATASTORE_LOCKS_DIR}/${datastore name}/${lock_file_path_helper(rpath)}.index.json.lck`
-    /// where rpath is the relative path of the snapshot.
-    fn manifest_lock_path(&self) -> PathBuf {
-        let path = Path::new(DATASTORE_LOCKS_DIR).join(self.store.name());
+    /// Acquires a shared lock on a snapshot.
+    pub fn lock_shared(&self) -> Result<BackupLockGuard, Error> {
+        if *OLD_LOCKING {
+            lock_dir_noblock_shared(
+                &self.full_path(),
+                "snapshot",
+                "backup is running or snapshot is in use, could not acquire shared lock",
+            )
+            .map(BackupLockGuard::from)
+        } else {
+            lock_helper(self.store.name(), &self.lock_path(), |p| {
+                open_backup_lockfile(p, Some(Duration::from_secs(0)), false)
+                    .with_context(|| format!("unable to acquire shared snapshot lock {p:?}"))
+            })
+        }
+    }
 
-        let rpath = Path::new(self.dir.group.ty.as_str())
-            .join(&self.dir.group.id)
-            .join(&self.backup_time_string)
-            .join(MANIFEST_LOCK_NAME);
+    /// Load the manifest without a lock. Must not be written back.
+    pub fn load_manifest(&self) -> Result<(BackupManifest, u64), Error> {
+        let blob = self.load_blob(MANIFEST_BLOB_NAME.as_ref())?;
+        let raw_size = blob.raw_size();
+        let manifest = BackupManifest::try_from(blob)?;
+        Ok((manifest, raw_size))
+    }
 
-        path.join(lock_file_path_helper(&self.ns, rpath))
+    /// Load the verify state from the manifest.
+    pub fn verify_state(&self) -> Result<Option<VerifyState>, anyhow::Error> {
+        Ok(self.load_manifest()?.0.verify_state()?.map(|svs| svs.state))
     }
+}
 
+impl<T: CanWrite> BackupDir<T> {
     /// Locks the manifest of a snapshot, for example, to update or delete it.
     pub(crate) fn lock_manifest(&self) -> Result<BackupLockGuard, Error> {
         let path = if *OLD_LOCKING {
@@ -523,21 +456,6 @@ impl BackupDir {
         })
     }
 
-    /// Returns a file name for locking a snapshot.
-    ///
-    /// The lock file will be located in:
-    /// `${DATASTORE_LOCKS_DIR}/${datastore name}/${lock_file_path_helper(rpath)}`
-    /// where `rpath` is the relative path of the snapshot.
-    fn lock_path(&self) -> PathBuf {
-        let path = Path::new(DATASTORE_LOCKS_DIR).join(self.store.name());
-
-        let rpath = Path::new(self.dir.group.ty.as_str())
-            .join(&self.dir.group.id)
-            .join(&self.backup_time_string);
-
-        path.join(lock_file_path_helper(&self.ns, rpath))
-    }
-
     /// Locks a snapshot exclusively.
     pub fn lock(&self) -> Result<BackupLockGuard, Error> {
         if *OLD_LOCKING {
@@ -555,23 +473,6 @@ impl BackupDir {
         }
     }
 
-    /// Acquires a shared lock on a snapshot.
-    pub fn lock_shared(&self) -> Result<BackupLockGuard, Error> {
-        if *OLD_LOCKING {
-            lock_dir_noblock_shared(
-                &self.full_path(),
-                "snapshot",
-                "backup is running or snapshot is in use, could not acquire shared lock",
-            )
-            .map(BackupLockGuard::from)
-        } else {
-            lock_helper(self.store.name(), &self.lock_path(), |p| {
-                open_backup_lockfile(p, Some(Duration::from_secs(0)), false)
-                    .with_context(|| format!("unable to acquire shared snapshot lock {p:?}"))
-            })
-        }
-    }
-
     /// Destroy the whole snapshot, bails if it's protected
     ///
     /// Setting `force` to true skips locking and thus ignores if the backup is currently in use.
@@ -624,31 +525,6 @@ impl BackupDir {
         Ok(())
     }
 
-    /// Get the datastore.
-    pub fn datastore(&self) -> &Arc<DataStore> {
-        &self.store
-    }
-
-    /// Returns the backup owner.
-    ///
-    /// The backup owner is the entity who first created the backup group.
-    pub fn get_owner(&self) -> Result<Authid, Error> {
-        self.store.get_owner(&self.ns, self.as_ref())
-    }
-
-    /// Lock the snapshot and open a reader.
-    pub fn locked_reader(&self) -> Result<crate::SnapshotReader, Error> {
-        crate::SnapshotReader::new_do(self.clone())
-    }
-
-    /// Load the manifest without a lock. Must not be written back.
-    pub fn load_manifest(&self) -> Result<(BackupManifest, u64), Error> {
-        let blob = self.load_blob(MANIFEST_BLOB_NAME.as_ref())?;
-        let raw_size = blob.raw_size();
-        let manifest = BackupManifest::try_from(blob)?;
-        Ok((manifest, raw_size))
-    }
-
     /// Update the manifest of the specified snapshot. Never write a manifest directly,
     /// only use this method - anything else may break locking guarantees.
     pub fn update_manifest(
@@ -706,68 +582,203 @@ impl BackupDir {
 
         Ok(())
     }
+}
+
+impl<T> BackupDir<T> {
+    /// Temporarily used for tests.
+    #[doc(hidden)]
+    pub fn new_test(dir: pbs_api_types::BackupDir) -> Self {
+        Self {
+            store: DataStore::new_test(),
+            backup_time_string: Self::backup_time_to_string(dir.time).unwrap(),
+            ns: BackupNamespace::root(),
+            dir,
+        }
+    }
 
-    /// Load the verify state from the manifest.
-    pub fn verify_state(&self) -> Result<Option<VerifyState>, anyhow::Error> {
-        Ok(self.load_manifest()?.0.verify_state()?.map(|svs| svs.state))
+    pub(crate) fn with_group(group: BackupGroup<T>, backup_time: i64) -> Result<Self, Error> {
+        let backup_time_string = Self::backup_time_to_string(backup_time)?;
+        Ok(Self {
+            store: group.store,
+            ns: group.ns,
+            dir: (group.group, backup_time).into(),
+            backup_time_string,
+        })
+    }
+
+    pub(crate) fn with_rfc3339(
+        group: BackupGroup<T>,
+        backup_time_string: String,
+    ) -> Result<Self, Error> {
+        let backup_time = proxmox_time::parse_rfc3339(&backup_time_string)?;
+        Ok(Self {
+            store: group.store,
+            ns: group.ns,
+            dir: (group.group, backup_time).into(),
+            backup_time_string,
+        })
+    }
+
+    #[inline]
+    pub fn backup_ns(&self) -> &BackupNamespace {
+        &self.ns
+    }
+
+    #[inline]
+    pub fn backup_type(&self) -> BackupType {
+        self.dir.group.ty
+    }
+
+    #[inline]
+    pub fn backup_id(&self) -> &str {
+        &self.dir.group.id
+    }
+
+    #[inline]
+    pub fn backup_time(&self) -> i64 {
+        self.dir.time
+    }
+
+    pub fn backup_time_string(&self) -> &str {
+        &self.backup_time_string
+    }
+
+    pub fn dir(&self) -> &pbs_api_types::BackupDir {
+        &self.dir
+    }
+
+    pub fn group(&self) -> &pbs_api_types::BackupGroup {
+        &self.dir.group
+    }
+
+    pub fn relative_path(&self) -> PathBuf {
+        let mut path = self.ns.path();
+        path.push(self.dir.group.ty.as_str());
+        path.push(&self.dir.group.id);
+        path.push(&self.backup_time_string);
+        path
+    }
+
+    /// Returns the absolute path for backup_dir, using the cached formatted time string.
+    pub fn full_path(&self) -> PathBuf {
+        let mut path = self.store.base_path();
+        path.push(self.relative_path());
+        path
+    }
+
+    pub fn protected_file(&self) -> PathBuf {
+        let mut path = self.full_path();
+        path.push(".protected");
+        path
+    }
+
+    pub fn is_protected(&self) -> bool {
+        let path = self.protected_file();
+        path.exists()
+    }
+
+    pub fn backup_time_to_string(backup_time: i64) -> Result<String, Error> {
+        // fixme: can this fail? (avoid unwrap)
+        proxmox_time::epoch_to_rfc3339_utc(backup_time)
+    }
+
+    /// Returns the filename to lock a manifest
+    ///
+    /// Also creates the basedir. The lockfile is located in
+    /// `${DATASTORE_LOCKS_DIR}/${datastore name}/${lock_file_path_helper(rpath)}.index.json.lck`
+    /// where rpath is the relative path of the snapshot.
+    fn manifest_lock_path(&self) -> PathBuf {
+        let path = Path::new(DATASTORE_LOCKS_DIR).join(self.store.name());
+
+        let rpath = Path::new(self.dir.group.ty.as_str())
+            .join(&self.dir.group.id)
+            .join(&self.backup_time_string)
+            .join(MANIFEST_LOCK_NAME);
+
+        path.join(lock_file_path_helper(&self.ns, rpath))
+    }
+
+    /// Returns a file name for locking a snapshot.
+    ///
+    /// The lock file will be located in:
+    /// `${DATASTORE_LOCKS_DIR}/${datastore name}/${lock_file_path_helper(rpath)}`
+    /// where `rpath` is the relative path of the snapshot.
+    fn lock_path(&self) -> PathBuf {
+        let path = Path::new(DATASTORE_LOCKS_DIR).join(self.store.name());
+
+        let rpath = Path::new(self.dir.group.ty.as_str())
+            .join(&self.dir.group.id)
+            .join(&self.backup_time_string);
+
+        path.join(lock_file_path_helper(&self.ns, rpath))
+    }
+
+    /// Get the datastore.
+    pub fn datastore(&self) -> &Arc<DataStore<T>> {
+        &self.store
+    }
+
+    /// Lock the snapshot and open a reader.
+    pub fn locked_reader(&self) -> Result<crate::SnapshotReader, Error> {
+        crate::SnapshotReader::new_do(self.clone())
     }
 }
 
-impl AsRef<pbs_api_types::BackupNamespace> for BackupDir {
+impl<T> AsRef<pbs_api_types::BackupNamespace> for BackupDir<T> {
     fn as_ref(&self) -> &pbs_api_types::BackupNamespace {
         &self.ns
     }
 }
 
-impl AsRef<pbs_api_types::BackupDir> for BackupDir {
+impl<T> AsRef<pbs_api_types::BackupDir> for BackupDir<T> {
     fn as_ref(&self) -> &pbs_api_types::BackupDir {
         &self.dir
     }
 }
 
-impl AsRef<pbs_api_types::BackupGroup> for BackupDir {
+impl<T> AsRef<pbs_api_types::BackupGroup> for BackupDir<T> {
     fn as_ref(&self) -> &pbs_api_types::BackupGroup {
         &self.dir.group
     }
 }
 
-impl From<&BackupDir> for pbs_api_types::BackupGroup {
-    fn from(dir: &BackupDir) -> pbs_api_types::BackupGroup {
+impl<T> From<&BackupDir<T>> for pbs_api_types::BackupGroup {
+    fn from(dir: &BackupDir<T>) -> pbs_api_types::BackupGroup {
         dir.dir.group.clone()
     }
 }
 
-impl From<BackupDir> for pbs_api_types::BackupGroup {
-    fn from(dir: BackupDir) -> pbs_api_types::BackupGroup {
+impl<T> From<BackupDir<T>> for pbs_api_types::BackupGroup {
+    fn from(dir: BackupDir<T>) -> pbs_api_types::BackupGroup {
         dir.dir.group
     }
 }
 
-impl From<&BackupDir> for pbs_api_types::BackupDir {
-    fn from(dir: &BackupDir) -> pbs_api_types::BackupDir {
+impl<T> From<&BackupDir<T>> for pbs_api_types::BackupDir {
+    fn from(dir: &BackupDir<T>) -> pbs_api_types::BackupDir {
         dir.dir.clone()
     }
 }
 
-impl From<BackupDir> for pbs_api_types::BackupDir {
-    fn from(dir: BackupDir) -> pbs_api_types::BackupDir {
+impl<T> From<BackupDir<T>> for pbs_api_types::BackupDir {
+    fn from(dir: BackupDir<T>) -> pbs_api_types::BackupDir {
         dir.dir
     }
 }
 
 /// Detailed Backup Information, lists files inside a BackupDir
 #[derive(Clone, Debug)]
-pub struct BackupInfo {
+pub struct BackupInfo<T> {
     /// the backup directory
-    pub backup_dir: BackupDir,
+    pub backup_dir: BackupDir<T>,
     /// List of data files
     pub files: Vec<String>,
     /// Protection Status
     pub protected: bool,
 }
 
-impl BackupInfo {
-    pub fn new(backup_dir: BackupDir) -> Result<BackupInfo, Error> {
+impl<T: CanRead> BackupInfo<T> {
+    pub fn new(backup_dir: BackupDir<T>) -> Result<BackupInfo<T>, Error> {
         let path = backup_dir.full_path();
 
         let files = list_backup_files(libc::AT_FDCWD, &path)?;
@@ -779,8 +790,10 @@ impl BackupInfo {
             protected,
         })
     }
+}
 
-    pub fn sort_list(list: &mut [BackupInfo], ascendending: bool) {
+impl<T> BackupInfo<T> {
+    pub fn sort_list(list: &mut [BackupInfo<T>], ascendending: bool) {
         if ascendending {
             // oldest first
             list.sort_unstable_by(|a, b| a.backup_dir.dir.time.cmp(&b.backup_dir.dir.time));
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 66a2e209..9356750b 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -303,7 +303,7 @@ impl<T: CanRead + 'static> DataStore<T> {
         self: &Arc<DataStore<T>>,
         ns: BackupNamespace,
         ty: BackupType,
-    ) -> Result<impl Iterator<Item = BackupGroup> + 'static, Error> {
+    ) -> Result<impl Iterator<Item = BackupGroup<T>> + 'static, Error> {
         Ok(self.iter_backup_type(ns, ty)?.ok())
     }
 
@@ -314,7 +314,7 @@ impl<T: CanRead + 'static> DataStore<T> {
     pub fn iter_backup_groups_ok(
         self: &Arc<DataStore<T>>,
         ns: BackupNamespace,
-    ) -> Result<impl Iterator<Item = BackupGroup> + 'static, Error> {
+    ) -> Result<impl Iterator<Item = BackupGroup<T>> + 'static, Error> {
         Ok(self.iter_backup_groups(ns)?.ok())
     }
 }
@@ -644,7 +644,7 @@ impl<T: CanRead> DataStore<T> {
     pub fn list_backup_groups(
         self: &Arc<DataStore<T>>,
         ns: BackupNamespace,
-    ) -> Result<Vec<BackupGroup>, Error> {
+    ) -> Result<Vec<BackupGroup<T>>, Error> {
         ListGroups::new(Arc::clone(self), ns)?.collect()
     }
 
@@ -837,7 +837,7 @@ impl<T: CanRead> DataStore<T> {
         ty: BackupType,
         id: D,
         time: i64,
-    ) -> Result<BackupDir, Error>
+    ) -> Result<BackupDir<T>, Error>
     where
         D: Into<String>,
     {
@@ -847,10 +847,10 @@ impl<T: CanRead> DataStore<T> {
     /// Open a snapshot (backup directory) from this datastore with a cached rfc3339 time string.
     pub fn backup_dir_with_rfc3339<D: Into<String>>(
         self: &Arc<Self>,
-        group: BackupGroup,
+        group: BackupGroup<T>,
         time_string: D,
-    ) -> Result<BackupDir, Error> {
-        BackupDir::with_rfc3339(group, time_string.into())
+    ) -> Result<BackupDir<T>, Error> {
+        BackupDir::<T>::with_rfc3339(group, time_string.into())
     }
 
     /// Open a backup group from this datastore.
@@ -859,7 +859,7 @@ impl<T: CanRead> DataStore<T> {
         ns: BackupNamespace,
         ty: BackupType,
         id: D,
-    ) -> BackupGroup
+    ) -> BackupGroup<T>
     where
         D: Into<String>,
     {
@@ -889,7 +889,7 @@ impl<T: CanRead> DataStore<T> {
         self: &Arc<Self>,
         ns: BackupNamespace,
         dir: pbs_api_types::BackupDir,
-    ) -> Result<BackupDir, Error> {
+    ) -> Result<BackupDir<T>, Error> {
         BackupDir::with_group(self.backup_group(ns, dir.group), dir.time)
     }
 }
@@ -1258,7 +1258,7 @@ impl<T: CanWrite> DataStore<T> {
                         _ => bail!("exhausted retries and unexpected counter overrun"),
                     };
 
-                    let mut snapshots = match group.list_backups() {
+                    let mut snapshots: Vec<BackupInfo<T>> = match group.list_backups() {
                         Ok(snapshots) => snapshots,
                         Err(err) => {
                             if group.exists() {
@@ -1526,7 +1526,11 @@ impl<T: CanWrite> DataStore<T> {
     }
 
     /// Updates the protection status of the specified snapshot.
-    pub fn update_protection(&self, backup_dir: &BackupDir, protection: bool) -> Result<(), Error> {
+    pub fn update_protection(
+        &self,
+        backup_dir: &BackupDir<T>,
+        protection: bool,
+    ) -> Result<(), Error> {
         let full_path = backup_dir.full_path();
 
         if !full_path.exists() {
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:42 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:42 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 09/12] examples/tests: add
 missing generics
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-10-h.laimer@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 pbs-datastore/examples/ls-snapshots.rs | 4 ++--
 tests/prune.rs                         | 8 ++++----
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/pbs-datastore/examples/ls-snapshots.rs b/pbs-datastore/examples/ls-snapshots.rs
index 2eeea489..cf860a05 100644
--- a/pbs-datastore/examples/ls-snapshots.rs
+++ b/pbs-datastore/examples/ls-snapshots.rs
@@ -2,7 +2,7 @@ use std::path::PathBuf;
 
 use anyhow::{bail, Error};
 
-use pbs_datastore::DataStore;
+use pbs_datastore::{chunk_store::Read as R, DataStore};
 
 fn run() -> Result<(), Error> {
     let base: PathBuf = match std::env::args().nth(1) {
@@ -18,7 +18,7 @@ fn run() -> Result<(), Error> {
         None => None,
     };
 
-    let store = unsafe { DataStore::open_path("", base, None)? };
+    let store = unsafe { DataStore::<R>::open_path("", base, None)? };
 
     for ns in store.recursive_iter_backup_ns_ok(Default::default(), max_depth)? {
         println!("found namespace store:/{}", ns);
diff --git a/tests/prune.rs b/tests/prune.rs
index b11449ca..02de1078 100644
--- a/tests/prune.rs
+++ b/tests/prune.rs
@@ -4,10 +4,10 @@ use anyhow::Error;
 
 use pbs_api_types::{PruneJobOptions, MANIFEST_BLOB_NAME};
 use pbs_datastore::prune::compute_prune_info;
-use pbs_datastore::{BackupDir, BackupInfo};
+use pbs_datastore::{chunk_store::Read as R, BackupDir, BackupInfo};
 
 fn get_prune_list(
-    list: Vec<BackupInfo>,
+    list: Vec<BackupInfo<R>>,
     return_kept: bool,
     options: &PruneJobOptions,
 ) -> Vec<PathBuf> {
@@ -27,7 +27,7 @@ fn get_prune_list(
         .collect()
 }
 
-fn create_info(snapshot: &str, partial: bool) -> BackupInfo {
+fn create_info(snapshot: &str, partial: bool) -> BackupInfo<R> {
     let backup_dir = BackupDir::new_test(snapshot.parse().unwrap());
 
     let mut files = Vec::new();
@@ -43,7 +43,7 @@ fn create_info(snapshot: &str, partial: bool) -> BackupInfo {
     }
 }
 
-fn create_info_protected(snapshot: &str, partial: bool) -> BackupInfo {
+fn create_info_protected(snapshot: &str, partial: bool) -> BackupInfo<R> {
     let mut info = create_info(snapshot, partial);
     info.protected = true;
     info
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:43 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:43 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 10/12] api: admin: pull
 datastore loading out of check_privs helper
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-11-h.laimer@proxmox.com>

We have to use different lookup functions depending on what we
plan to do with the reference, deciding what function to use
based on the passed operation type was problematic as
the different functions return differently typed datastore
references which is a problem for the return type of the helper
function.

Like this the loading and permission checking is separated, which
can definitely make debugging easier and may itself be a reason for
separating it in the first place, though that was not why it was done
here.

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 src/api2/admin/datastore.rs | 116 ++++++++++++++++++------------------
 1 file changed, 59 insertions(+), 57 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index e3f93cdd..218d7e73 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -41,7 +41,7 @@ use pbs_api_types::{
     BackupContent, BackupGroupDeleteStats, BackupNamespace, BackupType, Counts, CryptMode,
     DataStoreConfig, DataStoreListItem, DataStoreMountStatus, DataStoreStatus,
     GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
-    MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
+    MaintenanceType, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
     BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
     BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
     IGNORE_VERIFIED_BACKUPS_SCHEMA, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
@@ -92,27 +92,29 @@ fn get_group_note_path<T>(
 
 // helper to unify common sequence of checks:
 // 1. check privs on NS (full or limited access)
-// 2. load datastore
-// 3. if needed (only limited access), check owner of group
-fn check_privs_and_load_store(
-    store: &str,
+// 2. if needed (only limited access), check owner of group
+fn check_privs<T: CanRead>(
+    store: &Arc<DataStore<T>>,
     ns: &BackupNamespace,
     auth_id: &Authid,
     full_access_privs: u64,
     partial_access_privs: u64,
-    operation: Option<Operation>,
     backup_group: &pbs_api_types::BackupGroup,
-) -> Result<Arc<DataStore>, Error> {
-    let limited = check_ns_privs_full(store, ns, auth_id, full_access_privs, partial_access_privs)?;
-
-    let datastore = DataStore::lookup_datastore(store, operation)?;
+) -> Result<(), Error> {
+    let limited = check_ns_privs_full(
+        store.name(),
+        ns,
+        auth_id,
+        full_access_privs,
+        partial_access_privs,
+    )?;
 
     if limited {
-        let owner = datastore.get_owner(ns, backup_group)?;
+        let owner = store.get_owner(ns, backup_group)?;
         check_backup_owner(&owner, auth_id)?;
     }
 
-    Ok(datastore)
+    Ok(())
 }
 
 fn read_backup_index<T: CanRead>(
@@ -303,13 +305,13 @@ pub async fn delete_group(
     tokio::task::spawn_blocking(move || {
         let ns = ns.unwrap_or_default();
 
-        let datastore = check_privs_and_load_store(
-            &store,
+        let datastore = DataStore::lookup_datastore_write(&store)?;
+        check_privs(
+            &datastore,
             &ns,
             &auth_id,
             PRIV_DATASTORE_MODIFY,
             PRIV_DATASTORE_PRUNE,
-            Some(Operation::Write),
             &group,
         )?;
 
@@ -370,13 +372,13 @@ pub async fn list_snapshot_files(
     tokio::task::spawn_blocking(move || {
         let ns = ns.unwrap_or_default();
 
-        let datastore = check_privs_and_load_store(
-            &store,
+        let datastore = DataStore::lookup_datastore_read(&store)?;
+        check_privs(
+            &datastore,
             &ns,
             &auth_id,
             PRIV_DATASTORE_AUDIT | PRIV_DATASTORE_READ,
             PRIV_DATASTORE_BACKUP,
-            Some(Operation::Read),
             &backup_dir.group,
         )?;
 
@@ -424,13 +426,13 @@ pub async fn delete_snapshot(
     tokio::task::spawn_blocking(move || {
         let ns = ns.unwrap_or_default();
 
-        let datastore = check_privs_and_load_store(
-            &store,
+        let datastore = DataStore::lookup_datastore_write(&store)?;
+        check_privs(
+            &datastore,
             &ns,
             &auth_id,
             PRIV_DATASTORE_MODIFY,
             PRIV_DATASTORE_PRUNE,
-            Some(Operation::Write),
             &backup_dir.group,
         )?;
 
@@ -1001,13 +1003,13 @@ pub fn prune(
 ) -> Result<Value, Error> {
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
     let ns = ns.unwrap_or_default();
-    let datastore = check_privs_and_load_store(
-        &store,
+    let datastore = DataStore::lookup_datastore_write(&store)?;
+    check_privs(
+        &datastore,
         &ns,
         &auth_id,
         PRIV_DATASTORE_MODIFY,
         PRIV_DATASTORE_PRUNE,
-        Some(Operation::Write),
         &group,
     )?;
 
@@ -1405,13 +1407,13 @@ pub fn download_file(
         let backup_ns = optional_ns_param(&param)?;
 
         let backup_dir: pbs_api_types::BackupDir = Deserialize::deserialize(&param)?;
-        let datastore = check_privs_and_load_store(
-            store,
+        let datastore = DataStore::lookup_datastore_read(store)?;
+        check_privs(
+            &datastore,
             &backup_ns,
             &auth_id,
             PRIV_DATASTORE_READ,
             PRIV_DATASTORE_BACKUP,
-            Some(Operation::Read),
             &backup_dir.group,
         )?;
 
@@ -1490,13 +1492,13 @@ pub fn download_file_decoded(
         let backup_ns = optional_ns_param(&param)?;
 
         let backup_dir_api: pbs_api_types::BackupDir = Deserialize::deserialize(&param)?;
-        let datastore = check_privs_and_load_store(
-            store,
+        let datastore = DataStore::lookup_datastore_read(store)?;
+        check_privs(
+            &datastore,
             &backup_ns,
             &auth_id,
             PRIV_DATASTORE_READ,
             PRIV_DATASTORE_BACKUP,
-            Some(Operation::Read),
             &backup_dir_api.group,
         )?;
 
@@ -1617,13 +1619,13 @@ pub fn upload_backup_log(
 
         let backup_dir_api: pbs_api_types::BackupDir = Deserialize::deserialize(&param)?;
 
-        let datastore = check_privs_and_load_store(
-            store,
+        let datastore = DataStore::lookup_datastore_write(store)?;
+        check_privs(
+            &datastore,
             &backup_ns,
             &auth_id,
             0,
             PRIV_DATASTORE_BACKUP,
-            Some(Operation::Write),
             &backup_dir_api.group,
         )?;
         let backup_dir = datastore.backup_dir(backup_ns.clone(), backup_dir_api.clone())?;
@@ -1713,13 +1715,13 @@ pub async fn catalog(
 
     let ns = ns.unwrap_or_default();
 
-    let datastore = check_privs_and_load_store(
-        &store,
+    let datastore = DataStore::lookup_datastore_read(&store)?;
+    check_privs(
+        &datastore,
         &ns,
         &auth_id,
         PRIV_DATASTORE_READ,
         PRIV_DATASTORE_BACKUP,
-        Some(Operation::Read),
         &backup_dir.group,
     )?;
 
@@ -1833,13 +1835,13 @@ pub fn pxar_file_download(
         let ns = optional_ns_param(&param)?;
 
         let backup_dir: pbs_api_types::BackupDir = Deserialize::deserialize(&param)?;
-        let datastore = check_privs_and_load_store(
-            store,
+        let datastore = DataStore::lookup_datastore_read(store)?;
+        check_privs(
+            &datastore,
             &ns,
             &auth_id,
             PRIV_DATASTORE_READ,
             PRIV_DATASTORE_BACKUP,
-            Some(Operation::Read),
             &backup_dir.group,
         )?;
 
@@ -2044,13 +2046,13 @@ pub fn get_group_notes(
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
     let ns = ns.unwrap_or_default();
 
-    let datastore = check_privs_and_load_store(
-        &store,
+    let datastore = DataStore::lookup_datastore_read(&store)?;
+    check_privs(
+        &datastore,
         &ns,
         &auth_id,
         PRIV_DATASTORE_AUDIT,
         PRIV_DATASTORE_BACKUP,
-        Some(Operation::Read),
         &backup_group,
     )?;
 
@@ -2092,13 +2094,13 @@ pub fn set_group_notes(
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
     let ns = ns.unwrap_or_default();
 
-    let datastore = check_privs_and_load_store(
-        &store,
+    let datastore = DataStore::lookup_datastore_write(&store)?;
+    check_privs(
+        &datastore,
         &ns,
         &auth_id,
         PRIV_DATASTORE_MODIFY,
         PRIV_DATASTORE_BACKUP,
-        Some(Operation::Write),
         &backup_group,
     )?;
 
@@ -2138,13 +2140,13 @@ pub fn get_notes(
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
     let ns = ns.unwrap_or_default();
 
-    let datastore = check_privs_and_load_store(
-        &store,
+    let datastore = DataStore::lookup_datastore_read(&store)?;
+    check_privs(
+        &datastore,
         &ns,
         &auth_id,
         PRIV_DATASTORE_AUDIT,
         PRIV_DATASTORE_BACKUP,
-        Some(Operation::Read),
         &backup_dir.group,
     )?;
 
@@ -2191,13 +2193,13 @@ pub fn set_notes(
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
     let ns = ns.unwrap_or_default();
 
-    let datastore = check_privs_and_load_store(
-        &store,
+    let datastore = DataStore::lookup_datastore_write(&store)?;
+    check_privs(
+        &datastore,
         &ns,
         &auth_id,
         PRIV_DATASTORE_MODIFY,
         PRIV_DATASTORE_BACKUP,
-        Some(Operation::Write),
         &backup_dir.group,
     )?;
 
@@ -2241,13 +2243,13 @@ pub fn get_protection(
 ) -> Result<bool, Error> {
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
     let ns = ns.unwrap_or_default();
-    let datastore = check_privs_and_load_store(
-        &store,
+    let datastore = DataStore::lookup_datastore_read(&store)?;
+    check_privs(
+        &datastore,
         &ns,
         &auth_id,
         PRIV_DATASTORE_AUDIT,
         PRIV_DATASTORE_BACKUP,
-        Some(Operation::Read),
         &backup_dir.group,
     )?;
 
@@ -2291,13 +2293,13 @@ pub async fn set_protection(
 
     tokio::task::spawn_blocking(move || {
         let ns = ns.unwrap_or_default();
-        let datastore = check_privs_and_load_store(
-            &store,
+        let datastore = DataStore::lookup_datastore_write(&store)?;
+        check_privs(
+            &datastore,
             &ns,
             &auth_id,
             PRIV_DATASTORE_MODIFY,
             PRIV_DATASTORE_BACKUP,
-            Some(Operation::Write),
             &backup_dir.group,
         )?;
 
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:45 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:45 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 12/12] api/server: replace
 datastore_lookup with new, state-typed datastore returning functions
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-13-h.laimer@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 pbs-datastore/src/snapshot_reader.rs |  6 ++----
 src/api2/admin/datastore.rs          | 18 +++++++++---------
 src/api2/admin/namespace.rs          | 10 +++++-----
 src/api2/backup/mod.rs               |  8 ++++----
 src/api2/reader/mod.rs               |  8 ++++----
 src/api2/status/mod.rs               |  8 ++++----
 src/api2/tape/backup.rs              | 10 +++++-----
 src/api2/tape/restore.rs             | 12 ++++++------
 src/bin/proxmox-backup-proxy.rs      |  4 ++--
 src/server/prune_job.rs              |  4 ++--
 src/server/pull.rs                   |  9 ++++-----
 src/server/push.rs                   |  4 ++--
 src/server/verify_job.rs             |  4 ++--
 13 files changed, 51 insertions(+), 54 deletions(-)

diff --git a/pbs-datastore/src/snapshot_reader.rs b/pbs-datastore/src/snapshot_reader.rs
index d604507d..6ece122f 100644
--- a/pbs-datastore/src/snapshot_reader.rs
+++ b/pbs-datastore/src/snapshot_reader.rs
@@ -11,8 +11,7 @@ use nix::sys::stat::Mode;
 use pbs_config::BackupLockGuard;
 
 use pbs_api_types::{
-    print_store_and_ns, ArchiveType, BackupNamespace, Operation, CLIENT_LOG_BLOB_NAME,
-    MANIFEST_BLOB_NAME,
+    print_store_and_ns, ArchiveType, BackupNamespace, CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME,
 };
 
 use crate::backup_info::BackupDir;
@@ -163,9 +162,8 @@ impl<F: Fn(&[u8; 32]) -> bool, T: CanRead> Iterator for SnapshotChunkIterator<'_
                                 ),
                             };
 
-                        let datastore = DataStore::lookup_datastore(
+                        let datastore = DataStore::lookup_datastore_read(
                             self.snapshot_reader.datastore_name(),
-                            Some(Operation::Read),
                         )?;
                         let order =
                             datastore.get_chunks_in_order(&*index, &self.skip_fn, |_| Ok(()))?;
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 218d7e73..dfcb9123 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -203,7 +203,7 @@ pub fn list_groups(
         PRIV_DATASTORE_BACKUP,
     )?;
 
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
+    let datastore = DataStore::lookup_datastore_read(&store)?;
 
     datastore
         .iter_backup_groups(ns.clone())? // FIXME: Namespaces and recursion parameters!
@@ -508,7 +508,7 @@ unsafe fn list_snapshots_blocking(
         PRIV_DATASTORE_BACKUP,
     )?;
 
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
+    let datastore = DataStore::lookup_datastore_read(&store)?;
 
     // FIXME: filter also owner before collecting, for doing that nicely the owner should move into
     // backup group and provide an error free (Err -> None) accessor
@@ -720,7 +720,7 @@ pub async fn status(
         }
     };
 
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
+    let datastore = DataStore::lookup_datastore_read(&store)?;
 
     let (counts, gc_status) = if verbose {
         let filter_owner = if store_privs & PRIV_DATASTORE_AUDIT != 0 {
@@ -833,7 +833,7 @@ pub fn verify(
         PRIV_DATASTORE_BACKUP,
     )?;
 
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
+    let datastore = DataStore::lookup_datastore_write(&store)?;
     let ignore_verified = ignore_verified.unwrap_or(true);
 
     let worker_id;
@@ -1182,7 +1182,7 @@ pub fn prune_datastore(
         true,
     )?;
 
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
+    let datastore = DataStore::lookup_datastore_write(&store)?;
     let ns = prune_options.ns.clone().unwrap_or_default();
     let worker_id = format!("{}:{}", store, ns);
 
@@ -1220,7 +1220,7 @@ pub fn start_garbage_collection(
     _info: &ApiMethod,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Value, Error> {
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
+    let datastore = DataStore::lookup_datastore_write(&store)?;
     let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
 
     let job = Job::new("garbage_collection", &store)
@@ -1267,7 +1267,7 @@ pub fn garbage_collection_status(
         ..Default::default()
     };
 
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
+    let datastore = DataStore::lookup_datastore_read(&store)?;
     let status_in_memory = datastore.last_gc_status();
     let state_file = JobState::load("garbage_collection", &store)
         .map_err(|err| log::error!("could not open GC statefile for {store}: {err}"))
@@ -1973,7 +1973,7 @@ pub fn get_rrd_stats(
     cf: RrdMode,
     _param: Value,
 ) -> Result<Value, Error> {
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
+    let datastore = DataStore::lookup_datastore_read(&store)?;
     let disk_manager = crate::tools::disks::DiskManage::new();
 
     let mut rrd_fields = vec![
@@ -2353,7 +2353,7 @@ pub async fn set_backup_owner(
             PRIV_DATASTORE_BACKUP,
         )?;
 
-        let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
+        let datastore = DataStore::lookup_datastore_write(&store)?;
 
         let backup_group = datastore.backup_group(ns, backup_group);
         let owner = backup_group.get_owner()?;
diff --git a/src/api2/admin/namespace.rs b/src/api2/admin/namespace.rs
index 6cf88d89..44a31269 100644
--- a/src/api2/admin/namespace.rs
+++ b/src/api2/admin/namespace.rs
@@ -5,8 +5,8 @@ use proxmox_router::{http_bail, ApiMethod, Permission, Router, RpcEnvironment};
 use proxmox_schema::*;
 
 use pbs_api_types::{
-    Authid, BackupGroupDeleteStats, BackupNamespace, NamespaceListItem, Operation,
-    DATASTORE_SCHEMA, NS_MAX_DEPTH_SCHEMA, PROXMOX_SAFE_ID_FORMAT,
+    Authid, BackupGroupDeleteStats, BackupNamespace, NamespaceListItem, DATASTORE_SCHEMA,
+    NS_MAX_DEPTH_SCHEMA, PROXMOX_SAFE_ID_FORMAT,
 };
 
 use pbs_datastore::DataStore;
@@ -54,7 +54,7 @@ pub fn create_namespace(
 
     check_ns_modification_privs(&store, &ns, &auth_id)?;
 
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
+    let datastore = DataStore::lookup_datastore_write(&store)?;
 
     datastore.create_namespace(&parent, name)
 }
@@ -97,7 +97,7 @@ pub fn list_namespaces(
     // get result up-front to avoid cloning NS, it's relatively cheap anyway (no IO normally)
     let parent_access = check_ns_privs(&store, &parent, &auth_id, NS_PRIVS_OK);
 
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
+    let datastore = DataStore::lookup_datastore_read(&store)?;
 
     let iter = match datastore.recursive_iter_backup_ns_ok(parent, max_depth) {
         Ok(iter) => iter,
@@ -162,7 +162,7 @@ pub fn delete_namespace(
 
     check_ns_modification_privs(&store, &ns, &auth_id)?;
 
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
+    let datastore = DataStore::lookup_datastore_write(&store)?;
 
     let (removed_all, stats) = datastore.remove_namespace_recursive(&ns, delete_groups)?;
     if !removed_all {
diff --git a/src/api2/backup/mod.rs b/src/api2/backup/mod.rs
index 79354dbf..c160080e 100644
--- a/src/api2/backup/mod.rs
+++ b/src/api2/backup/mod.rs
@@ -19,9 +19,9 @@ use proxmox_schema::*;
 use proxmox_sortable_macro::sortable;
 
 use pbs_api_types::{
-    ArchiveType, Authid, BackupNamespace, BackupType, Operation, VerifyState,
-    BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
-    BACKUP_TYPE_SCHEMA, CHUNK_DIGEST_SCHEMA, DATASTORE_SCHEMA, PRIV_DATASTORE_BACKUP,
+    ArchiveType, Authid, BackupNamespace, BackupType, VerifyState, BACKUP_ARCHIVE_NAME_SCHEMA,
+    BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA,
+    CHUNK_DIGEST_SCHEMA, DATASTORE_SCHEMA, PRIV_DATASTORE_BACKUP,
 };
 use pbs_config::CachedUserInfo;
 use pbs_datastore::chunk_store::{Read as R, Write as W};
@@ -96,7 +96,7 @@ fn upgrade_to_backup_protocol(
             )
             .map_err(|err| http_err!(FORBIDDEN, "{err}"))?;
 
-        let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
+        let datastore = DataStore::lookup_datastore_write(&store)?;
 
         let protocols = parts
             .headers
diff --git a/src/api2/reader/mod.rs b/src/api2/reader/mod.rs
index 52f0953a..ec9fb751 100644
--- a/src/api2/reader/mod.rs
+++ b/src/api2/reader/mod.rs
@@ -18,9 +18,9 @@ use proxmox_schema::{BooleanSchema, ObjectSchema};
 use proxmox_sortable_macro::sortable;
 
 use pbs_api_types::{
-    ArchiveType, Authid, Operation, BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA,
-    BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, CHUNK_DIGEST_SCHEMA,
-    DATASTORE_SCHEMA, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_READ,
+    ArchiveType, Authid, BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA,
+    BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, CHUNK_DIGEST_SCHEMA, DATASTORE_SCHEMA,
+    PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_READ,
 };
 use pbs_config::CachedUserInfo;
 use pbs_datastore::chunk_store::Read as R;
@@ -92,7 +92,7 @@ fn upgrade_to_backup_reader_protocol(
             bail!("no permissions on /{}", acl_path.join("/"));
         }
 
-        let datastore = DataStore::lookup_datastore(&store, Some(Operation::Read))?;
+        let datastore = DataStore::lookup_datastore_read(&store)?;
 
         let backup_dir = pbs_api_types::BackupDir::deserialize(&param)?;
 
diff --git a/src/api2/status/mod.rs b/src/api2/status/mod.rs
index e066a99c..5a85cf80 100644
--- a/src/api2/status/mod.rs
+++ b/src/api2/status/mod.rs
@@ -10,8 +10,8 @@ use proxmox_schema::api;
 use proxmox_sortable_macro::sortable;
 
 use pbs_api_types::{
-    Authid, DataStoreConfig, DataStoreMountStatus, DataStoreStatusListItem, Operation,
-    PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP,
+    Authid, DataStoreConfig, DataStoreMountStatus, DataStoreStatusListItem, PRIV_DATASTORE_AUDIT,
+    PRIV_DATASTORE_BACKUP,
 };
 
 use pbs_config::CachedUserInfo;
@@ -69,7 +69,7 @@ pub async fn datastore_status(
         };
 
         if !allowed {
-            if let Ok(datastore) = DataStore::lookup_datastore(store, Some(Operation::Lookup)) {
+            if let Ok(datastore) = DataStore::lookup_datastore_read(store) {
                 if can_access_any_namespace(datastore, &auth_id, &user_info) {
                     list.push(DataStoreStatusListItem::empty(store, None, mount_status));
                 }
@@ -77,7 +77,7 @@ pub async fn datastore_status(
             continue;
         }
 
-        let datastore = match DataStore::lookup_datastore(store, Some(Operation::Read)) {
+        let datastore = match DataStore::lookup_datastore_read(store) {
             Ok(datastore) => datastore,
             Err(err) => {
                 list.push(DataStoreStatusListItem::empty(
diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
index 306d5936..d9ff7ba9 100644
--- a/src/api2/tape/backup.rs
+++ b/src/api2/tape/backup.rs
@@ -11,9 +11,9 @@ use proxmox_schema::api;
 use proxmox_worker_task::WorkerTaskContext;
 
 use pbs_api_types::{
-    print_ns_and_snapshot, print_store_and_ns, Authid, MediaPoolConfig, Operation,
-    TapeBackupJobConfig, TapeBackupJobSetup, TapeBackupJobStatus, JOB_ID_SCHEMA,
-    PRIV_DATASTORE_READ, PRIV_TAPE_AUDIT, PRIV_TAPE_WRITE, UPID_SCHEMA,
+    print_ns_and_snapshot, print_store_and_ns, Authid, MediaPoolConfig, TapeBackupJobConfig,
+    TapeBackupJobSetup, TapeBackupJobStatus, JOB_ID_SCHEMA, PRIV_DATASTORE_READ, PRIV_TAPE_AUDIT,
+    PRIV_TAPE_WRITE, UPID_SCHEMA,
 };
 
 use pbs_config::CachedUserInfo;
@@ -151,7 +151,7 @@ pub fn do_tape_backup_job(
 
     let worker_type = job.jobtype().to_string();
 
-    let datastore = DataStore::lookup_datastore(&setup.store, Some(Operation::Read))?;
+    let datastore = DataStore::lookup_datastore_read(&setup.store)?;
 
     let (config, _digest) = pbs_config::media_pool::config()?;
     let pool_config: MediaPoolConfig = config.lookup("pool", &setup.pool)?;
@@ -307,7 +307,7 @@ pub fn backup(
 
     check_backup_permission(&auth_id, &setup.store, &setup.pool, &setup.drive)?;
 
-    let datastore = DataStore::lookup_datastore(&setup.store, Some(Operation::Read))?;
+    let datastore = DataStore::lookup_datastore_read(&setup.store)?;
 
     let (config, _digest) = pbs_config::media_pool::config()?;
     let pool_config: MediaPoolConfig = config.lookup("pool", &setup.pool)?;
diff --git a/src/api2/tape/restore.rs b/src/api2/tape/restore.rs
index 8f089c20..1147623b 100644
--- a/src/api2/tape/restore.rs
+++ b/src/api2/tape/restore.rs
@@ -20,10 +20,10 @@ use proxmox_worker_task::WorkerTaskContext;
 
 use pbs_api_types::{
     parse_ns_and_snapshot, print_ns_and_snapshot, ArchiveType, Authid, BackupDir, BackupNamespace,
-    CryptMode, NotificationMode, Operation, TapeRestoreNamespace, Userid,
-    DATASTORE_MAP_ARRAY_SCHEMA, DATASTORE_MAP_LIST_SCHEMA, DRIVE_NAME_SCHEMA, MANIFEST_BLOB_NAME,
-    MAX_NAMESPACE_DEPTH, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_TAPE_READ,
-    TAPE_RESTORE_NAMESPACE_SCHEMA, TAPE_RESTORE_SNAPSHOT_SCHEMA, UPID_SCHEMA,
+    CryptMode, NotificationMode, TapeRestoreNamespace, Userid, DATASTORE_MAP_ARRAY_SCHEMA,
+    DATASTORE_MAP_LIST_SCHEMA, DRIVE_NAME_SCHEMA, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH,
+    PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_TAPE_READ, TAPE_RESTORE_NAMESPACE_SCHEMA,
+    TAPE_RESTORE_SNAPSHOT_SCHEMA, UPID_SCHEMA,
 };
 use pbs_client::pxar::tools::handle_root_with_optional_format_version_prelude;
 use pbs_config::CachedUserInfo;
@@ -145,10 +145,10 @@ impl TryFrom<String> for DataStoreMap<W> {
             if let Some(index) = store.find('=') {
                 let mut target = store.split_off(index);
                 target.remove(0); // remove '='
-                let datastore = DataStore::lookup_datastore(&target, Some(Operation::Write))?;
+                let datastore = DataStore::lookup_datastore_write(&target)?;
                 map.insert(store, datastore);
             } else if default.is_none() {
-                default = Some(DataStore::lookup_datastore(&store, Some(Operation::Write))?);
+                default = Some(DataStore::lookup_datastore_write(&store)?);
             } else {
                 bail!("multiple default stores given");
             }
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 643a2dbd..4f5d9681 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -40,7 +40,7 @@ use pbs_buildcfg::configdir;
 use proxmox_time::CalendarEvent;
 
 use pbs_api_types::{
-    Authid, DataStoreConfig, Operation, PruneJobConfig, SyncJobConfig, TapeBackupJobConfig,
+    Authid, DataStoreConfig, PruneJobConfig, SyncJobConfig, TapeBackupJobConfig,
     VerificationJobConfig,
 };
 
@@ -516,7 +516,7 @@ async fn schedule_datastore_garbage_collection() {
             Err(_) => continue, // could not get lock
         };
 
-        let datastore = match DataStore::lookup_datastore(&store, Some(Operation::Write)) {
+        let datastore = match DataStore::lookup_datastore_write(&store) {
             Ok(datastore) => datastore,
             Err(err) => {
                 log::warn!("skipping scheduled GC on {store}, could look it up - {err}");
diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
index 395aaee4..20cb9218 100644
--- a/src/server/prune_job.rs
+++ b/src/server/prune_job.rs
@@ -4,7 +4,7 @@ use anyhow::Error;
 use tracing::{info, warn};
 
 use pbs_api_types::{
-    print_store_and_ns, Authid, KeepOptions, Operation, PruneJobOptions, MAX_NAMESPACE_DEPTH,
+    print_store_and_ns, Authid, KeepOptions, PruneJobOptions, MAX_NAMESPACE_DEPTH,
     PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
 };
 use pbs_datastore::chunk_store::CanWrite;
@@ -128,7 +128,7 @@ pub fn do_prune_job(
     auth_id: &Authid,
     schedule: Option<String>,
 ) -> Result<String, Error> {
-    let datastore = DataStore::lookup_datastore(&store, Some(Operation::Write))?;
+    let datastore = DataStore::lookup_datastore_write(&store)?;
 
     let worker_type = job.jobtype().to_string();
     let auth_id = auth_id.clone();
diff --git a/src/server/pull.rs b/src/server/pull.rs
index 573aa805..d885a3a8 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -12,9 +12,8 @@ use tracing::info;
 
 use pbs_api_types::{
     print_store_and_ns, ArchiveType, Authid, BackupArchiveName, BackupDir, BackupGroup,
-    BackupNamespace, GroupFilter, Operation, RateLimitConfig, Remote, VerifyState,
-    CLIENT_LOG_BLOB_NAME, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, PRIV_DATASTORE_AUDIT,
-    PRIV_DATASTORE_BACKUP,
+    BackupNamespace, GroupFilter, RateLimitConfig, Remote, VerifyState, CLIENT_LOG_BLOB_NAME,
+    MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP,
 };
 use pbs_client::BackupRepository;
 use pbs_config::CachedUserInfo;
@@ -110,12 +109,12 @@ impl PullParameters {
             })
         } else {
             Arc::new(LocalSource {
-                store: DataStore::lookup_datastore(remote_store, Some(Operation::Read))?,
+                store: DataStore::lookup_datastore_read(remote_store)?,
                 ns: remote_ns,
             })
         };
         let target = PullTarget {
-            store: DataStore::lookup_datastore(store, Some(Operation::Write))?,
+            store: DataStore::lookup_datastore_write(store)?,
             ns,
         };
 
diff --git a/src/server/push.rs b/src/server/push.rs
index ff9d9358..532fc688 100644
--- a/src/server/push.rs
+++ b/src/server/push.rs
@@ -12,7 +12,7 @@ use tracing::{info, warn};
 use pbs_api_types::{
     print_store_and_ns, ApiVersion, ApiVersionInfo, ArchiveType, Authid, BackupArchiveName,
     BackupDir, BackupGroup, BackupGroupDeleteStats, BackupNamespace, GroupFilter, GroupListItem,
-    NamespaceListItem, Operation, RateLimitConfig, Remote, SnapshotListItem, CLIENT_LOG_BLOB_NAME,
+    NamespaceListItem, RateLimitConfig, Remote, SnapshotListItem, CLIENT_LOG_BLOB_NAME,
     MANIFEST_BLOB_NAME, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_READ, PRIV_REMOTE_DATASTORE_BACKUP,
     PRIV_REMOTE_DATASTORE_MODIFY, PRIV_REMOTE_DATASTORE_PRUNE,
 };
@@ -107,7 +107,7 @@ impl PushParameters {
         let remove_vanished = remove_vanished.unwrap_or(false);
         let encrypted_only = encrypted_only.unwrap_or(false);
         let verified_only = verified_only.unwrap_or(false);
-        let store = DataStore::lookup_datastore(store, Some(Operation::Read))?;
+        let store = DataStore::lookup_datastore_read(store)?;
 
         if !store.namespace_exists(&ns) {
             bail!(
diff --git a/src/server/verify_job.rs b/src/server/verify_job.rs
index a15a257d..bd5253ed 100644
--- a/src/server/verify_job.rs
+++ b/src/server/verify_job.rs
@@ -1,7 +1,7 @@
 use anyhow::{format_err, Error};
 use tracing::{error, info};
 
-use pbs_api_types::{Authid, Operation, VerificationJobConfig};
+use pbs_api_types::{Authid, VerificationJobConfig};
 use pbs_datastore::DataStore;
 use proxmox_rest_server::WorkerTask;
 
@@ -18,7 +18,7 @@ pub fn do_verification_job(
     schedule: Option<String>,
     to_stdout: bool,
 ) -> Result<String, Error> {
-    let datastore = DataStore::lookup_datastore(&verification_job.store, Some(Operation::Read))?;
+    let datastore = DataStore::lookup_datastore_write(&verification_job.store)?;
 
     let outdated_after = verification_job.outdated_after;
     let ignore_verified_snapshots = verification_job.ignore_verified.unwrap_or(true);
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:39 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:39 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 06/12] pbs-datastore: add
 generics and separate functions into impl blocks
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-7-h.laimer@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 pbs-datastore/src/backup_info.rs        | 18 ++---
 pbs-datastore/src/datastore.rs          | 12 ++--
 pbs-datastore/src/dynamic_index.rs      | 22 +++---
 pbs-datastore/src/fixed_index.rs        | 50 +++++++-------
 pbs-datastore/src/hierarchy.rs          | 92 ++++++++++++++-----------
 pbs-datastore/src/local_chunk_reader.rs | 13 ++--
 pbs-datastore/src/prune.rs              | 19 +++--
 pbs-datastore/src/snapshot_reader.rs    | 25 +++----
 8 files changed, 134 insertions(+), 117 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index f3ca283c..25d8fc08 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -160,6 +160,10 @@ impl<T: CanRead> BackupGroup<T> {
 
         Ok(last)
     }
+
+    pub fn iter_snapshots(&self) -> Result<crate::ListSnapshots<T>, Error> {
+        crate::ListSnapshots::new(self.clone())
+    }
 }
 
 impl<T: CanWrite> BackupGroup<T> {
@@ -293,10 +297,6 @@ impl<T: Clone> BackupGroup<T> {
         BackupDir::with_rfc3339(self.clone(), time_string.into())
     }
 
-    pub fn iter_snapshots(&self) -> Result<crate::ListSnapshots, Error> {
-        crate::ListSnapshots::new(self.clone())
-    }
-
     /// Returns a file name for locking a group.
     ///
     /// The lock file will be located in:
@@ -429,6 +429,11 @@ impl<T: CanRead> BackupDir<T> {
     pub fn verify_state(&self) -> Result<Option<VerifyState>, anyhow::Error> {
         Ok(self.load_manifest()?.0.verify_state()?.map(|svs| svs.state))
     }
+
+    /// Lock the snapshot and open a reader.
+    pub fn locked_reader(&self) -> Result<crate::SnapshotReader<T>, Error> {
+        crate::SnapshotReader::new_do(self.clone())
+    }
 }
 
 impl<T: CanWrite> BackupDir<T> {
@@ -717,11 +722,6 @@ impl<T> BackupDir<T> {
     pub fn datastore(&self) -> &Arc<DataStore<T>> {
         &self.store
     }
-
-    /// Lock the snapshot and open a reader.
-    pub fn locked_reader(&self) -> Result<crate::SnapshotReader, Error> {
-        crate::SnapshotReader::new_do(self.clone())
-    }
 }
 
 impl<T> AsRef<pbs_api_types::BackupNamespace> for BackupDir<T> {
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 9356750b..cb2d2172 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -600,7 +600,7 @@ impl<T: CanRead> DataStore<T> {
     pub fn iter_backup_ns(
         self: &Arc<DataStore<T>>,
         ns: BackupNamespace,
-    ) -> Result<ListNamespaces, Error> {
+    ) -> Result<ListNamespaces<T>, Error> {
         ListNamespaces::new(Arc::clone(self), ns)
     }
 
@@ -611,7 +611,7 @@ impl<T: CanRead> DataStore<T> {
     pub fn recursive_iter_backup_ns(
         self: &Arc<DataStore<T>>,
         ns: BackupNamespace,
-    ) -> Result<ListNamespacesRecursive, Error> {
+    ) -> Result<ListNamespacesRecursive<T>, Error> {
         ListNamespacesRecursive::new(Arc::clone(self), ns)
     }
 
@@ -623,7 +623,7 @@ impl<T: CanRead> DataStore<T> {
         self: &Arc<DataStore<T>>,
         ns: BackupNamespace,
         ty: BackupType,
-    ) -> Result<ListGroupsType, Error> {
+    ) -> Result<ListGroupsType<T>, Error> {
         ListGroupsType::new(Arc::clone(self), ns, ty)
     }
 
@@ -634,7 +634,7 @@ impl<T: CanRead> DataStore<T> {
     pub fn iter_backup_groups(
         self: &Arc<DataStore<T>>,
         ns: BackupNamespace,
-    ) -> Result<ListGroups, Error> {
+    ) -> Result<ListGroups<T>, Error> {
         ListGroups::new(Arc::clone(self), ns)
     }
 
@@ -900,7 +900,7 @@ impl<T: CanWrite> DataStore<T> {
         filename: P,
         size: usize,
         chunk_size: usize,
-    ) -> Result<FixedIndexWriter, Error> {
+    ) -> Result<FixedIndexWriter<T>, Error> {
         let index = FixedIndexWriter::create(
             self.inner.chunk_store.clone(),
             filename.as_ref(),
@@ -914,7 +914,7 @@ impl<T: CanWrite> DataStore<T> {
     pub fn create_dynamic_writer<P: AsRef<Path>>(
         &self,
         filename: P,
-    ) -> Result<DynamicIndexWriter, Error> {
+    ) -> Result<DynamicIndexWriter<T>, Error> {
         let index = DynamicIndexWriter::create(self.inner.chunk_store.clone(), filename.as_ref())?;
 
         Ok(index)
diff --git a/pbs-datastore/src/dynamic_index.rs b/pbs-datastore/src/dynamic_index.rs
index 8e9cb116..dbf99faa 100644
--- a/pbs-datastore/src/dynamic_index.rs
+++ b/pbs-datastore/src/dynamic_index.rs
@@ -18,7 +18,7 @@ use pxar::accessor::{MaybeReady, ReadAt, ReadAtOperation};
 use pbs_tools::lru_cache::LruCache;
 
 use crate::chunk_stat::ChunkStat;
-use crate::chunk_store::ChunkStore;
+use crate::chunk_store::{CanWrite, ChunkStore};
 use crate::data_blob::{DataBlob, DataChunkBuilder};
 use crate::file_formats;
 use crate::index::{ChunkReadInfo, IndexFile};
@@ -275,8 +275,8 @@ impl IndexFile for DynamicIndexReader {
 }
 
 /// Create dynamic index files (`.dixd`)
-pub struct DynamicIndexWriter {
-    store: Arc<ChunkStore>,
+pub struct DynamicIndexWriter<T> {
+    store: Arc<ChunkStore<T>>,
     _lock: ProcessLockSharedGuard,
     writer: BufWriter<File>,
     closed: bool,
@@ -287,14 +287,14 @@ pub struct DynamicIndexWriter {
     pub ctime: i64,
 }
 
-impl Drop for DynamicIndexWriter {
+impl<T> Drop for DynamicIndexWriter<T> {
     fn drop(&mut self) {
         let _ = std::fs::remove_file(&self.tmp_filename); // ignore errors
     }
 }
 
-impl DynamicIndexWriter {
-    pub fn create(store: Arc<ChunkStore>, path: &Path) -> Result<Self, Error> {
+impl<T: CanWrite> DynamicIndexWriter<T> {
+    pub fn create(store: Arc<ChunkStore<T>>, path: &Path) -> Result<Self, Error> {
         let shared_lock = store.try_shared_lock()?;
 
         let full_path = store.relative_path(path);
@@ -394,8 +394,8 @@ impl DynamicIndexWriter {
 /// Writer which splits a binary stream into dynamic sized chunks
 ///
 /// And store the resulting chunk list into the index file.
-pub struct DynamicChunkWriter {
-    index: DynamicIndexWriter,
+pub struct DynamicChunkWriter<T: CanWrite> {
+    index: DynamicIndexWriter<T>,
     closed: bool,
     chunker: ChunkerImpl,
     stat: ChunkStat,
@@ -404,8 +404,8 @@ pub struct DynamicChunkWriter {
     chunk_buffer: Vec<u8>,
 }
 
-impl DynamicChunkWriter {
-    pub fn new(index: DynamicIndexWriter, chunk_size: usize) -> Self {
+impl<T: CanWrite> DynamicChunkWriter<T> {
+    pub fn new(index: DynamicIndexWriter<T>, chunk_size: usize) -> Self {
         Self {
             index,
             closed: false,
@@ -490,7 +490,7 @@ impl DynamicChunkWriter {
     }
 }
 
-impl Write for DynamicChunkWriter {
+impl<T: CanWrite> Write for DynamicChunkWriter<T> {
     fn write(&mut self, data: &[u8]) -> std::result::Result<usize, std::io::Error> {
         let chunker = &mut self.chunker;
 
diff --git a/pbs-datastore/src/fixed_index.rs b/pbs-datastore/src/fixed_index.rs
index d4bfcb51..8db9f440 100644
--- a/pbs-datastore/src/fixed_index.rs
+++ b/pbs-datastore/src/fixed_index.rs
@@ -214,8 +214,8 @@ impl IndexFile for FixedIndexReader {
     }
 }
 
-pub struct FixedIndexWriter {
-    store: Arc<ChunkStore>,
+pub struct FixedIndexWriter<T> {
+    store: Arc<ChunkStore<T>>,
     file: File,
     _lock: ProcessLockSharedGuard,
     filename: PathBuf,
@@ -229,9 +229,9 @@ pub struct FixedIndexWriter {
 }
 
 // `index` is mmap()ed which cannot be thread-local so should be sendable
-unsafe impl Send for FixedIndexWriter {}
+unsafe impl<T> Send for FixedIndexWriter<T> {}
 
-impl Drop for FixedIndexWriter {
+impl<T> Drop for FixedIndexWriter<T> {
     fn drop(&mut self) {
         let _ = std::fs::remove_file(&self.tmp_filename); // ignore errors
         if let Err(err) = self.unmap() {
@@ -240,10 +240,30 @@ impl Drop for FixedIndexWriter {
     }
 }
 
-impl FixedIndexWriter {
+impl<T> FixedIndexWriter<T> {
+    fn unmap(&mut self) -> Result<(), Error> {
+        if self.index.is_null() {
+            return Ok(());
+        }
+
+        let index_size = self.index_length * 32;
+
+        if let Err(err) =
+            unsafe { nix::sys::mman::munmap(self.index as *mut std::ffi::c_void, index_size) }
+        {
+            bail!("unmap file {:?} failed - {}", self.tmp_filename, err);
+        }
+
+        self.index = std::ptr::null_mut();
+
+        Ok(())
+    }
+}
+
+impl<T: crate::chunk_store::CanWrite> FixedIndexWriter<T> {
     #[allow(clippy::cast_ptr_alignment)]
     pub fn create(
-        store: Arc<ChunkStore>,
+        store: Arc<ChunkStore<T>>,
         path: &Path,
         size: usize,
         chunk_size: usize,
@@ -320,24 +340,6 @@ impl FixedIndexWriter {
         self.index_length
     }
 
-    fn unmap(&mut self) -> Result<(), Error> {
-        if self.index.is_null() {
-            return Ok(());
-        }
-
-        let index_size = self.index_length * 32;
-
-        if let Err(err) =
-            unsafe { nix::sys::mman::munmap(self.index as *mut std::ffi::c_void, index_size) }
-        {
-            bail!("unmap file {:?} failed - {}", self.tmp_filename, err);
-        }
-
-        self.index = std::ptr::null_mut();
-
-        Ok(())
-    }
-
     pub fn close(&mut self) -> Result<[u8; 32], Error> {
         if self.index.is_null() {
             bail!("cannot close already closed index file.");
diff --git a/pbs-datastore/src/hierarchy.rs b/pbs-datastore/src/hierarchy.rs
index e0bf8441..b331d1de 100644
--- a/pbs-datastore/src/hierarchy.rs
+++ b/pbs-datastore/src/hierarchy.rs
@@ -9,16 +9,17 @@ use pbs_api_types::{BackupNamespace, BackupType, BACKUP_DATE_REGEX, BACKUP_ID_RE
 use proxmox_sys::fs::get_file_type;
 
 use crate::backup_info::{BackupDir, BackupGroup};
+use crate::chunk_store::CanRead;
 use crate::DataStore;
 
 /// A iterator for all BackupDir's (Snapshots) in a BackupGroup
-pub struct ListSnapshots {
-    group: BackupGroup,
+pub struct ListSnapshots<T: CanRead> {
+    group: BackupGroup<T>,
     fd: proxmox_sys::fs::ReadDir,
 }
 
-impl ListSnapshots {
-    pub fn new(group: BackupGroup) -> Result<Self, Error> {
+impl<T: CanRead> ListSnapshots<T> {
+    pub fn new(group: BackupGroup<T>) -> Result<Self, Error> {
         let group_path = group.full_group_path();
         Ok(ListSnapshots {
             fd: proxmox_sys::fs::read_subdir(libc::AT_FDCWD, &group_path)
@@ -28,8 +29,8 @@ impl ListSnapshots {
     }
 }
 
-impl Iterator for ListSnapshots {
-    type Item = Result<BackupDir, Error>;
+impl<T: CanRead> Iterator for ListSnapshots<T> {
+    type Item = Result<BackupDir<T>, Error>;
 
     fn next(&mut self) -> Option<Self::Item> {
         loop {
@@ -64,21 +65,25 @@ impl Iterator for ListSnapshots {
 }
 
 /// An iterator for a single backup group type.
-pub struct ListGroupsType {
-    store: Arc<DataStore>,
+pub struct ListGroupsType<T> {
+    store: Arc<DataStore<T>>,
     ns: BackupNamespace,
     ty: BackupType,
     dir: proxmox_sys::fs::ReadDir,
 }
 
-impl ListGroupsType {
-    pub fn new(store: Arc<DataStore>, ns: BackupNamespace, ty: BackupType) -> Result<Self, Error> {
+impl<T: CanRead> ListGroupsType<T> {
+    pub fn new(
+        store: Arc<DataStore<T>>,
+        ns: BackupNamespace,
+        ty: BackupType,
+    ) -> Result<Self, Error> {
         Self::new_at(libc::AT_FDCWD, store, ns, ty)
     }
 
     fn new_at(
         fd: RawFd,
-        store: Arc<DataStore>,
+        store: Arc<DataStore<T>>,
         ns: BackupNamespace,
         ty: BackupType,
     ) -> Result<Self, Error> {
@@ -90,13 +95,13 @@ impl ListGroupsType {
         })
     }
 
-    pub(crate) fn ok(self) -> ListGroupsOk<Self> {
+    pub(crate) fn ok(self) -> ListGroupsOk<Self, T> {
         ListGroupsOk::new(self)
     }
 }
 
-impl Iterator for ListGroupsType {
-    type Item = Result<BackupGroup, Error>;
+impl<T: CanRead> Iterator for ListGroupsType<T> {
+    type Item = Result<BackupGroup<T>, Error>;
 
     fn next(&mut self) -> Option<Self::Item> {
         loop {
@@ -134,15 +139,15 @@ impl Iterator for ListGroupsType {
 }
 
 /// A iterator for a (single) level of Backup Groups
-pub struct ListGroups {
-    store: Arc<DataStore>,
+pub struct ListGroups<T> {
+    store: Arc<DataStore<T>>,
     ns: BackupNamespace,
     type_fd: proxmox_sys::fs::ReadDir,
-    id_state: Option<ListGroupsType>,
+    id_state: Option<ListGroupsType<T>>,
 }
 
-impl ListGroups {
-    pub fn new(store: Arc<DataStore>, ns: BackupNamespace) -> Result<Self, Error> {
+impl<T: CanRead> ListGroups<T> {
+    pub fn new(store: Arc<DataStore<T>>, ns: BackupNamespace) -> Result<Self, Error> {
         Ok(Self {
             type_fd: proxmox_sys::fs::read_subdir(libc::AT_FDCWD, &store.namespace_path(&ns))?,
             store,
@@ -151,13 +156,13 @@ impl ListGroups {
         })
     }
 
-    pub(crate) fn ok(self) -> ListGroupsOk<Self> {
+    pub(crate) fn ok(self) -> ListGroupsOk<Self, T> {
         ListGroupsOk::new(self)
     }
 }
 
-impl Iterator for ListGroups {
-    type Item = Result<BackupGroup, Error>;
+impl<T: CanRead> Iterator for ListGroups<T> {
+    type Item = Result<BackupGroup<T>, Error>;
 
     fn next(&mut self) -> Option<Self::Item> {
         loop {
@@ -217,36 +222,36 @@ pub(crate) trait GroupIter {
     fn store_name(&self) -> &str;
 }
 
-impl GroupIter for ListGroups {
+impl<T: CanRead> GroupIter for ListGroups<T> {
     fn store_name(&self) -> &str {
         self.store.name()
     }
 }
 
-impl GroupIter for ListGroupsType {
+impl<T: CanRead> GroupIter for ListGroupsType<T> {
     fn store_name(&self) -> &str {
         self.store.name()
     }
 }
 
-pub(crate) struct ListGroupsOk<I>(Option<I>)
+pub(crate) struct ListGroupsOk<I, T: CanRead>(Option<I>)
 where
-    I: GroupIter + Iterator<Item = Result<BackupGroup, Error>>;
+    I: GroupIter + Iterator<Item = Result<BackupGroup<T>, Error>>;
 
-impl<I> ListGroupsOk<I>
+impl<I, T: CanRead> ListGroupsOk<I, T>
 where
-    I: GroupIter + Iterator<Item = Result<BackupGroup, Error>>,
+    I: GroupIter + Iterator<Item = Result<BackupGroup<T>, Error>>,
 {
     fn new(inner: I) -> Self {
         Self(Some(inner))
     }
 }
 
-impl<I> Iterator for ListGroupsOk<I>
+impl<I, T: CanRead> Iterator for ListGroupsOk<I, T>
 where
-    I: GroupIter + Iterator<Item = Result<BackupGroup, Error>>,
+    I: GroupIter + Iterator<Item = Result<BackupGroup<T>, Error>>,
 {
-    type Item = BackupGroup;
+    type Item = BackupGroup<T>;
 
     fn next(&mut self) -> Option<Self::Item> {
         if let Some(iter) = &mut self.0 {
@@ -269,19 +274,21 @@ where
 }
 
 /// A iterator for a (single) level of Namespaces
-pub struct ListNamespaces {
+pub struct ListNamespaces<T> {
     ns: BackupNamespace,
     base_path: PathBuf,
     ns_state: Option<proxmox_sys::fs::ReadDir>,
+    _marker: std::marker::PhantomData<T>,
 }
 
-impl ListNamespaces {
+impl<T: CanRead> ListNamespaces<T> {
     /// construct a new single-level namespace iterator on a datastore with an optional anchor ns
-    pub fn new(store: Arc<DataStore>, ns: BackupNamespace) -> Result<Self, Error> {
+    pub fn new(store: Arc<DataStore<T>>, ns: BackupNamespace) -> Result<Self, Error> {
         Ok(ListNamespaces {
             ns,
             base_path: store.base_path(),
             ns_state: None,
+            _marker: std::marker::PhantomData,
         })
     }
 
@@ -293,11 +300,12 @@ impl ListNamespaces {
             ns: ns.unwrap_or_default(),
             base_path: path,
             ns_state: None,
+            _marker: std::marker::PhantomData,
         })
     }
 }
 
-impl Iterator for ListNamespaces {
+impl<T: CanRead> Iterator for ListNamespaces<T> {
     type Item = Result<BackupNamespace, Error>;
 
     fn next(&mut self) -> Option<Self::Item> {
@@ -361,18 +369,18 @@ impl Iterator for ListNamespaces {
 /// can be useful for searching all backup groups from a certain anchor, as that can contain
 /// sub-namespaces but also groups on its own level, so otherwise one would need to special case
 /// the ones from the own level.
-pub struct ListNamespacesRecursive {
-    store: Arc<DataStore>,
+pub struct ListNamespacesRecursive<T> {
+    store: Arc<DataStore<T>>,
     /// the starting namespace we search downward from
     ns: BackupNamespace,
     /// the maximal recursion depth from the anchor start ns (depth == 0) downwards
     max_depth: u8,
-    state: Option<Vec<ListNamespaces>>, // vector to avoid code recursion
+    state: Option<Vec<ListNamespaces<T>>>, // vector to avoid code recursion
 }
 
-impl ListNamespacesRecursive {
+impl<T: CanRead> ListNamespacesRecursive<T> {
     /// Creates an recursive namespace iterator.
-    pub fn new(store: Arc<DataStore>, ns: BackupNamespace) -> Result<Self, Error> {
+    pub fn new(store: Arc<DataStore<T>>, ns: BackupNamespace) -> Result<Self, Error> {
         Self::new_max_depth(store, ns, pbs_api_types::MAX_NAMESPACE_DEPTH)
     }
 
@@ -383,7 +391,7 @@ impl ListNamespacesRecursive {
     /// Depth is counted relatively, that means not from the datastore as anchor, but from `ns`,
     /// and it will be clamped to `min(depth, MAX_NAMESPACE_DEPTH - ns.depth())` automatically.
     pub fn new_max_depth(
-        store: Arc<DataStore>,
+        store: Arc<DataStore<T>>,
         ns: BackupNamespace,
         max_depth: usize,
     ) -> Result<Self, Error> {
@@ -403,7 +411,7 @@ impl ListNamespacesRecursive {
     }
 }
 
-impl Iterator for ListNamespacesRecursive {
+impl<T: CanRead> Iterator for ListNamespacesRecursive<T> {
     type Item = Result<BackupNamespace, Error>;
 
     fn next(&mut self) -> Option<Self::Item> {
diff --git a/pbs-datastore/src/local_chunk_reader.rs b/pbs-datastore/src/local_chunk_reader.rs
index 05a70c06..ccdde9f1 100644
--- a/pbs-datastore/src/local_chunk_reader.rs
+++ b/pbs-datastore/src/local_chunk_reader.rs
@@ -7,20 +7,21 @@ use anyhow::{bail, Error};
 use pbs_api_types::CryptMode;
 use pbs_tools::crypt_config::CryptConfig;
 
+use crate::chunk_store::CanRead;
 use crate::data_blob::DataBlob;
 use crate::read_chunk::{AsyncReadChunk, ReadChunk};
 use crate::DataStore;
 
 #[derive(Clone)]
-pub struct LocalChunkReader {
-    store: Arc<DataStore>,
+pub struct LocalChunkReader<T: CanRead> {
+    store: Arc<DataStore<T>>,
     crypt_config: Option<Arc<CryptConfig>>,
     crypt_mode: CryptMode,
 }
 
-impl LocalChunkReader {
+impl<T: CanRead> LocalChunkReader<T> {
     pub fn new(
-        store: Arc<DataStore>,
+        store: Arc<DataStore<T>>,
         crypt_config: Option<Arc<CryptConfig>>,
         crypt_mode: CryptMode,
     ) -> Self {
@@ -47,7 +48,7 @@ impl LocalChunkReader {
     }
 }
 
-impl ReadChunk for LocalChunkReader {
+impl<T: CanRead> ReadChunk for LocalChunkReader<T> {
     fn read_raw_chunk(&self, digest: &[u8; 32]) -> Result<DataBlob, Error> {
         let chunk = self.store.load_chunk(digest)?;
         self.ensure_crypt_mode(chunk.crypt_mode()?)?;
@@ -63,7 +64,7 @@ impl ReadChunk for LocalChunkReader {
     }
 }
 
-impl AsyncReadChunk for LocalChunkReader {
+impl<T: CanRead + Send + Sync> AsyncReadChunk for LocalChunkReader<T> {
     fn read_raw_chunk<'a>(
         &'a self,
         digest: &'a [u8; 32],
diff --git a/pbs-datastore/src/prune.rs b/pbs-datastore/src/prune.rs
index 7b6f9f75..a8eb511b 100644
--- a/pbs-datastore/src/prune.rs
+++ b/pbs-datastore/src/prune.rs
@@ -5,6 +5,8 @@ use anyhow::Error;
 
 use pbs_api_types::KeepOptions;
 
+use crate::chunk_store::CanRead;
+
 use super::BackupInfo;
 
 #[derive(Clone, Copy, PartialEq, Eq)]
@@ -36,9 +38,9 @@ impl std::fmt::Display for PruneMark {
     }
 }
 
-fn mark_selections<F: Fn(&BackupInfo) -> Result<String, Error>>(
+fn mark_selections<F: Fn(&BackupInfo<T>) -> Result<String, Error>, T: CanRead>(
     mark: &mut HashMap<PathBuf, PruneMark>,
-    list: &[BackupInfo],
+    list: &[BackupInfo<T>],
     keep: usize,
     select_id: F,
 ) -> Result<(), Error> {
@@ -82,7 +84,10 @@ fn mark_selections<F: Fn(&BackupInfo) -> Result<String, Error>>(
     Ok(())
 }
 
-fn remove_incomplete_snapshots(mark: &mut HashMap<PathBuf, PruneMark>, list: &[BackupInfo]) {
+fn remove_incomplete_snapshots<T: CanRead>(
+    mark: &mut HashMap<PathBuf, PruneMark>,
+    list: &[BackupInfo<T>],
+) {
     let mut keep_unfinished = true;
     for info in list.iter() {
         // backup is considered unfinished if there is no manifest
@@ -104,10 +109,10 @@ fn remove_incomplete_snapshots(mark: &mut HashMap<PathBuf, PruneMark>, list: &[B
 }
 
 /// This filters incomplete and kept backups.
-pub fn compute_prune_info(
-    mut list: Vec<BackupInfo>,
+pub fn compute_prune_info<T: CanRead>(
+    mut list: Vec<BackupInfo<T>>,
     options: &KeepOptions,
-) -> Result<Vec<(BackupInfo, PruneMark)>, Error> {
+) -> Result<Vec<(BackupInfo<T>, PruneMark)>, Error> {
     let mut mark = HashMap::new();
 
     BackupInfo::sort_list(&mut list, false);
@@ -154,7 +159,7 @@ pub fn compute_prune_info(
         })?;
     }
 
-    let prune_info: Vec<(BackupInfo, PruneMark)> = list
+    let prune_info: Vec<(BackupInfo<T>, PruneMark)> = list
         .into_iter()
         .map(|info| {
             let backup_id = info.backup_dir.relative_path();
diff --git a/pbs-datastore/src/snapshot_reader.rs b/pbs-datastore/src/snapshot_reader.rs
index 5da0533c..d604507d 100644
--- a/pbs-datastore/src/snapshot_reader.rs
+++ b/pbs-datastore/src/snapshot_reader.rs
@@ -16,6 +16,7 @@ use pbs_api_types::{
 };
 
 use crate::backup_info::BackupDir;
+use crate::chunk_store::CanRead;
 use crate::dynamic_index::DynamicIndexReader;
 use crate::fixed_index::FixedIndexReader;
 use crate::index::IndexFile;
@@ -24,8 +25,8 @@ use crate::DataStore;
 /// Helper to access the contents of a datastore backup snapshot
 ///
 /// This make it easy to iterate over all used chunks and files.
-pub struct SnapshotReader {
-    snapshot: BackupDir,
+pub struct SnapshotReader<T: CanRead> {
+    snapshot: BackupDir<T>,
     datastore_name: String,
     file_list: Vec<String>,
     locked_dir: Dir,
@@ -35,17 +36,17 @@ pub struct SnapshotReader {
     _lock: BackupLockGuard,
 }
 
-impl SnapshotReader {
+impl<T: CanRead> SnapshotReader<T> {
     /// Lock snapshot, reads the manifest and returns a new instance
     pub fn new(
-        datastore: Arc<DataStore>,
+        datastore: Arc<DataStore<T>>,
         ns: BackupNamespace,
         snapshot: pbs_api_types::BackupDir,
     ) -> Result<Self, Error> {
         Self::new_do(datastore.backup_dir(ns, snapshot)?)
     }
 
-    pub(crate) fn new_do(snapshot: BackupDir) -> Result<Self, Error> {
+    pub(crate) fn new_do(snapshot: BackupDir<T>) -> Result<Self, Error> {
         let datastore = snapshot.datastore();
         let snapshot_path = snapshot.full_path();
 
@@ -94,7 +95,7 @@ impl SnapshotReader {
     }
 
     /// Return the snapshot directory
-    pub fn snapshot(&self) -> &BackupDir {
+    pub fn snapshot(&self) -> &BackupDir<T> {
         &self.snapshot
     }
 
@@ -124,7 +125,7 @@ impl SnapshotReader {
     pub fn chunk_iterator<F: Fn(&[u8; 32]) -> bool>(
         &self,
         skip_fn: F,
-    ) -> Result<SnapshotChunkIterator<F>, Error> {
+    ) -> Result<SnapshotChunkIterator<F, T>, Error> {
         SnapshotChunkIterator::new(self, skip_fn)
     }
 }
@@ -134,15 +135,15 @@ impl SnapshotReader {
 /// Note: The iterator returns a `Result`, and the iterator state is
 /// undefined after the first error. So it make no sense to continue
 /// iteration after the first error.
-pub struct SnapshotChunkIterator<'a, F: Fn(&[u8; 32]) -> bool> {
-    snapshot_reader: &'a SnapshotReader,
+pub struct SnapshotChunkIterator<'a, F: Fn(&[u8; 32]) -> bool, T: CanRead> {
+    snapshot_reader: &'a SnapshotReader<T>,
     todo_list: Vec<String>,
     skip_fn: F,
     #[allow(clippy::type_complexity)]
     current_index: Option<(Rc<Box<dyn IndexFile + Send>>, usize, Vec<(usize, u64)>)>,
 }
 
-impl<F: Fn(&[u8; 32]) -> bool> Iterator for SnapshotChunkIterator<'_, F> {
+impl<F: Fn(&[u8; 32]) -> bool, T: CanRead> Iterator for SnapshotChunkIterator<'_, F, T> {
     type Item = Result<[u8; 32], Error>;
 
     fn next(&mut self) -> Option<Self::Item> {
@@ -189,8 +190,8 @@ impl<F: Fn(&[u8; 32]) -> bool> Iterator for SnapshotChunkIterator<'_, F> {
     }
 }
 
-impl<'a, F: Fn(&[u8; 32]) -> bool> SnapshotChunkIterator<'a, F> {
-    pub fn new(snapshot_reader: &'a SnapshotReader, skip_fn: F) -> Result<Self, Error> {
+impl<'a, F: Fn(&[u8; 32]) -> bool, T: CanRead> SnapshotChunkIterator<'a, F, T> {
+    pub fn new(snapshot_reader: &'a SnapshotReader<T>, skip_fn: F) -> Result<Self, Error> {
         let mut todo_list = Vec::new();
 
         for filename in snapshot_reader.file_list() {
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:37 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:37 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 04/12] datastore: separate
 functions into impl block
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-5-h.laimer@proxmox.com>

... based on whether they are reading/writing.

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 1282 ++++++++++++++++----------------
 1 file changed, 643 insertions(+), 639 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 6936875e..66a2e209 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -27,7 +27,6 @@ use pbs_api_types::{
     DataStoreConfig, DatastoreFSyncLevel, DatastoreTuning, GarbageCollectionStatus,
     MaintenanceMode, MaintenanceType, Operation, UPID,
 };
-use pbs_config::BackupLockGuard;
 
 use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, OLD_LOCKING};
 use crate::chunk_store::{CanRead, CanWrite, ChunkStore, Lookup as L, Read as R, Write as W};
@@ -250,28 +249,87 @@ impl DataStore<W> {
     }
 }
 
-impl DataStore {
-    // This one just panics on everything
-    #[doc(hidden)]
-    pub(crate) unsafe fn new_test() -> Arc<Self> {
-        Arc::new(Self {
-            inner: unsafe { DataStoreImpl::new_test() },
-            operation: None,
-        })
+impl<T: CanRead + 'static> DataStore<T> {
+    /// Get a streaming iter over single-level backup namespaces of a datatstore, filtered by Ok
+    ///
+    /// The iterated item's result is already unwrapped, if it contained an error it will be
+    /// logged. Can be useful in iterator chain commands
+    pub fn iter_backup_ns_ok(
+        self: &Arc<DataStore<T>>,
+        ns: BackupNamespace,
+    ) -> Result<impl Iterator<Item = BackupNamespace> + 'static, Error> {
+        let this = Arc::clone(self);
+        Ok(
+            ListNamespaces::new(Arc::clone(self), ns)?.filter_map(move |ns| match ns {
+                Ok(ns) => Some(ns),
+                Err(err) => {
+                    log::error!("list groups error on datastore {} - {}", this.name(), err);
+                    None
+                }
+            }),
+        )
+    }
+
+    /// Get a streaming iter over single-level backup namespaces of a datatstore, filtered by Ok
+    ///
+    /// The iterated item's result is already unwrapped, if it contained an error it will be
+    /// logged. Can be useful in iterator chain commands
+    pub fn recursive_iter_backup_ns_ok(
+        self: &Arc<DataStore<T>>,
+        ns: BackupNamespace,
+        max_depth: Option<usize>,
+    ) -> Result<impl Iterator<Item = BackupNamespace> + 'static, Error> {
+        let this = Arc::clone(self);
+        Ok(if let Some(depth) = max_depth {
+            ListNamespacesRecursive::new_max_depth(Arc::clone(self), ns, depth)?
+        } else {
+            ListNamespacesRecursive::new(Arc::clone(self), ns)?
+        }
+        .filter_map(move |ns| match ns {
+            Ok(ns) => Some(ns),
+            Err(err) => {
+                log::error!("list groups error on datastore {} - {}", this.name(), err);
+                None
+            }
+        }))
+    }
+
+    /// Get a streaming iter over top-level backup groups of a datastore of a particular type,
+    /// filtered by `Ok` results
+    ///
+    /// The iterated item's result is already unwrapped, if it contained an error it will be
+    /// logged. Can be useful in iterator chain commands
+    pub fn iter_backup_type_ok(
+        self: &Arc<DataStore<T>>,
+        ns: BackupNamespace,
+        ty: BackupType,
+    ) -> Result<impl Iterator<Item = BackupGroup> + 'static, Error> {
+        Ok(self.iter_backup_type(ns, ty)?.ok())
+    }
+
+    /// Get a streaming iter over top-level backup groups of a datatstore, filtered by Ok results
+    ///
+    /// The iterated item's result is already unwrapped, if it contained an error it will be
+    /// logged. Can be useful in iterator chain commands
+    pub fn iter_backup_groups_ok(
+        self: &Arc<DataStore<T>>,
+        ns: BackupNamespace,
+    ) -> Result<impl Iterator<Item = BackupGroup> + 'static, Error> {
+        Ok(self.iter_backup_groups(ns)?.ok())
     }
+}
 
-    pub fn lookup_datastore(
+impl<T: CanRead> DataStore<T> {
+    pub fn open_datastore(
         name: &str,
         operation: Option<Operation>,
-    ) -> Result<Arc<DataStore>, Error> {
+        cache_entry: Option<Arc<DataStoreImpl<T>>>,
+    ) -> Result<Arc<DataStore<T>>, Error> {
         // Avoid TOCTOU between checking maintenance mode and updating active operation counter, as
         // we use it to decide whether it is okay to delete the datastore.
-        let _config_lock = pbs_config::datastore::lock_config()?;
-
         // we could use the ConfigVersionCache's generation for staleness detection, but  we load
         // the config anyway -> just use digest, additional benefit: manual changes get detected
-        let (config, digest) = pbs_config::datastore::config()?;
-        let config: DataStoreConfig = config.lookup("datastore", name)?;
+        let (config, digest, _lock) = Self::read_config(name)?;
 
         if let Some(maintenance_mode) = config.get_maintenance_mode() {
             if let Err(error) = maintenance_mode.check(operation) {
@@ -280,16 +338,11 @@ impl DataStore {
         }
 
         if get_datastore_mount_status(&config) == Some(false) {
-            let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
-            datastore_cache.remove(&config.name);
             bail!("datastore '{}' is not mounted", config.name);
         }
 
-        let mut datastore_cache = DATASTORE_MAP.lock().unwrap();
-        let entry = datastore_cache.get(name);
-
         // reuse chunk store so that we keep using the same process locker instance!
-        let chunk_store = if let Some(datastore) = &entry {
+        let chunk_store = if let Some(datastore) = &cache_entry {
             let last_digest = datastore.last_digest.as_ref();
             if let Some(true) = last_digest.map(|last_digest| last_digest == &digest) {
                 if let Some(operation) = operation {
@@ -306,73 +359,25 @@ impl DataStore {
                 DatastoreTuning::API_SCHEMA
                     .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
             )?;
-            Arc::new(ChunkStore::open(
+            Arc::new(ChunkStore::<T>::open(
                 name,
                 config.absolute_path(),
                 tuning.sync_level.unwrap_or_default(),
             )?)
         };
 
-        let datastore = DataStore::with_store_and_config(chunk_store, config, Some(digest))?;
-
-        let datastore = Arc::new(datastore);
-        datastore_cache.insert(name.to_string(), datastore.clone());
+        let datastore = Self::with_store_and_config(chunk_store, config, Some(digest))?;
 
         if let Some(operation) = operation {
             update_active_operations(name, operation, 1)?;
         }
 
         Ok(Arc::new(Self {
-            inner: datastore,
+            inner: datastore.into(),
             operation,
         }))
     }
 
-    /// removes all datastores that are not configured anymore
-    pub fn remove_unused_datastores() -> Result<(), Error> {
-        let (config, _digest) = pbs_config::datastore::config()?;
-
-        let mut map = DATASTORE_MAP.lock().unwrap();
-        // removes all elements that are not in the config
-        map.retain(|key, _| config.sections.contains_key(key));
-        Ok(())
-    }
-
-    /// trigger clearing cache entry based on maintenance mode. Entry will only
-    /// be cleared iff there is no other task running, if there is, the end of the
-    /// last running task will trigger the clearing of the cache entry.
-    pub fn update_datastore_cache(name: &str) -> Result<(), Error> {
-        let (config, _digest) = pbs_config::datastore::config()?;
-        let datastore: DataStoreConfig = config.lookup("datastore", name)?;
-        if datastore
-            .get_maintenance_mode()
-            .is_some_and(|m| m.clear_from_cache())
-        {
-            // the datastore drop handler does the checking if tasks are running and clears the
-            // cache entry, so we just have to trigger it here
-            let _ = DataStore::lookup_datastore(name, Some(Operation::Lookup));
-        }
-
-        Ok(())
-    }
-
-    /// Open a raw database given a name and a path.
-    ///
-    /// # Safety
-    /// See the safety section in `open_from_config`
-    pub unsafe fn open_path(
-        name: &str,
-        path: impl AsRef<Path>,
-        operation: Option<Operation>,
-    ) -> Result<Arc<Self>, Error> {
-        let path = path
-            .as_ref()
-            .to_str()
-            .ok_or_else(|| format_err!("non-utf8 paths not supported"))?
-            .to_owned();
-        unsafe { Self::open_from_config(DataStoreConfig::new(name.to_owned(), path), operation) }
-    }
-
     /// Open a datastore given a raw configuration.
     ///
     /// # Safety
@@ -394,7 +399,7 @@ impl DataStore {
             DatastoreTuning::API_SCHEMA
                 .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
         )?;
-        let chunk_store = ChunkStore::open(
+        let chunk_store = ChunkStore::<T>::open(
             &name,
             config.absolute_path(),
             tuning.sync_level.unwrap_or_default(),
@@ -413,10 +418,10 @@ impl DataStore {
     }
 
     fn with_store_and_config(
-        chunk_store: Arc<ChunkStore>,
+        chunk_store: Arc<ChunkStore<T>>,
         config: DataStoreConfig,
         last_digest: Option<[u8; 32]>,
-    ) -> Result<DataStoreImpl, Error> {
+    ) -> Result<DataStoreImpl<T>, Error> {
         let mut gc_status_path = chunk_store.base_path();
         gc_status_path.push(".gc-status");
 
@@ -448,6 +453,23 @@ impl DataStore {
         })
     }
 
+    /// Open a raw database given a name and a path.
+    ///
+    /// # Safety
+    /// See the safety section in `open_from_config`
+    pub unsafe fn open_path(
+        name: &str,
+        path: impl AsRef<Path>,
+        operation: Option<Operation>,
+    ) -> Result<Arc<Self>, Error> {
+        let path = path
+            .as_ref()
+            .to_str()
+            .ok_or_else(|| format_err!("non-utf8 paths not supported"))?
+            .to_owned();
+        unsafe { Self::open_from_config(DataStoreConfig::new(name.to_owned(), path), operation) }
+    }
+
     pub fn get_chunk_iterator(
         &self,
     ) -> Result<
@@ -457,53 +479,6 @@ impl DataStore {
         self.inner.chunk_store.get_chunk_iterator()
     }
 
-    pub fn create_fixed_writer<P: AsRef<Path>>(
-        &self,
-        filename: P,
-        size: usize,
-        chunk_size: usize,
-    ) -> Result<FixedIndexWriter, Error> {
-        let index = FixedIndexWriter::create(
-            self.inner.chunk_store.clone(),
-            filename.as_ref(),
-            size,
-            chunk_size,
-        )?;
-
-        Ok(index)
-    }
-
-    pub fn open_fixed_reader<P: AsRef<Path>>(
-        &self,
-        filename: P,
-    ) -> Result<FixedIndexReader, Error> {
-        let full_path = self.inner.chunk_store.relative_path(filename.as_ref());
-
-        let index = FixedIndexReader::open(&full_path)?;
-
-        Ok(index)
-    }
-
-    pub fn create_dynamic_writer<P: AsRef<Path>>(
-        &self,
-        filename: P,
-    ) -> Result<DynamicIndexWriter, Error> {
-        let index = DynamicIndexWriter::create(self.inner.chunk_store.clone(), filename.as_ref())?;
-
-        Ok(index)
-    }
-
-    pub fn open_dynamic_reader<P: AsRef<Path>>(
-        &self,
-        filename: P,
-    ) -> Result<DynamicIndexReader, Error> {
-        let full_path = self.inner.chunk_store.relative_path(filename.as_ref());
-
-        let index = DynamicIndexReader::open(&full_path)?;
-
-        Ok(index)
-    }
-
     pub fn open_index<P>(&self, filename: P) -> Result<Box<dyn IndexFile + Send>, Error>
     where
         P: AsRef<Path>,
@@ -543,73 +518,26 @@ impl DataStore {
         Ok(())
     }
 
-    pub fn name(&self) -> &str {
-        self.inner.chunk_store.name()
-    }
-
-    pub fn base_path(&self) -> PathBuf {
-        self.inner.chunk_store.base_path()
-    }
-
-    /// Returns the absolute path for a backup namespace on this datastore
-    pub fn namespace_path(&self, ns: &BackupNamespace) -> PathBuf {
-        let mut path = self.base_path();
-        path.reserve(ns.path_len());
-        for part in ns.components() {
-            path.push("ns");
-            path.push(part);
-        }
-        path
-    }
+    pub fn open_fixed_reader<P: AsRef<Path>>(
+        &self,
+        filename: P,
+    ) -> Result<FixedIndexReader, Error> {
+        let full_path = self.inner.chunk_store.relative_path(filename.as_ref());
 
-    /// Returns the absolute path for a backup_type
-    pub fn type_path(&self, ns: &BackupNamespace, backup_type: BackupType) -> PathBuf {
-        let mut full_path = self.namespace_path(ns);
-        full_path.push(backup_type.to_string());
-        full_path
-    }
+        let index = FixedIndexReader::open(&full_path)?;
 
-    /// Returns the absolute path for a backup_group
-    pub fn group_path(
-        &self,
-        ns: &BackupNamespace,
-        backup_group: &pbs_api_types::BackupGroup,
-    ) -> PathBuf {
-        let mut full_path = self.namespace_path(ns);
-        full_path.push(backup_group.to_string());
-        full_path
+        Ok(index)
     }
 
-    /// Returns the absolute path for backup_dir
-    pub fn snapshot_path(
+    pub fn open_dynamic_reader<P: AsRef<Path>>(
         &self,
-        ns: &BackupNamespace,
-        backup_dir: &pbs_api_types::BackupDir,
-    ) -> PathBuf {
-        let mut full_path = self.namespace_path(ns);
-        full_path.push(backup_dir.to_string());
-        full_path
-    }
-
-    /// Create a backup namespace.
-    pub fn create_namespace(
-        self: &Arc<Self>,
-        parent: &BackupNamespace,
-        name: String,
-    ) -> Result<BackupNamespace, Error> {
-        if !self.namespace_exists(parent) {
-            bail!("cannot create new namespace, parent {parent} doesn't already exists");
-        }
-
-        // construct ns before mkdir to enforce max-depth and name validity
-        let ns = BackupNamespace::from_parent_ns(parent, name)?;
-
-        let mut ns_full_path = self.base_path();
-        ns_full_path.push(ns.path());
+        filename: P,
+    ) -> Result<DynamicIndexReader, Error> {
+        let full_path = self.inner.chunk_store.relative_path(filename.as_ref());
 
-        std::fs::create_dir_all(ns_full_path)?;
+        let index = DynamicIndexReader::open(&full_path)?;
 
-        Ok(ns)
+        Ok(index)
     }
 
     /// Returns if the given namespace exists on the datastore
@@ -619,7 +547,401 @@ impl DataStore {
         path.exists()
     }
 
-    /// Remove all backup groups of a single namespace level but not the namespace itself.
+    /// Returns the time of the last successful backup
+    ///
+    /// Or None if there is no backup in the group (or the group dir does not exist).
+    pub fn last_successful_backup(
+        self: &Arc<Self>,
+        ns: &BackupNamespace,
+        backup_group: &pbs_api_types::BackupGroup,
+    ) -> Result<Option<i64>, Error> {
+        let backup_group = self.backup_group(ns.clone(), backup_group.clone());
+
+        let group_path = backup_group.full_group_path();
+
+        if group_path.exists() {
+            backup_group.last_successful_backup()
+        } else {
+            Ok(None)
+        }
+    }
+
+    /// Returns the backup owner.
+    ///
+    /// The backup owner is the entity who first created the backup group.
+    pub fn get_owner(
+        &self,
+        ns: &BackupNamespace,
+        backup_group: &pbs_api_types::BackupGroup,
+    ) -> Result<Authid, Error> {
+        let full_path = self.owner_path(ns, backup_group);
+        let owner = proxmox_sys::fs::file_read_firstline(full_path)?;
+        owner
+            .trim_end() // remove trailing newline
+            .parse()
+            .map_err(|err| format_err!("parsing owner for {backup_group} failed: {err}"))
+    }
+
+    pub fn owns_backup(
+        &self,
+        ns: &BackupNamespace,
+        backup_group: &pbs_api_types::BackupGroup,
+        auth_id: &Authid,
+    ) -> Result<bool, Error> {
+        let owner = self.get_owner(ns, backup_group)?;
+
+        Ok(check_backup_owner(&owner, auth_id).is_ok())
+    }
+
+    /// Get a streaming iter over single-level backup namespaces of a datatstore
+    ///
+    /// The iterated item is still a Result that can contain errors from rather unexptected FS or
+    /// parsing errors.
+    pub fn iter_backup_ns(
+        self: &Arc<DataStore<T>>,
+        ns: BackupNamespace,
+    ) -> Result<ListNamespaces, Error> {
+        ListNamespaces::new(Arc::clone(self), ns)
+    }
+
+    /// Get a streaming iter over single-level backup namespaces of a datatstore
+    ///
+    /// The iterated item is still a Result that can contain errors from rather unexptected FS or
+    /// parsing errors.
+    pub fn recursive_iter_backup_ns(
+        self: &Arc<DataStore<T>>,
+        ns: BackupNamespace,
+    ) -> Result<ListNamespacesRecursive, Error> {
+        ListNamespacesRecursive::new(Arc::clone(self), ns)
+    }
+
+    /// Get a streaming iter over top-level backup groups of a datatstore of a particular type.
+    ///
+    /// The iterated item is still a Result that can contain errors from rather unexptected FS or
+    /// parsing errors.
+    pub fn iter_backup_type(
+        self: &Arc<DataStore<T>>,
+        ns: BackupNamespace,
+        ty: BackupType,
+    ) -> Result<ListGroupsType, Error> {
+        ListGroupsType::new(Arc::clone(self), ns, ty)
+    }
+
+    /// Get a streaming iter over top-level backup groups of a datatstore
+    ///
+    /// The iterated item is still a Result that can contain errors from rather unexptected FS or
+    /// parsing errors.
+    pub fn iter_backup_groups(
+        self: &Arc<DataStore<T>>,
+        ns: BackupNamespace,
+    ) -> Result<ListGroups, Error> {
+        ListGroups::new(Arc::clone(self), ns)
+    }
+
+    /// Get a in-memory vector for all top-level backup groups of a datatstore
+    ///
+    /// NOTE: using the iterator directly is most often more efficient w.r.t. memory usage
+    pub fn list_backup_groups(
+        self: &Arc<DataStore<T>>,
+        ns: BackupNamespace,
+    ) -> Result<Vec<BackupGroup>, Error> {
+        ListGroups::new(Arc::clone(self), ns)?.collect()
+    }
+
+    /// Lookup all index files to be found in the datastore without taking any logical iteration
+    /// into account.
+    /// The filesystem is walked recursevly to detect index files based on their archive type based
+    /// on the filename. This however excludes the chunks folder, hidden files and does not follow
+    /// symlinks.
+    fn list_index_files(&self) -> Result<HashSet<PathBuf>, Error> {
+        let base = self.base_path();
+
+        let mut list = HashSet::new();
+
+        use walkdir::WalkDir;
+
+        let walker = WalkDir::new(base).into_iter();
+
+        // make sure we skip .chunks (and other hidden files to keep it simple)
+        fn is_hidden(entry: &walkdir::DirEntry) -> bool {
+            entry
+                .file_name()
+                .to_str()
+                .map(|s| s.starts_with('.'))
+                .unwrap_or(false)
+        }
+        let handle_entry_err = |err: walkdir::Error| {
+            // first, extract the actual IO error and the affected path
+            let (inner, path) = match (err.io_error(), err.path()) {
+                (None, _) => return Ok(()), // not an IO-error
+                (Some(inner), Some(path)) => (inner, path),
+                (Some(inner), None) => bail!("unexpected error on datastore traversal: {inner}"),
+            };
+            if inner.kind() == io::ErrorKind::PermissionDenied {
+                if err.depth() <= 1 && path.ends_with("lost+found") {
+                    // allow skipping of (root-only) ext4 fsck-directory on EPERM ..
+                    return Ok(());
+                }
+                // .. but do not ignore EPERM in general, otherwise we might prune too many chunks.
+                // E.g., if users messed up with owner/perms on a rsync
+                bail!("cannot continue garbage-collection safely, permission denied on: {path:?}");
+            } else if inner.kind() == io::ErrorKind::NotFound {
+                log::info!("ignoring vanished file: {path:?}");
+                return Ok(());
+            } else {
+                bail!("unexpected error on datastore traversal: {inner} - {path:?}");
+            }
+        };
+        for entry in walker.filter_entry(|e| !is_hidden(e)) {
+            let path = match entry {
+                Ok(entry) => entry.into_path(),
+                Err(err) => {
+                    handle_entry_err(err)?;
+                    continue;
+                }
+            };
+            if let Ok(archive_type) = ArchiveType::from_path(&path) {
+                if archive_type == ArchiveType::FixedIndex
+                    || archive_type == ArchiveType::DynamicIndex
+                {
+                    list.insert(path);
+                }
+            }
+        }
+
+        Ok(list)
+    }
+
+    // Similar to open index, but return with Ok(None) if index file vanished.
+    fn open_index_reader(&self, absolute_path: &Path) -> Result<Option<Box<dyn IndexFile>>, Error> {
+        let archive_type = match ArchiveType::from_path(absolute_path) {
+            // ignore archives with unknown archive type
+            Ok(ArchiveType::Blob) | Err(_) => bail!("unexpected archive type"),
+            Ok(archive_type) => archive_type,
+        };
+
+        if absolute_path.is_relative() {
+            bail!("expected absolute path, got '{absolute_path:?}'");
+        }
+
+        let file = match std::fs::File::open(absolute_path) {
+            Ok(file) => file,
+            // ignore vanished files
+            Err(err) if err.kind() == io::ErrorKind::NotFound => return Ok(None),
+            Err(err) => {
+                return Err(Error::from(err).context(format!("can't open file '{absolute_path:?}'")))
+            }
+        };
+
+        match archive_type {
+            ArchiveType::FixedIndex => {
+                let reader = FixedIndexReader::new(file)
+                    .with_context(|| format!("can't open fixed index '{absolute_path:?}'"))?;
+                Ok(Some(Box::new(reader)))
+            }
+            ArchiveType::DynamicIndex => {
+                let reader = DynamicIndexReader::new(file)
+                    .with_context(|| format!("can't open dynamic index '{absolute_path:?}'"))?;
+                Ok(Some(Box::new(reader)))
+            }
+            ArchiveType::Blob => bail!("unexpected archive type blob"),
+        }
+    }
+
+    pub fn last_gc_status(&self) -> GarbageCollectionStatus {
+        self.inner.last_gc_status.lock().unwrap().clone()
+    }
+
+    pub fn garbage_collection_running(&self) -> bool {
+        self.inner.gc_mutex.try_lock().is_err()
+    }
+
+    pub fn try_shared_chunk_store_lock(&self) -> Result<ProcessLockSharedGuard, Error> {
+        self.inner.chunk_store.try_shared_lock()
+    }
+
+    pub fn stat_chunk(&self, digest: &[u8; 32]) -> Result<std::fs::Metadata, Error> {
+        let (chunk_path, _digest_str) = self.inner.chunk_store.chunk_path(digest);
+        std::fs::metadata(chunk_path).map_err(Error::from)
+    }
+
+    pub fn load_chunk(&self, digest: &[u8; 32]) -> Result<DataBlob, Error> {
+        let (chunk_path, digest_str) = self.inner.chunk_store.chunk_path(digest);
+
+        proxmox_lang::try_block!({
+            let mut file = std::fs::File::open(&chunk_path)?;
+            DataBlob::load_from_reader(&mut file)
+        })
+        .map_err(|err| {
+            format_err!(
+                "store '{}', unable to load chunk '{}' - {}",
+                self.name(),
+                digest_str,
+                err,
+            )
+        })
+    }
+
+    /// returns a list of chunks sorted by their inode number on disk chunks that couldn't get
+    /// stat'ed are placed at the end of the list
+    pub fn get_chunks_in_order<F, A>(
+        &self,
+        index: &(dyn IndexFile + Send),
+        skip_chunk: F,
+        check_abort: A,
+    ) -> Result<Vec<(usize, u64)>, Error>
+    where
+        F: Fn(&[u8; 32]) -> bool,
+        A: Fn(usize) -> Result<(), Error>,
+    {
+        let index_count = index.index_count();
+        let mut chunk_list = Vec::with_capacity(index_count);
+        use std::os::unix::fs::MetadataExt;
+        for pos in 0..index_count {
+            check_abort(pos)?;
+
+            let info = index.chunk_info(pos).unwrap();
+
+            if skip_chunk(&info.digest) {
+                continue;
+            }
+
+            let ino = match self.inner.chunk_order {
+                ChunkOrder::Inode => {
+                    match self.stat_chunk(&info.digest) {
+                        Err(_) => u64::MAX, // could not stat, move to end of list
+                        Ok(metadata) => metadata.ino(),
+                    }
+                }
+                ChunkOrder::None => 0,
+            };
+
+            chunk_list.push((pos, ino));
+        }
+
+        match self.inner.chunk_order {
+            // sorting by inode improves data locality, which makes it lots faster on spinners
+            ChunkOrder::Inode => {
+                chunk_list.sort_unstable_by(|(_, ino_a), (_, ino_b)| ino_a.cmp(ino_b))
+            }
+            ChunkOrder::None => {}
+        }
+
+        Ok(chunk_list)
+    }
+
+    /// Open a snapshot (backup directory) from this datastore.
+    pub fn backup_dir_from_parts<D>(
+        self: &Arc<Self>,
+        ns: BackupNamespace,
+        ty: BackupType,
+        id: D,
+        time: i64,
+    ) -> Result<BackupDir, Error>
+    where
+        D: Into<String>,
+    {
+        self.backup_dir(ns, (ty, id.into(), time).into())
+    }
+
+    /// Open a snapshot (backup directory) from this datastore with a cached rfc3339 time string.
+    pub fn backup_dir_with_rfc3339<D: Into<String>>(
+        self: &Arc<Self>,
+        group: BackupGroup,
+        time_string: D,
+    ) -> Result<BackupDir, Error> {
+        BackupDir::with_rfc3339(group, time_string.into())
+    }
+
+    /// Open a backup group from this datastore.
+    pub fn backup_group_from_parts<D>(
+        self: &Arc<Self>,
+        ns: BackupNamespace,
+        ty: BackupType,
+        id: D,
+    ) -> BackupGroup
+    where
+        D: Into<String>,
+    {
+        self.backup_group(ns, (ty, id.into()).into())
+    }
+
+    /*
+    /// Open a backup group from this datastore by backup group path such as `vm/100`.
+    ///
+    /// Convenience method for `store.backup_group(path.parse()?)`
+    pub fn backup_group_from_path(self: &Arc<Self>, path: &str) -> Result<BackupGroup, Error> {
+        todo!("split out the namespace");
+    }
+    */
+
+    /// Open a backup group from this datastore.
+    pub fn backup_group(
+        self: &Arc<Self>,
+        ns: BackupNamespace,
+        group: pbs_api_types::BackupGroup,
+    ) -> BackupGroup<T> {
+        BackupGroup::new(Arc::clone(self), ns, group)
+    }
+
+    /// Open a snapshot (backup directory) from this datastore.
+    pub fn backup_dir(
+        self: &Arc<Self>,
+        ns: BackupNamespace,
+        dir: pbs_api_types::BackupDir,
+    ) -> Result<BackupDir, Error> {
+        BackupDir::with_group(self.backup_group(ns, dir.group), dir.time)
+    }
+}
+
+impl<T: CanWrite> DataStore<T> {
+    pub fn create_fixed_writer<P: AsRef<Path>>(
+        &self,
+        filename: P,
+        size: usize,
+        chunk_size: usize,
+    ) -> Result<FixedIndexWriter, Error> {
+        let index = FixedIndexWriter::create(
+            self.inner.chunk_store.clone(),
+            filename.as_ref(),
+            size,
+            chunk_size,
+        )?;
+
+        Ok(index)
+    }
+
+    pub fn create_dynamic_writer<P: AsRef<Path>>(
+        &self,
+        filename: P,
+    ) -> Result<DynamicIndexWriter, Error> {
+        let index = DynamicIndexWriter::create(self.inner.chunk_store.clone(), filename.as_ref())?;
+
+        Ok(index)
+    }
+
+    /// Create a backup namespace.
+    pub fn create_namespace(
+        self: &Arc<Self>,
+        parent: &BackupNamespace,
+        name: String,
+    ) -> Result<BackupNamespace, Error> {
+        if !self.namespace_exists(parent) {
+            bail!("cannot create new namespace, parent {parent} doesn't already exists");
+        }
+
+        // construct ns before mkdir to enforce max-depth and name validity
+        let ns = BackupNamespace::from_parent_ns(parent, name)?;
+
+        let mut ns_full_path = self.base_path();
+        ns_full_path.push(ns.path());
+
+        std::fs::create_dir_all(ns_full_path)?;
+
+        Ok(ns)
+    }
+
+    /// Remove all backup groups of a single namespace level but not the namespace itself.
     ///
     /// Does *not* descends into child-namespaces and doesn't remoes the namespace itself either.
     ///
@@ -719,85 +1041,30 @@ impl DataStore {
         Ok((removed_all_requested, stats))
     }
 
-    /// Remove a complete backup group including all snapshots.
-    ///
-    /// Returns `BackupGroupDeleteStats`, containing the number of deleted snapshots
-    /// and number of protected snaphsots, which therefore were not removed.
-    pub fn remove_backup_group(
-        self: &Arc<Self>,
-        ns: &BackupNamespace,
-        backup_group: &pbs_api_types::BackupGroup,
-    ) -> Result<BackupGroupDeleteStats, Error> {
-        let backup_group = self.backup_group(ns.clone(), backup_group.clone());
-
-        backup_group.destroy()
-    }
-
-    /// Remove a backup directory including all content
-    pub fn remove_backup_dir(
-        self: &Arc<Self>,
-        ns: &BackupNamespace,
-        backup_dir: &pbs_api_types::BackupDir,
-        force: bool,
-    ) -> Result<(), Error> {
-        let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
-
-        backup_dir.destroy(force)
-    }
-
-    /// Returns the time of the last successful backup
-    ///
-    /// Or None if there is no backup in the group (or the group dir does not exist).
-    pub fn last_successful_backup(
-        self: &Arc<Self>,
-        ns: &BackupNamespace,
-        backup_group: &pbs_api_types::BackupGroup,
-    ) -> Result<Option<i64>, Error> {
-        let backup_group = self.backup_group(ns.clone(), backup_group.clone());
-
-        let group_path = backup_group.full_group_path();
-
-        if group_path.exists() {
-            backup_group.last_successful_backup()
-        } else {
-            Ok(None)
-        }
-    }
-
-    /// Return the path of the 'owner' file.
-    pub(super) fn owner_path(
-        &self,
-        ns: &BackupNamespace,
-        group: &pbs_api_types::BackupGroup,
-    ) -> PathBuf {
-        self.group_path(ns, group).join("owner")
-    }
-
-    /// Returns the backup owner.
+    /// Remove a complete backup group including all snapshots.
     ///
-    /// The backup owner is the entity who first created the backup group.
-    pub fn get_owner(
-        &self,
+    /// Returns `BackupGroupDeleteStats`, containing the number of deleted snapshots
+    /// and number of protected snaphsots, which therefore were not removed.
+    pub fn remove_backup_group(
+        self: &Arc<Self>,
         ns: &BackupNamespace,
         backup_group: &pbs_api_types::BackupGroup,
-    ) -> Result<Authid, Error> {
-        let full_path = self.owner_path(ns, backup_group);
-        let owner = proxmox_sys::fs::file_read_firstline(full_path)?;
-        owner
-            .trim_end() // remove trailing newline
-            .parse()
-            .map_err(|err| format_err!("parsing owner for {backup_group} failed: {err}"))
+    ) -> Result<BackupGroupDeleteStats, Error> {
+        let backup_group = self.backup_group(ns.clone(), backup_group.clone());
+
+        backup_group.destroy()
     }
 
-    pub fn owns_backup(
-        &self,
+    /// Remove a backup directory including all content
+    pub fn remove_backup_dir(
+        self: &Arc<Self>,
         ns: &BackupNamespace,
-        backup_group: &pbs_api_types::BackupGroup,
-        auth_id: &Authid,
-    ) -> Result<bool, Error> {
-        let owner = self.get_owner(ns, backup_group)?;
+        backup_dir: &pbs_api_types::BackupDir,
+        force: bool,
+    ) -> Result<(), Error> {
+        let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
 
-        Ok(check_backup_owner(&owner, auth_id).is_ok())
+        backup_dir.destroy(force)
     }
 
     /// Set the backup owner.
@@ -900,229 +1167,6 @@ impl DataStore {
         }
     }
 
-    /// Get a streaming iter over single-level backup namespaces of a datatstore
-    ///
-    /// The iterated item is still a Result that can contain errors from rather unexptected FS or
-    /// parsing errors.
-    pub fn iter_backup_ns(
-        self: &Arc<DataStore>,
-        ns: BackupNamespace,
-    ) -> Result<ListNamespaces, Error> {
-        ListNamespaces::new(Arc::clone(self), ns)
-    }
-
-    /// Get a streaming iter over single-level backup namespaces of a datatstore, filtered by Ok
-    ///
-    /// The iterated item's result is already unwrapped, if it contained an error it will be
-    /// logged. Can be useful in iterator chain commands
-    pub fn iter_backup_ns_ok(
-        self: &Arc<DataStore>,
-        ns: BackupNamespace,
-    ) -> Result<impl Iterator<Item = BackupNamespace> + 'static, Error> {
-        let this = Arc::clone(self);
-        Ok(
-            ListNamespaces::new(Arc::clone(self), ns)?.filter_map(move |ns| match ns {
-                Ok(ns) => Some(ns),
-                Err(err) => {
-                    log::error!("list groups error on datastore {} - {}", this.name(), err);
-                    None
-                }
-            }),
-        )
-    }
-
-    /// Get a streaming iter over single-level backup namespaces of a datatstore
-    ///
-    /// The iterated item is still a Result that can contain errors from rather unexptected FS or
-    /// parsing errors.
-    pub fn recursive_iter_backup_ns(
-        self: &Arc<DataStore>,
-        ns: BackupNamespace,
-    ) -> Result<ListNamespacesRecursive, Error> {
-        ListNamespacesRecursive::new(Arc::clone(self), ns)
-    }
-
-    /// Get a streaming iter over single-level backup namespaces of a datatstore, filtered by Ok
-    ///
-    /// The iterated item's result is already unwrapped, if it contained an error it will be
-    /// logged. Can be useful in iterator chain commands
-    pub fn recursive_iter_backup_ns_ok(
-        self: &Arc<DataStore>,
-        ns: BackupNamespace,
-        max_depth: Option<usize>,
-    ) -> Result<impl Iterator<Item = BackupNamespace> + 'static, Error> {
-        let this = Arc::clone(self);
-        Ok(if let Some(depth) = max_depth {
-            ListNamespacesRecursive::new_max_depth(Arc::clone(self), ns, depth)?
-        } else {
-            ListNamespacesRecursive::new(Arc::clone(self), ns)?
-        }
-        .filter_map(move |ns| match ns {
-            Ok(ns) => Some(ns),
-            Err(err) => {
-                log::error!("list groups error on datastore {} - {}", this.name(), err);
-                None
-            }
-        }))
-    }
-
-    /// Get a streaming iter over top-level backup groups of a datatstore of a particular type.
-    ///
-    /// The iterated item is still a Result that can contain errors from rather unexptected FS or
-    /// parsing errors.
-    pub fn iter_backup_type(
-        self: &Arc<DataStore>,
-        ns: BackupNamespace,
-        ty: BackupType,
-    ) -> Result<ListGroupsType, Error> {
-        ListGroupsType::new(Arc::clone(self), ns, ty)
-    }
-
-    /// Get a streaming iter over top-level backup groups of a datastore of a particular type,
-    /// filtered by `Ok` results
-    ///
-    /// The iterated item's result is already unwrapped, if it contained an error it will be
-    /// logged. Can be useful in iterator chain commands
-    pub fn iter_backup_type_ok(
-        self: &Arc<DataStore>,
-        ns: BackupNamespace,
-        ty: BackupType,
-    ) -> Result<impl Iterator<Item = BackupGroup> + 'static, Error> {
-        Ok(self.iter_backup_type(ns, ty)?.ok())
-    }
-
-    /// Get a streaming iter over top-level backup groups of a datatstore
-    ///
-    /// The iterated item is still a Result that can contain errors from rather unexptected FS or
-    /// parsing errors.
-    pub fn iter_backup_groups(
-        self: &Arc<DataStore>,
-        ns: BackupNamespace,
-    ) -> Result<ListGroups, Error> {
-        ListGroups::new(Arc::clone(self), ns)
-    }
-
-    /// Get a streaming iter over top-level backup groups of a datatstore, filtered by Ok results
-    ///
-    /// The iterated item's result is already unwrapped, if it contained an error it will be
-    /// logged. Can be useful in iterator chain commands
-    pub fn iter_backup_groups_ok(
-        self: &Arc<DataStore>,
-        ns: BackupNamespace,
-    ) -> Result<impl Iterator<Item = BackupGroup> + 'static, Error> {
-        Ok(self.iter_backup_groups(ns)?.ok())
-    }
-
-    /// Get a in-memory vector for all top-level backup groups of a datatstore
-    ///
-    /// NOTE: using the iterator directly is most often more efficient w.r.t. memory usage
-    pub fn list_backup_groups(
-        self: &Arc<DataStore>,
-        ns: BackupNamespace,
-    ) -> Result<Vec<BackupGroup>, Error> {
-        ListGroups::new(Arc::clone(self), ns)?.collect()
-    }
-
-    /// Lookup all index files to be found in the datastore without taking any logical iteration
-    /// into account.
-    /// The filesystem is walked recursevly to detect index files based on their archive type based
-    /// on the filename. This however excludes the chunks folder, hidden files and does not follow
-    /// symlinks.
-    fn list_index_files(&self) -> Result<HashSet<PathBuf>, Error> {
-        let base = self.base_path();
-
-        let mut list = HashSet::new();
-
-        use walkdir::WalkDir;
-
-        let walker = WalkDir::new(base).into_iter();
-
-        // make sure we skip .chunks (and other hidden files to keep it simple)
-        fn is_hidden(entry: &walkdir::DirEntry) -> bool {
-            entry
-                .file_name()
-                .to_str()
-                .map(|s| s.starts_with('.'))
-                .unwrap_or(false)
-        }
-        let handle_entry_err = |err: walkdir::Error| {
-            // first, extract the actual IO error and the affected path
-            let (inner, path) = match (err.io_error(), err.path()) {
-                (None, _) => return Ok(()), // not an IO-error
-                (Some(inner), Some(path)) => (inner, path),
-                (Some(inner), None) => bail!("unexpected error on datastore traversal: {inner}"),
-            };
-            if inner.kind() == io::ErrorKind::PermissionDenied {
-                if err.depth() <= 1 && path.ends_with("lost+found") {
-                    // allow skipping of (root-only) ext4 fsck-directory on EPERM ..
-                    return Ok(());
-                }
-                // .. but do not ignore EPERM in general, otherwise we might prune too many chunks.
-                // E.g., if users messed up with owner/perms on a rsync
-                bail!("cannot continue garbage-collection safely, permission denied on: {path:?}");
-            } else if inner.kind() == io::ErrorKind::NotFound {
-                log::info!("ignoring vanished file: {path:?}");
-                return Ok(());
-            } else {
-                bail!("unexpected error on datastore traversal: {inner} - {path:?}");
-            }
-        };
-        for entry in walker.filter_entry(|e| !is_hidden(e)) {
-            let path = match entry {
-                Ok(entry) => entry.into_path(),
-                Err(err) => {
-                    handle_entry_err(err)?;
-                    continue;
-                }
-            };
-            if let Ok(archive_type) = ArchiveType::from_path(&path) {
-                if archive_type == ArchiveType::FixedIndex
-                    || archive_type == ArchiveType::DynamicIndex
-                {
-                    list.insert(path);
-                }
-            }
-        }
-
-        Ok(list)
-    }
-
-    // Similar to open index, but return with Ok(None) if index file vanished.
-    fn open_index_reader(&self, absolute_path: &Path) -> Result<Option<Box<dyn IndexFile>>, Error> {
-        let archive_type = match ArchiveType::from_path(absolute_path) {
-            // ignore archives with unknown archive type
-            Ok(ArchiveType::Blob) | Err(_) => bail!("unexpected archive type"),
-            Ok(archive_type) => archive_type,
-        };
-
-        if absolute_path.is_relative() {
-            bail!("expected absolute path, got '{absolute_path:?}'");
-        }
-
-        let file = match std::fs::File::open(absolute_path) {
-            Ok(file) => file,
-            // ignore vanished files
-            Err(err) if err.kind() == io::ErrorKind::NotFound => return Ok(None),
-            Err(err) => {
-                return Err(Error::from(err).context(format!("can't open file '{absolute_path:?}'")))
-            }
-        };
-
-        match archive_type {
-            ArchiveType::FixedIndex => {
-                let reader = FixedIndexReader::new(file)
-                    .with_context(|| format!("can't open fixed index '{absolute_path:?}'"))?;
-                Ok(Some(Box::new(reader)))
-            }
-            ArchiveType::DynamicIndex => {
-                let reader = DynamicIndexReader::new(file)
-                    .with_context(|| format!("can't open dynamic index '{absolute_path:?}'"))?;
-                Ok(Some(Box::new(reader)))
-            }
-            ArchiveType::Blob => bail!("unexpected archive type blob"),
-        }
-    }
-
     // mark chunks  used by ``index`` as used
     fn index_mark_used_chunks(
         &self,
@@ -1301,15 +1345,7 @@ impl DataStore {
             warn!("Found {strange_paths_count} index files outside of expected directory scheme");
         }
 
-        Ok(())
-    }
-
-    pub fn last_gc_status(&self) -> GarbageCollectionStatus {
-        self.inner.last_gc_status.lock().unwrap().clone()
-    }
-
-    pub fn garbage_collection_running(&self) -> bool {
-        self.inner.gc_mutex.try_lock().is_err()
+        Ok(())
     }
 
     pub fn garbage_collection(
@@ -1479,14 +1515,6 @@ impl DataStore {
         Ok(())
     }
 
-    pub fn try_shared_chunk_store_lock(&self) -> Result<ProcessLockSharedGuard, Error> {
-        self.inner.chunk_store.try_shared_lock()
-    }
-
-    pub fn chunk_path(&self, digest: &[u8; 32]) -> (PathBuf, String) {
-        self.inner.chunk_store.chunk_path(digest)
-    }
-
     pub fn cond_touch_chunk(&self, digest: &[u8; 32], assert_exists: bool) -> Result<bool, Error> {
         self.inner
             .chunk_store
@@ -1497,28 +1525,6 @@ impl DataStore {
         self.inner.chunk_store.insert_chunk(chunk, digest)
     }
 
-    pub fn stat_chunk(&self, digest: &[u8; 32]) -> Result<std::fs::Metadata, Error> {
-        let (chunk_path, _digest_str) = self.inner.chunk_store.chunk_path(digest);
-        std::fs::metadata(chunk_path).map_err(Error::from)
-    }
-
-    pub fn load_chunk(&self, digest: &[u8; 32]) -> Result<DataBlob, Error> {
-        let (chunk_path, digest_str) = self.inner.chunk_store.chunk_path(digest);
-
-        proxmox_lang::try_block!({
-            let mut file = std::fs::File::open(&chunk_path)?;
-            DataBlob::load_from_reader(&mut file)
-        })
-        .map_err(|err| {
-            format_err!(
-                "store '{}', unable to load chunk '{}' - {}",
-                self.name(),
-                digest_str,
-                err,
-            )
-        })
-    }
-
     /// Updates the protection status of the specified snapshot.
     pub fn update_protection(&self, backup_dir: &BackupDir, protection: bool) -> Result<(), Error> {
         let full_path = backup_dir.full_path();
@@ -1545,128 +1551,6 @@ impl DataStore {
         Ok(())
     }
 
-    pub fn verify_new(&self) -> bool {
-        self.inner.verify_new
-    }
-
-    /// returns a list of chunks sorted by their inode number on disk chunks that couldn't get
-    /// stat'ed are placed at the end of the list
-    pub fn get_chunks_in_order<F, A>(
-        &self,
-        index: &(dyn IndexFile + Send),
-        skip_chunk: F,
-        check_abort: A,
-    ) -> Result<Vec<(usize, u64)>, Error>
-    where
-        F: Fn(&[u8; 32]) -> bool,
-        A: Fn(usize) -> Result<(), Error>,
-    {
-        let index_count = index.index_count();
-        let mut chunk_list = Vec::with_capacity(index_count);
-        use std::os::unix::fs::MetadataExt;
-        for pos in 0..index_count {
-            check_abort(pos)?;
-
-            let info = index.chunk_info(pos).unwrap();
-
-            if skip_chunk(&info.digest) {
-                continue;
-            }
-
-            let ino = match self.inner.chunk_order {
-                ChunkOrder::Inode => {
-                    match self.stat_chunk(&info.digest) {
-                        Err(_) => u64::MAX, // could not stat, move to end of list
-                        Ok(metadata) => metadata.ino(),
-                    }
-                }
-                ChunkOrder::None => 0,
-            };
-
-            chunk_list.push((pos, ino));
-        }
-
-        match self.inner.chunk_order {
-            // sorting by inode improves data locality, which makes it lots faster on spinners
-            ChunkOrder::Inode => {
-                chunk_list.sort_unstable_by(|(_, ino_a), (_, ino_b)| ino_a.cmp(ino_b))
-            }
-            ChunkOrder::None => {}
-        }
-
-        Ok(chunk_list)
-    }
-
-    /// Open a backup group from this datastore.
-    pub fn backup_group(
-        self: &Arc<Self>,
-        ns: BackupNamespace,
-        group: pbs_api_types::BackupGroup,
-    ) -> BackupGroup {
-        BackupGroup::new(Arc::clone(self), ns, group)
-    }
-
-    /// Open a backup group from this datastore.
-    pub fn backup_group_from_parts<T>(
-        self: &Arc<Self>,
-        ns: BackupNamespace,
-        ty: BackupType,
-        id: T,
-    ) -> BackupGroup
-    where
-        T: Into<String>,
-    {
-        self.backup_group(ns, (ty, id.into()).into())
-    }
-
-    /*
-    /// Open a backup group from this datastore by backup group path such as `vm/100`.
-    ///
-    /// Convenience method for `store.backup_group(path.parse()?)`
-    pub fn backup_group_from_path(self: &Arc<Self>, path: &str) -> Result<BackupGroup, Error> {
-        todo!("split out the namespace");
-    }
-    */
-
-    /// Open a snapshot (backup directory) from this datastore.
-    pub fn backup_dir(
-        self: &Arc<Self>,
-        ns: BackupNamespace,
-        dir: pbs_api_types::BackupDir,
-    ) -> Result<BackupDir, Error> {
-        BackupDir::with_group(self.backup_group(ns, dir.group), dir.time)
-    }
-
-    /// Open a snapshot (backup directory) from this datastore.
-    pub fn backup_dir_from_parts<T>(
-        self: &Arc<Self>,
-        ns: BackupNamespace,
-        ty: BackupType,
-        id: T,
-        time: i64,
-    ) -> Result<BackupDir, Error>
-    where
-        T: Into<String>,
-    {
-        self.backup_dir(ns, (ty, id.into(), time).into())
-    }
-
-    /// Open a snapshot (backup directory) from this datastore with a cached rfc3339 time string.
-    pub fn backup_dir_with_rfc3339<T: Into<String>>(
-        self: &Arc<Self>,
-        group: BackupGroup,
-        time_string: T,
-    ) -> Result<BackupDir, Error> {
-        BackupDir::with_rfc3339(group, time_string.into())
-    }
-
-    /*
-    /// Open a snapshot (backup directory) from this datastore by a snapshot path.
-    pub fn backup_dir_from_path(self: &Arc<Self>, path: &str) -> Result<BackupDir, Error> {
-        todo!("split out the namespace");
-    }
-    */
-
     /// Syncs the filesystem of the datastore if 'sync_level' is set to
     /// [`DatastoreFSyncLevel::Filesystem`]. Uses syncfs(2).
     pub fn try_ensure_sync_level(&self) -> Result<(), Error> {
@@ -1786,6 +1670,126 @@ impl DataStore {
 
         Ok(())
     }
+}
+
+impl<T> DataStore<T> {
+    #[doc(hidden)]
+    pub(crate) fn new_test() -> Arc<Self> {
+        Arc::new(Self {
+            inner: DataStoreImpl::new_test(),
+            operation: None,
+        })
+    }
+
+    pub fn read_config(name: &str) -> Result<(DataStoreConfig, [u8; 32], BackupLockGuard), Error> {
+        let lock = pbs_config::datastore::lock_config()?;
+
+        let (config, digest) = pbs_config::datastore::config()?;
+        let config: DataStoreConfig = config.lookup("datastore", name)?;
+        Ok((config, digest, lock))
+    }
+
+    /// removes all datastores that are not configured anymore
+    pub fn remove_unused_datastores() -> Result<(), Error> {
+        let (config, _digest) = pbs_config::datastore::config()?;
+
+        let mut map_read = DATASTORE_MAP_READ.lock().unwrap();
+        let mut map_write = DATASTORE_MAP_READ.lock().unwrap();
+        // removes all elements that are not in the config
+        map_read.retain(|key, _| config.sections.contains_key(key));
+        map_write.retain(|key, _| config.sections.contains_key(key));
+        Ok(())
+    }
+
+    /// trigger clearing cache entry based on maintenance mode. Entry will only
+    /// be cleared iff there is no other task running, if there is, the end of the
+    /// last running task will trigger the clearing of the cache entry.
+    pub fn update_datastore_cache(name: &str) -> Result<(), Error> {
+        let (config, _digest) = pbs_config::datastore::config()?;
+        let datastore: DataStoreConfig = config.lookup("datastore", name)?;
+        if datastore
+            .get_maintenance_mode()
+            .is_some_and(|m| m.clear_from_cache())
+        {
+            // the datastore drop handler does the checking if tasks are running and clears the
+            // cache entry, so we just have to trigger it here
+            let _ = DataStore::<L>::lookup_datastore(name);
+        }
+
+        Ok(())
+    }
+
+    pub fn name(&self) -> &str {
+        self.inner.chunk_store.name()
+    }
+
+    pub fn base_path(&self) -> PathBuf {
+        self.inner.chunk_store.base_path()
+    }
+
+    /// Returns the absolute path for a backup namespace on this datastore
+    pub fn namespace_path(&self, ns: &BackupNamespace) -> PathBuf {
+        let mut path = self.base_path();
+        path.reserve(ns.path_len());
+        for part in ns.components() {
+            path.push("ns");
+            path.push(part);
+        }
+        path
+    }
+
+    /// Returns the absolute path for a backup_type
+    pub fn type_path(&self, ns: &BackupNamespace, backup_type: BackupType) -> PathBuf {
+        let mut full_path = self.namespace_path(ns);
+        full_path.push(backup_type.to_string());
+        full_path
+    }
+
+    /// Returns the absolute path for a backup_group
+    pub fn group_path(
+        &self,
+        ns: &BackupNamespace,
+        backup_group: &pbs_api_types::BackupGroup,
+    ) -> PathBuf {
+        let mut full_path = self.namespace_path(ns);
+        full_path.push(backup_group.to_string());
+        full_path
+    }
+
+    /// Returns the absolute path for backup_dir
+    pub fn snapshot_path(
+        &self,
+        ns: &BackupNamespace,
+        backup_dir: &pbs_api_types::BackupDir,
+    ) -> PathBuf {
+        let mut full_path = self.namespace_path(ns);
+        full_path.push(backup_dir.to_string());
+        full_path
+    }
+
+    /// Return the path of the 'owner' file.
+    pub(super) fn owner_path(
+        &self,
+        ns: &BackupNamespace,
+        group: &pbs_api_types::BackupGroup,
+    ) -> PathBuf {
+        self.group_path(ns, group).join("owner")
+    }
+
+    pub fn chunk_path(&self, digest: &[u8; 32]) -> (PathBuf, String) {
+        self.inner.chunk_store.chunk_path(digest)
+    }
+
+    pub fn verify_new(&self) -> bool {
+        self.inner.verify_new
+    }
+
+    /*
+    /// Open a snapshot (backup directory) from this datastore by a snapshot path.
+    pub fn backup_dir_from_path(self: &Arc<Self>, path: &str) -> Result<BackupDir, Error> {
+        todo!("split out the namespace");
+    }
+    */
 
     pub fn old_locking(&self) -> bool {
         *OLD_LOCKING
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:14:41 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:14:41 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 08/12]
 api/backup/bin/server/tape: add missing generics
In-Reply-To: <20250526141445.228717-1-h.laimer@proxmox.com>
References: <20250526141445.228717-1-h.laimer@proxmox.com>
Message-ID: <20250526141445.228717-9-h.laimer@proxmox.com>

Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
---
 src/api2/admin/datastore.rs                 | 27 ++++----
 src/api2/backup/mod.rs                      | 21 +++---
 src/api2/backup/upload_chunk.rs             | 19 +++---
 src/api2/config/datastore.rs                |  5 +-
 src/api2/reader/environment.rs              | 30 +++++----
 src/api2/reader/mod.rs                      |  5 +-
 src/api2/tape/backup.rs                     | 11 ++--
 src/api2/tape/drive.rs                      |  3 +-
 src/api2/tape/restore.rs                    | 71 +++++++++++----------
 src/backup/hierarchy.rs                     | 23 +++----
 src/backup/verify.rs                        | 53 +++++++--------
 src/bin/proxmox-backup-proxy.rs             |  7 +-
 src/server/gc_job.rs                        |  7 +-
 src/server/prune_job.rs                     |  5 +-
 src/server/pull.rs                          | 23 +++----
 src/server/push.rs                          |  3 +-
 src/server/sync.rs                          | 13 ++--
 src/tape/file_formats/snapshot_archive.rs   |  5 +-
 src/tape/pool_writer/mod.rs                 | 11 ++--
 src/tape/pool_writer/new_chunks_iterator.rs |  7 +-
 20 files changed, 189 insertions(+), 160 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 39249448..e3f93cdd 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -54,6 +54,7 @@ use pbs_config::CachedUserInfo;
 use pbs_datastore::backup_info::BackupInfo;
 use pbs_datastore::cached_chunk_reader::CachedChunkReader;
 use pbs_datastore::catalog::{ArchiveEntry, CatalogReader};
+use pbs_datastore::chunk_store::{CanRead, Read as R};
 use pbs_datastore::data_blob::DataBlob;
 use pbs_datastore::data_blob_reader::DataBlobReader;
 use pbs_datastore::dynamic_index::{BufferedDynamicReader, DynamicIndexReader, LocalDynamicReadAt};
@@ -79,8 +80,8 @@ use crate::server::jobstate::{compute_schedule_status, Job, JobState};
 
 const GROUP_NOTES_FILE_NAME: &str = "notes";
 
-fn get_group_note_path(
-    store: &DataStore,
+fn get_group_note_path<T>(
+    store: &DataStore<T>,
     ns: &BackupNamespace,
     group: &pbs_api_types::BackupGroup,
 ) -> PathBuf {
@@ -114,8 +115,8 @@ fn check_privs_and_load_store(
     Ok(datastore)
 }
 
-fn read_backup_index(
-    backup_dir: &BackupDir,
+fn read_backup_index<T: CanRead>(
+    backup_dir: &BackupDir<T>,
 ) -> Result<(BackupManifest, Vec<BackupContent>), Error> {
     let (manifest, index_size) = backup_dir.load_manifest()?;
 
@@ -140,8 +141,8 @@ fn read_backup_index(
     Ok((manifest, result))
 }
 
-fn get_all_snapshot_files(
-    info: &BackupInfo,
+fn get_all_snapshot_files<T: CanRead>(
+    info: &BackupInfo<T>,
 ) -> Result<(BackupManifest, Vec<BackupContent>), Error> {
     let (manifest, mut files) = read_backup_index(&info.backup_dir)?;
 
@@ -529,7 +530,7 @@ unsafe fn list_snapshots_blocking(
         (None, None) => datastore.list_backup_groups(ns.clone())?,
     };
 
-    let info_to_snapshot_list_item = |group: &BackupGroup, owner, info: BackupInfo| {
+    let info_to_snapshot_list_item = |group: &BackupGroup<R>, owner, info: BackupInfo<R>| {
         let backup = pbs_api_types::BackupDir {
             group: group.into(),
             time: info.backup_dir.backup_time(),
@@ -629,8 +630,8 @@ unsafe fn list_snapshots_blocking(
     })
 }
 
-async fn get_snapshots_count(
-    store: &Arc<DataStore>,
+async fn get_snapshots_count<T: CanRead + Send + Sync + 'static>(
+    store: &Arc<DataStore<T>>,
     owner: Option<&Authid>,
 ) -> Result<Counts, Error> {
     let store = Arc::clone(store);
@@ -1796,12 +1797,12 @@ pub const API_METHOD_PXAR_FILE_DOWNLOAD: ApiMethod = ApiMethod::new(
     &Permission::Anybody,
 );
 
-fn get_local_pxar_reader(
-    datastore: Arc<DataStore>,
+fn get_local_pxar_reader<T: CanRead>(
+    datastore: Arc<DataStore<T>>,
     manifest: &BackupManifest,
-    backup_dir: &BackupDir,
+    backup_dir: &BackupDir<T>,
     pxar_name: &BackupArchiveName,
-) -> Result<(LocalDynamicReadAt<LocalChunkReader>, u64), Error> {
+) -> Result<(LocalDynamicReadAt<LocalChunkReader<T>>, u64), Error> {
     let mut path = datastore.base_path();
     path.push(backup_dir.relative_path());
     path.push(pxar_name.as_ref());
diff --git a/src/api2/backup/mod.rs b/src/api2/backup/mod.rs
index 629df933..79354dbf 100644
--- a/src/api2/backup/mod.rs
+++ b/src/api2/backup/mod.rs
@@ -24,6 +24,7 @@ use pbs_api_types::{
     BACKUP_TYPE_SCHEMA, CHUNK_DIGEST_SCHEMA, DATASTORE_SCHEMA, PRIV_DATASTORE_BACKUP,
 };
 use pbs_config::CachedUserInfo;
+use pbs_datastore::chunk_store::{Read as R, Write as W};
 use pbs_datastore::index::IndexFile;
 use pbs_datastore::{DataStore, PROXMOX_BACKUP_PROTOCOL_ID_V1};
 use pbs_tools::json::{required_array_param, required_integer_param, required_string_param};
@@ -279,7 +280,7 @@ fn upgrade_to_backup_protocol(
                         return Ok(());
                     }
 
-                    let verify = |env: BackupEnvironment| {
+                    let verify = |env: BackupEnvironment<W>| {
                         if let Err(err) = env.verify_after_complete(snap_guard) {
                             env.log(format!(
                                 "backup finished, but starting the requested verify task failed: {}",
@@ -400,7 +401,7 @@ fn create_dynamic_index(
     _info: &ApiMethod,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Value, Error> {
-    let env: &BackupEnvironment = rpcenv.as_ref();
+    let env: &BackupEnvironment<W> = rpcenv.as_ref();
 
     let name = required_string_param(&param, "archive-name")?.to_owned();
 
@@ -450,7 +451,7 @@ fn create_fixed_index(
     _info: &ApiMethod,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Value, Error> {
-    let env: &BackupEnvironment = rpcenv.as_ref();
+    let env: &BackupEnvironment<W> = rpcenv.as_ref();
 
     let name = required_string_param(&param, "archive-name")?.to_owned();
     let size = required_integer_param(&param, "size")? as usize;
@@ -565,7 +566,7 @@ fn dynamic_append(
         );
     }
 
-    let env: &BackupEnvironment = rpcenv.as_ref();
+    let env: &BackupEnvironment<W> = rpcenv.as_ref();
 
     env.debug(format!("dynamic_append {} chunks", digest_list.len()));
 
@@ -639,7 +640,7 @@ fn fixed_append(
         );
     }
 
-    let env: &BackupEnvironment = rpcenv.as_ref();
+    let env: &BackupEnvironment<W> = rpcenv.as_ref();
 
     env.debug(format!("fixed_append {} chunks", digest_list.len()));
 
@@ -714,7 +715,7 @@ fn close_dynamic_index(
     let csum_str = required_string_param(&param, "csum")?;
     let csum = <[u8; 32]>::from_hex(csum_str)?;
 
-    let env: &BackupEnvironment = rpcenv.as_ref();
+    let env: &BackupEnvironment<W> = rpcenv.as_ref();
 
     env.dynamic_writer_close(wid, chunk_count, size, csum)?;
 
@@ -767,7 +768,7 @@ fn close_fixed_index(
     let csum_str = required_string_param(&param, "csum")?;
     let csum = <[u8; 32]>::from_hex(csum_str)?;
 
-    let env: &BackupEnvironment = rpcenv.as_ref();
+    let env: &BackupEnvironment<W> = rpcenv.as_ref();
 
     env.fixed_writer_close(wid, chunk_count, size, csum)?;
 
@@ -781,7 +782,7 @@ fn finish_backup(
     _info: &ApiMethod,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Value, Error> {
-    let env: &BackupEnvironment = rpcenv.as_ref();
+    let env: &BackupEnvironment<W> = rpcenv.as_ref();
 
     env.finish_backup()?;
     env.log("successfully finished backup");
@@ -800,7 +801,7 @@ fn get_previous_backup_time(
     _info: &ApiMethod,
     rpcenv: &mut dyn RpcEnvironment,
 ) -> Result<Value, Error> {
-    let env: &BackupEnvironment = rpcenv.as_ref();
+    let env: &BackupEnvironment<R> = rpcenv.as_ref();
 
     let backup_time = env
         .last_backup
@@ -827,7 +828,7 @@ fn download_previous(
     rpcenv: Box<dyn RpcEnvironment>,
 ) -> ApiResponseFuture {
     async move {
-        let env: &BackupEnvironment = rpcenv.as_ref();
+        let env: &BackupEnvironment<R> = rpcenv.as_ref();
 
         let archive_name = required_string_param(&param, "archive-name")?.to_owned();
 
diff --git a/src/api2/backup/upload_chunk.rs b/src/api2/backup/upload_chunk.rs
index 20259660..bb1566ae 100644
--- a/src/api2/backup/upload_chunk.rs
+++ b/src/api2/backup/upload_chunk.rs
@@ -14,25 +14,26 @@ use proxmox_schema::*;
 use proxmox_sortable_macro::sortable;
 
 use pbs_api_types::{BACKUP_ARCHIVE_NAME_SCHEMA, CHUNK_DIGEST_SCHEMA};
+use pbs_datastore::chunk_store::{CanWrite, Lookup as L, Write as W};
 use pbs_datastore::file_formats::{DataBlobHeader, EncryptedDataBlobHeader};
 use pbs_datastore::{DataBlob, DataStore};
 use pbs_tools::json::{required_integer_param, required_string_param};
 
 use super::environment::*;
 
-pub struct UploadChunk {
+pub struct UploadChunk<T> {
     stream: Body,
-    store: Arc<DataStore>,
+    store: Arc<DataStore<T>>,
     digest: [u8; 32],
     size: u32,
     encoded_size: u32,
     raw_data: Option<Vec<u8>>,
 }
 
-impl UploadChunk {
+impl<T> UploadChunk<T> {
     pub fn new(
         stream: Body,
-        store: Arc<DataStore>,
+        store: Arc<DataStore<T>>,
         digest: [u8; 32],
         size: u32,
         encoded_size: u32,
@@ -48,7 +49,7 @@ impl UploadChunk {
     }
 }
 
-impl Future for UploadChunk {
+impl<T: CanWrite> Future for UploadChunk<T> {
     type Output = Result<([u8; 32], u32, u32, bool), Error>;
 
     fn poll(self: Pin<&mut Self>, cx: &mut Context) -> Poll<Self::Output> {
@@ -159,7 +160,7 @@ fn upload_fixed_chunk(
         let digest_str = required_string_param(&param, "digest")?;
         let digest = <[u8; 32]>::from_hex(digest_str)?;
 
-        let env: &BackupEnvironment = rpcenv.as_ref();
+        let env: &BackupEnvironment<W> = rpcenv.as_ref();
 
         let (digest, size, compressed_size, is_duplicate) =
             UploadChunk::new(req_body, env.datastore.clone(), digest, size, encoded_size).await?;
@@ -228,7 +229,7 @@ fn upload_dynamic_chunk(
         let digest_str = required_string_param(&param, "digest")?;
         let digest = <[u8; 32]>::from_hex(digest_str)?;
 
-        let env: &BackupEnvironment = rpcenv.as_ref();
+        let env: &BackupEnvironment<W> = rpcenv.as_ref();
 
         let (digest, size, compressed_size, is_duplicate) =
             UploadChunk::new(req_body, env.datastore.clone(), digest, size, encoded_size).await?;
@@ -273,7 +274,7 @@ fn upload_speedtest(
                 println!("Upload error: {}", err);
             }
         }
-        let env: &BackupEnvironment = rpcenv.as_ref();
+        let env: &BackupEnvironment<L> = rpcenv.as_ref();
         Ok(env.format_response(Ok(Value::Null)))
     }
     .boxed()
@@ -312,7 +313,7 @@ fn upload_blob(
         let file_name = required_string_param(&param, "file-name")?.to_owned();
         let encoded_size = required_integer_param(&param, "encoded-size")? as usize;
 
-        let env: &BackupEnvironment = rpcenv.as_ref();
+        let env: &BackupEnvironment<W> = rpcenv.as_ref();
 
         if !file_name.ends_with(".blob") {
             bail!("wrong blob file extension: '{}'", file_name);
diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index b133be70..52fa6db1 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -30,6 +30,7 @@ use crate::api2::config::tape_backup_job::{delete_tape_backup_job, list_tape_bac
 use crate::api2::config::verify::delete_verification_job;
 use pbs_config::CachedUserInfo;
 
+use pbs_datastore::chunk_store::{Read as R, Write as W};
 use pbs_datastore::get_datastore_mount_status;
 use proxmox_rest_server::WorkerTask;
 
@@ -124,7 +125,7 @@ pub(crate) fn do_create_datastore(
     };
 
     let chunk_store = if reuse_datastore {
-        ChunkStore::verify_chunkstore(&path).and_then(|_| {
+        ChunkStore::<R>::verify_chunkstore(&path).and_then(|_| {
             // Must be the only instance accessing and locking the chunk store,
             // dropping will close all other locks from this process on the lockfile as well.
             ChunkStore::open(
@@ -666,7 +667,7 @@ pub async fn delete_datastore(
         auth_id.to_string(),
         to_stdout,
         move |_worker| {
-            pbs_datastore::DataStore::destroy(&name, destroy_data)?;
+            pbs_datastore::DataStore::<W>::destroy(&name, destroy_data)?;
 
             // ignore errors
             let _ = jobstate::remove_state_file("prune", &name);
diff --git a/src/api2/reader/environment.rs b/src/api2/reader/environment.rs
index 3b2f06f4..26f5bec6 100644
--- a/src/api2/reader/environment.rs
+++ b/src/api2/reader/environment.rs
@@ -14,25 +14,25 @@ use tracing::info;
 
 /// `RpcEnvironment` implementation for backup reader service
 #[derive(Clone)]
-pub struct ReaderEnvironment {
+pub struct ReaderEnvironment<T> {
     env_type: RpcEnvironmentType,
     result_attributes: Value,
     auth_id: Authid,
     pub debug: bool,
     pub formatter: &'static dyn OutputFormatter,
     pub worker: Arc<WorkerTask>,
-    pub datastore: Arc<DataStore>,
-    pub backup_dir: BackupDir,
+    pub datastore: Arc<DataStore<T>>,
+    pub backup_dir: BackupDir<T>,
     allowed_chunks: Arc<RwLock<HashSet<[u8; 32]>>>,
 }
 
-impl ReaderEnvironment {
+impl<T> ReaderEnvironment<T> {
     pub fn new(
         env_type: RpcEnvironmentType,
         auth_id: Authid,
         worker: Arc<WorkerTask>,
-        datastore: Arc<DataStore>,
-        backup_dir: BackupDir,
+        datastore: Arc<DataStore<T>>,
+        backup_dir: BackupDir<T>,
     ) -> Self {
         Self {
             result_attributes: json!({}),
@@ -71,7 +71,7 @@ impl ReaderEnvironment {
     }
 }
 
-impl RpcEnvironment for ReaderEnvironment {
+impl<T: Send + Sync + 'static> RpcEnvironment for ReaderEnvironment<T> {
     fn result_attrib_mut(&mut self) -> &mut Value {
         &mut self.result_attributes
     }
@@ -93,14 +93,18 @@ impl RpcEnvironment for ReaderEnvironment {
     }
 }
 
-impl AsRef<ReaderEnvironment> for dyn RpcEnvironment {
-    fn as_ref(&self) -> &ReaderEnvironment {
-        self.as_any().downcast_ref::<ReaderEnvironment>().unwrap()
+impl<T: 'static> AsRef<ReaderEnvironment<T>> for dyn RpcEnvironment {
+    fn as_ref(&self) -> &ReaderEnvironment<T> {
+        self.as_any()
+            .downcast_ref::<ReaderEnvironment<T>>()
+            .unwrap()
     }
 }
 
-impl AsRef<ReaderEnvironment> for Box<dyn RpcEnvironment> {
-    fn as_ref(&self) -> &ReaderEnvironment {
-        self.as_any().downcast_ref::<ReaderEnvironment>().unwrap()
+impl<T: 'static> AsRef<ReaderEnvironment<T>> for Box<dyn RpcEnvironment> {
+    fn as_ref(&self) -> &ReaderEnvironment<T> {
+        self.as_any()
+            .downcast_ref::<ReaderEnvironment<T>>()
+            .unwrap()
     }
 }
diff --git a/src/api2/reader/mod.rs b/src/api2/reader/mod.rs
index cc791299..52f0953a 100644
--- a/src/api2/reader/mod.rs
+++ b/src/api2/reader/mod.rs
@@ -23,6 +23,7 @@ use pbs_api_types::{
     DATASTORE_SCHEMA, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_READ,
 };
 use pbs_config::CachedUserInfo;
+use pbs_datastore::chunk_store::Read as R;
 use pbs_datastore::index::IndexFile;
 use pbs_datastore::{DataStore, PROXMOX_BACKUP_READER_PROTOCOL_ID_V1};
 use pbs_tools::json::required_string_param;
@@ -247,7 +248,7 @@ fn download_file(
     rpcenv: Box<dyn RpcEnvironment>,
 ) -> ApiResponseFuture {
     async move {
-        let env: &ReaderEnvironment = rpcenv.as_ref();
+        let env: &ReaderEnvironment<R> = rpcenv.as_ref();
 
         let file_name = required_string_param(&param, "file-name")?.to_owned();
 
@@ -303,7 +304,7 @@ fn download_chunk(
     rpcenv: Box<dyn RpcEnvironment>,
 ) -> ApiResponseFuture {
     async move {
-        let env: &ReaderEnvironment = rpcenv.as_ref();
+        let env: &ReaderEnvironment<R> = rpcenv.as_ref();
 
         let digest_str = required_string_param(&param, "digest")?;
         let digest = <[u8; 32]>::from_hex(digest_str)?;
diff --git a/src/api2/tape/backup.rs b/src/api2/tape/backup.rs
index 31293a9a..306d5936 100644
--- a/src/api2/tape/backup.rs
+++ b/src/api2/tape/backup.rs
@@ -18,6 +18,7 @@ use pbs_api_types::{
 
 use pbs_config::CachedUserInfo;
 use pbs_datastore::backup_info::{BackupDir, BackupInfo};
+use pbs_datastore::chunk_store::CanRead;
 use pbs_datastore::{DataStore, StoreProgress};
 
 use crate::tape::TapeNotificationMode;
@@ -360,9 +361,9 @@ enum SnapshotBackupResult {
     Ignored,
 }
 
-fn backup_worker(
+fn backup_worker<T: CanRead + Send + Sync + 'static>(
     worker: &WorkerTask,
-    datastore: Arc<DataStore>,
+    datastore: Arc<DataStore<T>>,
     pool_config: &MediaPoolConfig,
     setup: &TapeBackupJobSetup,
     summary: &mut TapeBackupJobSummary,
@@ -564,11 +565,11 @@ fn update_media_online_status(drive: &str) -> Result<Option<String>, Error> {
     }
 }
 
-fn backup_snapshot(
+fn backup_snapshot<T: CanRead + Send + Sync + 'static>(
     worker: &WorkerTask,
     pool_writer: &mut PoolWriter,
-    datastore: Arc<DataStore>,
-    snapshot: BackupDir,
+    datastore: Arc<DataStore<T>>,
+    snapshot: BackupDir<T>,
 ) -> Result<SnapshotBackupResult, Error> {
     let snapshot_path = snapshot.relative_path();
     info!("backup snapshot {snapshot_path:?}");
diff --git a/src/api2/tape/drive.rs b/src/api2/tape/drive.rs
index ba9051de..47fa06dc 100644
--- a/src/api2/tape/drive.rs
+++ b/src/api2/tape/drive.rs
@@ -24,6 +24,7 @@ use pbs_api_types::{
 use pbs_api_types::{PRIV_TAPE_AUDIT, PRIV_TAPE_READ, PRIV_TAPE_WRITE};
 
 use pbs_config::CachedUserInfo;
+use pbs_datastore::chunk_store::Write as W;
 use pbs_tape::{
     linux_list_drives::{lookup_device_identification, lto_tape_device_list, open_lto_tape_device},
     sg_tape::tape_alert_flags_critical,
@@ -1342,7 +1343,7 @@ pub fn catalog_media(
             drive.read_label()?; // skip over labels - we already read them above
 
             let mut checked_chunks = HashMap::new();
-            restore_media(
+            restore_media::<W>(
                 worker,
                 &mut drive,
                 &media_id,
diff --git a/src/api2/tape/restore.rs b/src/api2/tape/restore.rs
index 2cc1baab..8f089c20 100644
--- a/src/api2/tape/restore.rs
+++ b/src/api2/tape/restore.rs
@@ -27,6 +27,7 @@ use pbs_api_types::{
 };
 use pbs_client::pxar::tools::handle_root_with_optional_format_version_prelude;
 use pbs_config::CachedUserInfo;
+use pbs_datastore::chunk_store::{CanRead, CanWrite, Write as W};
 use pbs_datastore::dynamic_index::DynamicIndexReader;
 use pbs_datastore::fixed_index::FixedIndexReader;
 use pbs_datastore::index::IndexFile;
@@ -120,13 +121,13 @@ impl NamespaceMap {
     }
 }
 
-pub struct DataStoreMap {
-    map: HashMap<String, Arc<DataStore>>,
-    default: Option<Arc<DataStore>>,
+pub struct DataStoreMap<T> {
+    map: HashMap<String, Arc<DataStore<T>>>,
+    default: Option<Arc<DataStore<T>>>,
     ns_map: Option<NamespaceMap>,
 }
 
-impl TryFrom<String> for DataStoreMap {
+impl TryFrom<String> for DataStoreMap<W> {
     type Error = Error;
 
     fn try_from(value: String) -> Result<Self, Error> {
@@ -161,7 +162,7 @@ impl TryFrom<String> for DataStoreMap {
     }
 }
 
-impl DataStoreMap {
+impl<T> DataStoreMap<T> {
     fn add_namespaces_maps(&mut self, mappings: Vec<String>) -> Result<bool, Error> {
         let count = mappings.len();
         let ns_map = NamespaceMap::try_from(mappings)?;
@@ -169,7 +170,10 @@ impl DataStoreMap {
         Ok(count > 0)
     }
 
-    fn used_datastores(&self) -> HashMap<&str, (Arc<DataStore>, Option<HashSet<BackupNamespace>>)> {
+    #[allow(clippy::type_complexity)]
+    fn used_datastores(
+        &self,
+    ) -> HashMap<&str, (Arc<DataStore<T>>, Option<HashSet<BackupNamespace>>)> {
         let mut map = HashMap::new();
         for (source, target) in self.map.iter() {
             let ns = self.ns_map.as_ref().map(|map| map.used_namespaces(source));
@@ -189,18 +193,19 @@ impl DataStoreMap {
             .map(|mapping| mapping.get_namespaces(datastore, ns))
     }
 
-    fn target_store(&self, source_datastore: &str) -> Option<Arc<DataStore>> {
+    fn target_store(&self, source_datastore: &str) -> Option<Arc<DataStore<T>>> {
         self.map
             .get(source_datastore)
             .or(self.default.as_ref())
             .map(Arc::clone)
     }
 
+    #[allow(clippy::type_complexity)]
     fn get_targets(
         &self,
         source_datastore: &str,
         source_ns: &BackupNamespace,
-    ) -> Option<(Arc<DataStore>, Option<Vec<BackupNamespace>>)> {
+    ) -> Option<(Arc<DataStore<T>>, Option<Vec<BackupNamespace>>)> {
         self.target_store(source_datastore)
             .map(|store| (store, self.target_ns(source_datastore, source_ns)))
     }
@@ -237,9 +242,9 @@ fn check_datastore_privs(
     Ok(())
 }
 
-fn check_and_create_namespaces(
+fn check_and_create_namespaces<T: CanWrite>(
     user_info: &CachedUserInfo,
-    store: &Arc<DataStore>,
+    store: &Arc<DataStore<T>>,
     ns: &BackupNamespace,
     auth_id: &Authid,
     owner: Option<&Authid>,
@@ -449,13 +454,13 @@ pub fn restore(
 }
 
 #[allow(clippy::too_many_arguments)]
-fn restore_full_worker(
+fn restore_full_worker<T: CanWrite + Send + Sync + 'static>(
     worker: Arc<WorkerTask>,
     inventory: Inventory,
     media_set_uuid: Uuid,
     drive_config: SectionConfigData,
     drive_name: &str,
-    store_map: DataStoreMap,
+    store_map: DataStoreMap<T>,
     restore_owner: &Authid,
     notification_mode: &TapeNotificationMode,
     auth_id: &Authid,
@@ -529,8 +534,8 @@ fn restore_full_worker(
 }
 
 #[allow(clippy::too_many_arguments)]
-fn check_snapshot_restorable(
-    store_map: &DataStoreMap,
+fn check_snapshot_restorable<T: CanRead>(
+    store_map: &DataStoreMap<T>,
     store: &str,
     snapshot: &str,
     ns: &BackupNamespace,
@@ -618,14 +623,14 @@ fn log_required_tapes<'a>(inventory: &Inventory, list: impl Iterator<Item = &'a
 }
 
 #[allow(clippy::too_many_arguments)]
-fn restore_list_worker(
+fn restore_list_worker<T: CanWrite + Send + Sync + 'static>(
     worker: Arc<WorkerTask>,
     snapshots: Vec<String>,
     inventory: Inventory,
     media_set_uuid: Uuid,
     drive_config: SectionConfigData,
     drive_name: &str,
-    store_map: DataStoreMap,
+    store_map: DataStoreMap<T>,
     restore_owner: &Authid,
     notification_mode: &TapeNotificationMode,
     user_info: Arc<CachedUserInfo>,
@@ -955,16 +960,16 @@ fn get_media_set_catalog(
     Ok(catalog)
 }
 
-fn media_set_tmpdir(datastore: &DataStore, media_set_uuid: &Uuid) -> PathBuf {
+fn media_set_tmpdir<T>(datastore: &DataStore<T>, media_set_uuid: &Uuid) -> PathBuf {
     let mut path = datastore.base_path();
     path.push(".tmp");
     path.push(media_set_uuid.to_string());
     path
 }
 
-fn snapshot_tmpdir(
+fn snapshot_tmpdir<T>(
     source_datastore: &str,
-    datastore: &DataStore,
+    datastore: &DataStore<T>,
     snapshot: &str,
     media_set_uuid: &Uuid,
 ) -> PathBuf {
@@ -974,9 +979,9 @@ fn snapshot_tmpdir(
     path
 }
 
-fn restore_snapshots_to_tmpdir(
+fn restore_snapshots_to_tmpdir<T>(
     worker: Arc<WorkerTask>,
-    store_map: &DataStoreMap,
+    store_map: &DataStoreMap<T>,
     file_list: &[u64],
     mut drive: Box<dyn TapeDriver>,
     media_id: &MediaId,
@@ -1083,10 +1088,10 @@ fn restore_snapshots_to_tmpdir(
     Ok(tmp_paths)
 }
 
-fn restore_file_chunk_map(
+fn restore_file_chunk_map<T: CanWrite + Send + Sync + 'static>(
     worker: Arc<WorkerTask>,
     drive: &mut Box<dyn TapeDriver>,
-    store_map: &DataStoreMap,
+    store_map: &DataStoreMap<T>,
     file_chunk_map: &mut BTreeMap<u64, HashSet<[u8; 32]>>,
 ) -> Result<(), Error> {
     for (nr, chunk_map) in file_chunk_map.iter_mut() {
@@ -1133,10 +1138,10 @@ fn restore_file_chunk_map(
     Ok(())
 }
 
-fn restore_partial_chunk_archive<'a>(
+fn restore_partial_chunk_archive<'a, T: CanWrite + Send + Sync + 'static>(
     worker: Arc<WorkerTask>,
     reader: Box<dyn 'a + TapeRead>,
-    datastore: Arc<DataStore>,
+    datastore: Arc<DataStore<T>>,
     chunk_list: &mut HashSet<[u8; 32]>,
 ) -> Result<usize, Error> {
     let mut decoder = ChunkArchiveDecoder::new(reader);
@@ -1195,12 +1200,12 @@ fn restore_partial_chunk_archive<'a>(
 
 /// Request and restore complete media without using existing catalog (create catalog instead)
 #[allow(clippy::too_many_arguments)]
-pub fn request_and_restore_media(
+pub fn request_and_restore_media<T: CanWrite + Send + Sync + 'static>(
     worker: Arc<WorkerTask>,
     media_id: &MediaId,
     drive_config: &SectionConfigData,
     drive_name: &str,
-    store_map: &DataStoreMap,
+    store_map: &DataStoreMap<T>,
     checked_chunks_map: &mut HashMap<String, HashSet<[u8; 32]>>,
     restore_owner: &Authid,
     notification_mode: &TapeNotificationMode,
@@ -1253,11 +1258,11 @@ pub fn request_and_restore_media(
 /// Restore complete media content and catalog
 ///
 /// Only create the catalog if target is None.
-pub fn restore_media(
+pub fn restore_media<T: CanWrite + Send + Sync + 'static>(
     worker: Arc<WorkerTask>,
     drive: &mut Box<dyn TapeDriver>,
     media_id: &MediaId,
-    target: Option<(&DataStoreMap, &Authid)>,
+    target: Option<(&DataStoreMap<T>, &Authid)>,
     checked_chunks_map: &mut HashMap<String, HashSet<[u8; 32]>>,
     verbose: bool,
     auth_id: &Authid,
@@ -1301,11 +1306,11 @@ pub fn restore_media(
 }
 
 #[allow(clippy::too_many_arguments)]
-fn restore_archive<'a>(
+fn restore_archive<'a, T: CanWrite + Send + Sync + 'static>(
     worker: Arc<WorkerTask>,
     mut reader: Box<dyn 'a + TapeRead>,
     current_file_number: u64,
-    target: Option<(&DataStoreMap, &Authid)>,
+    target: Option<(&DataStoreMap<T>, &Authid)>,
     catalog: &mut MediaCatalog,
     checked_chunks_map: &mut HashMap<String, HashSet<[u8; 32]>>,
     verbose: bool,
@@ -1525,10 +1530,10 @@ fn scan_chunk_archive<'a>(
     Ok(Some(chunks))
 }
 
-fn restore_chunk_archive<'a>(
+fn restore_chunk_archive<'a, T: CanWrite + Send + Sync + 'static>(
     worker: Arc<WorkerTask>,
     reader: Box<dyn 'a + TapeRead>,
-    datastore: Arc<DataStore>,
+    datastore: Arc<DataStore<T>>,
     checked_chunks: &mut HashSet<[u8; 32]>,
     verbose: bool,
 ) -> Result<Option<Vec<[u8; 32]>>, Error> {
diff --git a/src/backup/hierarchy.rs b/src/backup/hierarchy.rs
index 8dd71fcf..039e32a6 100644
--- a/src/backup/hierarchy.rs
+++ b/src/backup/hierarchy.rs
@@ -7,6 +7,7 @@ use pbs_api_types::{
     PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_READ,
 };
 use pbs_config::CachedUserInfo;
+use pbs_datastore::chunk_store::CanRead;
 use pbs_datastore::{backup_info::BackupGroup, DataStore, ListGroups, ListNamespacesRecursive};
 
 /// Asserts that `privs` are fulfilled on datastore + (optional) namespace.
@@ -68,8 +69,8 @@ pub fn check_ns_privs_full(
     );
 }
 
-pub fn can_access_any_namespace(
-    store: Arc<DataStore>,
+pub fn can_access_any_namespace<T: CanRead + 'static>(
+    store: Arc<DataStore<T>>,
     auth_id: &Authid,
     user_info: &CachedUserInfo,
 ) -> bool {
@@ -95,8 +96,8 @@ pub fn can_access_any_namespace(
 ///
 /// Is basically just a filter-iter for pbs_datastore::ListNamespacesRecursive including access and
 /// optional owner checks.
-pub struct ListAccessibleBackupGroups<'a> {
-    store: &'a Arc<DataStore>,
+pub struct ListAccessibleBackupGroups<'a, T> {
+    store: &'a Arc<DataStore<T>>,
     auth_id: Option<&'a Authid>,
     user_info: Arc<CachedUserInfo>,
     /// The priv on NS level that allows auth_id trump the owner check
@@ -104,15 +105,15 @@ pub struct ListAccessibleBackupGroups<'a> {
     /// The priv that auth_id is required to have on NS level additionally to being owner
     owner_and_priv: u64,
     /// Contains the intertnal state, group iter and a bool flag for override_owner_priv
-    state: Option<(ListGroups, bool)>,
-    ns_iter: ListNamespacesRecursive,
+    state: Option<(ListGroups<T>, bool)>,
+    ns_iter: ListNamespacesRecursive<T>,
 }
 
-impl<'a> ListAccessibleBackupGroups<'a> {
+impl<'a, T: CanRead> ListAccessibleBackupGroups<'a, T> {
     // TODO: builder pattern
 
     pub fn new_owned(
-        store: &'a Arc<DataStore>,
+        store: &'a Arc<DataStore<T>>,
         ns: BackupNamespace,
         max_depth: usize,
         auth_id: Option<&'a Authid>,
@@ -122,7 +123,7 @@ impl<'a> ListAccessibleBackupGroups<'a> {
     }
 
     pub fn new_with_privs(
-        store: &'a Arc<DataStore>,
+        store: &'a Arc<DataStore<T>>,
         ns: BackupNamespace,
         max_depth: usize,
         override_owner_priv: Option<u64>,
@@ -145,8 +146,8 @@ impl<'a> ListAccessibleBackupGroups<'a> {
 pub static NS_PRIVS_OK: u64 =
     PRIV_DATASTORE_MODIFY | PRIV_DATASTORE_READ | PRIV_DATASTORE_BACKUP | PRIV_DATASTORE_AUDIT;
 
-impl Iterator for ListAccessibleBackupGroups<'_> {
-    type Item = Result<BackupGroup, Error>;
+impl<T: CanRead> Iterator for ListAccessibleBackupGroups<'_, T> {
+    type Item = Result<BackupGroup<T>, Error>;
 
     fn next(&mut self) -> Option<Self::Item> {
         loop {
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 3d2cba8a..15c2e9e4 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -15,6 +15,7 @@ use pbs_api_types::{
     UPID,
 };
 use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo};
+use pbs_datastore::chunk_store::{CanRead, CanWrite};
 use pbs_datastore::index::IndexFile;
 use pbs_datastore::manifest::{BackupManifest, FileInfo};
 use pbs_datastore::{DataBlob, DataStore, StoreProgress};
@@ -25,16 +26,16 @@ use crate::backup::hierarchy::ListAccessibleBackupGroups;
 
 /// A VerifyWorker encapsulates a task worker, datastore and information about which chunks have
 /// already been verified or detected as corrupt.
-pub struct VerifyWorker {
+pub struct VerifyWorker<T> {
     worker: Arc<dyn WorkerTaskContext>,
-    datastore: Arc<DataStore>,
+    datastore: Arc<DataStore<T>>,
     verified_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
     corrupt_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
 }
 
-impl VerifyWorker {
+impl<T> VerifyWorker<T> {
     /// Creates a new VerifyWorker for a given task worker and datastore.
-    pub fn new(worker: Arc<dyn WorkerTaskContext>, datastore: Arc<DataStore>) -> Self {
+    pub fn new(worker: Arc<dyn WorkerTaskContext>, datastore: Arc<DataStore<T>>) -> Self {
         Self {
             worker,
             datastore,
@@ -46,7 +47,7 @@ impl VerifyWorker {
     }
 }
 
-fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
+fn verify_blob<T: CanRead>(backup_dir: &BackupDir<T>, info: &FileInfo) -> Result<(), Error> {
     let blob = backup_dir.load_blob(&info.filename)?;
 
     let raw_size = blob.raw_size();
@@ -70,7 +71,7 @@ fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
     }
 }
 
-fn rename_corrupted_chunk(datastore: Arc<DataStore>, digest: &[u8; 32]) {
+fn rename_corrupted_chunk<T: CanWrite>(datastore: Arc<DataStore<T>>, digest: &[u8; 32]) {
     let (path, digest_str) = datastore.chunk_path(digest);
 
     let mut counter = 0;
@@ -97,8 +98,8 @@ fn rename_corrupted_chunk(datastore: Arc<DataStore>, digest: &[u8; 32]) {
     };
 }
 
-fn verify_index_chunks(
-    verify_worker: &VerifyWorker,
+fn verify_index_chunks<T: CanWrite + Send + Sync + 'static>(
+    verify_worker: &VerifyWorker<T>,
     index: Box<dyn IndexFile + Send>,
     crypt_mode: CryptMode,
 ) -> Result<(), Error> {
@@ -238,9 +239,9 @@ fn verify_index_chunks(
     Ok(())
 }
 
-fn verify_fixed_index(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
+fn verify_fixed_index<T: CanWrite + Send + Sync + 'static>(
+    verify_worker: &VerifyWorker<T>,
+    backup_dir: &BackupDir<T>,
     info: &FileInfo,
 ) -> Result<(), Error> {
     let mut path = backup_dir.relative_path();
@@ -260,9 +261,9 @@ fn verify_fixed_index(
     verify_index_chunks(verify_worker, Box::new(index), info.chunk_crypt_mode())
 }
 
-fn verify_dynamic_index(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
+fn verify_dynamic_index<T: CanWrite + Send + Sync + 'static>(
+    verify_worker: &VerifyWorker<T>,
+    backup_dir: &BackupDir<T>,
     info: &FileInfo,
 ) -> Result<(), Error> {
     let mut path = backup_dir.relative_path();
@@ -291,9 +292,9 @@ fn verify_dynamic_index(
 /// - Ok(true) if verify is successful
 /// - Ok(false) if there were verification errors
 /// - Err(_) if task was aborted
-pub fn verify_backup_dir(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
+pub fn verify_backup_dir<T: CanWrite + Send + Sync + 'static>(
+    verify_worker: &VerifyWorker<T>,
+    backup_dir: &BackupDir<T>,
     upid: UPID,
     filter: Option<&dyn Fn(&BackupManifest) -> bool>,
 ) -> Result<bool, Error> {
@@ -325,9 +326,9 @@ pub fn verify_backup_dir(
 }
 
 /// See verify_backup_dir
-pub fn verify_backup_dir_with_lock(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
+pub fn verify_backup_dir_with_lock<T: CanWrite + Send + Sync + 'static>(
+    verify_worker: &VerifyWorker<T>,
+    backup_dir: &BackupDir<T>,
     upid: UPID,
     filter: Option<&dyn Fn(&BackupManifest) -> bool>,
     _snap_lock: BackupLockGuard,
@@ -403,9 +404,9 @@ pub fn verify_backup_dir_with_lock(
 /// Returns
 /// - Ok((count, failed_dirs)) where failed_dirs had verification errors
 /// - Err(_) if task was aborted
-pub fn verify_backup_group(
-    verify_worker: &VerifyWorker,
-    group: &BackupGroup,
+pub fn verify_backup_group<T: CanWrite + Send + Sync + 'static>(
+    verify_worker: &VerifyWorker<T>,
+    group: &BackupGroup<T>,
     progress: &mut StoreProgress,
     upid: &UPID,
     filter: Option<&dyn Fn(&BackupManifest) -> bool>,
@@ -455,8 +456,8 @@ pub fn verify_backup_group(
 /// Returns
 /// - Ok(failed_dirs) where failed_dirs had verification errors
 /// - Err(_) if task was aborted
-pub fn verify_all_backups(
-    verify_worker: &VerifyWorker,
+pub fn verify_all_backups<T: CanWrite + Send + Sync + 'static>(
+    verify_worker: &VerifyWorker<T>,
     upid: &UPID,
     ns: BackupNamespace,
     max_depth: Option<usize>,
@@ -504,7 +505,7 @@ pub fn verify_all_backups(
             .filter(|group| {
                 !(group.backup_type() == BackupType::Host && group.backup_id() == "benchmark")
             })
-            .collect::<Vec<BackupGroup>>(),
+            .collect::<Vec<BackupGroup<T>>>(),
         Err(err) => {
             info!("unable to list backups: {err}");
             return Ok(errors);
diff --git a/src/bin/proxmox-backup-proxy.rs b/src/bin/proxmox-backup-proxy.rs
index 1d4cf37c..bda2f17b 100644
--- a/src/bin/proxmox-backup-proxy.rs
+++ b/src/bin/proxmox-backup-proxy.rs
@@ -20,7 +20,8 @@ use proxmox_router::{RpcEnvironment, RpcEnvironmentType};
 use proxmox_sys::fs::CreateOptions;
 use proxmox_sys::logrotate::LogRotate;
 
-use pbs_datastore::DataStore;
+use pbs_datastore::chunk_store::Lookup as L;
+use pbs_datastore::{is_garbage_collection_running, DataStore};
 
 use proxmox_rest_server::{
     cleanup_old_tasks, cookie_from_header, rotate_task_log_archive, ApiConfig, Redirector,
@@ -265,7 +266,7 @@ async fn run() -> Result<(), Error> {
 
     // to remove references for not configured datastores
     command_sock.register_command("datastore-removed".to_string(), |_value| {
-        if let Err(err) = DataStore::remove_unused_datastores() {
+        if let Err(err) = DataStore::<L>::remove_unused_datastores() {
             log::error!("could not refresh datastores: {err}");
         }
         Ok(Value::Null)
@@ -274,7 +275,7 @@ async fn run() -> Result<(), Error> {
     // clear cache entry for datastore that is in a specific maintenance mode
     command_sock.register_command("update-datastore-cache".to_string(), |value| {
         if let Some(name) = value.and_then(Value::as_str) {
-            if let Err(err) = DataStore::update_datastore_cache(name) {
+            if let Err(err) = DataStore::<L>::update_datastore_cache(name) {
                 log::error!("could not trigger update datastore cache: {err}");
             }
         }
diff --git a/src/server/gc_job.rs b/src/server/gc_job.rs
index 64835028..c2af6c67 100644
--- a/src/server/gc_job.rs
+++ b/src/server/gc_job.rs
@@ -4,15 +4,18 @@ use std::sync::Arc;
 use tracing::info;
 
 use pbs_api_types::Authid;
+use pbs_datastore::chunk_store::CanWrite;
 use pbs_datastore::DataStore;
 use proxmox_rest_server::WorkerTask;
 
 use crate::server::{jobstate::Job, send_gc_status};
 
 /// Runs a garbage collection job.
-pub fn do_garbage_collection_job(
+pub fn do_garbage_collection_job<
+    T: CanWrite + Send + Sync + std::panic::RefUnwindSafe + 'static,
+>(
     mut job: Job,
-    datastore: Arc<DataStore>,
+    datastore: Arc<DataStore<T>>,
     auth_id: &Authid,
     schedule: Option<String>,
     to_stdout: bool,
diff --git a/src/server/prune_job.rs b/src/server/prune_job.rs
index 1c86647a..395aaee4 100644
--- a/src/server/prune_job.rs
+++ b/src/server/prune_job.rs
@@ -7,6 +7,7 @@ use pbs_api_types::{
     print_store_and_ns, Authid, KeepOptions, Operation, PruneJobOptions, MAX_NAMESPACE_DEPTH,
     PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
 };
+use pbs_datastore::chunk_store::CanWrite;
 use pbs_datastore::prune::compute_prune_info;
 use pbs_datastore::DataStore;
 use proxmox_rest_server::WorkerTask;
@@ -14,10 +15,10 @@ use proxmox_rest_server::WorkerTask;
 use crate::backup::ListAccessibleBackupGroups;
 use crate::server::jobstate::Job;
 
-pub fn prune_datastore(
+pub fn prune_datastore<T: CanWrite>(
     auth_id: Authid,
     prune_options: PruneJobOptions,
-    datastore: Arc<DataStore>,
+    datastore: Arc<DataStore<T>>,
     dry_run: bool,
 ) -> Result<(), Error> {
     let store = &datastore.name();
diff --git a/src/server/pull.rs b/src/server/pull.rs
index b1724c14..573aa805 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -18,6 +18,7 @@ use pbs_api_types::{
 };
 use pbs_client::BackupRepository;
 use pbs_config::CachedUserInfo;
+use pbs_datastore::chunk_store::{CanWrite, Write as W};
 use pbs_datastore::data_blob::DataBlob;
 use pbs_datastore::dynamic_index::DynamicIndexReader;
 use pbs_datastore::fixed_index::FixedIndexReader;
@@ -34,8 +35,8 @@ use super::sync::{
 use crate::backup::{check_ns_modification_privs, check_ns_privs};
 use crate::tools::parallel_handler::ParallelHandler;
 
-pub(crate) struct PullTarget {
-    store: Arc<DataStore>,
+pub(crate) struct PullTarget<T> {
+    store: Arc<DataStore<T>>,
     ns: BackupNamespace,
 }
 
@@ -44,7 +45,7 @@ pub(crate) struct PullParameters {
     /// Where data is pulled from
     source: Arc<dyn SyncSource>,
     /// Where data should be pulled into
-    target: PullTarget,
+    target: PullTarget<W>,
     /// Owner of synced groups (needs to match local owner of pre-existing groups)
     owner: Authid,
     /// Whether to remove groups which exist locally, but not on the remote end
@@ -135,9 +136,9 @@ impl PullParameters {
     }
 }
 
-async fn pull_index_chunks<I: IndexFile>(
+async fn pull_index_chunks<I: IndexFile, T: CanWrite + Send + Sync + 'static>(
     chunk_reader: Arc<dyn AsyncReadChunk>,
-    target: Arc<DataStore>,
+    target: Arc<DataStore<T>>,
     index: I,
     downloaded_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
 ) -> Result<SyncStats, Error> {
@@ -260,9 +261,9 @@ fn verify_archive(info: &FileInfo, csum: &[u8; 32], size: u64) -> Result<(), Err
 ///   -- Verify tmp file checksum
 /// - if archive is an index, pull referenced chunks
 /// - Rename tmp file into real path
-async fn pull_single_archive<'a>(
+async fn pull_single_archive<'a, T: CanWrite + Send + Sync + 'static>(
     reader: Arc<dyn SyncSourceReader + 'a>,
-    snapshot: &'a pbs_datastore::BackupDir,
+    snapshot: &'a pbs_datastore::BackupDir<T>,
     archive_info: &'a FileInfo,
     downloaded_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
 ) -> Result<SyncStats, Error> {
@@ -343,10 +344,10 @@ async fn pull_single_archive<'a>(
 ///   -- if file already exists, verify contents
 ///   -- if not, pull it from the remote
 /// - Download log if not already existing
-async fn pull_snapshot<'a>(
+async fn pull_snapshot<'a, T: CanWrite + Send + Sync + 'static>(
     params: &PullParameters,
     reader: Arc<dyn SyncSourceReader + 'a>,
-    snapshot: &'a pbs_datastore::BackupDir,
+    snapshot: &'a pbs_datastore::BackupDir<T>,
     downloaded_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
     corrupt: bool,
     is_new: bool,
@@ -482,10 +483,10 @@ async fn pull_snapshot<'a>(
 ///
 /// The `reader` is configured to read from the source backup directory, while the
 /// `snapshot` is pointing to the local datastore and target namespace.
-async fn pull_snapshot_from<'a>(
+async fn pull_snapshot_from<'a, T: CanWrite + Send + Sync + 'static>(
     params: &PullParameters,
     reader: Arc<dyn SyncSourceReader + 'a>,
-    snapshot: &'a pbs_datastore::BackupDir,
+    snapshot: &'a pbs_datastore::BackupDir<T>,
     downloaded_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
     corrupt: bool,
 ) -> Result<SyncStats, Error> {
diff --git a/src/server/push.rs b/src/server/push.rs
index e71012ed..ff9d9358 100644
--- a/src/server/push.rs
+++ b/src/server/push.rs
@@ -18,6 +18,7 @@ use pbs_api_types::{
 };
 use pbs_client::{BackupRepository, BackupWriter, HttpClient, MergedChunkInfo, UploadOptions};
 use pbs_config::CachedUserInfo;
+use pbs_datastore::chunk_store::Read as R;
 use pbs_datastore::data_blob::ChunkInfo;
 use pbs_datastore::dynamic_index::DynamicIndexReader;
 use pbs_datastore::fixed_index::FixedIndexReader;
@@ -61,7 +62,7 @@ impl PushTarget {
 /// Parameters for a push operation
 pub(crate) struct PushParameters {
     /// Source of backups to be pushed to remote
-    source: Arc<LocalSource>,
+    source: Arc<LocalSource<R>>,
     /// Target for backups to be pushed to
     target: PushTarget,
     /// User used for permission checks on the source side, including potentially filtering visible
diff --git a/src/server/sync.rs b/src/server/sync.rs
index 09814ef0..96a73503 100644
--- a/src/server/sync.rs
+++ b/src/server/sync.rs
@@ -24,6 +24,7 @@ use pbs_api_types::{
     PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_READ,
 };
 use pbs_client::{BackupReader, BackupRepository, HttpClient, RemoteChunkReader};
+use pbs_datastore::chunk_store::CanRead;
 use pbs_datastore::data_blob::DataBlob;
 use pbs_datastore::read_chunk::AsyncReadChunk;
 use pbs_datastore::{BackupManifest, DataStore, ListNamespacesRecursive, LocalChunkReader};
@@ -105,10 +106,10 @@ pub(crate) struct RemoteSourceReader {
     pub(crate) dir: BackupDir,
 }
 
-pub(crate) struct LocalSourceReader {
+pub(crate) struct LocalSourceReader<T> {
     pub(crate) _dir_lock: Arc<Mutex<BackupLockGuard>>,
     pub(crate) path: PathBuf,
-    pub(crate) datastore: Arc<DataStore>,
+    pub(crate) datastore: Arc<DataStore<T>>,
 }
 
 #[async_trait::async_trait]
@@ -189,7 +190,7 @@ impl SyncSourceReader for RemoteSourceReader {
 }
 
 #[async_trait::async_trait]
-impl SyncSourceReader for LocalSourceReader {
+impl<T: CanRead + Send + Sync + 'static> SyncSourceReader for LocalSourceReader<T> {
     fn chunk_reader(&self, crypt_mode: CryptMode) -> Arc<dyn AsyncReadChunk> {
         Arc::new(LocalChunkReader::new(
             self.datastore.clone(),
@@ -266,8 +267,8 @@ pub(crate) struct RemoteSource {
     pub(crate) client: HttpClient,
 }
 
-pub(crate) struct LocalSource {
-    pub(crate) store: Arc<DataStore>,
+pub(crate) struct LocalSource<T> {
+    pub(crate) store: Arc<DataStore<T>>,
     pub(crate) ns: BackupNamespace,
 }
 
@@ -415,7 +416,7 @@ impl SyncSource for RemoteSource {
 }
 
 #[async_trait::async_trait]
-impl SyncSource for LocalSource {
+impl<T: CanRead + Send + Sync + 'static> SyncSource for LocalSource<T> {
     async fn list_namespaces(
         &self,
         max_depth: &mut Option<usize>,
diff --git a/src/tape/file_formats/snapshot_archive.rs b/src/tape/file_formats/snapshot_archive.rs
index 9d11c04b..7f4ef01f 100644
--- a/src/tape/file_formats/snapshot_archive.rs
+++ b/src/tape/file_formats/snapshot_archive.rs
@@ -5,6 +5,7 @@ use std::task::{Context, Poll};
 use proxmox_sys::error::SysError;
 use proxmox_uuid::Uuid;
 
+use pbs_datastore::chunk_store::CanRead;
 use pbs_datastore::SnapshotReader;
 use pbs_tape::{MediaContentHeader, TapeWrite, PROXMOX_TAPE_BLOCK_SIZE};
 
@@ -21,9 +22,9 @@ use crate::tape::file_formats::{
 /// `LEOM` was detected before all data was written. The stream is
 /// marked inclomplete in that case and does not contain all data (The
 /// backup task must rewrite the whole file on the next media).
-pub fn tape_write_snapshot_archive<'a>(
+pub fn tape_write_snapshot_archive<'a, T: CanRead>(
     writer: &mut (dyn TapeWrite + 'a),
-    snapshot_reader: &SnapshotReader,
+    snapshot_reader: &SnapshotReader<T>,
 ) -> Result<Option<Uuid>, std::io::Error> {
     let backup_dir = snapshot_reader.snapshot();
     let snapshot =
diff --git a/src/tape/pool_writer/mod.rs b/src/tape/pool_writer/mod.rs
index 54084421..17c20add 100644
--- a/src/tape/pool_writer/mod.rs
+++ b/src/tape/pool_writer/mod.rs
@@ -15,6 +15,7 @@ use tracing::{info, warn};
 
 use proxmox_uuid::Uuid;
 
+use pbs_datastore::chunk_store::CanRead;
 use pbs_datastore::{DataStore, SnapshotReader};
 use pbs_tape::{sg_tape::tape_alert_flags_critical, TapeWrite};
 use proxmox_rest_server::WorkerTask;
@@ -452,9 +453,9 @@ impl PoolWriter {
     /// archive is marked incomplete, and we do not use it. The caller
     /// should mark the media as full and try again using another
     /// media.
-    pub fn append_snapshot_archive(
+    pub fn append_snapshot_archive<T: CanRead>(
         &mut self,
-        snapshot_reader: &SnapshotReader,
+        snapshot_reader: &SnapshotReader<T>,
     ) -> Result<(bool, usize), Error> {
         let status = match self.status {
             Some(ref mut status) => status,
@@ -543,10 +544,10 @@ impl PoolWriter {
         Ok((leom, bytes_written))
     }
 
-    pub fn spawn_chunk_reader_thread(
+    pub fn spawn_chunk_reader_thread<T: CanRead + Send + Sync + 'static>(
         &self,
-        datastore: Arc<DataStore>,
-        snapshot_reader: Arc<Mutex<SnapshotReader>>,
+        datastore: Arc<DataStore<T>>,
+        snapshot_reader: Arc<Mutex<SnapshotReader<T>>>,
     ) -> Result<(std::thread::JoinHandle<()>, NewChunksIterator), Error> {
         NewChunksIterator::spawn(
             datastore,
diff --git a/src/tape/pool_writer/new_chunks_iterator.rs b/src/tape/pool_writer/new_chunks_iterator.rs
index e6f418df..e1a0da20 100644
--- a/src/tape/pool_writer/new_chunks_iterator.rs
+++ b/src/tape/pool_writer/new_chunks_iterator.rs
@@ -3,6 +3,7 @@ use std::sync::{Arc, Mutex};
 
 use anyhow::{format_err, Error};
 
+use pbs_datastore::chunk_store::CanRead;
 use pbs_datastore::{DataBlob, DataStore, SnapshotReader};
 
 use crate::tape::CatalogSet;
@@ -21,9 +22,9 @@ impl NewChunksIterator {
     /// Creates the iterator, spawning a new thread
     ///
     /// Make sure to join() the returned thread handle.
-    pub fn spawn(
-        datastore: Arc<DataStore>,
-        snapshot_reader: Arc<Mutex<SnapshotReader>>,
+    pub fn spawn<T: CanRead + Send + Sync + 'static>(
+        datastore: Arc<DataStore<T>>,
+        snapshot_reader: Arc<Mutex<SnapshotReader<T>>>,
         catalog_set: Arc<Mutex<CatalogSet>>,
         read_threads: usize,
     ) -> Result<(std::thread::JoinHandle<()>, Self), Error> {
-- 
2.39.5




From h.laimer at proxmox.com  Mon May 26 16:18:34 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Mon, 26 May 2025 16:18:34 +0200
Subject: [pbs-devel] superseded: [PATCH proxmox-backup RFC 00/10]
 introduce typestate for datastore/chunkstore
In-Reply-To: <20240903123401.91513-1-h.laimer@proxmox.com>
References: <20240903123401.91513-1-h.laimer@proxmox.com>
Message-ID: <e1a6415a-2555-41de-a231-1eef126afea7@proxmox.com>

superseded by 
https://lore.proxmox.com/pbs-devel/20240903123401.91513-1-h.laimer at proxmox.com/

On 9/3/24 14:33, Hannes Laimer wrote:
> This patch series introduces two traits, CanRead and CanWrite, to define whether
> a datastore reference is readable, writable, or neither. Functions that read
> or write are now implemented in `impl<T: CanRead>` or `impl<T: CanWrite>` blocks, ensuring
> that they are only available to references that are supposed to read/write.
> 
> Motivation:
> Currently, we track the number of read/write references of a datastore but we don't
> track Lookup operations as they don't read or write, they still need a chunkstore, so
> eventhough they don't neccessarily directly do IO, they hold an open file handle.
> This is a problem for things like unmounting, currently lookup operations are only really
> short, so you'd need really unlucky timing to actually run into problems, but still,
> if a datastore is in "offline" maintenance mode, we shouldn't open filehandles on it.
> 
> By encoding state in the type:
> 1. We can assign non-readable/writable references for lookup operations.
> 2. The compiler ensures correct usage of references. Since it is easy to miss
>      what might happen a few function calls down the line, having the compiler
>      yell at you for easily missed things like this, is a really good thing
>      I think.
> 
> Changes:
> * Added CanRead and CanWrite traits.
> * Separated functions into impl<T: CanRead> or impl<T: CanWrite>.
> * Introduced three new datastore lookup functions that return concrete types implementing
>     CanRead, CanWrite, or neither.
> * Renamed lookup_datastore() to open_datastore() and made it private.
> 
> The main downside is needing separate datastore caches for read and write references due to
> concrete type requirements in the cache HashMap.
> 
> Almost all changes are either adding generics or moving functions into the appropriate
> trait implementations. The logic itself is only touched twice, once in datastore_lookup()
> and once check_privs_and_load_store() in /api/admin/datastore, this function now only checks
> the privs, the datastore opening happens in the endpoint function directly.
> 
>   
> 
> Hannes Laimer (10):
>    chunkstore: add CanRead and CanWrite trait
>    chunkstore: separate functions into impl block
>    datastore: add generics and new lookup functions
>    datastore: separate functions into impl block
>    backup_info: add generics and separate functions into impl blocks
>    pbs-datastore: add generics and separate functions into impl blocks
>    api: replace datastore_lookup with new, state-typed datastore
>      returning functions
>    server/bin: replace datastore_lookup with new, state-typed datastore
>      returning functions
>    api: add generics and separate functions into impl blocks
>    backup/server/tape: add generics and separate functions into impl
>      blocks
> 
>   pbs-datastore/src/backup_info.rs            |  179 +-
>   pbs-datastore/src/chunk_store.rs            |  228 ++-
>   pbs-datastore/src/datastore.rs              | 1688 ++++++++++---------
>   pbs-datastore/src/dynamic_index.rs          |   22 +-
>   pbs-datastore/src/fixed_index.rs            |   50 +-
>   pbs-datastore/src/hierarchy.rs              |   92 +-
>   pbs-datastore/src/local_chunk_reader.rs     |   13 +-
>   pbs-datastore/src/prune.rs                  |   19 +-
>   pbs-datastore/src/snapshot_reader.rs        |   30 +-
>   src/api2/admin/datastore.rs                 |  170 +-
>   src/api2/admin/namespace.rs                 |    8 +-
>   src/api2/backup/environment.rs              |  176 +-
>   src/api2/backup/mod.rs                      |   25 +-
>   src/api2/backup/upload_chunk.rs             |   19 +-
>   src/api2/reader/environment.rs              |   31 +-
>   src/api2/reader/mod.rs                      |    9 +-
>   src/api2/status.rs                          |    8 +-
>   src/api2/tape/backup.rs                     |   21 +-
>   src/api2/tape/drive.rs                      |    2 +-
>   src/api2/tape/restore.rs                    |   75 +-
>   src/backup/hierarchy.rs                     |   26 +-
>   src/backup/verify.rs                        |   53 +-
>   src/bin/proxmox-backup-proxy.rs             |    4 +-
>   src/server/gc_job.rs                        |    8 +-
>   src/server/prune_job.rs                     |    9 +-
>   src/server/pull.rs                          |   29 +-
>   src/server/verify_job.rs                    |    4 +-
>   src/tape/file_formats/snapshot_archive.rs   |    5 +-
>   src/tape/pool_writer/mod.rs                 |   11 +-
>   src/tape/pool_writer/new_chunks_iterator.rs |    7 +-
>   30 files changed, 1585 insertions(+), 1436 deletions(-)
> 




From c.ebner at proxmox.com  Thu May 29 16:31:28 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:28 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 03/42] api: fix minor formatting
 issues
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-4-c.ebner@proxmox.com>

These are currently not shown by a `cargo fmt --check`.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/mod.rs | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/src/api2/backup/mod.rs b/src/api2/backup/mod.rs
index 629df933e..567bca3ef 100644
--- a/src/api2/backup/mod.rs
+++ b/src/api2/backup/mod.rs
@@ -166,7 +166,7 @@ fn upgrade_to_backup_protocol(
                     Ok(None) => {
                         // no verify state found, treat as valid
                         Some(info)
-                    },
+                    }
                     Err(err) => {
                         warn!("error parsing the snapshot manifest: {err:#}");
                         Some(info)
@@ -236,7 +236,8 @@ fn upgrade_to_backup_protocol(
                     .and_then(move |conn| {
                         env2.debug("protocol upgrade done");
 
-                        let mut http = hyper::server::conn::http2::Builder::new(ExecInheritLogContext);
+                        let mut http =
+                            hyper::server::conn::http2::Builder::new(ExecInheritLogContext);
                         // increase window size: todo - find optiomal size
                         let window_size = 32 * 1024 * 1024; // max = (1 << 31) - 2
                         http.initial_stream_window_size(window_size);
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:27 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:27 +0200
Subject: [pbs-devel] [RFC v2 proxmox/bookworm-stable 2/42] pbs-api-types:
 extend datastore config by backend config enum
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-3-c.ebner@proxmox.com>

Allows to configure a backend config variant for a datastore on
creation. The current default `Filesystem` backend variant is
introduced to be compatible with existing storages. A new S3 backend
variant allows to create datastores backed by an S3 compatible object
store instead.

For S3 backends, the id of the corresponding S3 client configuration
is storered. A valid datastore backend configuration for S3 therefore
contains:
```
    ...
    backend s3=<S3_CONFIG_ID>
    ...
```

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-api-types/src/datastore.rs | 58 +++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/pbs-api-types/src/datastore.rs b/pbs-api-types/src/datastore.rs
index 5bd953ac..2b983cb2 100644
--- a/pbs-api-types/src/datastore.rs
+++ b/pbs-api-types/src/datastore.rs
@@ -336,7 +336,11 @@ pub const DATASTORE_TUNING_STRING_SCHEMA: Schema = StringSchema::new("Datastore
             optional: true,
             format: &proxmox_schema::api_types::UUID_FORMAT,
             type: String,
-        }
+        },
+        backend: {
+            schema: DATASTORE_BACKEND_SCHEMA,
+            optional: true,
+        },
     }
 )]
 #[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
@@ -389,8 +393,59 @@ pub struct DataStoreConfig {
     #[updater(skip)]
     #[serde(skip_serializing_if = "Option::is_none")]
     pub backing_device: Option<String>,
+
+    /// Backend to be used by datastore
+    #[updater(skip)]
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub backend: Option<String>,
+}
+
+pub const DATASTORE_BACKEND_SCHEMA: Schema = StringSchema::new("Backend config to be used for datastore.")
+    .format(&ApiStringFormat::VerifyFn(verify_datastore_backend))
+    .type_text("<filesystem|s3=S3_CONFIG_ID>")
+    .schema();
+
+fn verify_datastore_backend(input: &str) -> Result<(), Error> {
+    DatastoreBackendConfig::from_str(input).map(|_| ())
+}
+
+#[derive(Clone, Default)]
+/// Available backend configurations for datastores.
+pub enum DatastoreBackendConfig {
+    #[default]
+    Filesystem,
+    S3(String),
 }
 
+impl std::str::FromStr for DatastoreBackendConfig {
+    type Err = Error;
+
+    fn from_str(s: &str) -> Result<Self, Self::Err> {
+        if s == "filesystem" {
+            return Ok(Self::Filesystem);
+        }
+        match s.split_once('=') {
+            Some(("s3", value)) => {
+                let s3_config_id = value.parse()?;
+                Ok(Self::S3(s3_config_id))
+            }
+            _ => bail!("invalid datastore backend configuration"),
+        }
+    }
+}
+
+impl std::fmt::Display for DatastoreBackendConfig {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            Self::Filesystem => write!(f, "filesystem"),
+            Self::S3(s3_config_id) => write!(f, "s3:{s3_config_id}"),
+        }
+    }
+}
+
+proxmox_serde::forward_serialize_to_display!(DatastoreBackendConfig);
+proxmox_serde::forward_deserialize_to_from_str!(DatastoreBackendConfig);
+
 #[api]
 #[derive(Serialize, Deserialize, Updater, Clone, PartialEq, Default)]
 #[serde(rename_all = "kebab-case")]
@@ -424,6 +479,7 @@ impl DataStoreConfig {
             tuning: None,
             maintenance_mode: None,
             backing_device: None,
+            backend: None,
         }
     }
 
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:30 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:30 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 05/42] datastore: ignore missing
 owner file when removing group directory
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-6-c.ebner@proxmox.com>

Since commit 23be00a4 ("fix #3336: datastore: remove group if the
last snapshot is removed"), a backup group directory is cleaned up
when the new locking mechanism is in use once:
- the group is requested to be destroyed and all the snapshots have
  been deleted
- the last snapshot of a group has been destroyed
Since then, the owner file is also cleaned up separately.

However, the owner file might be already missing due to removal of
the group directory executed when removing the last backup snapshot
of the group, making the subsequent call in the backup group destroy
method fail.

Fix this by ignoring a missing owner file and continue with trying to
emove the group directory itself.

Fixes: 23be00a4 ("fix #3336: datastore: remove group if the last snapshot is removed")
Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index d4732fdd9..1422fe865 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -246,9 +246,11 @@ impl BackupGroup {
     fn remove_group_dir(&self) -> Result<(), Error> {
         let owner_path = self.store.owner_path(&self.ns, &self.group);
 
-        std::fs::remove_file(&owner_path).map_err(|err| {
-            format_err!("removing the owner file '{owner_path:?}' failed - {err}")
-        })?;
+        if let Err(err) = std::fs::remove_file(&owner_path) {
+            if err.kind() != std::io::ErrorKind::NotFound {
+                bail!("removing the owner file '{owner_path:?}' failed - {err}");
+            }
+        }
 
         let path = self.full_group_path();
 
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:26 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:26 +0200
Subject: [pbs-devel] [RFC v2 proxmox/bookworm-stable 1/42] pbs-api-types:
 add types for S3 client configs and secrets
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-2-c.ebner@proxmox.com>

Adds the new config types `S3ClientConfig` and `S3ClientSecret` to
configure datastore backends using an S3 compatible object store.

Secrets are stored as different config to never be returned on api
calls, only allowing to set/update the values.

Use a different name (`secrets_id`) for the unique identifier in case
of the secrets type, although the same id should be used for storing
and lookup. By this, clashing of property names when using flattened
types as api parameters is avoided.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-api-types/src/lib.rs |   3 +
 pbs-api-types/src/s3.rs  | 138 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 141 insertions(+)
 create mode 100644 pbs-api-types/src/s3.rs

diff --git a/pbs-api-types/src/lib.rs b/pbs-api-types/src/lib.rs
index 99ec7961..7a5ea11d 100644
--- a/pbs-api-types/src/lib.rs
+++ b/pbs-api-types/src/lib.rs
@@ -147,6 +147,9 @@ pub use remote::*;
 mod pathpatterns;
 pub use pathpatterns::*;
 
+mod s3;
+pub use s3::*;
+
 mod tape;
 pub use tape::*;
 
diff --git a/pbs-api-types/src/s3.rs b/pbs-api-types/src/s3.rs
new file mode 100644
index 00000000..40c502ba
--- /dev/null
+++ b/pbs-api-types/src/s3.rs
@@ -0,0 +1,138 @@
+use anyhow::bail;
+use serde::{Deserialize, Serialize};
+
+use proxmox_schema::api_types::{
+    CERT_FINGERPRINT_SHA256_SCHEMA, DNS_NAME_OR_IP_SCHEMA, SAFE_ID_FORMAT,
+};
+use proxmox_schema::{api, const_regex, ApiStringFormat, Schema, StringSchema, Updater};
+
+#[rustfmt::skip]
+pub const S3_BUCKET_NAME_REGEX_STR: &str = r"^[a-z0-9]([a-z0-9\-]*[a-z0-9])?$";
+
+const_regex! {
+    /// Regex to match S3 bucket names.
+    ///
+    /// Be as strict as possible following the rules as described here:
+    /// https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html#general-purpose-bucket-names
+    pub S3_BUCKET_NAME_REGEX = r"^[a-z0-9]([a-z0-9\-]*[a-z0-9])?$";
+    /// Regex to match S3 regions.
+    pub S3_REGION_REGEX = r"^[a-z]{2}\-[a-z]{4,}\-[0-9]$";
+}
+
+pub const S3_REGION_FORMAT: ApiStringFormat = ApiStringFormat::Pattern(&S3_REGION_REGEX);
+
+pub const S3_CLIENT_ID_SCHEMA: Schema =
+    StringSchema::new("Unique ID to identify s3 client config.")
+        .format(&SAFE_ID_FORMAT)
+        .min_length(3)
+        .max_length(32)
+        .schema();
+
+pub const S3_REGION_SCHEMA: Schema = StringSchema::new("Region to access S3 object store.")
+    .format(&S3_REGION_FORMAT)
+    .min_length(3)
+    .max_length(32)
+    .schema();
+
+pub const S3_BUCKET_NAME_SCHEMA: Schema = StringSchema::new("Bucket name for S3 object store.")
+    .format(&ApiStringFormat::VerifyFn(|bucket_name| {
+        if !(S3_BUCKET_NAME_REGEX.regex_obj)().is_match(bucket_name) {
+            bail!("Bucket name does not match the regex pattern");
+        }
+
+        // Exclude pre- and postfixes described here:
+        // https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html#general-purpose-bucket-names
+        let forbidden_prefixes = ["xn--", "sthree-", "amzn-s3-demo-"];
+        for prefix in forbidden_prefixes {
+            if bucket_name.starts_with(prefix) {
+                bail!("Bucket name cannot start with '{prefix}'");
+            }
+        }
+
+        let forbidden_postfixes = ["--ol-s3", ".mrap", "--x-s3"];
+        for postfix in forbidden_postfixes {
+            if bucket_name.ends_with(postfix) {
+                bail!("Bucket name cannot end with '{postfix}'");
+            }
+        }
+
+        Ok(())
+    }))
+    .min_length(3)
+    .max_length(63)
+    .schema();
+
+#[api(
+    properties: {
+        id: {
+            schema: S3_CLIENT_ID_SCHEMA,
+        },
+        host: {
+            schema: DNS_NAME_OR_IP_SCHEMA,
+        },
+        bucket: {
+            schema: S3_BUCKET_NAME_SCHEMA,
+        },
+        port: {
+            type: u16,
+            description: "Port to access S3 object store.",
+            optional: true,
+        },
+        region: {
+            schema: S3_REGION_SCHEMA,
+            optional: true,
+        },
+        fingerprint: {
+            schema: CERT_FINGERPRINT_SHA256_SCHEMA,
+            optional: true,
+        },
+        "access-key": {
+            type: String,
+            description: "Access key for S3 object store.",
+        },
+    }
+)]
+#[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// S3 client configuration properties.
+pub struct S3ClientConfig {
+    #[updater(skip)]
+    pub id: String,
+    pub host: String,
+    pub bucket: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub port: Option<u16>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub region: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub fingerprint: Option<String>,
+    pub access_key: String,
+}
+
+impl S3ClientConfig {
+    pub fn acl_path(&self) -> Vec<&str> {
+        // Needs permissions on root path
+        Vec::new()
+    }
+}
+
+#[api(
+    properties: {
+        "secrets-id": {
+            type: String,
+            description: "Unique ID to identify s3 client secret config.",
+        },
+        "secret-key": {
+            type: String,
+            description: "Secret key for S3 object store.",
+        },
+    }
+)]
+#[derive(Serialize, Deserialize, Updater, Clone, PartialEq)]
+#[serde(rename_all = "kebab-case")]
+/// S3 client secrets configuration properties.
+pub struct S3ClientSecretsConfig {
+    #[updater(skip)]
+    pub secrets_id: String,
+    pub secret_key: String,
+}
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:25 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:25 +0200
Subject: [pbs-devel] [RFC v2 proxmox/bookworm-stable proxmox-backup 00/42]
 S3 storage backend for datastores
Message-ID: <20250529143207.694497-1-c.ebner@proxmox.com>

Disclaimer: These patches are in a development state and are not
intended for production use.

This patch series aims to add S3 compatible object stores as storage
backend for PBS datastores. A PBS local cache store using the regular
datastore layout is used for faster operation, bypassing requests to
the S3 api when possible. Further, the local cache store allows to
keep frequently used chunks and is used to avoid expensive metadata
updates on the object store, e.g. by using local marker file during
garbage collection.

Backups are created by upload chunks to the corresponding S3 bucket,
while keeping the index files in the local cache store, on backup
finish, the snapshot metadata are persisted to the S3 storage backend.

Snapshot restores read chunks preferably from the local cache store,
downloading and insterting them if not present from the S3 object
store.

Listing and snapsoht metadata operation currently rely soly on the
local cache store, with the intention to provide a mechanism to
re-sync and merge with object stored on the S3 backend if requested.

Sending this patch series as RFC to get some initial feedback, mostly
on the S3 client implementation part and the corresponding
configuration integration with PBS, which is already in an advanced
stage and warants initial review and real world testing.

Datastore operations on the S3 backend are still work in progress,
but feedback on that is appreciated very much as well.

Among the open points still being worked on are:
- Consistency between local cache and S3 store.
- Sync and merge of namespace, group snapshot and index files when
  required or requested.
- Advanced packing mechanism for chunks to significantly reduce the
  number of api requests and therefore be more cost effective.
- Reduction of in-memory copies for chunks/blobs and recalculation of
  checksums.

Testing:
For testing, an S3 compatible object store provided via Ceph RADOS
gateway can be used by the following setup. This was performed on a
pre-existing Ceph Reef 18.2 cluster.

Install radosgw on all the nodes:
```
apt install radosgw
```

On one node, generate client keyring:
```
ceph-authtool --create-keyring /etc/pve/priv/ceph.client.radosgw.keyring
```

For each node, generate key and add it to the keyring (adapt name
accordingly):
```
ceph-authtool /etc/pve/priv/ceph.client.radosgw.keyring -n client.radosgw.pve-c0-n1 --gen-key
```

Setup capabilities for client keys:
```
ceph-authtool -n client.radosgw.pve-c0-n1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/pve/priv/ceph.client.radosgw.keyring
```

Add the keys (repeat for each) to the cluster:
```
ceph -k /etc/pve/priv/ceph.client.admin.keyring auth add client.radosgw.pve-c0-n1 -i /etc/pve/priv/ceph.client.radosgw.keyring
```

For each client, add a config based on the one below to /etc/ceph/ceph.conf
```
[client.radosgw.pve-c0-n1]
        host = pve-c0-n1
        keyring = /etc/pve/priv/ceph.client.radosgw.keyring
        log file = /var/log/ceph/client.radosgw.$host.log
        rgw_dns_name = s3.pve-c0-n1.local
```

Restart the service for each node, e.g.
```
systemctl daemon-reload
systemctl restart radosgw.service
```

Setup a new user, generating access key and secret key shown in
output:
```
radosgw-admin user create --uid=testuser --display-name="TestUser" --email=your at mail.com
```

Since the configuration and keyring are located on the pmxcfs, add
the following override so the gateway service is only started after
pve-cluster by adding to
`/etc/systemd/system/radosgw.service.d/override.conf`:
```
[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/radosgw
Description=LSB: radosgw RESTful rados gateway
After=pve-cluster.service
Wants=pve-cluster.service
```

A custom certificate must be added since the client forces tls by
extending the config with a path to a custom generated certificate and
key:
```
[client.radosgw.pve-c0-n1]
	host = pve-c0-n1
	keyring = /etc/pve/priv/ceph.client.radosgw.keyring
	logfile = /var/log/ceph/client.radsgw.$host.log
	rgw_dns_name = s3.pve-c0-n1.local
	rgw_frontends = "beast ssl_port=7480 ssl_certificate=/etc/pve/ceph/server-cert.pem ssl_private_key=/etc/pve/ceph/server-key.pem"
```

A new bucket can finally be created using the `s3cmd` cli tool after
initial configuration.

Most notable changes since the previous RFC version 1 [0]:
- Improved and fixed various issues with consistency and locking,
  especially with respect to backup group/snapshot pruning
- Fix and improve listing and deletion of multiple object, also taking
  S3 api object count limits into account.
- Fix namespace handling, especially with respect to prune.
- Fix pull sync jobs not uploading chunks to S3 object store backend
- Fix permissions for s3 config api endpoints
- Fixed issues with hyper::Body not being consumed when skipping cached
  chunks, resulting in stream errors on upload
- Use md5 checksum for consistency checks, crc32 is not yet implemented
  and ignored by many S3 compatible apis, e.g. rados gateway.
- Use iso6801 parser for last modified timestamp parser over limited own
  implementation.
- Rework proxmox-backup-manager s3 command for basic sanity checking
- Smaller bugfixes, code style cleanups and refactoring

[0] https://lore.proxmox.com/pbs-devel/20250519114640.303640-1-c.ebner at proxmox.com/T/

proxmox:

Christian Ebner (2):
  pbs-api-types: add types for S3 client configs and secrets
  pbs-api-types: extend datastore config by backend config enum

 pbs-api-types/src/datastore.rs |  58 +++++++++++++-
 pbs-api-types/src/lib.rs       |   3 +
 pbs-api-types/src/s3.rs        | 138 +++++++++++++++++++++++++++++++++
 3 files changed, 198 insertions(+), 1 deletion(-)
 create mode 100644 pbs-api-types/src/s3.rs

proxmox-backup:

Christian Ebner (40):
  api: fix minor formatting issues
  bin: sort submodules alphabetically
  datastore: ignore missing owner file when removing group directory
  verify: refactor verify related functions to be methods of worker
  s3 client: add crate for AWS S3 compatible object store client
  s3 client: implement AWS signature v4 request authentication
  s3 client: add dedicated type for s3 object keys
  s3 client: add type for last modified timestamp in responses
  s3 client: add helper to parse http date headers
  s3 client: implement methods to operate on s3 objects in bucket
  config: introduce s3 object store client configuration
  api: config: implement endpoints to manipulate and list s3 configs
  api: datastore: check S3 backend bucket access on datastore create
  api/bin: add endpoint and command to check s3 client connection
  datastore: allow to get the backend for a datastore
  api: backup: store datastore backend in runtime environment
  api: backup: conditionally upload chunks to S3 object store backend
  api: backup: conditionally upload blobs to S3 object store backend
  api: backup: conditionally upload indices to S3 object store backend
  api: backup: conditionally upload manifest to S3 object store backend
  sync: pull: conditionally upload content to S3 backend
  api: reader: fetch chunks based on datastore backend
  datastore: local chunk reader: read chunks based on backend
  verify worker: add datastore backed to verify worker
  verify: implement chunk verification for stores with s3 backend
  datastore: create namespace marker in S3 backend
  datastore: create/delete protected marker file on S3 storage backend
  datastore: prune groups/snapshots from S3 object store backend
  datastore: get and set owner for S3 store backend
  datastore: implement garbage collection for s3 backend
  ui: add S3 client edit window for configuration create/edit
  ui: add S3 client view for configuration
  ui: expose the S3 client view in the navigation tree
  ui: add s3 bucket selector and allow to set s3 backend
  tools: lru cache: add removed callback for evicted cache nodes
  tools: async lru cache: implement insert, remove and contains methods
  datastore: add local datastore cache for network attached storages
  api: backup: use local datastore cache on S3 backend chunk upload
  api: reader: use local datastore cache on S3 backend chunk fetching
  api: backup: add no-cache flag to bypass local datastore cache

 Cargo.toml                                    |   8 +
 examples/upload-speed.rs                      |   1 +
 pbs-client/src/backup_writer.rs               |   4 +-
 pbs-config/src/lib.rs                         |   1 +
 pbs-config/src/s3.rs                          |  82 ++
 pbs-datastore/Cargo.toml                      |   3 +
 pbs-datastore/src/backup_info.rs              |  53 +-
 pbs-datastore/src/cached_chunk_reader.rs      |   6 +-
 pbs-datastore/src/datastore.rs                | 435 ++++++++-
 pbs-datastore/src/dynamic_index.rs            |   1 +
 pbs-datastore/src/lib.rs                      |   4 +
 pbs-datastore/src/local_chunk_reader.rs       |  37 +-
 .../src/local_datastore_lru_cache.rs          | 116 +++
 pbs-s3-client/Cargo.toml                      |  29 +
 pbs-s3-client/src/aws_sign_v4.rs              | 140 +++
 pbs-s3-client/src/client.rs                   | 594 ++++++++++++
 pbs-s3-client/src/lib.rs                      | 122 +++
 pbs-s3-client/src/object_key.rs               |  64 ++
 pbs-s3-client/src/response_reader.rs          | 343 +++++++
 pbs-tools/src/async_lru_cache.rs              |  46 +-
 pbs-tools/src/lru_cache.rs                    |  42 +-
 proxmox-backup-client/src/benchmark.rs        |   1 +
 proxmox-backup-client/src/main.rs             |   8 +
 src/api2/admin/datastore.rs                   |  52 +-
 src/api2/admin/mod.rs                         |   2 +
 src/api2/admin/s3.rs                          |  72 ++
 src/api2/backup/environment.rs                | 145 ++-
 src/api2/backup/mod.rs                        | 107 +--
 src/api2/backup/upload_chunk.rs               |  93 +-
 src/api2/config/datastore.rs                  |  41 +-
 src/api2/config/mod.rs                        |   2 +
 src/api2/config/s3.rs                         | 305 ++++++
 src/api2/reader/environment.rs                |  12 +-
 src/api2/reader/mod.rs                        |  59 +-
 src/backup/verify.rs                          | 879 +++++++++---------
 src/bin/proxmox-backup-manager.rs             |   1 +
 src/bin/proxmox_backup_manager/mod.rs         |  30 +-
 src/bin/proxmox_backup_manager/s3.rs          |  34 +
 src/server/pull.rs                            |  62 +-
 src/server/push.rs                            |   1 +
 src/server/verify_job.rs                      |  12 +-
 www/Makefile                                  |   3 +
 www/NavigationTree.js                         |   6 +
 www/config/S3BucketView.js                    | 144 +++
 www/form/S3BucketSelector.js                  |  40 +
 www/window/DataStoreEdit.js                   |  35 +
 www/window/S3BucketEdit.js                    | 125 +++
 47 files changed, 3753 insertions(+), 649 deletions(-)
 create mode 100644 pbs-config/src/s3.rs
 create mode 100644 pbs-datastore/src/local_datastore_lru_cache.rs
 create mode 100644 pbs-s3-client/Cargo.toml
 create mode 100644 pbs-s3-client/src/aws_sign_v4.rs
 create mode 100644 pbs-s3-client/src/client.rs
 create mode 100644 pbs-s3-client/src/lib.rs
 create mode 100644 pbs-s3-client/src/object_key.rs
 create mode 100644 pbs-s3-client/src/response_reader.rs
 create mode 100644 src/api2/admin/s3.rs
 create mode 100644 src/api2/config/s3.rs
 create mode 100644 src/bin/proxmox_backup_manager/s3.rs
 create mode 100644 www/config/S3BucketView.js
 create mode 100644 www/form/S3BucketSelector.js
 create mode 100644 www/window/S3BucketEdit.js

-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:34 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:34 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 09/42] s3 client: add dedicated
 type for s3 object keys
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-10-c.ebner@proxmox.com>

S3 objects are uniquely identified within a bucket by their object
key [0].

Implements conversion and utility traits to easily convert and encode
a string or a chunk digest as corresponding object key for the S3
storage backend. Adds type checking for s3 client operations requiring
an object key.

[0] https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-s3-client/src/lib.rs        |  2 ++
 pbs-s3-client/src/object_key.rs | 64 +++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+)
 create mode 100644 pbs-s3-client/src/object_key.rs

diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
index 5a60b92ec..a4081df15 100644
--- a/pbs-s3-client/src/lib.rs
+++ b/pbs-s3-client/src/lib.rs
@@ -1,3 +1,5 @@
 mod aws_sign_v4;
 mod client;
 pub use client::{S3Client, S3ClientOptions};
+mod object_key;
+pub use object_key::{S3ObjectKey, S3_CONTENT_PREFIX};
diff --git a/pbs-s3-client/src/object_key.rs b/pbs-s3-client/src/object_key.rs
new file mode 100644
index 000000000..362c0cd55
--- /dev/null
+++ b/pbs-s3-client/src/object_key.rs
@@ -0,0 +1,64 @@
+use crate::aws_sign_v4::aws_sign_v4_uri_encode;
+
+pub const S3_CONTENT_PREFIX: &str = ".content";
+
+#[derive(Clone)]
+pub struct S3ObjectKey {
+    object_key: String,
+}
+
+// All regular keys (non-digests) get prefixed by a `/.contents`, so that
+// content listing without all the chunks can be done by that prefix.
+impl core::convert::From<&str> for S3ObjectKey {
+    fn from(object_key: &str) -> Self {
+        let object_key = object_key.strip_prefix("/").unwrap_or(object_key);
+        let object_key = format!(
+            "/{S3_CONTENT_PREFIX}/{object_key}",
+            object_key = aws_sign_v4_uri_encode(object_key, true)
+        );
+
+        Self { object_key }
+    }
+}
+
+impl core::convert::From<&[u8; 32]> for S3ObjectKey {
+    fn from(digest: &[u8; 32]) -> Self {
+        // Use the same layout as on regular PBS datastores, including the 4 hex digit prefix
+        let object_key = hex::encode(digest);
+        let prefix = &object_key[..4];
+        Self {
+            object_key: format!("/.chunks/{prefix}/{object_key}"),
+        }
+    }
+}
+
+impl core::convert::From<[u8; 32]> for S3ObjectKey {
+    fn from(digest: [u8; 32]) -> Self {
+        Self::from(&digest)
+    }
+}
+
+impl std::ops::Deref for S3ObjectKey {
+    type Target = str;
+
+    fn deref(&self) -> &Self::Target {
+        &self.object_key
+    }
+}
+
+impl std::fmt::Display for S3ObjectKey {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", self.object_key)
+    }
+}
+
+impl S3ObjectKey {
+    /// Generate source key for copy object operations given the source bucket.
+    pub fn to_copy_source_key(&self, source_bucket: &str) -> Self {
+        Self {
+            // object key already contains the required separator slash in-between source bucket
+            // and source object key.
+            object_key: format!("{source_bucket}{}", self.object_key),
+        }
+    }
+}
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:32 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:32 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 07/42] s3 client: add crate for
 AWS S3 compatible object store client
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-8-c.ebner@proxmox.com>

Adds the client to connect to an AWS S3 compatible object store API.
Force the use of an TLS encrypted connection as the communication
with the object store will contain sensitive information.
For self-signed certificates, check the fingerprint against the one
configured. This follows along the lines of the PBS client, used to
connect to the PBS server API.

The `S3Client` stores the client state and has to be configured upon
instantiation by providing `S3ClientOptions`.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 Cargo.toml                  |   3 +
 pbs-s3-client/Cargo.toml    |  16 +++++
 pbs-s3-client/src/client.rs | 131 ++++++++++++++++++++++++++++++++++++
 pbs-s3-client/src/lib.rs    |   2 +
 4 files changed, 152 insertions(+)
 create mode 100644 pbs-s3-client/Cargo.toml
 create mode 100644 pbs-s3-client/src/client.rs
 create mode 100644 pbs-s3-client/src/lib.rs

diff --git a/Cargo.toml b/Cargo.toml
index 6de6a6527..c2b0029ac 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -36,6 +36,7 @@ members = [
     "pbs-fuse-loop",
     "pbs-key-config",
     "pbs-pxar-fuse",
+    "pbs-s3-client",
     "pbs-tape",
     "pbs-tools",
 
@@ -105,6 +106,7 @@ pbs-datastore = { path = "pbs-datastore" }
 pbs-fuse-loop = { path = "pbs-fuse-loop" }
 pbs-key-config = { path = "pbs-key-config" }
 pbs-pxar-fuse = { path = "pbs-pxar-fuse" }
+pbs-s3-client = { path = "pbs-s3-client" }
 pbs-tape = { path = "pbs-tape" }
 pbs-tools = { path = "pbs-tools" }
 
@@ -245,6 +247,7 @@ pbs-client.workspace = true
 pbs-config.workspace = true
 pbs-datastore.workspace = true
 pbs-key-config.workspace = true
+pbs-s3-client.workspace = true
 pbs-tape.workspace = true
 pbs-tools.workspace = true
 proxmox-rrd.workspace = true
diff --git a/pbs-s3-client/Cargo.toml b/pbs-s3-client/Cargo.toml
new file mode 100644
index 000000000..1999c3323
--- /dev/null
+++ b/pbs-s3-client/Cargo.toml
@@ -0,0 +1,16 @@
+[package]
+name = "pbs-s3-client"
+version = "0.1.0"
+authors.workspace = true
+edition.workspace = true
+description = "low level client for AWS S3 compatible object stores"
+rust-version.workspace = true
+
+[dependencies]
+anyhow.workspace = true
+hex = { workspace = true, features = [ "serde" ] }
+hyper.workspace = true
+openssl.workspace = true
+tracing.workspace = true
+
+proxmox-http.workspace = true
diff --git a/pbs-s3-client/src/client.rs b/pbs-s3-client/src/client.rs
new file mode 100644
index 000000000..e001cc7b0
--- /dev/null
+++ b/pbs-s3-client/src/client.rs
@@ -0,0 +1,131 @@
+use std::sync::{Arc, Mutex};
+use std::time::Duration;
+
+use anyhow::{bail, format_err, Context, Error};
+use hyper::client::{Client, HttpConnector};
+use hyper::http::uri::Authority;
+use hyper::Body;
+use openssl::hash::MessageDigest;
+use openssl::ssl::{SslConnector, SslMethod, SslVerifyMode};
+use openssl::x509::X509StoreContextRef;
+use tracing::error;
+
+use proxmox_http::client::HttpsConnector;
+
+const S3_HTTP_CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
+const S3_TCP_KEEPALIVE_TIME: u32 = 120;
+
+/// Configuration options for client
+pub struct S3ClientOptions {
+    pub host: String,
+    pub port: Option<u16>,
+    pub bucket: String,
+    pub secret_key: String,
+    pub access_key: String,
+    pub region: String,
+    pub fingerprint: Option<String>,
+}
+
+/// S3 client for object stores compatible with the AWS S3 API
+pub struct S3Client {
+    client: Client<HttpsConnector>,
+    options: S3ClientOptions,
+    authority: Authority,
+}
+
+impl S3Client {
+    pub fn new(options: S3ClientOptions) -> Result<Self, Error> {
+        let expected_fingerprint = options.fingerprint.clone();
+        let verified_fingerprint = Arc::new(Mutex::new(None));
+        let trust_openssl_valid = Arc::new(Mutex::new(true));
+        let mut ssl_connector_builder = SslConnector::builder(SslMethod::tls())?;
+        ssl_connector_builder.set_verify_callback(
+            SslVerifyMode::PEER,
+            move |openssl_valid, context| match Self::verify_certificate_fingerprint(
+                openssl_valid,
+                context,
+                expected_fingerprint.clone(),
+                trust_openssl_valid.clone(),
+            ) {
+                Ok(None) => true,
+                Ok(Some(fingerprint)) => {
+                    *verified_fingerprint.lock().unwrap() = Some(fingerprint);
+                    true
+                }
+                Err(err) => {
+                    error!("certificate validation failed {err:#}");
+                    false
+                }
+            },
+        );
+
+        let mut http_connector = HttpConnector::new();
+        // want communication to object store backend api to always use https
+        http_connector.enforce_http(false);
+        http_connector.set_connect_timeout(Some(S3_HTTP_CONNECT_TIMEOUT));
+        let https_connector = HttpsConnector::with_connector(
+            http_connector,
+            ssl_connector_builder.build(),
+            S3_TCP_KEEPALIVE_TIME,
+        );
+        let client = Client::builder().build::<_, Body>(https_connector);
+        let authority = if let Some(port) = options.port {
+            format!("{}:{port}", options.host)
+        } else {
+            options.host.clone()
+        };
+        let authority = Authority::try_from(authority)?;
+
+        Ok(Self {
+            client,
+            options,
+            authority,
+        })
+    }
+
+    fn verify_certificate_fingerprint(
+        openssl_valid: bool,
+        context: &mut X509StoreContextRef,
+        expected_fingerprint: Option<String>,
+        trust_openssl: Arc<Mutex<bool>>,
+    ) -> Result<Option<String>, Error> {
+        let mut trust_openssl_valid = trust_openssl.lock().unwrap();
+
+        // only rely on openssl prevalidation if was not forced earlier
+        if openssl_valid && *trust_openssl_valid {
+            return Ok(None);
+        }
+
+        let certificate = match context.current_cert() {
+            Some(certificate) => certificate,
+            None => bail!("context lacks current certificate."),
+        };
+
+        if context.error_depth() > 0 {
+            *trust_openssl_valid = false;
+            return Ok(None);
+        }
+
+        let certificate_digest = certificate
+            .digest(MessageDigest::sha256())
+            .context("failed to calculate certificate digest")?;
+        let certificate_fingerprint = hex::encode(certificate_digest);
+        let certificate_fingerprint = certificate_fingerprint
+            .as_bytes()
+            .chunks(2)
+            .map(|v| std::str::from_utf8(v).unwrap())
+            .collect::<Vec<&str>>()
+            .join(":");
+
+        if let Some(expected_fingerprint) = expected_fingerprint {
+            let expected_fingerprint = expected_fingerprint.to_lowercase();
+            if expected_fingerprint == certificate_fingerprint {
+                return Ok(Some(certificate_fingerprint));
+            }
+        }
+
+        Err(format_err!(
+            "unexpected certificate fingerprint {certificate_fingerprint}"
+        ))
+    }
+}
diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
new file mode 100644
index 000000000..533ceab8e
--- /dev/null
+++ b/pbs-s3-client/src/lib.rs
@@ -0,0 +1,2 @@
+mod client;
+pub use client::{S3Client, S3ClientOptions};
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:33 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:33 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 08/42] s3 client: implement AWS
 signature v4 request authentication
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-9-c.ebner@proxmox.com>

The S3 API authenticates client requests by checking the
authentication signature provided by the requests `Authorization`
header. The latest AWS signature v4 signature is required for the
newest AWS regions [0] and most widely adapted [1-4], so rely soly on
that, not implementing older versions.

Adds helper methods to sign client requests, this includes encoding
and normalization of the headers, digest calculation of the request body
(if any) and signature generation.

[0] https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html
[1] https://docs.ceph.com/en/reef/radosgw/s3/authentication/#aws-signature-v4
[2] https://cloud.google.com/storage/docs/interoperability
[3] https://docs.wasabi.com/v1/docs/how-do-i-use-aws-signature-version-4-with-wasabi
[4] https://min.io/product/s3-compatibility

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-s3-client/Cargo.toml         |   2 +
 pbs-s3-client/src/aws_sign_v4.rs | 140 +++++++++++++++++++++++++++++++
 pbs-s3-client/src/lib.rs         |   1 +
 3 files changed, 143 insertions(+)
 create mode 100644 pbs-s3-client/src/aws_sign_v4.rs

diff --git a/pbs-s3-client/Cargo.toml b/pbs-s3-client/Cargo.toml
index 1999c3323..11189ea50 100644
--- a/pbs-s3-client/Cargo.toml
+++ b/pbs-s3-client/Cargo.toml
@@ -12,5 +12,7 @@ hex = { workspace = true, features = [ "serde" ] }
 hyper.workspace = true
 openssl.workspace = true
 tracing.workspace = true
+url.workspace = true
 
 proxmox-http.workspace = true
+proxmox-time.workspace = true
diff --git a/pbs-s3-client/src/aws_sign_v4.rs b/pbs-s3-client/src/aws_sign_v4.rs
new file mode 100644
index 000000000..8a538e868
--- /dev/null
+++ b/pbs-s3-client/src/aws_sign_v4.rs
@@ -0,0 +1,140 @@
+//! Helpers for request authentication using AWS signature version 4
+
+use anyhow::Error;
+use hyper::Body;
+use hyper::Request;
+use openssl::hash::MessageDigest;
+use openssl::pkey::{PKey, Private};
+use openssl::sha::sha256;
+use openssl::sign::Signer;
+use url::Url;
+
+use super::client::S3ClientOptions;
+
+pub(crate) const AWS_SIGN_V4_DATETIME_FORMAT: &str = "%Y%m%dT%H%M%SZ";
+
+const AWS_SIGN_V4_DATE_FORMAT: &str = "%Y%m%d";
+const AWS_SIGN_V4_SERVICE_S3: &str = "s3";
+const AWS_SIGN_V4_REQUEST_POSTFIX: &str = "aws4_request";
+
+/// Generate signature for S3 request authentication using AWS signature version 4.
+/// See: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html
+pub(crate) fn aws_sign_v4_signature(
+    request: &Request<Body>,
+    options: &S3ClientOptions,
+    epoch: i64,
+    payload_digest: &str,
+) -> Result<String, Error> {
+    // Include all headers in signature calculation since the reference docs note:
+    // "For the purpose of calculating an authorization signature, only the 'host' and any 'x-amz-*'
+    // headers are required. however, in order to prevent data tampering, you should consider
+    // including all the headers in the signature calculation."
+    // See https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html
+    let mut canonical_headers = Vec::new();
+    let mut signed_headers = Vec::new();
+    for (key, value) in request.headers() {
+        canonical_headers.push(format!(
+            "{}:{}",
+            // Header name has to be lower case, key.as_str() does guarantee that, see
+            // https://docs.rs/http/0.2.0/http/header/struct.HeaderName.html
+            key.as_str(),
+            // No need to trim since `HeaderValue` only allows visible UTF8 chars, see
+            // https://docs.rs/http/0.2.0/http/header/struct.HeaderValue.html
+            value.to_str()?,
+        ));
+        signed_headers.push(key.as_str());
+    }
+    canonical_headers.sort();
+    signed_headers.sort();
+    let signed_headers_string = signed_headers.join(";");
+
+    let mut canonical_queries = Url::parse(&request.uri().to_string())?
+        .query_pairs()
+        .map(|(key, value)| {
+            format!(
+                "{}={}",
+                aws_sign_v4_uri_encode(&key, false),
+                aws_sign_v4_uri_encode(&value, false),
+            )
+        })
+        .collect::<Vec<String>>();
+    canonical_queries.sort();
+
+    let canonical_request = format!(
+        "{}\n{}\n{}\n{}\n\n{}\n{}",
+        request.method().as_str(),
+        request.uri().path(),
+        canonical_queries.join("&"),
+        canonical_headers.join("\n"),
+        signed_headers_string,
+        payload_digest,
+    );
+
+    let date = proxmox_time::strftime_utc(AWS_SIGN_V4_DATE_FORMAT, epoch)?;
+    let datetime = proxmox_time::strftime_utc(AWS_SIGN_V4_DATETIME_FORMAT, epoch)?;
+
+    let credential_scope = format!(
+        "{date}/{}/{AWS_SIGN_V4_SERVICE_S3}/{AWS_SIGN_V4_REQUEST_POSTFIX}",
+        options.region,
+    );
+    let canonical_request_hash = hex::encode(sha256(canonical_request.as_bytes()));
+    let string_to_sign =
+        format!("AWS4-HMAC-SHA256\n{datetime}\n{credential_scope}\n{canonical_request_hash}");
+
+    let date_sign_key = PKey::hmac(format!("AWS4{}", options.secret_key).as_bytes())?;
+    let date_tag = hmac_sha256(&date_sign_key, date.as_bytes())?;
+
+    let region_sign_key = PKey::hmac(&date_tag)?;
+    let region_tag = hmac_sha256(&region_sign_key, options.region.as_bytes())?;
+
+    let service_sign_key = PKey::hmac(&region_tag)?;
+    let service_tag = hmac_sha256(&service_sign_key, AWS_SIGN_V4_SERVICE_S3.as_bytes())?;
+
+    let signing_key = PKey::hmac(&service_tag)?;
+    let signing_tag = hmac_sha256(&signing_key, AWS_SIGN_V4_REQUEST_POSTFIX.as_bytes())?;
+
+    let signature_key = PKey::hmac(&signing_tag)?;
+    let signature = hmac_sha256(&signature_key, string_to_sign.as_bytes())?;
+    let signature = hex::encode(&signature);
+
+    Ok(format!(
+        "AWS4-HMAC-SHA256 Credential={}/{credential_scope},SignedHeaders={signed_headers_string},Signature={signature}",
+        options.access_key,
+    ))
+}
+// Custom `uri_encode` implementation as recommended by AWS docs, since possible implementation
+// incompatibility with uri encoding libraries.
+// See: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
+pub(crate) fn aws_sign_v4_uri_encode(input: &str, is_object_key_name: bool) -> String {
+    // Assume up to  2 bytes per char max in output
+    let mut accumulator = String::with_capacity(2 * input.len());
+
+    input.chars().for_each(|char| {
+        match char {
+            // Unreserved characters, do not uri encode these bytes
+            'A'..='Z' | 'a'..='z' | '0'..='9' | '-' | '.' | '_' | '~' => accumulator.push(char),
+            // Space character is reserved, must be encoded as '%20', not '+'
+            ' ' => accumulator.push_str("%20"),
+            // Encode the forward slash character, '/', everywhere except in the object key name
+            '/' if !is_object_key_name => accumulator.push_str("%2F"),
+            '/' if is_object_key_name => accumulator.push(char),
+            // URI encoded byte is formed by a '%' and the two-digit hexadecimal value of the byte
+            // Letters in the hexadecimal value must be uppercase
+            _ => {
+                for byte in char.to_string().as_bytes() {
+                    accumulator.push_str(&format!("%{byte:02X}"));
+                }
+            }
+        }
+    });
+
+    accumulator
+}
+
+// Helper for hmac sha256 calculation
+fn hmac_sha256(key: &PKey<Private>, data: &[u8]) -> Result<Vec<u8>, Error> {
+    let mut signer = Signer::new(MessageDigest::sha256(), key)?;
+    signer.update(data)?;
+    let hmac = signer.sign_to_vec()?;
+    Ok(hmac)
+}
diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
index 533ceab8e..5a60b92ec 100644
--- a/pbs-s3-client/src/lib.rs
+++ b/pbs-s3-client/src/lib.rs
@@ -1,2 +1,3 @@
+mod aws_sign_v4;
 mod client;
 pub use client::{S3Client, S3ClientOptions};
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:36 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:36 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 11/42] s3 client: add helper to
 parse http date headers
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-12-c.ebner@proxmox.com>

Add a helper to parse the preferred date/time format for http `Date`
headers as specified in RFC 2616 [0], which is a fixed-length subset
of the format specified in RFC 1123 [1], itself being a followup to
RFC 822 [2]. Does not implement the format as described in the
obsolete RFC 850 [3].

This allows to parse the `Date` and `Last-Modified` headers of S3 API
responses.

[0] https://datatracker.ietf.org/doc/html/rfc2616#section-3.3
[1] https://datatracker.ietf.org/doc/html/rfc1123#section-5.2.14
[2] https://datatracker.ietf.org/doc/html/rfc822#section-5
[3] https://datatracker.ietf.org/doc/html/rfc850

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-s3-client/src/lib.rs | 97 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 96 insertions(+), 1 deletion(-)

diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
index dbe4bebcc..b3e539bdd 100644
--- a/pbs-s3-client/src/lib.rs
+++ b/pbs-s3-client/src/lib.rs
@@ -6,7 +6,12 @@ pub use object_key::{S3ObjectKey, S3_CONTENT_PREFIX};
 
 use std::time::Duration;
 
-use anyhow::{anyhow, bail, Error};
+use anyhow::{anyhow, bail, Context, Error};
+
+const VALID_DAYS_OF_WEEK: [&str; 7] = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"];
+const VALID_MONTHS: [&str; 12] = [
+    "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec",
+];
 
 #[derive(Debug)]
 pub struct LastModifiedTimestamp {
@@ -23,3 +28,93 @@ impl std::str::FromStr for LastModifiedTimestamp {
 }
 
 serde_plain::derive_deserialize_from_fromstr!(LastModifiedTimestamp, "last modified timestamp");
+
+/// Preferred date format specified by RFC2616, given as fixed-length
+/// subset of RFC1123, which itself is a followup to RFC822.
+///
+/// https://datatracker.ietf.org/doc/html/rfc2616#section-3.3
+/// https://datatracker.ietf.org/doc/html/rfc1123#section-5.2.14
+/// https://datatracker.ietf.org/doc/html/rfc822#section-5
+#[derive(Debug)]
+pub struct HttpDate {
+    epoch: i64,
+}
+
+impl HttpDate {
+    pub fn to_duration(&self) -> Result<Duration, Error> {
+        let seconds = u64::try_from(self.epoch)?;
+        Ok(Duration::from_secs(seconds))
+    }
+}
+
+impl std::str::FromStr for HttpDate {
+    type Err = Error;
+
+    fn from_str(timestamp: &str) -> Result<Self, Self::Err> {
+        let input = timestamp.as_bytes();
+        if input.len() != 29 {
+            bail!("unexpected length: got {}", input.len());
+        }
+
+        let expect = |pos: usize, c: u8| {
+            if input[pos] != c {
+                bail!("unexpected char at pos {pos}");
+            }
+            Ok(())
+        };
+
+        let digit = |pos: usize| -> Result<i32, Error> {
+            let digit = input[pos] as i32;
+            if !(48..=57).contains(&digit) {
+                bail!("unexpected char at pos {pos}");
+            }
+            Ok(digit - 48)
+        };
+
+        fn check_max(i: i32, max: i32) -> Result<i32, Error> {
+            if i > max {
+                bail!("value too large ({i} > {max})");
+            }
+            Ok(i)
+        }
+
+        let mut tm = proxmox_time::TmEditor::new(true);
+
+        if !VALID_DAYS_OF_WEEK
+            .iter()
+            .any(|valid| valid.as_bytes() == &input[0..3])
+        {
+            bail!("unexpected day of week, got {:?}", &input[0..3]);
+        }
+
+        expect(3, b',').context("unexpected separator after day of week")?;
+        expect(4, b' ').context("missing space after day of week separator")?;
+        tm.set_mday(check_max(digit(5)? * 10 + digit(6)?, 31)?)?;
+        expect(7, b' ').context("unexpected separator after day")?;
+        if let Some(month) = VALID_MONTHS
+            .iter()
+            .position(|month| month.as_bytes() == &input[8..11])
+        {
+            // valid conversion to i32, position stems from fixed size array of 12 months.
+            tm.set_mon(check_max(month as i32 + 1, 12)?)?;
+        } else {
+            bail!("invalid month");
+        }
+        expect(11, b' ').context("unexpected separator after month")?;
+        tm.set_year(digit(12)? * 1000 + digit(13)? * 100 + digit(14)? * 10 + digit(15)?)?;
+        expect(16, b' ').context("unexpected separator after year")?;
+        tm.set_hour(check_max(digit(17)? * 10 + digit(18)?, 23)?)?;
+        expect(19, b':').context("unexpected separator after hour")?;
+        tm.set_min(check_max(digit(20)? * 10 + digit(21)?, 59)?)?;
+        expect(22, b':').context("unexpected separator after minute")?;
+        tm.set_sec(check_max(digit(23)? * 10 + digit(24)?, 60)?)?;
+        expect(25, b' ').context("unexpected separator after second")?;
+        if !input.ends_with(b"GMT") {
+            bail!("unexpected timezone");
+        }
+
+        let epoch = tm.into_epoch()?;
+
+        Ok(Self { epoch })
+    }
+}
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:38 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:38 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 13/42] config: introduce s3
 object store client configuration
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-14-c.ebner@proxmox.com>

Adds the client configuration for s3 object store as dedicated
configuration files, with secrets being stored separately from the
regular configuration and excluded from api responses for security
reasons.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-config/src/lib.rs |  1 +
 pbs-config/src/s3.rs  | 82 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)
 create mode 100644 pbs-config/src/s3.rs

diff --git a/pbs-config/src/lib.rs b/pbs-config/src/lib.rs
index 9c4d77c24..d03c079ab 100644
--- a/pbs-config/src/lib.rs
+++ b/pbs-config/src/lib.rs
@@ -10,6 +10,7 @@ pub mod network;
 pub mod notifications;
 pub mod prune;
 pub mod remote;
+pub mod s3;
 pub mod sync;
 pub mod tape_job;
 pub mod token_shadow;
diff --git a/pbs-config/src/s3.rs b/pbs-config/src/s3.rs
new file mode 100644
index 000000000..5fce5034d
--- /dev/null
+++ b/pbs-config/src/s3.rs
@@ -0,0 +1,82 @@
+use std::collections::HashMap;
+use std::sync::LazyLock;
+
+use anyhow::Error;
+
+use proxmox_schema::*;
+use proxmox_section_config::{SectionConfig, SectionConfigData, SectionConfigPlugin};
+
+use pbs_api_types::{S3ClientConfig, S3ClientSecretsConfig, JOB_ID_SCHEMA};
+
+use crate::{open_backup_lockfile, replace_backup_config, BackupLockGuard};
+
+pub static CONFIG: LazyLock<SectionConfig> = LazyLock::new(init);
+
+fn init() -> SectionConfig {
+    let obj_schema = match S3ClientConfig::API_SCHEMA {
+        Schema::Object(ref obj_schema) => obj_schema,
+        _ => unreachable!(),
+    };
+    let secrets_obj_schema = match S3ClientSecretsConfig::API_SCHEMA {
+        Schema::Object(ref obj_schema) => obj_schema,
+        _ => unreachable!(),
+    };
+
+    let plugin =
+        SectionConfigPlugin::new("s3client".to_string(), Some(String::from("id")), obj_schema);
+    let secrets_plugin = SectionConfigPlugin::new(
+        "s3secrets".to_string(),
+        Some(String::from("secrets-id")),
+        secrets_obj_schema,
+    );
+    let mut config = SectionConfig::new(&JOB_ID_SCHEMA);
+    config.register_plugin(plugin);
+    config.register_plugin(secrets_plugin);
+
+    config
+}
+
+pub const S3_CFG_FILENAME: &str = "/etc/proxmox-backup/s3.cfg";
+pub const S3_SECRETS_CFG_FILENAME: &str = "/etc/proxmox-backup/s3-secrets.cfg";
+pub const S3_CFG_LOCKFILE: &str = "/etc/proxmox-backup/.s3.lck";
+
+/// Get exclusive lock
+pub fn lock_config() -> Result<BackupLockGuard, Error> {
+    open_backup_lockfile(S3_CFG_LOCKFILE, None, true)
+}
+
+pub fn config() -> Result<(SectionConfigData, [u8; 32]), Error> {
+    parse_config(S3_CFG_FILENAME)
+}
+
+pub fn secrets_config() -> Result<(SectionConfigData, [u8; 32]), Error> {
+    parse_config(S3_SECRETS_CFG_FILENAME)
+}
+
+pub fn save_config(config: &SectionConfigData, secrets: &SectionConfigData) -> Result<(), Error> {
+    let raw = CONFIG.write(S3_CFG_FILENAME, config)?;
+    replace_backup_config(S3_CFG_FILENAME, raw.as_bytes())?;
+
+    let secrets_raw = CONFIG.write(S3_SECRETS_CFG_FILENAME, secrets)?;
+    // Secrets are stored with `backup` permissions to allow reading from
+    // not protected api endpoints as well.
+    replace_backup_config(S3_SECRETS_CFG_FILENAME, secrets_raw.as_bytes())?;
+
+    Ok(())
+}
+
+// shell completion helper
+pub fn complete_s3_client_id(_arg: &str, _param: &HashMap<String, String>) -> Vec<String> {
+    match config() {
+        Ok((data, _digest)) => data.sections.keys().map(|id| id.to_string()).collect(),
+        Err(_) => Vec::new(),
+    }
+}
+
+fn parse_config(path: &str) -> Result<(SectionConfigData, [u8; 32]), Error> {
+    let content = proxmox_sys::fs::file_read_optional_string(path)?;
+    let content = content.unwrap_or_default();
+    let digest = openssl::sha::sha256(content.as_bytes());
+    let data = CONFIG.parse(path, &content)?;
+    Ok((data, digest))
+}
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:45 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:45 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 20/42] api: backup:
 conditionally upload blobs to S3 object store backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-21-c.ebner@proxmox.com>

Upload blobs to both, the local datastore cache and the S3 object
store if s3 is configured as backend.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/environment.rs | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 8919b919a..393a8351d 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -581,6 +581,31 @@ impl BackupEnvironment {
         let blob = DataBlob::load_from_reader(&mut &data[..])?;
 
         let raw_data = blob.raw_data();
+        if let DatastoreBackend::S3(s3_client) = &self.backend {
+            let data = Body::from(raw_data.to_vec());
+            let mut object_key = self.backup_dir.relative_path();
+            object_key.push(file_name);
+            let object_key = object_key
+                .as_os_str()
+                .to_str()
+                .ok_or_else(|| format_err!("invalid path"))?;
+            match proxmox_async::runtime::block_on(s3_client.put_object(object_key.into(), data))? {
+                PutObjectResponse::PreconditionFailed => {
+                    self.log(format!(
+                        "Upload of blob failed, object {object_key} already present."
+                    ));
+                    bail!("upload of blob failed");
+                }
+                PutObjectResponse::NeedsRetry => {
+                    self.log("Upload of blob failed, reupload required.");
+                    bail!("concurrent operation, reupload required");
+                }
+                PutObjectResponse::Success(_content) => {
+                    self.log(format!("Uploaded blob to object store: {object_key}"))
+                }
+            }
+        }
+
         replace_file(&path, raw_data, CreateOptions::new(), false)?;
 
         self.log(format!(
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:44 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:44 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 19/42] api: backup:
 conditionally upload chunks to S3 object store backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-20-c.ebner@proxmox.com>

Upload fixed and dynamic sized chunks to either the filesystem or
the S3 object store, depending on the configured backend.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/upload_chunk.rs | 44 ++++++++++++++++++++++-----------
 1 file changed, 29 insertions(+), 15 deletions(-)

diff --git a/src/api2/backup/upload_chunk.rs b/src/api2/backup/upload_chunk.rs
index 20259660a..838eec1fa 100644
--- a/src/api2/backup/upload_chunk.rs
+++ b/src/api2/backup/upload_chunk.rs
@@ -15,7 +15,8 @@ use proxmox_sortable_macro::sortable;
 
 use pbs_api_types::{BACKUP_ARCHIVE_NAME_SCHEMA, CHUNK_DIGEST_SCHEMA};
 use pbs_datastore::file_formats::{DataBlobHeader, EncryptedDataBlobHeader};
-use pbs_datastore::{DataBlob, DataStore};
+use pbs_datastore::{DataBlob, DataStore, DatastoreBackend};
+use pbs_s3_client::PutObjectResponse;
 use pbs_tools::json::{required_integer_param, required_string_param};
 
 use super::environment::*;
@@ -153,16 +154,10 @@ fn upload_fixed_chunk(
 ) -> ApiResponseFuture {
     async move {
         let wid = required_integer_param(&param, "wid")? as usize;
-        let size = required_integer_param(&param, "size")? as u32;
-        let encoded_size = required_integer_param(&param, "encoded-size")? as u32;
-
-        let digest_str = required_string_param(&param, "digest")?;
-        let digest = <[u8; 32]>::from_hex(digest_str)?;
-
         let env: &BackupEnvironment = rpcenv.as_ref();
 
         let (digest, size, compressed_size, is_duplicate) =
-            UploadChunk::new(req_body, env.datastore.clone(), digest, size, encoded_size).await?;
+            upload_to_backend(req_body, param, env).await?;
 
         env.register_fixed_chunk(wid, digest, size, compressed_size, is_duplicate)?;
         let digest_str = hex::encode(digest);
@@ -222,16 +217,10 @@ fn upload_dynamic_chunk(
 ) -> ApiResponseFuture {
     async move {
         let wid = required_integer_param(&param, "wid")? as usize;
-        let size = required_integer_param(&param, "size")? as u32;
-        let encoded_size = required_integer_param(&param, "encoded-size")? as u32;
-
-        let digest_str = required_string_param(&param, "digest")?;
-        let digest = <[u8; 32]>::from_hex(digest_str)?;
-
         let env: &BackupEnvironment = rpcenv.as_ref();
 
         let (digest, size, compressed_size, is_duplicate) =
-            UploadChunk::new(req_body, env.datastore.clone(), digest, size, encoded_size).await?;
+            upload_to_backend(req_body, param, env).await?;
 
         env.register_dynamic_chunk(wid, digest, size, compressed_size, is_duplicate)?;
         let digest_str = hex::encode(digest);
@@ -243,6 +232,31 @@ fn upload_dynamic_chunk(
     .boxed()
 }
 
+async fn upload_to_backend(
+    req_body: Body,
+    param: Value,
+    env: &BackupEnvironment,
+) -> Result<([u8; 32], u32, u32, bool), Error> {
+    let size = required_integer_param(&param, "size")? as u32;
+    let encoded_size = required_integer_param(&param, "encoded-size")? as u32;
+    let digest_str = required_string_param(&param, "digest")?;
+    let digest = <[u8; 32]>::from_hex(digest_str)?;
+
+    match &env.backend {
+        DatastoreBackend::Filesystem => {
+            UploadChunk::new(req_body, env.datastore.clone(), digest, size, encoded_size).await
+        }
+        DatastoreBackend::S3(s3_client) => {
+            let is_duplicate = match s3_client.put_object(digest.into(), req_body).await? {
+                PutObjectResponse::PreconditionFailed => true,
+                PutObjectResponse::NeedsRetry => bail!("concurrent operation, reupload required"),
+                PutObjectResponse::Success(_content) => false,
+            };
+            Ok((digest, size, encoded_size, is_duplicate))
+        }
+    }
+}
+
 pub const API_METHOD_UPLOAD_SPEEDTEST: ApiMethod = ApiMethod::new(
     &ApiHandler::AsyncHttp(&upload_speedtest),
     &ObjectSchema::new("Test upload speed.", &[]),
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:41 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:41 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 16/42] api/bin: add endpoint and
 command to check s3 client connection
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-17-c.ebner@proxmox.com>

Adds a dedicated api endpoint and a proxmox-backup-manager command to
check if the configured S3 client can reach the bucket.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/mod.rs                 |  2 +
 src/api2/admin/s3.rs                  | 72 +++++++++++++++++++++++++++
 src/bin/proxmox-backup-manager.rs     |  1 +
 src/bin/proxmox_backup_manager/mod.rs |  2 +
 src/bin/proxmox_backup_manager/s3.rs  | 34 +++++++++++++
 5 files changed, 111 insertions(+)
 create mode 100644 src/api2/admin/s3.rs
 create mode 100644 src/bin/proxmox_backup_manager/s3.rs

diff --git a/src/api2/admin/mod.rs b/src/api2/admin/mod.rs
index a1c49f8e2..7694de4b9 100644
--- a/src/api2/admin/mod.rs
+++ b/src/api2/admin/mod.rs
@@ -9,6 +9,7 @@ pub mod gc;
 pub mod metrics;
 pub mod namespace;
 pub mod prune;
+pub mod s3;
 pub mod sync;
 pub mod traffic_control;
 pub mod verify;
@@ -19,6 +20,7 @@ const SUBDIRS: SubdirMap = &sorted!([
     ("metrics", &metrics::ROUTER),
     ("prune", &prune::ROUTER),
     ("gc", &gc::ROUTER),
+    ("s3", &s3::ROUTER),
     ("sync", &sync::ROUTER),
     ("traffic-control", &traffic_control::ROUTER),
     ("verify", &verify::ROUTER),
diff --git a/src/api2/admin/s3.rs b/src/api2/admin/s3.rs
new file mode 100644
index 000000000..229bcc535
--- /dev/null
+++ b/src/api2/admin/s3.rs
@@ -0,0 +1,72 @@
+//! S3 bucket operations
+
+use anyhow::{Context, Error};
+use hyper::Body;
+use serde_json::Value;
+
+use proxmox_router::{list_subdirs_api_method, Permission, Router, RpcEnvironment, SubdirMap};
+use proxmox_schema::*;
+use proxmox_sortable_macro::sortable;
+
+use pbs_api_types::{S3ClientConfig, S3ClientSecretsConfig, PRIV_SYS_MODIFY, S3_CLIENT_ID_SCHEMA};
+
+#[api(
+    input: {
+        properties: {
+            "s3-client-id": {
+                schema: S3_CLIENT_ID_SCHEMA ,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&[], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Perform basic sanity check for given s3 client configuration
+pub async fn check(s3_client_id: String, _rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+    let (config, _digest) = pbs_config::s3::config()?;
+    let config: S3ClientConfig = config
+        .lookup("s3client", &s3_client_id)
+        .context("config lookup failed")?;
+    let (secrets, _secrets_digest) = pbs_config::s3::secrets_config()?;
+    let secrets: S3ClientSecretsConfig = secrets
+        .lookup("s3secrets", &s3_client_id)
+        .context("secrets lookup failed")?;
+
+    let options = pbs_s3_client::S3ClientOptions {
+        host: config.host,
+        port: config.port,
+        bucket: config.bucket,
+        region: config.region.unwrap_or_default(),
+        fingerprint: config.fingerprint,
+        access_key: config.access_key,
+        secret_key: secrets.secret_key,
+    };
+
+    let test_object_key = ".s3-client-test";
+    let client = pbs_s3_client::S3Client::new(options).context("client creation failed")?;
+    client.head_bucket().await.context("head object failed")?;
+    client
+        .put_object(test_object_key.into(), Body::empty())
+        .await
+        .context("put object failed")?;
+    client
+        .get_object(test_object_key.into())
+        .await
+        .context("get object failed")?;
+    client
+        .delete_object(test_object_key.into())
+        .await
+        .context("delete object failed")?;
+
+    Ok(Value::Null)
+}
+
+#[sortable]
+const S3_OPERATION_SUBDIRS: SubdirMap = &[("check", &Router::new().get(&API_METHOD_CHECK))];
+
+const S3_OPERATION_ROUTER: Router = Router::new()
+    .get(&list_subdirs_api_method!(S3_OPERATION_SUBDIRS))
+    .subdirs(S3_OPERATION_SUBDIRS);
+
+pub const ROUTER: Router = Router::new().match_all("s3-client-id", &S3_OPERATION_ROUTER);
diff --git a/src/bin/proxmox-backup-manager.rs b/src/bin/proxmox-backup-manager.rs
index d4363e717..68d87c676 100644
--- a/src/bin/proxmox-backup-manager.rs
+++ b/src/bin/proxmox-backup-manager.rs
@@ -677,6 +677,7 @@ async fn run() -> Result<(), Error> {
         .insert("garbage-collection", garbage_collection_commands())
         .insert("acme", acme_mgmt_cli())
         .insert("cert", cert_mgmt_cli())
+        .insert("s3", s3_commands())
         .insert("subscription", subscription_commands())
         .insert("sync-job", sync_job_commands())
         .insert("verify-job", verify_job_commands())
diff --git a/src/bin/proxmox_backup_manager/mod.rs b/src/bin/proxmox_backup_manager/mod.rs
index 9b5c73e9a..312a6db6b 100644
--- a/src/bin/proxmox_backup_manager/mod.rs
+++ b/src/bin/proxmox_backup_manager/mod.rs
@@ -26,6 +26,8 @@ mod prune;
 pub use prune::*;
 mod remote;
 pub use remote::*;
+mod s3;
+pub use s3::*;
 mod subscription;
 pub use subscription::*;
 mod sync;
diff --git a/src/bin/proxmox_backup_manager/s3.rs b/src/bin/proxmox_backup_manager/s3.rs
new file mode 100644
index 000000000..a92d3d1b2
--- /dev/null
+++ b/src/bin/proxmox_backup_manager/s3.rs
@@ -0,0 +1,34 @@
+use pbs_api_types::S3_CLIENT_ID_SCHEMA;
+use proxmox_router::{cli::*, RpcEnvironment};
+use proxmox_schema::api;
+
+use proxmox_backup::api2;
+
+use anyhow::Error;
+use serde_json::Value;
+
+#[api(
+    input: {
+        properties: {
+            "s3-client-id": {
+                schema: S3_CLIENT_ID_SCHEMA,
+            },
+        },
+    },
+)]
+/// Perform basic sanity checks for given S3 client configuration
+async fn check(s3_client_id: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Error> {
+    api2::admin::s3::check(s3_client_id, rpcenv).await?;
+    Ok(Value::Null)
+}
+
+pub fn s3_commands() -> CommandLineInterface {
+    let cmd_def = CliCommandMap::new().insert(
+        "check",
+        CliCommand::new(&API_METHOD_CHECK)
+            .arg_param(&["s3-client-id"])
+            .completion_cb("s3-client-id", pbs_config::s3::complete_s3_client_id),
+    );
+
+    cmd_def.into()
+}
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:43 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:43 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 18/42] api: backup: store
 datastore backend in runtime environment
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-19-c.ebner@proxmox.com>

Get and store the datastore's backend during creation of the backup
runtime environment and upload the chunks to the local filesystem or
s3 object store based on the backend variant.

By storing the backend variant in the environment the s3 client is
instantiated only once and reused for all api calls in the same
backup http/2 connection.

Refactor the upgrade method by moving all logic into the async block,
such that the now possible error on backup environment creation gets
propagated to the thread spawn call side.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/environment.rs | 12 +++--
 src/api2/backup/mod.rs         | 99 +++++++++++++++++-----------------
 2 files changed, 57 insertions(+), 54 deletions(-)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 6cd29f512..8919b919a 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -15,7 +15,8 @@ use pbs_api_types::Authid;
 use pbs_datastore::backup_info::{BackupDir, BackupInfo};
 use pbs_datastore::dynamic_index::DynamicIndexWriter;
 use pbs_datastore::fixed_index::FixedIndexWriter;
-use pbs_datastore::{DataBlob, DataStore};
+use pbs_datastore::{DataBlob, DataStore, DatastoreBackend};
+use pbs_s3_client::PutObjectResponse;
 use proxmox_rest_server::{formatter::*, WorkerTask};
 
 use crate::backup::VerifyWorker;
@@ -115,6 +116,7 @@ pub struct BackupEnvironment {
     pub datastore: Arc<DataStore>,
     pub backup_dir: BackupDir,
     pub last_backup: Option<BackupInfo>,
+    pub backend: DatastoreBackend,
     state: Arc<Mutex<SharedBackupState>>,
 }
 
@@ -125,7 +127,7 @@ impl BackupEnvironment {
         worker: Arc<WorkerTask>,
         datastore: Arc<DataStore>,
         backup_dir: BackupDir,
-    ) -> Self {
+    ) -> Result<Self, Error> {
         let state = SharedBackupState {
             finished: false,
             uid_counter: 0,
@@ -137,7 +139,8 @@ impl BackupEnvironment {
             backup_stat: UploadStatistic::new(),
         };
 
-        Self {
+        let backend = datastore.backend()?;
+        Ok(Self {
             result_attributes: json!({}),
             env_type,
             auth_id,
@@ -147,8 +150,9 @@ impl BackupEnvironment {
             formatter: JSON_FORMATTER,
             backup_dir,
             last_backup: None,
+            backend,
             state: Arc::new(Mutex::new(state)),
-        }
+        })
     }
 
     /// Register a Chunk with associated length.
diff --git a/src/api2/backup/mod.rs b/src/api2/backup/mod.rs
index 567bca3ef..2c6afca41 100644
--- a/src/api2/backup/mod.rs
+++ b/src/api2/backup/mod.rs
@@ -185,7 +185,8 @@ fn upgrade_to_backup_protocol(
             }
 
             // lock last snapshot to prevent forgetting/pruning it during backup
-            let guard = last.backup_dir
+            let guard = last
+                .backup_dir
                 .lock_shared()
                 .with_context(|| format!("while locking last snapshot during backup '{last:?}'"))?;
             Some(guard)
@@ -204,14 +205,14 @@ fn upgrade_to_backup_protocol(
             Some(worker_id),
             auth_id.to_string(),
             true,
-            move |worker| {
+            move |worker| async move {
                 let mut env = BackupEnvironment::new(
                     env_type,
                     auth_id,
                     worker.clone(),
                     datastore,
                     backup_dir,
-                );
+                )?;
 
                 env.debug = debug;
                 env.last_backup = last_backup;
@@ -264,55 +265,53 @@ fn upgrade_to_backup_protocol(
                     });
                 let mut abort_future = abort_future.map(|_| Err(format_err!("task aborted")));
 
-                async move {
-                    // keep flock until task ends
-                    let _group_guard = _group_guard;
-                    let snap_guard = snap_guard;
-                    let _last_guard = _last_guard;
-
-                    let res = select! {
-                        req = req_fut => req,
-                        abrt = abort_future => abrt,
-                    };
-                    if benchmark {
-                        env.log("benchmark finished successfully");
-                        proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
-                        return Ok(());
+                // keep flock until task ends
+                let _group_guard = _group_guard;
+                let snap_guard = snap_guard;
+                let _last_guard = _last_guard;
+
+                let res = select! {
+                    req = req_fut => req,
+                    abrt = abort_future => abrt,
+                };
+                if benchmark {
+                    env.log("benchmark finished successfully");
+                    proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
+                    return Ok(());
+                }
+
+                let verify = |env: BackupEnvironment| {
+                    if let Err(err) = env.verify_after_complete(snap_guard) {
+                        env.log(format!(
+                            "backup finished, but starting the requested verify task failed: {}",
+                            err
+                        ));
                     }
+                };
 
-                    let verify = |env: BackupEnvironment| {
-                        if let Err(err) = env.verify_after_complete(snap_guard) {
-                            env.log(format!(
-                                "backup finished, but starting the requested verify task failed: {}",
-                                err
-                            ));
-                        }
-                    };
-
-                    match (res, env.ensure_finished()) {
-                        (Ok(_), Ok(())) => {
-                            env.log("backup finished successfully");
-                            verify(env);
-                            Ok(())
-                        }
-                        (Err(err), Ok(())) => {
-                            // ignore errors after finish
-                            env.log(format!("backup had errors but finished: {}", err));
-                            verify(env);
-                            Ok(())
-                        }
-                        (Ok(_), Err(err)) => {
-                            env.log(format!("backup ended and finish failed: {}", err));
-                            env.log("removing unfinished backup");
-                            proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
-                            Err(err)
-                        }
-                        (Err(err), Err(_)) => {
-                            env.log(format!("backup failed: {}", err));
-                            env.log("removing failed backup");
-                            proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
-                            Err(err)
-                        }
+                match (res, env.ensure_finished()) {
+                    (Ok(_), Ok(())) => {
+                        env.log("backup finished successfully");
+                        verify(env);
+                        Ok(())
+                    }
+                    (Err(err), Ok(())) => {
+                        // ignore errors after finish
+                        env.log(format!("backup had errors but finished: {}", err));
+                        verify(env);
+                        Ok(())
+                    }
+                    (Ok(_), Err(err)) => {
+                        env.log(format!("backup ended and finish failed: {}", err));
+                        env.log("removing unfinished backup");
+                        proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
+                        Err(err)
+                    }
+                    (Err(err), Err(_)) => {
+                        env.log(format!("backup failed: {}", err));
+                        env.log("removing failed backup");
+                        proxmox_async::runtime::block_in_place(|| env.remove_backup())?;
+                        Err(err)
                     }
                 }
             },
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:53 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:53 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 28/42] datastore: create
 namespace marker in S3 backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-29-c.ebner@proxmox.com>

The S3 object store only allows to store objects, referenced by their
key. For backup namespaces datastores however use directories, so
they cannot be represented as one to one mapping.

Instead, create an empty marker file for each namespace and operate
based on that.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 22 +++++++++++++++++++++-
 1 file changed, 21 insertions(+), 1 deletion(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 42d27d249..ab5c22501 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -8,7 +8,7 @@ use std::time::Duration;
 
 use anyhow::{bail, format_err, Context, Error};
 use nix::unistd::{unlinkat, UnlinkatFlags};
-use pbs_s3_client::{S3Client, S3ClientOptions};
+use pbs_s3_client::{PutObjectResponse, S3Client, S3ClientOptions};
 use pbs_tools::lru_cache::LruCache;
 use tracing::{info, warn};
 
@@ -42,6 +42,8 @@ use crate::DataBlob;
 static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
     LazyLock::new(|| Mutex::new(HashMap::new()));
 
+const NAMESPACE_MARKER_FILENAME: &str = ".namespace";
+
 /// checks if auth_id is owner, or, if owner is a token, if
 /// auth_id is the user of the token
 pub fn check_backup_owner(owner: &Authid, auth_id: &Authid) -> Result<(), Error> {
@@ -590,6 +592,24 @@ impl DataStore {
         // construct ns before mkdir to enforce max-depth and name validity
         let ns = BackupNamespace::from_parent_ns(parent, name)?;
 
+        if let DatastoreBackend::S3(s3_client) = self.backend()? {
+            let marker = ns.path().join(NAMESPACE_MARKER_FILENAME);
+            let namespace_marker = marker
+                .to_str()
+                .ok_or_else(|| format_err!("unexpected namespace path"))?;
+
+            let response = proxmox_async::runtime::block_on(
+                s3_client.put_object(namespace_marker.into(), hyper::body::Body::empty()),
+            )?;
+            match response {
+                PutObjectResponse::NeedsRetry => bail!("failed to create namespace, needs retry"),
+                PutObjectResponse::PreconditionFailed => {
+                    bail!("failed to create namespace, precondition failed")
+                }
+                PutObjectResponse::Success(_) => (),
+            }
+        }
+
         let mut ns_full_path = self.base_path();
         ns_full_path.push(ns.path());
 
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:48 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:48 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 23/42] sync: pull: conditionally
 upload content to S3 backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-24-c.ebner@proxmox.com>

If the datastore is backed by an S3 object store, not only insert the
pulled contents to the local cache store, but also upload it to the
S3 backend.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/server/pull.rs | 58 ++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 56 insertions(+), 2 deletions(-)

diff --git a/src/server/pull.rs b/src/server/pull.rs
index b1724c142..f36efd7c8 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -8,6 +8,7 @@ use std::time::SystemTime;
 
 use anyhow::{bail, format_err, Error};
 use proxmox_human_byte::HumanByte;
+use tokio::io::AsyncReadExt;
 use tracing::info;
 
 use pbs_api_types::{
@@ -24,7 +25,7 @@ use pbs_datastore::fixed_index::FixedIndexReader;
 use pbs_datastore::index::IndexFile;
 use pbs_datastore::manifest::{BackupManifest, FileInfo};
 use pbs_datastore::read_chunk::AsyncReadChunk;
-use pbs_datastore::{check_backup_owner, DataStore, StoreProgress};
+use pbs_datastore::{check_backup_owner, DataStore, DatastoreBackend, StoreProgress};
 use pbs_tools::sha::sha256;
 
 use super::sync::{
@@ -167,7 +168,18 @@ async fn pull_index_chunks<I: IndexFile>(
         move |(chunk, digest, size): (DataBlob, [u8; 32], u64)| {
             // println!("verify and write {}", hex::encode(&digest));
             chunk.verify_unencrypted(size as usize, &digest)?;
-            target2.insert_chunk(&chunk, &digest)?;
+            match target2.backend()? {
+                DatastoreBackend::Filesystem => {
+                    target2.insert_chunk(&chunk, &digest)?;
+                }
+                DatastoreBackend::S3(s3_client) => {
+                    let data = chunk.raw_data().to_vec();
+                    let upload_body = hyper::Body::from(data);
+                    proxmox_async::runtime::block_on(
+                        s3_client.put_object(digest.into(), upload_body),
+                    )?;
+                }
+            }
             Ok(())
         },
     );
@@ -331,6 +343,20 @@ async fn pull_single_archive<'a>(
     if let Err(err) = std::fs::rename(&tmp_path, &path) {
         bail!("Atomic rename file {:?} failed - {}", path, err);
     }
+    if let DatastoreBackend::S3(s3_client) = snapshot.datastore().backend()? {
+        let archive_path = snapshot.relative_path().join(archive_name);
+        let object_key = archive_path
+            .as_os_str()
+            .to_str()
+            .ok_or_else(|| format_err!("invalid archive path"))?;
+
+        let archive = tokio::fs::File::open(&path).await?;
+        let mut reader = tokio::io::BufReader::new(archive);
+        let mut contents = Vec::new();
+        reader.read_to_end(&mut contents).await?;
+        let data = hyper::body::Body::from(contents);
+        s3_client.put_object(object_key.into(), data).await?;
+    }
     Ok(sync_stats)
 }
 
@@ -401,6 +427,7 @@ async fn pull_snapshot<'a>(
         }
     }
 
+    let manifest_data = tmp_manifest_blob.raw_data().to_vec();
     let manifest = BackupManifest::try_from(tmp_manifest_blob)?;
 
     if ignore_not_verified_or_encrypted(
@@ -467,9 +494,36 @@ async fn pull_snapshot<'a>(
     if let Err(err) = std::fs::rename(&tmp_manifest_name, &manifest_name) {
         bail!("Atomic rename file {:?} failed - {}", manifest_name, err);
     }
+    if let DatastoreBackend::S3(s3_client) = snapshot.datastore().backend()? {
+        let object_path = snapshot.relative_path().join(MANIFEST_BLOB_NAME.as_ref());
+        let object_key = object_path
+            .as_os_str()
+            .to_str()
+            .ok_or_else(|| format_err!("invalid archive path"))?;
+
+        let data = hyper::body::Body::from(manifest_data);
+        s3_client.put_object(object_key.into(), data).await?;
+    }
 
     if !client_log_name.exists() {
         reader.try_download_client_log(&client_log_name).await?;
+        if client_log_name.exists() {
+            if let DatastoreBackend::S3(s3_client) = snapshot.datastore().backend()? {
+                let object_path = snapshot.relative_path().join(CLIENT_LOG_BLOB_NAME.as_ref());
+                let object_key = object_path
+                    .as_os_str()
+                    .to_str()
+                    .ok_or_else(|| format_err!("invalid archive path"))?;
+
+                let log_file = tokio::fs::File::open(&client_log_name).await?;
+                let mut reader = tokio::io::BufReader::new(log_file);
+                let mut contents = Vec::new();
+                reader.read_to_end(&mut contents).await?;
+
+                let data = hyper::body::Body::from(contents);
+                s3_client.put_object(object_key.into(), data).await?;
+            }
+        }
     };
     snapshot
         .cleanup_unreferenced_files(&manifest)
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:51 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:51 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 26/42] verify worker: add
 datastore backed to verify worker
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-27-c.ebner@proxmox.com>

In order to fetch chunks from an S3 compatible object store,
instantiate and store the s3 client in the verify worker by storing
the datastore's backend. This allows to reuse the same instance for
the whole verification task.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs    |  2 +-
 src/api2/backup/environment.rs |  2 +-
 src/backup/verify.rs           | 14 ++++++++++----
 src/server/verify_job.rs       |  2 +-
 4 files changed, 13 insertions(+), 7 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 7dc881ade..7b7f79b22 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -893,7 +893,7 @@ pub fn verify(
         auth_id.to_string(),
         to_stdout,
         move |worker| {
-            let verify_worker = VerifyWorker::new(worker.clone(), datastore);
+            let verify_worker = VerifyWorker::new(worker.clone(), datastore)?;
             let failed_dirs = if let Some(backup_dir) = backup_dir {
                 let mut res = Vec::new();
                 if !verify_worker.verify_backup_dir(
diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 685b78e89..384e8a73f 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -796,7 +796,7 @@ impl BackupEnvironment {
             move |worker| {
                 worker.log_message("Automatically verifying newly added snapshot");
 
-                let verify_worker = VerifyWorker::new(worker.clone(), datastore);
+                let verify_worker = VerifyWorker::new(worker.clone(), datastore)?;
                 if !verify_worker.verify_backup_dir_with_lock(
                     &backup_dir,
                     worker.upid().clone(),
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 0b954ae23..a01ddcca3 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -17,7 +17,7 @@ use pbs_api_types::{
 use pbs_datastore::backup_info::{BackupDir, BackupGroup, BackupInfo};
 use pbs_datastore::index::IndexFile;
 use pbs_datastore::manifest::{BackupManifest, FileInfo};
-use pbs_datastore::{DataBlob, DataStore, StoreProgress};
+use pbs_datastore::{DataBlob, DataStore, DatastoreBackend, StoreProgress};
 
 use crate::tools::parallel_handler::ParallelHandler;
 
@@ -30,19 +30,25 @@ pub struct VerifyWorker {
     datastore: Arc<DataStore>,
     verified_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
     corrupt_chunks: Arc<Mutex<HashSet<[u8; 32]>>>,
+    backend: DatastoreBackend,
 }
 
 impl VerifyWorker {
     /// Creates a new VerifyWorker for a given task worker and datastore.
-    pub fn new(worker: Arc<dyn WorkerTaskContext>, datastore: Arc<DataStore>) -> Self {
-        Self {
+    pub fn new(
+        worker: Arc<dyn WorkerTaskContext>,
+        datastore: Arc<DataStore>,
+    ) -> Result<Self, Error> {
+        let backend = datastore.backend()?;
+        Ok(Self {
             worker,
             datastore,
             // start with 16k chunks == up to 64G data
             verified_chunks: Arc::new(Mutex::new(HashSet::with_capacity(16 * 1024))),
             // start with 64 chunks since we assume there are few corrupt ones
             corrupt_chunks: Arc::new(Mutex::new(HashSet::with_capacity(64))),
-        }
+            backend,
+        })
     }
 
     fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
diff --git a/src/server/verify_job.rs b/src/server/verify_job.rs
index 95a7b2a9b..c8792174b 100644
--- a/src/server/verify_job.rs
+++ b/src/server/verify_job.rs
@@ -41,7 +41,7 @@ pub fn do_verification_job(
                 None => Default::default(),
             };
 
-            let verify_worker = VerifyWorker::new(worker.clone(), datastore);
+            let verify_worker = VerifyWorker::new(worker.clone(), datastore)?;
             let result = verify_worker.verify_all_backups(
                 worker.upid(),
                 ns,
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:39 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:39 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 14/42] api: config: implement
 endpoints to manipulate and list s3 configs
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-15-c.ebner@proxmox.com>

Allows to create, list, modify and delete configurations for s3
clients via the api.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/config/mod.rs |   2 +
 src/api2/config/s3.rs  | 305 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 307 insertions(+)
 create mode 100644 src/api2/config/s3.rs

diff --git a/src/api2/config/mod.rs b/src/api2/config/mod.rs
index 15dc5db92..1cd9ead76 100644
--- a/src/api2/config/mod.rs
+++ b/src/api2/config/mod.rs
@@ -14,6 +14,7 @@ pub mod metrics;
 pub mod notifications;
 pub mod prune;
 pub mod remote;
+pub mod s3;
 pub mod sync;
 pub mod tape_backup_job;
 pub mod tape_encryption_keys;
@@ -32,6 +33,7 @@ const SUBDIRS: SubdirMap = &sorted!([
     ("notifications", &notifications::ROUTER),
     ("prune", &prune::ROUTER),
     ("remote", &remote::ROUTER),
+    ("s3", &s3::ROUTER),
     ("sync", &sync::ROUTER),
     ("tape-backup-job", &tape_backup_job::ROUTER),
     ("tape-encryption-keys", &tape_encryption_keys::ROUTER),
diff --git a/src/api2/config/s3.rs b/src/api2/config/s3.rs
new file mode 100644
index 000000000..aa6d0fa81
--- /dev/null
+++ b/src/api2/config/s3.rs
@@ -0,0 +1,305 @@
+use ::serde::{Deserialize, Serialize};
+use anyhow::Error;
+use hex::FromHex;
+use serde_json::Value;
+
+use proxmox_router::{http_bail, Permission, Router, RpcEnvironment};
+use proxmox_schema::{api, param_bail};
+
+use pbs_api_types::{
+    S3ClientConfig, S3ClientConfigUpdater, S3ClientSecretsConfig, S3ClientSecretsConfigUpdater,
+    JOB_ID_SCHEMA, PRIV_SYS_AUDIT, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA,
+};
+use pbs_config::s3;
+
+#[api(
+    input: {
+        properties: {},
+    },
+    returns: {
+        description: "List configured s3 clients.",
+        type: Array,
+        items: { type: S3ClientConfig },
+    },
+    access: {
+        permission: &Permission::Privilege(&[], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// List all s3 client configurations.
+pub fn list_s3_client_config(
+    _param: Value,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<Vec<S3ClientConfig>, Error> {
+    let (config, digest) = s3::config()?;
+    let list = config.convert_to_typed_array("s3client")?;
+
+    let (_secrets, secrets_digest) = s3::secrets_config()?;
+    let digest = digest_with_secrets(&digest, &secrets_digest);
+    rpcenv["digest"] = hex::encode(digest).into();
+
+    Ok(list)
+}
+
+#[api(
+    protected: true,
+    input: {
+        properties: {
+            config: {
+                type: S3ClientConfig,
+                flatten: true,
+            },
+            secrets: {
+                type: S3ClientSecretsConfig,
+                flatten: true,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&[], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Create a new s3 client configuration.
+pub fn create_s3_client_config(
+    config: S3ClientConfig,
+    secrets: S3ClientSecretsConfig,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    // Asssure both, config and secrets are referenced by the same `id`
+    if config.id != secrets.secrets_id {
+        param_bail!(
+            "id",
+            "config and secrets must use the same id ({} != {})",
+            config.id,
+            secrets.secrets_id
+        );
+    }
+
+    let _lock = s3::lock_config()?;
+    let (mut section_config, _digest) = s3::config()?;
+    if section_config.sections.contains_key(&config.id) {
+        param_bail!("id", "s3 client config '{}' already exists.", config.id);
+    }
+
+    let (mut section_secrets, _secrets_digest) = s3::secrets_config()?;
+    if section_secrets.sections.contains_key(&config.id) {
+        param_bail!("id", "s3 secrets config '{}' already exists.", config.id);
+    }
+
+    section_config.set_data(&config.id, "s3client", &config)?;
+    section_secrets.set_data(&config.id, "s3secrets", &secrets)?;
+    s3::save_config(&section_config, &section_secrets)?;
+
+    Ok(())
+}
+
+#[api(
+    input: {
+        properties: {
+            id: {
+                schema: JOB_ID_SCHEMA,
+            },
+        },
+    },
+    returns: { type: S3ClientConfig },
+    access: {
+        permission: &Permission::Privilege(&[], PRIV_SYS_AUDIT, false),
+    },
+)]
+/// Read an s3 client configuration.
+pub fn read_s3_client_config(
+    id: String,
+    rpcenv: &mut dyn RpcEnvironment,
+) -> Result<S3ClientConfig, Error> {
+    let (config, digest) = s3::config()?;
+    let s3_client_config: S3ClientConfig = config.lookup("s3client", &id)?;
+
+    let (_secrets, secrets_digest) = s3::secrets_config()?;
+    let digest = digest_with_secrets(&digest, &secrets_digest);
+    rpcenv["digest"] = hex::encode(digest).into();
+
+    Ok(s3_client_config)
+}
+
+#[api()]
+#[derive(Serialize, Deserialize)]
+#[serde(rename_all = "kebab-case")]
+/// Deletable property name
+pub enum DeletableProperty {
+    /// Delete the port property.
+    Port,
+    /// Delete the region property.
+    Region,
+    /// Delete the fingerprint property.
+    Fingerprint,
+}
+
+#[api(
+    protected: true,
+    input: {
+        properties: {
+            id: {
+                schema: JOB_ID_SCHEMA,
+            },
+            update: {
+                type: S3ClientConfigUpdater,
+                flatten: true,
+            },
+            "update-secrets": {
+                type: S3ClientSecretsConfigUpdater,
+                flatten: true,
+            },
+            delete: {
+                description: "List of properties to delete.",
+                type: Array,
+                optional: true,
+                items: {
+                    type: DeletableProperty,
+                }
+            },
+            digest: {
+                optional: true,
+                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&[], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Update an s3 client configuration.
+#[allow(clippy::too_many_arguments)]
+pub fn update_s3_client_config(
+    id: String,
+    update: S3ClientConfigUpdater,
+    update_secrets: S3ClientSecretsConfigUpdater,
+    delete: Option<Vec<DeletableProperty>>,
+    digest: Option<String>,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let _lock = s3::lock_config()?;
+    let (mut config, expected_digest) = s3::config()?;
+    let (mut secrets, secrets_digest) = s3::secrets_config()?;
+    let expected_digest = digest_with_secrets(&expected_digest, &secrets_digest);
+
+    // Secrets are not included in digest concurrent changes therefore not detected.
+    if let Some(ref digest) = digest {
+        let digest = <[u8; 32]>::from_hex(digest)?;
+        crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
+    }
+
+    let mut data: S3ClientConfig = config.lookup("s3client", &id)?;
+
+    if let Some(delete) = delete {
+        for delete_prop in delete {
+            match delete_prop {
+                DeletableProperty::Port => {
+                    data.port = None;
+                }
+                DeletableProperty::Region => {
+                    data.region = None;
+                }
+                DeletableProperty::Fingerprint => {
+                    data.fingerprint = None;
+                }
+            }
+        }
+    }
+
+    if let Some(host) = update.host {
+        data.host = host;
+    }
+    if let Some(bucket) = update.bucket {
+        data.bucket = bucket;
+    }
+    if let Some(port) = update.port {
+        data.port = Some(port);
+    }
+    if let Some(region) = update.region {
+        data.region = Some(region);
+    }
+    if let Some(access_key) = update.access_key {
+        data.access_key = access_key;
+    }
+    if let Some(fingerprint) = update.fingerprint {
+        data.fingerprint = Some(fingerprint);
+    }
+
+    let mut secrets_data: S3ClientSecretsConfig = secrets.lookup("s3secrets", &id)?;
+    if let Some(secret_key) = update_secrets.secret_key {
+        secrets_data.secret_key = secret_key;
+    }
+
+    config.set_data(&id, "s3client", &data)?;
+    secrets.set_data(&id, "s3secrets", &secrets_data)?;
+    s3::save_config(&config, &secrets)?;
+
+    Ok(())
+}
+
+#[api(
+    protected: true,
+    input: {
+        properties: {
+            id: {
+                schema: JOB_ID_SCHEMA,
+            },
+            digest: {
+                optional: true,
+                schema: PROXMOX_CONFIG_DIGEST_SCHEMA,
+            },
+        },
+    },
+    access: {
+        permission: &Permission::Privilege(&[], PRIV_SYS_MODIFY, false),
+    },
+)]
+/// Remove an s3 client configuration.
+pub fn delete_s3_client_config(
+    id: String,
+    digest: Option<String>,
+    _rpcenv: &mut dyn RpcEnvironment,
+) -> Result<(), Error> {
+    let _lock = s3::lock_config()?;
+    let (mut config, expected_digest) = s3::config()?;
+    let (mut secrets, secrets_digest) = s3::secrets_config()?;
+    let expected_digest = digest_with_secrets(&expected_digest, &secrets_digest);
+
+    if let Some(ref digest) = digest {
+        let digest = <[u8; 32]>::from_hex(digest)?;
+        crate::tools::detect_modified_configuration_file(&digest, &expected_digest)?;
+    }
+
+    match (config.sections.remove(&id), secrets.sections.remove(&id)) {
+        (Some(_), Some(_)) => {}
+        (None, None) => http_bail!(
+            NOT_FOUND,
+            "s3 client config and secrets '{id}' do not exist."
+        ),
+        (Some(_), None) => http_bail!(
+            NOT_FOUND,
+            "removed s3 client config, but no secrets for '{id}' found."
+        ),
+        (None, Some(_)) => http_bail!(
+            NOT_FOUND,
+            "removed s3 client secrets, but no config for '{id}' found."
+        ),
+    }
+    s3::save_config(&config, &secrets)
+}
+
+// Calculate the digest based on the digest of config and secrets to detect changes for both
+fn digest_with_secrets(digest: &[u8; 32], secrets_digest: &[u8; 32]) -> [u8; 32] {
+    let mut digest = digest.to_vec();
+    digest.append(&mut secrets_digest.to_vec());
+    openssl::sha::sha256(&digest)
+}
+
+const ITEM_ROUTER: Router = Router::new()
+    .get(&API_METHOD_READ_S3_CLIENT_CONFIG)
+    .put(&API_METHOD_UPDATE_S3_CLIENT_CONFIG)
+    .delete(&API_METHOD_DELETE_S3_CLIENT_CONFIG);
+
+pub const ROUTER: Router = Router::new()
+    .get(&API_METHOD_LIST_S3_CLIENT_CONFIG)
+    .post(&API_METHOD_CREATE_S3_CLIENT_CONFIG)
+    .match_all("id", &ITEM_ROUTER);
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:32:01 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:32:01 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 36/42] ui: add s3 bucket
 selector and allow to set s3 backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-37-c.ebner@proxmox.com>

In order to be able to create datastore with an s3 object store
backend. Implements a bucket selector and exposes it in the advanced
options of the datastore edit window.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/Makefile                 |  1 +
 www/form/S3BucketSelector.js | 40 ++++++++++++++++++++++++++++++++++++
 www/window/DataStoreEdit.js  | 35 +++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+)
 create mode 100644 www/form/S3BucketSelector.js

diff --git a/www/Makefile b/www/Makefile
index ca4683941..41deeee00 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -42,6 +42,7 @@ JSSRC=							\
 	Schema.js					\
 	form/TokenSelector.js				\
 	form/AuthidSelector.js				\
+	form/S3BucketSelector.js			\
 	form/RemoteSelector.js				\
 	form/RemoteTargetSelector.js   			\
 	form/DataStoreSelector.js			\
diff --git a/www/form/S3BucketSelector.js b/www/form/S3BucketSelector.js
new file mode 100644
index 000000000..c9905feb9
--- /dev/null
+++ b/www/form/S3BucketSelector.js
@@ -0,0 +1,40 @@
+Ext.define('PBS.form.S3BucketSelector', {
+    extend: 'Proxmox.form.ComboGrid',
+    alias: 'widget.pbsS3BucketSelector',
+
+    allowBlank: false,
+    autoSelect: false,
+    valueField: 'id',
+    displayField: 'id',
+
+    store: {
+	model: 'pmx-s3bucket',
+	autoLoad: true,
+	sorters: 'id',
+    },
+
+    listConfig: {
+	columns: [
+	    {
+		header: gettext('S3 Bucket ID'),
+		sortable: true,
+		dataIndex: 'id',
+		renderer: Ext.String.htmlEncode,
+		flex: 1,
+	    },
+	    {
+		header: gettext('Bucket'),
+		sortable: true,
+		dataIndex: 'bucket',
+		renderer: Ext.String.htmlEncode,
+		flex: 1,
+	    },
+	    {
+		header: gettext('Host'),
+		sortable: true,
+		dataIndex: 'host',
+		flex: 1,
+	    },
+	],
+    },
+});
diff --git a/www/window/DataStoreEdit.js b/www/window/DataStoreEdit.js
index 4a0b8d819..dffd2b2e0 100644
--- a/www/window/DataStoreEdit.js
+++ b/www/window/DataStoreEdit.js
@@ -101,6 +101,7 @@ Ext.define('PBS.DataStoreEdit', {
 		columnB: [
 		    {
 			xtype: 'checkbox',
+			name: 'removable-datastore',
 			boxLabel: gettext('Removable datastore'),
 			submitValue: false,
 			listeners: {
@@ -135,6 +136,37 @@ Ext.define('PBS.DataStoreEdit', {
 			fieldLabel: gettext('Reuse existing datastore'),
 		    },
 		],
+		advancedColumn2: [
+		    {
+			xtype: 'checkbox',
+			boxLabel: gettext('With S3 object store'),
+			submitValue: false,
+			listeners: {
+			    change: function(checkbox, withS3Backend) {
+				let inputPanel = checkbox.up('inputpanel');
+
+				let bucketSelector = inputPanel.down('[name=backend]');
+				bucketSelector.setDisabled(!withS3Backend);
+				bucketSelector.allowBlank = !withS3Backend;
+				bucketSelector.setValue('');
+
+				let removableDatastore = inputPanel.down('[name=removable-datastore]');
+				removableDatastore.setDisabled(withS3Backend);
+				removableDatastore.allowBlank = withS3Backend;
+				removableDatastore.setValue('');
+			    },
+			},
+		    },
+		    {
+			xtype: 'pbsS3BucketSelector',
+			name: 'backend',
+			fieldLabel: gettext('S3 Bucket ID'),
+			disabled: true,
+			cbind: {
+			    editable: '{isCreate}',
+			},
+		    },
+		],
 
 		onGetValues: function(values) {
 		    let me = this;
@@ -143,6 +175,9 @@ Ext.define('PBS.DataStoreEdit', {
 			// New datastores default to using the notification system
 			values['notification-mode'] = 'notification-system';
 		    }
+		    if (values.backend) {
+			values.backend = PBS.Utils.printPropertyString({ 's3': values.backend });
+		    }
 		    return values;
 		},
 	    },
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:32:05 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:32:05 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 40/42] api: backup: use local
 datastore cache on S3 backend chunk upload
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-41-c.ebner@proxmox.com>

Take advantage of the local datastore cache to avoid re-uploading of
already known chunks. This not only helps improve the backup/upload
speeds, but also avoids additionally costs by reducing the number of
requests and transferred payload data to the S3 object store api.

If the cache is present, lookup if it contains the chunk, skipping
upload altogether if it is. Otherwise, upload the chunk into memory,
upload it to the S3 object store api and insert it into the local
datastore cache.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/upload_chunk.rs | 46 ++++++++++++++++++++++++++++++---
 src/server/pull.rs              |  4 +++
 2 files changed, 46 insertions(+), 4 deletions(-)

diff --git a/src/api2/backup/upload_chunk.rs b/src/api2/backup/upload_chunk.rs
index 838eec1fa..7a80fd0eb 100644
--- a/src/api2/backup/upload_chunk.rs
+++ b/src/api2/backup/upload_chunk.rs
@@ -247,10 +247,48 @@ async fn upload_to_backend(
             UploadChunk::new(req_body, env.datastore.clone(), digest, size, encoded_size).await
         }
         DatastoreBackend::S3(s3_client) => {
-            let is_duplicate = match s3_client.put_object(digest.into(), req_body).await? {
-                PutObjectResponse::PreconditionFailed => true,
-                PutObjectResponse::NeedsRetry => bail!("concurrent operation, reupload required"),
-                PutObjectResponse::Success(_content) => false,
+            // Load chunk data into memory, need to write it twice, to S3 object store and
+            // local cache store. Further, body needs to be consumed also if chunks insert
+            // can be skipped since cached.
+            let data = req_body
+                .map_err(Error::from)
+                .try_fold(Vec::new(), |mut acc, chunk| {
+                    acc.extend_from_slice(&chunk);
+                    future::ok::<_, Error>(acc)
+                })
+                .await?;
+
+            if encoded_size != data.len() as u32 {
+                bail!(
+                    "got blob with unexpected length ({encoded_size} != {})",
+                    data.len()
+                );
+            }
+
+            if env.datastore.cache_contains(&digest) {
+                return Ok((digest, size, encoded_size, true));
+            }
+
+            let datastore = env.datastore.clone();
+            let upload_body = hyper::Body::from(data.clone());
+            let upload = s3_client.put_object(digest.into(), upload_body);
+            let cache_insert = tokio::task::spawn_blocking(move || {
+                let chunk = DataBlob::from_raw(data)?;
+                datastore.cache_insert(&digest, &chunk)
+            });
+            let is_duplicate = match futures::join!(upload, cache_insert) {
+                (Ok(upload_response), Ok(Ok(()))) => match upload_response {
+                    PutObjectResponse::PreconditionFailed => true,
+                    PutObjectResponse::NeedsRetry => {
+                        bail!("concurrent operation, reupload required")
+                    }
+                    PutObjectResponse::Success(_content) => false,
+                },
+                (Ok(_), Ok(Err(err))) => return Err(err.context("chunk cache insert failed")),
+                (Ok(_), Err(err)) => {
+                    return Err(Error::from(err).context("chunk cache insert task failed"))
+                }
+                (Err(err), _) => return Err(err.context("chunk upload failed")),
             };
             Ok((digest, size, encoded_size, is_duplicate))
         }
diff --git a/src/server/pull.rs b/src/server/pull.rs
index f36efd7c8..85d3154eb 100644
--- a/src/server/pull.rs
+++ b/src/server/pull.rs
@@ -173,6 +173,10 @@ async fn pull_index_chunks<I: IndexFile>(
                     target2.insert_chunk(&chunk, &digest)?;
                 }
                 DatastoreBackend::S3(s3_client) => {
+                    if target2.cache_contains(&digest) {
+                        return Ok(());
+                    }
+                    target2.cache_insert(&digest, &chunk)?;
                     let data = chunk.raw_data().to_vec();
                     let upload_body = hyper::Body::from(data);
                     proxmox_async::runtime::block_on(
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:32:07 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:32:07 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 42/42] api: backup: add no-cache
 flag to bypass local datastore cache
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-43-c.ebner@proxmox.com>

Adds the `no-cache` flag so the client can request to bypass the
local datastore cache for chunk uploads. This is mainly intended for
debugging and benchmarking, but can be used in cases the caching is
known to be ineffective (no possible deduplication).

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 examples/upload-speed.rs               |  1 +
 pbs-client/src/backup_writer.rs        |  4 +++-
 proxmox-backup-client/src/benchmark.rs |  1 +
 proxmox-backup-client/src/main.rs      |  8 ++++++++
 src/api2/backup/environment.rs         |  3 +++
 src/api2/backup/mod.rs                 |  3 +++
 src/api2/backup/upload_chunk.rs        | 11 +++++++++++
 src/server/push.rs                     |  1 +
 8 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/examples/upload-speed.rs b/examples/upload-speed.rs
index e4b570ec5..8a6594a47 100644
--- a/examples/upload-speed.rs
+++ b/examples/upload-speed.rs
@@ -25,6 +25,7 @@ async fn upload_speed() -> Result<f64, Error> {
         &(BackupType::Host, "speedtest".to_string(), backup_time).into(),
         false,
         true,
+        false,
     )
     .await?;
 
diff --git a/pbs-client/src/backup_writer.rs b/pbs-client/src/backup_writer.rs
index 325425069..a91880720 100644
--- a/pbs-client/src/backup_writer.rs
+++ b/pbs-client/src/backup_writer.rs
@@ -82,6 +82,7 @@ impl BackupWriter {
         backup: &BackupDir,
         debug: bool,
         benchmark: bool,
+        no_cache: bool,
     ) -> Result<Arc<BackupWriter>, Error> {
         let mut param = json!({
             "backup-type": backup.ty(),
@@ -89,7 +90,8 @@ impl BackupWriter {
             "backup-time": backup.time,
             "store": datastore,
             "debug": debug,
-            "benchmark": benchmark
+            "benchmark": benchmark,
+            "no-cache": no_cache,
         });
 
         if !ns.is_root() {
diff --git a/proxmox-backup-client/src/benchmark.rs b/proxmox-backup-client/src/benchmark.rs
index a6f24d745..ed21c7a91 100644
--- a/proxmox-backup-client/src/benchmark.rs
+++ b/proxmox-backup-client/src/benchmark.rs
@@ -236,6 +236,7 @@ async fn test_upload_speed(
         &(BackupType::Host, "benchmark".to_string(), backup_time).into(),
         false,
         true,
+        true,
     )
     .await?;
 
diff --git a/proxmox-backup-client/src/main.rs b/proxmox-backup-client/src/main.rs
index 44f4f5db5..83fc9309a 100644
--- a/proxmox-backup-client/src/main.rs
+++ b/proxmox-backup-client/src/main.rs
@@ -742,6 +742,12 @@ fn spawn_catalog_upload(
                 optional: true,
                 default: false,
             },
+            "no-cache": {
+                type: Boolean,
+                description: "Bypass local datastore cache for network storages.",
+                optional: true,
+                default: false,
+            },
         }
     }
  )]
@@ -754,6 +760,7 @@ async fn create_backup(
     change_detection_mode: Option<BackupDetectionMode>,
     dry_run: bool,
     skip_e2big_xattr: bool,
+    no_cache: bool,
     limit: ClientRateLimitConfig,
     _info: &ApiMethod,
     _rpcenv: &mut dyn RpcEnvironment,
@@ -960,6 +967,7 @@ async fn create_backup(
         &snapshot,
         true,
         false,
+        no_cache,
     )
     .await?;
 
diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 384e8a73f..874f0c44d 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -112,6 +112,7 @@ pub struct BackupEnvironment {
     result_attributes: Value,
     auth_id: Authid,
     pub debug: bool,
+    pub no_cache: bool,
     pub formatter: &'static dyn OutputFormatter,
     pub worker: Arc<WorkerTask>,
     pub datastore: Arc<DataStore>,
@@ -128,6 +129,7 @@ impl BackupEnvironment {
         worker: Arc<WorkerTask>,
         datastore: Arc<DataStore>,
         backup_dir: BackupDir,
+        no_cache: bool,
     ) -> Result<Self, Error> {
         let state = SharedBackupState {
             finished: false,
@@ -148,6 +150,7 @@ impl BackupEnvironment {
             worker,
             datastore,
             debug: tracing::enabled!(tracing::Level::DEBUG),
+            no_cache,
             formatter: JSON_FORMATTER,
             backup_dir,
             last_backup: None,
diff --git a/src/api2/backup/mod.rs b/src/api2/backup/mod.rs
index 2c6afca41..0913d4264 100644
--- a/src/api2/backup/mod.rs
+++ b/src/api2/backup/mod.rs
@@ -51,6 +51,7 @@ pub const API_METHOD_UPGRADE_BACKUP: ApiMethod = ApiMethod::new(
             ("backup-time", false, &BACKUP_TIME_SCHEMA),
             ("debug", true, &BooleanSchema::new("Enable verbose debug logging.").schema()),
             ("benchmark", true, &BooleanSchema::new("Job is a benchmark (do not keep data).").schema()),
+            ("no-cache", true, &BooleanSchema::new("Disable local datastore cache for network storages").schema()),
         ]),
     )
 ).access(
@@ -77,6 +78,7 @@ fn upgrade_to_backup_protocol(
     async move {
         let debug = param["debug"].as_bool().unwrap_or(false);
         let benchmark = param["benchmark"].as_bool().unwrap_or(false);
+        let no_cache = param["no-cache"].as_bool().unwrap_or(false);
 
         let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
 
@@ -212,6 +214,7 @@ fn upgrade_to_backup_protocol(
                     worker.clone(),
                     datastore,
                     backup_dir,
+                    no_cache,
                 )?;
 
                 env.debug = debug;
diff --git a/src/api2/backup/upload_chunk.rs b/src/api2/backup/upload_chunk.rs
index 7a80fd0eb..4e949a073 100644
--- a/src/api2/backup/upload_chunk.rs
+++ b/src/api2/backup/upload_chunk.rs
@@ -247,6 +247,17 @@ async fn upload_to_backend(
             UploadChunk::new(req_body, env.datastore.clone(), digest, size, encoded_size).await
         }
         DatastoreBackend::S3(s3_client) => {
+            if env.no_cache {
+                let is_duplicate = match s3_client.put_object(digest.into(), req_body).await? {
+                    PutObjectResponse::PreconditionFailed => true,
+                    PutObjectResponse::NeedsRetry => {
+                        bail!("concurrent operation, reupload required")
+                    }
+                    PutObjectResponse::Success(_content) => false,
+                };
+                return Ok((digest, size, encoded_size, is_duplicate));
+            }
+
             // Load chunk data into memory, need to write it twice, to S3 object store and
             // local cache store. Further, body needs to be consumed also if chunks insert
             // can be skipped since cached.
diff --git a/src/server/push.rs b/src/server/push.rs
index e71012ed8..6a31d2abe 100644
--- a/src/server/push.rs
+++ b/src/server/push.rs
@@ -828,6 +828,7 @@ pub(crate) async fn push_snapshot(
         snapshot,
         false,
         false,
+        false,
     )
     .await?;
 
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:29 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:29 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 04/42] bin: sort submodules
 alphabetically
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-5-c.ebner@proxmox.com>

Makes it easier to find existing entries or insert new modules.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/bin/proxmox_backup_manager/mod.rs | 28 +++++++++++++--------------
 1 file changed, 14 insertions(+), 14 deletions(-)

diff --git a/src/bin/proxmox_backup_manager/mod.rs b/src/bin/proxmox_backup_manager/mod.rs
index 11fb6dd3b..9b5c73e9a 100644
--- a/src/bin/proxmox_backup_manager/mod.rs
+++ b/src/bin/proxmox_backup_manager/mod.rs
@@ -8,31 +8,31 @@ mod cert;
 pub use cert::*;
 mod datastore;
 pub use datastore::*;
+mod disk;
+pub use disk::*;
 mod dns;
 pub use dns::*;
 mod ldap;
 pub use ldap::*;
 mod network;
 pub use network::*;
-mod prune;
-pub use prune::*;
-mod remote;
-pub use remote::*;
-mod sync;
-pub use sync::*;
-mod verify;
-pub use verify::*;
-mod user;
-pub use user::*;
-mod subscription;
-pub use subscription::*;
-mod disk;
-pub use disk::*;
 mod node;
 pub use node::*;
 mod notifications;
 pub use notifications::*;
 mod openid;
 pub use openid::*;
+mod prune;
+pub use prune::*;
+mod remote;
+pub use remote::*;
+mod subscription;
+pub use subscription::*;
+mod sync;
+pub use sync::*;
 mod traffic_control;
 pub use traffic_control::*;
+mod user;
+pub use user::*;
+mod verify;
+pub use verify::*;
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:37 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:37 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 12/42] s3 client: implement
 methods to operate on s3 objects in bucket
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-13-c.ebner@proxmox.com>

Adds the basic implementation of the client to use s3 object stores
as backend for PBS datastores.

This implements the basic client actions on a bucket and objects
stored within given bucket.

This is not feature complete and intended to be extended on a
per-demand fashion rather than implementing the whole client at once.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 Cargo.toml                           |   3 +
 pbs-s3-client/Cargo.toml             |   8 +
 pbs-s3-client/src/client.rs          | 463 +++++++++++++++++++++++++++
 pbs-s3-client/src/lib.rs             |   2 +
 pbs-s3-client/src/response_reader.rs | 343 ++++++++++++++++++++
 5 files changed, 819 insertions(+)
 create mode 100644 pbs-s3-client/src/response_reader.rs

diff --git a/Cargo.toml b/Cargo.toml
index aaa79c2aa..1bc3bb88b 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -137,15 +137,18 @@ log = "0.4.17"
 nix = "0.26.1"
 nom = "7"
 num-traits = "0.2"
+md5 = "0.7.0"
 once_cell = "1.3.1"
 openssl = "0.10.40"
 percent-encoding = "2.1"
 pin-project-lite = "0.2"
+quick-xml = "0.26"
 regex = "1.5.5"
 rustyline = "9"
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
 serde_plain = "1.0"
+serde-xml-rs = "0.5"
 siphasher = "0.3"
 syslog = "6"
 tar = "0.4"
diff --git a/pbs-s3-client/Cargo.toml b/pbs-s3-client/Cargo.toml
index 3261a32bb..d20cd8f38 100644
--- a/pbs-s3-client/Cargo.toml
+++ b/pbs-s3-client/Cargo.toml
@@ -8,12 +8,20 @@ rust-version.workspace = true
 
 [dependencies]
 anyhow.workspace = true
+base64.workspace = true
+bytes.workspace = true
+futures.workspace = true
 hex = { workspace = true, features = [ "serde" ] }
 hyper.workspace = true
 iso8601.workspace = true
+md5.workspace = true
 openssl.workspace = true
+quick-xml  = { workspace = true, features = ["async-tokio"] }
 serde.workspace = true
 serde_plain.workspace = true
+serde-xml-rs.workspace = true
+tokio = { workspace = true, features = [] }
+tokio-util = { workspace = true, features = ["compat"] }
 tracing.workspace = true
 url.workspace = true
 
diff --git a/pbs-s3-client/src/client.rs b/pbs-s3-client/src/client.rs
index e001cc7b0..b7ca4e298 100644
--- a/pbs-s3-client/src/client.rs
+++ b/pbs-s3-client/src/client.rs
@@ -1,17 +1,36 @@
+use std::collections::HashMap;
+use std::io::Cursor;
 use std::sync::{Arc, Mutex};
 use std::time::Duration;
 
 use anyhow::{bail, format_err, Context, Error};
+use bytes::{Bytes, BytesMut};
+use hyper::body::HttpBody;
 use hyper::client::{Client, HttpConnector};
+use hyper::http::method::Method;
 use hyper::http::uri::Authority;
+use hyper::http::StatusCode;
+use hyper::http::{header, HeaderValue, Uri};
 use hyper::Body;
+use hyper::{Request, Response};
 use openssl::hash::MessageDigest;
+use openssl::sha::Sha256;
 use openssl::ssl::{SslConnector, SslMethod, SslVerifyMode};
 use openssl::x509::X509StoreContextRef;
+use quick_xml::events::BytesText;
+use quick_xml::writer::Writer;
 use tracing::error;
 
 use proxmox_http::client::HttpsConnector;
 
+use crate::aws_sign_v4::aws_sign_v4_signature;
+use crate::aws_sign_v4::AWS_SIGN_V4_DATETIME_FORMAT;
+use crate::object_key::S3ObjectKey;
+use crate::response_reader::{
+    CopyObjectResponse, DeleteObjectsResponse, GetObjectResponse, HeadObjectResponse,
+    ListObjectsV2Response, PutObjectResponse, ResponseReader,
+};
+
 const S3_HTTP_CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
 const S3_TCP_KEEPALIVE_TIME: u32 = 120;
 
@@ -128,4 +147,448 @@ impl S3Client {
             "unexpected certificate fingerprint {certificate_fingerprint}"
         ))
     }
+
+    async fn prepare(&self, mut request: Request<Body>) -> Result<Request<Body>, Error> {
+        let host_header = request
+            .uri()
+            .authority()
+            .ok_or_else(|| format_err!("request missing authority"))?
+            .to_string();
+
+        // Content verification for aws s3 signature
+        let mut hasher = Sha256::new();
+        // Load payload into memory, needed as the hash and checksum have to be calculated a-priori
+        let buffer: Bytes = {
+            let body = request.body_mut();
+            let mut buf = BytesMut::with_capacity(body.size_hint().lower() as usize);
+            while let Some(chunk) = body.data().await {
+                let chunk = chunk?;
+                hasher.update(&chunk);
+                buf.extend_from_slice(&chunk);
+            }
+            buf.freeze()
+        };
+        // Use MD5 as upload integrity check, as other methods are not supported by all S3 object
+        // store providers and might be ignored and this is recommended by AWS as described in
+        // https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_RequestSyntax
+        let payload_md5 = md5::compute(&buffer);
+        let payload_digest = hex::encode(hasher.finish());
+        let payload_len = buffer.len();
+        *request.body_mut() = Body::from(buffer);
+
+        let epoch = proxmox_time::epoch_i64();
+        let datetime = proxmox_time::strftime_utc(AWS_SIGN_V4_DATETIME_FORMAT, epoch)?;
+
+        request
+            .headers_mut()
+            .insert("x-amz-date", HeaderValue::from_str(&datetime)?);
+        request
+            .headers_mut()
+            .insert("host", HeaderValue::from_str(&host_header)?);
+        request.headers_mut().insert(
+            "x-amz-content-sha256",
+            HeaderValue::from_str(&payload_digest)?,
+        );
+        request.headers_mut().insert(
+            header::CONTENT_LENGTH,
+            HeaderValue::from_str(&payload_len.to_string())?,
+        );
+        if payload_len > 0 {
+            let md5_digest = base64::encode(*payload_md5);
+            request
+                .headers_mut()
+                .insert("Content-MD5", HeaderValue::from_str(&md5_digest)?);
+        }
+
+        let signature = aws_sign_v4_signature(&request, &self.options, epoch, &payload_digest)?;
+
+        request
+            .headers_mut()
+            .insert(header::AUTHORIZATION, HeaderValue::from_str(&signature)?);
+
+        Ok(request)
+    }
+
+    pub async fn send(&self, request: Request<Body>) -> Result<Response<Body>, Error> {
+        let request = self.prepare(request).await?;
+        let response = tokio::time::timeout(S3_HTTP_CONNECT_TIMEOUT, self.client.request(request))
+            .await
+            .context("request timeout")??;
+        Ok(response)
+    }
+
+    /// Check if bucket exists and got permissions to access it.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html
+    pub async fn head_bucket(&self) -> Result<(), Error> {
+        let request = Request::builder()
+            .method(Method::HEAD)
+            .uri(self.uri_builder("/")?)
+            .body(Body::empty())?;
+        let response = self.send(request).await?;
+        let (parts, _body) = response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::BAD_REQUEST | StatusCode::FORBIDDEN | StatusCode::NOT_FOUND => {
+                bail!("bucket does not exist or no permission to access it")
+            }
+            status_code => bail!("unexpected status code {status_code}"),
+        }
+
+        Ok(())
+    }
+
+    /// Fetch metadata from an object without returning the object itself.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html
+    pub async fn head_object(
+        &self,
+        object_key: S3ObjectKey,
+    ) -> Result<Option<HeadObjectResponse>, Error> {
+        let request = Request::builder()
+            .method(Method::HEAD)
+            .uri(self.uri_builder(&object_key)?)
+            .body(Body::empty())?;
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.head_object_response().await
+    }
+
+    /// Fetch an object from object store.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
+    pub async fn get_object(
+        &self,
+        object_key: S3ObjectKey,
+    ) -> Result<Option<GetObjectResponse>, Error> {
+        let request = Request::builder()
+            .method(Method::GET)
+            .uri(self.uri_builder(&object_key)?)
+            .body(Body::empty())?;
+
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.get_object_response().await
+    }
+
+    /// Returns some or all (up to 1,000) of the objects in a bucket with each request.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectTagging.html
+    pub async fn list_objects_v2(
+        &self,
+        prefix: Option<&str>,
+        max_keys: Option<u64>,
+        continuation_token: Option<&str>,
+    ) -> Result<ListObjectsV2Response, Error> {
+        let mut path_and_query = String::from("/?list-type=2");
+        if let Some(prefix) = prefix {
+            path_and_query.push_str("&prefix=");
+            path_and_query.push_str(prefix);
+        }
+        if let Some(max_keys) = max_keys {
+            path_and_query.push_str("&max-keys=");
+            path_and_query.push_str(&max_keys.to_string());
+        }
+        if let Some(token) = continuation_token {
+            path_and_query.push_str("&continuation-token=");
+            path_and_query.push_str(token);
+        }
+        let request = Request::builder()
+            .method(Method::GET)
+            .uri(self.uri_builder(&path_and_query)?)
+            .body(Body::empty())?;
+
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.list_objects_v2_response().await
+    }
+
+    /// Add a new object to a bucket.
+    ///
+    /// Do not reupload if an object with matching key already exists in the bucket.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
+    pub async fn put_object(
+        &self,
+        object_key: S3ObjectKey,
+        object_data: Body,
+    ) -> Result<PutObjectResponse, Error> {
+        let request = Request::builder()
+            .method(Method::PUT)
+            .uri(self.uri_builder(&object_key)?)
+            .header(header::CONTENT_TYPE, "binary/octet")
+            // Never overwrite pre-existing objects with the same key.
+            //.header(header::IF_NONE_MATCH, "*")
+            .body(object_data)?;
+
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.put_object_response().await
+    }
+
+    /// Sets the supplied tag-set to an object that already exists in a bucket. A tag is a key-value pair.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObjectTagging.html
+    pub async fn put_object_tagging(
+        &self,
+        object_key: S3ObjectKey,
+        tagset: &HashMap<String, String>,
+    ) -> Result<bool, Error> {
+        let mut writer = Writer::new(Cursor::new(Vec::new()));
+        writer
+            .create_element("Tagging")
+            .with_attribute(("xmlns", "http://s3.amazonaws.com/doc/2006-03-01/"))
+            .write_inner_content(|writer| {
+                writer
+                    .create_element("TagSet")
+                    .write_inner_content(|writer| {
+                        for (key, value) in tagset.iter() {
+                            writer.create_element("Tag").write_inner_content(|writer| {
+                                writer
+                                    .create_element("Key")
+                                    .write_text_content(BytesText::new(key))?;
+                                writer
+                                    .create_element("Value")
+                                    .write_text_content(BytesText::new(value))?;
+                                Ok(())
+                            })?;
+                        }
+                        Ok(())
+                    })?;
+                Ok(())
+            })?;
+
+        let body: Body = writer.into_inner().into_inner().into();
+        let request = Request::builder()
+            .method(Method::PUT)
+            .uri(self.uri_builder(&format!("{object_key}?tagging"))?)
+            .body(body)?;
+
+        let response = self.send(request).await?;
+        Ok(response.status().is_success())
+    }
+
+    /// Sets the supplied tag to an object that already exists in a bucket. A tag is a key-value pair.
+    /// Optimized version of the `put_object_tagging` to only set a single tag.
+    pub async fn put_object_tag(
+        &self,
+        object_key: S3ObjectKey,
+        tag_key: &str,
+        tag_value: &str,
+    ) -> Result<bool, Error> {
+        let body: Body = format!(
+            r#"<Tagging xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
+                <TagSet>
+                    <Tag>
+                        <Key>{tag_key}</Key>
+                        <Value>{tag_value}</Value>
+                    </Tag>
+                </TagSet>
+            </Tagging>"#
+        )
+        .into();
+
+        let request = Request::builder()
+            .method(Method::PUT)
+            .uri(self.uri_builder(&format!("{object_key}?tagging"))?)
+            .body(body)?;
+
+        let response = self.send(request).await?;
+        //TODO: Response and error handling!
+        Ok(response.status().is_success())
+    }
+
+    /// Creates a copy of an object that is already stored in Amazon S3.
+    /// Uses the `x-amz-metadata-directive` set to `REPLACE`, therefore resulting in updated metadata.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
+    pub async fn copy_object(
+        &self,
+        destination_key: S3ObjectKey,
+        source_bucket: &str,
+        source_key: S3ObjectKey,
+    ) -> Result<CopyObjectResponse, Error> {
+        let copy_source = source_key.to_copy_source_key(source_bucket);
+        let request = Request::builder()
+            .method(Method::PUT)
+            .uri(self.uri_builder(&destination_key)?)
+            .header("x-amz-copy-source", HeaderValue::from_str(&copy_source)?)
+            .header(
+                "x-amz-metadata-directive",
+                HeaderValue::from_str("REPLACE")?,
+            )
+            .body(Body::empty())?;
+
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.copy_object_response().await
+    }
+
+    /// Helper to update the metadata for an object by copying it to itself. This will not cause
+    /// any additional costs other than the request cost itself.
+    ///
+    /// Note: This will actually create a new object for buckets with versioning enabled.
+    /// Return with error if that is the case, detected by checking the presence of the
+    /// `x-amz-version-id` header in the response.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html
+    pub async fn update_object_metadata(
+        &self,
+        object_key: S3ObjectKey,
+    ) -> Result<CopyObjectResponse, Error> {
+        let response = self
+            .copy_object(object_key.clone(), &self.options.bucket, object_key)
+            .await?;
+        if response.x_amz_version_id.is_some() {
+            // Return an error if the response contains an `x-amz-version-id`, indicating that the
+            // bucket has versioning enabled, as that will bloat the bucket size and therefore cost.
+            bail!("Failed to update object metadata as versioning is enabled");
+        }
+        Ok(response)
+    }
+
+    /// Removes an object from a bucket.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html
+    pub async fn delete_object(&self, object_key: S3ObjectKey) -> Result<(), Error> {
+        let request = Request::builder()
+            .method(Method::DELETE)
+            .uri(self.uri_builder(&object_key)?)
+            .body(Body::empty())?;
+
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.delete_object_response().await
+    }
+
+    /// Delete multiple objects from a bucket using a single HTTP request.
+    /// See reference docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
+    pub async fn delete_objects(
+        &self,
+        object_keys: &[String],
+    ) -> Result<DeleteObjectsResponse, Error> {
+        let mut body = String::from(r#"<Delete xmlns="http://s3.amazonaws.com/doc/2006-03-01/">"#);
+        for object_key in object_keys {
+            let object = format!("<Object><Key>{object_key}</Key></Object>");
+            body.push_str(&object);
+        }
+        body.push_str("</Delete>");
+        let request = Request::builder()
+            .method(Method::POST)
+            .uri(self.uri_builder("/?delete")?)
+            .body(Body::from(body))?;
+
+        let response = self.send(request).await?;
+        let response_reader = ResponseReader::new(response);
+        response_reader.delete_objects_response().await
+    }
+
+    /// Delete objects by given key prefix.
+    /// Requires at least 2 api calls.
+    pub async fn delete_objects_by_prefix(&self, prefix: &str) -> Result<bool, Error> {
+        // S3 API does not provide a convenient way to delete objects by key prefix.
+        // List all objects with given group prefix and delete all objects found, so this
+        // requires at least 2 API calls.
+        let mut next_continuation_token: Option<String> = None;
+        let mut delete_errors = false;
+        loop {
+            let list_objects_result = self
+                .list_objects_v2(Some(prefix), None, next_continuation_token.as_deref())
+                .await?;
+            let objects_to_delete: Vec<String> = list_objects_result
+                .contents
+                .into_iter()
+                .map(|item| item.key)
+                .collect();
+            let response = self.delete_objects(&objects_to_delete).await?;
+            if response.error.is_some() {
+                delete_errors = true;
+            }
+
+            if list_objects_result.is_truncated {
+                next_continuation_token = list_objects_result
+                    .next_continuation_token
+                    .as_ref()
+                    .cloned();
+                continue;
+            }
+            break;
+        }
+        Ok(delete_errors)
+    }
+
+    /// Delete objects by given key prefix, but exclude items pre-filter based on suffix
+    /// (including the parent component of the matched suffix). E.g. do not remove items in a
+    /// snapshot directory, by matching based on the protected file marker (given as suffix).
+    ///
+    /// Requires at least 2 api calls.
+    pub async fn delete_objects_by_prefix_with_suffix_filter(
+        &self,
+        prefix: &str,
+        suffix: &str,
+    ) -> Result<bool, Error> {
+        // S3 API does not provide a convenient way to delete objects by key prefix.
+        // List all objects with given group prefix and delete all objects found, so this
+        // requires at least 2 API calls.
+        let mut next_continuation_token: Option<String> = None;
+        let mut delete_errors = false;
+        let mut prefix_filters = Vec::new();
+        let mut list_objects = Vec::new();
+        loop {
+            let list_objects_result = self
+                .list_objects_v2(Some(prefix), None, next_continuation_token.as_deref())
+                .await?;
+            let mut prefixes: Vec<String> = list_objects_result
+                .contents
+                .iter()
+                .filter_map(|item| {
+                    let prefix_filter = item
+                        .key
+                        .strip_suffix(suffix)
+                        .map(|prefix| prefix.to_string());
+                    if prefix_filter.is_none() {
+                        list_objects.push(item.key.clone());
+                    }
+                    prefix_filter
+                })
+                .collect();
+            prefix_filters.append(&mut prefixes);
+
+            if list_objects_result.is_truncated {
+                next_continuation_token = list_objects_result
+                    .next_continuation_token
+                    .as_ref()
+                    .cloned();
+                continue;
+            }
+            break;
+        }
+
+        // Re-filter in case the 1000 items per request boundary lead to the prefix not being
+        // filtered for some items
+        let objects_to_delete: Vec<String> = list_objects
+            .into_iter()
+            .filter_map(|item| {
+                for prefix in &prefix_filters {
+                    if item.strip_prefix(prefix).is_some() {
+                        return None;
+                    }
+                }
+                Some(item)
+            })
+            .collect();
+
+        for objects in objects_to_delete.chunks(1000) {
+            let result = self.delete_objects(objects).await?;
+            if result.error.is_some() {
+                delete_errors = true;
+            }
+        }
+
+        Ok(delete_errors)
+    }
+
+    #[inline(always)]
+    /// Helper to generate [`Uri`] instance with common properties based on given path and query
+    /// string
+    fn uri_builder(&self, path_and_query: &str) -> Result<Uri, Error> {
+        Uri::builder()
+            .scheme("https")
+            .authority(self.authority.clone())
+            .path_and_query(path_and_query)
+            .build()
+            .context("failed to build uri")
+    }
 }
diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
index b3e539bdd..b4e7eb497 100644
--- a/pbs-s3-client/src/lib.rs
+++ b/pbs-s3-client/src/lib.rs
@@ -3,6 +3,8 @@ mod client;
 pub use client::{S3Client, S3ClientOptions};
 mod object_key;
 pub use object_key::{S3ObjectKey, S3_CONTENT_PREFIX};
+mod response_reader;
+pub use response_reader::PutObjectResponse;
 
 use std::time::Duration;
 
diff --git a/pbs-s3-client/src/response_reader.rs b/pbs-s3-client/src/response_reader.rs
new file mode 100644
index 000000000..8c553ce8f
--- /dev/null
+++ b/pbs-s3-client/src/response_reader.rs
@@ -0,0 +1,343 @@
+use std::str::FromStr;
+
+use anyhow::{anyhow, bail, Context, Error};
+use hyper::body::HttpBody;
+use hyper::header::HeaderName;
+use hyper::http::header;
+use hyper::http::StatusCode;
+use hyper::{Body, HeaderMap, Response};
+use serde::Deserialize;
+
+use crate::{HttpDate, LastModifiedTimestamp};
+
+pub(crate) struct ResponseReader {
+    response: Response<Body>,
+}
+
+#[derive(Debug)]
+pub struct ListObjectsV2Response {
+    pub date: HttpDate,
+    pub name: String,
+    pub prefix: String,
+    pub key_count: u64,
+    pub max_keys: u64,
+    pub is_truncated: bool,
+    pub continuation_token: Option<String>,
+    pub next_continuation_token: Option<String>,
+    pub contents: Vec<ListObjectsV2Contents>,
+}
+
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+struct ListObjectsV2ResponseBody {
+    pub name: String,
+    pub prefix: String,
+    pub key_count: u64,
+    pub max_keys: u64,
+    pub is_truncated: bool,
+    pub continuation_token: Option<String>,
+    pub next_continuation_token: Option<String>,
+    pub contents: Option<Vec<ListObjectsV2Contents>>,
+}
+
+impl ListObjectsV2ResponseBody {
+    fn with_date(self, date: HttpDate) -> ListObjectsV2Response {
+        ListObjectsV2Response {
+            date,
+            name: self.name,
+            prefix: self.prefix,
+            key_count: self.key_count,
+            max_keys: self.max_keys,
+            is_truncated: self.is_truncated,
+            continuation_token: self.continuation_token,
+            next_continuation_token: self.next_continuation_token,
+            contents: self.contents.unwrap_or_default(),
+        }
+    }
+}
+
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+pub struct ListObjectsV2Contents {
+    pub key: String,
+    pub last_modified: LastModifiedTimestamp,
+    pub e_tag: String,
+    pub size: u64,
+    pub storage_class: String,
+}
+
+#[derive(Debug)]
+/// Subset of the head object response (headers only, there is no body)
+/// See https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html#API_HeadObject_ResponseSyntax
+pub struct HeadObjectResponse {
+    pub content_length: u64,
+    pub content_type: String,
+    pub date: HttpDate,
+    pub e_tag: String,
+    pub last_modified: HttpDate,
+}
+
+#[derive(Debug)]
+/// Subset of the get object response
+/// https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_ResponseSyntax
+pub struct GetObjectResponse {
+    pub content_length: u64,
+    pub content_type: String,
+    pub date: HttpDate,
+    pub e_tag: String,
+    pub last_modified: HttpDate,
+    pub content: Body,
+}
+
+#[derive(Debug)]
+pub struct CopyObjectResponse {
+    pub copy_object_result: CopyObjectResult,
+    pub x_amz_version_id: Option<String>,
+}
+
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+pub struct CopyObjectResult {
+    pub e_tag: String,
+    pub last_modified: LastModifiedTimestamp,
+}
+
+/// Subset of the put object response
+/// https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_ResponseSyntax
+#[derive(Debug)]
+pub enum PutObjectResponse {
+    NeedsRetry,
+    PreconditionFailed,
+    Success(String),
+}
+
+/// Subset of the delete objects response
+/// https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html#API_DeleteObjects_ResponseElements
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+pub struct DeleteObjectsResponse {
+    pub deleted: Option<Vec<DeletedObject>>,
+    pub error: Option<Vec<DeleteObjectError>>,
+}
+
+/// https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeletedObject.html
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+pub struct DeletedObject {
+    pub delete_marker: Option<bool>,
+    pub delete_marker_version_id: Option<String>,
+    pub key: Option<String>,
+    pub version_id: Option<String>,
+}
+
+/// https://docs.aws.amazon.com/AmazonS3/latest/API/API_Error.html
+#[derive(Deserialize, Debug)]
+#[serde(rename_all = "PascalCase")]
+pub struct DeleteObjectError {
+    pub code: Option<String>,
+    pub key: Option<String>,
+    pub message: Option<String>,
+    pub version_id: Option<String>,
+}
+
+impl ResponseReader {
+    pub(crate) fn new(response: Response<Body>) -> Self {
+        Self { response }
+    }
+
+    pub(crate) async fn list_objects_v2_response(self) -> Result<ListObjectsV2Response, Error> {
+        let (parts, body) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::NOT_FOUND => bail!("bucket does not exist"),
+            status_code => bail!("unexpected status code {status_code}"),
+        }
+
+        let body = body.collect().await?.to_bytes();
+        let body = String::from_utf8(body.to_vec())?;
+
+        let date: HttpDate = Self::parse_header(header::DATE, &parts.headers)?;
+
+        let response: ListObjectsV2ResponseBody =
+            serde_xml_rs::from_str(&body).context("failed to parse response body")?;
+
+        Ok(response.with_date(date))
+    }
+
+    pub(crate) async fn head_object_response(self) -> Result<Option<HeadObjectResponse>, Error> {
+        let (parts, body) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::NOT_FOUND => return Ok(None),
+            status_code => bail!("unexpected status code {status_code}"),
+        }
+        let body = body.collect().await?.to_bytes();
+        if !body.is_empty() {
+            bail!("got unexpected non-empty response body");
+        }
+        println!("Headers {:?}", parts.headers);
+
+        let content_length: u64 = Self::parse_header(header::CONTENT_LENGTH, &parts.headers)?;
+        let content_type = Self::parse_header(header::CONTENT_TYPE, &parts.headers)?;
+        let e_tag = Self::parse_header(header::ETAG, &parts.headers)?;
+        let date = Self::parse_header(header::DATE, &parts.headers)?;
+        let last_modified = Self::parse_header(header::LAST_MODIFIED, &parts.headers)?;
+
+        Ok(Some(HeadObjectResponse {
+            content_length,
+            content_type,
+            date,
+            e_tag,
+            last_modified,
+        }))
+    }
+
+    pub(crate) async fn get_object_response(self) -> Result<Option<GetObjectResponse>, Error> {
+        let (parts, content) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::NOT_FOUND => return Ok(None),
+            StatusCode::FORBIDDEN => bail!("object is archived and inaccessible until restored"),
+            status_code => bail!("unexpected status code {status_code}"),
+        }
+
+        let content_length: u64 = Self::parse_header(header::CONTENT_LENGTH, &parts.headers)?;
+        let content_type = Self::parse_header(header::CONTENT_TYPE, &parts.headers)?;
+        let e_tag = Self::parse_header(header::ETAG, &parts.headers)?;
+        let date = Self::parse_header(header::DATE, &parts.headers)?;
+        let last_modified = Self::parse_header(header::LAST_MODIFIED, &parts.headers)?;
+
+        Ok(Some(GetObjectResponse {
+            content_length,
+            content_type,
+            date,
+            e_tag,
+            last_modified,
+            content,
+        }))
+    }
+
+    pub(crate) async fn copy_object_response(self) -> Result<CopyObjectResponse, Error> {
+        let (parts, content) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::NOT_FOUND => bail!("object not found"),
+            StatusCode::FORBIDDEN => bail!("the source object is not in the active tier"),
+            status_code => bail!("unexpected status code {status_code}"),
+        }
+
+        let body = content.collect().await?.to_bytes();
+        let body = String::from_utf8(body.to_vec())?;
+
+        let x_amz_version_id = match parts.headers.get("x-amz-version-id") {
+            Some(version_id) => Some(
+                version_id
+                    .to_str()
+                    .context("failed to parse version id header")?
+                    .to_owned(),
+            ),
+            None => None,
+        };
+
+        let copy_object_result: CopyObjectResult =
+            serde_xml_rs::from_str(&body).context("failed to parse response body")?;
+
+        Ok(CopyObjectResponse {
+            copy_object_result,
+            x_amz_version_id,
+        })
+    }
+
+    pub(crate) async fn put_object_response(self) -> Result<PutObjectResponse, Error> {
+        let (parts, body) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            // If-None-Match precondition failed, an object with same key already present.
+            // FIXME: Should this be dropped in favor of re-uploading and rely on the local
+            // cache to detect duplicates to increase data safety guarantees?
+            StatusCode::PRECONDITION_FAILED => return Ok(PutObjectResponse::PreconditionFailed),
+            StatusCode::CONFLICT => return Ok(PutObjectResponse::NeedsRetry),
+            StatusCode::BAD_REQUEST => bail!("invalid request: {body:?}"),
+            status_code => bail!("unexpected status code {status_code}"),
+        };
+
+        let body = body.collect().await?.to_bytes();
+        if !body.is_empty() {
+            bail!("got unexpected non-empty response body");
+        }
+
+        let e_tag = Self::parse_header(header::ETAG, &parts.headers)?;
+
+        Ok(PutObjectResponse::Success(e_tag))
+    }
+
+    pub(crate) async fn delete_object_response(self) -> Result<(), Error> {
+        let (parts, _body) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::NO_CONTENT => (),
+            status_code => bail!("unexpected status code {status_code}"),
+        };
+
+        Ok(())
+    }
+
+    pub(crate) async fn delete_objects_response(self) -> Result<DeleteObjectsResponse, Error> {
+        let (parts, body) = self.response.into_parts();
+
+        match parts.status {
+            StatusCode::OK => (),
+            StatusCode::BAD_REQUEST => bail!("invalid request: {body:?}"),
+            status_code => bail!("unexpected status code {status_code}"),
+        };
+
+        let body = body.collect().await?.to_bytes();
+        let body = String::from_utf8(body.to_vec())?;
+
+        let delete_objects_response: DeleteObjectsResponse =
+            serde_xml_rs::from_str(&body).context("failed to parse response body")?;
+
+        Ok(delete_objects_response)
+    }
+
+    fn parse_header<T: FromStr>(name: HeaderName, headers: &HeaderMap) -> Result<T, Error>
+    where
+        <T as FromStr>::Err: Send + Sync + 'static,
+        Result<T, <T as FromStr>::Err>: Context<T, <T as FromStr>::Err>,
+    {
+        let header_value = headers
+            .get(&name)
+            .ok_or_else(|| anyhow!("missing header '{name}'"))?;
+        let header_str = header_value
+            .to_str()
+            .with_context(|| format!("non UTF-8 header '{name}'"))?;
+        let value = header_str
+            .parse()
+            .with_context(|| format!("failed to parse header '{name}'"))?;
+        Ok(value)
+    }
+
+    // TODO: Integrity checks via CRC32 or SHA265 currently cannot be performed, since not
+    // supported by all S3 object store providers.
+    // See also:
+    // https://tracker.ceph.com/issues/63951
+    // https://tracker.ceph.com/issues/69105
+    // https://www.backblaze.com/docs/cloud-storage-s3-compatible-api
+    fn parse_x_amz_checksum_crc32_header(headers: &HeaderMap) -> Result<Option<u32>, Error> {
+        let x_amz_checksum_crc32 = match headers.get("x-amz-checksum-crc32") {
+            Some(x_amz_checksum_crc32) => x_amz_checksum_crc32,
+            None => return Ok(None),
+        };
+        let x_amz_checksum_crc32 = base64::decode(x_amz_checksum_crc32.to_str()?)?;
+        let x_amz_checksum_crc32: [u8; 4] = x_amz_checksum_crc32
+            .try_into()
+            .map_err(|_e| anyhow!("failed to convert x-amz-checksum-crc32 header"))?;
+        let x_amz_checksum_crc32 = u32::from_be_bytes(x_amz_checksum_crc32);
+        Ok(Some(x_amz_checksum_crc32))
+    }
+}
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:35 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:35 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 10/42] s3 client: add type for
 last modified timestamp in responses
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-11-c.ebner@proxmox.com>

Adds a helper to parse modified timestamps as encountered in s3 list
objects v2 and copy object api calls.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 Cargo.toml               |  2 ++
 pbs-s3-client/Cargo.toml |  3 +++
 pbs-s3-client/src/lib.rs | 20 ++++++++++++++++++++
 3 files changed, 25 insertions(+)

diff --git a/Cargo.toml b/Cargo.toml
index c2b0029ac..aaa79c2aa 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -131,6 +131,7 @@ handlebars = "3.0"
 hex = "0.4.3"
 hickory-resolver = { version = "0.24.1", default-features = false, features = [ "system-config", "tokio-runtime" ] }
 hyper = { version = "0.14", features = [ "backports", "deprecated", "full" ] }
+iso8601 = "0.4.1"
 libc = "0.2"
 log = "0.4.17"
 nix = "0.26.1"
@@ -144,6 +145,7 @@ regex = "1.5.5"
 rustyline = "9"
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
+serde_plain = "1.0"
 siphasher = "0.3"
 syslog = "6"
 tar = "0.4"
diff --git a/pbs-s3-client/Cargo.toml b/pbs-s3-client/Cargo.toml
index 11189ea50..3261a32bb 100644
--- a/pbs-s3-client/Cargo.toml
+++ b/pbs-s3-client/Cargo.toml
@@ -10,7 +10,10 @@ rust-version.workspace = true
 anyhow.workspace = true
 hex = { workspace = true, features = [ "serde" ] }
 hyper.workspace = true
+iso8601.workspace = true
 openssl.workspace = true
+serde.workspace = true
+serde_plain.workspace = true
 tracing.workspace = true
 url.workspace = true
 
diff --git a/pbs-s3-client/src/lib.rs b/pbs-s3-client/src/lib.rs
index a4081df15..dbe4bebcc 100644
--- a/pbs-s3-client/src/lib.rs
+++ b/pbs-s3-client/src/lib.rs
@@ -3,3 +3,23 @@ mod client;
 pub use client::{S3Client, S3ClientOptions};
 mod object_key;
 pub use object_key::{S3ObjectKey, S3_CONTENT_PREFIX};
+
+use std::time::Duration;
+
+use anyhow::{anyhow, bail, Error};
+
+#[derive(Debug)]
+pub struct LastModifiedTimestamp {
+    _datetime: iso8601::DateTime,
+}
+
+impl std::str::FromStr for LastModifiedTimestamp {
+    type Err = Error;
+
+    fn from_str(timestamp: &str) -> Result<Self, Self::Err> {
+        let _datetime = iso8601::datetime(timestamp).map_err(|err| anyhow!(err))?;
+        Ok(Self { _datetime })
+    }
+}
+
+serde_plain::derive_deserialize_from_fromstr!(LastModifiedTimestamp, "last modified timestamp");
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:40 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:40 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 15/42] api: datastore: check S3
 backend bucket access on datastore create
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-16-c.ebner@proxmox.com>

Check if the configured S3 object store backend can be reached and
the provided secrets have the permissions to access the bucket.

Perform the check before creating the chunk store, so it is not left
behind if the bucket cannot be reached.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/config/datastore.rs | 41 ++++++++++++++++++++++++++++++++----
 1 file changed, 37 insertions(+), 4 deletions(-)

diff --git a/src/api2/config/datastore.rs b/src/api2/config/datastore.rs
index b133be707..19b08b7e4 100644
--- a/src/api2/config/datastore.rs
+++ b/src/api2/config/datastore.rs
@@ -3,6 +3,7 @@ use std::path::{Path, PathBuf};
 use ::serde::{Deserialize, Serialize};
 use anyhow::{bail, Context, Error};
 use hex::FromHex;
+use pbs_s3_client::{S3Client, S3ClientOptions};
 use serde_json::Value;
 use tracing::{info, warn};
 
@@ -12,10 +13,10 @@ use proxmox_section_config::SectionConfigData;
 use proxmox_uuid::Uuid;
 
 use pbs_api_types::{
-    Authid, DataStoreConfig, DataStoreConfigUpdater, DatastoreNotify, DatastoreTuning, KeepOptions,
-    MaintenanceMode, PruneJobConfig, PruneJobOptions, DATASTORE_SCHEMA, PRIV_DATASTORE_ALLOCATE,
-    PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_MODIFY, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA,
-    UPID_SCHEMA,
+    Authid, DataStoreConfig, DataStoreConfigUpdater, DatastoreBackendConfig, DatastoreNotify,
+    DatastoreTuning, KeepOptions, MaintenanceMode, PruneJobConfig, PruneJobOptions, S3ClientConfig,
+    S3ClientSecretsConfig, DATASTORE_SCHEMA, PRIV_DATASTORE_ALLOCATE, PRIV_DATASTORE_AUDIT,
+    PRIV_DATASTORE_MODIFY, PRIV_SYS_MODIFY, PROXMOX_CONFIG_DIGEST_SCHEMA, UPID_SCHEMA,
 };
 use pbs_config::BackupLockGuard;
 use pbs_datastore::chunk_store::ChunkStore;
@@ -116,6 +117,38 @@ pub(crate) fn do_create_datastore(
             .parse_property_string(datastore.tuning.as_deref().unwrap_or(""))?,
     )?;
 
+    if let Some(ref backend_config) = datastore.backend {
+        let backend_config: DatastoreBackendConfig = backend_config.parse()?;
+        match backend_config {
+            DatastoreBackendConfig::Filesystem => (),
+            DatastoreBackendConfig::S3(ref s3_client_id) => {
+                let (config, _config_digest) =
+                    pbs_config::s3::config().context("failed to get s3 config")?;
+                let (secrets, _secrets_digest) =
+                    pbs_config::s3::secrets_config().context("failed to get s3 secrets")?;
+                let config: S3ClientConfig = config
+                    .lookup("s3client", s3_client_id)
+                    .with_context(|| format!("no '{s3_client_id}' in config"))?;
+                let secrets: S3ClientSecretsConfig = secrets
+                    .lookup("s3secrets", s3_client_id)
+                    .with_context(|| format!("no '{s3_client_id}' in secrets"))?;
+                let options = S3ClientOptions {
+                    host: config.host,
+                    port: config.port,
+                    bucket: config.bucket,
+                    region: config.region.unwrap_or("us-west-1".to_string()),
+                    fingerprint: config.fingerprint,
+                    access_key: config.access_key,
+                    secret_key: secrets.secret_key,
+                };
+                let s3_client = S3Client::new(options).context("failed to create s3 client")?;
+                // Fine to block since this runs in worker task
+                proxmox_async::runtime::block_on(s3_client.head_bucket())
+                    .context("failed to access bucket")?;
+            }
+        }
+    }
+
     let unmount_guard = if datastore.backing_device.is_some() {
         do_mount_device(datastore.clone())?;
         UnmountGuard::new(Some(path.clone()))
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:47 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:47 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 22/42] api: backup:
 conditionally upload manifest to S3 object store backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-23-c.ebner@proxmox.com>

Upload the manifest to the S3 object store backend after it has been
finished in the backup api call handler, if s3 is configured as
backend. Keep also the locally cached version for fast and efficient
listing of contents without the need to perform expensive (as in
monetary cost and IO latency) requests.

The datastore's metadata contents will be synced from the S3 backend
during datastore opening.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/environment.rs | 33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 72e369bcf..685b78e89 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -12,7 +12,7 @@ use serde_json::{json, Value};
 use proxmox_router::{RpcEnvironment, RpcEnvironmentType};
 use proxmox_sys::fs::{replace_file, CreateOptions};
 
-use pbs_api_types::Authid;
+use pbs_api_types::{Authid, MANIFEST_BLOB_NAME};
 use pbs_datastore::backup_info::{BackupDir, BackupInfo};
 use pbs_datastore::dynamic_index::DynamicIndexWriter;
 use pbs_datastore::fixed_index::FixedIndexWriter;
@@ -719,6 +719,37 @@ impl BackupEnvironment {
             }
         }
 
+        if let DatastoreBackend::S3(s3_client) = &self.backend {
+            // Upload manifest to S3 object store
+            let mut object_key = self.backup_dir.relative_path();
+            object_key.push(MANIFEST_BLOB_NAME.as_ref());
+            let mut path = self.datastore.base_path();
+            path.push(&object_key);
+            let mut manifest = std::fs::File::open(&path)?;
+            let mut buffer = Vec::new();
+            manifest.read_to_end(&mut buffer)?;
+            let data = Body::from(buffer);
+            let object_key = object_key
+                .as_os_str()
+                .to_str()
+                .ok_or_else(|| format_err!("invalid path"))?;
+            match proxmox_async::runtime::block_on(s3_client.put_object(object_key.into(), data))? {
+                PutObjectResponse::PreconditionFailed => {
+                    self.log(format!(
+                        "Upload of manifest failed, object {object_key} already present."
+                    ));
+                    bail!("upload of manifest failed");
+                }
+                PutObjectResponse::NeedsRetry => {
+                    self.log("Upload of manifest failed, reupload required.");
+                    bail!("concurrent operation, reupload required");
+                }
+                PutObjectResponse::Success(_content) => {
+                    self.log(format!("Uploaded manifest to object store: {object_key}"))
+                }
+            }
+        }
+
         self.datastore.try_ensure_sync_level()?;
 
         // marks the backup as successful
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:42 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:42 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 17/42] datastore: allow to get
 the backend for a datastore
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-18-c.ebner@proxmox.com>

Implements an enum with variants Filesystem and S3 to distinguish
between available backends. Filesystem will be used as default, if no
backend is configured in the datastores configuration. If the
datastore has an s3 backend configured, the backend method will
instantiate and s3 client and return it with the S3 variant.

This allows to instantiate the client once, keeping and reusing the
same open connection to the api for the lifetime of task or job, e.g.
in the backup writer/readers runtime environment.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/Cargo.toml       |  1 +
 pbs-datastore/src/datastore.rs | 46 ++++++++++++++++++++++++++++++++--
 pbs-datastore/src/lib.rs       |  1 +
 3 files changed, 46 insertions(+), 2 deletions(-)

diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index 7623adc28..3ee06c9bb 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -44,4 +44,5 @@ pbs-api-types.workspace = true
 pbs-buildcfg.workspace = true
 pbs-config.workspace = true
 pbs-key-config.workspace = true
+pbs-s3-client.workspace = true
 pbs-tools.workspace = true
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index cbf78ecb6..42d27d249 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -8,6 +8,7 @@ use std::time::Duration;
 
 use anyhow::{bail, format_err, Context, Error};
 use nix::unistd::{unlinkat, UnlinkatFlags};
+use pbs_s3_client::{S3Client, S3ClientOptions};
 use pbs_tools::lru_cache::LruCache;
 use tracing::{info, warn};
 
@@ -23,8 +24,9 @@ use proxmox_worker_task::WorkerTaskContext;
 
 use pbs_api_types::{
     ArchiveType, Authid, BackupGroupDeleteStats, BackupNamespace, BackupType, ChunkOrder,
-    DataStoreConfig, DatastoreFSyncLevel, DatastoreTuning, GarbageCollectionStatus,
-    MaintenanceMode, MaintenanceType, Operation, UPID,
+    DataStoreConfig, DatastoreBackendConfig, DatastoreFSyncLevel, DatastoreTuning,
+    GarbageCollectionStatus, MaintenanceMode, MaintenanceType, Operation, S3ClientConfig,
+    S3ClientSecretsConfig, UPID,
 };
 use pbs_config::BackupLockGuard;
 
@@ -125,6 +127,7 @@ pub struct DataStoreImpl {
     chunk_order: ChunkOrder,
     last_digest: Option<[u8; 32]>,
     sync_level: DatastoreFSyncLevel,
+    backend_config: DatastoreBackendConfig,
 }
 
 impl DataStoreImpl {
@@ -139,6 +142,7 @@ impl DataStoreImpl {
             chunk_order: Default::default(),
             last_digest: None,
             sync_level: Default::default(),
+            backend_config: Default::default(),
         })
     }
 }
@@ -194,6 +198,12 @@ impl Drop for DataStore {
     }
 }
 
+#[derive(Clone)]
+pub enum DatastoreBackend {
+    Filesystem,
+    S3(Arc<S3Client>),
+}
+
 impl DataStore {
     // This one just panics on everything
     #[doc(hidden)]
@@ -204,6 +214,32 @@ impl DataStore {
         })
     }
 
+    /// Get the backend for this datastore based on it's configuration
+    pub fn backend(&self) -> Result<DatastoreBackend, Error> {
+        let backend_type = match self.inner.backend_config {
+            DatastoreBackendConfig::Filesystem => DatastoreBackend::Filesystem,
+            DatastoreBackendConfig::S3(ref s3_client_id) => {
+                let (config, _config_digest) = pbs_config::s3::config()?;
+                let (secrets, _secrets_digest) = pbs_config::s3::secrets_config()?;
+                let config: S3ClientConfig = config.lookup("s3client", s3_client_id)?;
+                let secrets: S3ClientSecretsConfig = secrets.lookup("s3secrets", s3_client_id)?;
+                let options = S3ClientOptions {
+                    host: config.host,
+                    port: config.port,
+                    bucket: config.bucket,
+                    region: config.region.unwrap_or("us-west-1".to_string()),
+                    fingerprint: config.fingerprint,
+                    access_key: config.access_key,
+                    secret_key: secrets.secret_key,
+                };
+                let s3_client = S3Client::new(options)?;
+                DatastoreBackend::S3(Arc::new(s3_client))
+            }
+        };
+
+        Ok(backend_type)
+    }
+
     pub fn lookup_datastore(
         name: &str,
         operation: Option<Operation>,
@@ -381,6 +417,11 @@ impl DataStore {
                 .parse_property_string(config.tuning.as_deref().unwrap_or(""))?,
         )?;
 
+        let backend_config = match config.backend {
+            Some(config) => config.parse()?,
+            None => Default::default(),
+        };
+
         Ok(DataStoreImpl {
             chunk_store,
             gc_mutex: Mutex::new(()),
@@ -389,6 +430,7 @@ impl DataStore {
             chunk_order: tuning.chunk_order.unwrap_or_default(),
             last_digest,
             sync_level: tuning.sync_level.unwrap_or_default(),
+            backend_config,
         })
     }
 
diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index 5014b6c09..e6f65575b 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -203,6 +203,7 @@ pub use store_progress::StoreProgress;
 mod datastore;
 pub use datastore::{
     check_backup_owner, ensure_datastore_is_mounted, get_datastore_mount_status, DataStore,
+    DatastoreBackend,
 };
 
 mod hierarchy;
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:46 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:46 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 21/42] api: backup:
 conditionally upload indices to S3 object store backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-22-c.ebner@proxmox.com>

If the datastore is backed by an S3 compatible object store, upload
the dynamic or fixed index files to the object store after closing
them. The local index files are kept in the local caching datastore
to allow for fast and efficient content lookups, avoiding expensive
(as in monetary cost and IO latency) requests.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/backup/environment.rs | 65 ++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)

diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 393a8351d..72e369bcf 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -2,6 +2,7 @@ use anyhow::{bail, format_err, Context, Error};
 use pbs_config::BackupLockGuard;
 
 use std::collections::HashMap;
+use std::io::Read;
 use std::sync::{Arc, Mutex};
 use tracing::info;
 
@@ -479,6 +480,38 @@ impl BackupEnvironment {
             );
         }
 
+        // For S3 backends, upload the index file to the object store after closing
+        if let DatastoreBackend::S3(s3_client) = &self.backend {
+            let mut object_key = self.backup_dir.relative_path();
+            object_key.push(&data.name);
+            let object_key = object_key
+                .as_os_str()
+                .to_str()
+                .ok_or_else(|| format_err!("invalid file name"))?;
+
+            let mut full_path = self.datastore.base_path();
+            full_path.push(object_key);
+            let mut file = std::fs::File::open(&full_path)?;
+            let mut buffer = Vec::new();
+            file.read_to_end(&mut buffer)?;
+            let data = Body::from(buffer);
+            match proxmox_async::runtime::block_on(s3_client.put_object(object_key.into(), data))? {
+                PutObjectResponse::PreconditionFailed => {
+                    self.log(format!(
+                        "Upload of dynamic index failed, object key {object_key} already present"
+                    ));
+                    bail!("Upload of dynamic index failed");
+                }
+                PutObjectResponse::NeedsRetry => {
+                    self.log("Upload of dynamic index failed, reupload required");
+                    bail!("concurrent operation, reupload required");
+                }
+                PutObjectResponse::Success(_content) => {
+                    self.log(format!("Uploaded index file to object store: {object_key}"))
+                }
+            }
+        }
+
         self.log_upload_stat(
             &data.name,
             &csum,
@@ -553,6 +586,38 @@ impl BackupEnvironment {
             );
         }
 
+        // For S3 backends, upload the index file to the object store after closing
+        if let DatastoreBackend::S3(s3_client) = &self.backend {
+            let mut object_key = self.backup_dir.relative_path();
+            object_key.push(&data.name);
+            let object_key = object_key
+                .as_os_str()
+                .to_str()
+                .ok_or_else(|| format_err!("invalid file name"))?;
+
+            let mut full_path = self.datastore.base_path();
+            full_path.push(object_key);
+            let mut file = std::fs::File::open(&full_path)?;
+            let mut buffer = Vec::new();
+            file.read_to_end(&mut buffer)?;
+            let data = Body::from(buffer);
+            match proxmox_async::runtime::block_on(s3_client.put_object(object_key.into(), data))? {
+                PutObjectResponse::PreconditionFailed => {
+                    self.log(format!(
+                        "Upload of fixed index failed, object {object_key} already present."
+                    ));
+                    bail!("upload of fixed index failed");
+                }
+                PutObjectResponse::NeedsRetry => {
+                    self.log("Upload of fixed index failed, reupload required.");
+                    bail!("concurrent operation, reupload required");
+                }
+                PutObjectResponse::Success(_content) => {
+                    self.log(format!("Uploaded index file to object store: {object_key}"))
+                }
+            }
+        }
+
         self.log_upload_stat(
             &data.name,
             &expected_csum,
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:50 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:50 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 25/42] datastore: local chunk
 reader: read chunks based on backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-26-c.ebner@proxmox.com>

Get and store the datastore's backend on local chunk reader
instantiantion and fetch chunks based on the variant from either the
filesystem or the s3 object store.

By storing the backend variant, the s3 client is instantiated only
once and reused until the local chunk reader instance is dropped.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/Cargo.toml                |  2 ++
 pbs-datastore/src/local_chunk_reader.rs | 37 +++++++++++++++++++++----
 2 files changed, 33 insertions(+), 6 deletions(-)

diff --git a/pbs-datastore/Cargo.toml b/pbs-datastore/Cargo.toml
index 3ee06c9bb..323f5e270 100644
--- a/pbs-datastore/Cargo.toml
+++ b/pbs-datastore/Cargo.toml
@@ -13,6 +13,7 @@ crc32fast.workspace = true
 endian_trait.workspace = true
 futures.workspace = true
 hex = { workspace = true, features = [ "serde" ] }
+hyper.workspace = true
 libc.workspace = true
 log.workspace = true
 nix.workspace = true
@@ -28,6 +29,7 @@ zstd-safe.workspace = true
 pathpatterns.workspace = true
 pxar.workspace = true
 
+proxmox-async.workspace = true
 proxmox-borrow.workspace = true
 proxmox-human-byte.workspace = true
 proxmox-io.workspace = true
diff --git a/pbs-datastore/src/local_chunk_reader.rs b/pbs-datastore/src/local_chunk_reader.rs
index 05a70c068..a363059a1 100644
--- a/pbs-datastore/src/local_chunk_reader.rs
+++ b/pbs-datastore/src/local_chunk_reader.rs
@@ -3,17 +3,21 @@ use std::pin::Pin;
 use std::sync::Arc;
 
 use anyhow::{bail, Error};
+use hyper::body::HttpBody;
 
 use pbs_api_types::CryptMode;
+use pbs_s3_client::S3Client;
 use pbs_tools::crypt_config::CryptConfig;
 
 use crate::data_blob::DataBlob;
+use crate::datastore::DatastoreBackend;
 use crate::read_chunk::{AsyncReadChunk, ReadChunk};
 use crate::DataStore;
 
 #[derive(Clone)]
 pub struct LocalChunkReader {
     store: Arc<DataStore>,
+    backend: DatastoreBackend,
     crypt_config: Option<Arc<CryptConfig>>,
     crypt_mode: CryptMode,
 }
@@ -24,8 +28,11 @@ impl LocalChunkReader {
         crypt_config: Option<Arc<CryptConfig>>,
         crypt_mode: CryptMode,
     ) -> Self {
+        // TODO: Error handling!
+        let backend = store.backend().unwrap();
         Self {
             store,
+            backend,
             crypt_config,
             crypt_mode,
         }
@@ -47,10 +54,25 @@ impl LocalChunkReader {
     }
 }
 
+async fn fetch(s3_client: Arc<S3Client>, digest: &[u8; 32]) -> Result<DataBlob, Error> {
+    if let Some(response) = s3_client.get_object(digest.into()).await? {
+        let bytes = response.content.collect().await?.to_bytes();
+        DataBlob::from_raw(bytes.to_vec())
+    } else {
+        bail!("no object with digest {}", hex::encode(digest));
+    }
+}
+
 impl ReadChunk for LocalChunkReader {
     fn read_raw_chunk(&self, digest: &[u8; 32]) -> Result<DataBlob, Error> {
-        let chunk = self.store.load_chunk(digest)?;
+        let chunk = match &self.backend {
+            DatastoreBackend::Filesystem => self.store.load_chunk(digest)?,
+            DatastoreBackend::S3(s3_client) => {
+                proxmox_async::runtime::block_on(fetch(s3_client.clone(), digest))?
+            }
+        };
         self.ensure_crypt_mode(chunk.crypt_mode()?)?;
+
         Ok(chunk)
     }
 
@@ -69,11 +91,14 @@ impl AsyncReadChunk for LocalChunkReader {
         digest: &'a [u8; 32],
     ) -> Pin<Box<dyn Future<Output = Result<DataBlob, Error>> + Send + 'a>> {
         Box::pin(async move {
-            let (path, _) = self.store.chunk_path(digest);
-
-            let raw_data = tokio::fs::read(&path).await?;
-
-            let chunk = DataBlob::load_from_reader(&mut &raw_data[..])?;
+            let chunk = match &self.backend {
+                DatastoreBackend::Filesystem => {
+                    let (path, _) = self.store.chunk_path(digest);
+                    let raw_data = tokio::fs::read(&path).await?;
+                    DataBlob::load_from_reader(&mut &raw_data[..])?
+                }
+                DatastoreBackend::S3(s3_client) => fetch(s3_client.clone(), digest).await?,
+            };
             self.ensure_crypt_mode(chunk.crypt_mode()?)?;
 
             Ok(chunk)
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:49 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:49 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 24/42] api: reader: fetch chunks
 based on datastore backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-25-c.ebner@proxmox.com>

Read the chunk based on the datastores backend, reading from local
filesystem or fetching from S3 object store.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/reader/environment.rs | 12 +++++++----
 src/api2/reader/mod.rs         | 38 ++++++++++++++++++++++------------
 2 files changed, 33 insertions(+), 17 deletions(-)

diff --git a/src/api2/reader/environment.rs b/src/api2/reader/environment.rs
index 3b2f06f43..8924352b0 100644
--- a/src/api2/reader/environment.rs
+++ b/src/api2/reader/environment.rs
@@ -1,13 +1,14 @@
 use std::collections::HashSet;
 use std::sync::{Arc, RwLock};
 
+use anyhow::Error;
 use serde_json::{json, Value};
 
 use proxmox_router::{RpcEnvironment, RpcEnvironmentType};
 
 use pbs_api_types::Authid;
 use pbs_datastore::backup_info::BackupDir;
-use pbs_datastore::DataStore;
+use pbs_datastore::{DataStore, DatastoreBackend};
 use proxmox_rest_server::formatter::*;
 use proxmox_rest_server::WorkerTask;
 use tracing::info;
@@ -23,6 +24,7 @@ pub struct ReaderEnvironment {
     pub worker: Arc<WorkerTask>,
     pub datastore: Arc<DataStore>,
     pub backup_dir: BackupDir,
+    pub backend: DatastoreBackend,
     allowed_chunks: Arc<RwLock<HashSet<[u8; 32]>>>,
 }
 
@@ -33,8 +35,9 @@ impl ReaderEnvironment {
         worker: Arc<WorkerTask>,
         datastore: Arc<DataStore>,
         backup_dir: BackupDir,
-    ) -> Self {
-        Self {
+    ) -> Result<Self, Error> {
+        let backend = datastore.backend()?;
+        Ok(Self {
             result_attributes: json!({}),
             env_type,
             auth_id,
@@ -43,8 +46,9 @@ impl ReaderEnvironment {
             debug: tracing::enabled!(tracing::Level::DEBUG),
             formatter: JSON_FORMATTER,
             backup_dir,
+            backend,
             allowed_chunks: Arc::new(RwLock::new(HashSet::new())),
-        }
+        })
     }
 
     pub fn log<S: AsRef<str>>(&self, msg: S) {
diff --git a/src/api2/reader/mod.rs b/src/api2/reader/mod.rs
index cc791299c..3417f49be 100644
--- a/src/api2/reader/mod.rs
+++ b/src/api2/reader/mod.rs
@@ -24,7 +24,8 @@ use pbs_api_types::{
 };
 use pbs_config::CachedUserInfo;
 use pbs_datastore::index::IndexFile;
-use pbs_datastore::{DataStore, PROXMOX_BACKUP_READER_PROTOCOL_ID_V1};
+use pbs_datastore::{DataStore, DatastoreBackend, PROXMOX_BACKUP_READER_PROTOCOL_ID_V1};
+use pbs_s3_client::S3Client;
 use pbs_tools::json::required_string_param;
 
 use crate::api2::backup::optional_ns_param;
@@ -159,7 +160,7 @@ fn upgrade_to_backup_reader_protocol(
                     worker.clone(),
                     datastore,
                     backup_dir,
-                );
+                )?;
 
                 env.debug = debug;
 
@@ -320,17 +321,10 @@ fn download_chunk(
             ));
         }
 
-        let (path, _) = env.datastore.chunk_path(&digest);
-        let path2 = path.clone();
-
-        env.debug(format!("download chunk {:?}", path));
-
-        let data =
-            proxmox_async::runtime::block_in_place(|| std::fs::read(path)).map_err(move |err| {
-                http_err!(BAD_REQUEST, "reading file {:?} failed: {}", path2, err)
-            })?;
-
-        let body = Body::from(data);
+        let body = match &env.backend {
+            DatastoreBackend::Filesystem => load_from_filesystem(env, &digest)?,
+            DatastoreBackend::S3(s3_client) => fetch_from_object_store(s3_client, &digest).await?,
+        };
 
         // fixme: set other headers ?
         Ok(Response::builder()
@@ -342,6 +336,24 @@ fn download_chunk(
     .boxed()
 }
 
+async fn fetch_from_object_store(s3_client: &S3Client, digest: &[u8; 32]) -> Result<Body, Error> {
+    if let Some(response) = s3_client.get_object(digest.into()).await? {
+        return Ok(response.content);
+    }
+    bail!("cannot find chunk with digest {}", hex::encode(digest));
+}
+
+fn load_from_filesystem(env: &ReaderEnvironment, digest: &[u8; 32]) -> Result<Body, Error> {
+    let (path, _) = env.datastore.chunk_path(digest);
+    let path2 = path.clone();
+
+    env.debug(format!("download chunk {path:?}"));
+
+    let data = proxmox_async::runtime::block_in_place(|| std::fs::read(path))
+        .map_err(move |err| http_err!(BAD_REQUEST, "reading file {path2:?} failed: {err}"))?;
+    Ok(Body::from(data))
+}
+
 /* this is too slow
 fn download_chunk_old(
     _parts: Parts,
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:52 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:52 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 27/42] verify: implement chunk
 verification for stores with s3 backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-28-c.ebner@proxmox.com>

For datastores backed by an S3 compatible object store, rather than
reading the chunks to be verified from the local filesystem, fetch
them via the s3 client from the configured bucket.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/backup/verify.rs | 59 +++++++++++++++++++++++++++++++++++---------
 1 file changed, 47 insertions(+), 12 deletions(-)

diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index a01ddcca3..2c28c6af5 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -5,6 +5,7 @@ use std::sync::{Arc, Mutex};
 use std::time::Instant;
 
 use anyhow::{bail, Error};
+use hyper::body::HttpBody;
 use tracing::{error, info, warn};
 
 use proxmox_worker_task::WorkerTaskContext;
@@ -189,18 +190,52 @@ impl VerifyWorker {
                 continue; // already verified or marked corrupt
             }
 
-            match self.datastore.load_chunk(&info.digest) {
-                Err(err) => {
-                    self.corrupt_chunks.lock().unwrap().insert(info.digest);
-                    error!("can't verify chunk, load failed - {err}");
-                    errors.fetch_add(1, Ordering::SeqCst);
-                    Self::rename_corrupted_chunk(self.datastore.clone(), &info.digest);
-                }
-                Ok(chunk) => {
-                    let size = info.size();
-                    read_bytes += chunk.raw_size();
-                    decoder_pool.send((chunk, info.digest, size))?;
-                    decoded_bytes += size;
+            match &self.backend {
+                DatastoreBackend::Filesystem => match self.datastore.load_chunk(&info.digest) {
+                    Err(err) => {
+                        self.corrupt_chunks.lock().unwrap().insert(info.digest);
+                        error!("can't verify chunk, load failed - {err}");
+                        errors.fetch_add(1, Ordering::SeqCst);
+                        Self::rename_corrupted_chunk(self.datastore.clone(), &info.digest);
+                    }
+                    Ok(chunk) => {
+                        let size = info.size();
+                        read_bytes += chunk.raw_size();
+                        decoder_pool.send((chunk, info.digest, size))?;
+                        decoded_bytes += size;
+                    }
+                },
+                DatastoreBackend::S3(s3_client) => {
+                    //TODO: How to avoid all these requests? Does the AWS api offer other means
+                    // to verify the contents/integrity of objects?
+                    match proxmox_async::runtime::block_on(s3_client.get_object(info.digest.into()))
+                    {
+                        Ok(Some(response)) => {
+                            let bytes =
+                                proxmox_async::runtime::block_on(response.content.collect())?
+                                    .to_bytes();
+                            let chunk = DataBlob::from_raw(bytes.to_vec())?;
+                            let size = info.size();
+                            read_bytes += chunk.raw_size();
+                            decoder_pool.send((chunk, info.digest, size))?;
+                            decoded_bytes += size;
+                        }
+                        Ok(None) => {
+                            self.corrupt_chunks.lock().unwrap().insert(info.digest);
+                            error!(
+                                "can't verify missing chunk with digest {}",
+                                hex::encode(info.digest)
+                            );
+                            errors.fetch_add(1, Ordering::SeqCst);
+                        }
+                        Err(err) => {
+                            self.corrupt_chunks.lock().unwrap().insert(info.digest);
+                            error!("can't verify chunk, load failed - {err}");
+                            errors.fetch_add(1, Ordering::SeqCst);
+                            //TODO: How to handle corrupt chunks for S3 store?
+                            //Self::rename_corrupted_chunk(self.datastore.clone(), &info.digest);
+                        }
+                    }
                 }
             }
         }
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:56 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:56 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 31/42] datastore: get and set
 owner for S3 store backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-32-c.ebner@proxmox.com>

Read or write the ownership information from/to the corresponding
object in the S3 object store. Keep that information available if
the bucket is reused as datastore.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 39 ++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index d016e2139..52ec8218e 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -816,6 +816,25 @@ impl DataStore {
         backup_group: &pbs_api_types::BackupGroup,
     ) -> Result<Authid, Error> {
         let full_path = self.owner_path(ns, backup_group);
+
+        if let DatastoreBackend::S3(s3_client) = self.backend()? {
+            let object_key = format!(
+                "{}/{backup_group}/owner",
+                ns.path()
+                    .to_str()
+                    .ok_or_else(|| format_err!("unexpected owner path"))?,
+            );
+            let response =
+                proxmox_async::runtime::block_on(s3_client.get_object(object_key.as_str().into()))?
+                    .ok_or_else(|| format_err!("fetching owner failed"))?;
+            let content =
+                proxmox_async::runtime::block_on(hyper::body::HttpBody::collect(response.content))?;
+            let owner = String::from_utf8(content.to_bytes().trim_ascii_end().to_vec())?;
+            return owner
+                .parse()
+                .map_err(|err| format_err!("parsing owner for {backup_group} failed: {err}"));
+        }
+
         let owner = proxmox_sys::fs::file_read_firstline(full_path)?;
         owner
             .trim_end() // remove trailing newline
@@ -844,6 +863,26 @@ impl DataStore {
     ) -> Result<(), Error> {
         let path = self.owner_path(ns, backup_group);
 
+        if let DatastoreBackend::S3(s3_client) = self.backend()? {
+            let object_key = format!(
+                "{}/{backup_group}/owner",
+                ns.path()
+                    .to_str()
+                    .ok_or_else(|| format_err!("unexpected owner path"))?,
+            );
+            let data = hyper::body::Body::from(format!("{auth_id}\n"));
+            let response = proxmox_async::runtime::block_on(
+                s3_client.put_object(object_key.as_str().into(), data),
+            )?;
+            match response {
+                PutObjectResponse::NeedsRetry => bail!("failed to set owner, needs retry"),
+                PutObjectResponse::PreconditionFailed => {
+                    bail!("failed to set owner, precondition failed")
+                }
+                PutObjectResponse::Success(_) => (),
+            }
+        }
+
         let mut open_options = std::fs::OpenOptions::new();
         open_options.write(true);
 
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:54 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:54 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 29/42] datastore: create/delete
 protected marker file on S3 storage backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-30-c.ebner@proxmox.com>

Commit 8292d3d2 ("api2/admin/datastore: add get/set_protection")
introduced the protected flag for backup snapshots, considering
snapshots as protected based on the presence/absence of the
`.protected` marker file in the corresponding snapshot directory.

To allow independent recovery of a datastore backed by an S3 bucket,
also create/delete the marker file on the object store backend. For
actual checks, still rely on the marker as encountered in the local
cache store.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 45 ++++++++++++++++++++++++++++++----
 1 file changed, 40 insertions(+), 5 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index ab5c22501..5c8b49947 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1539,12 +1539,47 @@ impl DataStore {
 
         let protected_path = backup_dir.protected_file();
         if protection {
-            std::fs::File::create(protected_path)
+            std::fs::File::create(&protected_path)
                 .map_err(|err| format_err!("could not create protection file: {}", err))?;
-        } else if let Err(err) = std::fs::remove_file(protected_path) {
-            // ignore error for non-existing file
-            if err.kind() != std::io::ErrorKind::NotFound {
-                bail!("could not remove protection file: {}", err);
+            if let DatastoreBackend::S3(s3_client) = self.backend()? {
+                let marker = backup_dir.relative_path().join(".protected");
+                let protected_marker = marker
+                    .to_str()
+                    .ok_or_else(|| format_err!("unexpected protected marker path"))?;
+                let response = proxmox_async::runtime::block_on(
+                    s3_client.put_object(protected_marker.into(), hyper::body::Body::empty()),
+                )?;
+                match response {
+                    PutObjectResponse::NeedsRetry => {
+                        let _ = std::fs::remove_file(protected_path);
+                        bail!("failed to mark snapshot as protected, needs retry")
+                    }
+                    PutObjectResponse::PreconditionFailed => {
+                        let _ = std::fs::remove_file(protected_path);
+                        bail!("failed to mark snapshot as protected, precondition failed")
+                    }
+                    PutObjectResponse::Success(_) => (),
+                }
+            }
+        } else {
+            if let Err(err) = std::fs::remove_file(&protected_path) {
+                // ignore error for non-existing file
+                if err.kind() != std::io::ErrorKind::NotFound {
+                    bail!("could not remove protection file: {err}");
+                }
+            }
+            if let DatastoreBackend::S3(s3_client) = self.backend()? {
+                let marker = backup_dir.relative_path().join(".protected");
+                let protected_marker = marker
+                    .to_str()
+                    .ok_or_else(|| format_err!("unexpected protected marker path"))?;
+                if let Err(err) = proxmox_async::runtime::block_on(
+                    s3_client.delete_object(protected_marker.into()),
+                ) {
+                    std::fs::File::create(&protected_path)
+                        .map_err(|err| format_err!("could not re-create protection file: {err}"))?;
+                    return Err(err);
+                }
             }
         }
 
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:58 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:58 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 33/42] ui: add S3 client edit
 window for configuration create/edit
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-34-c.ebner@proxmox.com>

Adds an edit window for creating or editing S3 client configurations.
Loosely based on the same edit window for the remote configuration.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/window/S3BucketEdit.js | 125 +++++++++++++++++++++++++++++++++++++
 1 file changed, 125 insertions(+)
 create mode 100644 www/window/S3BucketEdit.js

diff --git a/www/window/S3BucketEdit.js b/www/window/S3BucketEdit.js
new file mode 100644
index 000000000..1491ddbe5
--- /dev/null
+++ b/www/window/S3BucketEdit.js
@@ -0,0 +1,125 @@
+Ext.define('PBS.window.S3BucketEdit', {
+    extend: 'Proxmox.window.Edit',
+    alias: 'widget.pbsS3BucketEdit',
+    mixins: ['Proxmox.Mixin.CBind'],
+
+    onlineHelp: 'backup_s3bucket',
+
+    isAdd: true,
+
+    subject: gettext('S3 Bucket'),
+
+    fieldDefaults: { labelWidth: 120 },
+
+    cbindData: function(initialConfig) {
+	let me = this;
+
+	let baseurl = '/api2/extjs/config/s3';
+	let id = initialConfig.id;
+
+	me.isCreate = !id;
+	me.url = id ? `${baseurl}/${id}` : baseurl;
+	me.method = id ? 'PUT' : 'POST';
+	me.autoLoad = !!id;
+	return {
+	    passwordEmptyText: me.isCreate ? '' : gettext('Unchanged'),
+	};
+    },
+
+    items: {
+	xtype: 'inputpanel',
+	column1: [
+	    {
+		xtype: 'pmxDisplayEditField',
+		name: 'id',
+		fieldLabel: gettext('Unique Identifier'),
+		renderer: Ext.htmlEncode,
+		allowBlank: false,
+		minLength: 4,
+		cbind: {
+		    editable: '{isCreate}',
+		},
+	    },
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'host',
+		fieldLabel: gettext('Host'),
+		allowBlank: false,
+		emptyText: gettext('FQDN or IP-address'),
+	    },
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'port',
+		fieldLabel: gettext('Port'),
+		emptyText: gettext("default"),
+		cbind: {
+		    deleteEmpty: '{!isCreate}',
+		},
+	    },
+	],
+
+	column2: [
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'bucket',
+		fieldLabel: gettext('Bucket'),
+		allowBlank: false,
+	    },
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'region',
+		fieldLabel: gettext('Region'),
+		emptyText: gettext("default"),
+	    },
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'access-key',
+		fieldLabel: gettext('Access Key'),
+		cbind: {
+		    emptyText: '{passwordEmptyText}',
+		    allowBlank: '{!isCreate}',
+		},
+	    },
+	    {
+		xtype: 'textfield',
+		name: 'secret-key',
+		inputType: 'password',
+		fieldLabel: gettext('Secret Key'),
+		cbind: {
+		    emptyText: '{passwordEmptyText}',
+		    allowBlank: '{!isCreate}',
+		},
+	    },
+	],
+
+	columnB: [
+	    {
+		xtype: 'proxmoxtextfield',
+		name: 'fingerprint',
+		fieldLabel: gettext('Fingerprint'),
+		emptyText: gettext("Server certificate's SHA-256 fingerprint, required for self-signed certificates"),
+		cbind: {
+		    deleteEmpty: '{!isCreate}',
+		},
+	    },
+	],
+    },
+
+    getValues: function() {
+	let me = this;
+	let values = me.callParent(arguments);
+
+	if (me.isCreate) {
+	    /// Secrets are stored into separate config, but set the same id for both configs
+	    values['secrets-id'] = values.id;
+	}
+	if (values['access-key'] === '') {
+	    delete values['access-key']
+	}
+	if (values['secret-key'] === '') {
+	    delete values['secret-key']
+	}
+
+	return values;
+    },
+});
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:55 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:55 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 30/42] datastore: prune
 groups/snapshots from S3 object store backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-31-c.ebner@proxmox.com>

When pruning a backup group or a backup snapshot for a datastore with
S3 object store backend, remove the associated objects by removing
them based on the prefix.

In order to exclude protected contents, add a filtering based on the
presence of the protected marker.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/backup_info.rs | 45 +++++++++++++++++++++++++++++---
 pbs-datastore/src/datastore.rs   | 34 +++++++++++++++++++++---
 src/api2/admin/datastore.rs      | 24 +++++++++++------
 3 files changed, 88 insertions(+), 15 deletions(-)

diff --git a/pbs-datastore/src/backup_info.rs b/pbs-datastore/src/backup_info.rs
index 1422fe865..b9ac286ad 100644
--- a/pbs-datastore/src/backup_info.rs
+++ b/pbs-datastore/src/backup_info.rs
@@ -8,6 +8,7 @@ use std::time::Duration;
 
 use anyhow::{bail, format_err, Context, Error};
 
+use pbs_s3_client::S3_CONTENT_PREFIX;
 use proxmox_sys::fs::{lock_dir_noblock, lock_dir_noblock_shared, replace_file, CreateOptions};
 use proxmox_systemd::escape_unit;
 
@@ -18,7 +19,7 @@ use pbs_api_types::{
 use pbs_config::{open_backup_lockfile, BackupLockGuard};
 
 use crate::manifest::{BackupManifest, MANIFEST_LOCK_NAME};
-use crate::{DataBlob, DataStore};
+use crate::{DataBlob, DataStore, DatastoreBackend};
 
 pub const DATASTORE_LOCKS_DIR: &str = "/run/proxmox-backup/locks";
 
@@ -214,7 +215,7 @@ impl BackupGroup {
     ///
     /// Returns `BackupGroupDeleteStats`, containing the number of deleted snapshots
     /// and number of protected snaphsots, which therefore were not removed.
-    pub fn destroy(&self) -> Result<BackupGroupDeleteStats, Error> {
+    pub fn destroy(&self, backend: &DatastoreBackend) -> Result<BackupGroupDeleteStats, Error> {
         let _guard = self
             .lock()
             .with_context(|| format!("while destroying group '{self:?}'"))?;
@@ -228,10 +229,26 @@ impl BackupGroup {
                 delete_stats.increment_protected_snapshots();
                 continue;
             }
-            snap.destroy(false)?;
+            // also for S3 cleanup local only, the actual S3 objects will be removed below,
+            // reducing the number of required API calls.
+            snap.destroy(false, &DatastoreBackend::Filesystem)?;
             delete_stats.increment_removed_snapshots();
         }
 
+        if let DatastoreBackend::S3(s3_client) = backend {
+            let path = self.relative_group_path();
+            let group_prefix = path
+                .to_str()
+                .ok_or_else(|| format_err!("invalid group path prefix"))?;
+            let prefix = format!("{S3_CONTENT_PREFIX}/{group_prefix}");
+            let delete_objects_error = proxmox_async::runtime::block_on(
+                s3_client.delete_objects_by_prefix_with_suffix_filter(&prefix, ".protected"),
+            )?;
+            if delete_objects_error {
+                bail!("deleting objects failed");
+            }
+        }
+
         // Note: make sure the old locking mechanism isn't used as `remove_dir_all` is not safe in
         // that case
         if delete_stats.all_removed() && !*OLD_LOCKING {
@@ -577,7 +594,7 @@ impl BackupDir {
     /// Destroy the whole snapshot, bails if it's protected
     ///
     /// Setting `force` to true skips locking and thus ignores if the backup is currently in use.
-    pub fn destroy(&self, force: bool) -> Result<(), Error> {
+    pub fn destroy(&self, force: bool, backend: &DatastoreBackend) -> Result<(), Error> {
         let (_guard, _manifest_guard);
         if !force {
             _guard = self
@@ -590,6 +607,19 @@ impl BackupDir {
             bail!("cannot remove protected snapshot"); // use special error type?
         }
 
+        if let DatastoreBackend::S3(s3_client) = backend {
+            let path = self.relative_path();
+            let snapshot_prefix = path
+                .to_str()
+                .ok_or_else(|| format_err!("invalid snapshot path"))?;
+            let prefix = format!("{S3_CONTENT_PREFIX}/{snapshot_prefix}");
+            let delete_objects_error =
+                proxmox_async::runtime::block_on(s3_client.delete_objects_by_prefix(&prefix))?;
+            if delete_objects_error {
+                bail!("deleting objects failed");
+            }
+        }
+
         let full_path = self.full_path();
         log::info!("removing backup snapshot {:?}", full_path);
         std::fs::remove_dir_all(&full_path).map_err(|err| {
@@ -619,6 +649,13 @@ impl BackupDir {
         // do to rectify the situation.
         if guard.is_ok() && group.list_backups()?.is_empty() && !*OLD_LOCKING {
             group.remove_group_dir()?;
+            if let DatastoreBackend::S3(s3_client) = backend {
+                let path = group.relative_group_path().join("owner");
+                let owner_key = path
+                    .to_str()
+                    .ok_or_else(|| format_err!("invalid group path prefix"))?;
+                proxmox_async::runtime::block_on(s3_client.delete_object(owner_key.into()))?;
+            }
         } else if let Err(err) = guard {
             log::debug!("{err:#}");
         }
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 5c8b49947..d016e2139 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -29,6 +29,7 @@ use pbs_api_types::{
     S3ClientSecretsConfig, UPID,
 };
 use pbs_config::BackupLockGuard;
+use pbs_s3_client::S3_CONTENT_PREFIX;
 
 use crate::backup_info::{BackupDir, BackupGroup, BackupInfo, OLD_LOCKING};
 use crate::chunk_store::ChunkStore;
@@ -643,7 +644,9 @@ impl DataStore {
         let mut stats = BackupGroupDeleteStats::default();
 
         for group in self.iter_backup_groups(ns.to_owned())? {
-            let delete_stats = group?.destroy()?;
+            let group = group?;
+            let backend = self.backend()?;
+            let delete_stats = group.destroy(&backend)?;
             stats.add(&delete_stats);
             removed_all_groups = removed_all_groups && delete_stats.all_removed();
         }
@@ -677,6 +680,8 @@ impl DataStore {
         let store = self.name();
         let mut removed_all_requested = true;
         let mut stats = BackupGroupDeleteStats::default();
+        let backend = self.backend()?;
+
         if delete_groups {
             log::info!("removing whole namespace recursively below {store}:/{ns}",);
             for ns in self.recursive_iter_backup_ns(ns.to_owned())? {
@@ -684,6 +689,20 @@ impl DataStore {
                 stats.add(&delete_stats);
                 removed_all_requested = removed_all_requested && removed_ns_groups;
             }
+
+            if let DatastoreBackend::S3(s3_client) = &backend {
+                let ns_dir = ns.path();
+                let ns_prefix = ns_dir
+                    .to_str()
+                    .ok_or_else(|| format_err!("invalid namespace path prefix"))?;
+                let prefix = format!("{S3_CONTENT_PREFIX}/{ns_prefix}");
+                let delete_objects_error = proxmox_async::runtime::block_on(
+                    s3_client.delete_objects_by_prefix_with_suffix_filter(&prefix, ".protected"),
+                )?;
+                if delete_objects_error {
+                    bail!("deleting objects failed");
+                }
+            }
         } else {
             log::info!("pruning empty namespace recursively below {store}:/{ns}");
         }
@@ -719,6 +738,15 @@ impl DataStore {
                         log::warn!("failed to remove namespace {ns} - {err}")
                     }
                 }
+                if let DatastoreBackend::S3(s3_client) = &backend {
+                    // Only remove the namespace marker, if it was empty,
+                    // than this is the same as the namespace being removed.
+                    let ns_dir = ns.path().join(NAMESPACE_MARKER_FILENAME);
+                    let ns_key = ns_dir
+                        .to_str()
+                        .ok_or_else(|| format_err!("invalid namespace path"))?;
+                    proxmox_async::runtime::block_on(s3_client.delete_object(ns_key.into()))?;
+                }
             }
         }
 
@@ -736,7 +764,7 @@ impl DataStore {
     ) -> Result<BackupGroupDeleteStats, Error> {
         let backup_group = self.backup_group(ns.clone(), backup_group.clone());
 
-        backup_group.destroy()
+        backup_group.destroy(&self.backend()?)
     }
 
     /// Remove a backup directory including all content
@@ -748,7 +776,7 @@ impl DataStore {
     ) -> Result<(), Error> {
         let backup_dir = self.backup_dir(ns.clone(), backup_dir.clone())?;
 
-        backup_dir.destroy(force)
+        backup_dir.destroy(force, &self.backend()?)
     }
 
     /// Returns the time of the last successful backup
diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 7b7f79b22..c62b980d1 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -432,7 +432,7 @@ pub async fn delete_snapshot(
 
         let snapshot = datastore.backup_dir(ns, backup_dir)?;
 
-        snapshot.destroy(false)?;
+        snapshot.destroy(false, &datastore.backend()?)?;
 
         Ok(Value::Null)
     })
@@ -1098,13 +1098,21 @@ pub fn prune(
             });
 
             if !keep {
-                if let Err(err) = backup_dir.destroy(false) {
-                    warn!(
-                        "failed to remove dir {:?}: {}",
-                        backup_dir.relative_path(),
-                        err,
-                    );
-                }
+                match datastore.backend() {
+                    Ok(backend) => {
+                        if let Err(err) = backup_dir.destroy(false, &backend) {
+                            warn!(
+                                "failed to remove dir {:?}: {}",
+                                backup_dir.relative_path(),
+                                err,
+                            );
+                        }
+                    }
+                    Err(err) => warn!(
+                        "failed to remove dir {:?}: {err}",
+                        backup_dir.relative_path()
+                    ),
+                };
             }
         }
         prune_result
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:57 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:57 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 32/42] datastore: implement
 garbage collection for s3 backend
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-33-c.ebner@proxmox.com>

Implements the garbage collection for datastore's backed by an s3
object store.
Take advantage of the local datastore by placing marker files in the
chunk store during phase 1 of the garbage collection, updating their
atime if already present. By this expensive api calls can be avoided
to update the object metadata (only possible via a copy object
operation).

The phase 2 is implemented by fetching a list of all the chunks via
the ListObjectsV2 api call, filtered by the chunk folder prefix.
This operation has to be performed in patches of 1000 objects, given
by the api's response limits.
For each object key, lookup the marker file and decide based on the
marker existence and it's atime if the chunk object needs to be
removed. Deletion happens via the delete objects operation, allowing
to delete multiple chunks by a single request.

This allows to efficiently lookup chunks which are not in use
anymore while being performant and cost effective.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs | 203 +++++++++++++++++++++++++++++----
 1 file changed, 178 insertions(+), 25 deletions(-)

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index 52ec8218e..c940c935e 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -4,7 +4,7 @@ use std::os::unix::ffi::OsStrExt;
 use std::os::unix::io::AsRawFd;
 use std::path::{Path, PathBuf};
 use std::sync::{Arc, LazyLock, Mutex};
-use std::time::Duration;
+use std::time::{Duration, SystemTime};
 
 use anyhow::{bail, format_err, Context, Error};
 use nix::unistd::{unlinkat, UnlinkatFlags};
@@ -1204,6 +1204,7 @@ impl DataStore {
         chunk_lru_cache: &mut LruCache<[u8; 32], ()>,
         status: &mut GarbageCollectionStatus,
         worker: &dyn WorkerTaskContext,
+        s3_client: Option<Arc<S3Client>>,
     ) -> Result<(), Error> {
         status.index_file_count += 1;
         status.index_data_bytes += index.index_bytes();
@@ -1218,21 +1219,41 @@ impl DataStore {
                 continue;
             }
 
-            if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
-                let hex = hex::encode(digest);
-                warn!(
-                    "warning: unable to access non-existent chunk {hex}, required by {file_name:?}"
-                );
-
-                // touch any corresponding .bad files to keep them around, meaning if a chunk is
-                // rewritten correctly they will be removed automatically, as well as if no index
-                // file requires the chunk anymore (won't get to this loop then)
-                for i in 0..=9 {
-                    let bad_ext = format!("{}.bad", i);
-                    let mut bad_path = PathBuf::new();
-                    bad_path.push(self.chunk_path(digest).0);
-                    bad_path.set_extension(bad_ext);
-                    self.inner.chunk_store.cond_touch_path(&bad_path, false)?;
+            match s3_client {
+                None => {
+                    // Filesystem backend
+                    if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
+                        let hex = hex::encode(digest);
+                        warn!(
+                            "warning: unable to access non-existent chunk {hex}, required by {file_name:?}"
+                        );
+
+                        // touch any corresponding .bad files to keep them around, meaning if a chunk is
+                        // rewritten correctly they will be removed automatically, as well as if no index
+                        // file requires the chunk anymore (won't get to this loop then)
+                        for i in 0..=9 {
+                            let bad_ext = format!("{}.bad", i);
+                            let mut bad_path = PathBuf::new();
+                            bad_path.push(self.chunk_path(digest).0);
+                            bad_path.set_extension(bad_ext);
+                            self.inner.chunk_store.cond_touch_path(&bad_path, false)?;
+                        }
+                    }
+                }
+                Some(ref _s3_client) => {
+                    // Update atime on local cache marker files.
+                    if !self.inner.chunk_store.cond_touch_chunk(digest, false)? {
+                        let (chunk_path, _digest) = self.chunk_path(digest);
+                        // Insert empty file as marker to tell GC phase2 that this is
+                        // a chunk still in-use, so to keep in the S3 object store.
+                        std::fs::File::options()
+                            .write(true)
+                            .create_new(true)
+                            .open(chunk_path)
+                            .with_context(|| {
+                                format!("failed to create marker for chunk {}", hex::encode(digest))
+                            })?;
+                    }
                 }
             }
         }
@@ -1244,6 +1265,7 @@ impl DataStore {
         status: &mut GarbageCollectionStatus,
         worker: &dyn WorkerTaskContext,
         cache_capacity: usize,
+        s3_client: Option<Arc<S3Client>>,
     ) -> Result<(), Error> {
         // Iterate twice over the datastore to fetch index files, even if this comes with an
         // additional runtime cost:
@@ -1333,6 +1355,7 @@ impl DataStore {
                                 &mut chunk_lru_cache,
                                 status,
                                 worker,
+                                s3_client.as_ref().cloned(),
                             )?;
 
                             if !unprocessed_index_list.remove(&path) {
@@ -1367,7 +1390,14 @@ impl DataStore {
                     continue;
                 }
             };
-            self.index_mark_used_chunks(index, &path, &mut chunk_lru_cache, status, worker)?;
+            self.index_mark_used_chunks(
+                index,
+                &path,
+                &mut chunk_lru_cache,
+                status,
+                worker,
+                s3_client.as_ref().cloned(),
+            )?;
             warn!("Marked chunks for unexpected index file at '{path:?}'");
         }
         if strange_paths_count > 0 {
@@ -1465,18 +1495,141 @@ impl DataStore {
                 1024 * 1024
             };
 
-            info!("Start GC phase1 (mark used chunks)");
+            let s3_client = match self.backend()? {
+                DatastoreBackend::Filesystem => None,
+                DatastoreBackend::S3(s3_client) => {
+                    proxmox_async::runtime::block_on(s3_client.head_bucket())
+                        .context("failed to reach bucket")?;
+                    Some(s3_client)
+                }
+            };
 
-            self.mark_used_chunks(&mut gc_status, worker, gc_cache_capacity)
-                .context("marking used chunks failed")?;
+            info!("Start GC phase1 (mark used chunks)");
 
-            info!("Start GC phase2 (sweep unused chunks)");
-            self.inner.chunk_store.sweep_unused_chunks(
-                oldest_writer,
-                min_atime,
+            self.mark_used_chunks(
                 &mut gc_status,
                 worker,
-            )?;
+                gc_cache_capacity,
+                s3_client.as_ref().cloned(),
+            )
+            .context("marking used chunks failed")?;
+
+            info!("Start GC phase2 (sweep unused chunks)");
+
+            if let Some(ref s3_client) = s3_client {
+                let mut chunk_count = 0;
+                let prefix = Some(".chunks/");
+                // Operates in batches of 1000 objects max per request
+                let mut list_bucket_result = proxmox_async::runtime::block_on(
+                    s3_client.list_objects_v2(prefix, None, None),
+                )?;
+
+                let mut delete_list = Vec::with_capacity(1000);
+                loop {
+                    for content in list_bucket_result.contents {
+                        // Check object is actually a chunk
+                        let digest = match Path::new(&content.key).file_name() {
+                            Some(file_name) => file_name,
+                            // should never be the case as objects will have a filename
+                            None => continue,
+                        };
+                        let bytes = digest.as_bytes();
+                        if bytes.len() != 64 && bytes.len() != 64 + ".0.bad".len() {
+                            continue;
+                        }
+                        if !bytes.iter().take(64).all(u8::is_ascii_hexdigit) {
+                            continue;
+                        }
+
+                        let bad = bytes.ends_with(b".bad");
+
+                        // Check local markers (created or atime updated during phase1) and
+                        // keep or delete chunk based on that.
+
+                        let mut chunk_path = self.base_path();
+                        chunk_path.push(&content.key);
+                        let atime = match std::fs::metadata(chunk_path) {
+                            Ok(stat) => stat.accessed()?,
+                            Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
+                                // File not found, delete by setting atime to unix epoch
+                                info!("Not found, mark for deletion: {}", content.key);
+                                SystemTime::UNIX_EPOCH
+                            }
+                            Err(err) => return Err(err.into()),
+                        };
+                        let atime = atime.duration_since(SystemTime::UNIX_EPOCH)?.as_secs() as i64;
+
+                        chunk_count += 1;
+
+                        if atime < min_atime {
+                            delete_list.push(content.key);
+                            if bad {
+                                gc_status.removed_bad += 1;
+                            } else {
+                                gc_status.removed_chunks += 1;
+                            }
+                            gc_status.removed_bytes += content.size;
+                        } else if atime < oldest_writer {
+                            if bad {
+                                gc_status.still_bad += 1;
+                            } else {
+                                gc_status.pending_chunks += 1;
+                            }
+                            gc_status.pending_bytes += content.size;
+                        } else {
+                            if !bad {
+                                gc_status.disk_chunks += 1;
+                            }
+                            gc_status.disk_bytes += content.size;
+                        }
+                    }
+
+                    if !delete_list.is_empty() {
+                        let delete_objects_result = proxmox_async::runtime::block_on(
+                            s3_client.delete_objects(&delete_list),
+                        )?;
+                        if let Some(_err) = delete_objects_result.error {
+                            bail!("failed to delete some objects");
+                        }
+                        delete_list.clear();
+                    }
+
+                    // Process next batch of chunks if there is more
+                    if list_bucket_result.is_truncated {
+                        list_bucket_result =
+                            proxmox_async::runtime::block_on(s3_client.list_objects_v2(
+                                prefix,
+                                None,
+                                list_bucket_result.next_continuation_token.as_deref(),
+                            ))?;
+                        continue;
+                    }
+
+                    break;
+                }
+                info!("processed {chunk_count} total chunks");
+
+                // Phase 2 GC of Filesystem backed storage is phase 3 for S3 backed GC
+                info!("Start GC phase3 (sweep unused chunk markers)");
+
+                let mut tmp_gc_status = GarbageCollectionStatus {
+                    upid: Some(upid.to_string()),
+                    ..Default::default()
+                };
+                self.inner.chunk_store.sweep_unused_chunks(
+                    oldest_writer,
+                    min_atime,
+                    &mut tmp_gc_status,
+                    worker,
+                )?;
+            } else {
+                self.inner.chunk_store.sweep_unused_chunks(
+                    oldest_writer,
+                    min_atime,
+                    &mut gc_status,
+                    worker,
+                )?;
+            }
 
             info!(
                 "Removed garbage: {}",
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:32:04 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:32:04 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 39/42] datastore: add local
 datastore cache for network attached storages
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-40-c.ebner@proxmox.com>

Use a local datastore as cache using LRU cache replacement policy for
operations on a datastore backed by a network, e.g. by an S3 object
store backend. The goal is to reduce number of requests to the
backend and thereby save costs (monetary as well as time).

The cacher allows to fetch cache items on cache misses via the access
method.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/datastore.rs                |  46 ++++++-
 pbs-datastore/src/lib.rs                      |   3 +
 .../src/local_datastore_lru_cache.rs          | 116 ++++++++++++++++++
 3 files changed, 164 insertions(+), 1 deletion(-)
 create mode 100644 pbs-datastore/src/local_datastore_lru_cache.rs

diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index c3ac63b32..409aec74c 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -37,8 +37,9 @@ use crate::dynamic_index::{DynamicIndexReader, DynamicIndexWriter};
 use crate::fixed_index::{FixedIndexReader, FixedIndexWriter};
 use crate::hierarchy::{ListGroups, ListGroupsType, ListNamespaces, ListNamespacesRecursive};
 use crate::index::IndexFile;
+use crate::local_datastore_lru_cache::S3Cacher;
 use crate::task_tracking::{self, update_active_operations};
-use crate::DataBlob;
+use crate::{DataBlob, LocalDatastoreLruCache};
 
 static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
     LazyLock::new(|| Mutex::new(HashMap::new()));
@@ -131,6 +132,7 @@ pub struct DataStoreImpl {
     last_digest: Option<[u8; 32]>,
     sync_level: DatastoreFSyncLevel,
     backend_config: DatastoreBackendConfig,
+    lru_store_caching: Option<LocalDatastoreLruCache>,
 }
 
 impl DataStoreImpl {
@@ -146,6 +148,7 @@ impl DataStoreImpl {
             last_digest: None,
             sync_level: Default::default(),
             backend_config: Default::default(),
+            lru_store_caching: None,
         })
     }
 }
@@ -243,6 +246,37 @@ impl DataStore {
         Ok(backend_type)
     }
 
+    pub fn cache(&self) -> Option<&LocalDatastoreLruCache> {
+        self.inner.lru_store_caching.as_ref()
+    }
+
+    /// Check if the digest is present in the local datastore cache.
+    /// Always returns false if there is no cache configured for this datastore.
+    pub fn cache_contains(&self, digest: &[u8; 32]) -> bool {
+        if let Some(cache) = self.inner.lru_store_caching.as_ref() {
+            return cache.contains(digest);
+        }
+        false
+    }
+
+    /// Insert digest as most recently used on in the cache.
+    /// Returns with success if there is no cache configured for this datastore.
+    pub fn cache_insert(&self, digest: &[u8; 32], chunk: &DataBlob) -> Result<(), Error> {
+        if let Some(cache) = self.inner.lru_store_caching.as_ref() {
+            return cache.insert(digest, chunk);
+        }
+        Ok(())
+    }
+
+    pub fn cacher(&self) -> Result<Option<S3Cacher>, Error> {
+        self.backend().map(|backend| match backend {
+            DatastoreBackend::S3(s3_client) => {
+                Some(S3Cacher::new(s3_client, self.inner.chunk_store.clone()))
+            }
+            DatastoreBackend::Filesystem => None,
+        })
+    }
+
     pub fn lookup_datastore(
         name: &str,
         operation: Option<Operation>,
@@ -425,6 +459,15 @@ impl DataStore {
             None => Default::default(),
         };
 
+        const LOCAL_DATASTORE_CACHE_SIZE: usize = 10_000_000;
+        let lru_store_caching = if let DatastoreBackendConfig::S3(_) = backend_config {
+            let cache =
+                LocalDatastoreLruCache::new(LOCAL_DATASTORE_CACHE_SIZE, chunk_store.clone());
+            Some(cache)
+        } else {
+            None
+        };
+
         Ok(DataStoreImpl {
             chunk_store,
             gc_mutex: Mutex::new(()),
@@ -434,6 +477,7 @@ impl DataStore {
             last_digest,
             sync_level: tuning.sync_level.unwrap_or_default(),
             backend_config,
+            lru_store_caching,
         })
     }
 
diff --git a/pbs-datastore/src/lib.rs b/pbs-datastore/src/lib.rs
index e6f65575b..f1ad3d4c2 100644
--- a/pbs-datastore/src/lib.rs
+++ b/pbs-datastore/src/lib.rs
@@ -216,3 +216,6 @@ pub use snapshot_reader::SnapshotReader;
 
 mod local_chunk_reader;
 pub use local_chunk_reader::LocalChunkReader;
+
+mod local_datastore_lru_cache;
+pub use local_datastore_lru_cache::LocalDatastoreLruCache;
diff --git a/pbs-datastore/src/local_datastore_lru_cache.rs b/pbs-datastore/src/local_datastore_lru_cache.rs
new file mode 100644
index 000000000..c711c5208
--- /dev/null
+++ b/pbs-datastore/src/local_datastore_lru_cache.rs
@@ -0,0 +1,116 @@
+//! Use a local datastore as cache for operations on a datastore attached via
+//! a network layer (e.g. via the S3 backend).
+
+use std::future::Future;
+use std::sync::Arc;
+
+use anyhow::{bail, Error};
+use hyper::body::HttpBody;
+
+use pbs_s3_client::S3Client;
+use pbs_tools::async_lru_cache::{AsyncCacher, AsyncLruCache};
+
+use crate::ChunkStore;
+use crate::DataBlob;
+
+#[derive(Clone)]
+pub struct S3Cacher {
+    client: Arc<S3Client>,
+    store: Arc<ChunkStore>,
+}
+
+impl AsyncCacher<[u8; 32], ()> for S3Cacher {
+    fn fetch(
+        &self,
+        key: [u8; 32],
+    ) -> Box<dyn Future<Output = Result<Option<()>, Error>> + Send + 'static> {
+        let client = self.client.clone();
+        let store = self.store.clone();
+        Box::new(async move {
+            match client.get_object(key.into()).await? {
+                None => bail!("could not fetch object with key {}", hex::encode(key)),
+                Some(response) => {
+                    let bytes = response.content.collect().await?.to_bytes();
+                    let chunk = DataBlob::from_raw(bytes.to_vec())?;
+                    store.insert_chunk(&chunk, &key)?;
+                    Ok(Some(()))
+                }
+            }
+        })
+    }
+}
+
+impl S3Cacher {
+    pub fn new(client: Arc<S3Client>, store: Arc<ChunkStore>) -> Self {
+        Self { client, store }
+    }
+}
+
+/// LRU cache using local datastore for caching chunks
+///
+/// Uses a LRU cache, but without storing the values in-memory but rather
+/// on the filesystem
+pub struct LocalDatastoreLruCache {
+    cache: AsyncLruCache<[u8; 32], ()>,
+    store: Arc<ChunkStore>,
+}
+
+impl LocalDatastoreLruCache {
+    pub fn new(capacity: usize, store: Arc<ChunkStore>) -> Self {
+        Self {
+            cache: AsyncLruCache::new(capacity),
+            store,
+        }
+    }
+
+    /// Insert a new chunk into the local datastore cache.
+    ///
+    /// Fails if the chunk cannot be inserted successfully.
+    pub fn insert(&self, digest: &[u8; 32], chunk: &DataBlob) -> Result<(), Error> {
+        self.store.insert_chunk(chunk, digest)?;
+        self.cache.insert(*digest, (), |digest| {
+            let (path, _digest_str) = self.store.chunk_path(&digest);
+            // Truncate to free up space but keep the inode around, since that
+            // is used as marker for chunks in use by garbage collection.
+            nix::unistd::truncate(&path, 0).map_err(Error::from)
+        })
+    }
+
+    /// Remove a chunk from the local datastore cache.
+    ///
+    /// Fails if the chunk cannot be deleted successfully.
+    pub fn remove(&self, digest: &[u8; 32]) -> Result<(), Error> {
+        self.cache.remove(*digest);
+        let (path, _digest_str) = self.store.chunk_path(digest);
+        std::fs::remove_file(path).map_err(Error::from)
+    }
+
+    pub async fn access(
+        &self,
+        digest: &[u8; 32],
+        cacher: &mut S3Cacher,
+    ) -> Result<Option<DataBlob>, Error> {
+        if self
+            .cache
+            .access(*digest, cacher, |digest| {
+                let (path, _digest_str) = self.store.chunk_path(&digest);
+                // Truncate to free up space but keep the inode around, since that
+                // is used as marker for chunks in use by garbage collection.
+                nix::unistd::truncate(&path, 0).map_err(Error::from)
+            })
+            .await?
+            .is_some()
+        {
+            let (path, _digest_str) = self.store.chunk_path(digest);
+            let mut file = std::fs::File::open(&path)?;
+            let chunk = DataBlob::load_from_reader(&mut file)?;
+            Ok(Some(chunk))
+        } else {
+            Ok(None)
+        }
+    }
+
+    pub fn contains(&self, digest: &[u8; 32]) -> bool {
+        self.cache.contains(*digest)
+    }
+}
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:32:02 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:32:02 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 37/42] tools: lru cache: add
 removed callback for evicted cache nodes
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-38-c.ebner@proxmox.com>

Add a callback function to be executed on evicted cache nodes. The
callback gets the key of the removed node, allowing to externally act
based on that value.

Since the callback might fail, extend the current LRU cache api to
return an error on insert, covering the error for the `removed`
callback.

Async lru cache, callsites and tests are adapted to include the
additional callback parameter accordingly.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-datastore/src/cached_chunk_reader.rs |  6 +++-
 pbs-datastore/src/datastore.rs           |  2 +-
 pbs-datastore/src/dynamic_index.rs       |  1 +
 pbs-tools/src/async_lru_cache.rs         | 23 +++++++++----
 pbs-tools/src/lru_cache.rs               | 42 +++++++++++++++---------
 5 files changed, 50 insertions(+), 24 deletions(-)

diff --git a/pbs-datastore/src/cached_chunk_reader.rs b/pbs-datastore/src/cached_chunk_reader.rs
index be7f2a1e2..95ac23a54 100644
--- a/pbs-datastore/src/cached_chunk_reader.rs
+++ b/pbs-datastore/src/cached_chunk_reader.rs
@@ -81,7 +81,11 @@ impl<I: IndexFile, R: AsyncReadChunk + Send + Sync + 'static> CachedChunkReader<
                 let info = self.index.chunk_info(chunk.0).unwrap();
 
                 // will never be None, see AsyncChunkCacher
-                let data = self.cache.access(info.digest, &self.cacher).await?.unwrap();
+                let data = self
+                    .cache
+                    .access(info.digest, &self.cacher, |_| Ok(()))
+                    .await?
+                    .unwrap();
 
                 let want_bytes = ((info.range.end - cur_offset) as usize).min(size - read);
                 let slice = &mut buf[read..(read + want_bytes)];
diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
index c940c935e..c3ac63b32 100644
--- a/pbs-datastore/src/datastore.rs
+++ b/pbs-datastore/src/datastore.rs
@@ -1215,7 +1215,7 @@ impl DataStore {
             let digest = index.index_digest(pos).unwrap();
 
             // Avoid multiple expensive atime updates by utimensat
-            if chunk_lru_cache.insert(*digest, ()) {
+            if chunk_lru_cache.insert(*digest, (), |_| Ok(()))? {
                 continue;
             }
 
diff --git a/pbs-datastore/src/dynamic_index.rs b/pbs-datastore/src/dynamic_index.rs
index 8e9cb1163..e9d28c7de 100644
--- a/pbs-datastore/src/dynamic_index.rs
+++ b/pbs-datastore/src/dynamic_index.rs
@@ -599,6 +599,7 @@ impl<S: ReadChunk> BufferedDynamicReader<S> {
                     store: &mut self.store,
                     index: &self.index,
                 },
+                |_| Ok(()),
             )?
             .ok_or_else(|| format_err!("chunk not found by cacher"))?;
 
diff --git a/pbs-tools/src/async_lru_cache.rs b/pbs-tools/src/async_lru_cache.rs
index c43b87717..141114933 100644
--- a/pbs-tools/src/async_lru_cache.rs
+++ b/pbs-tools/src/async_lru_cache.rs
@@ -42,7 +42,16 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V: Clone + Send + 'static> AsyncL
     /// Access an item either via the cache or by calling cacher.fetch. A return value of Ok(None)
     /// means the item requested has no representation, Err(_) means a call to fetch() failed,
     /// regardless of whether it was initiated by this call or a previous one.
-    pub async fn access(&self, key: K, cacher: &dyn AsyncCacher<K, V>) -> Result<Option<V>, Error> {
+    /// Calls the removed callback on  the evicted item, if any.
+    pub async fn access<F>(
+        &self,
+        key: K,
+        cacher: &dyn AsyncCacher<K, V>,
+        removed: F,
+    ) -> Result<Option<V>, Error>
+    where
+        F: Fn(K) -> Result<(), Error>,
+    {
         let (owner, result_fut) = {
             // check if already requested
             let mut maps = self.maps.lock().unwrap();
@@ -71,7 +80,7 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V: Clone + Send + 'static> AsyncL
             // this call was the one initiating the request, put into LRU and remove from map
             let mut maps = self.maps.lock().unwrap();
             if let Ok(Some(ref value)) = result {
-                maps.0.insert(key, value.clone());
+                maps.0.insert(key, value.clone(), removed)?;
             }
             maps.1.remove(&key);
         }
@@ -106,15 +115,15 @@ mod test {
             let cache: AsyncLruCache<i32, String> = AsyncLruCache::new(2);
 
             assert_eq!(
-                cache.access(10, &cacher).await.unwrap(),
+                cache.access(10, &cacher, |_| Ok(())).await.unwrap(),
                 Some("x10".to_string())
             );
             assert_eq!(
-                cache.access(20, &cacher).await.unwrap(),
+                cache.access(20, &cacher, |_| Ok(())).await.unwrap(),
                 Some("x20".to_string())
             );
             assert_eq!(
-                cache.access(30, &cacher).await.unwrap(),
+                cache.access(30, &cacher, |_| Ok(())).await.unwrap(),
                 Some("x30".to_string())
             );
 
@@ -123,14 +132,14 @@ mod test {
                 tokio::spawn(async move {
                     let cacher = TestAsyncCacher { prefix: "y" };
                     assert_eq!(
-                        c.access(40, &cacher).await.unwrap(),
+                        c.access(40, &cacher, |_| Ok(())).await.unwrap(),
                         Some("y40".to_string())
                     );
                 });
             }
 
             assert_eq!(
-                cache.access(20, &cacher).await.unwrap(),
+                cache.access(20, &cacher, |_| Ok(())).await.unwrap(),
                 Some("x20".to_string())
             );
         });
diff --git a/pbs-tools/src/lru_cache.rs b/pbs-tools/src/lru_cache.rs
index 9e0112647..53b84ec41 100644
--- a/pbs-tools/src/lru_cache.rs
+++ b/pbs-tools/src/lru_cache.rs
@@ -60,10 +60,10 @@ impl<K, V> CacheNode<K, V> {
 /// assert_eq!(cache.get_mut(1), None);
 /// assert_eq!(cache.len(), 0);
 ///
-/// cache.insert(1, 1);
-/// cache.insert(2, 2);
-/// cache.insert(3, 3);
-/// cache.insert(4, 4);
+/// cache.insert(1, 1, |_| Ok(()));
+/// cache.insert(2, 2, |_| Ok(()));
+/// cache.insert(3, 3, |_| Ok(()));
+/// cache.insert(4, 4, |_| Ok(()));
 /// assert_eq!(cache.len(), 3);
 ///
 /// assert_eq!(cache.get_mut(1), None);
@@ -77,9 +77,9 @@ impl<K, V> CacheNode<K, V> {
 /// assert_eq!(cache.len(), 0);
 /// assert_eq!(cache.get_mut(2), None);
 /// // access will fill in missing cache entry by fetching from LruCacher
-/// assert_eq!(cache.access(2, &mut LruCacher {}).unwrap(), Some(&mut 2));
+/// assert_eq!(cache.access(2, &mut LruCacher {}, |_| Ok(())).unwrap(), Some(&mut 2));
 ///
-/// cache.insert(1, 1);
+/// cache.insert(1, 1, |_| Ok(()));
 /// assert_eq!(cache.get_mut(1), Some(&mut 1));
 ///
 /// cache.clear();
@@ -133,7 +133,10 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
 
     /// Insert or update an entry identified by `key` with the given `value`.
     /// This entry is placed as the most recently used node at the head.
-    pub fn insert(&mut self, key: K, value: V) -> bool {
+    pub fn insert<F>(&mut self, key: K, value: V, removed: F) -> Result<bool, anyhow::Error>
+    where
+        F: Fn(K) -> Result<(), anyhow::Error>,
+    {
         match self.map.entry(key) {
             Entry::Occupied(mut o) => {
                 // Node present, update value
@@ -142,7 +145,7 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
                 let mut node = unsafe { Box::from_raw(node_ptr) };
                 node.value = value;
                 let _node_ptr = Box::into_raw(node);
-                true
+                Ok(true)
             }
             Entry::Vacant(v) => {
                 // Node not present, insert a new one
@@ -158,9 +161,11 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
                 // avoid borrow conflict. This means there are temporarily
                 // self.capacity + 1 cache nodes.
                 if self.map.len() > self.capacity {
-                    self.pop_tail();
+                    if let Some(removed_node) = self.pop_tail() {
+                        removed(removed_node)?;
+                    }
                 }
-                false
+                Ok(false)
             }
         }
     }
@@ -174,11 +179,12 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
     }
 
     /// Remove the least recently used node from the cache.
-    fn pop_tail(&mut self) {
+    fn pop_tail(&mut self) -> Option<K> {
         if let Some(old_tail) = self.list.pop_tail() {
             // Remove HashMap entry for old tail
-            self.map.remove(&old_tail.key);
+            return self.map.remove(&old_tail.key).map(|_| old_tail.key);
         }
+        None
     }
 
     /// Get a mutable reference to the value identified by `key`.
@@ -206,11 +212,15 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
     /// value.
     /// If fetch returns a value, it is inserted as the most recently used entry
     /// in the cache.
-    pub fn access<'a>(
+    pub fn access<'a, F>(
         &'a mut self,
         key: K,
         cacher: &mut dyn Cacher<K, V>,
-    ) -> Result<Option<&'a mut V>, anyhow::Error> {
+        removed: F,
+    ) -> Result<Option<&'a mut V>, anyhow::Error>
+    where
+        F: Fn(K) -> Result<(), anyhow::Error>,
+    {
         match self.map.entry(key) {
             Entry::Occupied(mut o) => {
                 // Cache hit, birng node to front of list
@@ -234,7 +244,9 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V> LruCache<K, V> {
                         // avoid borrow conflict. This means there are temporarily
                         // self.capacity + 1 cache nodes.
                         if self.map.len() > self.capacity {
-                            self.pop_tail();
+                            if let Some(removed_node) = self.pop_tail() {
+                                removed(removed_node)?;
+                            }
                         }
                     }
                 }
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:31 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:31 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 06/42] verify: refactor verify
 related functions to be methods of worker
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-7-c.ebner@proxmox.com>

Instead of passing the VerifyWorker state as reference to the various
verification related functions, implement them as methods or
associated functions of the VerifyWorker. This does not only make
their correlation more clear, but it also reduces the number of
function call parameters and improves readability.

No functional changes intended.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/admin/datastore.rs    |  28 +-
 src/api2/backup/environment.rs |   7 +-
 src/backup/verify.rs           | 830 ++++++++++++++++-----------------
 src/server/verify_job.rs       |  12 +-
 4 files changed, 423 insertions(+), 454 deletions(-)

diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
index 392494488..7dc881ade 100644
--- a/src/api2/admin/datastore.rs
+++ b/src/api2/admin/datastore.rs
@@ -70,10 +70,7 @@ use proxmox_rest_server::{formatter, WorkerTask};
 
 use crate::api2::backup::optional_ns_param;
 use crate::api2::node::rrd::create_value_from_rrd;
-use crate::backup::{
-    check_ns_privs_full, verify_all_backups, verify_backup_dir, verify_backup_group, verify_filter,
-    ListAccessibleBackupGroups, NS_PRIVS_OK,
-};
+use crate::backup::{check_ns_privs_full, ListAccessibleBackupGroups, VerifyWorker, NS_PRIVS_OK};
 
 use crate::server::jobstate::{compute_schedule_status, Job, JobState};
 
@@ -896,14 +893,15 @@ pub fn verify(
         auth_id.to_string(),
         to_stdout,
         move |worker| {
-            let verify_worker = crate::backup::VerifyWorker::new(worker.clone(), datastore);
+            let verify_worker = VerifyWorker::new(worker.clone(), datastore);
             let failed_dirs = if let Some(backup_dir) = backup_dir {
                 let mut res = Vec::new();
-                if !verify_backup_dir(
-                    &verify_worker,
+                if !verify_worker.verify_backup_dir(
                     &backup_dir,
                     worker.upid().clone(),
-                    Some(&move |manifest| verify_filter(ignore_verified, outdated_after, manifest)),
+                    Some(&move |manifest| {
+                        VerifyWorker::verify_filter(ignore_verified, outdated_after, manifest)
+                    }),
                 )? {
                     res.push(print_ns_and_snapshot(
                         backup_dir.backup_ns(),
@@ -912,12 +910,13 @@ pub fn verify(
                 }
                 res
             } else if let Some(backup_group) = backup_group {
-                verify_backup_group(
-                    &verify_worker,
+                verify_worker.verify_backup_group(
                     &backup_group,
                     &mut StoreProgress::new(1),
                     worker.upid(),
-                    Some(&move |manifest| verify_filter(ignore_verified, outdated_after, manifest)),
+                    Some(&move |manifest| {
+                        VerifyWorker::verify_filter(ignore_verified, outdated_after, manifest)
+                    }),
                 )?
             } else {
                 let owner = if owner_check_required {
@@ -926,13 +925,14 @@ pub fn verify(
                     None
                 };
 
-                verify_all_backups(
-                    &verify_worker,
+                verify_worker.verify_all_backups(
                     worker.upid(),
                     ns,
                     max_depth,
                     owner,
-                    Some(&move |manifest| verify_filter(ignore_verified, outdated_after, manifest)),
+                    Some(&move |manifest| {
+                        VerifyWorker::verify_filter(ignore_verified, outdated_after, manifest)
+                    }),
                 )?
             };
             if !failed_dirs.is_empty() {
diff --git a/src/api2/backup/environment.rs b/src/api2/backup/environment.rs
index 3d541b461..6cd29f512 100644
--- a/src/api2/backup/environment.rs
+++ b/src/api2/backup/environment.rs
@@ -18,7 +18,7 @@ use pbs_datastore::fixed_index::FixedIndexWriter;
 use pbs_datastore::{DataBlob, DataStore};
 use proxmox_rest_server::{formatter::*, WorkerTask};
 
-use crate::backup::verify_backup_dir_with_lock;
+use crate::backup::VerifyWorker;
 
 use hyper::{Body, Response};
 
@@ -671,9 +671,8 @@ impl BackupEnvironment {
             move |worker| {
                 worker.log_message("Automatically verifying newly added snapshot");
 
-                let verify_worker = crate::backup::VerifyWorker::new(worker.clone(), datastore);
-                if !verify_backup_dir_with_lock(
-                    &verify_worker,
+                let verify_worker = VerifyWorker::new(worker.clone(), datastore);
+                if !verify_worker.verify_backup_dir_with_lock(
                     &backup_dir,
                     worker.upid().clone(),
                     None,
diff --git a/src/backup/verify.rs b/src/backup/verify.rs
index 3d2cba8ac..0b954ae23 100644
--- a/src/backup/verify.rs
+++ b/src/backup/verify.rs
@@ -44,517 +44,491 @@ impl VerifyWorker {
             corrupt_chunks: Arc::new(Mutex::new(HashSet::with_capacity(64))),
         }
     }
-}
-
-fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
-    let blob = backup_dir.load_blob(&info.filename)?;
 
-    let raw_size = blob.raw_size();
-    if raw_size != info.size {
-        bail!("wrong size ({} != {})", info.size, raw_size);
-    }
-
-    let csum = openssl::sha::sha256(blob.raw_data());
-    if csum != info.csum {
-        bail!("wrong index checksum");
-    }
+    fn verify_blob(backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
+        let blob = backup_dir.load_blob(&info.filename)?;
 
-    match blob.crypt_mode()? {
-        CryptMode::Encrypt => Ok(()),
-        CryptMode::None => {
-            // digest already verified above
-            blob.decode(None, None)?;
-            Ok(())
+        let raw_size = blob.raw_size();
+        if raw_size != info.size {
+            bail!("wrong size ({} != {})", info.size, raw_size);
         }
-        CryptMode::SignOnly => bail!("Invalid CryptMode for blob"),
-    }
-}
-
-fn rename_corrupted_chunk(datastore: Arc<DataStore>, digest: &[u8; 32]) {
-    let (path, digest_str) = datastore.chunk_path(digest);
 
-    let mut counter = 0;
-    let mut new_path = path.clone();
-    loop {
-        new_path.set_file_name(format!("{}.{}.bad", digest_str, counter));
-        if new_path.exists() && counter < 9 {
-            counter += 1;
-        } else {
-            break;
+        let csum = openssl::sha::sha256(blob.raw_data());
+        if csum != info.csum {
+            bail!("wrong index checksum");
         }
-    }
 
-    match std::fs::rename(&path, &new_path) {
-        Ok(_) => {
-            info!("corrupted chunk renamed to {:?}", &new_path);
-        }
-        Err(err) => {
-            match err.kind() {
-                std::io::ErrorKind::NotFound => { /* ignored */ }
-                _ => info!("could not rename corrupted chunk {:?} - {err}", &path),
+        match blob.crypt_mode()? {
+            CryptMode::Encrypt => Ok(()),
+            CryptMode::None => {
+                // digest already verified above
+                blob.decode(None, None)?;
+                Ok(())
             }
+            CryptMode::SignOnly => bail!("Invalid CryptMode for blob"),
         }
-    };
-}
+    }
 
-fn verify_index_chunks(
-    verify_worker: &VerifyWorker,
-    index: Box<dyn IndexFile + Send>,
-    crypt_mode: CryptMode,
-) -> Result<(), Error> {
-    let errors = Arc::new(AtomicUsize::new(0));
+    fn rename_corrupted_chunk(datastore: Arc<DataStore>, digest: &[u8; 32]) {
+        let (path, digest_str) = datastore.chunk_path(digest);
 
-    let start_time = Instant::now();
+        let mut counter = 0;
+        let mut new_path = path.clone();
+        loop {
+            new_path.set_file_name(format!("{}.{}.bad", digest_str, counter));
+            if new_path.exists() && counter < 9 {
+                counter += 1;
+            } else {
+                break;
+            }
+        }
 
-    let mut read_bytes = 0;
-    let mut decoded_bytes = 0;
+        match std::fs::rename(&path, &new_path) {
+            Ok(_) => {
+                info!("corrupted chunk renamed to {:?}", &new_path);
+            }
+            Err(err) => {
+                match err.kind() {
+                    std::io::ErrorKind::NotFound => { /* ignored */ }
+                    _ => info!("could not rename corrupted chunk {:?} - {err}", &path),
+                }
+            }
+        };
+    }
 
-    let datastore2 = Arc::clone(&verify_worker.datastore);
-    let corrupt_chunks2 = Arc::clone(&verify_worker.corrupt_chunks);
-    let verified_chunks2 = Arc::clone(&verify_worker.verified_chunks);
-    let errors2 = Arc::clone(&errors);
+    fn verify_index_chunks(
+        &self,
+        index: Box<dyn IndexFile + Send>,
+        crypt_mode: CryptMode,
+    ) -> Result<(), Error> {
+        let errors = Arc::new(AtomicUsize::new(0));
+
+        let start_time = Instant::now();
+
+        let mut read_bytes = 0;
+        let mut decoded_bytes = 0;
+
+        let datastore2 = Arc::clone(&self.datastore);
+        let corrupt_chunks2 = Arc::clone(&self.corrupt_chunks);
+        let verified_chunks2 = Arc::clone(&self.verified_chunks);
+        let errors2 = Arc::clone(&errors);
+
+        let decoder_pool = ParallelHandler::new(
+            "verify chunk decoder",
+            4,
+            move |(chunk, digest, size): (DataBlob, [u8; 32], u64)| {
+                let chunk_crypt_mode = match chunk.crypt_mode() {
+                    Err(err) => {
+                        corrupt_chunks2.lock().unwrap().insert(digest);
+                        info!("can't verify chunk, unknown CryptMode - {err}");
+                        errors2.fetch_add(1, Ordering::SeqCst);
+                        return Ok(());
+                    }
+                    Ok(mode) => mode,
+                };
+
+                if chunk_crypt_mode != crypt_mode {
+                    info!(
+                    "chunk CryptMode {chunk_crypt_mode:?} does not match index CryptMode {crypt_mode:?}"
+                );
+                    errors2.fetch_add(1, Ordering::SeqCst);
+                }
 
-    let decoder_pool = ParallelHandler::new(
-        "verify chunk decoder",
-        4,
-        move |(chunk, digest, size): (DataBlob, [u8; 32], u64)| {
-            let chunk_crypt_mode = match chunk.crypt_mode() {
-                Err(err) => {
+                if let Err(err) = chunk.verify_unencrypted(size as usize, &digest) {
                     corrupt_chunks2.lock().unwrap().insert(digest);
-                    info!("can't verify chunk, unknown CryptMode - {err}");
+                    info!("{err}");
                     errors2.fetch_add(1, Ordering::SeqCst);
-                    return Ok(());
+                    Self::rename_corrupted_chunk(datastore2.clone(), &digest);
+                } else {
+                    verified_chunks2.lock().unwrap().insert(digest);
                 }
-                Ok(mode) => mode,
-            };
 
-            if chunk_crypt_mode != crypt_mode {
-                info!(
-                    "chunk CryptMode {chunk_crypt_mode:?} does not match index CryptMode {crypt_mode:?}"
-                );
-                errors2.fetch_add(1, Ordering::SeqCst);
-            }
+                Ok(())
+            },
+        );
 
-            if let Err(err) = chunk.verify_unencrypted(size as usize, &digest) {
-                corrupt_chunks2.lock().unwrap().insert(digest);
-                info!("{err}");
-                errors2.fetch_add(1, Ordering::SeqCst);
-                rename_corrupted_chunk(datastore2.clone(), &digest);
+        let skip_chunk = |digest: &[u8; 32]| -> bool {
+            if self.verified_chunks.lock().unwrap().contains(digest) {
+                true
+            } else if self.corrupt_chunks.lock().unwrap().contains(digest) {
+                let digest_str = hex::encode(digest);
+                info!("chunk {digest_str} was marked as corrupt");
+                errors.fetch_add(1, Ordering::SeqCst);
+                true
             } else {
-                verified_chunks2.lock().unwrap().insert(digest);
+                false
             }
+        };
 
+        let check_abort = |pos: usize| -> Result<(), Error> {
+            if pos & 1023 == 0 {
+                self.worker.check_abort()?;
+                self.worker.fail_on_shutdown()?;
+            }
             Ok(())
-        },
-    );
-
-    let skip_chunk = |digest: &[u8; 32]| -> bool {
-        if verify_worker
-            .verified_chunks
-            .lock()
-            .unwrap()
-            .contains(digest)
-        {
-            true
-        } else if verify_worker
-            .corrupt_chunks
-            .lock()
-            .unwrap()
-            .contains(digest)
-        {
-            let digest_str = hex::encode(digest);
-            info!("chunk {digest_str} was marked as corrupt");
-            errors.fetch_add(1, Ordering::SeqCst);
-            true
-        } else {
-            false
-        }
-    };
-
-    let check_abort = |pos: usize| -> Result<(), Error> {
-        if pos & 1023 == 0 {
-            verify_worker.worker.check_abort()?;
-            verify_worker.worker.fail_on_shutdown()?;
-        }
-        Ok(())
-    };
+        };
 
-    let chunk_list =
-        verify_worker
+        let chunk_list = self
             .datastore
             .get_chunks_in_order(&*index, skip_chunk, check_abort)?;
 
-    for (pos, _) in chunk_list {
-        verify_worker.worker.check_abort()?;
-        verify_worker.worker.fail_on_shutdown()?;
+        for (pos, _) in chunk_list {
+            self.worker.check_abort()?;
+            self.worker.fail_on_shutdown()?;
 
-        let info = index.chunk_info(pos).unwrap();
+            let info = index.chunk_info(pos).unwrap();
 
-        // we must always recheck this here, the parallel worker below alter it!
-        if skip_chunk(&info.digest) {
-            continue; // already verified or marked corrupt
-        }
-
-        match verify_worker.datastore.load_chunk(&info.digest) {
-            Err(err) => {
-                verify_worker
-                    .corrupt_chunks
-                    .lock()
-                    .unwrap()
-                    .insert(info.digest);
-                error!("can't verify chunk, load failed - {err}");
-                errors.fetch_add(1, Ordering::SeqCst);
-                rename_corrupted_chunk(verify_worker.datastore.clone(), &info.digest);
+            // we must always recheck this here, the parallel worker below alter it!
+            if skip_chunk(&info.digest) {
+                continue; // already verified or marked corrupt
             }
-            Ok(chunk) => {
-                let size = info.size();
-                read_bytes += chunk.raw_size();
-                decoder_pool.send((chunk, info.digest, size))?;
-                decoded_bytes += size;
+
+            match self.datastore.load_chunk(&info.digest) {
+                Err(err) => {
+                    self.corrupt_chunks.lock().unwrap().insert(info.digest);
+                    error!("can't verify chunk, load failed - {err}");
+                    errors.fetch_add(1, Ordering::SeqCst);
+                    Self::rename_corrupted_chunk(self.datastore.clone(), &info.digest);
+                }
+                Ok(chunk) => {
+                    let size = info.size();
+                    read_bytes += chunk.raw_size();
+                    decoder_pool.send((chunk, info.digest, size))?;
+                    decoded_bytes += size;
+                }
             }
         }
-    }
 
-    decoder_pool.complete()?;
+        decoder_pool.complete()?;
+
+        let elapsed = start_time.elapsed().as_secs_f64();
 
-    let elapsed = start_time.elapsed().as_secs_f64();
+        let read_bytes_mib = (read_bytes as f64) / (1024.0 * 1024.0);
+        let decoded_bytes_mib = (decoded_bytes as f64) / (1024.0 * 1024.0);
 
-    let read_bytes_mib = (read_bytes as f64) / (1024.0 * 1024.0);
-    let decoded_bytes_mib = (decoded_bytes as f64) / (1024.0 * 1024.0);
+        let read_speed = read_bytes_mib / elapsed;
+        let decode_speed = decoded_bytes_mib / elapsed;
 
-    let read_speed = read_bytes_mib / elapsed;
-    let decode_speed = decoded_bytes_mib / elapsed;
+        let error_count = errors.load(Ordering::SeqCst);
 
-    let error_count = errors.load(Ordering::SeqCst);
+        info!(
+            "  verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors)"
+        );
 
-    info!(
-        "  verified {read_bytes_mib:.2}/{decoded_bytes_mib:.2} MiB in {elapsed:.2} seconds, speed {read_speed:.2}/{decode_speed:.2} MiB/s ({error_count} errors)"
-    );
+        if errors.load(Ordering::SeqCst) > 0 {
+            bail!("chunks could not be verified");
+        }
 
-    if errors.load(Ordering::SeqCst) > 0 {
-        bail!("chunks could not be verified");
+        Ok(())
     }
 
-    Ok(())
-}
+    fn verify_fixed_index(&self, backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
+        let mut path = backup_dir.relative_path();
+        path.push(&info.filename);
 
-fn verify_fixed_index(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
-    info: &FileInfo,
-) -> Result<(), Error> {
-    let mut path = backup_dir.relative_path();
-    path.push(&info.filename);
+        let index = self.datastore.open_fixed_reader(&path)?;
 
-    let index = verify_worker.datastore.open_fixed_reader(&path)?;
+        let (csum, size) = index.compute_csum();
+        if size != info.size {
+            bail!("wrong size ({} != {})", info.size, size);
+        }
 
-    let (csum, size) = index.compute_csum();
-    if size != info.size {
-        bail!("wrong size ({} != {})", info.size, size);
-    }
+        if csum != info.csum {
+            bail!("wrong index checksum");
+        }
 
-    if csum != info.csum {
-        bail!("wrong index checksum");
+        self.verify_index_chunks(Box::new(index), info.chunk_crypt_mode())
     }
 
-    verify_index_chunks(verify_worker, Box::new(index), info.chunk_crypt_mode())
-}
-
-fn verify_dynamic_index(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
-    info: &FileInfo,
-) -> Result<(), Error> {
-    let mut path = backup_dir.relative_path();
-    path.push(&info.filename);
+    fn verify_dynamic_index(&self, backup_dir: &BackupDir, info: &FileInfo) -> Result<(), Error> {
+        let mut path = backup_dir.relative_path();
+        path.push(&info.filename);
 
-    let index = verify_worker.datastore.open_dynamic_reader(&path)?;
+        let index = self.datastore.open_dynamic_reader(&path)?;
 
-    let (csum, size) = index.compute_csum();
-    if size != info.size {
-        bail!("wrong size ({} != {})", info.size, size);
-    }
-
-    if csum != info.csum {
-        bail!("wrong index checksum");
-    }
+        let (csum, size) = index.compute_csum();
+        if size != info.size {
+            bail!("wrong size ({} != {})", info.size, size);
+        }
 
-    verify_index_chunks(verify_worker, Box::new(index), info.chunk_crypt_mode())
-}
+        if csum != info.csum {
+            bail!("wrong index checksum");
+        }
 
-/// Verify a single backup snapshot
-///
-/// This checks all archives inside a backup snapshot.
-/// Errors are logged to the worker log.
-///
-/// Returns
-/// - Ok(true) if verify is successful
-/// - Ok(false) if there were verification errors
-/// - Err(_) if task was aborted
-pub fn verify_backup_dir(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
-    upid: UPID,
-    filter: Option<&dyn Fn(&BackupManifest) -> bool>,
-) -> Result<bool, Error> {
-    if !backup_dir.full_path().exists() {
-        info!(
-            "SKIPPED: verify {}:{} - snapshot does not exist (anymore).",
-            verify_worker.datastore.name(),
-            backup_dir.dir(),
-        );
-        return Ok(true);
+        self.verify_index_chunks(Box::new(index), info.chunk_crypt_mode())
     }
 
-    let snap_lock = backup_dir.lock_shared();
-
-    match snap_lock {
-        Ok(snap_lock) => {
-            verify_backup_dir_with_lock(verify_worker, backup_dir, upid, filter, snap_lock)
-        }
-        Err(err) => {
+    /// Verify a single backup snapshot
+    ///
+    /// This checks all archives inside a backup snapshot.
+    /// Errors are logged to the worker log.
+    ///
+    /// Returns
+    /// - Ok(true) if verify is successful
+    /// - Ok(false) if there were verification errors
+    /// - Err(_) if task was aborted
+    pub fn verify_backup_dir(
+        &self,
+        backup_dir: &BackupDir,
+        upid: UPID,
+        filter: Option<&dyn Fn(&BackupManifest) -> bool>,
+    ) -> Result<bool, Error> {
+        if !backup_dir.full_path().exists() {
             info!(
-                "SKIPPED: verify {}:{} - could not acquire snapshot lock: {}",
-                verify_worker.datastore.name(),
+                "SKIPPED: verify {}:{} - snapshot does not exist (anymore).",
+                self.datastore.name(),
                 backup_dir.dir(),
-                err,
             );
-            Ok(true)
+            return Ok(true);
         }
-    }
-}
 
-/// See verify_backup_dir
-pub fn verify_backup_dir_with_lock(
-    verify_worker: &VerifyWorker,
-    backup_dir: &BackupDir,
-    upid: UPID,
-    filter: Option<&dyn Fn(&BackupManifest) -> bool>,
-    _snap_lock: BackupLockGuard,
-) -> Result<bool, Error> {
-    let datastore_name = verify_worker.datastore.name();
-    let backup_dir_name = backup_dir.dir();
-
-    let manifest = match backup_dir.load_manifest() {
-        Ok((manifest, _)) => manifest,
-        Err(err) => {
-            info!("verify {datastore_name}:{backup_dir_name} - manifest load error: {err}");
-            return Ok(false);
-        }
-    };
+        let snap_lock = backup_dir.lock_shared();
 
-    if let Some(filter) = filter {
-        if !filter(&manifest) {
-            info!("SKIPPED: verify {datastore_name}:{backup_dir_name} (recently verified)");
-            return Ok(true);
+        match snap_lock {
+            Ok(snap_lock) => self.verify_backup_dir_with_lock(backup_dir, upid, filter, snap_lock),
+            Err(err) => {
+                info!(
+                    "SKIPPED: verify {}:{} - could not acquire snapshot lock: {}",
+                    self.datastore.name(),
+                    backup_dir.dir(),
+                    err,
+                );
+                Ok(true)
+            }
         }
     }
 
-    info!("verify {datastore_name}:{backup_dir_name}");
-
-    let mut error_count = 0;
+    /// See verify_backup_dir
+    pub fn verify_backup_dir_with_lock(
+        &self,
+        backup_dir: &BackupDir,
+        upid: UPID,
+        filter: Option<&dyn Fn(&BackupManifest) -> bool>,
+        _snap_lock: BackupLockGuard,
+    ) -> Result<bool, Error> {
+        let datastore_name = self.datastore.name();
+        let backup_dir_name = backup_dir.dir();
+
+        let manifest = match backup_dir.load_manifest() {
+            Ok((manifest, _)) => manifest,
+            Err(err) => {
+                info!("verify {datastore_name}:{backup_dir_name} - manifest load error: {err}");
+                return Ok(false);
+            }
+        };
 
-    let mut verify_result = VerifyState::Ok;
-    for info in manifest.files() {
-        let result = proxmox_lang::try_block!({
-            info!("  check {}", info.filename);
-            match ArchiveType::from_path(&info.filename)? {
-                ArchiveType::FixedIndex => verify_fixed_index(verify_worker, backup_dir, info),
-                ArchiveType::DynamicIndex => verify_dynamic_index(verify_worker, backup_dir, info),
-                ArchiveType::Blob => verify_blob(backup_dir, info),
+        if let Some(filter) = filter {
+            if !filter(&manifest) {
+                info!("SKIPPED: verify {datastore_name}:{backup_dir_name} (recently verified)");
+                return Ok(true);
             }
-        });
+        }
 
-        verify_worker.worker.check_abort()?;
-        verify_worker.worker.fail_on_shutdown()?;
+        info!("verify {datastore_name}:{backup_dir_name}");
 
-        if let Err(err) = result {
-            info!(
-                "verify {datastore_name}:{backup_dir_name}/{file_name} failed: {err}",
-                file_name = info.filename,
-            );
-            error_count += 1;
-            verify_result = VerifyState::Failed;
-        }
-    }
+        let mut error_count = 0;
 
-    let verify_state = SnapshotVerifyState {
-        state: verify_result,
-        upid,
-    };
-
-    if let Err(err) = {
-        let verify_state = serde_json::to_value(verify_state)?;
-        backup_dir.update_manifest(|manifest| {
-            manifest.unprotected["verify_state"] = verify_state;
-        })
-    } {
-        info!("verify {datastore_name}:{backup_dir_name} - manifest update error: {err}");
-        return Ok(false);
-    }
+        let mut verify_result = VerifyState::Ok;
+        for info in manifest.files() {
+            let result = proxmox_lang::try_block!({
+                info!("  check {}", info.filename);
+                match ArchiveType::from_path(&info.filename)? {
+                    ArchiveType::FixedIndex => self.verify_fixed_index(backup_dir, info),
+                    ArchiveType::DynamicIndex => self.verify_dynamic_index(backup_dir, info),
+                    ArchiveType::Blob => Self::verify_blob(backup_dir, info),
+                }
+            });
 
-    Ok(error_count == 0)
-}
+            self.worker.check_abort()?;
+            self.worker.fail_on_shutdown()?;
 
-/// Verify all backups inside a backup group
-///
-/// Errors are logged to the worker log.
-///
-/// Returns
-/// - Ok((count, failed_dirs)) where failed_dirs had verification errors
-/// - Err(_) if task was aborted
-pub fn verify_backup_group(
-    verify_worker: &VerifyWorker,
-    group: &BackupGroup,
-    progress: &mut StoreProgress,
-    upid: &UPID,
-    filter: Option<&dyn Fn(&BackupManifest) -> bool>,
-) -> Result<Vec<String>, Error> {
-    let mut errors = Vec::new();
-    let mut list = match group.list_backups() {
-        Ok(list) => list,
-        Err(err) => {
-            info!(
-                "verify {}, group {} - unable to list backups: {}",
-                print_store_and_ns(verify_worker.datastore.name(), group.backup_ns()),
-                group.group(),
-                err,
-            );
-            return Ok(errors);
-        }
-    };
-
-    let snapshot_count = list.len();
-    info!(
-        "verify group {}:{} ({} snapshots)",
-        verify_worker.datastore.name(),
-        group.group(),
-        snapshot_count
-    );
-
-    progress.group_snapshots = snapshot_count as u64;
-
-    BackupInfo::sort_list(&mut list, false); // newest first
-    for (pos, info) in list.into_iter().enumerate() {
-        if !verify_backup_dir(verify_worker, &info.backup_dir, upid.clone(), filter)? {
-            errors.push(print_ns_and_snapshot(
-                info.backup_dir.backup_ns(),
-                info.backup_dir.as_ref(),
-            ));
+            if let Err(err) = result {
+                info!(
+                    "verify {datastore_name}:{backup_dir_name}/{file_name} failed: {err}",
+                    file_name = info.filename,
+                );
+                error_count += 1;
+                verify_result = VerifyState::Failed;
+            }
         }
-        progress.done_snapshots = pos as u64 + 1;
-        info!("percentage done: {progress}");
-    }
-    Ok(errors)
-}
 
-/// Verify all (owned) backups inside a datastore
-///
-/// Errors are logged to the worker log.
-///
-/// Returns
-/// - Ok(failed_dirs) where failed_dirs had verification errors
-/// - Err(_) if task was aborted
-pub fn verify_all_backups(
-    verify_worker: &VerifyWorker,
-    upid: &UPID,
-    ns: BackupNamespace,
-    max_depth: Option<usize>,
-    owner: Option<&Authid>,
-    filter: Option<&dyn Fn(&BackupManifest) -> bool>,
-) -> Result<Vec<String>, Error> {
-    let mut errors = Vec::new();
-
-    info!("verify datastore {}", verify_worker.datastore.name());
-
-    let owner_filtered = if let Some(owner) = &owner {
-        info!("limiting to backups owned by {owner}");
-        true
-    } else {
-        false
-    };
-
-    // FIXME: This should probably simply enable recursion (or the call have a recursion parameter)
-    let store = &verify_worker.datastore;
-    let max_depth = max_depth.unwrap_or(pbs_api_types::MAX_NAMESPACE_DEPTH);
-
-    let mut list = match ListAccessibleBackupGroups::new_with_privs(
-        store,
-        ns.clone(),
-        max_depth,
-        Some(PRIV_DATASTORE_VERIFY),
-        Some(PRIV_DATASTORE_BACKUP),
-        owner,
-    ) {
-        Ok(list) => list
-            .filter_map(|group| match group {
-                Ok(group) => Some(group),
-                Err(err) if owner_filtered => {
-                    // intentionally not in task log, the user might not see this group!
-                    println!("error on iterating groups in ns '{ns}' - {err}");
-                    None
-                }
-                Err(err) => {
-                    // we don't filter by owner, but we want to log the error
-                    info!("error on iterating groups in ns '{ns}' - {err}");
-                    errors.push(err.to_string());
-                    None
-                }
-            })
-            .filter(|group| {
-                !(group.backup_type() == BackupType::Host && group.backup_id() == "benchmark")
+        let verify_state = SnapshotVerifyState {
+            state: verify_result,
+            upid,
+        };
+
+        if let Err(err) = {
+            let verify_state = serde_json::to_value(verify_state)?;
+            backup_dir.update_manifest(|manifest| {
+                manifest.unprotected["verify_state"] = verify_state;
             })
-            .collect::<Vec<BackupGroup>>(),
-        Err(err) => {
-            info!("unable to list backups: {err}");
-            return Ok(errors);
+        } {
+            info!("verify {datastore_name}:{backup_dir_name} - manifest update error: {err}");
+            return Ok(false);
         }
-    };
 
-    list.sort_unstable_by(|a, b| a.group().cmp(b.group()));
+        Ok(error_count == 0)
+    }
 
-    let group_count = list.len();
-    info!("found {group_count} groups");
+    /// Verify all backups inside a backup group
+    ///
+    /// Errors are logged to the worker log.
+    ///
+    /// Returns
+    /// - Ok((count, failed_dirs)) where failed_dirs had verification errors
+    /// - Err(_) if task was aborted
+    pub fn verify_backup_group(
+        &self,
+        group: &BackupGroup,
+        progress: &mut StoreProgress,
+        upid: &UPID,
+        filter: Option<&dyn Fn(&BackupManifest) -> bool>,
+    ) -> Result<Vec<String>, Error> {
+        let mut errors = Vec::new();
+        let mut list = match group.list_backups() {
+            Ok(list) => list,
+            Err(err) => {
+                info!(
+                    "verify {}, group {} - unable to list backups: {}",
+                    print_store_and_ns(self.datastore.name(), group.backup_ns()),
+                    group.group(),
+                    err,
+                );
+                return Ok(errors);
+            }
+        };
 
-    let mut progress = StoreProgress::new(group_count as u64);
+        let snapshot_count = list.len();
+        info!(
+            "verify group {}:{} ({} snapshots)",
+            self.datastore.name(),
+            group.group(),
+            snapshot_count
+        );
 
-    for (pos, group) in list.into_iter().enumerate() {
-        progress.done_groups = pos as u64;
-        progress.done_snapshots = 0;
-        progress.group_snapshots = 0;
+        progress.group_snapshots = snapshot_count as u64;
 
-        let mut group_errors =
-            verify_backup_group(verify_worker, &group, &mut progress, upid, filter)?;
-        errors.append(&mut group_errors);
+        BackupInfo::sort_list(&mut list, false); // newest first
+        for (pos, info) in list.into_iter().enumerate() {
+            if !self.verify_backup_dir(&info.backup_dir, upid.clone(), filter)? {
+                errors.push(print_ns_and_snapshot(
+                    info.backup_dir.backup_ns(),
+                    info.backup_dir.as_ref(),
+                ));
+            }
+            progress.done_snapshots = pos as u64 + 1;
+            info!("percentage done: {progress}");
+        }
+        Ok(errors)
     }
 
-    Ok(errors)
-}
+    /// Verify all (owned) backups inside a datastore
+    ///
+    /// Errors are logged to the worker log.
+    ///
+    /// Returns
+    /// - Ok(failed_dirs) where failed_dirs had verification errors
+    /// - Err(_) if task was aborted
+    pub fn verify_all_backups(
+        &self,
+        upid: &UPID,
+        ns: BackupNamespace,
+        max_depth: Option<usize>,
+        owner: Option<&Authid>,
+        filter: Option<&dyn Fn(&BackupManifest) -> bool>,
+    ) -> Result<Vec<String>, Error> {
+        let mut errors = Vec::new();
+
+        info!("verify datastore {}", self.datastore.name());
+
+        let owner_filtered = if let Some(owner) = &owner {
+            info!("limiting to backups owned by {owner}");
+            true
+        } else {
+            false
+        };
+
+        // FIXME: This should probably simply enable recursion (or the call have a recursion parameter)
+        let store = &self.datastore;
+        let max_depth = max_depth.unwrap_or(pbs_api_types::MAX_NAMESPACE_DEPTH);
+
+        let mut list = match ListAccessibleBackupGroups::new_with_privs(
+            store,
+            ns.clone(),
+            max_depth,
+            Some(PRIV_DATASTORE_VERIFY),
+            Some(PRIV_DATASTORE_BACKUP),
+            owner,
+        ) {
+            Ok(list) => list
+                .filter_map(|group| match group {
+                    Ok(group) => Some(group),
+                    Err(err) if owner_filtered => {
+                        // intentionally not in task log, the user might not see this group!
+                        println!("error on iterating groups in ns '{ns}' - {err}");
+                        None
+                    }
+                    Err(err) => {
+                        // we don't filter by owner, but we want to log the error
+                        info!("error on iterating groups in ns '{ns}' - {err}");
+                        errors.push(err.to_string());
+                        None
+                    }
+                })
+                .filter(|group| {
+                    !(group.backup_type() == BackupType::Host && group.backup_id() == "benchmark")
+                })
+                .collect::<Vec<BackupGroup>>(),
+            Err(err) => {
+                info!("unable to list backups: {err}");
+                return Ok(errors);
+            }
+        };
+
+        list.sort_unstable_by(|a, b| a.group().cmp(b.group()));
+
+        let group_count = list.len();
+        info!("found {group_count} groups");
 
-/// Filter out any snapshot from being (re-)verified where this fn returns false.
-pub fn verify_filter(
-    ignore_verified_snapshots: bool,
-    outdated_after: Option<i64>,
-    manifest: &BackupManifest,
-) -> bool {
-    if !ignore_verified_snapshots {
-        return true;
+        let mut progress = StoreProgress::new(group_count as u64);
+
+        for (pos, group) in list.into_iter().enumerate() {
+            progress.done_groups = pos as u64;
+            progress.done_snapshots = 0;
+            progress.group_snapshots = 0;
+
+            let mut group_errors = self.verify_backup_group(&group, &mut progress, upid, filter)?;
+            errors.append(&mut group_errors);
+        }
+
+        Ok(errors)
     }
 
-    match manifest.verify_state() {
-        Err(err) => {
-            warn!("error reading manifest: {err:#}");
-            true
+    /// Filter out any snapshot from being (re-)verified where this fn returns false.
+    pub fn verify_filter(
+        ignore_verified_snapshots: bool,
+        outdated_after: Option<i64>,
+        manifest: &BackupManifest,
+    ) -> bool {
+        if !ignore_verified_snapshots {
+            return true;
         }
-        Ok(None) => true, // no last verification, always include
-        Ok(Some(last_verify)) => {
-            match outdated_after {
-                None => false, // never re-verify if ignored and no max age
-                Some(max_age) => {
-                    let now = proxmox_time::epoch_i64();
-                    let days_since_last_verify = (now - last_verify.upid.starttime) / 86400;
-
-                    days_since_last_verify > max_age
+
+        match manifest.verify_state() {
+            Err(err) => {
+                warn!("error reading manifest: {err:#}");
+                true
+            }
+            Ok(None) => true, // no last verification, always include
+            Ok(Some(last_verify)) => {
+                match outdated_after {
+                    None => false, // never re-verify if ignored and no max age
+                    Some(max_age) => {
+                        let now = proxmox_time::epoch_i64();
+                        let days_since_last_verify = (now - last_verify.upid.starttime) / 86400;
+
+                        days_since_last_verify > max_age
+                    }
                 }
             }
         }
diff --git a/src/server/verify_job.rs b/src/server/verify_job.rs
index a15a257da..95a7b2a9b 100644
--- a/src/server/verify_job.rs
+++ b/src/server/verify_job.rs
@@ -5,10 +5,7 @@ use pbs_api_types::{Authid, Operation, VerificationJobConfig};
 use pbs_datastore::DataStore;
 use proxmox_rest_server::WorkerTask;
 
-use crate::{
-    backup::{verify_all_backups, verify_filter},
-    server::jobstate::Job,
-};
+use crate::{backup::VerifyWorker, server::jobstate::Job};
 
 /// Runs a verification job.
 pub fn do_verification_job(
@@ -44,15 +41,14 @@ pub fn do_verification_job(
                 None => Default::default(),
             };
 
-            let verify_worker = crate::backup::VerifyWorker::new(worker.clone(), datastore);
-            let result = verify_all_backups(
-                &verify_worker,
+            let verify_worker = VerifyWorker::new(worker.clone(), datastore);
+            let result = verify_worker.verify_all_backups(
                 worker.upid(),
                 ns,
                 verification_job.max_depth,
                 None,
                 Some(&move |manifest| {
-                    verify_filter(ignore_verified_snapshots, outdated_after, manifest)
+                    VerifyWorker::verify_filter(ignore_verified_snapshots, outdated_after, manifest)
                 }),
             );
             let job_result = match result {
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:33:45 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:33:45 +0200
Subject: [pbs-devel] superseded: [RFC proxmox proxmox-backup 00/39] S3
 storage backend for datastores
In-Reply-To: <20250519114640.303640-1-c.ebner@proxmox.com>
References: <20250519114640.303640-1-c.ebner@proxmox.com>
Message-ID: <81d5cd62-9769-4be5-a86e-2c4aa44ef81c@proxmox.com>

superseded-by RFC version 2:
https://lore.proxmox.com/pbs-devel/20250529143207.694497-1-c.ebner at proxmox.com/T/



From c.ebner at proxmox.com  Thu May 29 16:32:03 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:32:03 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 38/42] tools: async lru cache:
 implement insert, remove and contains methods
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-39-c.ebner@proxmox.com>

Add methods to insert new cache entries without using the cacher,
remove cache entries given their key and check if the cache contains
a key, marking it the most recently used one if it does.

These methods will be used to implement the local datastore cache
which stores the values (chunks) on the filesystem rather than
keeping track of them by storing them in-memory in the cache. The lru
cache will only be used to allow for fast lookup and keep track of
the lookup order.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 pbs-tools/src/async_lru_cache.rs | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/pbs-tools/src/async_lru_cache.rs b/pbs-tools/src/async_lru_cache.rs
index 141114933..3a975de32 100644
--- a/pbs-tools/src/async_lru_cache.rs
+++ b/pbs-tools/src/async_lru_cache.rs
@@ -87,6 +87,29 @@ impl<K: std::cmp::Eq + std::hash::Hash + Copy, V: Clone + Send + 'static> AsyncL
 
         result
     }
+
+    /// Insert an item as the most recently used one into the cache, calling the removed callback
+    /// on the evicted cache item, if any.
+    pub fn insert<F>(&self, key: K, value: V, removed: F) -> Result<(), Error>
+    where
+        F: Fn(K) -> Result<(), Error>,
+    {
+        let mut maps = self.maps.lock().unwrap();
+        maps.0.insert(key, value.clone(), removed)?;
+        Ok(())
+    }
+
+    /// Check if the item exists and if so, mark it as the most recently uses one.
+    pub fn contains(&self, key: K) -> bool {
+        let mut maps = self.maps.lock().unwrap();
+        maps.0.get_mut(key).is_some()
+    }
+
+    /// Remove the item from the cache.
+    pub fn remove(&self, key: K) {
+        let mut maps = self.maps.lock().unwrap();
+        maps.0.remove(key);
+    }
 }
 
 #[cfg(test)]
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:32:00 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:32:00 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 35/42] ui: expose the S3 client
 view in the navigation tree
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-36-c.ebner@proxmox.com>

Add a `S3 Clients` item to the navigation tree to allow accessing the
S3 client configuration view and edit windows.

Adds the required source files to the Makefile.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/Makefile          | 2 ++
 www/NavigationTree.js | 6 ++++++
 2 files changed, 8 insertions(+)

diff --git a/www/Makefile b/www/Makefile
index 44c5fa133..ca4683941 100644
--- a/www/Makefile
+++ b/www/Makefile
@@ -61,6 +61,7 @@ JSSRC=							\
 	config/RemoteView.js				\
 	config/TrafficControlView.js			\
 	config/ACLView.js				\
+	config/S3BucketView.js				\
 	config/SyncView.js				\
 	config/VerifyView.js				\
 	config/PruneView.js				\
@@ -85,6 +86,7 @@ JSSRC=							\
 	window/PruneJobEdit.js				\
 	window/GCJobEdit.js				\
 	window/UserEdit.js				\
+	window/S3BucketEdit.js				\
 	window/Settings.js				\
 	window/TokenEdit.js				\
 	window/VerifyJobEdit.js				\
diff --git a/www/NavigationTree.js b/www/NavigationTree.js
index f10b0cd63..c79797d79 100644
--- a/www/NavigationTree.js
+++ b/www/NavigationTree.js
@@ -80,6 +80,12 @@ Ext.define('PBS.store.NavigationStore', {
 			path: 'pbsSubscription',
 			leaf: true,
 		    },
+		    {
+			text: gettext('S3 Buckets'),
+			iconCls: 'fa fa-trash',
+			path: 'pbsS3BucketView',
+			leaf: true,
+		    },
 		],
 	    },
 	    {
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:31:59 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:31:59 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 34/42] ui: add S3 client view
 for configuration
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-35-c.ebner@proxmox.com>

Adds the view to configure S3 clients in the Configuration section of
the UI.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 www/config/S3BucketView.js | 144 +++++++++++++++++++++++++++++++++++++
 1 file changed, 144 insertions(+)
 create mode 100644 www/config/S3BucketView.js

diff --git a/www/config/S3BucketView.js b/www/config/S3BucketView.js
new file mode 100644
index 000000000..85ac6c49c
--- /dev/null
+++ b/www/config/S3BucketView.js
@@ -0,0 +1,144 @@
+Ext.define('pmx-s3bucket', {
+    extend: 'Ext.data.Model',
+    fields: ['id', 'host', 'bucket', 'port', 'access-key', 'secret-key', 'region', 'fingerprint'],
+    idProperty: 'id',
+    proxy: {
+	type: 'proxmox',
+	url: '/api2/json/config/s3',
+    },
+});
+
+Ext.define('PBS.config.S3BucketView', {
+    extend: 'Ext.grid.GridPanel',
+    alias: 'widget.pbsS3BucketView',
+
+    title: gettext('S3 Buckets'),
+
+    stateful: true,
+    stateId: 'grid-s3buckets',
+    tools: [PBS.Utils.get_help_tool("backup-s3-bucket")],
+
+    controller: {
+	xclass: 'Ext.app.ViewController',
+
+	addS3Bucket: function() {
+	    let me = this;
+            Ext.create('PBS.window.S3BucketEdit', {
+		listeners: {
+		    destroy: function() {
+			me.reload();
+		    },
+		},
+            }).show();
+	},
+
+	editS3Bucket: function() {
+	    let me = this;
+	    let view = me.getView();
+	    let selection = view.getSelection();
+	    if (selection.length < 1) return;
+
+            Ext.create('PBS.window.S3BucketEdit', {
+                id: selection[0].data.id,
+		listeners: {
+		    destroy: function() {
+			me.reload();
+		    },
+		},
+            }).show();
+	},
+
+	reload: function() { this.getView().getStore().rstore.load(); },
+
+	init: function(view) {
+	    Proxmox.Utils.monStoreErrors(view, view.getStore().rstore);
+	},
+    },
+
+    listeners: {
+	activate: 'reload',
+	itemdblclick: 'editS3Bucket',
+    },
+
+    store: {
+	type: 'diff',
+	autoDestroy: true,
+	autoDestroyRstore: true,
+	sorters: 'id',
+	rstore: {
+	    type: 'update',
+	    storeid: 'pmx-s3bucket',
+	    model: 'pmx-s3bucket',
+	    autoStart: true,
+	    interval: 5000,
+	},
+    },
+
+    tbar: [
+	{
+	    xtype: 'proxmoxButton',
+	    text: gettext('Add'),
+	    handler: 'addS3Bucket',
+	    selModel: false,
+	},
+	{
+	    xtype: 'proxmoxButton',
+	    text: gettext('Edit'),
+	    handler: 'editS3Bucket',
+	    disabled: true,
+	},
+	{
+	    xtype: 'proxmoxStdRemoveButton',
+	    baseurl: '/config/s3',
+	    callback: 'reload',
+	},
+    ],
+
+    viewConfig: {
+	trackOver: false,
+    },
+
+    columns: [
+	{
+	    dataIndex: 'id',
+	    header: gettext('Unique Identifier'),
+	    renderer: Ext.String.htmlEncode,
+	    sortable: true,
+	    width: 200,
+	},
+	{
+	    dataIndex: 'bucket',
+	    header: gettext('Bucket'),
+	    renderer: Ext.String.htmlEncode,
+	    sortable: true,
+	    width: 200,
+	},
+	{
+	    dataIndex: 'host',
+	    header: gettext('Host'),
+	    sortable: true,
+	    width: 200,
+	},
+	{
+	    dataIndex: 'port',
+	    header: gettext('Port'),
+	    renderer: Ext.String.htmlEncode,
+	    sortable: true,
+	    width: 100,
+	},
+	{
+	    dataIndex: 'region',
+	    header: gettext('Region'),
+	    renderer: Ext.String.htmlEncode,
+	    sortable: true,
+	    width: 100,
+	},
+	{
+	    dataIndex: 'fingerprint',
+	    header: gettext('Fingerprint'),
+	    renderer: Ext.String.htmlEncode,
+	    sortable: false,
+	    flex: 1,
+	},
+    ],
+});
-- 
2.39.5




From c.ebner at proxmox.com  Thu May 29 16:32:06 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Thu, 29 May 2025 16:32:06 +0200
Subject: [pbs-devel] [RFC v2 proxmox-backup 41/42] api: reader: use local
 datastore cache on S3 backend chunk fetching
In-Reply-To: <20250529143207.694497-1-c.ebner@proxmox.com>
References: <20250529143207.694497-1-c.ebner@proxmox.com>
Message-ID: <20250529143207.694497-42-c.ebner@proxmox.com>

Take advantage of the local datastore filesystem cache for datastores
backed by an s3 object store in order to reduce number of requests
and latency, and increase throughput.

Also, reducing the number of requests is cost beneficial for S3 object
stores charging for fetching of objects.

Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
---
 src/api2/reader/mod.rs | 23 ++++++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/src/api2/reader/mod.rs b/src/api2/reader/mod.rs
index 3417f49be..24962a136 100644
--- a/src/api2/reader/mod.rs
+++ b/src/api2/reader/mod.rs
@@ -323,7 +323,28 @@ fn download_chunk(
 
         let body = match &env.backend {
             DatastoreBackend::Filesystem => load_from_filesystem(env, &digest)?,
-            DatastoreBackend::S3(s3_client) => fetch_from_object_store(s3_client, &digest).await?,
+            DatastoreBackend::S3(s3_client) => {
+                match env.datastore.cache() {
+                    None => fetch_from_object_store(s3_client, &digest).await?,
+                    Some(cache) => {
+                        let mut cacher = env
+                            .datastore
+                            .cacher()?
+                            .ok_or(format_err!("no cacher for datastore"))?;
+                        // Download from object store, insert to local cache store and read from
+                        // file. Can this be optimized?
+                        let chunk =
+                            cache
+                                .access(&digest, &mut cacher)
+                                .await?
+                                .ok_or(format_err!(
+                                    "unable to access chunk with digest {}",
+                                    hex::encode(digest)
+                                ))?;
+                        Body::from(chunk.raw_data().to_owned())
+                    }
+                }
+            }
         };
 
         // fixme: set other headers ?
-- 
2.39.5




From c.ebner at proxmox.com  Fri May 30 12:02:36 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 30 May 2025 12:02:36 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 4/8] api: admin: run
 configured sync jobs when a datastore is mounted
In-Reply-To: <20250515124138.55436-5-h.laimer@proxmox.com>
References: <20250515124138.55436-1-h.laimer@proxmox.com>
 <20250515124138.55436-5-h.laimer@proxmox.com>
Message-ID: <4566b2e4-1736-4a30-bd65-0a457d6efd59@proxmox.com>

please see some comments inline

On 5/15/25 14:41, Hannes Laimer wrote:
> When a datastore is mounted, spawn a new task to run all sync jobs
> marked with `run-on-mount`. These jobs run sequentially and include
> any job for which the mounted datastore is:
> 
> - The source or target in a local push/pull job
> - The source in a push job to a remote datastore
> - The target in a pull job from a remote datastore
> 
> Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
> ---
>   src/api2/admin/datastore.rs | 91 +++++++++++++++++++++++++++++++++++--
>   src/api2/admin/sync.rs      |  2 +-
>   src/server/sync.rs          |  7 +--
>   3 files changed, 91 insertions(+), 9 deletions(-)
> 
> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
> index 392494488..8463adb6a 100644
> --- a/src/api2/admin/datastore.rs
> +++ b/src/api2/admin/datastore.rs
> @@ -42,8 +42,8 @@ use pbs_api_types::{
>       DataStoreConfig, DataStoreListItem, DataStoreMountStatus, DataStoreStatus,
>       GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, KeepOptions, MaintenanceMode,
>       MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, SnapshotVerifyState,
> -    BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
> -    BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
> +    SyncJobConfig, BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, BACKUP_NAMESPACE_SCHEMA,
> +    BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
>       IGNORE_VERIFIED_BACKUPS_SCHEMA, MANIFEST_BLOB_NAME, MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
>       PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
>       PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, PRIV_SYS_MODIFY, UPID, UPID_SCHEMA,
> @@ -2510,6 +2510,51 @@ pub fn do_mount_device(datastore: DataStoreConfig) -> Result<(), Error> {
>       Ok(())
>   }
>   
> +async fn do_sync_jobs(
> +    jobs_to_run: Vec<SyncJobConfig>,
> +    worker: Arc<WorkerTask>,
> +) -> Result<(), Error> {
> +    let count = jobs_to_run.len();
> +    info!(
> +        "will run {} sync jobs: {}",
> +        count,
> +        jobs_to_run
> +            .iter()
> +            .map(|j| j.id.clone())
> +            .collect::<Vec<String>>()
> +            .join(", ")
> +    );

nit:

above can be rewritten without cloning the job ids and `count` in-lined as:
```
     info!(
         "will run {count} sync jobs: {}",
         jobs_to_run
             .iter()
             .map(|sync_job_config| sync_job_config.id.as_str())
             .collect::<Vec<&str>>()
             .join(", ")
     );
```

> +
> +    for (i, job_config) in jobs_to_run.into_iter().enumerate() {
> +        if worker.abort_requested() {
> +            bail!("aborted due to user request");
> +        }
> +        let job_id = job_config.id.clone();
> +        let Ok(job) = Job::new("syncjob", &job_id) else {

nit: this should log an error/warning and the status progress, as this 
will fail if the job lock cannot be acquired. That is something which is 
of interest for debugging.

> +            continue;
> +        };
> +        let auth_id = Authid::root_auth_id().clone();

nit: auth_id does not need to be cloned here ...

> +        info!("[{}/{count}] starting '{job_id}'...", i + 1);
> +        match crate::server::do_sync_job(
> +            job,
> +            job_config,
> +            &auth_id,

... since only passed as reference here.

> +            Some("mount".to_string()),
> +            false,
> +        ) {
> +            Ok((_upid, handle)) => {
> +                tokio::select! {
> +                    sync_done = handle.fuse() => if let Err(err) = sync_done { warn!("could not wait for job to finish: {err}"); },

question: should this be logged as error rather than a warning ...

> +                    _abort = worker.abort_future() => bail!("aborted due to user request"),
> +                };
> +            }
> +            Err(err) => warn!("unable to start sync job {job_id} - {err}"),

... same here?

> +        }
> +    }
> +
> +    Ok(())
> +}
> +
>   #[api(
>       protected: true,
>       input: {
> @@ -2541,12 +2586,48 @@ pub fn mount(store: String, rpcenv: &mut dyn RpcEnvironment) -> Result<Value, Er
>       let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
>       let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
>   
> -    let upid = WorkerTask::new_thread(
> +    let upid = WorkerTask::spawn(

question: is it okay to run this on the same thread here? 
`do_mount_device` does some blocking calls after all?

>           "mount-device",
> -        Some(store),
> +        Some(store.clone()),
>           auth_id.to_string(),
>           to_stdout,
> -        move |_worker| do_mount_device(datastore),
> +        move |_worker| async move {
> +            do_mount_device(datastore.clone())?;
> +            let Ok((sync_config, _digest)) = pbs_config::sync::config() else {
> +                warn!("unable to read sync job config, won't run any sync jobs");
> +                return Ok(());
> +            };
> +            let Ok(list) = sync_config.convert_to_typed_array("sync") else {
> +                warn!("unable to parse sync job config, won't run any sync jobs");
> +                return Ok(());
> +            };
> +            let jobs_to_run: Vec<SyncJobConfig> = list
> +                .into_iter()
> +                .filter(|job: &SyncJobConfig| {
> +                    // add job iff (running on mount is enabled and) any of these apply
> +                    //   - the jobs is local and we are source or target
> +                    //   - we are the source of a push to a remote
> +                    //   - we are the target of a pull from a remote
> +                    //
> +                    // `job.store == datastore.name` iff we are the target for pull from remote or we
> +                    // are the source for push to remote, therefore we don't have to check for the
> +                    // direction of the job.
> +                    job.run_on_mount.unwrap_or(false)
> +                        && (job.remote.is_none() && job.remote_store == datastore.name
> +                            || job.store == datastore.name)
> +                })
> +                .collect();
> +            if !jobs_to_run.is_empty() {

comment: an additional log info to the mount task log would be nice, so 
one sees from it as well that some sync jobs were triggered.

> +                let _ = WorkerTask::spawn(
> +                    "mount-sync-jobs",
> +                    Some(store),
> +                    auth_id.to_string(),
> +                    false,
> +                    move |worker| async move { do_sync_jobs(jobs_to_run, worker).await },
> +                );
> +            }
> +            Ok(())
> +        },

comment: all of above is executed within a api endpoint flagged as 
protected! This however leads to the sync job to be executed by the 
proxmox-backup-proxy, all the synced contents therefore written and 
owner by the root user instead of the backup user.

>       )?;
>   
>       Ok(json!(upid))
> diff --git a/src/api2/admin/sync.rs b/src/api2/admin/sync.rs
> index 6722ebea0..01dea5126 100644
> --- a/src/api2/admin/sync.rs
> +++ b/src/api2/admin/sync.rs
> @@ -161,7 +161,7 @@ pub fn run_sync_job(
>   
>       let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
>   
> -    let upid_str = do_sync_job(job, sync_job, &auth_id, None, to_stdout)?;
> +    let (upid_str, _) = do_sync_job(job, sync_job, &auth_id, None, to_stdout)?;
>   
>       Ok(upid_str)
>   }
> diff --git a/src/server/sync.rs b/src/server/sync.rs
> index 09814ef0c..c45a8975e 100644
> --- a/src/server/sync.rs
> +++ b/src/server/sync.rs
> @@ -12,6 +12,7 @@ use futures::{future::FutureExt, select};
>   use hyper::http::StatusCode;
>   use pbs_config::BackupLockGuard;
>   use serde_json::json;
> +use tokio::task::JoinHandle;
>   use tracing::{info, warn};
>   
>   use proxmox_human_byte::HumanByte;
> @@ -598,7 +599,7 @@ pub fn do_sync_job(
>       auth_id: &Authid,
>       schedule: Option<String>,
>       to_stdout: bool,
> -) -> Result<String, Error> {
> +) -> Result<(String, JoinHandle<()>), Error> {
>       let job_id = format!(
>           "{}:{}:{}:{}:{}",
>           sync_job.remote.as_deref().unwrap_or("-"),
> @@ -614,7 +615,7 @@ pub fn do_sync_job(
>           bail!("can't sync to same datastore");
>       }
>   
> -    let upid_str = WorkerTask::spawn(
> +    let (upid_str, handle) = WorkerTask::spawn_with_handle(
>           &worker_type,
>           Some(job_id.clone()),
>           auth_id.to_string(),
> @@ -730,7 +731,7 @@ pub fn do_sync_job(
>           },
>       )?;
>   
> -    Ok(upid_str)
> +    Ok((upid_str, handle))
>   }
>   
>   pub(super) fn ignore_not_verified_or_encrypted(




From h.laimer at proxmox.com  Fri May 30 13:45:26 2025
From: h.laimer at proxmox.com (Hannes Laimer)
Date: Fri, 30 May 2025 13:45:26 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 4/8] api: admin: run
 configured sync jobs when a datastore is mounted
In-Reply-To: <4566b2e4-1736-4a30-bd65-0a457d6efd59@proxmox.com>
References: <20250515124138.55436-1-h.laimer@proxmox.com>
 <20250515124138.55436-5-h.laimer@proxmox.com>
 <4566b2e4-1736-4a30-bd65-0a457d6efd59@proxmox.com>
Message-ID: <9b24918c-0f4a-489d-8bf8-d3e2a423749b@proxmox.com>

added some inline comments, I'll probably send a v3 addressing some of
the things you mentioned

On 5/30/25 12:02, Christian Ebner wrote:
> please see some comments inline
> 
> On 5/15/25 14:41, Hannes Laimer wrote:
>> When a datastore is mounted, spawn a new task to run all sync jobs
>> marked with `run-on-mount`. These jobs run sequentially and include
>> any job for which the mounted datastore is:
>>
>> - The source or target in a local push/pull job
>> - The source in a push job to a remote datastore
>> - The target in a pull job from a remote datastore
>>
>> Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
>> ---
>> ? src/api2/admin/datastore.rs | 91 +++++++++++++++++++++++++++++++++++--
>> ? src/api2/admin/sync.rs????? |? 2 +-
>> ? src/server/sync.rs????????? |? 7 +--
>> ? 3 files changed, 91 insertions(+), 9 deletions(-)
>>
>> diff --git a/src/api2/admin/datastore.rs b/src/api2/admin/datastore.rs
>> index 392494488..8463adb6a 100644
>> --- a/src/api2/admin/datastore.rs
>> +++ b/src/api2/admin/datastore.rs
>> @@ -42,8 +42,8 @@ use pbs_api_types::{
>> ????? DataStoreConfig, DataStoreListItem, DataStoreMountStatus, 
>> DataStoreStatus,
>> ????? GarbageCollectionJobStatus, GroupListItem, JobScheduleStatus, 
>> KeepOptions, MaintenanceMode,
>> ????? MaintenanceType, Operation, PruneJobOptions, SnapshotListItem, 
>> SnapshotVerifyState,
>> -??? BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, 
>> BACKUP_NAMESPACE_SCHEMA, BACKUP_TIME_SCHEMA,
>> -??? BACKUP_TYPE_SCHEMA, CATALOG_NAME, CLIENT_LOG_BLOB_NAME, 
>> DATASTORE_SCHEMA,
>> +??? SyncJobConfig, BACKUP_ARCHIVE_NAME_SCHEMA, BACKUP_ID_SCHEMA, 
>> BACKUP_NAMESPACE_SCHEMA,
>> +??? BACKUP_TIME_SCHEMA, BACKUP_TYPE_SCHEMA, CATALOG_NAME, 
>> CLIENT_LOG_BLOB_NAME, DATASTORE_SCHEMA,
>> ????? IGNORE_VERIFIED_BACKUPS_SCHEMA, MANIFEST_BLOB_NAME, 
>> MAX_NAMESPACE_DEPTH, NS_MAX_DEPTH_SCHEMA,
>> ????? PRIV_DATASTORE_AUDIT, PRIV_DATASTORE_BACKUP, 
>> PRIV_DATASTORE_MODIFY, PRIV_DATASTORE_PRUNE,
>> ????? PRIV_DATASTORE_READ, PRIV_DATASTORE_VERIFY, PRIV_SYS_MODIFY, 
>> UPID, UPID_SCHEMA,
>> @@ -2510,6 +2510,51 @@ pub fn do_mount_device(datastore: 
>> DataStoreConfig) -> Result<(), Error> {
>> ????? Ok(())
>> ? }
>> +async fn do_sync_jobs(
>> +??? jobs_to_run: Vec<SyncJobConfig>,
>> +??? worker: Arc<WorkerTask>,
>> +) -> Result<(), Error> {
>> +??? let count = jobs_to_run.len();
>> +??? info!(
>> +??????? "will run {} sync jobs: {}",
>> +??????? count,
>> +??????? jobs_to_run
>> +??????????? .iter()
>> +??????????? .map(|j| j.id.clone())
>> +??????????? .collect::<Vec<String>>()
>> +??????????? .join(", ")
>> +??? );
> 
> nit:
> 
> above can be rewritten without cloning the job ids and `count` in-lined as:
> ```
>  ??? info!(
>  ??????? "will run {count} sync jobs: {}",
>  ??????? jobs_to_run
>  ??????????? .iter()
>  ??????????? .map(|sync_job_config| sync_job_config.id.as_str())
>  ??????????? .collect::<Vec<&str>>()
>  ??????????? .join(", ")
>  ??? );
> ```
> 
>> +
>> +??? for (i, job_config) in jobs_to_run.into_iter().enumerate() {
>> +??????? if worker.abort_requested() {
>> +??????????? bail!("aborted due to user request");
>> +??????? }
>> +??????? let job_id = job_config.id.clone();
>> +??????? let Ok(job) = Job::new("syncjob", &job_id) else {
> 
> nit: this should log an error/warning and the status progress, as this 
> will fail if the job lock cannot be acquired. That is something which is 
> of interest for debugging.
> 
makes sense, will do

>> +??????????? continue;
>> +??????? };
>> +??????? let auth_id = Authid::root_auth_id().clone();
> 
> nit: auth_id does not need to be cloned here ...
> 
>> +??????? info!("[{}/{count}] starting '{job_id}'...", i + 1);
>> +??????? match crate::server::do_sync_job(
>> +??????????? job,
>> +??????????? job_config,
>> +??????????? &auth_id,
> 
> ... since only passed as reference here.
> 
>> +??????????? Some("mount".to_string()),
>> +??????????? false,
>> +??????? ) {
>> +??????????? Ok((_upid, handle)) => {
>> +??????????????? tokio::select! {
>> +??????????????????? sync_done = handle.fuse() => if let Err(err) = 
>> sync_done { warn!("could not wait for job to finish: {err}"); },
> 
> question: should this be logged as error rather than a warning ...
> 

hmm, maybe. But it doesn't mean anything necessarily failed, the sync
itself will continue. We'll just start the next one without waiting.

But I'm not super sure exactly when this would fail, maybe there are
some cases where an error would be warranted. But with my current
understanding it is not really.

>> +??????????????????? _abort = worker.abort_future() => bail!("aborted 
>> due to user request"),
>> +??????????????? };
>> +??????????? }
>> +??????????? Err(err) => warn!("unable to start sync job {job_id} - 
>> {err}"),
> 
> ... same here?
> 

in my head error means task could not continue, which is not the case
here, we'll just skip this jobs. But no hard feelings, could also make
an error here.

>> +??????? }
>> +??? }
>> +
>> +??? Ok(())
>> +}
>> +
>> ? #[api(
>> ????? protected: true,
>> ????? input: {
>> @@ -2541,12 +2586,48 @@ pub fn mount(store: String, rpcenv: &mut dyn 
>> RpcEnvironment) -> Result<Value, Er
>> ????? let auth_id: Authid = rpcenv.get_auth_id().unwrap().parse()?;
>> ????? let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
>> -??? let upid = WorkerTask::new_thread(
>> +??? let upid = WorkerTask::spawn(
> 
> question: is it okay to run this on the same thread here? 
> `do_mount_device` does some blocking calls after all?
> 

yes, there's no real reason to look for job configs unless mounting is
successful and done. So the blocking is not a problem, actually quite
the opposite.

>> ????????? "mount-device",
>> -??????? Some(store),
>> +??????? Some(store.clone()),
>> ????????? auth_id.to_string(),
>> ????????? to_stdout,
>> -??????? move |_worker| do_mount_device(datastore),
>> +??????? move |_worker| async move {
>> +??????????? do_mount_device(datastore.clone())?;
>> +??????????? let Ok((sync_config, _digest)) = 
>> pbs_config::sync::config() else {
>> +??????????????? warn!("unable to read sync job config, won't run any 
>> sync jobs");
>> +??????????????? return Ok(());
>> +??????????? };
>> +??????????? let Ok(list) = sync_config.convert_to_typed_array("sync") 
>> else {
>> +??????????????? warn!("unable to parse sync job config, won't run any 
>> sync jobs");
>> +??????????????? return Ok(());
>> +??????????? };
>> +??????????? let jobs_to_run: Vec<SyncJobConfig> = list
>> +??????????????? .into_iter()
>> +??????????????? .filter(|job: &SyncJobConfig| {
>> +??????????????????? // add job iff (running on mount is enabled and) 
>> any of these apply
>> +??????????????????? //?? - the jobs is local and we are source or target
>> +??????????????????? //?? - we are the source of a push to a remote
>> +??????????????????? //?? - we are the target of a pull from a remote
>> +??????????????????? //
>> +??????????????????? // `job.store == datastore.name` iff we are the 
>> target for pull from remote or we
>> +??????????????????? // are the source for push to remote, therefore 
>> we don't have to check for the
>> +??????????????????? // direction of the job.
>> +??????????????????? job.run_on_mount.unwrap_or(false)
>> +??????????????????????? && (job.remote.is_none() && job.remote_store 
>> == datastore.name
>> +??????????????????????????? || job.store == datastore.name)
>> +??????????????? })
>> +??????????????? .collect();
>> +??????????? if !jobs_to_run.is_empty() {
> 
> comment: an additional log info to the mount task log would be nice, so 
> one sees from it as well that some sync jobs were triggered.
> 

sure

>> +??????????????? let _ = WorkerTask::spawn(
>> +??????????????????? "mount-sync-jobs",
>> +??????????????????? Some(store),
>> +??????????????????? auth_id.to_string(),
>> +??????????????????? false,
>> +??????????????????? move |worker| async move 
>> { do_sync_jobs(jobs_to_run, worker).await },
>> +??????????????? );
>> +??????????? }
>> +??????????? Ok(())
>> +??????? },
> 
> comment: all of above is executed within a api endpoint flagged as 
> protected! This however leads to the sync job to be executed by the 
> proxmox-backup-proxy, all the synced contents therefore written and 
> owner by the root user instead of the backup user.
> 

actually true, good catch! I didn't even check that, I just assumed we'd 
explicitly set the owner whenever we create dirs/files during sync. We 
do this for garbage collection and in basically all other places IIRC.

So, I think we should also do that for syncs, can you think of a reason
to not do that? If not, I'll prepare a patch for that.

>> ????? )?;
>> ????? Ok(json!(upid))
>> diff --git a/src/api2/admin/sync.rs b/src/api2/admin/sync.rs
>> index 6722ebea0..01dea5126 100644
>> --- a/src/api2/admin/sync.rs
>> +++ b/src/api2/admin/sync.rs
>> @@ -161,7 +161,7 @@ pub fn run_sync_job(
>> ????? let to_stdout = rpcenv.env_type() == RpcEnvironmentType::CLI;
>> -??? let upid_str = do_sync_job(job, sync_job, &auth_id, None, 
>> to_stdout)?;
>> +??? let (upid_str, _) = do_sync_job(job, sync_job, &auth_id, None, 
>> to_stdout)?;
>> ????? Ok(upid_str)
>> ? }
>> diff --git a/src/server/sync.rs b/src/server/sync.rs
>> index 09814ef0c..c45a8975e 100644
>> --- a/src/server/sync.rs
>> +++ b/src/server/sync.rs
>> @@ -12,6 +12,7 @@ use futures::{future::FutureExt, select};
>> ? use hyper::http::StatusCode;
>> ? use pbs_config::BackupLockGuard;
>> ? use serde_json::json;
>> +use tokio::task::JoinHandle;
>> ? use tracing::{info, warn};
>> ? use proxmox_human_byte::HumanByte;
>> @@ -598,7 +599,7 @@ pub fn do_sync_job(
>> ????? auth_id: &Authid,
>> ????? schedule: Option<String>,
>> ????? to_stdout: bool,
>> -) -> Result<String, Error> {
>> +) -> Result<(String, JoinHandle<()>), Error> {
>> ????? let job_id = format!(
>> ????????? "{}:{}:{}:{}:{}",
>> ????????? sync_job.remote.as_deref().unwrap_or("-"),
>> @@ -614,7 +615,7 @@ pub fn do_sync_job(
>> ????????? bail!("can't sync to same datastore");
>> ????? }
>> -??? let upid_str = WorkerTask::spawn(
>> +??? let (upid_str, handle) = WorkerTask::spawn_with_handle(
>> ????????? &worker_type,
>> ????????? Some(job_id.clone()),
>> ????????? auth_id.to_string(),
>> @@ -730,7 +731,7 @@ pub fn do_sync_job(
>> ????????? },
>> ????? )?;
>> -??? Ok(upid_str)
>> +??? Ok((upid_str, handle))
>> ? }
>> ? pub(super) fn ignore_not_verified_or_encrypted(
> 




From c.ebner at proxmox.com  Fri May 30 15:08:43 2025
From: c.ebner at proxmox.com (Christian Ebner)
Date: Fri, 30 May 2025 15:08:43 +0200
Subject: [pbs-devel] [PATCH proxmox-backup v2 4/8] api: admin: run
 configured sync jobs when a datastore is mounted
In-Reply-To: <9b24918c-0f4a-489d-8bf8-d3e2a423749b@proxmox.com>
References: <20250515124138.55436-1-h.laimer@proxmox.com>
 <20250515124138.55436-5-h.laimer@proxmox.com>
 <4566b2e4-1736-4a30-bd65-0a457d6efd59@proxmox.com>
 <9b24918c-0f4a-489d-8bf8-d3e2a423749b@proxmox.com>
Message-ID: <f32ecd97-977d-46da-9ee9-47db5bd69853@proxmox.com>

On 5/30/25 13:45, Hannes Laimer wrote:
> added some inline comments, I'll probably send a v3 addressing some of
> the things you mentioned
> 
> On 5/30/25 12:02, Christian Ebner wrote:
>> please see some comments inline
>>
>> On 5/15/25 14:41, Hannes Laimer wrote:
>>> When a datastore is mounted, spawn a new task to run all sync jobs
>>> marked with `run-on-mount`. These jobs run sequentially and include
>>> any job for which the mounted datastore is:
>>>
>>> - The source or target in a local push/pull job
>>> - The source in a push job to a remote datastore
>>> - The target in a pull job from a remote datastore
>>>
>>> Signed-off-by: Hannes Laimer <h.laimer at proxmox.com>
>>> +??????????????? let _ = WorkerTask::spawn(
>>> +??????????????????? "mount-sync-jobs",
>>> +??????????????????? Some(store),
>>> +??????????????????? auth_id.to_string(),
>>> +??????????????????? false,
>>> +??????????????????? move |worker| async move 
>>> { do_sync_jobs(jobs_to_run, worker).await },
>>> +??????????????? );
>>> +??????????? }
>>> +??????????? Ok(())
>>> +??????? },
>>
>> comment: all of above is executed within a api endpoint flagged as 
>> protected! This however leads to the sync job to be executed by the 
>> proxmox-backup-proxy, all the synced contents therefore written and 
>> owner by the root user instead of the backup user.
>>
> 
> actually true, good catch! I didn't even check that, I just assumed we'd 
> explicitly set the owner whenever we create dirs/files during sync. We 
> do this for garbage collection and in basically all other places IIRC.
> 
> So, I think we should also do that for syncs, can you think of a reason
> to not do that? If not, I'll prepare a patch for that.

As discussed off-list a bit, IMHO running the whole sync job code paths 
in the privileged api server is not ideal and would weaken the privilege 
separation. But I do agree that this is already been done in some places 
and would be probably easy to approach.

An option would be to add an (unprivileged) api endpoint which allows to 
pass a list of sync jobs to be executed sequentially, e.g. `POST 
/api2/json/admin/sync/run-jobs`.

Given that we found no better approach other then the two proposed ones,
maybe somebody else has an opinion/suggestion on this?