[pbs-devel] [PATCH proxmox-backup v9 18/46] datastore: create namespace marker in s3 backend

[Hannes Laimer]

On Sat Jul 19, 2025 at 2:50 PM CEST, Christian Ebner wrote:
> The S3 object store only allows to store objects, referenced by their
> key. For backup namespaces datastores however use directories, so
> they cannot be represented as one to one mapping.
>
> Instead, create an empty marker file for each namespace and operate
> based on that.
>
> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
> ---
> changes since version 8:
> - use upload_on_replace_with_retry
>
>  pbs-datastore/src/datastore.rs | 10 ++++++++++
>  1 file changed, 10 insertions(+)
>
> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
> index bc829c5b8..5bb4e1777 100644
> --- a/pbs-datastore/src/datastore.rs
> +++ b/pbs-datastore/src/datastore.rs
> @@ -44,6 +44,7 @@ static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
>      LazyLock::new(|| Mutex::new(HashMap::new()));
>  
>  const GROUP_NOTES_FILE_NAME: &str = "notes";
> +const NAMESPACE_MARKER_FILENAME: &str = ".namespace";
>  
>  /// checks if auth_id is owner, or, if owner is a token, if
>  /// auth_id is the user of the token
> @@ -613,6 +614,15 @@ impl DataStore {
>          // construct ns before mkdir to enforce max-depth and name validity
>          let ns = BackupNamespace::from_parent_ns(parent, name)?;
>  
> +        if let DatastoreBackend::S3(s3_client) = self.backend()? {
> +            let object_key = crate::s3::object_key_from_path(&ns.path(), NAMESPACE_MARKER_FILENAME)
> +                .context("invalid namespace marker object key")?;


hmmm, I didn't test this, but with our NS max depth of 7 (IIRC) and the
linux max filename length of ~250 this "could" be larger than the 1024
bytes an object key can be. Maybe we should check for that, even though
I doubt this would ever be a problem...

> +            let _is_duplicate = proxmox_async::runtime::block_on(
> +                s3_client.upload_no_replace_with_retry(object_key, hyper::body::Bytes::from("")),
> +            )
> +            .context("failed to create namespace on s3 backend")?;
> +        }
> +
>          let mut ns_full_path = self.base_path();
>          ns_full_path.push(ns.path());
>