[pbs-devel] [PATCH proxmox-backup v9 18/46] datastore: create namespace marker in s3 backend

[Christian Ebner]

On 7/21/25 3:52 PM, Hannes Laimer wrote:
> On Sat Jul 19, 2025 at 2:50 PM CEST, Christian Ebner wrote:
>> The S3 object store only allows to store objects, referenced by their
>> key. For backup namespaces datastores however use directories, so
>> they cannot be represented as one to one mapping.
>>
>> Instead, create an empty marker file for each namespace and operate
>> based on that.
>>
>> Signed-off-by: Christian Ebner <c.ebner at proxmox.com>
>> ---
>> changes since version 8:
>> - use upload_on_replace_with_retry
>>
>>   pbs-datastore/src/datastore.rs | 10 ++++++++++
>>   1 file changed, 10 insertions(+)
>>
>> diff --git a/pbs-datastore/src/datastore.rs b/pbs-datastore/src/datastore.rs
>> index bc829c5b8..5bb4e1777 100644
>> --- a/pbs-datastore/src/datastore.rs
>> +++ b/pbs-datastore/src/datastore.rs
>> @@ -44,6 +44,7 @@ static DATASTORE_MAP: LazyLock<Mutex<HashMap<String, Arc<DataStoreImpl>>>> =
>>       LazyLock::new(|| Mutex::new(HashMap::new()));
>>   
>>   const GROUP_NOTES_FILE_NAME: &str = "notes";
>> +const NAMESPACE_MARKER_FILENAME: &str = ".namespace";
>>   
>>   /// checks if auth_id is owner, or, if owner is a token, if
>>   /// auth_id is the user of the token
>> @@ -613,6 +614,15 @@ impl DataStore {
>>           // construct ns before mkdir to enforce max-depth and name validity
>>           let ns = BackupNamespace::from_parent_ns(parent, name)?;
>>   
>> +        if let DatastoreBackend::S3(s3_client) = self.backend()? {
>> +            let object_key = crate::s3::object_key_from_path(&ns.path(), NAMESPACE_MARKER_FILENAME)
>> +                .context("invalid namespace marker object key")?;
> 
> 
> hmmm, I didn't test this, but with our NS max depth of 7 (IIRC) and the
> linux max filename length of ~250 this "could" be larger than the 1024
> bytes an object key can be. Maybe we should check for that, even though
> I doubt this would ever be a problem..

Okay, can add a check to cover that as well.