[pbs-devel] [PATCH] tape: forbid operations on a s3 datastore

[Dominik Csapak]

Hi

On 7/23/25 21:51, Thomas Lamprecht wrote:
> This was fine and I applied it but still have some comments on one
> of the arguments that would indicate that this never makes sense to
> have, as IMO it can.
> 
> Am 23.07.25 um 16:31 schrieb Dominik Csapak:
>> namely:
>> * backup to tape from s3 (including a configuring such a job)
>> * restore to s3 from tape
>>
>> It does not work currently, but it probably does not make sense to allow
>> that at all for several reasons:
>> * both are designed to be 'off-site', so copying data from one off-site
>>    location to another directly does not make sense most of the time
> 
> S3 can be off-site, but it can be also the single main storage for
> huge amount of data in a environment. W.g., if one's applications
> and use cases all support S3 and one has a performant local storage
> providing that, why not use that also for the "hot" backup store, one
> can backup directly onto a S3-backed datastore after all, and then
> sync to tape for relatively cheap off-site cold storage.
> 
> And not all countries have slow internet, 5 to 25 Gbps connections are a
> thing (e.g. provided by the swiss ISP Fiber7), as are Dark Fiber connections
> between sites, which means that even if the S3 storage is indeed offsite,
> it might be faster to pull data from there than the tape can be written too.

true>
> 
>> * (modern) tape operations can reach relatively high speeds (> 300MB/s)
>>    and up/downloading to an (most likely remote) s3 storage will slow
>>    down the tape
> 
> Does it really? I'd expect that local S3 storage backed by good HW should
> be able to get at least to 300 MB/s.
> 
> And even if, what's better, not being able to make a tape backup because
> one e.g. has not an intermediate storage with multiple TB of space available
> on the PBS directly, or having to wait a bit longer?
> 
> Again, totally justified stop-gap for now, but I would not write this
> off as not making sense in general, as while for some setups your
> points will be true, they doesn't necessarily have to be. So if there
> is actual user demand, which is a different topic, I'd see it worthwhile
> into looking more closely in what it would take to somewhat sanely support
> such an use case.

yeah, i agree with you. I probably was too eager to find a rationale to
not allow this at all, since I feared that making tape -> s3 backups
(or vice versa) will make problems for too many people (that don't
have high bandwidth uplinks, etc.)