[pbs-devel] [PATCH proxmox-backup] docs: add note for not using remote storages

[Thomas Lamprecht]

Am 12/06/2024 um 08:39 schrieb Dominik Csapak:
> 
> On 6/11/24 8:05 PM, Thomas Lamprecht wrote:
>> This section is a quite central and important one, so I'm being a bit
>> more nitpicking with it than other content. NFS boxes are still quite
>> popular, a blanket recommendation against them quite probably won't
>> help our cause or reducing noise in our getting help channels.
>>
>> Dietmar already applied this, so would need a follow-up please.
> 
> sure
> 
>>
>> Am 11/06/2024 um 11:30 schrieb Dominik Csapak:
>>> such as NFS or SMB. They will not provide the expected performance
>>> and it's better to recommend against them.
>>
>> Not so sure about doing recommending against them as a blanket statement,
>> the "remote" part might adjective is a bit subtle and, e.g., using a local
>> full flash NVMe storage attached over a 100G link with latency in the µs
>> surely beats basically any local spinner only storage and probably even
>> a lot of SATA attached SSD ones.
> 
> well alone the fact of using nfs makes some operations a few magnitudes 
> slower. e.g. here locally creating a datastore locally takes a few 
> seconds (probably fast due to the page cache) but a locally
> mounted nfs (so no network involved) on the same disk takes
> a few minutes. so at least some file creation/deletion operations
> are some magnitudes slower just by using nfs (though i guess
> there are some options/ipmlementations that can influence that
> such as async/sync export options)
>
> also a remote SMB share from windows (same physical host though, so
> again, no real network) takes ~ a minute for the same operation
> 
> so yes, while I generally agree that using remote storage can be fast 
> enough, using any of them increases some file operations by a 
> significant amount, even when using fast storage and fast network

Just because there is some overhead (that is the result of a trade-off
to get a parallel/simultaneous accessible FS) doesn't mean that we
should recommend against an FS, which is IMO a bit strange to do
in a system requirement recommendation list anyway (there's a huge
list of things that'd need to get added then here, from not using
USB 1.0 pen drives as backing storage to not sliding strong magnets
over the server).

> 
> (i know that datastore creation is not the best benchmark for this,
> but shows that there is significant overhead on some operations)

Yeah, one creates a datastore only once, and on actual backup there
are at max a few mkdirs, not 65k, so not really relevant here.
Also, just because there's some overhead, allowing simultaneous mounts
doesn't come for free, it doesn't mean that it's actually a problem for
actual backup. As said, a blanket recommendation against a setup that
is already rather frequent is IMO just deterring (future) users.


>>
>> Also, it can be totally fine to use as second datastore, i.e. in a setup
>> with a (smaller) datastore backed by (e.g. local) fast storage that is
>> then periodically synced to a slower remote.
>>
>>> Signed-off-by: Dominik Csapak <d.csapak at proxmox.com>
>>> ---
>>> if we want to discourage users even more, we could also detect it on
>>> datastore creation and put a warning into the task log
>>
>> I would avoid that, at least not without actually measuring how the
>> storage performs (which is probably quite prone to errors, or would
>> require periodic measurements).
> 
> fine with me
> 
>>
>>>
>>> also if we ever come around to implementing the 'health' page thomas
>>> wished for, we can put a warning/error there too
>>>
>>>   docs/system-requirements.rst | 3 +++
>>>   1 file changed, 3 insertions(+)
>>>
>>> diff --git a/docs/system-requirements.rst b/docs/system-requirements.rst
>>> index fb920865..17756b7b 100644
>>> --- a/docs/system-requirements.rst
>>> +++ b/docs/system-requirements.rst
>>> @@ -41,6 +41,9 @@ Recommended Server System Requirements
>>>     * Use only SSDs, for best results
>>>     * If HDDs are used: Using a metadata cache is highly recommended, for example,
>>>       add a ZFS :ref:`special device mirror <local_zfs_special_device>`.
>>> +  * While it's technically possible to use remote storages such as NFS or SMB,
>>
>> Up-front, I wrote some possible smaller improvements upfront but then
>> a replacement (see below), but I kept the others
>>
>> Would do s/remote storages/remote storage/
>>
>> (We use "storages" quite a few times already, but if possible keeping it
>> singular sounds nicer IMO)
> 
> ok
> 
>>
>>> +    the additional latency and overhead drastically reduces performance and it's
>>
>> s/additional latency and overhead/additional latency overhead/ ?
>>
>> or "network overhead"
>>
>> If it'd stay as is, the "reduces" should be changed to "reduce" ("latency and
>> overhead" is plural).
>>
> 
> i meant actually two things here, the network latency and the additional
> overhead of the second filesystem layer

Then it'd have helped me if to avoid mixing a specific overhead (latency) with
a generic mentioning of the word overhead, like:

"... the added overhead of networking and providing concurrent file system access
drastically reduces performance ..."

But that sounds a bit convoluted, so the best option here might be to just
use "added overhead".


>>
>> But I'd rather reword the whole thing to focus more on what the actual issue is,
>> i.e., not NFS or SMB/CIFS per se, but if the network accessing them is slow.
>> Maybe something like:
>>
>> * Avoid using remote storage, like NFS or SMB/CIFS, connected over a slow
>>    (< 10 Gbps) and/or high latency (> 1 ms) link. Such a storage can
>>    dramatically reduce performance and may even negatively impact the
>>    backup source, e.g. by causing IO hangs.
>>
>> I pulled the numbers in parentheses out of thin air, but IMO they shouldn't be too far
>> off from 2024 Slow™, no hard feelings on adapting them though.
> 
> IMHO i'd not mention any specific numbers at all, unless we actually
> benchmarked such a setup. so what about:

Not sure what numbers from a benchmark would be of use here? One knows what
fast storage can do latency wise and how much bandwidth is a good baseline
– granted, the numbers are not helping for every specific setup, but doing
some benchmark won't change that either.
Anyway, won't matter, see below.

> 
> * Avoid using remote storage, like NFS or SMB/CIFS, connected over a 
> slow and/or high latency link. Such a storage can dramatically reduce 
> performance and may even negatively impact the backup source, e.g. by
> causing IO hangs. If you want to use such a storage, make sure it
> performs as expected by testing it before using it in production.
> 

That starts to get rather convoluted, tbh., the more I think about this,
the more I prefer just reverting the whole thing, I see no gain in
"bashing" NFS/SMB just because they have some overhead.

If, we could simply adapt the "Use only SSDs, for best results" point to:

"Prefer fast local storage that delivers high IOPS for random IO workloads; use only enterprise SSDs for best results."

Would be a better fit to convey that fast local storage should be preferred,
especially in a "recommended" (not "recommended against") list.


> 
> By adding that additional sentence we hopefully nudge some users
> into actually testing before deploying it, instead of then
> complaining that it's slow.

If only; from forum and office request it's quite sensible to assume
that a good amount of users already have their storage box, and they'd
need to do so to be able to test it in any way, so already too late.

It might be better to describe a setup how to still be able to use their
existing, NFS/SMB/... attached storage in the best way possible. E.g., by
doing a fast small local storage for incoming backups and use the bigger
remote storage only through syncing to it. This has a few benefits beside
getting good performance with existing, slower storage (of any type), like
having already an extra copy of most recent data.