[PVE-User] Shared storage recommendations

[Ronny Aasen]

On 27.02.2019 16:49, Frederic Van Espen wrote:
> Hi,
>
>> tips: you want 5 monitors, so you can have dual failures or one fail
>> while another is under maintenance/reboot;  Good networking is
>> important, 25gbps is better then 40gbps due to lower latency;
> Thanks for these tips! Do you have any reference documentation for the
> 25gbps vs 40gbps statement? I find no immediate results for that on
> google. In any case, we were thinking of starting out with a 10gbps
> network. Are there any solid reasons not to do that for a light
> workload?

Not sure what documentation exist. 40gbps links are internally 4x 10gbps 
waves so a single io will flow on one of the 10gbps links at 10gbps 
latency.  a 25 Gbps link is a faster single wave, and 100Gbps is 
4x25Gbps waves.  since IOPS is usually much more important then 
bandwidth (for VM's) you may get more iops with 25Gbps then with 
40Gbps   basically it depends on the workload, if the load is mostly 
bandwidth you might need the thru put of 40Gbps more.

quote from
https://ceph.com/planet/making-ceph-faster-lessons-from-performance-testing/
"20 HDDs or 2-3 SSDs can exceed the bandwidth of single 10GbE link for 
read throughput. "
so if your nodes are on this size, a single 10gbps will be the 
bottleneck. but if the load is below the bottleneck it is not a problem.

even if you design network with 2x10gbps links for HA you may need  need 
to run the cluster on a single 10gbps in case of a failure on part of 
the network. workload and SLA's you provide to your clients determine if 
that is acceptable or not.  monitoring is important


good luck!

Ronny Aasen