[PVE-User] replication bandwith limit not respected

[Stefan Radman]

Hi Tonči

I am not familiar with PVE storage replication but with networks.
Your answers to the questions below might help everyone on this list (including me) to better understand the issues behind the problem you are facing.

> lowered replication speed down to 80MB/s

Have you tried much lower rates (e.g. 10MB/s)?
Did anything change in the bandwidth utilization?

> replcation still uses whole NIC bandwith (1G)


How do you know? 
How do know that storage replication cloggs your network and not some other apllication?
Where do you measure the replication traffic and how?

Are you running Jumbo frames (e.g. MTU 9000) between the replication targets or standard MTU 1500?

If you are not using a dedicated NIC for storage replication you are likely (and should be) running on standard MTU. 
Using the standard MTU will generate more packets than jumbo frames, thus more headers, thus more overhead.
A bandwidth limit of 80MB/s translates to rougly 640Mbit/s but I assume that this does not include the overhead of protocols used by PVE storage replication (Ethernet, TCP/IP, SSH).

How much overhead in % do you expect?

My conservative assumption (rule of thumb) for SSH with standard MTU is 20% overhead. 
That would be 768Mbit/s, leaving only 232Mbit/s for other traffic.

What is your output of “pveversion”?

Regards

Stefan

> On Sep 6, 2024, at 16:00, Tonči Stipičević <tonci at suma-informatika.hr> wrote:
> 
> Hello,
> 
> I'm running latest cluster (community support)
> 
> And today I've lowered replication speed down to 80MB/s  but replcation still uses whole NIC bandwith (1G)  ... after few host restarts  still the same
> 
> 2024-09-06 14:37:31 using a bandwidth limit of 80000000 bytes per second for transferring 'data2:subvol-1007-disk-0'
> 
> Does somebody else experience that too ?
> 
> 
> Thank you in advance
> 
> BR
> 
> Tonči
> 
> -- 
> srdačan pozdrav / best regards
> 
> Tonči Stipičević, dipl. ing. elektr.
> direktor / manager
> 
> SUMA Informatika d.o.o., Badalićeva 27, OIB 93926415263
> 
> Podrška / Upravljanje IT sustavima za male i srednje tvrtke
> Small & Medium Business IT Support / Management
> 
> mob: 091 1234003
> www.suma-informatika.hr
> 
> 
> _______________________________________________
> pve-user mailing list
> pve-user at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user