PVE 6.1 incorrect MTU

[Stefan M. Radman]

Recently I upgraded the first node of a 5.4 cluster to 6.1.
Everything went smooth (thanks for pve5to6!) until I restarted the first node after the upgrade and started to get weird problems with the LVM/multipath/iSCSI based storage (hung PVE and LVM processes etc).

After a while of digging I found that the issues were due to an incorrect MTU on the storage interface (Jumbo frames got truncated).

The network configuration (see /etc/network/interfaces further below) worked very well with 5.4 but it does not work with 6.1.

After booting with PVE 6.1, some of the MTUs are not as configured.
Here is what "ip link" shows after boot.

root at seisram04:~# ip link | fgrep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
8: vmbr533: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
9: bond0.533 at bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr533 state UP mode DEFAULT group default qlen 1000
10: vmbr683: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
11: bond0.683 at bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr683 state UP mode DEFAULT group default qlen 1000
12: vmbr686: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
13: bond0.686 at bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr686 state UP mode DEFAULT group default qlen 1000

The only interface that is correctly configured for Jumbo frames is  vmbr686, the bridge serving the storage VLAN but the underlying bond0 interface and it's slaves eno1 and eno2 have the default MTU of 1500 and Jumbo frames get truncated on the way.

Why did this configuration work on 5.4 but not on 6.1?
In the interface configuration, all MTU values are explicitly configured.

Why are some of them ignored in 6.1? What am I missing here?

How can I make this configuration work in PVE 6.1 (and later)?

Any suggestions welcome.

Thank you

Stefan

/etc/network/interfaces:

auto lo
iface lo inet loopback

iface eno1 inet manual
mtu 9000
#Gb1 - Trunk - Jumbo Frames

iface eno2 inet manual
mtu 9000
#Gb2 - Trunk - Jumbo Frames

auto eno3
iface eno3 inet static
address 192.168.84.4
netmask 255.255.255.0
mtu 1500
#Gb3 - COROSYNC1 - VLAN684

auto eno4
iface eno4 inet static
address 192.168.85.4
netmask 255.255.255.0
mtu 1500
#Gb4 - COROSYNC2 - VLAN685

auto bond0
iface bond0 inet manual
slaves eno1 eno2
bond_miimon 100
bond_mode active-backup
mtu 9000
#HA Bundle Gb1/Gb2 - Trunk - Jumbo Frames

auto vmbr0
iface vmbr0 inet static
address  172.21.54.204
netmask  255.255.255.0
gateway  172.21.54.254
bridge_ports bond0
bridge_stp off
bridge_fd 0
mtu 1500
#PRIVATE - VLAN 682 - Native

auto vmbr533
iface vmbr533 inet manual
bridge_ports bond0.533
bridge_stp off
bridge_fd 0
mtu 1500
#PUBLIC - VLAN 533

auto vmbr683
iface vmbr683 inet manual
        bridge_ports bond0.683
        bridge_stp off
        bridge_fd 0
        mtu 1500
#VPN - VLAN 683

auto vmbr686
iface vmbr686 inet static
address  192.168.86.204
netmask  255.255.255.0
bridge_ports bond0.686
bridge_stp off
bridge_fd 0
mtu 9000
#STORAGE - VLAN 686 - Jumbo Frames



CONFIDENTIALITY NOTICE: This communication may contain privileged and confidential information, or may otherwise be protected from disclosure, and is intended solely for use of the intended recipient(s). If you are not the intended recipient of this communication, please notify the sender that you have received this communication in error and delete and destroy all copies in your possession.