[PVE-User] MTU size changed on a running cluster

[Marco M. Gabriel]

Hi there,

on a productive 5 node Proxmox VE Ceph cluster, we experienced some strange
behaviour:

Based on
http://pve.proxmox.com/wiki/Open_vSwitch#Example_2:_Bond_.2B_Bridge_.2B_Internal_Ports
we
have an internal network for cluster/corosync communication and another
internal network for Ceph Storage traffic. The Ceph OVS bridge was set to
MTU 9000 in /etc/network/interfaces and ran without a problem since a week.

Today we've seen Ceph errors like "x requests are blocked > 32 sec".

After a troubleshooting, we's seen that packets got dropped because they
were > 1500 bytes on the Ceph interface. That was strange as we had set
them to MTU 9000 and it was running since a week.

We checked the Interfaces and on two nodes, we saw a MTU of 1500 while the
other three nodes still had MTU 9000.

Has anybody experiences something like that? I read that an OVS bridge
automatically sets it's own MTU according to the lowest MTU of the member
interfaces, but I am not sure if this could be a problem here.

Any hints appreciated,
Marco