[pve-devel] [PATCH access-control/cluster/docs/gui-tests/manager/network/proxmox{, -ve-rs, -perl-rs} v2 00/57] Add SDN Fabrics

[Friedrich Weber]

On 04/04/2025 18:28, Gabriel Goller wrote:
> This series allows the user to add fabrics such as OpenFabric and OSPF over
> their clusters.
> 
> This series relies on: 
> https://lore.proxmox.com/pve-devel/20250404135522.2603272-1-s.hanreich@proxmox.com/T/#mf4cf46c066d856cea819ac3e79d115a290f47466

Thanks for the v2, I like this feature a lot!

Unfortunately, one problem I noticed while testing this is that it may
break pre-existing FRR configs (such as full-mesh Ceph clusters set up
according to [1]) when making seemingly unrelated SDN changes. I already
quickly discussed this with Stefan, posting here in case others have
input as well.

Steps to reproduce:

- on PVE 8.3 (without these patches), set up Ceph full mesh with
OpenFabric as described in [1], includes custom /etc/frr/frr.conf
- also use some SDN feature, e.g. a VLAN zone with a Vnet
- install patched packages, systemctl restart pveproxy pvedaemon
- make a fabric-unrelated change in the SDN config, e.g. change tag of
the VLAN zone Vnet
- apply SDN config

=>
SDN stack writes out a nearly-empty /etc/frr/frr.conf on all nodes and
thus takes down the full mesh:

# cat /etc/frr/frr.conf
frr version 10.2.1
frr defaults datacenter
hostname fabric159
log syslog informational
service integrated-vtysh-config
!
!
line vty

It seems to also disable the fabricd daemon in /etc/frr/daemons:

# grep fabric /etc/frr/daemons
fabricd=no
fabricd_options="-A 127.0.0.1 --dummy_as_loopback"
# vtysh -c 'show openfabric route'
fabricd is not running

It makes sense that one cannot use both our fabrics integration and
custom FRR configs, but the above SDN config change is not related to
fabrics, so we should probably avoid touching the frr.conf if possible.
The wiki article [1] does warn that the full mesh doesn't work in
combination with EVPN, but unfortunately doesn't mention an inherent
incompatibility with the SDN stack as a whole.

[1]
https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Routed_Setup_(with_Fallback)