[pve-devel] [PATCH cluster/docs/manager/network/proxmox{, -ve-rs, -firewall, -perl-rs} 00/52] Add SDN Fabrics
Gabriel Goller
g.goller at proxmox.com
Fri Mar 28 18:12:48 CET 2025
This series allows the user to add fabrics such as OpenFabric and OSPF over
their clusters.
Overview
========
This series allows the user to create routed networks ('fabrics') across their
clusters, which can be used as the underlay network for a EVPN cluster, or for
creating Ceph full mesh clusters easily.
This patch series adds the initial support for two routing protocols:
* OpenFabric
* OSPF
In the future we plan on moving the existing IS-IS and BGP controllers into the
fabric structure. There are also plans for adding a new Wireguard fabric to
this.
Implementation
==============
Every fabric consists of one or more nodes, which themselves consists of one or
more interfaces. Fabrics and nodes are modeled as different section config
types, interfaces are an array contained in a node section. We have a separate
configuration file for each fabric type. This is because the basic structure
(fabric, node, interface) is the same, but the specific options vary wildly.
This makes serialization / validation from the Rust side a lot easier.
For now, nodes in the fabric configuration are always PVE nodes, but in the
future nodes could also represent external members of the fabric (e.g. in a
potential wireguard fabric).
Settings can be configured on a fabric-level, so they apply to all interfaces,
or overridden on a interface-level (hidden in the UI by default).
Most of the functionality is implemented by rust and exposed to the existing SDN
module via perlmod. This includes configuration reading / writing, FRR config
generation from the section config and API CRUD methods.
The API provides one common GET method, to get the configuration of all
different fabric types (for the tree overview), but otherwise there are separate
CRUD endpoints for every fabric type, to mimic the split of configuration files.
Another upside of this is, that the generated rust structs for the API endpoints
(for PDM) will be much nicer.
For the FRR-specific functionality we introduced a new proxmox-frr crate that
models the different entities in the FRR configuration format (routers,
interfaces, route-maps, ...) and provides serializers for those structs. For
more information see the respective FRR commits. When applying the SDN
configuration, perl calls into perlmod to utilize the proxmox-frr crate for
generating the FRR configuration of the fabrics.
We also introduce a proxmox-network-types crate, where we extracted generic
network types (Cidr for instance), so we can reuse them across multiple crates
(proxmox-frr, proxmox-firewall, proxmox-ve-config, ..).
The hierarchical nature of the configuration and the relationship between nodes
inside the fabrics requires validation of sections relative to other sections.
For this matter we introduced an intermediate configuration in the initial RFC,
but that turned out to be unwieldy (lots of additional code & conversions).
Because of this we introduced a Validation trait, that handles validation of
section config data.
The UI allows users to easily create different types of fabrics. One can add
Nodes to the fabrics by selecting them from a dropdown which shows all the nodes
in the cluster. Additionally the user can then select the interfaces of the node
which should be added to the fabric. There are also protocol-specific options
such as "passive", "hello-interval" etc. available to select on the interface.
There are also options spanning whole fabrics: the "hello-interval" option on
openfabric for example, can be set on the fabric and will be applied to every
interface.
Refactoring
===========
This patch series required some rework of existing functionality, mostly how SDN
generates the FRR configuration and writes /etc/network/interfaces. Prior the
FRR configuration was generated exclusively from the controllers, but fabrics
need to write it as well. Same goes for the interfaces file, which got written
by the Zone plugin, but Fabrics need to write this file as well.
For this we moved the FRR and ifupdown config generation one level up to the SDN
module, which now calls into the respective child modules to generate the FRR /
ifupdown configuration.
Dependencies
============
pve-manager depends on pve-docs
pve-manager depends on pve-network
pve-network depends on proxmox-perl-rs
pve-network depends on pve-cluster
proxmox-perl-rs depends on proxmox-ve-config
proxmox-perl-rs depends on proxmox-frr
proxmox-perl-rs depends on proxmox-network-types
proxmox-ve-config depends on proxmox-frr
proxmox-ve-config depends on proxmox-network-types
proxmox-frr depends on proxmox-network-types
proxmox-ve-config depends on proxmox-serde
proxmox-firewall depends on proxmox-network-types
Changelog v1:
=============
proxmox-ve-rs
-------------
* remove intermediate-config, convert section-config directly to frr-types.
* add validation layer to validate the section-config
* simplify openfabric `net` to `router-id`
* add loopback prefixes to ensure that all router-ids are in a specific subnet
* generate router-map and access-lists to rewrite the source address of all
the routes received through openfabric and ospf
* add integration tests
* add option for ospf unnumbered
* only allow ipv4 on ospf
pve-network
-------------
* rework frr config generation
* rework etc/network/interfaces config generation
* revert "return loopback interface"
proxmox-perl-rs
-------------
* generate /etc/network/interfaces config to set ip-addresses
* auto-generate dummy interface for every fabric
pve-manager
-------------
* simplify a lot
* remove interface entries in tree
* hide specific openfabric/ospf options (hello-interval, passive etc.)
frr (external)
--------------
* fix --dummy_as_loopback bug (already on staging)
RFC
===
Changelog v2:
=============
proxmox-ve-rs
-------------
* serialize internal representation directly to the frr format
* add integration tests to proxmox-frr
* change internal representation to use BTreeMap instead of HashMap (so that
the test output is ordered)
* move some stuff from proxmox-frr and proxmox-ve-config to proxmox-network-types
pve-network
-----------
* generate frr config and append to running config directly (without going
through perl frr merging)
* check permissions on each fabric when listing
pve-manager
-----------
* autogenerate net and router-id when selecting the first interface
pve-cluster
-----------
* update the config files in status.c (pve-cluster) (thanks @Thomas)
frr (external)
--------------
* got this one merged: https://github.com/FRRouting/frr/pull/18242, so we
*could* automatically add dummy interfaces
Big thanks to Stefan Hanreich for his help and support throughout this series!
proxmox:
Gabriel Goller (1):
serde: add string_as_bool module for boolean string parsing
proxmox-serde/src/lib.rs | 84 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 84 insertions(+)
proxmox-ve-rs:
Gabriel Goller (17):
add proxmox-network-types crate
network-types: add common hostname and openfabric types
network-types: add openfabric NET type
network-types: move Ipv4Cidr and Ipv6Cidr types
frr: create proxmox-frr crate
frr: add common frr types
frr: add openfabric types
frr: add ospf types
frr: add route-map types
frr: add generic types over openfabric and ospf
frr: add serializer for all FRR types
ve-config: add openfabric section-config
ve-config: add ospf section-config
ve-config: add FRR conversion helpers for openfabric and ospf
ve-config: add validation for section-config
ve-config: add section-config to frr types conversion
ve-config: add integrations tests
.gitignore | 1 +
Cargo.toml | 7 +
proxmox-frr/Cargo.toml | 22 +
proxmox-frr/debian/changelog | 5 +
proxmox-frr/debian/control | 47 ++
proxmox-frr/debian/copyright | 18 +
proxmox-frr/debian/debcargo.toml | 7 +
proxmox-frr/src/lib.rs | 218 +++++++++
proxmox-frr/src/openfabric.rs | 93 ++++
proxmox-frr/src/ospf.rs | 135 +++++
proxmox-frr/src/route_map.rs | 128 +++++
proxmox-frr/src/serializer.rs | 192 ++++++++
proxmox-network-types/Cargo.toml | 15 +
proxmox-network-types/debian/changelog | 5 +
proxmox-network-types/debian/control | 41 ++
proxmox-network-types/debian/copyright | 18 +
proxmox-network-types/debian/debcargo.toml | 7 +
.../src}/address.rs | 14 +-
proxmox-network-types/src/hostname.rs | 34 ++
proxmox-network-types/src/lib.rs | 4 +
proxmox-network-types/src/net.rs | 382 +++++++++++++++
proxmox-network-types/src/openfabric.rs | 89 ++++
proxmox-ve-config/Cargo.toml | 17 +-
proxmox-ve-config/debian/control | 37 +-
proxmox-ve-config/src/firewall/cluster.rs | 3 +-
proxmox-ve-config/src/firewall/ct_helper.rs | 8 +-
proxmox-ve-config/src/firewall/host.rs | 3 +-
proxmox-ve-config/src/firewall/types/alias.rs | 3 +-
proxmox-ve-config/src/firewall/types/ipset.rs | 26 +-
proxmox-ve-config/src/firewall/types/mod.rs | 2 -
proxmox-ve-config/src/firewall/types/rule.rs | 3 +-
.../src/firewall/types/rule_match.rs | 5 +-
proxmox-ve-config/src/guest/vm.rs | 4 +-
proxmox-ve-config/src/host/utils.rs | 2 +-
proxmox-ve-config/src/sdn/config.rs | 8 +-
proxmox-ve-config/src/sdn/fabric/mod.rs | 460 ++++++++++++++++++
.../src/sdn/fabric/openfabric/frr.rs | 24 +
.../src/sdn/fabric/openfabric/mod.rs | 291 +++++++++++
.../src/sdn/fabric/openfabric/validation.rs | 56 +++
proxmox-ve-config/src/sdn/fabric/ospf/frr.rs | 29 ++
proxmox-ve-config/src/sdn/fabric/ospf/mod.rs | 245 ++++++++++
.../src/sdn/fabric/ospf/validation.rs | 53 ++
proxmox-ve-config/src/sdn/ipam.rs | 4 +-
proxmox-ve-config/src/sdn/mod.rs | 3 +-
proxmox-ve-config/tests/fabric/helper.rs | 44 ++
proxmox-ve-config/tests/fabric/main.rs | 80 +++
.../resources/cfg/openfabric_default.cfg | 17 +
.../cfg/openfabric_verification_fail.cfg | 11 +
.../fabric/resources/cfg/ospf_default.cfg | 10 +
.../resources/cfg/ospf_verification_fail.cfg | 11 +
.../resources/frr/openfabric_default.pve.frr | 32 ++
.../resources/frr/openfabric_default.pve1.frr | 28 ++
.../fabric/resources/frr/ospf_default.pve.frr | 26 +
.../resources/frr/ospf_default.pve1.frr | 21 +
proxmox-ve-config/tests/sdn/main.rs | 3 +-
55 files changed, 3011 insertions(+), 40 deletions(-)
create mode 100644 proxmox-frr/Cargo.toml
create mode 100644 proxmox-frr/debian/changelog
create mode 100644 proxmox-frr/debian/control
create mode 100644 proxmox-frr/debian/copyright
create mode 100644 proxmox-frr/debian/debcargo.toml
create mode 100644 proxmox-frr/src/lib.rs
create mode 100644 proxmox-frr/src/openfabric.rs
create mode 100644 proxmox-frr/src/ospf.rs
create mode 100644 proxmox-frr/src/route_map.rs
create mode 100644 proxmox-frr/src/serializer.rs
create mode 100644 proxmox-network-types/Cargo.toml
create mode 100644 proxmox-network-types/debian/changelog
create mode 100644 proxmox-network-types/debian/control
create mode 100644 proxmox-network-types/debian/copyright
create mode 100644 proxmox-network-types/debian/debcargo.toml
rename {proxmox-ve-config/src/firewall/types => proxmox-network-types/src}/address.rs (99%)
create mode 100644 proxmox-network-types/src/hostname.rs
create mode 100644 proxmox-network-types/src/lib.rs
create mode 100644 proxmox-network-types/src/net.rs
create mode 100644 proxmox-network-types/src/openfabric.rs
create mode 100644 proxmox-ve-config/src/sdn/fabric/mod.rs
create mode 100644 proxmox-ve-config/src/sdn/fabric/openfabric/frr.rs
create mode 100644 proxmox-ve-config/src/sdn/fabric/openfabric/mod.rs
create mode 100644 proxmox-ve-config/src/sdn/fabric/openfabric/validation.rs
create mode 100644 proxmox-ve-config/src/sdn/fabric/ospf/frr.rs
create mode 100644 proxmox-ve-config/src/sdn/fabric/ospf/mod.rs
create mode 100644 proxmox-ve-config/src/sdn/fabric/ospf/validation.rs
create mode 100644 proxmox-ve-config/tests/fabric/helper.rs
create mode 100644 proxmox-ve-config/tests/fabric/main.rs
create mode 100644 proxmox-ve-config/tests/fabric/resources/cfg/openfabric_default.cfg
create mode 100644 proxmox-ve-config/tests/fabric/resources/cfg/openfabric_verification_fail.cfg
create mode 100644 proxmox-ve-config/tests/fabric/resources/cfg/ospf_default.cfg
create mode 100644 proxmox-ve-config/tests/fabric/resources/cfg/ospf_verification_fail.cfg
create mode 100644 proxmox-ve-config/tests/fabric/resources/frr/openfabric_default.pve.frr
create mode 100644 proxmox-ve-config/tests/fabric/resources/frr/openfabric_default.pve1.frr
create mode 100644 proxmox-ve-config/tests/fabric/resources/frr/ospf_default.pve.frr
create mode 100644 proxmox-ve-config/tests/fabric/resources/frr/ospf_default.pve1.frr
proxmox-firewall:
Stefan Hanreich (1):
firewall: nftables: migrate to proxmox-network-types
Cargo.toml | 1 +
proxmox-firewall/Cargo.toml | 1 +
proxmox-firewall/src/firewall.rs | 2 +-
proxmox-firewall/src/object.rs | 4 +++-
proxmox-firewall/src/rule.rs | 3 ++-
proxmox-nftables/Cargo.toml | 3 ++-
proxmox-nftables/src/expression.rs | 5 +----
proxmox-nftables/src/types.rs | 2 +-
8 files changed, 12 insertions(+), 9 deletions(-)
proxmox-perl-rs:
Gabriel Goller (7):
perl-rs: sdn: initial fabric infrastructure
perl-rs: sdn: add CRUD helpers for OpenFabric fabric management
perl-rs: sdn: OpenFabric perlmod methods
perl-rs: sdn: implement OSPF interface file configuration generation
perl-rs: sdn: add CRUD helpers for OSPF fabric management
perl-rs: sdn: OSPF perlmod methods
perl-rs: sdn: implement OSPF interface file configuration generation
pve-rs/Cargo.toml | 6 +-
pve-rs/Makefile | 3 +
pve-rs/src/lib.rs | 1 +
pve-rs/src/sdn/fabrics.rs | 50 ++++
pve-rs/src/sdn/mod.rs | 3 +
pve-rs/src/sdn/openfabric.rs | 462 +++++++++++++++++++++++++++++++++++
pve-rs/src/sdn/ospf.rs | 427 ++++++++++++++++++++++++++++++++
7 files changed, 951 insertions(+), 1 deletion(-)
create mode 100644 pve-rs/src/sdn/fabrics.rs
create mode 100644 pve-rs/src/sdn/mod.rs
create mode 100644 pve-rs/src/sdn/openfabric.rs
create mode 100644 pve-rs/src/sdn/ospf.rs
pve-cluster:
Gabriel Goller (1):
cluster: add sdn fabrics config files
src/PVE/Cluster.pm | 2 ++
src/pmxcfs/status.c | 2 ++
2 files changed, 4 insertions(+)
pve-network:
Gabriel Goller (1):
debian: add dependency to proxmox-perl-rs
Stefan Hanreich (16):
sdn: fix value returned by pending_config
fabrics: add fabrics module
refactor: controller: move frr methods into helper
controllers: implement new api for frr config generation
sdn: add frr config generation helper
test: isis: add test for standalone configuration
sdn: frr: add daemon status to frr helper
sdn: running: apply fabrics config
fabrics: generate ifupdown configuration
api: add fabrics subfolder
api: fabrics: add common helpers
fabric: openfabric: add api endpoints
fabric: ospf: add api endpoints
test: fabrics: add test cases for ospf and openfabric + evpn
frr: bump frr config version to 10.2.1
frr: fix reloading frr configuration
debian/control | 2 +
src/PVE/API2/Network/SDN.pm | 7 +
src/PVE/API2/Network/SDN/Fabrics.pm | 294 ++++++++++++
src/PVE/API2/Network/SDN/Fabrics/Common.pm | 80 ++++
src/PVE/API2/Network/SDN/Fabrics/Makefile | 9 +
.../API2/Network/SDN/Fabrics/OpenFabric.pm | 348 +++++++++++++++
src/PVE/API2/Network/SDN/Fabrics/Ospf.pm | 345 ++++++++++++++
src/PVE/API2/Network/SDN/Makefile | 3 +-
src/PVE/Network/SDN.pm | 138 +++++-
src/PVE/Network/SDN/Controllers.pm | 67 +--
src/PVE/Network/SDN/Controllers/BgpPlugin.pm | 21 +-
src/PVE/Network/SDN/Controllers/EvpnPlugin.pm | 295 +-----------
src/PVE/Network/SDN/Controllers/IsisPlugin.pm | 21 +-
src/PVE/Network/SDN/Controllers/Plugin.pm | 31 +-
src/PVE/Network/SDN/Fabrics.pm | 130 ++++++
src/PVE/Network/SDN/Frr.pm | 420 ++++++++++++++++++
src/PVE/Network/SDN/Makefile | 2 +-
src/PVE/Network/SDN/Zones.pm | 10 -
src/test/run_test_zones.pl | 11 +-
.../expected_controller_config | 2 +-
.../expected_controller_config | 2 +-
.../evpn/ebgp/expected_controller_config | 2 +-
.../ebgp_loopback/expected_controller_config | 2 +-
.../evpn/exitnode/expected_controller_config | 2 +-
.../expected_controller_config | 2 +-
.../expected_controller_config | 2 +-
.../exitnode_snat/expected_controller_config | 2 +-
.../expected_controller_config | 2 +-
.../evpn/ipv4/expected_controller_config | 2 +-
.../evpn/ipv4ipv6/expected_controller_config | 2 +-
.../expected_controller_config | 2 +-
.../evpn/ipv6/expected_controller_config | 2 +-
.../ipv6underlay/expected_controller_config | 2 +-
.../evpn/isis/expected_controller_config | 2 +-
.../isis_loopback/expected_controller_config | 2 +-
.../expected_controller_config | 22 +
.../isis_standalone/expected_sdn_interfaces | 1 +
.../zones/evpn/isis_standalone/interfaces | 12 +
.../zones/evpn/isis_standalone/sdn_config | 21 +
.../expected_controller_config | 2 +-
.../multiplezones/expected_controller_config | 2 +-
.../expected_controller_config | 72 +++
.../openfabric_fabric/expected_sdn_interfaces | 56 +++
.../zones/evpn/openfabric_fabric/interfaces | 6 +
.../zones/evpn/openfabric_fabric/sdn_config | 79 ++++
.../ospf_fabric/expected_controller_config | 66 +++
.../evpn/ospf_fabric/expected_sdn_interfaces | 53 +++
src/test/zones/evpn/ospf_fabric/interfaces | 6 +
src/test/zones/evpn/ospf_fabric/sdn_config | 75 ++++
.../evpn/rt_import/expected_controller_config | 2 +-
.../evpn/vxlanport/expected_controller_config | 2 +-
51 files changed, 2270 insertions(+), 473 deletions(-)
create mode 100644 src/PVE/API2/Network/SDN/Fabrics.pm
create mode 100644 src/PVE/API2/Network/SDN/Fabrics/Common.pm
create mode 100644 src/PVE/API2/Network/SDN/Fabrics/Makefile
create mode 100644 src/PVE/API2/Network/SDN/Fabrics/OpenFabric.pm
create mode 100644 src/PVE/API2/Network/SDN/Fabrics/Ospf.pm
create mode 100644 src/PVE/Network/SDN/Fabrics.pm
create mode 100644 src/PVE/Network/SDN/Frr.pm
create mode 100644 src/test/zones/evpn/isis_standalone/expected_controller_config
create mode 100644 src/test/zones/evpn/isis_standalone/expected_sdn_interfaces
create mode 100644 src/test/zones/evpn/isis_standalone/interfaces
create mode 100644 src/test/zones/evpn/isis_standalone/sdn_config
create mode 100644 src/test/zones/evpn/openfabric_fabric/expected_controller_config
create mode 100644 src/test/zones/evpn/openfabric_fabric/expected_sdn_interfaces
create mode 100644 src/test/zones/evpn/openfabric_fabric/interfaces
create mode 100644 src/test/zones/evpn/openfabric_fabric/sdn_config
create mode 100644 src/test/zones/evpn/ospf_fabric/expected_controller_config
create mode 100644 src/test/zones/evpn/ospf_fabric/expected_sdn_interfaces
create mode 100644 src/test/zones/evpn/ospf_fabric/interfaces
create mode 100644 src/test/zones/evpn/ospf_fabric/sdn_config
pve-manager:
Gabriel Goller (7):
api: use new generalized frr and etc network config helper functions
fabrics: add common interface panel
fabrics: add additional interface fields for openfabric and ospf
fabrics: add FabricEdit components
fabrics: add NodeEdit components
fabrics: Add main FabricView
utils: avoid line-break in pending changes message
PVE/API2/Network.pm | 6 +-
www/manager6/Makefile | 8 +
www/manager6/Utils.js | 2 +-
www/manager6/dc/Config.js | 8 +
www/manager6/sdn/FabricsView.js | 430 ++++++++++++++++++
www/manager6/sdn/fabrics/Common.js | 285 ++++++++++++
.../sdn/fabrics/openfabric/FabricEdit.js | 71 +++
.../sdn/fabrics/openfabric/InterfaceEdit.js | 64 +++
.../sdn/fabrics/openfabric/NodeEdit.js | 205 +++++++++
www/manager6/sdn/fabrics/ospf/FabricEdit.js | 64 +++
.../sdn/fabrics/ospf/InterfaceEdit.js | 27 ++
www/manager6/sdn/fabrics/ospf/NodeEdit.js | 207 +++++++++
12 files changed, 1374 insertions(+), 3 deletions(-)
create mode 100644 www/manager6/sdn/FabricsView.js
create mode 100644 www/manager6/sdn/fabrics/Common.js
create mode 100644 www/manager6/sdn/fabrics/openfabric/FabricEdit.js
create mode 100644 www/manager6/sdn/fabrics/openfabric/InterfaceEdit.js
create mode 100644 www/manager6/sdn/fabrics/openfabric/NodeEdit.js
create mode 100644 www/manager6/sdn/fabrics/ospf/FabricEdit.js
create mode 100644 www/manager6/sdn/fabrics/ospf/InterfaceEdit.js
create mode 100644 www/manager6/sdn/fabrics/ospf/NodeEdit.js
pve-docs:
Gabriel Goller (1):
fabrics: add initial documentation for sdn fabrics
pvesdn.adoc | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 155 insertions(+)
Summary over all repositories:
137 files changed, 7861 insertions(+), 526 deletions(-)
--
Generated by git-murpp 0.8.0
More information about the pve-devel
mailing list