[pve-devel] [PATCH container/docs/firewall/manager/proxmox-firewall/qemu-server v3 00/39] proxmox firewall nftables implementation

[Stefan Hanreich]

## Introduction
This RFC provides a drop-in replacement for the current pve-firewall package
that is based on Rust and nftables.

It consists of three crates:
* proxmox-ve-config
  for parsing firewall and guest configuration files, as well as some helpers
  to access host configuration (particularly networking)
* proxmox-nftables
  contains bindings for libnftables as well as types that implement the JSON
  schema defined by libnftables-json
* proxmox-firewall
  uses the other two crates to read the firewall configuration and create the
  respective nftables configuration


## Installation
* Build & install all deb packages on your PVE instance
* Enable the nftables firewall by going to
  Web UI > <Host> > Firewall > Options > nftables
* Enable the firewall datacenter-wide if you haven't already
* Restarting running VMs/CTs is required so the changes to the fwbr creation
  go into effect

For your convenience I have provided pre-built packages on our share under
`shanreich-proxmox-firewall`.

The source code is also available on my staff repo as `proxmox-firewall`.


## Configuration
The firewall should work as a drop-in replacement for the pve-firewall, so you
should be able to configure the firewall as usual via the Web UI or
configuration files.


## Known Issues
There is currently one major issue that we still need to solve: REJECTing
packets from the guest firewalls is currently not possible for incoming traffic
(it will instead be dropped).

This is due to the fact that we are using the postrouting hook of nftables in a
table with type bridge for incoming traffic. In the bridge table in the
postrouting hook we cannot tell whether the packet has also been sent to other
ports in the bridge (e.g. when a MAC has not yet been learned and the packet
then gets flooded to all bridge ports). If we would then REJECT a packet in the
postrouting hook this can lead to a bug where the firewall rules for one guest
REJECT a packet and send a response (RST for TCP, ICMP port/host-unreachable
otherwise).

This has also been explained in the respective commit introducing the
restriction [1].

We were able to circumvent this restriction in the old firewall due to using
firewall bridges and rejecting in the firewall bridge itself. Doing this leads
to the behavior described above, which has tripped up some of our users before
[2] [3] and which is, frankly, wrong.

I currently see two possible solutions for this, both of which carry downsides.
Your input on this matter would be much appreciated, particularly if you can
think of another solution which I cannot currently see:

1. Only REJECT packets in the prerouting chain of the firewall bridge with the
destination MAC address set to the MAC address of the network device, otherwise
DROP

The downside of this is that we, once again, will have to resort to using
firewall bridges, which we wanted to eliminate. This would also be the sole
reason for still having to resort to using firewall bridges.

2. Only allow DROP in the guest firewall for incoming traffic

This would be quite awkward since, well, rejecting traffic would be quite nice
for a firewall I'd say ;)

I'm happy for all input regarding this matter.


## Useful Commands

You can check if firewall rules got created by running

```
nft list ruleset
```

You can also check that `iptables` rules are not created via
```
iptables-save
```

Further info about the services:
```
systemctl status proxmox-firewall.{service,timer}
```

You can grab the debug output from the new firewall like so:

```
RUST_LOG=trace proxmox-firewall
```

## Upcoming
There are many ideas for further features, here is a (non-exhaustive) list:
* SNAT/DNAT handling
* SDN integration + bridge/vnet-level firewalling
* counters
* flow offloading
* synproxy support
* rate-limiting for rules
* connlimit support
* brouting support

The first thing I'll be working on though is proper rustdocs for both libraries
as well as the firewall.


Changes from v2 -> v3:
* move from libnftables to nft CLI
* daemonize proxmox-firewall
* prettify chains.json
* move location of proxmox-firewall to /usr/libexec/proxmox
* update documentation to reflect the changes

Changes from v1 -> v2:
* now builds cleanly in sbuild (thanks @Fabian)
* made base ruleset more efficient
* fixed issues with some rules in the base ruleset
* refactored config loading in order to make it mockable (thanks @Lukas)
* added an integration test that spans the whole pipeline from config ->
  nftables rules
* changed many maps from HashMap to BTreeMap to make the generated nftables
  output stable (particularly important for the integration test)
* improved logging output
* implemented ARP spoofing prevention
* failing to change sysctl settings now only emits a warning instead of aborting
  the rule generation process
* fixed some bugs wrt rule generation
* added a few macros (HTTP/3, PBS)
* added documentation to pve-docs
* EUI64 link-local addresses are now added to the automatically generated ip
  filters
* incorporated suggestions from @Max and @Lukas (tyvm!)


[1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/net/bridge/netfilter/nft_reject_bridge.c?h=v6.8.2&id=127917c29a432c3b798e014a1714e9c1af0f87fe
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=4964
[3] https://forum.proxmox.com/threads/proxmox-claiming-mac-address.52601/page-2#post-415493

proxmox-firewall:

Stefan Hanreich (34):
  config: add proxmox-ve-config crate
  config: firewall: add types for ip addresses
  config: firewall: add types for ports
  config: firewall: add types for log level and rate limit
  config: firewall: add types for aliases
  config: host: add helpers for host network configuration
  config: guest: add helpers for parsing guest network config
  config: firewall: add types for ipsets
  config: firewall: add types for rules
  config: firewall: add types for security groups
  config: firewall: add generic parser for firewall configs
  config: firewall: add cluster-specific config + option types
  config: firewall: add host specific config + option types
  config: firewall: add guest-specific config + option types
  config: firewall: add firewall macros
  config: firewall: add conntrack helper types
  nftables: add crate for libnftables bindings
  nftables: add helpers
  nftables: expression: add types
  nftables: expression: implement conversion traits for firewall config
  nftables: statement: add types
  nftables: statement: add conversion traits for config types
  nftables: commands: add types
  nftables: types: add conversion traits
  nftables: add nft client
  firewall: add firewall crate
  firewall: add base ruleset
  firewall: add config loader
  firewall: add rule generation logic
  firewall: add object generation logic
  firewall: add ruleset generation logic
  firewall: add proxmox-firewall binary and move existing code into lib
  firewall: add files for debian packaging
  firewall: add integration test


qemu-server:

Stefan Hanreich (1):
  firewall: add handling for new nft firewall

 vm-network-scripts/pve-bridge | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)


pve-container:

Stefan Hanreich (1):
  firewall: add handling for new nft firewall

 src/PVE/LXC.pm | 5 +++++
 1 file changed, 5 insertions(+)


pve-firewall:

Stefan Hanreich (1):
  add configuration option for new nftables firewall

 src/PVE/Firewall.pm | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)


pve-manager:

Stefan Hanreich (1):
  firewall: expose configuration option for new nftables firewall

 www/manager6/grid/FirewallOptions.js | 1 +
 1 file changed, 1 insertion(+)


pve-docs:

Stefan Hanreich (1):
  firewall: add documentation for proxmox-firewall

 pve-firewall.adoc | 185 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 185 insertions(+)


Summary over all repositories:
  5 files changed, 214 insertions(+), 6 deletions(-)

-- 
Generated by git-murpp 0.6.0