[pve-devel] [PATCH ve-rs/firewall/qemu-server/manager v2 00/13] fix #5180: migrate conntrack state on live migration

[Stefan Hanreich]

Gave this a spin on my test cluster, and it worked as advertised. As
already mentioned and talked off-list we should imo add documentation
for this setting and mention the current restrictions (no migration of
NAT entries). We might also want to mention that only CT entries created
*after* the firewall is enabled for a VM get migrated, although that is
probably rather uncommon.

Other than that consider this:

Tested-by: Stefan Hanreich <s.hanreich at proxmox.com>

I've also mainly looked at the firewall + pve-manager code, maybe
someone with more knowledge about QEMU can chime in on the DBus server
parts. Consider the firewall and ui patches:

Reviewed-by: Stefan Hanreich <s.hanreich at proxmox.com>


On 4/24/25 13:19, Christoph Heiss wrote:
> Fixes #5180 [0].
> 
> This implements migration of per-VM conntrack state on live-migration.
> 
> The core of the implementation are in patch #7 & #8. See there for more
> details.
> 
> Patch #1 - #3 implement CONNMARK'ing any VM traffic with their unique
> VMID. This is needed later on to filter conntrack entries for the
> migration. These three patches can be applied independently,
> CONNMARK'ing traffic does not have any visible impact.
> 
> Currently, remote/inter-cluster migration is not supported and indicated
> to the user with a warning. See also patch #8 for a bit more in-depth
> explanation.
> 
> Needed dependency bumps between packages are indicated in the notes
> appropriately.
> 
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5180
> 
> Testing
> =======
> 
> I've primarily tested intra-cluster live-migrations, with both the
> iptables-based and nftables-based firewall), using the reproducer as
> described in #5180. I further verified that the D-Bus servers get
> started as expected and are _always_ stopped, even in the case of some
> migration error.
> 
> Finally, I also checked using `conntrack -L -m <vmid>` tool that the
> conntrack entries are 
> a) added/updated on the target node and 
> b) removed from the source node afterwards
> 
> Also tested was the migration from/to an "old" (unpatched) node, which
> results in the issue as per #5180 & appropriate warnings in the UI.
> 
> For remote migrations, only tested that the warning is logged as
> expected.
> 
> History
> =======
> 
> v1: https://lore.proxmox.com/pve-devel/20250317141152.1247324-1-c.heiss@proxmox.com/
> 
> Changes v1 -> v2:
>   * rebased as necessary
>   * "un-rfc'd" firewall conntrack flushing patches
>   * use an instanced systemd service instead of fork+exec for the
>     pve-dbus-vmstate helper
> 
> Diffstat
> ========
> 
> pve-firewall:
> 
> Christoph Heiss (2):
>   firewall: add connmark rule with VMID to all guest chains
>   firewall: helpers: add sub for flushing conntrack entries by mark
> 
>  debian/control              |  3 ++-
>  src/PVE/Firewall.pm         |  7 +++++--
>  src/PVE/Firewall/Helpers.pm | 11 +++++++++++
>  3 files changed, 18 insertions(+), 3 deletions(-)
> 
> proxmox-firewall:
> 
> Christoph Heiss (1):
>   firewall: add connmark rule with VMID to all guest chains
> 
>  proxmox-firewall/src/firewall.rs              | 14 +++-
>  .../integration_tests__firewall.snap          | 84 +++++++++++++++++++
>  proxmox-nftables/src/expression.rs            |  9 ++
>  proxmox-nftables/src/statement.rs             | 10 ++-
>  4 files changed, 114 insertions(+), 3 deletions(-)
> 
> proxmox-ve-rs:
> 
> Christoph Heiss (1):
>   config: guest: allow access to raw Vmid value
> 
>  proxmox-ve-config/src/guest/types.rs | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> qemu-server:
> 
> Christoph Heiss (5):
>   qmp helpers: allow passing structured args via qemu_objectadd()
>   api2: qemu: add module exposing node migration capabilities
>   fix #5180: libexec: add QEMU dbus-vmstate daemon for migrating
>     conntrack
>   fix #5180: migrate: integrate helper for live-migrating conntrack info
>   migrate: flush old VM conntrack entries after successful migration
> 
>  Makefile                               |   7 +-
>  PVE/API2/Qemu.pm                       |  72 +++++++++++
>  PVE/API2/Qemu/Makefile                 |   2 +-
>  PVE/API2/Qemu/Migration.pm             |  46 +++++++
>  PVE/CLI/qm.pm                          |   5 +
>  PVE/QemuMigrate.pm                     |  69 ++++++++++
>  PVE/QemuServer.pm                      |   6 +
>  PVE/QemuServer/DBusVMState.pm          | 120 ++++++++++++++++++
>  PVE/QemuServer/Makefile                |   1 +
>  PVE/QemuServer/QMPHelpers.pm           |   4 +-
>  dbus-vmstate/Makefile                  |   7 ++
>  dbus-vmstate/dbus-vmstate              | 168 +++++++++++++++++++++++++
>  dbus-vmstate/org.qemu.VMState1.conf    |  11 ++
>  dbus-vmstate/pve-dbus-vmstate at .service |  10 ++
>  debian/control                         |   7 +-
>  15 files changed, 530 insertions(+), 5 deletions(-)
>  create mode 100644 PVE/API2/Qemu/Migration.pm
>  create mode 100644 PVE/QemuServer/DBusVMState.pm
>  create mode 100644 dbus-vmstate/Makefile
>  create mode 100755 dbus-vmstate/dbus-vmstate
>  create mode 100644 dbus-vmstate/org.qemu.VMState1.conf
>  create mode 100644 dbus-vmstate/pve-dbus-vmstate at .service
> 
> pve-manager:
> 
> Christoph Heiss (4):
>   api2: capabilities: explicitly import CPU capabilities module
>   api2: capabilities: proxy index endpoints to respective nodes
>   api2: capabilities: expose new qemu/migration endpoint
>   ui: window: Migrate: add checkbox for migrating VM conntrack state
> 
>  PVE/API2/Capabilities.pm       |  9 +++++
>  www/manager6/window/Migrate.js | 73 ++++++++++++++++++++++++++++++++--
>  2 files changed, 78 insertions(+), 4 deletions(-)
>