[pve-devel] [PATCH ve-rs/firewall/qemu-server/manager/docs v3 00/14] fix #5180: migrate conntrack state on live migration

[Christoph Heiss]

Fixes #5180 [0].

This implements migration of per-VM conntrack state on live-migration.

The core of the implementation are in patch #7 & #8. See there for more
details.

Patch #1 - #3 implement CONNMARK'ing any VM traffic with their unique
VMID. This is needed later on to filter conntrack entries for the
migration. These three patches can be applied independently,
CONNMARK'ing traffic does not have any visible impact.

Regarding the use of connlabels instead, as was discussed in v2:
Connlabels are (unfortunately) pure bitmasks and cannot be used for
storing arbitrary values such as VMIDs. Thus we are basically stuck with
using connmarks.

Currently, remote/inter-cluster migration is not supported and indicated
to the user with a warning. See also patch #8 for a bit more in-depth
explanation.

[0] https://bugzilla.proxmox.com/show_bug.cgi?id=5180

Dependencies
============

proxmox-firewall depends on the proxmox-ve-rs changes.
qemu-server depends on the pve-firewall/proxmox-firewall changes.

pve-manager only soft-depends on the other, as it will detect whether
conntrack migration is supported.

Testing
=======

I've primarily tested intra-cluster live-migrations, with both the
iptables-based and nftables-based firewall), using the reproducer as
described in #5180. I further verified that the D-Bus services get
started as expected and are _always_ stopped, even in the case of some
migration error.

Finally, I also checked using `conntrack -L -m <vmid>` tool that the
conntrack entries are 
a) added/updated on the target node and 
b) removed from the source node afterwards

Also tested was the migration from/to an "old" (unpatched) node, which
results in the issue as per #5180 & appropriate warnings in the UI.

For remote migrations, tested that the warning is logged as expected.

History
=======

v1: https://lore.proxmox.com/pve-devel/20250317141152.1247324-1-c.heiss@proxmox.com/
v2: https://lore.proxmox.com/pve-devel/20250424111941.730528-1-c.heiss@proxmox.com/

Changes v1 -> v2:
  * rebased as necessary
  * "un-rfc'd" firewall conntrack flushing patches
  * use an instanced systemd service instead of fork+exec for the
    pve-dbus-vmstate helper

Changes v2 -> v3:
  * rebased on trixie/latest masters
  * added documentation patch
  * moved node capability module to 
    PVE::API2::NodeCapabilities::Qemu::Migration, based on Fiona's 
    suggestion

Diffstat
========

proxmox-ve-rs:

Christoph Heiss (1):
  config: guest: allow access to raw Vmid value

 proxmox-ve-config/src/guest/types.rs | 4 ++++
 1 file changed, 4 insertions(+)

proxmox-firewall:

Christoph Heiss (1):
  firewall: add connmark rule with VMID to all guest chains

 proxmox-firewall/src/firewall.rs              | 14 +++-
 .../integration_tests__firewall.snap          | 84 +++++++++++++++++++
 proxmox-nftables/src/expression.rs            |  9 ++
 proxmox-nftables/src/statement.rs             | 10 ++-
 4 files changed, 114 insertions(+), 3 deletions(-)

pve-firewall:

Christoph Heiss (2):
  firewall: add connmark rule with VMID to all guest chains
  firewall: helpers: add sub for flushing conntrack entries by mark

 debian/control              |  3 ++-
 src/PVE/Firewall.pm         | 14 ++++++++++++--
 src/PVE/Firewall/Helpers.pm | 12 ++++++++++++
 3 files changed, 26 insertions(+), 3 deletions(-)

qemu-server:

Christoph Heiss (5):
  qmp helpers: allow passing structured args via qemu_objectadd()
  api2: qemu: add module exposing node migration capabilities
  fix #5180: dbus-vmstate: add daemon for QEMUs dbus-vmstate interface
  fix #5180: migrate: integrate helper for live-migrating conntrack info
  migrate: flush old VM conntrack entries after successful migration

 Makefile                                      |   4 +-
 debian/control                                |   7 +-
 src/Makefile                                  |   1 +
 src/PVE/API2/Makefile                         |   1 +
 src/PVE/API2/NodeCapabilities/Makefile        |   9 +
 .../API2/NodeCapabilities/Qemu/Migration.pm   |  48 +++++
 src/PVE/API2/Qemu.pm                          |  75 ++++++++
 src/PVE/CLI/qm.pm                             |   5 +
 src/PVE/QemuMigrate.pm                        |  78 ++++++++
 src/PVE/QemuServer.pm                         |   6 +
 src/PVE/QemuServer/DBusVMState.pm             | 125 +++++++++++++
 src/PVE/QemuServer/Makefile                   |   1 +
 src/PVE/QemuServer/QMPHelpers.pm              |   4 +-
 src/dbus-vmstate/Makefile                     |  11 ++
 src/dbus-vmstate/dbus-vmstate                 | 168 ++++++++++++++++++
 src/dbus-vmstate/org.qemu.VMState1.conf       |  11 ++
 src/dbus-vmstate/pve-dbus-vmstate at .service    |  10 ++
 17 files changed, 560 insertions(+), 4 deletions(-)
 create mode 100644 src/PVE/API2/NodeCapabilities/Makefile
 create mode 100644 src/PVE/API2/NodeCapabilities/Qemu/Migration.pm
 create mode 100644 src/PVE/QemuServer/DBusVMState.pm
 create mode 100644 src/dbus-vmstate/Makefile
 create mode 100755 src/dbus-vmstate/dbus-vmstate
 create mode 100644 src/dbus-vmstate/org.qemu.VMState1.conf
 create mode 100644 src/dbus-vmstate/pve-dbus-vmstate at .service

pve-manager:

Christoph Heiss (4):
  api2: capabilities: explicitly import CPU capabilities module
  api2: capabilities: proxy index endpoints to respective nodes
  api2: capabilities: expose new qemu/migration endpoint
  ui: window: Migrate: add checkbox for migrating VM conntrack state

 PVE/API2/Capabilities.pm       | 11 ++++-
 www/manager6/window/Migrate.js | 82 ++++++++++++++++++++++++++++++++--
 2 files changed, 89 insertions(+), 4 deletions(-)

pve-docs:

Christoph Heiss (1):
  qm: document conntrack state migration for live migrations

 qm.adoc | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

-- 
2.47.1