[pve-devel] [PATCH ve-rs/firewall/qemu-server/manager/docs v3 00/14] fix #5180: migrate conntrack state on live migration
Christoph Heiss
c.heiss at proxmox.com
Thu Jul 17 16:16:49 CEST 2025
v4 posted: https://lore.proxmox.com/pve-devel/20250717141530.1471199-1-c.heiss@proxmox.com/
On Thu Jul 3, 2025 at 1:54 PM CEST, Christoph Heiss wrote:
> Fixes #5180 [0].
>
> This implements migration of per-VM conntrack state on live-migration.
>
> The core of the implementation are in patch #7 & #8. See there for more
> details.
>
> Patch #1 - #3 implement CONNMARK'ing any VM traffic with their unique
> VMID. This is needed later on to filter conntrack entries for the
> migration. These three patches can be applied independently,
> CONNMARK'ing traffic does not have any visible impact.
>
> Regarding the use of connlabels instead, as was discussed in v2:
> Connlabels are (unfortunately) pure bitmasks and cannot be used for
> storing arbitrary values such as VMIDs. Thus we are basically stuck with
> using connmarks.
>
> Currently, remote/inter-cluster migration is not supported and indicated
> to the user with a warning. See also patch #8 for a bit more in-depth
> explanation.
>
> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5180
>
> Dependencies
> ============
>
> proxmox-firewall depends on the proxmox-ve-rs changes.
> qemu-server depends on the pve-firewall/proxmox-firewall changes.
>
> pve-manager only soft-depends on the other, as it will detect whether
> conntrack migration is supported.
>
> Testing
> =======
>
> I've primarily tested intra-cluster live-migrations, with both the
> iptables-based and nftables-based firewall), using the reproducer as
> described in #5180. I further verified that the D-Bus services get
> started as expected and are _always_ stopped, even in the case of some
> migration error.
>
> Finally, I also checked using `conntrack -L -m <vmid>` tool that the
> conntrack entries are
> a) added/updated on the target node and
> b) removed from the source node afterwards
>
> Also tested was the migration from/to an "old" (unpatched) node, which
> results in the issue as per #5180 & appropriate warnings in the UI.
>
> For remote migrations, tested that the warning is logged as expected.
>
> History
> =======
>
> v1: https://lore.proxmox.com/pve-devel/20250317141152.1247324-1-c.heiss@proxmox.com/
> v2: https://lore.proxmox.com/pve-devel/20250424111941.730528-1-c.heiss@proxmox.com/
>
> Changes v1 -> v2:
> * rebased as necessary
> * "un-rfc'd" firewall conntrack flushing patches
> * use an instanced systemd service instead of fork+exec for the
> pve-dbus-vmstate helper
>
> Changes v2 -> v3:
> * rebased on trixie/latest masters
> * added documentation patch
> * moved node capability module to
> PVE::API2::NodeCapabilities::Qemu::Migration, based on Fiona's
> suggestion
>
> Diffstat
> ========
>
> proxmox-ve-rs:
>
> Christoph Heiss (1):
> config: guest: allow access to raw Vmid value
>
> proxmox-ve-config/src/guest/types.rs | 4 ++++
> 1 file changed, 4 insertions(+)
>
> proxmox-firewall:
>
> Christoph Heiss (1):
> firewall: add connmark rule with VMID to all guest chains
>
> proxmox-firewall/src/firewall.rs | 14 +++-
> .../integration_tests__firewall.snap | 84 +++++++++++++++++++
> proxmox-nftables/src/expression.rs | 9 ++
> proxmox-nftables/src/statement.rs | 10 ++-
> 4 files changed, 114 insertions(+), 3 deletions(-)
>
> pve-firewall:
>
> Christoph Heiss (2):
> firewall: add connmark rule with VMID to all guest chains
> firewall: helpers: add sub for flushing conntrack entries by mark
>
> debian/control | 3 ++-
> src/PVE/Firewall.pm | 14 ++++++++++++--
> src/PVE/Firewall/Helpers.pm | 12 ++++++++++++
> 3 files changed, 26 insertions(+), 3 deletions(-)
>
> qemu-server:
>
> Christoph Heiss (5):
> qmp helpers: allow passing structured args via qemu_objectadd()
> api2: qemu: add module exposing node migration capabilities
> fix #5180: dbus-vmstate: add daemon for QEMUs dbus-vmstate interface
> fix #5180: migrate: integrate helper for live-migrating conntrack info
> migrate: flush old VM conntrack entries after successful migration
>
> Makefile | 4 +-
> debian/control | 7 +-
> src/Makefile | 1 +
> src/PVE/API2/Makefile | 1 +
> src/PVE/API2/NodeCapabilities/Makefile | 9 +
> .../API2/NodeCapabilities/Qemu/Migration.pm | 48 +++++
> src/PVE/API2/Qemu.pm | 75 ++++++++
> src/PVE/CLI/qm.pm | 5 +
> src/PVE/QemuMigrate.pm | 78 ++++++++
> src/PVE/QemuServer.pm | 6 +
> src/PVE/QemuServer/DBusVMState.pm | 125 +++++++++++++
> src/PVE/QemuServer/Makefile | 1 +
> src/PVE/QemuServer/QMPHelpers.pm | 4 +-
> src/dbus-vmstate/Makefile | 11 ++
> src/dbus-vmstate/dbus-vmstate | 168 ++++++++++++++++++
> src/dbus-vmstate/org.qemu.VMState1.conf | 11 ++
> src/dbus-vmstate/pve-dbus-vmstate at .service | 10 ++
> 17 files changed, 560 insertions(+), 4 deletions(-)
> create mode 100644 src/PVE/API2/NodeCapabilities/Makefile
> create mode 100644 src/PVE/API2/NodeCapabilities/Qemu/Migration.pm
> create mode 100644 src/PVE/QemuServer/DBusVMState.pm
> create mode 100644 src/dbus-vmstate/Makefile
> create mode 100755 src/dbus-vmstate/dbus-vmstate
> create mode 100644 src/dbus-vmstate/org.qemu.VMState1.conf
> create mode 100644 src/dbus-vmstate/pve-dbus-vmstate at .service
>
> pve-manager:
>
> Christoph Heiss (4):
> api2: capabilities: explicitly import CPU capabilities module
> api2: capabilities: proxy index endpoints to respective nodes
> api2: capabilities: expose new qemu/migration endpoint
> ui: window: Migrate: add checkbox for migrating VM conntrack state
>
> PVE/API2/Capabilities.pm | 11 ++++-
> www/manager6/window/Migrate.js | 82 ++++++++++++++++++++++++++++++++--
> 2 files changed, 89 insertions(+), 4 deletions(-)
>
> pve-docs:
>
> Christoph Heiss (1):
> qm: document conntrack state migration for live migrations
>
> qm.adoc | 16 ++++++++++++++++
> 1 file changed, 16 insertions(+)
More information about the pve-devel
mailing list