[pve-devel] [PATCH ve-rs/firewall/qemu-server/manager v2 00/13] fix #5180: migrate conntrack state on live migration

[Christoph Heiss]

On Tue Jun 3, 2025 at 11:35 AM CEST, Stefan Hanreich wrote:
> Gave this a spin on my test cluster, and it worked as advertised. As
> already mentioned and talked off-list we should imo add documentation
> for this setting and mention the current restrictions (no migration of
> NAT entries). We might also want to mention that only CT entries created
> *after* the firewall is enabled for a VM get migrated, although that is
> probably rather uncommon.

Thanks for the (pretty extensive, as also talked off-list) testing!

I'll definitely add documentation in the next revision, mentioning
especially all the limitations.

>
> Other than that consider this:
>
> Tested-by: Stefan Hanreich <s.hanreich at proxmox.com>
>
> I've also mainly looked at the firewall + pve-manager code, maybe
> someone with more knowledge about QEMU can chime in on the DBus server
> parts. Consider the firewall and ui patches:
>
> Reviewed-by: Stefan Hanreich <s.hanreich at proxmox.com>
>
>
> On 4/24/25 13:19, Christoph Heiss wrote:
>> Fixes #5180 [0].
>>
>> This implements migration of per-VM conntrack state on live-migration.
>>
>> The core of the implementation are in patch #7 & #8. See there for more
>> details.
>>
>> Patch #1 - #3 implement CONNMARK'ing any VM traffic with their unique
>> VMID. This is needed later on to filter conntrack entries for the
>> migration. These three patches can be applied independently,
>> CONNMARK'ing traffic does not have any visible impact.
>>
>> Currently, remote/inter-cluster migration is not supported and indicated
>> to the user with a warning. See also patch #8 for a bit more in-depth
>> explanation.
>>
>> Needed dependency bumps between packages are indicated in the notes
>> appropriately.
>>
>> [0] https://bugzilla.proxmox.com/show_bug.cgi?id=5180
>>
>> Testing
>> =======
>>
>> I've primarily tested intra-cluster live-migrations, with both the
>> iptables-based and nftables-based firewall), using the reproducer as
>> described in #5180. I further verified that the D-Bus servers get
>> started as expected and are _always_ stopped, even in the case of some
>> migration error.
>>
>> Finally, I also checked using `conntrack -L -m <vmid>` tool that the
>> conntrack entries are
>> a) added/updated on the target node and
>> b) removed from the source node afterwards
>>
>> Also tested was the migration from/to an "old" (unpatched) node, which
>> results in the issue as per #5180 & appropriate warnings in the UI.
>>
>> For remote migrations, only tested that the warning is logged as
>> expected.
>>
>> History
>> =======
>>
>> v1: https://lore.proxmox.com/pve-devel/20250317141152.1247324-1-c.heiss@proxmox.com/
>>
>> Changes v1 -> v2:
>>   * rebased as necessary
>>   * "un-rfc'd" firewall conntrack flushing patches
>>   * use an instanced systemd service instead of fork+exec for the
>>     pve-dbus-vmstate helper
>>
>> Diffstat
>> ========
>>
>> pve-firewall:
>>
>> Christoph Heiss (2):
>>   firewall: add connmark rule with VMID to all guest chains
>>   firewall: helpers: add sub for flushing conntrack entries by mark
>>
>>  debian/control              |  3 ++-
>>  src/PVE/Firewall.pm         |  7 +++++--
>>  src/PVE/Firewall/Helpers.pm | 11 +++++++++++
>>  3 files changed, 18 insertions(+), 3 deletions(-)
>>
>> proxmox-firewall:
>>
>> Christoph Heiss (1):
>>   firewall: add connmark rule with VMID to all guest chains
>>
>>  proxmox-firewall/src/firewall.rs              | 14 +++-
>>  .../integration_tests__firewall.snap          | 84 +++++++++++++++++++
>>  proxmox-nftables/src/expression.rs            |  9 ++
>>  proxmox-nftables/src/statement.rs             | 10 ++-
>>  4 files changed, 114 insertions(+), 3 deletions(-)
>>
>> proxmox-ve-rs:
>>
>> Christoph Heiss (1):
>>   config: guest: allow access to raw Vmid value
>>
>>  proxmox-ve-config/src/guest/types.rs | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> qemu-server:
>>
>> Christoph Heiss (5):
>>   qmp helpers: allow passing structured args via qemu_objectadd()
>>   api2: qemu: add module exposing node migration capabilities
>>   fix #5180: libexec: add QEMU dbus-vmstate daemon for migrating
>>     conntrack
>>   fix #5180: migrate: integrate helper for live-migrating conntrack info
>>   migrate: flush old VM conntrack entries after successful migration
>>
>>  Makefile                               |   7 +-
>>  PVE/API2/Qemu.pm                       |  72 +++++++++++
>>  PVE/API2/Qemu/Makefile                 |   2 +-
>>  PVE/API2/Qemu/Migration.pm             |  46 +++++++
>>  PVE/CLI/qm.pm                          |   5 +
>>  PVE/QemuMigrate.pm                     |  69 ++++++++++
>>  PVE/QemuServer.pm                      |   6 +
>>  PVE/QemuServer/DBusVMState.pm          | 120 ++++++++++++++++++
>>  PVE/QemuServer/Makefile                |   1 +
>>  PVE/QemuServer/QMPHelpers.pm           |   4 +-
>>  dbus-vmstate/Makefile                  |   7 ++
>>  dbus-vmstate/dbus-vmstate              | 168 +++++++++++++++++++++++++
>>  dbus-vmstate/org.qemu.VMState1.conf    |  11 ++
>>  dbus-vmstate/pve-dbus-vmstate at .service |  10 ++
>>  debian/control                         |   7 +-
>>  15 files changed, 530 insertions(+), 5 deletions(-)
>>  create mode 100644 PVE/API2/Qemu/Migration.pm
>>  create mode 100644 PVE/QemuServer/DBusVMState.pm
>>  create mode 100644 dbus-vmstate/Makefile
>>  create mode 100755 dbus-vmstate/dbus-vmstate
>>  create mode 100644 dbus-vmstate/org.qemu.VMState1.conf
>>  create mode 100644 dbus-vmstate/pve-dbus-vmstate at .service
>>
>> pve-manager:
>>
>> Christoph Heiss (4):
>>   api2: capabilities: explicitly import CPU capabilities module
>>   api2: capabilities: proxy index endpoints to respective nodes
>>   api2: capabilities: expose new qemu/migration endpoint
>>   ui: window: Migrate: add checkbox for migrating VM conntrack state
>>
>>  PVE/API2/Capabilities.pm       |  9 +++++
>>  www/manager6/window/Migrate.js | 73 ++++++++++++++++++++++++++++++++--
>>  2 files changed, 78 insertions(+), 4 deletions(-)
>>