[pve-devel] [PATCH ha-manager v2 00/12] HA rules follow up (part 1)

Daniel Kral d.kral at proxmox.com
Fri Aug 1 18:22:15 CEST 2025


Here's a follow up on the HA rules and especially the HA resource
affinity rules.

The first three patches haven't changed as they were lower priority for
me than the last part about loosening restrictions on mixed resource
references.

Patches #1 - #8 are rather independent patches but still have some
ordering / dependencies in between (e.g. patch #1 - #3 depend on each
other). The larger part here are patches #9 - #12, which allow mixed
resource references between node and resource affinity rules.

Also did a

    git rebase HEAD~12 --exec 'make clean && make deb'

on the series.

Changelog to v1
---------------

- add missing return schema to read_rule api

- ignore disable parameter as otherwise rule feasibility tests will be
  skipped

- do not check ignored resources as otherwise migrations are not
  possible or co-migrations are done even if the resource is not
  ha-managed (could be an use case in the future, but not for now)

- change behavior for single resource fails for resources in positive
  affinity rules: try to recover resource by migrating all resources to
  another available node

- loosen restrictions on mixed resource references, i.e., if a resource
  is used in a node affinity rule and a resource affinity rule at the
  same time (see more information in the patch)

TODO (more important)
---------------------

- more test cases and dev/user testing

- do not include ignored resources in feasibility check

- updates to pve-docs about the new behavior and missing docs:

  - new: mixing node and resource affinity rules are allowed now

  - new: interactions between node and resource affinity rules

  - new: ha rule conflicts for mixed usage

  - missing: for positive resource affinity rules, if resources are
    separated across nodes, the most populated one will be chosen as the
    one where all are migrated to

  - maybe missing: strictness behavior of negative resource affinity
    rules, i.e., that one resource cannot be migrated to either where
    another negative affinity resource is located nor migrated to

- using node affinity rules and negative resource affinity rules should
  still be improved as there are quite a few cases where manual
  intervention is needed to follow the resources node affinity rules
  (for now, resource affinity rules often times overwrite node affinity
  rules; the former should get more information about the latter in the
  scheduler; but might not be a blocker as its rather an edge case)

  - some of these cases are also represented in the added test cases, so
    that changes to them are documented in the future if those cases are
    accomodated for as well

TODO (less important)
---------------------

- adding node affinity rule blockers as well (but not really important)

- maybe log not-migrated groups because they had no group members?

- other suggestions to the original patch series'

- cleaning up


Daniel Kral (12):
  manager: fix ~revision version check for ha groups migration
  test: ha tester: add ha groups migration tests with runtime upgrades
  tree-wide: pass optional parameters as hash values for for_each_rule
    helper
  api: rules: add missing return schema for the read_rule api endpoint
  api: rules: ignore disable parameter if it is set to a falsy value
  rules: resource affinity: make message in inter-consistency check
    clearer
  config, manager: do not check ignored resources with affinity when
    migrating
  rules: make positive affinity resources migrate on single resource
    fail
  rules: allow same resources in node and resource affinity rules
  rules: restrict inter-plugin resource references to simple cases
  test: rules: add test cases for inter-plugin checks allowing simple
    use cases
  test: ha tester: add resource affinity test cases mixed with node
    affinity rules

 src/PVE/API2/HA/Rules.pm                      |  12 +-
 src/PVE/HA/Config.pm                          |   2 +
 src/PVE/HA/Manager.pm                         |  11 +-
 src/PVE/HA/Rules.pm                           | 289 +++++++++++++--
 src/PVE/HA/Rules/NodeAffinity.pm              |  14 +-
 src/PVE/HA/Rules/ResourceAffinity.pm          |  30 +-
 src/PVE/HA/Sim/Hardware.pm                    |   8 +
 ...onsistent-node-resource-affinity-rules.cfg |  54 +++
 ...nt-node-resource-affinity-rules.cfg.expect |  73 ++++
 ...sistent-resource-affinity-rules.cfg.expect |   8 +-
 ...egative-resource-affinity-rules.cfg.expect |   4 +-
 ...y-for-positive-resource-affinity-rules.cfg |  37 ++
 ...ositive-resource-affinity-rules.cfg.expect | 111 ++++++
 ...-affinity-with-resource-affinity-rules.cfg |  35 ++
 ...ty-with-resource-affinity-rules.cfg.expect |  48 +++
 .../multiple-resource-refs-in-rules.cfg       |  52 ---
 ...multiple-resource-refs-in-rules.cfg.expect | 111 ------
 src/test/test-group-migrate3/README           |   7 +
 src/test/test-group-migrate3/cmdlist          |  17 +
 src/test/test-group-migrate3/groups           |   7 +
 src/test/test-group-migrate3/hardware_status  |   5 +
 src/test/test-group-migrate3/log.expect       | 344 ++++++++++++++++++
 src/test/test-group-migrate3/manager_status   |   1 +
 src/test/test-group-migrate3/service_config   |   5 +
 src/test/test-group-migrate4/README           |   8 +
 src/test/test-group-migrate4/cmdlist          |  15 +
 src/test/test-group-migrate4/groups           |   7 +
 src/test/test-group-migrate4/hardware_status  |   5 +
 src/test/test-group-migrate4/log.expect       | 277 ++++++++++++++
 src/test/test-group-migrate4/manager_status   |   1 +
 src/test/test-group-migrate4/service_config   |   5 +
 .../README                                    |   9 +-
 .../log.expect                                |  26 +-
 .../README                                    |  14 +
 .../cmdlist                                   |   3 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  41 +++
 .../manager_status                            |   1 +
 .../rules_config                              |   7 +
 .../service_config                            |   5 +
 .../README                                    |  13 +
 .../cmdlist                                   |   3 +
 .../hardware_status                           |   7 +
 .../log.expect                                |  63 ++++
 .../manager_status                            |   1 +
 .../rules_config                              |   7 +
 .../service_config                            |   5 +
 .../README                                    |  17 +
 .../cmdlist                                   |   6 +
 .../hardware_status                           |   6 +
 .../log.expect                                |  67 ++++
 .../manager_status                            |   1 +
 .../rules_config                              |  15 +
 .../service_config                            |   5 +
 .../README                                    |  15 +
 .../cmdlist                                   |   3 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  49 +++
 .../manager_status                            |   1 +
 .../rules_config                              |   7 +
 .../service_config                            |   5 +
 .../README                                    |  15 +
 .../cmdlist                                   |   3 +
 .../hardware_status                           |   5 +
 .../log.expect                                |  49 +++
 .../manager_status                            |   1 +
 .../rules_config                              |   7 +
 .../service_config                            |   5 +
 68 files changed, 1852 insertions(+), 248 deletions(-)
 create mode 100644 src/test/rules_cfgs/inconsistent-node-resource-affinity-rules.cfg
 create mode 100644 src/test/rules_cfgs/inconsistent-node-resource-affinity-rules.cfg.expect
 create mode 100644 src/test/rules_cfgs/infer-node-affinity-for-positive-resource-affinity-rules.cfg
 create mode 100644 src/test/rules_cfgs/infer-node-affinity-for-positive-resource-affinity-rules.cfg.expect
 create mode 100644 src/test/rules_cfgs/multi-priority-node-affinity-with-resource-affinity-rules.cfg
 create mode 100644 src/test/rules_cfgs/multi-priority-node-affinity-with-resource-affinity-rules.cfg.expect
 delete mode 100644 src/test/rules_cfgs/multiple-resource-refs-in-rules.cfg
 delete mode 100644 src/test/rules_cfgs/multiple-resource-refs-in-rules.cfg.expect
 create mode 100644 src/test/test-group-migrate3/README
 create mode 100644 src/test/test-group-migrate3/cmdlist
 create mode 100644 src/test/test-group-migrate3/groups
 create mode 100644 src/test/test-group-migrate3/hardware_status
 create mode 100644 src/test/test-group-migrate3/log.expect
 create mode 100644 src/test/test-group-migrate3/manager_status
 create mode 100644 src/test/test-group-migrate3/service_config
 create mode 100644 src/test/test-group-migrate4/README
 create mode 100644 src/test/test-group-migrate4/cmdlist
 create mode 100644 src/test/test-group-migrate4/groups
 create mode 100644 src/test/test-group-migrate4/hardware_status
 create mode 100644 src/test/test-group-migrate4/log.expect
 create mode 100644 src/test/test-group-migrate4/manager_status
 create mode 100644 src/test/test-group-migrate4/service_config
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative1/README
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative1/cmdlist
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative1/hardware_status
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative1/log.expect
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative1/manager_status
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative1/rules_config
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative1/service_config
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative2/README
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative2/cmdlist
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative2/hardware_status
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative2/log.expect
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative2/manager_status
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative2/rules_config
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative2/service_config
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative3/README
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative3/cmdlist
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative3/hardware_status
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative3/log.expect
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative3/manager_status
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative3/rules_config
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-negative3/service_config
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive1/README
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive1/cmdlist
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive1/hardware_status
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive1/log.expect
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive1/manager_status
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive1/rules_config
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive1/service_config
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive2/README
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive2/cmdlist
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive2/hardware_status
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive2/log.expect
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive2/manager_status
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive2/rules_config
 create mode 100644 src/test/test-resource-affinity-with-node-affinity-strict-positive2/service_config

-- 
2.47.2





More information about the pve-devel mailing list