[pve-devel] [PATCH docs/ha-manager/manager v5 00/29] HA Rules

Daniel Kral d.kral at proxmox.com
Wed Jul 30 19:59:27 CEST 2025


Here's a (hopefully) final update on the core HA rules series.

@Michael and @Hannes Duerr have been a great help for testing this
series, thank you very much!

For maintainers: ha-manager patch #19 should be updated to the correct
pve-manager version that is dependent on the pve-ha-manager package
which can interpret the HA rules config.

Changelog since v4
------------------

- rebased on newest available master

- postponed persistent rules migration only if all ha groups migration
  checks passed

- also check the lrm status of all nodes as additional ha groups
  migration checks

- do not allow modifying ha rules api calls unless the ha groups have
  been fully migrated

- do not allow reading/modifying ha group api calls after the ha groups
  have been fully migrated (after thought: we could be more loose on the
  reading api endpoints?)

- do not allow 'group' parameter for ha resources api calls after the ha
  groups have been fully migrated

- the migrated ha groups' comment is not overwritten anymore

- fixed the ha-manager CLI for 'ha-manager rules config' to display an
  overview of the ha rules on the CLI too

ha-manager:

Daniel Kral (23):
  tree-wide: make arguments for select_service_node explicit
  manager: improve signature of select_service_node
  introduce rules base plugin
  rules: introduce node affinity rule plugin
  config, env, hw: add rules read and parse methods
  config: delete services from rules if services are deleted from config
  manager: read and update rules config
  test: ha tester: add test cases for future node affinity rules
  resources: introduce failback property in ha resource config
  manager: migrate ha groups to node affinity rules in-memory
  manager: apply node affinity rules when selecting service nodes
  test: add test cases for rules config
  api: introduce ha rules api endpoints
  cli: expose ha rules api endpoints to ha-manager cli
  sim: do not create default groups config
  test: ha tester: migrate groups to service and rules config
  test: ha tester: replace any reference to groups with node affinity
    rules
  env: add property delete for update_service_config
  manager: persistently migrate ha groups to ha rules
  api: groups: disallow calls to ha groups endpoints if fully migrated
  api: resources: exclude group property in reading endpoints if
    migrated
  api: resources: disallow group prop in modifying endpoints if migrated
  api: rules: disallow modifying api calls if ha groups not migrated

 .gitignore                                    |   1 +
 debian/pve-ha-manager.install                 |   3 +
 src/PVE/API2/HA/Groups.pm                     |  25 +-
 src/PVE/API2/HA/Makefile                      |   2 +-
 src/PVE/API2/HA/Resources.pm                  |  24 +-
 src/PVE/API2/HA/Rules.pm                      | 394 +++++++++++++++
 src/PVE/API2/HA/Status.pm                     |  11 +-
 src/PVE/CLI/ha_manager.pm                     |  36 ++
 src/PVE/HA/Config.pm                          |  67 ++-
 src/PVE/HA/Env.pm                             |  34 +-
 src/PVE/HA/Env/PVE2.pm                        |  45 +-
 src/PVE/HA/Groups.pm                          |  49 ++
 src/PVE/HA/Makefile                           |   3 +-
 src/PVE/HA/Manager.pm                         | 321 ++++++++----
 src/PVE/HA/Resources.pm                       |   9 +
 src/PVE/HA/Resources/PVECT.pm                 |   1 +
 src/PVE/HA/Resources/PVEVM.pm                 |   1 +
 src/PVE/HA/Rules.pm                           | 455 ++++++++++++++++++
 src/PVE/HA/Rules/Makefile                     |   6 +
 src/PVE/HA/Rules/NodeAffinity.pm              | 296 ++++++++++++
 src/PVE/HA/Sim/Env.pm                         |  48 +-
 src/PVE/HA/Sim/Hardware.pm                    |  70 ++-
 src/PVE/HA/Tools.pm                           |  46 ++
 src/test/Makefile                             |   4 +-
 .../defaults-for-node-affinity-rules.cfg      |  22 +
 ...efaults-for-node-affinity-rules.cfg.expect |  60 +++
 ...e-resource-refs-in-node-affinity-rules.cfg |  31 ++
 ...rce-refs-in-node-affinity-rules.cfg.expect |  63 +++
 src/test/test-basic5/groups                   |   2 -
 src/test/test-basic5/rules_config             |   3 +
 src/test/test-basic5/service_config           |   2 +-
 src/test/test-crs-static2/README              |   3 +-
 src/test/test-crs-static2/groups              |   2 -
 src/test/test-crs-static2/rules_config        |   3 +
 src/test/test-crs-static2/service_config      |   2 +-
 src/test/test-group-migrate1/README           |   4 +
 src/test/test-group-migrate1/cmdlist          |   3 +
 src/test/test-group-migrate1/groups           |   7 +
 src/test/test-group-migrate1/hardware_status  |   5 +
 src/test/test-group-migrate1/log.expect       | 101 ++++
 src/test/test-group-migrate1/manager_status   |   1 +
 src/test/test-group-migrate1/service_config   |   5 +
 src/test/test-group-migrate2/README           |   3 +
 src/test/test-group-migrate2/cmdlist          |   3 +
 src/test/test-group-migrate2/groups           |   7 +
 src/test/test-group-migrate2/hardware_status  |   5 +
 src/test/test-group-migrate2/log.expect       |  50 ++
 src/test/test-group-migrate2/manager_status   |   1 +
 src/test/test-group-migrate2/service_config   |   5 +
 src/test/test-node-affinity-nonstrict1/README |  11 +
 .../test-node-affinity-nonstrict1/cmdlist     |   4 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-nonstrict1/log.expect  |  40 ++
 .../manager_status                            |   1 +
 .../rules_config                              |   3 +
 .../service_config                            |   3 +
 src/test/test-node-affinity-nonstrict2/README |  12 +
 .../test-node-affinity-nonstrict2/cmdlist     |   4 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-nonstrict2/log.expect  |  35 ++
 .../manager_status                            |   1 +
 .../rules_config                              |   3 +
 .../service_config                            |   3 +
 src/test/test-node-affinity-nonstrict3/README |  10 +
 .../test-node-affinity-nonstrict3/cmdlist     |   4 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-nonstrict3/log.expect  |  56 +++
 .../manager_status                            |   1 +
 .../rules_config                              |   3 +
 .../service_config                            |   5 +
 src/test/test-node-affinity-nonstrict4/README |  14 +
 .../test-node-affinity-nonstrict4/cmdlist     |   4 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-nonstrict4/log.expect  |  54 +++
 .../manager_status                            |   1 +
 .../rules_config                              |   3 +
 .../service_config                            |   5 +
 src/test/test-node-affinity-nonstrict5/README |  16 +
 .../test-node-affinity-nonstrict5/cmdlist     |   5 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-nonstrict5/log.expect  |  66 +++
 .../manager_status                            |   1 +
 .../rules_config                              |   3 +
 .../service_config                            |   3 +
 src/test/test-node-affinity-nonstrict6/README |  14 +
 .../test-node-affinity-nonstrict6/cmdlist     |   5 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-nonstrict6/log.expect  |  52 ++
 .../manager_status                            |   1 +
 .../rules_config                              |   3 +
 .../service_config                            |   3 +
 src/test/test-node-affinity-strict1/README    |  11 +
 src/test/test-node-affinity-strict1/cmdlist   |   4 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-strict1/log.expect     |  40 ++
 .../test-node-affinity-strict1/manager_status |   1 +
 .../test-node-affinity-strict1/rules_config   |   4 +
 .../test-node-affinity-strict1/service_config |   3 +
 src/test/test-node-affinity-strict2/README    |  11 +
 src/test/test-node-affinity-strict2/cmdlist   |   4 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-strict2/log.expect     |  40 ++
 .../test-node-affinity-strict2/manager_status |   1 +
 .../test-node-affinity-strict2/rules_config   |   4 +
 .../test-node-affinity-strict2/service_config |   3 +
 src/test/test-node-affinity-strict3/README    |  10 +
 src/test/test-node-affinity-strict3/cmdlist   |   4 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-strict3/log.expect     |  74 +++
 .../test-node-affinity-strict3/manager_status |   1 +
 .../test-node-affinity-strict3/rules_config   |   4 +
 .../test-node-affinity-strict3/service_config |   5 +
 src/test/test-node-affinity-strict4/README    |  14 +
 src/test/test-node-affinity-strict4/cmdlist   |   4 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-strict4/log.expect     |  54 +++
 .../test-node-affinity-strict4/manager_status |   1 +
 .../test-node-affinity-strict4/rules_config   |   4 +
 .../test-node-affinity-strict4/service_config |   5 +
 src/test/test-node-affinity-strict5/README    |  16 +
 src/test/test-node-affinity-strict5/cmdlist   |   5 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-strict5/log.expect     |  66 +++
 .../test-node-affinity-strict5/manager_status |   1 +
 .../test-node-affinity-strict5/rules_config   |   4 +
 .../test-node-affinity-strict5/service_config |   3 +
 src/test/test-node-affinity-strict6/README    |  15 +
 src/test/test-node-affinity-strict6/cmdlist   |   5 +
 .../hardware_status                           |   5 +
 .../test-node-affinity-strict6/log.expect     |  52 ++
 .../test-node-affinity-strict6/manager_status |   1 +
 .../test-node-affinity-strict6/rules_config   |   4 +
 .../test-node-affinity-strict6/service_config |   3 +
 src/test/test-recovery1/groups                |   4 -
 src/test/test-recovery1/rules_config          |   4 +
 src/test/test-recovery1/service_config        |   2 +-
 src/test/test-recovery2/README                |   4 +-
 src/test/test-recovery2/groups                |   4 -
 src/test/test-recovery2/rules_config          |   4 +
 src/test/test-recovery2/service_config        |   2 +-
 src/test/test-recovery3/groups                |   4 -
 src/test/test-recovery3/rules_config          |   4 +
 src/test/test-recovery3/service_config        |   2 +-
 src/test/test-recovery4/groups                |   4 -
 src/test/test-recovery4/rules_config          |   4 +
 src/test/test-recovery4/service_config        |   2 +-
 .../test-relocate-policy-default-group/README |   6 +-
 src/test/test-resource-failure2/groups        |   2 -
 src/test/test-resource-failure2/rules_config  |   3 +
 .../test-resource-failure2/service_config     |   2 +-
 .../test-resource-failure3/service_config     |   2 +-
 src/test/test-resource-failure6/README        |   6 +-
 src/test/test-shutdown2/groups                |   2 -
 src/test/test-shutdown2/rules_config          |   3 +
 src/test/test-shutdown2/service_config        |   4 +-
 src/test/test-shutdown3/groups                |   2 -
 src/test/test-shutdown3/rules_config          |   3 +
 src/test/test-shutdown3/service_config        |   4 +-
 src/test/test_failover1.pl                    |  27 +-
 src/test/test_rules_config.pl                 | 100 ++++
 160 files changed, 3374 insertions(+), 173 deletions(-)
 create mode 100644 src/PVE/API2/HA/Rules.pm
 create mode 100644 src/PVE/HA/Rules.pm
 create mode 100644 src/PVE/HA/Rules/Makefile
 create mode 100644 src/PVE/HA/Rules/NodeAffinity.pm
 create mode 100644 src/test/rules_cfgs/defaults-for-node-affinity-rules.cfg
 create mode 100644 src/test/rules_cfgs/defaults-for-node-affinity-rules.cfg.expect
 create mode 100644 src/test/rules_cfgs/multiple-resource-refs-in-node-affinity-rules.cfg
 create mode 100644 src/test/rules_cfgs/multiple-resource-refs-in-node-affinity-rules.cfg.expect
 delete mode 100644 src/test/test-basic5/groups
 create mode 100644 src/test/test-basic5/rules_config
 delete mode 100644 src/test/test-crs-static2/groups
 create mode 100644 src/test/test-crs-static2/rules_config
 create mode 100644 src/test/test-group-migrate1/README
 create mode 100644 src/test/test-group-migrate1/cmdlist
 create mode 100644 src/test/test-group-migrate1/groups
 create mode 100644 src/test/test-group-migrate1/hardware_status
 create mode 100644 src/test/test-group-migrate1/log.expect
 create mode 100644 src/test/test-group-migrate1/manager_status
 create mode 100644 src/test/test-group-migrate1/service_config
 create mode 100644 src/test/test-group-migrate2/README
 create mode 100644 src/test/test-group-migrate2/cmdlist
 create mode 100644 src/test/test-group-migrate2/groups
 create mode 100644 src/test/test-group-migrate2/hardware_status
 create mode 100644 src/test/test-group-migrate2/log.expect
 create mode 100644 src/test/test-group-migrate2/manager_status
 create mode 100644 src/test/test-group-migrate2/service_config
 create mode 100644 src/test/test-node-affinity-nonstrict1/README
 create mode 100644 src/test/test-node-affinity-nonstrict1/cmdlist
 create mode 100644 src/test/test-node-affinity-nonstrict1/hardware_status
 create mode 100644 src/test/test-node-affinity-nonstrict1/log.expect
 create mode 100644 src/test/test-node-affinity-nonstrict1/manager_status
 create mode 100644 src/test/test-node-affinity-nonstrict1/rules_config
 create mode 100644 src/test/test-node-affinity-nonstrict1/service_config
 create mode 100644 src/test/test-node-affinity-nonstrict2/README
 create mode 100644 src/test/test-node-affinity-nonstrict2/cmdlist
 create mode 100644 src/test/test-node-affinity-nonstrict2/hardware_status
 create mode 100644 src/test/test-node-affinity-nonstrict2/log.expect
 create mode 100644 src/test/test-node-affinity-nonstrict2/manager_status
 create mode 100644 src/test/test-node-affinity-nonstrict2/rules_config
 create mode 100644 src/test/test-node-affinity-nonstrict2/service_config
 create mode 100644 src/test/test-node-affinity-nonstrict3/README
 create mode 100644 src/test/test-node-affinity-nonstrict3/cmdlist
 create mode 100644 src/test/test-node-affinity-nonstrict3/hardware_status
 create mode 100644 src/test/test-node-affinity-nonstrict3/log.expect
 create mode 100644 src/test/test-node-affinity-nonstrict3/manager_status
 create mode 100644 src/test/test-node-affinity-nonstrict3/rules_config
 create mode 100644 src/test/test-node-affinity-nonstrict3/service_config
 create mode 100644 src/test/test-node-affinity-nonstrict4/README
 create mode 100644 src/test/test-node-affinity-nonstrict4/cmdlist
 create mode 100644 src/test/test-node-affinity-nonstrict4/hardware_status
 create mode 100644 src/test/test-node-affinity-nonstrict4/log.expect
 create mode 100644 src/test/test-node-affinity-nonstrict4/manager_status
 create mode 100644 src/test/test-node-affinity-nonstrict4/rules_config
 create mode 100644 src/test/test-node-affinity-nonstrict4/service_config
 create mode 100644 src/test/test-node-affinity-nonstrict5/README
 create mode 100644 src/test/test-node-affinity-nonstrict5/cmdlist
 create mode 100644 src/test/test-node-affinity-nonstrict5/hardware_status
 create mode 100644 src/test/test-node-affinity-nonstrict5/log.expect
 create mode 100644 src/test/test-node-affinity-nonstrict5/manager_status
 create mode 100644 src/test/test-node-affinity-nonstrict5/rules_config
 create mode 100644 src/test/test-node-affinity-nonstrict5/service_config
 create mode 100644 src/test/test-node-affinity-nonstrict6/README
 create mode 100644 src/test/test-node-affinity-nonstrict6/cmdlist
 create mode 100644 src/test/test-node-affinity-nonstrict6/hardware_status
 create mode 100644 src/test/test-node-affinity-nonstrict6/log.expect
 create mode 100644 src/test/test-node-affinity-nonstrict6/manager_status
 create mode 100644 src/test/test-node-affinity-nonstrict6/rules_config
 create mode 100644 src/test/test-node-affinity-nonstrict6/service_config
 create mode 100644 src/test/test-node-affinity-strict1/README
 create mode 100644 src/test/test-node-affinity-strict1/cmdlist
 create mode 100644 src/test/test-node-affinity-strict1/hardware_status
 create mode 100644 src/test/test-node-affinity-strict1/log.expect
 create mode 100644 src/test/test-node-affinity-strict1/manager_status
 create mode 100644 src/test/test-node-affinity-strict1/rules_config
 create mode 100644 src/test/test-node-affinity-strict1/service_config
 create mode 100644 src/test/test-node-affinity-strict2/README
 create mode 100644 src/test/test-node-affinity-strict2/cmdlist
 create mode 100644 src/test/test-node-affinity-strict2/hardware_status
 create mode 100644 src/test/test-node-affinity-strict2/log.expect
 create mode 100644 src/test/test-node-affinity-strict2/manager_status
 create mode 100644 src/test/test-node-affinity-strict2/rules_config
 create mode 100644 src/test/test-node-affinity-strict2/service_config
 create mode 100644 src/test/test-node-affinity-strict3/README
 create mode 100644 src/test/test-node-affinity-strict3/cmdlist
 create mode 100644 src/test/test-node-affinity-strict3/hardware_status
 create mode 100644 src/test/test-node-affinity-strict3/log.expect
 create mode 100644 src/test/test-node-affinity-strict3/manager_status
 create mode 100644 src/test/test-node-affinity-strict3/rules_config
 create mode 100644 src/test/test-node-affinity-strict3/service_config
 create mode 100644 src/test/test-node-affinity-strict4/README
 create mode 100644 src/test/test-node-affinity-strict4/cmdlist
 create mode 100644 src/test/test-node-affinity-strict4/hardware_status
 create mode 100644 src/test/test-node-affinity-strict4/log.expect
 create mode 100644 src/test/test-node-affinity-strict4/manager_status
 create mode 100644 src/test/test-node-affinity-strict4/rules_config
 create mode 100644 src/test/test-node-affinity-strict4/service_config
 create mode 100644 src/test/test-node-affinity-strict5/README
 create mode 100644 src/test/test-node-affinity-strict5/cmdlist
 create mode 100644 src/test/test-node-affinity-strict5/hardware_status
 create mode 100644 src/test/test-node-affinity-strict5/log.expect
 create mode 100644 src/test/test-node-affinity-strict5/manager_status
 create mode 100644 src/test/test-node-affinity-strict5/rules_config
 create mode 100644 src/test/test-node-affinity-strict5/service_config
 create mode 100644 src/test/test-node-affinity-strict6/README
 create mode 100644 src/test/test-node-affinity-strict6/cmdlist
 create mode 100644 src/test/test-node-affinity-strict6/hardware_status
 create mode 100644 src/test/test-node-affinity-strict6/log.expect
 create mode 100644 src/test/test-node-affinity-strict6/manager_status
 create mode 100644 src/test/test-node-affinity-strict6/rules_config
 create mode 100644 src/test/test-node-affinity-strict6/service_config
 delete mode 100644 src/test/test-recovery1/groups
 create mode 100644 src/test/test-recovery1/rules_config
 delete mode 100644 src/test/test-recovery2/groups
 create mode 100644 src/test/test-recovery2/rules_config
 delete mode 100644 src/test/test-recovery3/groups
 create mode 100644 src/test/test-recovery3/rules_config
 delete mode 100644 src/test/test-recovery4/groups
 create mode 100644 src/test/test-recovery4/rules_config
 delete mode 100644 src/test/test-resource-failure2/groups
 create mode 100644 src/test/test-resource-failure2/rules_config
 delete mode 100644 src/test/test-shutdown2/groups
 create mode 100644 src/test/test-shutdown2/rules_config
 delete mode 100644 src/test/test-shutdown3/groups
 create mode 100644 src/test/test-shutdown3/rules_config
 create mode 100755 src/test/test_rules_config.pl


docs:

Daniel Kral (2):
  ha: add documentation about ha rules and ha node affinity rules
  ha: crs: add effects of ha node affinity rule on the crs scheduler

 Makefile                           |   2 +
 gen-ha-rules-node-affinity-opts.pl |  20 +++++
 gen-ha-rules-opts.pl               |  17 +++++
 ha-manager.adoc                    | 113 +++++++++++++++++++++++++++++
 ha-rules-node-affinity-opts.adoc   |  18 +++++
 ha-rules-opts.adoc                 |  12 +++
 pmxcfs.adoc                        |   1 +
 7 files changed, 183 insertions(+)
 create mode 100755 gen-ha-rules-node-affinity-opts.pl
 create mode 100755 gen-ha-rules-opts.pl
 create mode 100644 ha-rules-node-affinity-opts.adoc
 create mode 100644 ha-rules-opts.adoc


manager:

Daniel Kral (4):
  api: ha: add ha rules api endpoints
  ui: ha: remove ha groups from ha resource components
  ui: ha: show failback flag in resources status view
  ui: ha: replace ha groups with ha node affinity rules

 PVE/API2/HAConfig.pm                          |   8 +-
 www/manager6/Makefile                         |   8 +-
 www/manager6/StateProvider.js                 |   2 +-
 www/manager6/dc/Config.js                     |   8 +-
 www/manager6/ha/GroupSelector.js              |  71 -------
 www/manager6/ha/Groups.js                     | 117 -----------
 www/manager6/ha/ResourceEdit.js               |  16 +-
 www/manager6/ha/Resources.js                  |  17 +-
 www/manager6/ha/RuleEdit.js                   | 146 +++++++++++++
 www/manager6/ha/RuleErrorsModal.js            |  50 +++++
 www/manager6/ha/Rules.js                      | 196 ++++++++++++++++++
 www/manager6/ha/StatusView.js                 |   5 +-
 .../NodeAffinityRuleEdit.js}                  | 105 ++--------
 www/manager6/ha/rules/NodeAffinityRules.js    |  36 ++++
 14 files changed, 481 insertions(+), 304 deletions(-)
 delete mode 100644 www/manager6/ha/GroupSelector.js
 delete mode 100644 www/manager6/ha/Groups.js
 create mode 100644 www/manager6/ha/RuleEdit.js
 create mode 100644 www/manager6/ha/RuleErrorsModal.js
 create mode 100644 www/manager6/ha/Rules.js
 rename www/manager6/ha/{GroupEdit.js => rules/NodeAffinityRuleEdit.js} (67%)
 create mode 100644 www/manager6/ha/rules/NodeAffinityRules.js


Summary over all repositories:
  181 files changed, 4038 insertions(+), 477 deletions(-)

-- 
Generated by git-murpp 0.8.0




More information about the pve-devel mailing list