[pve-devel] [PATCH docs/ha-manager/manager v4 00/25] HA Rules
Daniel Kral
d.kral at proxmox.com
Tue Jul 29 20:00:40 CEST 2025
Here's a quick update on the core HA rules series. This cleans up the
series so that all tests are running again and includes the missing ui
patch that I didn't see missing last time.
The persistent migration path has been tested for at least four full
upgrade runs now, always with one node being behind and checking that
the group config is only removed as soon as all nodes are on the right
version.
I'll wait for tomorrow if something comes up and will do some testing
myself, so I'm anticipating to follow up on this tomorrow. I'll also
want to get a more mature version of the HA resource affinity series
ready for tomorrow on the mailing list.
For maintainers: ha-manager patch #19 should be updated to the correct
pve-manager version that is dependent on the pve-ha-manager package
which can interpret the HA rules config.
Changelog since v3
------------------
- rebased on newest available master
- included missing ui patch for web interface
- correction in failback property description (does not influence the ha
node affinity rules)
- migrated the groups configs in the test cases to node affinity rules
in rules configs (except two test cases for the persistent migration)
- improved persistent ha group migration process
- try a persistent upgrade only every 10 HA manager rounds
- various other minor touches
TODO
----
- More testing on edge cases for the HA Manager migration path
- Some more testing of the ha-manager CLI and adding a deprecation
warning on the HA Groups API and disallowing requests as soon as the
groups config is fully migrated
ha-manager:
Daniel Kral (19):
tree-wide: make arguments for select_service_node explicit
manager: improve signature of select_service_node
introduce rules base plugin
rules: introduce node affinity rule plugin
config, env, hw: add rules read and parse methods
config: delete services from rules if services are deleted from config
manager: read and update rules config
test: ha tester: add test cases for future node affinity rules
resources: introduce failback property in ha resource config
manager: migrate ha groups to node affinity rules in-memory
manager: apply node affinity rules when selecting service nodes
test: add test cases for rules config
api: introduce ha rules api endpoints
cli: expose ha rules api endpoints to ha-manager cli
sim: do not create default groups for test cases
test: ha tester: migrate groups to service and rules config
test: ha tester: replace any reference to groups with node affinity
rules
env: add property delete for update_service_config
manager: persistently migrate ha groups to ha rules
.gitignore | 1 +
debian/pve-ha-manager.install | 3 +
src/PVE/API2/HA/Makefile | 2 +-
src/PVE/API2/HA/Resources.pm | 9 +
src/PVE/API2/HA/Rules.pm | 391 +++++++++++++++
src/PVE/API2/HA/Status.pm | 11 +-
src/PVE/CLI/ha_manager.pm | 32 ++
src/PVE/HA/Config.pm | 58 ++-
src/PVE/HA/Env.pm | 34 +-
src/PVE/HA/Env/PVE2.pm | 45 +-
src/PVE/HA/Groups.pm | 47 ++
src/PVE/HA/Makefile | 3 +-
src/PVE/HA/Manager.pm | 279 +++++++----
src/PVE/HA/Resources.pm | 9 +
src/PVE/HA/Resources/PVECT.pm | 1 +
src/PVE/HA/Resources/PVEVM.pm | 1 +
src/PVE/HA/Rules.pm | 455 ++++++++++++++++++
src/PVE/HA/Rules/Makefile | 6 +
src/PVE/HA/Rules/NodeAffinity.pm | 296 ++++++++++++
src/PVE/HA/Sim/Env.pm | 48 +-
src/PVE/HA/Sim/Hardware.pm | 70 ++-
src/PVE/HA/Sim/RTHardware.pm | 18 +
src/PVE/HA/Tools.pm | 46 ++
src/test/Makefile | 4 +-
.../defaults-for-node-affinity-rules.cfg | 22 +
...efaults-for-node-affinity-rules.cfg.expect | 60 +++
...e-resource-refs-in-node-affinity-rules.cfg | 31 ++
...rce-refs-in-node-affinity-rules.cfg.expect | 63 +++
src/test/test-basic5/groups | 2 -
src/test/test-basic5/rules_config | 3 +
src/test/test-basic5/service_config | 2 +-
src/test/test-crs-static2/README | 3 +-
src/test/test-crs-static2/groups | 2 -
src/test/test-crs-static2/rules_config | 3 +
src/test/test-crs-static2/service_config | 2 +-
src/test/test-group-migrate1/README | 4 +
src/test/test-group-migrate1/cmdlist | 4 +
src/test/test-group-migrate1/groups | 7 +
src/test/test-group-migrate1/hardware_status | 5 +
src/test/test-group-migrate1/log.expect | 87 ++++
src/test/test-group-migrate1/manager_status | 1 +
src/test/test-group-migrate1/service_config | 5 +
src/test/test-group-migrate2/README | 3 +
src/test/test-group-migrate2/cmdlist | 3 +
src/test/test-group-migrate2/groups | 7 +
src/test/test-group-migrate2/hardware_status | 5 +
src/test/test-group-migrate2/log.expect | 47 ++
src/test/test-group-migrate2/manager_status | 1 +
src/test/test-group-migrate2/service_config | 5 +
src/test/test-node-affinity-nonstrict1/README | 11 +
.../test-node-affinity-nonstrict1/cmdlist | 4 +
.../hardware_status | 5 +
.../test-node-affinity-nonstrict1/log.expect | 40 ++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 3 +
src/test/test-node-affinity-nonstrict2/README | 12 +
.../test-node-affinity-nonstrict2/cmdlist | 4 +
.../hardware_status | 5 +
.../test-node-affinity-nonstrict2/log.expect | 35 ++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 3 +
src/test/test-node-affinity-nonstrict3/README | 10 +
.../test-node-affinity-nonstrict3/cmdlist | 4 +
.../hardware_status | 5 +
.../test-node-affinity-nonstrict3/log.expect | 56 +++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 5 +
src/test/test-node-affinity-nonstrict4/README | 14 +
.../test-node-affinity-nonstrict4/cmdlist | 4 +
.../hardware_status | 5 +
.../test-node-affinity-nonstrict4/log.expect | 54 +++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 5 +
src/test/test-node-affinity-nonstrict5/README | 16 +
.../test-node-affinity-nonstrict5/cmdlist | 5 +
.../hardware_status | 5 +
.../test-node-affinity-nonstrict5/log.expect | 66 +++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 3 +
src/test/test-node-affinity-nonstrict6/README | 14 +
.../test-node-affinity-nonstrict6/cmdlist | 5 +
.../hardware_status | 5 +
.../test-node-affinity-nonstrict6/log.expect | 52 ++
.../manager_status | 1 +
.../rules_config | 3 +
.../service_config | 3 +
src/test/test-node-affinity-strict1/README | 11 +
src/test/test-node-affinity-strict1/cmdlist | 4 +
.../hardware_status | 5 +
.../test-node-affinity-strict1/log.expect | 40 ++
.../test-node-affinity-strict1/manager_status | 1 +
.../test-node-affinity-strict1/rules_config | 4 +
.../test-node-affinity-strict1/service_config | 3 +
src/test/test-node-affinity-strict2/README | 11 +
src/test/test-node-affinity-strict2/cmdlist | 4 +
.../hardware_status | 5 +
.../test-node-affinity-strict2/log.expect | 40 ++
.../test-node-affinity-strict2/manager_status | 1 +
.../test-node-affinity-strict2/rules_config | 4 +
.../test-node-affinity-strict2/service_config | 3 +
src/test/test-node-affinity-strict3/README | 10 +
src/test/test-node-affinity-strict3/cmdlist | 4 +
.../hardware_status | 5 +
.../test-node-affinity-strict3/log.expect | 74 +++
.../test-node-affinity-strict3/manager_status | 1 +
.../test-node-affinity-strict3/rules_config | 4 +
.../test-node-affinity-strict3/service_config | 5 +
src/test/test-node-affinity-strict4/README | 14 +
src/test/test-node-affinity-strict4/cmdlist | 4 +
.../hardware_status | 5 +
.../test-node-affinity-strict4/log.expect | 54 +++
.../test-node-affinity-strict4/manager_status | 1 +
.../test-node-affinity-strict4/rules_config | 4 +
.../test-node-affinity-strict4/service_config | 5 +
src/test/test-node-affinity-strict5/README | 16 +
src/test/test-node-affinity-strict5/cmdlist | 5 +
.../hardware_status | 5 +
.../test-node-affinity-strict5/log.expect | 66 +++
.../test-node-affinity-strict5/manager_status | 1 +
.../test-node-affinity-strict5/rules_config | 4 +
.../test-node-affinity-strict5/service_config | 3 +
src/test/test-node-affinity-strict6/README | 15 +
src/test/test-node-affinity-strict6/cmdlist | 5 +
.../hardware_status | 5 +
.../test-node-affinity-strict6/log.expect | 52 ++
.../test-node-affinity-strict6/manager_status | 1 +
.../test-node-affinity-strict6/rules_config | 4 +
.../test-node-affinity-strict6/service_config | 3 +
src/test/test-recovery1/groups | 4 -
src/test/test-recovery1/rules_config | 4 +
src/test/test-recovery1/service_config | 2 +-
src/test/test-recovery2/README | 4 +-
src/test/test-recovery2/groups | 4 -
src/test/test-recovery2/rules_config | 4 +
src/test/test-recovery2/service_config | 2 +-
src/test/test-recovery3/groups | 4 -
src/test/test-recovery3/rules_config | 4 +
src/test/test-recovery3/service_config | 2 +-
src/test/test-recovery4/groups | 4 -
src/test/test-recovery4/rules_config | 4 +
src/test/test-recovery4/service_config | 2 +-
.../test-relocate-policy-default-group/README | 6 +-
src/test/test-resource-failure2/groups | 2 -
src/test/test-resource-failure2/rules_config | 3 +
.../test-resource-failure2/service_config | 2 +-
.../test-resource-failure3/service_config | 2 +-
src/test/test-resource-failure6/README | 6 +-
src/test/test-shutdown2/groups | 2 -
src/test/test-shutdown2/rules_config | 3 +
src/test/test-shutdown2/service_config | 4 +-
src/test/test-shutdown3/groups | 2 -
src/test/test-shutdown3/rules_config | 3 +
src/test/test-shutdown3/service_config | 4 +-
src/test/test_failover1.pl | 27 +-
src/test/test_rules_config.pl | 100 ++++
160 files changed, 3284 insertions(+), 165 deletions(-)
create mode 100644 src/PVE/API2/HA/Rules.pm
create mode 100644 src/PVE/HA/Rules.pm
create mode 100644 src/PVE/HA/Rules/Makefile
create mode 100644 src/PVE/HA/Rules/NodeAffinity.pm
create mode 100644 src/test/rules_cfgs/defaults-for-node-affinity-rules.cfg
create mode 100644 src/test/rules_cfgs/defaults-for-node-affinity-rules.cfg.expect
create mode 100644 src/test/rules_cfgs/multiple-resource-refs-in-node-affinity-rules.cfg
create mode 100644 src/test/rules_cfgs/multiple-resource-refs-in-node-affinity-rules.cfg.expect
delete mode 100644 src/test/test-basic5/groups
create mode 100644 src/test/test-basic5/rules_config
delete mode 100644 src/test/test-crs-static2/groups
create mode 100644 src/test/test-crs-static2/rules_config
create mode 100644 src/test/test-group-migrate1/README
create mode 100644 src/test/test-group-migrate1/cmdlist
create mode 100644 src/test/test-group-migrate1/groups
create mode 100644 src/test/test-group-migrate1/hardware_status
create mode 100644 src/test/test-group-migrate1/log.expect
create mode 100644 src/test/test-group-migrate1/manager_status
create mode 100644 src/test/test-group-migrate1/service_config
create mode 100644 src/test/test-group-migrate2/README
create mode 100644 src/test/test-group-migrate2/cmdlist
create mode 100644 src/test/test-group-migrate2/groups
create mode 100644 src/test/test-group-migrate2/hardware_status
create mode 100644 src/test/test-group-migrate2/log.expect
create mode 100644 src/test/test-group-migrate2/manager_status
create mode 100644 src/test/test-group-migrate2/service_config
create mode 100644 src/test/test-node-affinity-nonstrict1/README
create mode 100644 src/test/test-node-affinity-nonstrict1/cmdlist
create mode 100644 src/test/test-node-affinity-nonstrict1/hardware_status
create mode 100644 src/test/test-node-affinity-nonstrict1/log.expect
create mode 100644 src/test/test-node-affinity-nonstrict1/manager_status
create mode 100644 src/test/test-node-affinity-nonstrict1/rules_config
create mode 100644 src/test/test-node-affinity-nonstrict1/service_config
create mode 100644 src/test/test-node-affinity-nonstrict2/README
create mode 100644 src/test/test-node-affinity-nonstrict2/cmdlist
create mode 100644 src/test/test-node-affinity-nonstrict2/hardware_status
create mode 100644 src/test/test-node-affinity-nonstrict2/log.expect
create mode 100644 src/test/test-node-affinity-nonstrict2/manager_status
create mode 100644 src/test/test-node-affinity-nonstrict2/rules_config
create mode 100644 src/test/test-node-affinity-nonstrict2/service_config
create mode 100644 src/test/test-node-affinity-nonstrict3/README
create mode 100644 src/test/test-node-affinity-nonstrict3/cmdlist
create mode 100644 src/test/test-node-affinity-nonstrict3/hardware_status
create mode 100644 src/test/test-node-affinity-nonstrict3/log.expect
create mode 100644 src/test/test-node-affinity-nonstrict3/manager_status
create mode 100644 src/test/test-node-affinity-nonstrict3/rules_config
create mode 100644 src/test/test-node-affinity-nonstrict3/service_config
create mode 100644 src/test/test-node-affinity-nonstrict4/README
create mode 100644 src/test/test-node-affinity-nonstrict4/cmdlist
create mode 100644 src/test/test-node-affinity-nonstrict4/hardware_status
create mode 100644 src/test/test-node-affinity-nonstrict4/log.expect
create mode 100644 src/test/test-node-affinity-nonstrict4/manager_status
create mode 100644 src/test/test-node-affinity-nonstrict4/rules_config
create mode 100644 src/test/test-node-affinity-nonstrict4/service_config
create mode 100644 src/test/test-node-affinity-nonstrict5/README
create mode 100644 src/test/test-node-affinity-nonstrict5/cmdlist
create mode 100644 src/test/test-node-affinity-nonstrict5/hardware_status
create mode 100644 src/test/test-node-affinity-nonstrict5/log.expect
create mode 100644 src/test/test-node-affinity-nonstrict5/manager_status
create mode 100644 src/test/test-node-affinity-nonstrict5/rules_config
create mode 100644 src/test/test-node-affinity-nonstrict5/service_config
create mode 100644 src/test/test-node-affinity-nonstrict6/README
create mode 100644 src/test/test-node-affinity-nonstrict6/cmdlist
create mode 100644 src/test/test-node-affinity-nonstrict6/hardware_status
create mode 100644 src/test/test-node-affinity-nonstrict6/log.expect
create mode 100644 src/test/test-node-affinity-nonstrict6/manager_status
create mode 100644 src/test/test-node-affinity-nonstrict6/rules_config
create mode 100644 src/test/test-node-affinity-nonstrict6/service_config
create mode 100644 src/test/test-node-affinity-strict1/README
create mode 100644 src/test/test-node-affinity-strict1/cmdlist
create mode 100644 src/test/test-node-affinity-strict1/hardware_status
create mode 100644 src/test/test-node-affinity-strict1/log.expect
create mode 100644 src/test/test-node-affinity-strict1/manager_status
create mode 100644 src/test/test-node-affinity-strict1/rules_config
create mode 100644 src/test/test-node-affinity-strict1/service_config
create mode 100644 src/test/test-node-affinity-strict2/README
create mode 100644 src/test/test-node-affinity-strict2/cmdlist
create mode 100644 src/test/test-node-affinity-strict2/hardware_status
create mode 100644 src/test/test-node-affinity-strict2/log.expect
create mode 100644 src/test/test-node-affinity-strict2/manager_status
create mode 100644 src/test/test-node-affinity-strict2/rules_config
create mode 100644 src/test/test-node-affinity-strict2/service_config
create mode 100644 src/test/test-node-affinity-strict3/README
create mode 100644 src/test/test-node-affinity-strict3/cmdlist
create mode 100644 src/test/test-node-affinity-strict3/hardware_status
create mode 100644 src/test/test-node-affinity-strict3/log.expect
create mode 100644 src/test/test-node-affinity-strict3/manager_status
create mode 100644 src/test/test-node-affinity-strict3/rules_config
create mode 100644 src/test/test-node-affinity-strict3/service_config
create mode 100644 src/test/test-node-affinity-strict4/README
create mode 100644 src/test/test-node-affinity-strict4/cmdlist
create mode 100644 src/test/test-node-affinity-strict4/hardware_status
create mode 100644 src/test/test-node-affinity-strict4/log.expect
create mode 100644 src/test/test-node-affinity-strict4/manager_status
create mode 100644 src/test/test-node-affinity-strict4/rules_config
create mode 100644 src/test/test-node-affinity-strict4/service_config
create mode 100644 src/test/test-node-affinity-strict5/README
create mode 100644 src/test/test-node-affinity-strict5/cmdlist
create mode 100644 src/test/test-node-affinity-strict5/hardware_status
create mode 100644 src/test/test-node-affinity-strict5/log.expect
create mode 100644 src/test/test-node-affinity-strict5/manager_status
create mode 100644 src/test/test-node-affinity-strict5/rules_config
create mode 100644 src/test/test-node-affinity-strict5/service_config
create mode 100644 src/test/test-node-affinity-strict6/README
create mode 100644 src/test/test-node-affinity-strict6/cmdlist
create mode 100644 src/test/test-node-affinity-strict6/hardware_status
create mode 100644 src/test/test-node-affinity-strict6/log.expect
create mode 100644 src/test/test-node-affinity-strict6/manager_status
create mode 100644 src/test/test-node-affinity-strict6/rules_config
create mode 100644 src/test/test-node-affinity-strict6/service_config
delete mode 100644 src/test/test-recovery1/groups
create mode 100644 src/test/test-recovery1/rules_config
delete mode 100644 src/test/test-recovery2/groups
create mode 100644 src/test/test-recovery2/rules_config
delete mode 100644 src/test/test-recovery3/groups
create mode 100644 src/test/test-recovery3/rules_config
delete mode 100644 src/test/test-recovery4/groups
create mode 100644 src/test/test-recovery4/rules_config
delete mode 100644 src/test/test-resource-failure2/groups
create mode 100644 src/test/test-resource-failure2/rules_config
delete mode 100644 src/test/test-shutdown2/groups
create mode 100644 src/test/test-shutdown2/rules_config
delete mode 100644 src/test/test-shutdown3/groups
create mode 100644 src/test/test-shutdown3/rules_config
create mode 100755 src/test/test_rules_config.pl
docs:
Daniel Kral (2):
ha: add documentation about ha rules and ha node affinity rules
ha: crs: add effects of ha node affinity rule on the crs scheduler
Makefile | 2 +
gen-ha-rules-node-affinity-opts.pl | 20 +++++
gen-ha-rules-opts.pl | 17 +++++
ha-manager.adoc | 113 +++++++++++++++++++++++++++++
ha-rules-node-affinity-opts.adoc | 18 +++++
ha-rules-opts.adoc | 12 +++
pmxcfs.adoc | 1 +
7 files changed, 183 insertions(+)
create mode 100755 gen-ha-rules-node-affinity-opts.pl
create mode 100755 gen-ha-rules-opts.pl
create mode 100644 ha-rules-node-affinity-opts.adoc
create mode 100644 ha-rules-opts.adoc
manager:
Daniel Kral (4):
api: ha: add ha rules api endpoints
ui: ha: remove ha groups from ha resource components
ui: ha: show failback flag in resources status view
ui: ha: replace ha groups with ha node affinity rules
PVE/API2/HAConfig.pm | 8 +-
www/manager6/Makefile | 8 +-
www/manager6/StateProvider.js | 2 +-
www/manager6/dc/Config.js | 8 +-
www/manager6/ha/GroupSelector.js | 71 -------
www/manager6/ha/Groups.js | 117 -----------
www/manager6/ha/ResourceEdit.js | 16 +-
www/manager6/ha/Resources.js | 17 +-
www/manager6/ha/RuleEdit.js | 146 +++++++++++++
www/manager6/ha/RuleErrorsModal.js | 50 +++++
www/manager6/ha/Rules.js | 196 ++++++++++++++++++
www/manager6/ha/StatusView.js | 5 +-
.../NodeAffinityRuleEdit.js} | 105 ++--------
www/manager6/ha/rules/NodeAffinityRules.js | 36 ++++
14 files changed, 481 insertions(+), 304 deletions(-)
delete mode 100644 www/manager6/ha/GroupSelector.js
delete mode 100644 www/manager6/ha/Groups.js
create mode 100644 www/manager6/ha/RuleEdit.js
create mode 100644 www/manager6/ha/RuleErrorsModal.js
create mode 100644 www/manager6/ha/Rules.js
rename www/manager6/ha/{GroupEdit.js => rules/NodeAffinityRuleEdit.js} (67%)
create mode 100644 www/manager6/ha/rules/NodeAffinityRules.js
base-commit: 58f080d41079a0c2494a61cdaf382e7e7adf39db
prerequisite-patch-id: f7b4b46d65fb44865ff09e2cc59cdf7fd659b48c
prerequisite-patch-id: d9576d6a257f301a2396e2874c734b0d892a5761
prerequisite-patch-id: d580d2af8dba82526b35ccce0b0e943667666040
prerequisite-patch-id: 2ad790fd968ed7da823222103e17605498a1e5c8
prerequisite-patch-id: 30dd0ce7ccbc9c5b8a3153515ab320c9bcbff3b7
prerequisite-patch-id: 0420d015785effb38262f336c2f30566fa7f0ef5
prerequisite-patch-id: 5a975adc780d83762ebb2be29fc03f1cd3e6eb69
prerequisite-patch-id: 9a86ed00a5f9c565259de761bc1a141e49820874
prerequisite-patch-id: 84bba3312f1c47432c6d86075e422509653b8263
Summary over all repositories:
181 files changed, 3948 insertions(+), 469 deletions(-)
--
Generated by git-murpp 0.8.0
More information about the pve-devel
mailing list