[pve-devel] [PATCH docs v5 1/1] ha: add documentation about ha resource affinity rules
Daniel Kral
d.kral at proxmox.com
Wed Jul 30 20:14:19 CEST 2025
Add documentation about HA Resource Affinity rules, what effects those
have on the CRS scheduler, and what users can expect when those are
changed.
There are also a few points on the rule conflicts/errors list which
describe some conflicts that can arise from a mixed usage of HA Node
Affinity rules and HA Resource Affinity rules.
Signed-off-by: Daniel Kral <d.kral at proxmox.com>
---
Makefile | 1 +
gen-ha-rules-resource-affinity-opts.pl | 20 +++++
ha-manager.adoc | 116 +++++++++++++++++++++++++
ha-rules-resource-affinity-opts.adoc | 8 ++
4 files changed, 145 insertions(+)
create mode 100755 gen-ha-rules-resource-affinity-opts.pl
create mode 100644 ha-rules-resource-affinity-opts.adoc
diff --git a/Makefile b/Makefile
index c5e506e..4d9e2f0 100644
--- a/Makefile
+++ b/Makefile
@@ -51,6 +51,7 @@ GEN_SCRIPTS= \
gen-ha-resources-opts.pl \
gen-ha-rules-node-affinity-opts.pl \
gen-ha-rules-opts.pl \
+ gen-ha-rules-resource-affinity-opts.pl \
gen-datacenter.cfg.5-opts.pl \
gen-pct.conf.5-opts.pl \
gen-pct-network-opts.pl \
diff --git a/gen-ha-rules-resource-affinity-opts.pl b/gen-ha-rules-resource-affinity-opts.pl
new file mode 100755
index 0000000..5abed50
--- /dev/null
+++ b/gen-ha-rules-resource-affinity-opts.pl
@@ -0,0 +1,20 @@
+#!/usr/bin/perl
+
+use lib '.';
+use strict;
+use warnings;
+use PVE::RESTHandler;
+
+use Data::Dumper;
+
+use PVE::HA::Rules;
+use PVE::HA::Rules::ResourceAffinity;
+
+my $private = PVE::HA::Rules::private();
+my $resource_affinity_rule_props = PVE::HA::Rules::ResourceAffinity::properties();
+my $properties = {
+ resources => $private->{propertyList}->{resources},
+ $resource_affinity_rule_props->%*,
+};
+
+print PVE::RESTHandler::dump_properties($properties);
diff --git a/ha-manager.adoc b/ha-manager.adoc
index 20eeb88..77d7a65 100644
--- a/ha-manager.adoc
+++ b/ha-manager.adoc
@@ -690,6 +690,10 @@ include::ha-rules-opts.adoc[]
| HA Rule Type | Description
| `node-affinity` | Places affinity from one or more HA resources to one or
more nodes.
+| `resource-affinity` | Places affinity between two or more HA resources. The
+affinity `positive` specifies that HA resources are to be kept on separate
+nodes, while the affinity `separate` specifies that HA resources are to be kept
+on the same node.
|===========================================================
[[ha_manager_node_affinity_rules]]
@@ -756,6 +760,88 @@ Node Affinity Rule Properties
include::ha-rules-node-affinity-opts.adoc[]
+[[ha_manager_resource_affinity_rules]]
+Resource Affinity Rules
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Another common requirement is that two or more HA resources should run on
+either the same node, or should be distributed on separate nodes. These are
+also commonly called "Affinity/Anti-Affinity constraints".
+
+For example, suppose there is a lot of communication traffic between the HA
+resources `vm:100` and `vm:200`, for example, a web server communicating with a
+database server. If those HA resources are on separate nodes, this could
+potentially result in a higher latency and unnecessary network load. Resource
+affinity rules with the affinity `positive` implement the constraint to keep
+the HA resources on the same node:
+
+----
+# ha-manager rules add resource-affinity keep-together \
+ --affinity positive --resources vm:100,vm:200
+----
+
+NOTE: If there are two or more positive resource affinity rules, which have
+common HA resources, then these are treated as a single positive resource
+affinity rule. For example, if the HA resources `vm:100` and `vm:101` and the
+HA resources `vm:101` and `vm:102` are each in a positive resource affinity
+rule, then it is the same as if `vm:100`, `vm:101` and `vm:102` would have been
+in a single positive resource affinity rule.
+
+However, suppose there are computationally expensive, and/or distributed
+programs running on the HA resources `vm:200` and `ct:300`, for example,
+sharded database instances. In that case, running them on the same node could
+potentially result in pressure on the hardware resources of the node and will
+slow down the operations of these HA resources. Resource affinity rules with
+the affinity `negative` implement the constraint to spread the HA resources on
+separate nodes:
+
+----
+# ha-manager rules add resource-affinity keep-separate \
+ --affinity negative --resources vm:200,ct:300
+----
+
+Other than node affinity rules, resource affinity rules are strict by default,
+i.e., if the constraints imposed by the resource affinity rules cannot be met
+for a HA resource, the HA Manager will put the HA resource in recovery state in
+case of a failover or in error state elsewhere.
+
+The above commands created the following rules in the rules configuration file:
+
+.Resource Affinity Rules Configuration Example (`/etc/pve/ha/rules.cfg`)
+----
+resource-affinity: keep-together
+ resources vm:100,vm:200
+ affinity positive
+
+resource-affinity: keep-separate
+ resources vm:200,ct:300
+ affinity negative
+----
+
+Interactions between Positive and Negative Resource Affinity Rules
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+If there are HA resources in a positive resource affinity rule, which are also
+part of a negative resource affinity rule, then all the other HA resources in
+the positive resource affinity rule are in negative affinity with the HA
+resources of these negative resource affinity rules as well.
+
+For example, if the HA resources `vm:100`, `vm:101`, and `vm:102` are in a
+positive resource affinity rule, and `vm:100` is in a negative resource affinity
+rule with the HA resource `ct:200`, then `vm:101` and `vm:102` are each in
+negative resource affinity with `ct:200` as well.
+
+Note that if there are two or more HA resources in both a positive and negative
+resource affinity rule, then those will be disabled as they cause a conflict:
+Two or more HA resources cannot be kept on the same node and separated on
+different nodes at the same time. For more information on these cases, see the
+section about xref:ha_manager_rule_conflicts[rule conflicts and errors] below.
+
+Resource Affinity Rule Properties
++++++++++++++++++++++++++++++++++
+
+include::ha-rules-resource-affinity-opts.adoc[]
+
[[ha_manager_rule_conflicts]]
Rule Conflicts and Errors
~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -772,6 +858,26 @@ Currently, HA rules are checked for the following feasibility tests:
total. If two or more HA node affinity rules specify the same HA resource,
these HA node affinity rules will be disabled.
+* A HA resource affinity rule must specify at least two HA resources to be
+ feasible. If a HA resource affinity rule specifies only one HA resource, the
+ HA resource affinity rule will be disabled.
+
+* A HA resource affinity rule must specify no more HA resources than there are
+ nodes in the cluster. If a HA resource affinity rule specifies more HA
+ resources than there are in the cluster, the HA resource affinity rule will
+ be disabled.
+
+* A positive HA resource affinity rule cannot specify the same two or more HA
+ resources as a negative HA resources affinity rule. That is, two or more HA
+ resources cannot be kept together and separate at the same time. If any pair
+ of positive and negative HA resource affinity rules do specify the same two
+ or more HA resources, both HA resource affinity rules will be disabled.
+
+* A HA resource can only be referenced by either a HA node affinity rule or HA
+ resource affinity rules, but not both at the same time. If there is at least
+ one HA node affinity rule and at least one HA resource affinity rule, which
+ specify the same HA resource, these rules will be disabled.
+
[[ha_manager_fencing]]
Fencing
-------
@@ -1203,6 +1309,16 @@ The CRS is currently used at the following scheduling points:
algorithm to ensure that these HA resources are assigned according to their
node and priority constraints.
+** Positive resource affinity rules: If a positive resource affinity rule is
+ created or HA resources are added to an existing positive resource affinity
+ rule, the HA stack will use the CRS algorithm to ensure that these HA
+ resources are moved to a common node.
+
+** Negative resource affinity rules: If a negative resource affinity rule is
+ created or HA resources are added to an existing negative resource affinity
+ rule, the HA stack will use the CRS algorithm to ensure that these HA
+ resources are moved to separate nodes.
+
- HA service stopped -> start transition (opt-in). Requesting that a stopped
service should be started is an good opportunity to check for the best suited
node as per the CRS algorithm, as moving stopped services is cheaper to do
diff --git a/ha-rules-resource-affinity-opts.adoc b/ha-rules-resource-affinity-opts.adoc
new file mode 100644
index 0000000..596ec3c
--- /dev/null
+++ b/ha-rules-resource-affinity-opts.adoc
@@ -0,0 +1,8 @@
+`affinity`: `<negative | positive>` ::
+
+Describes whether the HA resources are supposed to be kept on the same node ('positive'), or are supposed to be kept on separate nodes ('negative').
+
+`resources`: `<type>:<name>{,<type>:<name>}*` ::
+
+List of HA resource IDs. This consists of a list of resource types followed by a resource specific name separated with a colon (example: vm:100,ct:101).
+
--
2.47.2
More information about the pve-devel
mailing list