[pve-devel] [PATCH docs v3 1/1] ha: add documentation about ha rules and ha node affinity rules
Daniel Kral
d.kral at proxmox.com
Fri Jul 4 20:16:56 CEST 2025
Add documentation about HA Node Affinity rules and general documentation
what HA rules are for in a format that is extendable with other HA rule
types in the future.
Signed-off-by: Daniel Kral <d.kral at proxmox.com>
append to ha intro
Signed-off-by: Daniel Kral <d.kral at proxmox.com>
---
Makefile | 2 +
gen-ha-rules-node-affinity-opts.pl | 20 ++++++
gen-ha-rules-opts.pl | 17 +++++
ha-manager.adoc | 103 +++++++++++++++++++++++++++++
ha-rules-node-affinity-opts.adoc | 18 +++++
ha-rules-opts.adoc | 12 ++++
pmxcfs.adoc | 1 +
7 files changed, 173 insertions(+)
create mode 100755 gen-ha-rules-node-affinity-opts.pl
create mode 100755 gen-ha-rules-opts.pl
create mode 100644 ha-rules-node-affinity-opts.adoc
create mode 100644 ha-rules-opts.adoc
diff --git a/Makefile b/Makefile
index f30d77a..c5e506e 100644
--- a/Makefile
+++ b/Makefile
@@ -49,6 +49,8 @@ GEN_DEB_SOURCES= \
GEN_SCRIPTS= \
gen-ha-groups-opts.pl \
gen-ha-resources-opts.pl \
+ gen-ha-rules-node-affinity-opts.pl \
+ gen-ha-rules-opts.pl \
gen-datacenter.cfg.5-opts.pl \
gen-pct.conf.5-opts.pl \
gen-pct-network-opts.pl \
diff --git a/gen-ha-rules-node-affinity-opts.pl b/gen-ha-rules-node-affinity-opts.pl
new file mode 100755
index 0000000..e2f07fa
--- /dev/null
+++ b/gen-ha-rules-node-affinity-opts.pl
@@ -0,0 +1,20 @@
+#!/usr/bin/perl
+
+use lib '.';
+use strict;
+use warnings;
+use PVE::RESTHandler;
+
+use Data::Dumper;
+
+use PVE::HA::Rules;
+use PVE::HA::Rules::NodeAffinity;
+
+my $private = PVE::HA::Rules::private();
+my $node_affinity_rule_props = PVE::HA::Rules::NodeAffinity::properties();
+my $properties = {
+ resources => $private->{propertyList}->{resources},
+ $node_affinity_rule_props->%*,
+};
+
+print PVE::RESTHandler::dump_properties($properties);
diff --git a/gen-ha-rules-opts.pl b/gen-ha-rules-opts.pl
new file mode 100755
index 0000000..66dd174
--- /dev/null
+++ b/gen-ha-rules-opts.pl
@@ -0,0 +1,17 @@
+#!/usr/bin/perl
+
+use lib '.';
+use strict;
+use warnings;
+use PVE::RESTHandler;
+
+use Data::Dumper;
+
+use PVE::HA::Rules;
+
+my $private = PVE::HA::Rules::private();
+my $properties = $private->{propertyList};
+delete $properties->{type};
+delete $properties->{rule};
+
+print PVE::RESTHandler::dump_properties($properties);
diff --git a/ha-manager.adoc b/ha-manager.adoc
index 3d6fc4a..d316bb7 100644
--- a/ha-manager.adoc
+++ b/ha-manager.adoc
@@ -670,6 +670,109 @@ up online again to investigate the cause of failure and check if it runs
stably again. Setting the `nofailback` flag prevents the recovered services from
moving straight back to the fenced node.
+[[ha_manager_rules]]
+Rules
+~~~~~
+
+HA rules are used to put certain constraints on HA-managed resources, which are
+defined in the HA rules configuration file `/etc/pve/ha/rules.cfg`.
+
+----
+<type>: <rule>
+ resources <resources_list>
+ <property> <value>
+ ...
+----
+
+include::ha-rules-opts.adoc[]
+
+.Available HA Rule Types
+[width="100%",cols="1,3",options="header"]
+|===========================================================
+| HA Rule Type | Description
+| `node-affinity` | Places affinity from one or more HA resources to one or
+more nodes.
+|===========================================================
+
+[[ha_manager_node_affinity_rules]]
+Node Affinity Rules
+^^^^^^^^^^^^^^^^^^^
+
+NOTE: HA Node Affinity rules are equivalent to HA Groups and will replace them
+in an upcoming major release.
+
+By default, a HA resource is able to run on any cluster node, but a common
+requirement is that a HA resource should run on a specific node. That can be
+implemented by defining a HA node affinity rule to make the HA resource
+`vm:100` prefer the node `node1`:
+
+----
+# ha-manager rules add node-affinity ha-rule-vm100 --resources vm:100 --nodes node1
+----
+
+By default, node affinity rules are not strict, i.e., if there is none of the
+specified nodes available, the HA resource can also be moved to other nodes.
+If, on the other hand, a HA resource must be restricted to the specified nodes,
+then the node affinity rule must be set to be strict.
+
+In the previous example, the node affinity rule can be modified to restrict the
+resource `vm:100` to be only on `node1`:
+
+----
+# ha-manager rules set node-affinity ha-rule-vm100 --strict 1
+----
+
+For bigger clusters or specific use cases, it makes sense to define a more
+detailed failover behavior. For example, the resources `vm:200` and `ct:300`
+should run on `node1`. If `node1` becomes unavailable, the resources should be
+distributed on `node2` and `node3`. If `node2` and `node3` are also
+unavailable, the resources should run on `node4`.
+
+To implement this behavior in a node affinity rule, nodes can be paired with
+priorities to order the preference for nodes. If two or more nodes have the same
+priority, the resources can run on any of them. For the above example, `node1`
+gets the highest priority, `node2` and `node3` get the same priority, and at
+last `node4` gets the lowest priority, which can be omitted to default to `0`:
+
+----
+# ha-manager rules add node-affinity priority-cascade \
+ --resources vm:200,ct:300 --nodes "node1:2,node2:1,node3:1,node4"
+----
+
+The above commands create the following rules in the rules configuration file:
+
+.Node Affinity Rules Configuration Example (`/etc/pve/ha/rules.cfg`)
+----
+node-affinity: ha-rule-vm100
+ resources vm:100
+ nodes node1
+ strict 1
+
+node-affinity: priority-cascade
+ resources vm:200,ct:300
+ nodes node1:2,node2:1,node3:1,node4
+----
+
+Node Affinity Rule Properties
++++++++++++++++++++++++++++++
+
+include::ha-rules-node-affinity-opts.adoc[]
+
+[[ha_manager_rule_conflicts]]
+Rule Conflicts and Errors
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+HA rules can impose rather complex constraints on the HA resources. To ensure
+that a new or modified HA rule does not introduce uncertainty into the HA
+stack's CRS scheduler, HA rules are tested for feasibility before these are
+applied. If a rule does fail any of these tests, the rule is disabled until the
+conflicts and errors is resolved.
+
+Currently, HA rules are checked for the following feasibility tests:
+
+* A HA resource can only be referenced by a single HA node affinity rule in
+ total. If two or more HA node affinity rules specify the same HA resource,
+ these HA node affinity rules will be disabled.
[[ha_manager_fencing]]
Fencing
diff --git a/ha-rules-node-affinity-opts.adoc b/ha-rules-node-affinity-opts.adoc
new file mode 100644
index 0000000..852636c
--- /dev/null
+++ b/ha-rules-node-affinity-opts.adoc
@@ -0,0 +1,18 @@
+`nodes`: `<node>[:<pri>]{,<node>[:<pri>]}*` ::
+
+List of cluster node members, where a priority can be given to each node. A resource bound to a group will run on the available nodes with the highest priority. If there are more nodes in the highest priority class, the services will get distributed to those nodes. The priorities have a relative meaning only. The higher the number, the higher the priority.
+
+`resources`: `<type>:<name>{,<type>:<name>}*` ::
+
+List of HA resource IDs. This consists of a list of resource types followed by a resource specific name separated with a colon (example: vm:100,ct:101).
+
+`strict`: `<boolean>` ('default =' `0`)::
+
+Describes whether the node affinity rule is strict or non-strict.
++
+A non-strict node affinity rule makes resources prefer to be on the defined nodes.
+If none of the defined nodes are available, the resource may run on any other node.
++
+A strict node affinity rule makes resources be restricted to the defined nodes. If
+none of the defined nodes are available, the resource will be stopped.
+
diff --git a/ha-rules-opts.adoc b/ha-rules-opts.adoc
new file mode 100644
index 0000000..b50b289
--- /dev/null
+++ b/ha-rules-opts.adoc
@@ -0,0 +1,12 @@
+`comment`: `<string>` ::
+
+HA rule description.
+
+`disable`: `<boolean>` ('default =' `0`)::
+
+Whether the HA rule is disabled.
+
+`resources`: `<type>:<name>{,<type>:<name>}*` ::
+
+List of HA resource IDs. This consists of a list of resource types followed by a resource specific name separated with a colon (example: vm:100,ct:101).
+
diff --git a/pmxcfs.adoc b/pmxcfs.adoc
index f4aa847..8ca7284 100644
--- a/pmxcfs.adoc
+++ b/pmxcfs.adoc
@@ -104,6 +104,7 @@ Files
|`ha/crm_commands` | Displays HA operations that are currently being carried out by the CRM
|`ha/manager_status` | JSON-formatted information regarding HA services on the cluster
|`ha/resources.cfg` | Resources managed by high availability, and their current state
+|`ha/rules.cfg` | Rules putting constraints on the HA manager's scheduling of HA resources
|`nodes/<NAME>/config` | Node-specific configuration
|`nodes/<NAME>/lxc/<VMID>.conf` | VM configuration data for LXC containers
|`nodes/<NAME>/openvz/` | Prior to {pve} 4.0, used for container configuration data (deprecated, removed soon)
--
2.39.5
More information about the pve-devel
mailing list