[pve-devel] [PATCH docs v2 1/5] ha: config: add section about ha rules
Daniel Kral
d.kral at proxmox.com
Fri Jun 20 16:31:39 CEST 2025
Add section about how to create and modify ha rules, describing their
use cases and document their common and plugin-specific properties.
As of now, HA Location rules are controlled by the feature flag
'use-location-rules' in the datacenter config to replace HA Groups.
Signed-off-by: Daniel Kral <d.kral at proxmox.com>
---
There's still a few things missing here, e.g., what happens during
migration and pointing out that "connected" positive colocation rules
are handled as one positive colocation.
changes since v1:
- NEW!
Makefile | 3 +
gen-ha-rules-colocation-opts.pl | 20 +++
gen-ha-rules-location-opts.pl | 20 +++
gen-ha-rules-opts.pl | 17 +++
ha-manager.adoc | 231 +++++++++++++++++++++++++++++++-
ha-rules-colocation-opts.adoc | 8 ++
ha-rules-location-opts.adoc | 14 ++
ha-rules-opts.adoc | 12 ++
8 files changed, 318 insertions(+), 7 deletions(-)
create mode 100755 gen-ha-rules-colocation-opts.pl
create mode 100755 gen-ha-rules-location-opts.pl
create mode 100755 gen-ha-rules-opts.pl
create mode 100644 ha-rules-colocation-opts.adoc
create mode 100644 ha-rules-location-opts.adoc
create mode 100644 ha-rules-opts.adoc
diff --git a/Makefile b/Makefile
index f30d77a..b6da924 100644
--- a/Makefile
+++ b/Makefile
@@ -49,6 +49,9 @@ GEN_DEB_SOURCES= \
GEN_SCRIPTS= \
gen-ha-groups-opts.pl \
gen-ha-resources-opts.pl \
+ gen-ha-rules-opts.pl \
+ gen-ha-rules-colocation-opts.pl \
+ gen-ha-rules-location-opts.pl \
gen-datacenter.cfg.5-opts.pl \
gen-pct.conf.5-opts.pl \
gen-pct-network-opts.pl \
diff --git a/gen-ha-rules-colocation-opts.pl b/gen-ha-rules-colocation-opts.pl
new file mode 100755
index 0000000..203cbb6
--- /dev/null
+++ b/gen-ha-rules-colocation-opts.pl
@@ -0,0 +1,20 @@
+#!/usr/bin/perl
+
+use lib '.';
+use strict;
+use warnings;
+use PVE::RESTHandler;
+
+use Data::Dumper;
+
+use PVE::HA::Rules;
+use PVE::HA::Rules::Colocation;
+
+my $private = PVE::HA::Rules::private();
+my $colocation_props = PVE::HA::Rules::Colocation::properties();
+my $properies = {
+ services => $private->{propertyList}->{services},
+ $colocation_props->%*,
+};
+
+print PVE::RESTHandler::dump_properties($properies);
diff --git a/gen-ha-rules-location-opts.pl b/gen-ha-rules-location-opts.pl
new file mode 100755
index 0000000..0564385
--- /dev/null
+++ b/gen-ha-rules-location-opts.pl
@@ -0,0 +1,20 @@
+#!/usr/bin/perl
+
+use lib '.';
+use strict;
+use warnings;
+use PVE::RESTHandler;
+
+use Data::Dumper;
+
+use PVE::HA::Rules;
+use PVE::HA::Rules::Location;
+
+my $private = PVE::HA::Rules::private();
+my $location_props = PVE::HA::Rules::Location::properties();
+my $properies = {
+ services => $private->{propertyList}->{services},
+ $location_props->%*,
+};
+
+print PVE::RESTHandler::dump_properties($properies);
diff --git a/gen-ha-rules-opts.pl b/gen-ha-rules-opts.pl
new file mode 100755
index 0000000..012cb1a
--- /dev/null
+++ b/gen-ha-rules-opts.pl
@@ -0,0 +1,17 @@
+#!/usr/bin/perl
+
+use lib '.';
+use strict;
+use warnings;
+use PVE::RESTHandler;
+
+use Data::Dumper;
+
+use PVE::HA::Rules;
+
+my $private = PVE::HA::Rules::private();
+my $properies = $private->{propertyList};
+delete $properies->{type};
+delete $properies->{rule};
+
+print PVE::RESTHandler::dump_properties($properies);
diff --git a/ha-manager.adoc b/ha-manager.adoc
index 3d6fc4a..12f73f1 100644
--- a/ha-manager.adoc
+++ b/ha-manager.adoc
@@ -670,6 +670,205 @@ up online again to investigate the cause of failure and check if it runs
stably again. Setting the `nofailback` flag prevents the recovered services from
moving straight back to the fenced node.
+[[ha_manager_rules]]
+Rules
+~~~~~
+
+HA rules are used to put certain constraints on HA-managed resources, which are
+defined in the HA rules configuration file `/etc/pve/ha/rules.cfg`.
+
+----
+<type>: <rule>
+ services <services_list>
+ <property> <value>
+ ...
+----
+
+include::ha-rules-opts.adoc[]
+
+.Available HA Rule Types
+[width="100%",cols="1,3",options="header"]
+|===========================================================
+| HA Rule Type | Description
+| `location` | Places affinity from one or more HA resources to one or more
+nodes.
+| `colocation` | Places affinity between two or more HA resources. The affinity
+`separate` specifies that HA resources are to be kept on separate nodes, while
+the affinity `together` specifies that HA resources are to be kept on the same
+node.
+|===========================================================
+
+[[ha_manager_location_rules]]
+Location
+^^^^^^^^
+
+NOTE: HA Location rules are equivalent to HA Groups and will replace them in an
+upcoming major release. They can be used by enabling the `use-location-rules`
+option in the web interface under __Datacenter -> Options -> HA__ or in the
+`datacenter.cfg`. The HA Groups will not be used anymore if Location rules are
+enabled.
+
+A common requirement is that a HA resource should run on a specific node.
+Usually, the resource is able to run on any cluster node, so you can define a
+location rule to make the resource `vm:100` prefer the node `node1`:
+
+----
+# ha-manager rules add location vm100_prefers_node1 --services vm:100 --nodes node1
+----
+
+By default, location rules are not strict, i.e., if there is no specified node
+available, the resource can be moved to other nodes. In the previous example,
+the rule can be modified to restrict the resource `vm:100` to be only on `node1`:
+
+----
+# ha-manager rules set location vm100_prefers_node1 --strict 1
+----
+
+For bigger clusters, it makes sense to define a more detailed failover behavior.
+For example, the resources `vm:200` and `ct:300` should run on `node1`. If
+`node1` becomes unavailable, the resources should be distributed on `node2` and
+`node3`. If `node2` and `node3` are also unavailable, the resources should run
+on `node4`.
+
+To implement this behavior in a location rule, nodes can be paired with
+priorities to order the preference for nodes. If two or more nodes have the same
+priority, the resources can run on any of them. For the above example, `node1`
+gets the highest priority, `node2` and `node3` get the same priority, and at
+last `node4` gets the lowest priority, which can be omitted to default to `0`:
+
+----
+# ha-manager rules add location priority_cascade \
+ --services vm:200,ct:300 --nodes "node1:2,node2:1,node3:1,node4"
+----
+
+The above commands created the following rules in the rules configuration file:
+
+.Location Rules Configuration Example (`/etc/pve/ha/rules.cfg`)
+----
+location: vm100_prefers_node1
+ services vm:100
+ nodes node1
+ strict 1
+
+location: priority_cascade
+ services vm:200,ct:300
+ nodes node1:2,node2:1,node3:1,node4
+----
+
+Location Rule Properties
+++++++++++++++++++++++++
+
+include::ha-rules-location-opts.adoc[]
+
+[[ha_manager_colocation_rules]]
+Colocation
+^^^^^^^^^^
+
+A common requirement is that two or more HA resources should run on either the
+same node, or should be distributed on separate nodes. These are also commonly
+called "Affinity/Anti-Affinity" constraints.
+
+For example, suppose there is a lot of communication traffic between the
+HA resources `vm:100` and `vm:200`, say a web server communicating with a
+database server. If those HA resources are on separate nodes, this could
+potentially result in a higher latency and unnecessary network load. Colocation
+rules with the affinity `together` implement this constraint to colocate the
+HA resources on the same node:
+
+----
+# ha-manager rules add colocation keep_together \
+ --affinity together --services vm:100,vm:200
+----
+
+However, suppose there are computationally expensive, and/or distributed
+programs running on the HA resources `vm:200` and `ct:300`, say sharded
+database instances. In that case, running them on the same node could
+potentially result in pressure on the hardware resources of the node and will
+slow down the operations of these HA resources. Colocation rules with the
+affinity `separate` implement this constraint to colocate the HA resources on
+separate nodes:
+
+----
+# ha-manager rules add colocation keep_separate \
+ --affinity separate --services vm:200,ct:300
+----
+
+Other than HA location rules, colocation rules are strict by default,
+i.e., if the constraints imposed by the colocation rules cannot be met,
+the HA Manager will put these in recovery state in case of a failover or
+in error state elsewhere.
+
+The above commands created the following rules in the rules configuration file:
+
+.Colocation Rules Configuration Example (`/etc/pve/ha/rules.cfg`)
+----
+colocation: keep_together
+ affinity together
+ services vm:100,vm:200
+
+colocation: keep_separate
+ affinity separate
+ services vm:200,ct:300
+----
+
+Colocation Rule Properties
+++++++++++++++++++++++++++
+
+include::ha-rules-colocation-opts.adoc[]
+
+[[ha_manager_rule_conflicts]]
+Rule Conflicts
+~~~~~~~~~~~~~~
+
+HA rules can impose rather complex constraints on the HA resources. To
+ensure that a new or modified HA rule does not introduce uncertainty
+into the HA stack's CRS scheduler, HA rules are tested for feasibility
+before these are applied. If a rule does not pass any of these tests,
+the rule is disabled until the conflict is resolved.
+
+Currently, HA rules are checked for the following feasibility tests:
+
+* A HA resource can only be referenced by a single HA location rule in
+ total. If two or more HA location rules specify the same HA service,
+ these HA location rules will be disabled.
+
+* A HA colocation rule must specify at least two HA resources to be
+ feasible. If a HA colocation rule does specify only one HA resource,
+ the HA colocation rule will be disabled.
+
+* A positive HA colocation rule cannot specify the same two or more HA
+ resources as a negative HA colocation rule. That is, two or more HA
+ resources cannot be kept together and separate at the same time. If
+ any pair of positive and negative HA colocation rules do specify the
+ same two or more HA resources, both HA colocation rules will be
+ disabled.
+
+* A HA resource, which is already constrained by a HA group or a HA
+ location rule, can only be referenced by a HA colocation rule, if the
+ HA group or HA location rule does only use a single priority group,
+ i.e., the specified nodes all have the same priority. If one of the HA
+ resources of a HA colocation rule is constrained by a HA group or HA
+ location rule with multiple priorities, the HA colocation rule will be
+ disabled.
+
+* A HA resource, which is already constrained by a HA group or a HA
+ location rule, can only be referenced by a positive HA colocation
+ rule, if the HA group or HA location rule specifies at least one
+ common node, where the other positively colocated HA resources are
+ also allowed to run on. Otherwise, the positively colocated HA
+ resources could only run on separate nodes. In other words, if two or
+ more HA resources of a positive HA colocation rule are constrained to
+ different nodes, the positive HA colocation rule will be disabled.
+
+* A HA resource, which is already constrained by a HA group or a HA
+ location rule, can only be referenced by a negative HA colocation
+ rule, if the HA group or HA location rule specifies at least one node,
+ where the other negatively colocated HA resources are not allowed to
+ run on. Otherwise, the negatively colocated HA resources do not have
+ enough nodes to be separated on. In other words, if two or more HA
+ resources of a negative HA colocation rule are constrained to less
+ nodes than needed to separate them on, the negative HA colocation rule
+ will be disabled.
[[ha_manager_fencing]]
Fencing
@@ -752,14 +951,15 @@ After a node failed and its fencing was successful, the CRM tries to
move services from the failed node to nodes which are still online.
The selection of nodes, on which those services gets recovered, is
-influenced by the resource `group` settings, the list of currently active
-nodes, and their respective active service count.
+influenced by the resource `group` settings, HA rules the service is
+specified in, the list of currently active nodes, and their respective
+active service count.
The CRM first builds a set out of the intersection between user selected
nodes (from `group` setting) and available nodes. It then choose the
-subset of nodes with the highest priority, and finally select the node
-with the lowest active service count. This minimizes the possibility
-of an overloaded node.
+subset of nodes with the highest priority, applies the colocation rules,
+and finally select the node with the lowest active service count. This
+minimizes the possibility of an overloaded node.
CAUTION: On node failure, the CRM distributes services to the
remaining nodes. This increases the service count on those nodes, and
@@ -874,8 +1074,8 @@ You can use the manual maintenance mode to mark the node as unavailable for HA
operation, prompting all services managed by HA to migrate to other nodes.
The target nodes for these migrations are selected from the other currently
-available nodes, and determined by the HA group configuration and the configured
-cluster resource scheduler (CRS) mode.
+available nodes, and determined by the HA group configuration, HA rules
+configuration, and the configured cluster resource scheduler (CRS) mode.
During each migration, the original node will be recorded in the HA managers'
state, so that the service can be moved back again automatically once the
maintenance mode is disabled and the node is back online.
@@ -1092,6 +1292,23 @@ The CRS is currently used at the following scheduling points:
new target node for the HA services in that group, matching the adapted
priority constraints.
+- HA rule config changes (always active). If a rule emposes different
+ constraints on the services, the HA stack will use the CRS algorithm to find
+ a new target node for the HA services affected by these rules depending on
+ the type of the new rules:
+
+** Location rules: Identical to HA group config changes.
+
+** Positive Colocation rules (`together`): If a positive colocation rule is
+ created or services are added to an existing positive colocation rule, the
+ HA stack will use the CRS algorithm to ensure that these positively
+ colocated services are moved to a common node.
+
+** Negative Colocation rules (`separate`): If a negative colocation rule is
+ created or services are added to an existing negative colocation rule, the
+ HA stack will use the CRS algorithm to ensure that these negatively
+ colocated services are moved to separate nodes.
+
- HA service stopped -> start transition (opt-in). Requesting that a stopped
service should be started is an good opportunity to check for the best suited
node as per the CRS algorithm, as moving stopped services is cheaper to do
diff --git a/ha-rules-colocation-opts.adoc b/ha-rules-colocation-opts.adoc
new file mode 100644
index 0000000..4340187
--- /dev/null
+++ b/ha-rules-colocation-opts.adoc
@@ -0,0 +1,8 @@
+`affinity`: `<separate | together>` ::
+
+Describes whether the services are supposed to be kept on separate nodes, or are supposed to be kept together on the same node.
+
+`services`: `<type>:<name>{,<type>:<name>}*` ::
+
+List of HA resource IDs. This consists of a list of resource types followed by a resource specific name separated with a colon (example: vm:100,ct:101).
+
diff --git a/ha-rules-location-opts.adoc b/ha-rules-location-opts.adoc
new file mode 100644
index 0000000..603f8db
--- /dev/null
+++ b/ha-rules-location-opts.adoc
@@ -0,0 +1,14 @@
+`nodes`: `<node>[:<pri>]{,<node>[:<pri>]}*` ::
+
+List of cluster node members, where a priority can be given to each node. A resource bound to a group will run on the available nodes with the highest priority. If there are more nodes in the highest priority class, the services will get distributed to those nodes. The priorities have a relative meaning only. The higher the number, the higher the priority.
+
+`services`: `<type>:<name>{,<type>:<name>}*` ::
+
+List of HA resource IDs. This consists of a list of resource types followed by a resource specific name separated with a colon (example: vm:100,ct:101).
+
+`strict`: `<boolean>` ('default =' `0`)::
+
+Describes whether the location rule is mandatory or optional.
+A mandatory location rule makes services be restricted to the defined nodes. If none of the nodes are available, the service will be stopped.
+An optional location rule makes services prefer to be on the defined nodes. If none of the nodes are available, the service may run on any other node.
+
diff --git a/ha-rules-opts.adoc b/ha-rules-opts.adoc
new file mode 100644
index 0000000..58c1bd7
--- /dev/null
+++ b/ha-rules-opts.adoc
@@ -0,0 +1,12 @@
+`comment`: `<string>` ::
+
+HA rule description.
+
+`services`: `<type>:<name>{,<type>:<name>}*` ::
+
+List of HA resource IDs. This consists of a list of resource types followed by a resource specific name separated with a colon (example: vm:100,ct:101).
+
+`state`: `<disabled | enabled>` ('default =' `enabled`)::
+
+State of the HA rule.
+
--
2.39.5
More information about the pve-devel
mailing list