[pve-devel] [PATCH v3 many 00/66] fix #4156: introduce new notification system
Lukas Wagner
l.wagner at proxmox.com
Mon Jul 17 16:59:45 CEST 2023
# Overview
The purpose of this patch series is to overhaul the existing mail
notification infrastructure in Proxmox VE.
The series replaces calls to 'sendmail' with calls to a
new, configurable notification module. The module was designed to
support multiple notification endpoints, 'sendmail' using the system's
sendmail command being the first one. As a proof of the extensibility
of the current approach, the 'gotify' [1] plugin was also implemented.
The patch series also includes groups. They allow to send a notification
to multiple endpoints at the same time. Furthermore, there are filters.
Endpoints and groups can configure filters to determine if a notification
should be sent. For now, filters can only be configured based on notification
severity.
A short summary of what is included in this patch series:
- Sendmail endpoint plugin: uses the system's `sendmail` command
to send - well - mail. The sendmail plugin sends multi-part mails
containing HTML as well as plain text.
- Gotify endpoint plugin: sends a notification to a gotify server
- Groups: As for any notification event one is only able to select a single
target, groups can be created to notify multiply endpoints at the same time
- Filters: Endpoints and groups can also have filtering: The filter
can match on the notification's metadata (only severity for now) to
determine if it will be sent or not. Filters can be easily extended in
the future to match on other structured metadata as well.
- REST API for managing endpoints, groups and filters
- Overhauled GUI for backup jobs/one-off backups - here the use can now
select a notification target
- GUI for configuring the other notification events
(APT, replication, fencing) - here the user can configure *when* and
*where* to send a notification
- Notification rendering based on templates:
From a single template, the system can render notifications to either
plain text or HTML.
# Configuration
For every single notification event (backup jobs, replication, APT, fencing),
it is now possible to configure
a.) whether to send a notification at all
b.) where to send the notification (target)
For backup jobs, this is configured in the respective backup job config in
`jobs.cfg`, everything else is configured using an extended `notify`
configuration key in `datacenter.cfg`:
root at pve:~# cat /etc/pve/datacenter.cfg
keyboard: en-us
notify: target-fencing=foo,fencing=always,package-updates=auto,target-package-updates=bar
root at pve:~# cat /etc/pve/jobs.cfg
vzdump: backup-2e021835-88ea
...
notes-template {{guestname}}
notification-policy always
notification-target mail-to-root
...
Targets (endpoint plugins or groups) and filters are configured in two new
configuration files: `notifications.cfg` and `priv/notifications.cfg` -
the latter is only readable by root and is used to store any sensitive
information for notification endpoint plugins (tokens, passwords, etc.).
These two new configuration files are opaque to the Perl code, it only
interacts with it through the bindings to the Rust implementation.
root at pve:~# cat /etc/pve/notifications.cfg
filter: only-errors
min-severity error
sendmail: sm1
mailto-user root at pam
mailto user1 at example.com
mailto user2 at example.com
filter on-errors
gotify: gt1
filter only-errors
server https://<IP:port>
root at pve:~# cat /etc/pve/priv/notifications.cfg
gotify: gt1
token foobar
# Permissions
The new notification module is fully integrated in the permission system.
For every target or filter, there exists a corresponding ACL path
/mapping/notifications/<name>. A user must have the Mapping.Use permissions
on that path to send the notification/use the filter. Mapping.Modify and
Mapping.Audit are needed for writing/reading a target's/filter's config.
For groups, the user must have the Mapping.Use permission for every
single endpoint included in the group. If a group/endpoint has a filter,
the user must have the Mapping.Use permissions for that as well.
# Backwards Compatibility
Great care was taken so that existing setups work exactly as before.
Also, every new feature of this series is strictly opt-in, meaning that
if no settings are changed, the system *should* behave exactly as before with
regards to sending notifications. The formatting of the notification emails
has changed *slightly*. If any users relied on that, they might require some
changes (e.g. if they scraped the mails to get the results from backup jobs).
# A note to testers
Since this patch series covers a lot of different repos, it's a bit
cumbersome to set it up correctly. Thus, I've built .deb files that you can
just install (staging repos should be configured).
The packages are at `iso/packages/notification` on nasi. I'll try
to rebuild them regularly with my changes rebased on the current master
branches. Holler at me if you think anything is broken or if the packages
do not install properly.
The changes are also available on my staff repos. Development happens on the
`notification` branches, the version posted here corresponds to the
`notification_v3` tag.
The recommend setup to test this is a 3 node cluster (fencing notifications)
with ZFS storage (for replication notifications). For update notifications,
you can use `pvesh create /node/<hostname>/apt/update --notify 1`.
You might need to clear/rm `/var/lib/pve-manager/pkgupdates` first.
# Follow-up work (in no particular order)
- Revisit the template helpers. Once they are considered 'stable', we
could offer an API for users to send notifications themselves.
- Maybe add some fancier styling to the HTML mails?
e.g. a light, but recognizable Proxmox-colored style? Unfortunately
HTML mails have no proper support for CSS and require inline styling
- In the future, the API might be changed/extended so that supports
"registering" notifications.
This also allows us to specify a 'contract' on what properties
are guaranteed to be included with a specific notification.
Consequently, this allows us to:
a.) generate a list of all possible notification sources in the system
b.) add more sophisticated filtering (e.g. match property for
c.) add other endpoints types where a fixed set of metadata fields makes
sense (e.g. webhook)
In my head, using the notification module could look like this
then:
# Global context
my backup_failed_notification = PVE::Notify::register({
'id' => 'backup-failed',
'severity' => 'error',
'properties' => ['host', 'vmlist', 'logs'],
'title' => '{{ host }}: Backup failed'
'body' => <<'EOF'
A backup has failed for the following VMs: {{ vmlist }}
{{ logs }}
EOF
});
# Later, to send the notification:
PVE::Notify::send(target, backup_failed_notification->instantiate({
'host' => 'earth',
'vmlist' => ... ,
'logs' => ... ,
}));
- proxmox-mail-forward could be integrated as well. This would feed
e.g. zfs-zed events into our notification infrastructure. Special
care must be taken to not create recursive notification loops
(e.g. zed sends to root, forwarder uses notification module, a
configured sendmail endpoint sends to root, forwarder uses module
--> loop)
- Maybe add some CLI so that admins can send notifications in
scripts (an API endpoint callable via pvesh might be enough for a
start).
- Add more notification events - the 'chattier' ones should probably be
disabled by default
- Add other endpoints, e.g. webhook, a generic mail endpoint that
sends emails directly via SMTP, etc.
- Integrate the new module into the other products
- for PBS, this should be relatively straight-forward
[1] https://gotify.net/
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=4526
# Changelog
Note: I omitted per-commit changelogs due to already huge complexity managing
this patch series.
Changes since v2:
- Transformed 'channels' into 'groups'. Allow to notify via a single endpoint
or via a group (containing multiple endpoints)
- Dropped some of the features for filters for now, such as sub-filters and
property-matchers. The property matching would require use to stabilize
notification properties, which will be done later. Right now, the
notifications only have the properties required for rendering the
notification template.
- Groups can now also have filters
- Check if a filter/group/endpoint is still used before deleting it
- Ensure that filter/group/endpoint names are unique
- Add new options to datacenter.cfg, allowing to configure the target
and whether to notify at all for all notification events
- Added new GUI panels, one for modifying the new settings in
datacenter.cfg, the other one for adding/modifying/deleting notification
targets and filters
- Integrate notification targets in the backup job UI
- Full integration with the permission system
- Allow users as mail recipients, looking up their email address in the user
configuration file `user.cfg`.
- Look up mail-from and author from datacenter.cfg, if it is not configured
for a sendmail endpoint
- Support proxies for gotify endpoints. Proxy config is read from
datacenter.cfg
- Move `PVE::Notify` to `pve-cluster`, building a new `libpve-notify-perl`
package
- The API now uses the new array type
- Add always-available `mail-to-root` target, which is used as a fallback
if no other target is configured for an event
- Template rendering: Don't HTML-escape if rendering to plain text
- many other minor changes
Changes since v1:
- Some renaming:
- PVE::Notification -> PVE::Notify
- proxmox-notification -> proxmox-notify
- Split configuration for gotify endpoints into a public part in
`notifications.cfg` and a private part for the token in
`priv/notifications.cfg`
- Add template-based notification rendering (`proxmox`), including helpers
for:
- tables
- pretty printed JSON
- duration, timestamps
- byte sizes
- Add notification channels (repo `proxmox`)
- Add API routes for channels, endpoints, filters (implementation in
`proxmox-notify`, glue code in `proxmox-perl-rs` and handler in
`pve-manager`)
- Integrated new notification channels in backup jobs/one-off backups (repo
`pve-manager`)
- Replication/APT/Fencing use an 'anonymous' channel with a temporary
sendmail endpoint, sending mails to `root`
- Added new options for backup jobs
- Reworked git history
Versions of this patch series:
v2: https://lists.proxmox.com/pipermail/pve-devel/2023-May/056927.html
v1: https://lists.proxmox.com/pipermail/pve-devel/2023-March/056445.html
proxmox:
Lukas Wagner (23):
add proxmox-notify crate
notify: preparation for the first endpoint plugin
notify: preparation for the API
notify: api: add API for sending notifications/testing endpoints
notify: add sendmail plugin
notify: api: add API for sendmail endpoints
notify: add gotify endpoint
notify: api: add API for gotify endpoints
notify: add notification groups
notify: api: add API for groups
notify: add notification filter mechanism
notify: api: add API for filters
notify: add template rendering
notify: add example for template rendering
notify: add context
notify: sendmail: allow users as recipients
notify: sendmail: query default author/mailfrom from context
notify: gotify: add proxy support
notify: api: allow to query entities referenced by filter/target
notify: on deletion, check if a filter/endp. is still used by anything
notify: ensure that filter/group/endpoint names are unique
notify: additional logging when sending a notification
notify: add debian packaging
Cargo.toml | 3 +
proxmox-notify/Cargo.toml | 28 ++
proxmox-notify/debian/changelog | 5 +
proxmox-notify/debian/control | 108 ++++
proxmox-notify/debian/copyright | 16 +
proxmox-notify/debian/debcargo.toml | 7 +
proxmox-notify/examples/render.rs | 63 +++
proxmox-notify/src/api/common.rs | 55 +++
proxmox-notify/src/api/filter.rs | 227 +++++++++
proxmox-notify/src/api/gotify.rs | 292 +++++++++++
proxmox-notify/src/api/group.rs | 261 ++++++++++
proxmox-notify/src/api/mod.rs | 372 ++++++++++++++
proxmox-notify/src/api/sendmail.rs | 288 +++++++++++
proxmox-notify/src/config.rs | 103 ++++
proxmox-notify/src/context.rs | 18 +
proxmox-notify/src/endpoints/gotify.rs | 152 ++++++
proxmox-notify/src/endpoints/mod.rs | 4 +
proxmox-notify/src/endpoints/sendmail.rs | 146 ++++++
proxmox-notify/src/filter.rs | 198 ++++++++
proxmox-notify/src/group.rs | 48 ++
proxmox-notify/src/lib.rs | 602 +++++++++++++++++++++++
proxmox-notify/src/renderer/html.rs | 100 ++++
proxmox-notify/src/renderer/mod.rs | 366 ++++++++++++++
proxmox-notify/src/renderer/plaintext.rs | 141 ++++++
proxmox-notify/src/renderer/table.rs | 24 +
proxmox-notify/src/schema.rs | 49 ++
26 files changed, 3676 insertions(+)
create mode 100644 proxmox-notify/Cargo.toml
create mode 100644 proxmox-notify/debian/changelog
create mode 100644 proxmox-notify/debian/control
create mode 100644 proxmox-notify/debian/copyright
create mode 100644 proxmox-notify/debian/debcargo.toml
create mode 100644 proxmox-notify/examples/render.rs
create mode 100644 proxmox-notify/src/api/common.rs
create mode 100644 proxmox-notify/src/api/filter.rs
create mode 100644 proxmox-notify/src/api/gotify.rs
create mode 100644 proxmox-notify/src/api/group.rs
create mode 100644 proxmox-notify/src/api/mod.rs
create mode 100644 proxmox-notify/src/api/sendmail.rs
create mode 100644 proxmox-notify/src/config.rs
create mode 100644 proxmox-notify/src/context.rs
create mode 100644 proxmox-notify/src/endpoints/gotify.rs
create mode 100644 proxmox-notify/src/endpoints/mod.rs
create mode 100644 proxmox-notify/src/endpoints/sendmail.rs
create mode 100644 proxmox-notify/src/filter.rs
create mode 100644 proxmox-notify/src/group.rs
create mode 100644 proxmox-notify/src/lib.rs
create mode 100644 proxmox-notify/src/renderer/html.rs
create mode 100644 proxmox-notify/src/renderer/mod.rs
create mode 100644 proxmox-notify/src/renderer/plaintext.rs
create mode 100644 proxmox-notify/src/renderer/table.rs
create mode 100644 proxmox-notify/src/schema.rs
proxmox-perl-rs:
Lukas Wagner (10):
add PVE::RS::Notify module
notify: add api for sending notifications/testing endpoints
notify: add api for notification groups
notify: add api for sendmail endpoints
notify: add api for gotify endpoints
notify: add api for notification filters
notify: sendmail: support the `mailto-user` parameter
notify: implement context for getting default author/mailfrom
notify: add context for getting http_proxy from datacenter.cfg
notify: add wrapper for `get_referenced_entities`
pve-rs/Cargo.toml | 2 +
pve-rs/Makefile | 1 +
pve-rs/src/lib.rs | 4 +-
pve-rs/src/notify.rs | 555 +++++++++++++++++++++++++++++++++++++++++++
4 files changed, 561 insertions(+), 1 deletion(-)
create mode 100644 pve-rs/src/notify.rs
pve-cluster:
Lukas Wagner (3):
cluster files: add notifications.cfg
datacenter: add APT/fencing/replication notification configuration
add libpve-notify-perl package
debian/control | 9 ++
debian/libpve-notify-perl.docs | 1 +
debian/libpve-notify-perl.install | 1 +
src/PVE/Cluster.pm | 2 +
src/PVE/DataCenterConfig.pm | 63 ++++++++++++-
src/PVE/Makefile | 2 +-
src/PVE/Notify.pm | 145 ++++++++++++++++++++++++++++++
src/pmxcfs/status.c | 2 +
8 files changed, 222 insertions(+), 3 deletions(-)
create mode 100644 debian/libpve-notify-perl.docs
create mode 100644 debian/libpve-notify-perl.install
create mode 100644 src/PVE/Notify.pm
pve-guest-common:
Lukas Wagner (1):
vzdump: add config options for new notification backend
src/PVE/VZDump/Common.pm | 21 +++++++++++++++++++--
1 file changed, 19 insertions(+), 2 deletions(-)
pve-common:
Lukas Wagner (1):
JSONSchema: increase maxLength of config-digest to 64
src/PVE/JSONSchema.pm | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
pve-ha-manager:
Lukas Wagner (1):
manager: send notifications via new notification module
debian/control | 2 ++
src/PVE/HA/Env.pm | 6 ++---
src/PVE/HA/Env/PVE2.pm | 21 +++++++++-------
src/PVE/HA/NodeStatus.pm | 52 ++++++++++++++++++++++++----------------
src/PVE/HA/Sim/Env.pm | 10 ++++++--
5 files changed, 57 insertions(+), 34 deletions(-)
pve-manager:
Lukas Wagner (21):
test: fix names of .PHONY targets
d/control: add dependency to `libpve-notify-perl`
vzdump: send notifications via new notification module
test: rename mail_test.pl to vzdump_notification_test.pl
api: apt: send notification via new notification module
api: replication: send notifications via new notification module
api: prepare api handler module for notification config
api: notification: add api routes for groups
api: notification: add api routes for sendmail endpoints
api: notification: add api routes for gotify endpoints
api: notification: add api routes for filters
api: notification: allow fetching notification targets
api: notification: allow to test targets
api: notification: disallow removing targets if they are used
ui: backup: allow to select notification target for jobs
ui: backup: adapt backup job details to new notification params
ui: backup: allow to set notification-target for one-off backups
ui: allow to configure notification event -> target mapping
ui: add notification target configuration panel
ui: perm path: load notification target/filter acl entries
ui: perm path: increase width of the perm path selector combobox
PVE/API2/APT.pm | 99 +-
PVE/API2/Cluster.pm | 7 +
PVE/API2/Cluster/Makefile | 1 +
PVE/API2/Cluster/Notifications.pm | 1356 +++++++++++++++++
PVE/API2/Replication.pm | 63 +-
PVE/API2/VZDump.pm | 10 +-
PVE/VZDump.pm | 335 ++--
debian/control | 2 +
test/Makefile | 16 +-
...il_test.pl => vzdump_notification_test.pl} | 36 +-
www/manager6/Makefile | 5 +-
www/manager6/data/PermPathStore.js | 26 +-
www/manager6/dc/Backup.js | 84 +-
www/manager6/dc/BackupJobDetail.js | 20 +-
www/manager6/dc/Config.js | 28 +
www/manager6/dc/NotificationEvents.js | 238 +++
www/manager6/form/NotificationModeSelector.js | 8 +
...ector.js => NotificationPolicySelector.js} | 1 +
.../form/NotificationTargetSelector.js | 54 +
www/manager6/form/PermPathSelector.js | 1 +
www/manager6/window/Backup.js | 35 +-
21 files changed, 2190 insertions(+), 235 deletions(-)
create mode 100644 PVE/API2/Cluster/Notifications.pm
rename test/{mail_test.pl => vzdump_notification_test.pl} (62%)
create mode 100644 www/manager6/dc/NotificationEvents.js
create mode 100644 www/manager6/form/NotificationModeSelector.js
rename www/manager6/form/{EmailNotificationSelector.js => NotificationPolicySelector.js} (87%)
create mode 100644 www/manager6/form/NotificationTargetSelector.js
proxmox-widget-toolkit:
Lukas Wagner (5):
notification: add gui for sendmail notification endpoints
notification: add gui for gotify notification endpoints
notification: add gui for notification groups
notification: allow to select filter for notification targets
notification: add ui for managing notification filters
src/Makefile | 8 +
src/Schema.js | 18 ++
src/data/model/NotificationConfig.js | 17 ++
src/form/NotificationFilterSelector.js | 58 +++++
src/panel/GotifyEditPanel.js | 61 +++++
src/panel/NotificationConfigView.js | 315 ++++++++++++++++++++++++
src/panel/NotificationGroupEditPanel.js | 186 ++++++++++++++
src/panel/SendmailEditPanel.js | 149 +++++++++++
src/window/EndpointEditBase.js | 55 +++++
src/window/NotificationFilterEdit.js | 115 +++++++++
10 files changed, 982 insertions(+)
create mode 100644 src/data/model/NotificationConfig.js
create mode 100644 src/form/NotificationFilterSelector.js
create mode 100644 src/panel/GotifyEditPanel.js
create mode 100644 src/panel/NotificationConfigView.js
create mode 100644 src/panel/NotificationGroupEditPanel.js
create mode 100644 src/panel/SendmailEditPanel.js
create mode 100644 src/window/EndpointEditBase.js
create mode 100644 src/window/NotificationFilterEdit.js
pve-docs:
Lukas Wagner (1):
add documentation for the new notification system
notifications.adoc | 159 +++++++++++++++++++++++++++++++++++++++++++
pve-admin-guide.adoc | 2 +
pve-gui.adoc | 2 +
vzdump.adoc | 5 ++
4 files changed, 168 insertions(+)
create mode 100644 notifications.adoc
Summary over all repositories:
80 files changed, 7880 insertions(+), 277 deletions(-)
--
murpp v0.4.0
More information about the pve-devel
mailing list