[pve-devel] [PATCH v2 cluster/guest-common/manager/ha-manager/proxmox{, -perl-rs} 00/42] fix #4156: introduce new notification module
Lukas Wagner
l.wagner at proxmox.com
Wed May 24 15:56:07 CEST 2023
The purpose of this patch series is to overhaul the existing mail
notification infrastructure in Proxmox VE.
The series replaces calls to 'sendmail' with calls to a
new, configurable notification module. The module was designed to
support multiple notification endpoints, 'sendmail' using the system's
sendmail command being the first one. As a proof of the extensibility
of the current approach, the 'gotify' [1] plugin was also implemented
in this series.
Concepts:
- Endpoints:
An endpoint is responsible for sending a notification to some external
entity, e.g. by calling `sendmail` to send a mail, or by performing REST
API calls to a gotify server.
Currently, there are two types of endpoints, `sendmail` and
`gotify`.
- Channels:
Logically, channel can be thought of as a 'group of endpoints'. Each
endpoint can be included in one or more channels. If one is using the
notification API to send a notification, a channel has to be specified.
The notification will then be forwarded to all endpoints included in that
channel.
Logically they decouple endpoints from notification senders - for instance,
a backup job configuration would need to contain references to potentially
multiple endpoints, or, a alternatively, always notify via *all* endpoints.
The latter would potentially shift more configuration effort to filters, for
instance if some backup jobs should only notify via *some* endpoints.
I think the group/channel-based approach provides a relatively nice middle
ground.
- Filters:
Every endpoint can also have a filter. Filters allow filtering
notifications based on severity (info, notice, warning, error) or
notification properties (metadata included in a notification, they are
also the base for the template rendering).
Filters allow AND/OR/NOT conditions and using sub-filters to allow
arbitrarily complex filter structures.
Conceptually, the new notification backend consists of three separate parts:
- A new `proxmox-notify` crate, implemented in Rust. The crate contains
the endpoint/filter/channel implementations, configuration parsing/writing
(passed in/out as a string), template rendering, etc.
- Glue code in `proxmox-perl-rs`, in order to be able to make calls to the
`proxmox-notify` crate from Perl
- A light-weight wrapper module `PVE::Notify`, implemented in Perl and
living in `pve-manager` for now. It provides some helper functions and
is responsible for reading/writing the configuration files, passing the
configuration to the Rust part as a string.
As of now, there were four different event sources:
- Backup Jobs/One-off backups
- APT update notifications
- Replication failures
- Node Fencing
As a part of this patch series, all four were switched over to use the new
`PVE::Notify` package to send notifications.
For backup jobs, it is now possible to choose between 'E-Mail' or
'channel-based' notifications.
This was done so that
- we don't break existing configurations where the `mailto` option is set
- there is a shortcut in case somebody really only ever cares about email
notifications.
Under the hood, both use the new notification backend. The 'E-Mail' option
simply creates a temporary channel as well a temporary 'sendmail' endpoint.
Since there is no way to configure endpoints/channels from the GUI yet,
the control field for backup jobs where one can choose between
"E-Mail" and "Channel" based notifications is disabled right now and always
set to email. IMO it felt a bit weird being able to select a notification
without being able to create/configure one from the GUI.
APT/Replication/Node fencing do not yet have a way to configure a notification
channel, so they use the same 'E-Mail' approach, sending mails to `root` via
a temporary channel.
Follow-up work (in no particular order):
- Documentation (once the current approach has been approved)
- Add a GUI/CLI for managing channels/endpoints, later also filters
- Allow configuring a notification channel for APT/Repl/Fencing
- In the future, the API might be changed/extended so that supports
"registering" notifications. This allows us to a.) generate a
list of all possible notification sources in the system b.) allows
users to easily create filters for specific notification events.
In my head, using the notification module could look like this
then:
# Global context
my backup_failed_notification = PVE::Notify::register({
'id' => 'backup-failed',
'severity' => 'error',
'properties' => ['host', 'vmlist', 'logs'],
'title' => '{{ host }}: Backup failed'
'body' => <<'EOF'
A backup has failed for the following VMs: {{ vmlist }}
{{ logs }}
EOF
});
# Later, to send the notification:
PVE::Notify::send(backup_failed_notification->instantiate({
'host' => 'earth',
'vmlist' => ... ,
'logs' => ... ,
}));
- proxmox-mail-forward could be integrated as well. This would feed
e.g. zfs-zed events into our notification infrastructure. Special
care must be taken to not create recursive notification loops
(e.g. zed sends to root, forwarder uses notification module, a
configured sendmail endpoint sends to root, forwarder uses module
--> loop)
- Maybe add some CLI so that admins can send notifications in
scripts (an API endpoint callable via pvesh might be enough for a
start). This should be done once everything is sufficiently stable
(e.g. templating helpers, etc.)
- Add more notification events
- Add other endpoints, e.g. webhook, a generic SMTP, etc.
- Integrate the new module into the other products
[1] https://gotify.net/
[2] https://bugzilla.proxmox.com/show_bug.cgi?id=4526
Changes from v1:
- Some renaming:
- PVE::Notification -> PVE::Notify
- proxmox-notification -> proxmox-notify
- Split configuration for gotify endpoints into a public part in
`notifications.cfg` and a private part for the token in
`priv/notifications.cfg`
- Add template-based notification rendering (`proxmox`), including helpers
for:
- tables
- pretty printed JSON
- duration, timestamps
- byte sizes
- Add notification channels (repo `proxmox`)
- Add API routes for channels, endpoints, filters (implementation in
`proxmox-notify`, glue code in `proxmox-perl-rs` and handler in
`pve-manager`)
- Integrated new notification channels in backup jobs/one-off backups (repo
`pve-manager`)
- Replication/APT/Fencing use an 'anonymous' channel with a temporary
sendmail endpoint, sending mails to `root`
- Added new options for backup jobs
- Reworked git history
Versions of this patch series:
v1: https://lists.proxmox.com/pipermail/pve-devel/2023-March/056445.html
proxmox:
Lukas Wagner (17):
add `proxmox-human-byte` crate
human-byte: move tests to their own sub-module
add proxmox-notify crate
notify: add debian packaging
notify: preparation for the first endpoint plugin
notify: preparation for the API
notify: api: add API for sending notifications/testing endpoints
notify: add notification channels
notify: api: add API for channels
notify: add sendmail plugin
notify: api: add API for sendmail endpoints
notify: add gotify endpoint
notify: api: add API for gotify endpoints
notify: add notification filter mechanism
notify: api: add API for filters
notify: add template rendering
notify: add example for template rendering
Cargo.toml | 4 +
proxmox-human-byte/Cargo.toml | 15 +
proxmox-human-byte/debian/changelog | 5 +
proxmox-human-byte/debian/control | 43 ++
proxmox-human-byte/debian/copyright | 16 +
proxmox-human-byte/debian/debcargo.toml | 7 +
proxmox-human-byte/src/lib.rs | 363 +++++++++++++++
proxmox-notify/Cargo.toml | 28 ++
proxmox-notify/debian/changelog | 5 +
proxmox-notify/debian/control | 31 ++
proxmox-notify/debian/copyright | 16 +
proxmox-notify/debian/debcargo.toml | 7 +
proxmox-notify/examples/render.rs | 63 +++
proxmox-notify/src/api/channel.rs | 253 ++++++++++
proxmox-notify/src/api/common.rs | 46 ++
proxmox-notify/src/api/filter.rs | 366 +++++++++++++++
proxmox-notify/src/api/gotify.rs | 294 ++++++++++++
proxmox-notify/src/api/mod.rs | 111 +++++
proxmox-notify/src/api/sendmail.rs | 263 +++++++++++
proxmox-notify/src/channel.rs | 53 +++
proxmox-notify/src/config.rs | 103 ++++
proxmox-notify/src/endpoints/gotify.rs | 139 ++++++
proxmox-notify/src/endpoints/mod.rs | 4 +
proxmox-notify/src/endpoints/sendmail.rs | 106 +++++
proxmox-notify/src/filter.rs | 498 ++++++++++++++++++++
proxmox-notify/src/lib.rs | 567 +++++++++++++++++++++++
proxmox-notify/src/renderer/html.rs | 100 ++++
proxmox-notify/src/renderer/mod.rs | 359 ++++++++++++++
proxmox-notify/src/renderer/plaintext.rs | 141 ++++++
proxmox-notify/src/renderer/table.rs | 24 +
proxmox-notify/src/schema.rs | 43 ++
31 files changed, 4073 insertions(+)
create mode 100644 proxmox-human-byte/Cargo.toml
create mode 100644 proxmox-human-byte/debian/changelog
create mode 100644 proxmox-human-byte/debian/control
create mode 100644 proxmox-human-byte/debian/copyright
create mode 100644 proxmox-human-byte/debian/debcargo.toml
create mode 100644 proxmox-human-byte/src/lib.rs
create mode 100644 proxmox-notify/Cargo.toml
create mode 100644 proxmox-notify/debian/changelog
create mode 100644 proxmox-notify/debian/control
create mode 100644 proxmox-notify/debian/copyright
create mode 100644 proxmox-notify/debian/debcargo.toml
create mode 100644 proxmox-notify/examples/render.rs
create mode 100644 proxmox-notify/src/api/channel.rs
create mode 100644 proxmox-notify/src/api/common.rs
create mode 100644 proxmox-notify/src/api/filter.rs
create mode 100644 proxmox-notify/src/api/gotify.rs
create mode 100644 proxmox-notify/src/api/mod.rs
create mode 100644 proxmox-notify/src/api/sendmail.rs
create mode 100644 proxmox-notify/src/channel.rs
create mode 100644 proxmox-notify/src/config.rs
create mode 100644 proxmox-notify/src/endpoints/gotify.rs
create mode 100644 proxmox-notify/src/endpoints/mod.rs
create mode 100644 proxmox-notify/src/endpoints/sendmail.rs
create mode 100644 proxmox-notify/src/filter.rs
create mode 100644 proxmox-notify/src/lib.rs
create mode 100644 proxmox-notify/src/renderer/html.rs
create mode 100644 proxmox-notify/src/renderer/mod.rs
create mode 100644 proxmox-notify/src/renderer/plaintext.rs
create mode 100644 proxmox-notify/src/renderer/table.rs
create mode 100644 proxmox-notify/src/schema.rs
proxmox-perl-rs:
Lukas Wagner (7):
log: set default log level to 'info', add product specific logging env
var
add PVE::RS::Notify module
notify: add api for sending notifications/testing endpoints
notify: add api for notification channels
notify: add api for sendmail endpoints
notify: add api for gotify endpoints
notify: add api for notification filters
common/src/logger.rs | 12 +-
pmg-rs/src/lib.rs | 2 +-
pve-rs/Cargo.toml | 1 +
pve-rs/Makefile | 1 +
pve-rs/src/lib.rs | 3 +-
pve-rs/src/notify.rs | 411 +++++++++++++++++++++++++++++++++++++++++++
6 files changed, 426 insertions(+), 4 deletions(-)
create mode 100644 pve-rs/src/notify.rs
pve-cluster:
Lukas Wagner (1):
cluster files: add notifications.cfg
src/PVE/Cluster.pm | 2 ++
src/pmxcfs/status.c | 2 ++
2 files changed, 4 insertions(+)
pve-guest-common:
Lukas Wagner (1):
vzdump: add config options for new notification backend
src/PVE/VZDump/Common.pm | 28 ++++++++++++++++++++++++++--
1 file changed, 26 insertions(+), 2 deletions(-)
pve-manager:
Lukas Wagner (15):
test: fix names of .PHONY targets
add PVE::Notify module
vzdump: send notifications via new notification module
test: rename mail_test.pl to vzdump_notification_test.pl
api: apt: send notification via new notification module
api: replication: send notifications via new notification module
ui: backup: allow to select notification channel for notifications
ui: backup: adapt backup job details to new notification params
ui: backup: allow to set notification-{channel,mode} for one-off
backups
api: prepare api handler module for notification config
api: add api routes for notification channels
api: add api routes for sendmail endpoints
api: add api routes for gotify endpoints
api: add api routes for notification filters
ui: backup: disable notification mode selector for now
PVE/API2/APT.pm | 73 +-
PVE/API2/Cluster.pm | 7 +
PVE/API2/Cluster/Makefile | 1 +
PVE/API2/Cluster/Notifications.pm | 1262 +++++++++++++++++
PVE/API2/Replication.pm | 75 +-
PVE/API2/VZDump.pm | 2 +-
PVE/Makefile | 1 +
PVE/Notify.pm | 84 ++
PVE/VZDump.pm | 323 +++--
test/Makefile | 16 +-
...il_test.pl => vzdump_notification_test.pl} | 36 +-
www/manager6/Makefile | 4 +-
www/manager6/dc/Backup.js | 78 +-
www/manager6/dc/BackupJobDetail.js | 24 +-
.../form/NotificationChannelSelector.js | 47 +
www/manager6/form/NotificationModeSelector.js | 8 +
...ector.js => NotificationPolicySelector.js} | 1 +
www/manager6/window/Backup.js | 35 +-
18 files changed, 1863 insertions(+), 214 deletions(-)
create mode 100644 PVE/API2/Cluster/Notifications.pm
create mode 100644 PVE/Notify.pm
rename test/{mail_test.pl => vzdump_notification_test.pl} (62%)
create mode 100644 www/manager6/form/NotificationChannelSelector.js
create mode 100644 www/manager6/form/NotificationModeSelector.js
rename www/manager6/form/{EmailNotificationSelector.js => NotificationPolicySelector.js} (87%)
pve-ha-manager:
Lukas Wagner (1):
manager: send notifications via new notification module
src/PVE/HA/Env.pm | 6 ++---
src/PVE/HA/Env/PVE2.pm | 27 ++++++++++++++++++---
src/PVE/HA/NodeStatus.pm | 52 ++++++++++++++++++++++++----------------
src/PVE/HA/Sim/Env.pm | 10 ++++++--
4 files changed, 66 insertions(+), 29 deletions(-)
Summary over all repositories:
62 files changed, 6458 insertions(+), 249 deletions(-)
Generated by murpp v0.3.0
--
2.30.2
More information about the pve-devel
mailing list