[pve-devel] [PATCH v2 many 00/52] revamp notifications; smtp endpoints; system mail
Lukas Wagner
l.wagner at proxmox.com
Tue Nov 14 13:59:08 CET 2023
Note: For simplicity, the series merges the three series' into one
large one. Otherwise the cross-deps would have been really messy
to manage on the list.
## Notification revamp:
This series replaces notification filters and groups with notification
matchers. Instead of having a per-notification event target/policy
setting (at the moment stored in datacenter.cfg and jobs.cfg), this
shifts the routing part into the matcher completely.
Config example, I think this demonstrates the principle quite nicely:
sendmail: default-target
mailto-user root at pam
matcher: fencing-for-node
mode all # all match-directives have to match, default
match-field exact:hostname=pve.example.com
match-field exact:type=fencing
target default-target
--> Send all fencing notifications for a certain host to a certain
target.
Right now, there are three different match-directives:
- match-field: exact/regex match for notification metadata fields
- match-severity: match notification severities (info,notice,warning,error)
- match-calender: match notification timestamp
example: match-calendar mon..fri 8-12
The old target/policy based notification was already in the pvetest repository.
Thus we take special care that there is no breakage when the notification
system encounters old settings/configuration keys. It will clean them
out/migrate them if possible.
What I tested:
- Made sure existing notifications continue to work
(replication/fencing in a cluster setup, backups, system updates)
- Made sure that the 'legacy' mailto parameter for backups also works
- Tested the new UI for notification matchers
- Tested whether old config keys for filters and groups break anything
Followup work in the near future:
- UI code for notification matcher config is a bit messy, I will
send a cleanup-patch - main focus right now was to get it working
- Mark 'mailto' in backup jobs as deprecated in UI - while also
migrating automatically to the new system (create an endpoint/matcher
when creating/updating a backup job)
Changes in this series revision:
- Added built-in default config. builtins can be freely modified
(or disabled, if one has no use for them). If changed, the settings
are stored in /etc/pve/notifications.cfg. To reset to defaults, one
can simply delete this entry (from config, via API, via GUI).
- We also return a 'origin' paramter from certain API calls,
which tells us if it is a built-in, a user-created config entry or
modified built-in entry.
- Simplied permission system, only evaluate perms for
/mapping/notifications. Also switch from plural to singular form.
## SMTP Endpoints:
This patch series adds support for a new notification endpoint type,
smtp. As the name suggests, this new endpoint allows PVE to talk
to SMTP server directly, without using the system's MTA (postfix).
On the Rust side, these patches add a new dependency to the `lettre`
crate for SMTP communication. This crate was chosen as it is:
- by far the most popular mailing crate for Rust
- well maintained
- has reasonable dependencies
- has async support, enabling us to asyncify the proxmox-notify
crate at some point, if needed
Tested against:
- the gmail SMTP server
- the posteo SMTP server
- our own webmail SMTP server
This series also required updating the 'lettre' crate since
one of lettre's deps was bumped to a new version by us.
Changes in new, merged patch series:
- Added origin/disabled params
Changes since v3:
- Rebased on top of the matcher-based notification revamp
- Removed 'filter' setting from target configuration
- Pulled in required patches from 'system mail forwarding' patch series
Changes since v2:
- Rebased proxmox-widget-toolkit onto the latest master to avoid
any conflicts.
Changes since v1:
- Rebased on top of [1]
- Added a mechanism for mails forwarded by `proxmox-mail-forward`
These are forwarded inline as "message/rfc822" to avoid having
to rewrite mail headers (otherwise, some SMTP relays might reject the
mail, because the `From` header of the forwarded mail does not match the
mail account)
[1] https://lists.proxmox.com/pipermail/pve-devel/2023-August/058956.html
[2] https://lists.proxmox.com/pipermail/pve-devel/2023-October/059299.html
[3] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059818.html
[4] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059843.html
[5] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059872.html
## System mail forwarding
The aim of this patch series is to adapt `proxmox-mail-forward`
so that it forwards emails that were sent to the local root user
through the `proxmox_notify` crate.
A short summary of the status quo:
Any mail that is sent to the local `root` user is forwarded by
postfix to the `proxmox-mail-forward` binary, which receives the
mail via STDIN. `proxmox-mail-forward` looks up the email address
configured for the `root at pam` user in /etc/{proxmox-backup,pve}/user.cfg
and then forwards the mail to this address by calling `sendmail`
This patch series modifies `proxmox-mail-forward` in the following way:
`proxmox-mail-forward` instantiates the configuration for `proxmox_notify`
by reading `/etc/{proxmox-backup,pve}/notifications.cfg.
The forwarding behavior is the following:
- PVE installed: Use PVE's notifications.cfg
- PBS installed: Use PBS's notifications.cfg if present. If not,
use an empty configuration and add a default sendmail target and
a matcher - this is needed because notifications are not yet
integrated in PBS. In that way, the forwarding behavior is still
the same as before on PBS (forward to root at pam via sendmail).
- PVE/PBS co-installed: Use PVE's config *and* PBS's config.
If PBS's notifications.cfg does not exist,
a default sendmail target will *not* be added, to avoid
forwarding the same mail twice.
For co-installations we assume for now that PVE has a sensible
matcher/target config for forwarded mails.
Changelog:
- Merged series: no changes
- v1 -> v2:
- Rebased
- Apply the same fix for the PVE context as in [1]
- v2 -> v3:
- Rebased on top of matcher-based notification system:
This simplifies proxmox-mail-forward by a great deal, since
notification routing is moved into the matcher. This means
proxmox-mail-forward does not need to read /etc/pve/datacenter.cfg
any more to determine the target for the notification.
[1] https://lists.proxmox.com/pipermail/pve-devel/2023-October/059294.html
[2] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059818.html
[3] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059872.html
[4] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059894.html
[5] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059899.html
[6] https://lists.proxmox.com/pipermail/pve-devel/2023-November/059900.html
debcargo-conf:
Lukas Wagner (2):
cherry-pick chumsky 0.9.2 from debian unstable
update lettre to 0.11.1
src/chumsky/debian/changelog | 5 ++
src/chumsky/debian/copyright | 39 +++++++++++
src/chumsky/debian/copyright.debcargo.hint | 51 ++++++++++++++
src/chumsky/debian/debcargo.toml | 2 +
src/lettre/debian/changelog | 10 +++
.../debian/patches/downgrade_fastrand.patch | 13 ++++
.../debian/patches/downgrade_idna.patch | 13 ++++
src/lettre/debian/patches/downgrade_url.patch | 13 ++++
.../patches/remove_unused_features.patch | 69 ++++++++++---------
src/lettre/debian/patches/series | 4 +-
.../patches/upgrade_quoted_printable.patch | 13 ----
11 files changed, 185 insertions(+), 47 deletions(-)
create mode 100644 src/chumsky/debian/changelog
create mode 100644 src/chumsky/debian/copyright
create mode 100644 src/chumsky/debian/copyright.debcargo.hint
create mode 100644 src/chumsky/debian/debcargo.toml
create mode 100644 src/lettre/debian/patches/downgrade_fastrand.patch
create mode 100644 src/lettre/debian/patches/downgrade_idna.patch
create mode 100644 src/lettre/debian/patches/downgrade_url.patch
delete mode 100644 src/lettre/debian/patches/upgrade_quoted_printable.patch
proxmox:
Lukas Wagner (13):
notify: introduce Error::Generic
notify: factor out notification content into its own type
notify: replace filters and groups with matcher-based system
notify: add calendar matcher
notify: matcher: introduce common trait for match directives
notify: let a matcher always match if it has no matching directives
sys: email: add `forward`
notify: add mechanisms for email message forwarding
notify: add PVE/PBS context
notify: add 'smtp' endpoint
notify: add api for smtp endpoints
notify: add 'disable' parameter for matchers and targets.
notify: add built-in config and 'origin' parameter
Cargo.toml | 2 +
proxmox-notify/Cargo.toml | 11 +-
proxmox-notify/examples/render.rs | 4 +-
proxmox-notify/src/api/common.rs | 6 +-
proxmox-notify/src/api/filter.rs | 231 ---------
proxmox-notify/src/api/gotify.rs | 22 +-
proxmox-notify/src/api/group.rs | 259 ----------
proxmox-notify/src/api/matcher.rs | 265 ++++++++++
proxmox-notify/src/api/mod.rs | 146 ++----
proxmox-notify/src/api/sendmail.rs | 24 +-
proxmox-notify/src/api/smtp.rs | 362 ++++++++++++++
proxmox-notify/src/config.rs | 57 ++-
proxmox-notify/src/context.rs | 21 -
proxmox-notify/src/context/common.rs | 27 +
proxmox-notify/src/context/mod.rs | 43 ++
proxmox-notify/src/context/pbs.rs | 146 ++++++
proxmox-notify/src/context/pve.rs | 98 ++++
proxmox-notify/src/endpoints/common/mail.rs | 24 +
proxmox-notify/src/endpoints/common/mod.rs | 2 +
proxmox-notify/src/endpoints/gotify.rs | 53 +-
proxmox-notify/src/endpoints/mod.rs | 4 +
proxmox-notify/src/endpoints/sendmail.rs | 114 ++---
proxmox-notify/src/endpoints/smtp.rs | 263 ++++++++++
proxmox-notify/src/filter.rs | 193 +------
proxmox-notify/src/group.rs | 40 +-
proxmox-notify/src/lib.rs | 526 +++++++++++---------
proxmox-notify/src/matcher.rs | 500 +++++++++++++++++++
proxmox-notify/src/renderer/mod.rs | 15 +-
proxmox-notify/src/schema.rs | 11 +-
proxmox-sys/src/email.rs | 52 +-
30 files changed, 2341 insertions(+), 1180 deletions(-)
delete mode 100644 proxmox-notify/src/api/filter.rs
delete mode 100644 proxmox-notify/src/api/group.rs
create mode 100644 proxmox-notify/src/api/matcher.rs
create mode 100644 proxmox-notify/src/api/smtp.rs
delete mode 100644 proxmox-notify/src/context.rs
create mode 100644 proxmox-notify/src/context/common.rs
create mode 100644 proxmox-notify/src/context/mod.rs
create mode 100644 proxmox-notify/src/context/pbs.rs
create mode 100644 proxmox-notify/src/context/pve.rs
create mode 100644 proxmox-notify/src/endpoints/common/mail.rs
create mode 100644 proxmox-notify/src/endpoints/common/mod.rs
create mode 100644 proxmox-notify/src/endpoints/smtp.rs
create mode 100644 proxmox-notify/src/matcher.rs
proxmox-perl-rs:
Lukas Wagner (5):
notify: adapt to new matcher-based notification routing
notify: add bindings for smtp API calls
pve-rs: notify: remove notify_context for PVE
notify: add 'disable' parameter
notify: support 'origin' paramter
common/src/notify.rs | 288 +++++++++++++++++++++--------------
pve-rs/Cargo.toml | 2 +-
pve-rs/src/lib.rs | 7 +-
pve-rs/src/notify_context.rs | 117 --------------
4 files changed, 180 insertions(+), 234 deletions(-)
delete mode 100644 pve-rs/src/notify_context.rs
pve-cluster:
Lukas Wagner (1):
notify: adapt to matcher based notification system
src/PVE/Notify.pm | 101 +++++++++++++++++++++-------------------------
1 file changed, 47 insertions(+), 54 deletions(-)
pve-guest-common:
Lukas Wagner (1):
vzdump: deprecate mailto/mailnotification/notification-{target,policy}
src/PVE/VZDump/Common.pm | 16 +++++++---------
1 file changed, 7 insertions(+), 9 deletions(-)
pve-ha-manager:
Lukas Wagner (1):
env: switch to matcher-based notification system
src/PVE/HA/Env/PVE2.pm | 10 ++--------
src/PVE/HA/NodeStatus.pm | 11 +++++++++--
2 files changed, 11 insertions(+), 10 deletions(-)
pve-manager:
Lukas Wagner (12):
api: notification: remove notification groups
api: notification: add new matcher-based notification API
ui: dc: remove unneeded notification events panel
vzdump: adapt to new matcher based notification system
api: apt: adapt to matcher-based notifications
api: replication: adapt to matcher-based notification system
test: fix vzdump notification test
ui: vzdump: remove left-overs from target/policy based notifications
ui: dc: config: show notification panel again
notify: add API routes for smtp endpoints
api: notification: add disable and origin params
api: notification: simplify ACLs for notification
PVE/API2/APT.pm | 27 +-
PVE/API2/Cluster/Notifications.pm | 815 ++++++++++--------
PVE/API2/Replication.pm | 25 +-
PVE/API2/VZDump.pm | 8 +-
PVE/VZDump.pm | 40 +-
test/vzdump_notification_test.pl | 6 +-
www/manager6/Makefile | 4 -
www/manager6/dc/Backup.js | 81 +-
www/manager6/dc/Config.js | 28 +-
www/manager6/dc/NotificationEvents.js | 276 ------
.../form/NotificationPolicySelector.js | 1 -
www/manager6/window/Backup.js | 35 +-
12 files changed, 527 insertions(+), 819 deletions(-)
delete mode 100644 www/manager6/dc/NotificationEvents.js
proxmox-widget-toolkit:
Lukas Wagner (10):
notification ui: add target selector for matcher
notification ui: remove filter setting for targets
notification ui: remove notification groups
notification ui: rename filter to matcher
notification: matcher: add UI for matcher editing
notification ui: unprotected mailto-root target
noficiation: matcher edit: make 'field' an editable combobox
panel: notification: add gui for SMTP endpoints
notification ui: add enable checkbox for targets/matchers
notification ui: add column for 'origin'
src/Makefile | 6 +-
src/Schema.js | 10 +-
src/data/model/NotificationConfig.js | 6 +-
src/form/NotificationFilterSelector.js | 58 --
src/panel/EmailRecipientPanel.js | 88 ++
src/panel/GotifyEditPanel.js | 39 +-
src/panel/NotificationConfigView.js | 80 +-
src/panel/NotificationGroupEditPanel.js | 183 ----
src/panel/SendmailEditPanel.js | 91 +-
src/panel/SmtpEditPanel.js | 204 +++++
src/window/NotificationFilterEdit.js | 109 ---
src/window/NotificationMatcherEdit.js | 1066 +++++++++++++++++++++++
12 files changed, 1482 insertions(+), 458 deletions(-)
delete mode 100644 src/form/NotificationFilterSelector.js
create mode 100644 src/panel/EmailRecipientPanel.js
delete mode 100644 src/panel/NotificationGroupEditPanel.js
create mode 100644 src/panel/SmtpEditPanel.js
delete mode 100644 src/window/NotificationFilterEdit.js
create mode 100644 src/window/NotificationMatcherEdit.js
pve-docs:
Lukas Wagner (5):
notifications: update docs to for matcher-based notifications
notifications: document SMTP endpoints
notifications: document 'comment' option for targets/matchers
notifications: add documentation for system mail forwarding
notifications: change to simplified ACL structure.
notifications.adoc | 332 +++++++++++++++++++++++++++++++++------------
1 file changed, 245 insertions(+), 87 deletions(-)
proxmox-mail-forward:
Lukas Wagner (2):
feed forwarded mails into proxmox_notify
update d/control
Cargo.toml | 6 +-
debian/control | 6 +-
src/main.rs | 255 +++++++++++++++++++++++--------------------------
3 files changed, 125 insertions(+), 142 deletions(-)
Summary over all repositories:
77 files changed, 5150 insertions(+), 3040 deletions(-)
--
murpp v0.4.0
More information about the pve-devel
mailing list