From f.ebner at proxmox.com Mon Dec 5 10:51:46 2022 From: f.ebner at proxmox.com (Fiona Ebner) Date: Mon, 5 Dec 2022 10:51:46 +0100 Subject: [pmg-devel] [PATCH pmg-api v2] backup: restore: keep directories in /etc/pmg for inotify In-Reply-To: <20221130170727.85325-1-s.ivanov@proxmox.com> References: <20221130170727.85325-1-s.ivanov@proxmox.com> Message-ID: Am 30.11.22 um 18:07 schrieb Stoiko Ivanov: > By wiping the subdirectories in /etc/pmg/, we lose the inotify > watchers upon restore (/etc/pmg itself and thus most configs are > currently handled by the keep_root flag to rmtree) > This can lead to inconsistencies after restoring for parts relying on > config in a subdirectory (e.g. /etc/pmg/pbs/pbs.conf). > > This patch uses File::Find (included in perl-modules-$perlver) to keep > all directories an unlink everything else. > This was chosen for future robustness over keeping an explicit list of > directories to keep, in case a new directory gets added. > > quickly tested with a fifo, chardev, and socket in the directory. > > an alternative approach would be to simply reload pmgdaemon/pmgproxy > upon config-restore, but that feels more likely to miss some > (potentially future) service, expecting inotify to work. > > Reported-by: Fiona Ebner Tested-by: Fiona Ebner > Signed-off-by: Stoiko Ivanov From t.lamprecht at proxmox.com Mon Dec 12 13:18:11 2022 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 12 Dec 2022 13:18:11 +0100 Subject: [pmg-devel] applied: [PATCH pmg-api v2] backup: restore: keep directories in /etc/pmg for inotify In-Reply-To: <20221130170727.85325-1-s.ivanov@proxmox.com> References: <20221130170727.85325-1-s.ivanov@proxmox.com> Message-ID: <718c3996-85c5-2f54-bcd2-af479d71241c@proxmox.com> Am 30/11/2022 um 18:07 schrieb Stoiko Ivanov: > By wiping the subdirectories in /etc/pmg/, we lose the inotify > watchers upon restore (/etc/pmg itself and thus most configs are > currently handled by the keep_root flag to rmtree) > This can lead to inconsistencies after restoring for parts relying on > config in a subdirectory (e.g. /etc/pmg/pbs/pbs.conf). > > This patch uses File::Find (included in perl-modules-$perlver) to keep > all directories an unlink everything else. > This was chosen for future robustness over keeping an explicit list of > directories to keep, in case a new directory gets added. > > quickly tested with a fifo, chardev, and socket in the directory. > > an alternative approach would be to simply reload pmgdaemon/pmgproxy > upon config-restore, but that feels more likely to miss some > (potentially future) service, expecting inotify to work. > > Reported-by: Fiona Ebner > Signed-off-by: Stoiko Ivanov > --- > v1->v2: > * do not track an explicit list of directories, but simply keep all > (as suggested by Fiona) > * use File::Find, since it's present if perl-modules is installed > > I did not put this in a helper by itself in pve-common, because it seems > short/direct enough to not warrant it. > > src/PMG/Backup.pm | 10 ++++++++-- > 1 file changed, 8 insertions(+), 2 deletions(-) > > applied, with Fiona's T-b, thanks! I changed the find call to directly pass the code ref sub, as I found using the "wanted" option not really helpful (name makes it sound like it would have some influence on descending, but it really is just a dumb callback). Also caught ENOENT error explicitly from the unlink invocation, while not _that_ likely IMO still a good practice. From s.ivanov at proxmox.com Tue Dec 13 12:45:48 2022 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Tue, 13 Dec 2022 12:45:48 +0100 Subject: [pmg-devel] [PATCH pmg-docs] pmgqm: add example for systemd.timer edit Message-ID: <20221213114548.6504-1-s.ivanov@proxmox.com> The resetting of OnCalendar can be a bit confusing for users, who are not familiar with the intricacies of systemd overrides. Add a simple example to address a common use-case, and point to the relevant man pages for more details. Signed-off-by: Stoiko Ivanov --- pmgqm.adoc | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/pmgqm.adoc b/pmgqm.adoc index 046c91e..2a73d9f 100644 --- a/pmgqm.adoc +++ b/pmgqm.adoc @@ -44,6 +44,18 @@ The timer can be edited with the command below: systemctl edit pmgspamreport.timer ---- +Keep in mind that just adding another `OnCalendar` event will cause the report +to be sent out additionally at the specified time, if you want to prevent the +default mail at 00:05, you need to reset the `OnCalendar` setting - for example +to only send out the mails at 06:00: +---- +[Timer] +OnCalendar= +OnCalendar=06:00 +---- + +For details see the systemd man pages: `systemd.unit(5)`, `systemd.timer(5)`. + ifdef::manvolnum[] include::pmg-copyright.adoc[] endif::manvolnum[] -- 2.30.2 From s.ivanov at proxmox.com Tue Dec 13 18:01:59 2022 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Tue, 13 Dec 2022 18:01:59 +0100 Subject: [pmg-devel] [PATCH pmg-api] utils: fix mailflow if smtputf8 is disabled Message-ID: <20221213170159.18049-1-s.ivanov@proxmox.com> with the recent addition of smtputf8 support for the rulesystem setups explicitly disabling smtputf8 in postfix got broken. This is mostly noticeable for the spamreports (the receivers are taken from the database and potentially decoded from utf-8, which sets the 'is_utf8' flag, and then tries to use the smtputf8 extension when reinjecting the mail, which fails (since smtputf8 is disabled) Instead of checking for the internal flag explicitly check if the address contains only ascii printable characters (everything excluding controlcharacters - '[\x20-\x7E]') - see https://perldoc.perl.org/perlunifaq#What-is-%22the-UTF8-flag%22? and https://perldoc.perl.org/perlrecharclass#POSIX-Character-Classes reported in our community forum: https://forum.proxmox.com/threads/.119387/ issue is reproducible by setting `smtputf8_enable = no` in postfix main.cf and sending a spamreport using `pmgqm` regular mailflow should not be affected in those setups (as no utf-8 addresses would come into the system) Signed-off-by: Stoiko Ivanov --- src/PMG/Utils.pm | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/PMG/Utils.pm b/src/PMG/Utils.pm index 10193f6..b0a3c52 100644 --- a/src/PMG/Utils.pm +++ b/src/PMG/Utils.pm @@ -247,7 +247,7 @@ sub reinject_mail { my $has_utf8_targets = 0; foreach my $target (@$targets) { - if (utf8::is_utf8($target)) { + if ($target =~ /[^\p{PosixPrint}]/) { $has_utf8_targets = 1; last; } @@ -255,7 +255,7 @@ sub reinject_mail { my $mail_opts = " BODY=8BITMIME"; my $sender_addr; - if (utf8::is_utf8($sender)) { + if ($sender =~ /[^\p{PosixPrint}]/) { $sender_addr = encode('UTF-8', $smtp->_addr($sender)); $mail_opts .= " SMTPUTF8"; } else { @@ -285,7 +285,7 @@ sub reinject_mail { } } - if (utf8::is_utf8($target)) { + if ($sender =~ /[^\p{PosixPrint}]/) { $rcpt_addr = encode('UTF-8', $smtp->_addr($target)); } else { $rcpt_addr = $smtp->_addr($target); -- 2.30.2 From d.csapak at proxmox.com Thu Dec 15 12:10:01 2022 From: d.csapak at proxmox.com (Dominik Csapak) Date: Thu, 15 Dec 2022 12:10:01 +0100 Subject: [pmg-devel] [PATCH pmg-api] rulecache: sort rules additionally by id In-Reply-To: <20221130132054.66009-1-s.ivanov@proxmox.com> References: <20221130132054.66009-1-s.ivanov@proxmox.com> Message-ID: <3560812a-fdf5-fb0d-f329-bf2baf94290f@proxmox.com> On 11/30/22 14:20, Stoiko Ivanov wrote: > When more rules have the same priority currently their order is not > stable - postgres returns them in a stable way, based on their last > changetime - e.g. disabling and reenabling a rule puts it in the front > of evaluation. Sortin by id (the primary key) in addition should make > rule evaluation robust to such updates > > While there is no guarantee of ordering (within the same priority) > unexpected changes in which rule fires can cause confusion (at least > it confused me quite a bit). > > Reported-by: Mira Limbeck > Signed-off-by: Stoiko Ivanov > --- > With this applied I got stable ruleaction executions across rule-changes and > pmg-smtp-filter restarts, but would appreciate some additional test > > src/PMG/RuleCache.pm | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/src/PMG/RuleCache.pm b/src/PMG/RuleCache.pm > index d35e9c1..ebce98e 100644 > --- a/src/PMG/RuleCache.pm > +++ b/src/PMG/RuleCache.pm > @@ -37,7 +37,7 @@ sub new { > my $sth = $dbh->prepare( > "SELECT ID, Name, Priority, Active, Direction FROM Rule " . > "where Active > 0 " . > - "ORDER BY Priority DESC"); > + "ORDER BY Priority, ID DESC"); this sadly loses the 'DESC' ordering for the priority and all rules are now loaded in the reverse order ? > > $sth->execute(); > From d.csapak at proxmox.com Thu Dec 15 12:10:35 2022 From: d.csapak at proxmox.com (Dominik Csapak) Date: Thu, 15 Dec 2022 12:10:35 +0100 Subject: [pmg-devel] [PATCH pmg-api] pmgdb dump: encode ruledata before printing In-Reply-To: <20221130132129.66108-1-s.ivanov@proxmox.com> References: <20221130132129.66108-1-s.ivanov@proxmox.com> Message-ID: <25c1ac4f-5a96-9be3-1dd6-3e74e2a0956b@proxmox.com> Reviewed-by: Dominik Csapak Tested-by: Dominik Csapak From d.csapak at proxmox.com Thu Dec 15 12:17:37 2022 From: d.csapak at proxmox.com (Dominik Csapak) Date: Thu, 15 Dec 2022 12:17:37 +0100 Subject: [pmg-devel] [PATCH pmg-api] utils: fix mailflow if smtputf8 is disabled In-Reply-To: <20221213170159.18049-1-s.ivanov@proxmox.com> References: <20221213170159.18049-1-s.ivanov@proxmox.com> Message-ID: <5acbd21d-ecf2-f96e-d2b6-4a956ed916dc@proxmox.com> the approach seems sensible, but theoretically it can lose the SMTPUTF8 information if neither sender nor recipients contain any non ascii characters and then it's not allowed to send utf8 headers according to rfc6531 section 3.2[0]: --- If the SMTPUTF8 SMTP extension is not offered by the SMTP server, the SMTPUTF8-aware SMTP client MUST NOT transmit an internationalized email address and MUST NOT transmit a mail message containing internationalized mail headers as described in RFC 6532 [RFC6532] at any level within its MIME structure [RFC2045]. --- also one comment inline 0: https://www.rfc-editor.org/rfc/rfc6531#section-3.2 On 12/13/22 18:01, Stoiko Ivanov wrote: > with the recent addition of smtputf8 support for the rulesystem setups > explicitly disabling smtputf8 in postfix got broken. > > This is mostly noticeable for the spamreports (the receivers are taken > from the database and potentially decoded from utf-8, which sets the > 'is_utf8' flag, and then tries to use the smtputf8 extension when > reinjecting the mail, which fails (since smtputf8 is disabled) > > Instead of checking for the internal flag explicitly check if the > address contains only ascii printable characters (everything excluding > controlcharacters - '[\x20-\x7E]') - see > https://perldoc.perl.org/perlunifaq#What-is-%22the-UTF8-flag%22? > and > https://perldoc.perl.org/perlrecharclass#POSIX-Character-Classes > > reported in our community forum: > https://forum.proxmox.com/threads/.119387/ > > issue is reproducible by setting > `smtputf8_enable = no` in postfix main.cf > and sending a spamreport using `pmgqm` > > regular mailflow should not be affected in those setups (as no utf-8 > addresses would come into the system) > > Signed-off-by: Stoiko Ivanov > --- > src/PMG/Utils.pm | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > > diff --git a/src/PMG/Utils.pm b/src/PMG/Utils.pm > index 10193f6..b0a3c52 100644 > --- a/src/PMG/Utils.pm > +++ b/src/PMG/Utils.pm > @@ -247,7 +247,7 @@ sub reinject_mail { > > my $has_utf8_targets = 0; > foreach my $target (@$targets) { > - if (utf8::is_utf8($target)) { > + if ($target =~ /[^\p{PosixPrint}]/) { > $has_utf8_targets = 1; > last; > } > @@ -255,7 +255,7 @@ sub reinject_mail { > > my $mail_opts = " BODY=8BITMIME"; > my $sender_addr; > - if (utf8::is_utf8($sender)) { > + if ($sender =~ /[^\p{PosixPrint}]/) { > $sender_addr = encode('UTF-8', $smtp->_addr($sender)); > $mail_opts .= " SMTPUTF8"; > } else { > @@ -285,7 +285,7 @@ sub reinject_mail { > } > } > > - if (utf8::is_utf8($target)) { > + if ($sender =~ /[^\p{PosixPrint}]/) { > $rcpt_addr = encode('UTF-8', $smtp->_addr($target)); probably c&p error, is_utf8($target) becomes $sender =~ > } else { > $rcpt_addr = $smtp->_addr($target); From s.ivanov at proxmox.com Tue Dec 20 11:57:35 2022 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Tue, 20 Dec 2022 11:57:35 +0100 Subject: [pmg-devel] [PATCH pmg-api v2] rulecache: sort rules additionally by id Message-ID: <20221220105735.21928-1-s.ivanov@proxmox.com> When more rules have the same priority currently their order is not stable - postgres returns them in a stable way, based on their last changetime - e.g. disabling and reenabling a rule puts it in the front of evaluation. Sortin by id (the primary key) in addition should make rule evaluation robust to such updates While there is no guarantee of ordering (within the same priority) unexpected changes in which rule fires can cause confusion (at least it confused me quite a bit). Signed-off-by: Stoiko Ivanov --- v1->v2: * changed the ordering so that the sorting by priority did not get reversed huge thx to Dominik for testing and noticing! [0] [0] https://www.postgresql.org/docs/current/queries-order.html src/PMG/RuleCache.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/PMG/RuleCache.pm b/src/PMG/RuleCache.pm index d35e9c1..b8690ea 100644 --- a/src/PMG/RuleCache.pm +++ b/src/PMG/RuleCache.pm @@ -37,7 +37,7 @@ sub new { my $sth = $dbh->prepare( "SELECT ID, Name, Priority, Active, Direction FROM Rule " . "where Active > 0 " . - "ORDER BY Priority DESC"); + "ORDER BY Priority DESC, ID DESC"); $sth->execute(); -- 2.30.2 From s.ivanov at proxmox.com Wed Dec 21 15:53:43 2022 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Wed, 21 Dec 2022 15:53:43 +0100 Subject: [pmg-devel] [PATCH pmg-api v2] utils: fix mailflow if smtputf8 is disabled Message-ID: <20221221145343.80373-1-s.ivanov@proxmox.com> with the recent addition of smtputf8 support for the rulesystem setups explicitly disabling smtputf8 in postfix got broken. This is mostly noticeable for the spamreports (the receivers are taken from the database and potentially decoded from utf-8, which sets the 'is_utf8' flag, and then tries to use the smtputf8 extension when reinjecting the mail, which fails (since smtputf8 is disabled) Instead of checking for the internal flag, we check for occurence of characters which are not ascii printable (everything excluding controlcharacters - '[\x20-\x7E]') in the envelope-addresses and headers (there also for [\r\n\t], due to searching all headers and folding). - see https://perldoc.perl.org/perlunifaq#What-is-%22the-UTF8-flag%22? and https://perldoc.perl.org/perlrecharclass#POSIX-Character-Classes The only diversion from the requirements in the smptutf8 rfc https://www.rfc-editor.org/rfc/rfc6531 is that we do not check the headers of all parts of a multipart message (think suggested filename for an attachment), but I assume that this should not be an issue in mail-transit the addresses now always get encoded as UTF-8, as this is robust for aascii-only addresses. reported in our community forum: https://forum.proxmox.com/threads/.119387/ issue is reproducible by setting `smtputf8_enable = no` in postfix main.cf and sending a spamreport using `pmgqm` regular mailflow should not be affected in those setups (as no utf-8 addresses would come into the system) Signed-off-by: Stoiko Ivanov --- v1->v2: * as suggested by Dominik (huge thanks for the thorough review and the suggestions!) the (top-level) mail headers are also scanned for non-ascii printable characters (and \n\r\t, since those occur in headers as strings) * put the test in a sub of its own * addresses are now always encoded as utf-8 (since for ascii only addresses this should be identity src/PMG/Utils.pm | 42 +++++++++++++++++++++--------------------- 1 file changed, 21 insertions(+), 21 deletions(-) diff --git a/src/PMG/Utils.pm b/src/PMG/Utils.pm index 10193f6..825b8d9 100644 --- a/src/PMG/Utils.pm +++ b/src/PMG/Utils.pm @@ -221,6 +221,24 @@ sub subst_values_for_header { return $res; } +sub mail_needs_smtputf8 { + my ($entity, $sender, $targets) = @_; + + return 1 if ($sender =~ /[^\p{PosixPrint}]/); + + foreach my $target (@$targets) { + if ($target =~ /[^\p{PosixPrint}]/) { + return 1; + } + } + + if ($entity->head()->as_string() =~ /([^\p{PosixPrint}\n\r\t])/) { + return 1; + } + + return 0; +} + sub reinject_mail { my ($entity, $sender, $targets, $xforward, $me, $params) = @_; @@ -245,23 +263,9 @@ sub reinject_mail { } } - my $has_utf8_targets = 0; - foreach my $target (@$targets) { - if (utf8::is_utf8($target)) { - $has_utf8_targets = 1; - last; - } - } - my $mail_opts = " BODY=8BITMIME"; - my $sender_addr; - if (utf8::is_utf8($sender)) { - $sender_addr = encode('UTF-8', $smtp->_addr($sender)); - $mail_opts .= " SMTPUTF8"; - } else { - $sender_addr = $smtp->_addr($sender); - $mail_opts .= " SMTPUTF8" if $has_utf8_targets; - } + $mail_opts .= " SMTPUTF8" if mail_needs_smtputf8($entity, $sender, $targets); + my $sender_addr = encode('UTF-8', $smtp->_addr($sender)); if (defined($params->{mail})) { my $mailparams = $params->{mail}; @@ -284,12 +288,8 @@ sub reinject_mail { $rcpt_opts .= " $p=$rcptparams->{$p}"; } } + $rcpt_addr = encode('UTF-8', $smtp->_addr($target)); - if (utf8::is_utf8($target)) { - $rcpt_addr = encode('UTF-8', $smtp->_addr($target)); - } else { - $rcpt_addr = $smtp->_addr($target); - } if (!$smtp->_RCPT("TO:" . $rcpt_addr . $rcpt_opts)) { syslog ('err', "smtp error - got: %s %s", $smtp->code, scalar($smtp->message)); die "smtp to: ERROR"; -- 2.30.2 From c.heiss at proxmox.com Thu Dec 22 11:19:40 2022 From: c.heiss at proxmox.com (Christoph Heiss) Date: Thu, 22 Dec 2022 11:19:40 +0100 Subject: [pmg-devel] [PATCH pmg-api] fix #4410: Remove non-null host-bits from CIDR when reading `mynetworks` Message-ID: <20221222101940.3610215-1-c.heiss@proxmox.com> This will simply drop non-null host bits when reading the config file, thus preserving backwards-compatibility. When creating new entries, invalid CIDRs are now rejected to prevent creation of such entries in the future. Signed-off-by: Christoph Heiss --- src/PMG/API2/MyNetworks.pm | 4 ++++ src/PMG/Config.pm | 21 +++++++++++---------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/src/PMG/API2/MyNetworks.pm b/src/PMG/API2/MyNetworks.pm index 975ca2e..aff4041 100644 --- a/src/PMG/API2/MyNetworks.pm +++ b/src/PMG/API2/MyNetworks.pm @@ -3,6 +3,7 @@ package PMG::API2::MyNetworks; use strict; use warnings; use Data::Dumper; +use Net::IP; use PVE::SafeSyslog; use PVE::Tools qw(extract_param); @@ -83,6 +84,9 @@ __PACKAGE__->register_method ({ die "trusted network '$param->{cidr}' already exists\n" if $mynetworks->{$param->{cidr}}; + die "invalid network adress '$param->{cidr}', host-bits must be null\n" + if !Net::IP::ip_normalize($param->{cidr}); + $mynetworks->{$param->{cidr}} = { comment => $param->{comment} // '', }; diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm index 9ba5c76..f03f102 100755 --- a/src/PMG/Config.pm +++ b/src/PMG/Config.pm @@ -730,6 +730,7 @@ use PVE::SafeSyslog; use PVE::Tools qw($IPV4RE $IPV6RE); use PVE::INotify; use PVE::JSONSchema; +use PVE::Network; use PMG::Cluster; use PMG::Utils; @@ -1008,13 +1009,13 @@ sub read_pmg_mynetworks { while (defined(my $line = <$fh>)) { chomp $line; next if $line =~ m/^\s*$/; - if ($line =~ m!^((?:$IPV4RE|$IPV6RE))/(\d+)\s*(?:#(.*)\s*)?$!) { - my ($network, $prefix_size, $comment) = ($1, $2, $3); - my $cidr = "$network/${prefix_size}"; - $mynetworks->{$cidr} = { - cidr => $cidr, - network_address => $network, - prefix_size => $prefix_size, + if ($line =~ m!^((?:$IPV4RE|$IPV6RE)/\d+)\s*(?:#(.*)\s*)?$!) { + my ($cidr, $comment) = ($1, $2); + my $ip = PVE::Network::IP_from_cidr($cidr); + $mynetworks->{$ip->prefix()} = { + cidr => $ip->prefix(), + network_address => $ip->ip(), + prefix_size => $ip->prefixlen(), comment => $comment // '', }; } else { @@ -1336,11 +1337,11 @@ sub get_template_vars { } my $netlist = PVE::INotify::read_file('mynetworks'); - foreach my $cidr (keys %$netlist) { - if ($cidr =~ m/^($IPV6RE)\/(\d+)$/) { + foreach my $ip (values %$netlist) { + if ($ip->{cidr} =~ m/^($IPV6RE)\/(\d+)$/) { $mynetworks->{"[$1]/$2"} = 1; } else { - $mynetworks->{$cidr} = 1; + $mynetworks->{$ip->{cidr}} = 1; } } -- 2.30.2 From s.ivanov at proxmox.com Thu Dec 22 16:25:01 2022 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Thu, 22 Dec 2022 16:25:01 +0100 Subject: [pmg-devel] [PATCH pmg-api] fix #4410: Remove non-null host-bits from CIDR when reading `mynetworks` In-Reply-To: <20221222101940.3610215-1-c.heiss@proxmox.com> References: <20221222101940.3610215-1-c.heiss@proxmox.com> Message-ID: <20221222162501.1cd288c5@rosa.proxmox.com> Huge thanks for addressing this! I like the approach in general - two comments inline: On Thu, 22 Dec 2022 11:19:40 +0100 Christoph Heiss wrote: > This will simply drop non-null host bits when reading the config file, > thus preserving backwards-compatibility. > When creating new entries, invalid CIDRs are now rejected to prevent > creation of such entries in the future. > > Signed-off-by: Christoph Heiss > --- > src/PMG/API2/MyNetworks.pm | 4 ++++ > src/PMG/Config.pm | 21 +++++++++++---------- > 2 files changed, 15 insertions(+), 10 deletions(-) > > diff --git a/src/PMG/API2/MyNetworks.pm b/src/PMG/API2/MyNetworks.pm > index 975ca2e..aff4041 100644 > --- a/src/PMG/API2/MyNetworks.pm > +++ b/src/PMG/API2/MyNetworks.pm > @@ -3,6 +3,7 @@ package PMG::API2::MyNetworks; > use strict; > use warnings; > use Data::Dumper; > +use Net::IP; > > use PVE::SafeSyslog; > use PVE::Tools qw(extract_param); > @@ -83,6 +84,9 @@ __PACKAGE__->register_method ({ > die "trusted network '$param->{cidr}' already exists\n" > if $mynetworks->{$param->{cidr}}; > > + die "invalid network adress '$param->{cidr}', host-bits must be null\n" > + if !Net::IP::ip_normalize($param->{cidr}); > + > $mynetworks->{$param->{cidr}} = { > comment => $param->{comment} // '', > }; > diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm > index 9ba5c76..f03f102 100755 > --- a/src/PMG/Config.pm > +++ b/src/PMG/Config.pm > @@ -730,6 +730,7 @@ use PVE::SafeSyslog; > use PVE::Tools qw($IPV4RE $IPV6RE); > use PVE::INotify; > use PVE::JSONSchema; > +use PVE::Network; > > use PMG::Cluster; > use PMG::Utils; > @@ -1008,13 +1009,13 @@ sub read_pmg_mynetworks { > while (defined(my $line = <$fh>)) { > chomp $line; > next if $line =~ m/^\s*$/; > - if ($line =~ m!^((?:$IPV4RE|$IPV6RE))/(\d+)\s*(?:#(.*)\s*)?$!) { > - my ($network, $prefix_size, $comment) = ($1, $2, $3); > - my $cidr = "$network/${prefix_size}"; > - $mynetworks->{$cidr} = { > - cidr => $cidr, > - network_address => $network, > - prefix_size => $prefix_size, > + if ($line =~ m!^((?:$IPV4RE|$IPV6RE)/\d+)\s*(?:#(.*)\s*)?$!) { > + my ($cidr, $comment) = ($1, $2); > + my $ip = PVE::Network::IP_from_cidr($cidr); this call expands the prefix to full-length - which I wouldn't have noticed for ipv4 - but is quite visible with ipv6: entering `2001:db8::/32` results in `2001:0db8:0000:0000:0000:0000:0000:0000/32` IIUC - Net::IP::ip_compress_prefix($ip->prefix(), $ip->version()) might be an approach - but even that adds the last quad of zeros... If at all possible it would be great to keep the data as the user entered it. (In this case - in all situations where it's actually a valid prefix w/o host-bits set) > + $mynetworks->{$ip->prefix()} = { > + cidr => $ip->prefix(), > + network_address => $ip->ip(), > + prefix_size => $ip->prefixlen(), > comment => $comment // '', > }; > } else { > @@ -1336,11 +1337,11 @@ sub get_template_vars { > } > > my $netlist = PVE::INotify::read_file('mynetworks'); > - foreach my $cidr (keys %$netlist) { > - if ($cidr =~ m/^($IPV6RE)\/(\d+)$/) { > + foreach my $ip (values %$netlist) { why switch here to iterating over the values - and then accessing the cidr field twice, if it's by construction above the same as the key? > + if ($ip->{cidr} =~ m/^($IPV6RE)\/(\d+)$/) { > $mynetworks->{"[$1]/$2"} = 1; > } else { > - $mynetworks->{$cidr} = 1; > + $mynetworks->{$ip->{cidr}} = 1; > } > } > > -- > 2.30.2 > > > > _______________________________________________ > pmg-devel mailing list > pmg-devel at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel > > From d.csapak at proxmox.com Fri Dec 23 13:41:08 2022 From: d.csapak at proxmox.com (Dominik Csapak) Date: Fri, 23 Dec 2022 13:41:08 +0100 Subject: [pmg-devel] applied: [PATCH pmg-api v2] rulecache: sort rules additionally by id In-Reply-To: <20221220105735.21928-1-s.ivanov@proxmox.com> References: <20221220105735.21928-1-s.ivanov@proxmox.com> Message-ID: <76c0d1c7-a0f6-c6cb-2acb-4c03bf539615@proxmox.com> applied, thanks From d.csapak at proxmox.com Fri Dec 23 13:41:18 2022 From: d.csapak at proxmox.com (Dominik Csapak) Date: Fri, 23 Dec 2022 13:41:18 +0100 Subject: [pmg-devel] applied: [PATCH pmg-api v2] utils: fix mailflow if smtputf8 is disabled In-Reply-To: <20221221145343.80373-1-s.ivanov@proxmox.com> References: <20221221145343.80373-1-s.ivanov@proxmox.com> Message-ID: applied, thanks From c.heiss at proxmox.com Tue Dec 27 10:27:32 2022 From: c.heiss at proxmox.com (Christoph Heiss) Date: Tue, 27 Dec 2022 10:27:32 +0100 Subject: [pmg-devel] [PATCH pmg-api] fix #4410: Remove non-null host-bits from CIDR when reading `mynetworks` In-Reply-To: <20221222162501.1cd288c5@rosa.proxmox.com> References: <20221222101940.3610215-1-c.heiss@proxmox.com> <20221222162501.1cd288c5@rosa.proxmox.com> Message-ID: <7d9764c5-59d7-ad7f-e5b1-d08ffc0a414a@proxmox.com> On 12/22/22 16:25, Stoiko Ivanov wrote: > Huge thanks for addressing this! It was a nice opportunity to get into Perl and PMG too :^) > > I like the approach in general - two comments inline: > > On Thu, 22 Dec 2022 11:19:40 +0100 > Christoph Heiss wrote: > [..] >> @@ -1008,13 +1009,13 @@ sub read_pmg_mynetworks { >> while (defined(my $line = <$fh>)) { >> chomp $line; >> next if $line =~ m/^\s*$/; >> - if ($line =~ m!^((?:$IPV4RE|$IPV6RE))/(\d+)\s*(?:#(.*)\s*)?$!) { >> - my ($network, $prefix_size, $comment) = ($1, $2, $3); >> - my $cidr = "$network/${prefix_size}"; >> - $mynetworks->{$cidr} = { >> - cidr => $cidr, >> - network_address => $network, >> - prefix_size => $prefix_size, >> + if ($line =~ m!^((?:$IPV4RE|$IPV6RE)/\d+)\s*(?:#(.*)\s*)?$!) { >> + my ($cidr, $comment) = ($1, $2); >> + my $ip = PVE::Network::IP_from_cidr($cidr); > this call expands the prefix to full-length - which I wouldn't have > noticed for ipv4 - but is quite visible with ipv6: > entering `2001:db8::/32` results in > `2001:0db8:0000:0000:0000:0000:0000:0000/32` > IIUC - Net::IP::ip_compress_prefix($ip->prefix(), $ip->version()) might > be an approach - but even that adds the last quad of zeros... > > If at all possible it would be great to keep the data as the user entered it. > (In this case - in all situations where it's actually a valid prefix w/o > host-bits set) Ack, I didn't really test it all that extensively with IPv6. I'll look into it again and send a v2. > > > >> + $mynetworks->{$ip->prefix()} = { >> + cidr => $ip->prefix(), >> + network_address => $ip->ip(), >> + prefix_size => $ip->prefixlen(), >> comment => $comment // '', >> }; >> } else { >> @@ -1336,11 +1337,11 @@ sub get_template_vars { >> } >> >> my $netlist = PVE::INotify::read_file('mynetworks'); >> - foreach my $cidr (keys %$netlist) { >> - if ($cidr =~ m/^($IPV6RE)\/(\d+)$/) { >> + foreach my $ip (values %$netlist) { > why switch here to iterating over the values - and then accessing the cidr > field twice, if it's by construction above the same as the key? This was a left-over from when working on the code. I'll remove it. > >> + if ($ip->{cidr} =~ m/^($IPV6RE)\/(\d+)$/) { >> $mynetworks->{"[$1]/$2"} = 1; >> } else { >> - $mynetworks->{$cidr} = 1; >> + $mynetworks->{$ip->{cidr}} = 1; >> } >> } >> >> -- >> 2.30.2 >> >> >> >> _______________________________________________ >> pmg-devel mailing list >> pmg-devel at lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel >> >> > From c.heiss at proxmox.com Tue Dec 27 13:29:15 2022 From: c.heiss at proxmox.com (Christoph Heiss) Date: Tue, 27 Dec 2022 13:29:15 +0100 Subject: [pmg-devel] [PATCH pmg-api v2] fix #4410: Remove non-null host-bits from CIDR when reading `mynetworks` Message-ID: <20221227122915.218159-1-c.heiss@proxmox.com> This will simply drop non-null host bits when reading the config file, thus preserving backwards-compatibility. When creating new entries, invalid CIDRs are now rejected to prevent creation of such entries in the future. In the GUI, the entries are displayed as the user entered them (as suggested my Stoiko). This is done by considering /etc/pmg/mynetworks as the "source of truth" - all entries are saved there verbatim, but when writing the postfix config the right ones are picked. Signed-off-by: Christoph Heiss --- src/PMG/API2/MyNetworks.pm | 10 ++++++++-- src/PMG/Config.pm | 15 ++++++++------- 2 files changed, 16 insertions(+), 9 deletions(-) diff --git a/src/PMG/API2/MyNetworks.pm b/src/PMG/API2/MyNetworks.pm index 975ca2e..325e59b 100644 --- a/src/PMG/API2/MyNetworks.pm +++ b/src/PMG/API2/MyNetworks.pm @@ -3,6 +3,7 @@ package PMG::API2::MyNetworks; use strict; use warnings; use Data::Dumper; +use Net::IP; use PVE::SafeSyslog; use PVE::Tools qw(extract_param); @@ -10,6 +11,7 @@ use HTTP::Status qw(:constants); use PVE::JSONSchema qw(get_standard_option); use PVE::RESTHandler; use PVE::INotify; +use PVE::Network; use PMG::Config; @@ -77,13 +79,17 @@ __PACKAGE__->register_method ({ my ($param) = @_; my $code = sub { + die "invalid network adress '$param->{cidr}', host-bits must be null\n" + if !Net::IP::ip_normalize($param->{cidr}); my $mynetworks = PVE::INotify::read_file('mynetworks'); + my $ip = PVE::Network::IP_from_cidr($param->{cidr}); die "trusted network '$param->{cidr}' already exists\n" - if $mynetworks->{$param->{cidr}}; + if $mynetworks->{$ip->prefix()}; - $mynetworks->{$param->{cidr}} = { + $mynetworks->{$ip->prefix()} = { + cidr => $param->{cidr}, comment => $param->{comment} // '', }; diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm index 9ba5c76..a29060b 100755 --- a/src/PMG/Config.pm +++ b/src/PMG/Config.pm @@ -730,6 +730,7 @@ use PVE::SafeSyslog; use PVE::Tools qw($IPV4RE $IPV6RE); use PVE::INotify; use PVE::JSONSchema; +use PVE::Network; use PMG::Cluster; use PMG::Utils; @@ -1008,13 +1009,13 @@ sub read_pmg_mynetworks { while (defined(my $line = <$fh>)) { chomp $line; next if $line =~ m/^\s*$/; - if ($line =~ m!^((?:$IPV4RE|$IPV6RE))/(\d+)\s*(?:#(.*)\s*)?$!) { - my ($network, $prefix_size, $comment) = ($1, $2, $3); - my $cidr = "$network/${prefix_size}"; - $mynetworks->{$cidr} = { + if ($line =~ m!^((?:$IPV4RE|$IPV6RE)/\d+)\s*(?:#(.*)\s*)?$!) { + my ($cidr, $comment) = ($1, $2); + my $ip = PVE::Network::IP_from_cidr($cidr); + $mynetworks->{$ip->prefix()} = { cidr => $cidr, - network_address => $network, - prefix_size => $prefix_size, + network_address => $ip->short(), + prefix_size => $ip->prefixlen(), comment => $comment // '', }; } else { @@ -1032,7 +1033,7 @@ sub write_pmg_mynetworks { foreach my $cidr (sort keys %$mynetworks) { my $data = $mynetworks->{$cidr}; my $comment = $data->{comment} // '*'; - PVE::Tools::safe_print($filename, $fh, "$cidr #$comment\n"); + PVE::Tools::safe_print($filename, $fh, "$data->{cidr} #$comment\n"); } } -- 2.30.2 From s.ivanov at proxmox.com Tue Dec 27 19:21:35 2022 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Tue, 27 Dec 2022 19:21:35 +0100 Subject: [pmg-devel] [PATCH pmg-api v2] fix #4410: Remove non-null host-bits from CIDR when reading `mynetworks` In-Reply-To: <20221227122915.218159-1-c.heiss@proxmox.com> References: <20221227122915.218159-1-c.heiss@proxmox.com> Message-ID: <20221227192135.7f7bdf4f@rosa.proxmox.com> On Tue, 27 Dec 2022 13:29:15 +0100 Christoph Heiss wrote: > This will simply drop non-null host bits when reading the config file, > thus preserving backwards-compatibility. > When creating new entries, invalid CIDRs are now rejected to prevent > creation of such entries in the future. > > In the GUI, the entries are displayed as the user entered them (as > suggested my Stoiko). This is done by considering /etc/pmg/mynetworks > as the "source of truth" - all entries are saved there verbatim, but > when writing the postfix config the right ones are picked. This sounds good! Currently the code breaks the GET,PUT,DELETE api calls for mynetworks (mismatch between the key (which is the 'computed/long string' and the provided parameter (which is the the data from the file) (tested with an ipv6 prefix - creating works, getting/setting/deleting does not work) maybe keep the user-entered value as key - and add the host-bit-clean cidr (address/mask string) as additional field - then get this field when serializing the data in get_template_vars > > Signed-off-by: Christoph Heiss > --- > src/PMG/API2/MyNetworks.pm | 10 ++++++++-- > src/PMG/Config.pm | 15 ++++++++------- > 2 files changed, 16 insertions(+), 9 deletions(-) > > diff --git a/src/PMG/API2/MyNetworks.pm b/src/PMG/API2/MyNetworks.pm > index 975ca2e..325e59b 100644 > --- a/src/PMG/API2/MyNetworks.pm > +++ b/src/PMG/API2/MyNetworks.pm > @@ -3,6 +3,7 @@ package PMG::API2::MyNetworks; > use strict; > use warnings; > use Data::Dumper; > +use Net::IP; > > use PVE::SafeSyslog; > use PVE::Tools qw(extract_param); > @@ -10,6 +11,7 @@ use HTTP::Status qw(:constants); > use PVE::JSONSchema qw(get_standard_option); > use PVE::RESTHandler; > use PVE::INotify; > +use PVE::Network; > > use PMG::Config; > > @@ -77,13 +79,17 @@ __PACKAGE__->register_method ({ > my ($param) = @_; > > my $code = sub { > + die "invalid network adress '$param->{cidr}', host-bits must be null\n" > + if !Net::IP::ip_normalize($param->{cidr}); > > my $mynetworks = PVE::INotify::read_file('mynetworks'); > + my $ip = PVE::Network::IP_from_cidr($param->{cidr}); > > die "trusted network '$param->{cidr}' already exists\n" > - if $mynetworks->{$param->{cidr}}; > + if $mynetworks->{$ip->prefix()}; > > - $mynetworks->{$param->{cidr}} = { > + $mynetworks->{$ip->prefix()} = { > + cidr => $param->{cidr}, > comment => $param->{comment} // '', > }; > > diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm > index 9ba5c76..a29060b 100755 > --- a/src/PMG/Config.pm > +++ b/src/PMG/Config.pm > @@ -730,6 +730,7 @@ use PVE::SafeSyslog; > use PVE::Tools qw($IPV4RE $IPV6RE); > use PVE::INotify; > use PVE::JSONSchema; > +use PVE::Network; > > use PMG::Cluster; > use PMG::Utils; > @@ -1008,13 +1009,13 @@ sub read_pmg_mynetworks { > while (defined(my $line = <$fh>)) { > chomp $line; > next if $line =~ m/^\s*$/; > - if ($line =~ m!^((?:$IPV4RE|$IPV6RE))/(\d+)\s*(?:#(.*)\s*)?$!) { > - my ($network, $prefix_size, $comment) = ($1, $2, $3); > - my $cidr = "$network/${prefix_size}"; > - $mynetworks->{$cidr} = { > + if ($line =~ m!^((?:$IPV4RE|$IPV6RE)/\d+)\s*(?:#(.*)\s*)?$!) { > + my ($cidr, $comment) = ($1, $2); > + my $ip = PVE::Network::IP_from_cidr($cidr); > + $mynetworks->{$ip->prefix()} = { > cidr => $cidr, > - network_address => $network, > - prefix_size => $prefix_size, > + network_address => $ip->short(), the short() method yields IPv4 addresses in a somewhat uncommon format (at least for me -> 10.2.2.0/24 -> '10.2.2') - however at a quick glance it seems that the 'network_address' field is not really used anywhere - so we should probably just drop it. > + prefix_size => $ip->prefixlen(), > comment => $comment // '', > }; > } else { > @@ -1032,7 +1033,7 @@ sub write_pmg_mynetworks { > foreach my $cidr (sort keys %$mynetworks) { > my $data = $mynetworks->{$cidr}; > my $comment = $data->{comment} // '*'; > - PVE::Tools::safe_print($filename, $fh, "$cidr #$comment\n"); > + PVE::Tools::safe_print($filename, $fh, "$data->{cidr} #$comment\n"); > } > } > > -- > 2.30.2 > > > > _______________________________________________ > pmg-devel mailing list > pmg-devel at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel > > From c.heiss at proxmox.com Wed Dec 28 10:31:52 2022 From: c.heiss at proxmox.com (Christoph Heiss) Date: Wed, 28 Dec 2022 10:31:52 +0100 Subject: [pmg-devel] [PATCH pmg-api v2] fix #4410: Remove non-null host-bits from CIDR when reading `mynetworks` In-Reply-To: <20221227192135.7f7bdf4f@rosa.proxmox.com> References: <20221227122915.218159-1-c.heiss@proxmox.com> <20221227192135.7f7bdf4f@rosa.proxmox.com> Message-ID: <20221228093152.r2f5kbhnutaxrszc@maui.proxmox.com> On Tue, Dec 27, 2022 at 07:21:35PM +0100, Stoiko Ivanov wrote: > On Tue, 27 Dec 2022 13:29:15 +0100 > Christoph Heiss wrote: > > > This will simply drop non-null host bits when reading the config file, > > thus preserving backwards-compatibility. > > When creating new entries, invalid CIDRs are now rejected to prevent > > creation of such entries in the future. > > > > In the GUI, the entries are displayed as the user entered them (as > > suggested my Stoiko). This is done by considering /etc/pmg/mynetworks > > as the "source of truth" - all entries are saved there verbatim, but > > when writing the postfix config the right ones are picked. > This sounds good! > Currently the code breaks the GET,PUT,DELETE api calls for mynetworks > (mismatch between the key (which is the 'computed/long string' and the > provided parameter (which is the the data from the file) > > (tested with an ipv6 prefix - creating works, getting/setting/deleting > does not work) Weird, I though I tested at least deleting extensively enough. Well, back to the drawing board. > > maybe keep the user-entered value as key - and add the host-bit-clean cidr > (address/mask string) as additional field - then get this field when > serializing the data in get_template_vars Had that same thought too already, but then a new problem arises - duplicate (IPv6) entries can be created, since the check then just compare the user-entered value. E.g. both 2001:db8::/32 and 2001:db8::0/32 could be created with issue. That's why I chose to use the normalized prefixes as keys. But this was also possible before .. I'll see what I can come up with. > > > > [..] > > > > diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm > > index 9ba5c76..a29060b 100755 > > --- a/src/PMG/Config.pm > > +++ b/src/PMG/Config.pm > > @@ -730,6 +730,7 @@ use PVE::SafeSyslog; > > use PVE::Tools qw($IPV4RE $IPV6RE); > > use PVE::INotify; > > use PVE::JSONSchema; > > +use PVE::Network; > > > > use PMG::Cluster; > > use PMG::Utils; > > @@ -1008,13 +1009,13 @@ sub read_pmg_mynetworks { > > while (defined(my $line = <$fh>)) { > > chomp $line; > > next if $line =~ m/^\s*$/; > > - if ($line =~ m!^((?:$IPV4RE|$IPV6RE))/(\d+)\s*(?:#(.*)\s*)?$!) { > > - my ($network, $prefix_size, $comment) = ($1, $2, $3); > > - my $cidr = "$network/${prefix_size}"; > > - $mynetworks->{$cidr} = { > > + if ($line =~ m!^((?:$IPV4RE|$IPV6RE)/\d+)\s*(?:#(.*)\s*)?$!) { > > + my ($cidr, $comment) = ($1, $2); > > + my $ip = PVE::Network::IP_from_cidr($cidr); > > + $mynetworks->{$ip->prefix()} = { > > cidr => $cidr, > > - network_address => $network, > > - prefix_size => $prefix_size, > > + network_address => $ip->short(), > the short() method yields IPv4 addresses in a somewhat uncommon format (at > least for me -> 10.2.2.0/24 -> '10.2.2') - however at a quick glance it > seems that the 'network_address' field is not really used anywhere - so we > should probably just drop it. Yeah, ->short() also abbreviates IPv4 addresses. At first I had a check differentiating between v4 and v6 and only ->short()'ening v6 addresses. I'll investigate if and where this field is actually used (and prefix_size too, while at it). Removing it would change the API though - is stability / backwards compatibility something we have to be wary of or can I just drop it if it's really not used anywhere? > > > > > + prefix_size => $ip->prefixlen(), > > comment => $comment // '', > > }; > > } else { > > @@ -1032,7 +1033,7 @@ sub write_pmg_mynetworks { > > foreach my $cidr (sort keys %$mynetworks) { > > my $data = $mynetworks->{$cidr}; > > my $comment = $data->{comment} // '*'; > > - PVE::Tools::safe_print($filename, $fh, "$cidr #$comment\n"); > > + PVE::Tools::safe_print($filename, $fh, "$data->{cidr} #$comment\n"); > > } > > } > > > > -- > > 2.30.2 > > > > > > > > _______________________________________________ > > pmg-devel mailing list > > pmg-devel at lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel > > > > > From s.ivanov at proxmox.com Wed Dec 28 10:51:21 2022 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Wed, 28 Dec 2022 10:51:21 +0100 Subject: [pmg-devel] [PATCH pmg-api v2] fix #4410: Remove non-null host-bits from CIDR when reading `mynetworks` In-Reply-To: <20221228093152.r2f5kbhnutaxrszc@maui.proxmox.com> References: <20221227122915.218159-1-c.heiss@proxmox.com> <20221227192135.7f7bdf4f@rosa.proxmox.com> <20221228093152.r2f5kbhnutaxrszc@maui.proxmox.com> Message-ID: <20221228105121.7e3ff1f1@rosa.proxmox.com> On Wed, 28 Dec 2022 10:31:52 +0100 Christoph Heiss wrote: > On Tue, Dec 27, 2022 at 07:21:35PM +0100, Stoiko Ivanov wrote: > > On Tue, 27 Dec 2022 13:29:15 +0100 > > Christoph Heiss wrote: > > > > > This will simply drop non-null host bits when reading the config file, > > > thus preserving backwards-compatibility. > > > When creating new entries, invalid CIDRs are now rejected to prevent > > > creation of such entries in the future. > > > > > > In the GUI, the entries are displayed as the user entered them (as > > > suggested my Stoiko). This is done by considering /etc/pmg/mynetworks > > > as the "source of truth" - all entries are saved there verbatim, but > > > when writing the postfix config the right ones are picked. > > This sounds good! > > Currently the code breaks the GET,PUT,DELETE api calls for mynetworks > > (mismatch between the key (which is the 'computed/long string' and the > > provided parameter (which is the the data from the file) > > > > (tested with an ipv6 prefix - creating works, getting/setting/deleting > > does not work) > Weird, I though I tested at least deleting extensively enough. Well, > back to the drawing board. > > > > > maybe keep the user-entered value as key - and add the host-bit-clean cidr > > (address/mask string) as additional field - then get this field when > > serializing the data in get_template_vars > Had that same thought too already, but then a new problem arises - > duplicate (IPv6) entries can be created, since the check then just > compare the user-entered value. E.g. both 2001:db8::/32 and > 2001:db8::0/32 could be created with issue. That's why I chose to use > the normalized prefixes as keys. > But this was also possible before .. I'll see what I can come up with. One option that came to my mind was ... just keeping the API part and the values in /etc/pmg/mynetworks as they are .. only normalize the prefixes in get_template_vars for use in the postfix config It has the downside of users getting different values there than they entered - but I personally never understood why programs forced me to clear out the host-bits for their use - so I'm not sure if there's any upside to requiring our users to clear that up? > > > > > > > > [..] > > > > > > diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm > > > index 9ba5c76..a29060b 100755 > > > --- a/src/PMG/Config.pm > > > +++ b/src/PMG/Config.pm > > > @@ -730,6 +730,7 @@ use PVE::SafeSyslog; > > > use PVE::Tools qw($IPV4RE $IPV6RE); > > > use PVE::INotify; > > > use PVE::JSONSchema; > > > +use PVE::Network; > > > > > > use PMG::Cluster; > > > use PMG::Utils; > > > @@ -1008,13 +1009,13 @@ sub read_pmg_mynetworks { > > > while (defined(my $line = <$fh>)) { > > > chomp $line; > > > next if $line =~ m/^\s*$/; > > > - if ($line =~ m!^((?:$IPV4RE|$IPV6RE))/(\d+)\s*(?:#(.*)\s*)?$!) { > > > - my ($network, $prefix_size, $comment) = ($1, $2, $3); > > > - my $cidr = "$network/${prefix_size}"; > > > - $mynetworks->{$cidr} = { > > > + if ($line =~ m!^((?:$IPV4RE|$IPV6RE)/\d+)\s*(?:#(.*)\s*)?$!) { > > > + my ($cidr, $comment) = ($1, $2); > > > + my $ip = PVE::Network::IP_from_cidr($cidr); > > > + $mynetworks->{$ip->prefix()} = { > > > cidr => $cidr, > > > - network_address => $network, > > > - prefix_size => $prefix_size, > > > + network_address => $ip->short(), > > the short() method yields IPv4 addresses in a somewhat uncommon format (at > > least for me -> 10.2.2.0/24 -> '10.2.2') - however at a quick glance it > > seems that the 'network_address' field is not really used anywhere - so we > > should probably just drop it. > Yeah, ->short() also abbreviates IPv4 addresses. At first I had a check > differentiating between v4 and v6 and only ->short()'ening v6 addresses. > > I'll investigate if and where this field is actually used (and > prefix_size too, while at it). > > Removing it would change the API though - is stability / backwards > compatibility something we have to be wary of or can I just drop it if > it's really not used anywhere? good point! - we do return it from the API - so let's keep it for now (a 'FIXME: drop network_address and prefix_size with PMG 8.0' comment might make sense though - then we can clear it up with the next major release) > > > > > > > > > > + prefix_size => $ip->prefixlen(), > > > comment => $comment // '', > > > }; > > > } else { > > > @@ -1032,7 +1033,7 @@ sub write_pmg_mynetworks { > > > foreach my $cidr (sort keys %$mynetworks) { > > > my $data = $mynetworks->{$cidr}; > > > my $comment = $data->{comment} // '*'; > > > - PVE::Tools::safe_print($filename, $fh, "$cidr #$comment\n"); > > > + PVE::Tools::safe_print($filename, $fh, "$data->{cidr} #$comment\n"); > > > } > > > } > > > > > > -- > > > 2.30.2 > > > > > > > > > > > > _______________________________________________ > > > pmg-devel mailing list > > > pmg-devel at lists.proxmox.com > > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel > > > > > > > > From c.heiss at proxmox.com Wed Dec 28 11:17:15 2022 From: c.heiss at proxmox.com (Christoph Heiss) Date: Wed, 28 Dec 2022 11:17:15 +0100 Subject: [pmg-devel] [PATCH pmg-api v2] fix #4410: Remove non-null host-bits from CIDR when reading `mynetworks` In-Reply-To: <20221228105121.7e3ff1f1@rosa.proxmox.com> References: <20221227122915.218159-1-c.heiss@proxmox.com> <20221227192135.7f7bdf4f@rosa.proxmox.com> <20221228093152.r2f5kbhnutaxrszc@maui.proxmox.com> <20221228105121.7e3ff1f1@rosa.proxmox.com> Message-ID: <20221228101715.h4hsz25nyr3k33fl@maui.proxmox.com> On Wed, Dec 28, 2022 at 10:51:21AM +0100, Stoiko Ivanov wrote: > [..] > > One option that came to my mind was ... just keeping the API part and the > values in /etc/pmg/mynetworks as they are .. > only normalize the prefixes in get_template_vars for use in the postfix > config > > It has the downside of users getting different values there than they > entered - but I personally never understood why programs forced me to > clear out the host-bits for their use - so I'm not sure if there's any > upside to requiring our users to clear that up? One upside I can think of would be that users cannot _that_ easily typo the prefix, since they need to match up the prefix length to the network address. Then again, we can only do so much if someone puts in wrong values. But otherwise, the above suggestion seems very reasonable to me. I don't think a whole lot of users will read the postfix config directly anyway (why would they?), so IMHO that is a non-issue anyway if the values differ. I'll try it out and send a v3 with the above implemented, maybe it is the better way in the end anyway. > > [..] > > > > Removing it would change the API though - is stability / backwards > > compatibility something we have to be wary of or can I just drop it if > > it's really not used anywhere? > good point! - we do return it from the API - so let's keep it for now > (a 'FIXME: drop network_address and prefix_size with PMG 8.0' comment > might make sense though - then we can clear it up with the next major > release) Ack, I'll just add a comment and leave it as it is for now. From c.heiss at proxmox.com Wed Dec 28 12:52:59 2022 From: c.heiss at proxmox.com (Christoph Heiss) Date: Wed, 28 Dec 2022 12:52:59 +0100 Subject: [pmg-devel] [PATCH pmg-api v3] fix #4410: Remove non-null host bits from CIDR when writing postfix config Message-ID: <20221228115259.215030-1-c.heiss@proxmox.com> This will drop non-null host bits from `mynetworks` CIDRs when writing the `main.cf` postfix template. Backwards-compatibility with old entries in `/etc/pmg/mynetworks` is thus also preserved. Add an additional comment to the mynetworks API, indicating that unused fields can/should be dropped with the next PMG version. No GUI changes. The entries are written to `/etc/pmg/mynetworks` as the user enters them. Suggested by Stoiko, see discussion in v2 thread [0]. [0] https://lists.proxmox.com/pipermail/pmg-devel/2022-December/002247.html Signed-off-by: Christoph Heiss --- Changes v2 -> v3: * Dropped validation of host-bits of new entries on creation * Entries are now again written verbatim to `/etc/pmg/mynetworks` * Host bits are now dropped when writing the postfix template Changes v1 -> v2: * Reverted unneeded loop iterator change * Display CIDRs in GUI as the user entered them src/PMG/Config.pm | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm index 9ba5c76..c702394 100755 --- a/src/PMG/Config.pm +++ b/src/PMG/Config.pm @@ -8,6 +8,7 @@ use Data::Dumper; use PVE::Tools; use PVE::JSONSchema qw(get_standard_option); use PVE::SectionConfig; +use PVE::Network; use base qw(PVE::SectionConfig); @@ -1011,6 +1012,7 @@ sub read_pmg_mynetworks { if ($line =~ m!^((?:$IPV4RE|$IPV6RE))/(\d+)\s*(?:#(.*)\s*)?$!) { my ($network, $prefix_size, $comment) = ($1, $2, $3); my $cidr = "$network/${prefix_size}"; + # FIXME: Drop unused `network_address` and `prefix_size` with PMG 8.0 $mynetworks->{$cidr} = { cidr => $cidr, network_address => $network, @@ -1337,10 +1339,12 @@ sub get_template_vars { my $netlist = PVE::INotify::read_file('mynetworks'); foreach my $cidr (keys %$netlist) { - if ($cidr =~ m/^($IPV6RE)\/(\d+)$/) { - $mynetworks->{"[$1]/$2"} = 1; + my $ip = PVE::Network::IP_from_cidr($cidr); + if ($ip->version() == 4) { + $mynetworks->{$ip->prefix()} = 1; } else { - $mynetworks->{$cidr} = 1; + my $address = '[' . $ip->short() . ']/' . $ip->prefixlen(); + $mynetworks->{$address} = 1; } } -- 2.30.2 From s.ivanov at proxmox.com Wed Dec 28 18:08:43 2022 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Wed, 28 Dec 2022 18:08:43 +0100 Subject: [pmg-devel] [PATCH pmg-api v3] fix #4410: Remove non-null host bits from CIDR when writing postfix config In-Reply-To: <20221228115259.215030-1-c.heiss@proxmox.com> References: <20221228115259.215030-1-c.heiss@proxmox.com> Message-ID: <20221228180843.0c7671b7@rosa.proxmox.com> Looks good and minimal - one tiny nit/improvement: On Wed, 28 Dec 2022 12:52:59 +0100 Christoph Heiss wrote: > This will drop non-null host bits from `mynetworks` CIDRs when writing > the `main.cf` postfix template. > Backwards-compatibility with old entries in `/etc/pmg/mynetworks` is > thus also preserved. > > Add an additional comment to the mynetworks API, indicating that unused > fields can/should be dropped with the next PMG version. > > No GUI changes. The entries are written to `/etc/pmg/mynetworks` as the > user enters them. Suggested by Stoiko, see discussion in v2 thread [0]. > > [0] https://lists.proxmox.com/pipermail/pmg-devel/2022-December/002247.html > > Signed-off-by: Christoph Heiss > --- > > Changes v2 -> v3: > * Dropped validation of host-bits of new entries on creation > * Entries are now again written verbatim to `/etc/pmg/mynetworks` > * Host bits are now dropped when writing the postfix template > > Changes v1 -> v2: > * Reverted unneeded loop iterator change > * Display CIDRs in GUI as the user entered them > > src/PMG/Config.pm | 10 +++++++--- > 1 file changed, 7 insertions(+), 3 deletions(-) > > diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm > index 9ba5c76..c702394 100755 > --- a/src/PMG/Config.pm > +++ b/src/PMG/Config.pm > @@ -8,6 +8,7 @@ use Data::Dumper; > use PVE::Tools; > use PVE::JSONSchema qw(get_standard_option); > use PVE::SectionConfig; > +use PVE::Network; > > use base qw(PVE::SectionConfig); > > @@ -1011,6 +1012,7 @@ sub read_pmg_mynetworks { > if ($line =~ m!^((?:$IPV4RE|$IPV6RE))/(\d+)\s*(?:#(.*)\s*)?$!) { > my ($network, $prefix_size, $comment) = ($1, $2, $3); > my $cidr = "$network/${prefix_size}"; > + # FIXME: Drop unused `network_address` and `prefix_size` with PMG 8.0 > $mynetworks->{$cidr} = { > cidr => $cidr, > network_address => $network, > @@ -1337,10 +1339,12 @@ sub get_template_vars { > > my $netlist = PVE::INotify::read_file('mynetworks'); > foreach my $cidr (keys %$netlist) { > - if ($cidr =~ m/^($IPV6RE)\/(\d+)$/) { > - $mynetworks->{"[$1]/$2"} = 1; > + my $ip = PVE::Network::IP_from_cidr($cidr); this can return undef and we should check for it while our config-parser takes care of many edge-cases and broken cidrs I managed to get passed it with a mask of 148 (max would be 128 for ipv6) I would expect that at least part of our users do edit the config files manually and end up with invalid data there. Probably a `warn` and ignoring the entry might be appropriate here (afaict this is what happens when the mynetworks parser runs into a broken line as well) > + if ($ip->version() == 4) { > + $mynetworks->{$ip->prefix()} = 1; > } else { > - $mynetworks->{$cidr} = 1; > + my $address = '[' . $ip->short() . ']/' . $ip->prefixlen(); > + $mynetworks->{$address} = 1; > } > } > > -- > 2.30.2 > > > > _______________________________________________ > pmg-devel mailing list > pmg-devel at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel > > From c.heiss at proxmox.com Thu Dec 29 10:21:19 2022 From: c.heiss at proxmox.com (Christoph Heiss) Date: Thu, 29 Dec 2022 10:21:19 +0100 Subject: [pmg-devel] [PATCH pmg-api v3] fix #4410: Remove non-null host bits from CIDR when writing postfix config In-Reply-To: <20221228180843.0c7671b7@rosa.proxmox.com> References: <20221228115259.215030-1-c.heiss@proxmox.com> <20221228180843.0c7671b7@rosa.proxmox.com> Message-ID: <20221229092119.ajzs3ujegdtvukig@maui.proxmox.com> On Wed, Dec 28, 2022 at 06:08:43PM +0100, Stoiko Ivanov wrote: > Looks good and minimal - one tiny nit/improvement: > > On Wed, 28 Dec 2022 12:52:59 +0100 > Christoph Heiss wrote: > > > [..] > > @@ -1337,10 +1339,12 @@ sub get_template_vars { > > > > my $netlist = PVE::INotify::read_file('mynetworks'); > > foreach my $cidr (keys %$netlist) { > > - if ($cidr =~ m/^($IPV6RE)\/(\d+)$/) { > > - $mynetworks->{"[$1]/$2"} = 1; > > + my $ip = PVE::Network::IP_from_cidr($cidr); > this can return undef and we should check for it > while our config-parser takes care of many edge-cases and broken cidrs I > managed to get passed it with a mask of 148 (max would be 128 for ipv6) > > I would expect that at least part of our users do edit the config files > manually and end up with invalid data there. > > Probably a `warn` and ignoring the entry might be appropriate here (afaict > this is what happens when the mynetworks parser runs into a broken line as > well) Good point, really did not think of that. Will send a v4 shortly! > > > > + if ($ip->version() == 4) { > > + $mynetworks->{$ip->prefix()} = 1; > > } else { > > - $mynetworks->{$cidr} = 1; > > + my $address = '[' . $ip->short() . ']/' . $ip->prefixlen(); > > + $mynetworks->{$address} = 1; > > } > > } > > > > -- > > 2.30.2 > > > > > > > > _______________________________________________ > > pmg-devel mailing list > > pmg-devel at lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel > > > > > From c.heiss at proxmox.com Thu Dec 29 10:45:17 2022 From: c.heiss at proxmox.com (Christoph Heiss) Date: Thu, 29 Dec 2022 10:45:17 +0100 Subject: [pmg-devel] [PATCH pmg-api v4] fix #4410: Remove non-null host bits from CIDR when writing postfix config Message-ID: <20221229094515.1295216-1-c.heiss@proxmox.com> This will drop non-null host bits from `mynetworks` CIDRs when writing the `main.cf` postfix template. Backwards-compatibility with old entries in `/etc/pmg/mynetworks` is thus also preserved. Add an additional comment to the mynetworks API, indicating that unused fields can/should be dropped with the next PMG version. No GUI changes. The entries are written to `/etc/pmg/mynetworks` as the user enters them. Suggested by Stoiko, see discussion in v2 thread [0]. [0] https://lists.proxmox.com/pipermail/pmg-devel/2022-December/002247.html Signed-off-by: Christoph Heiss --- Changes v3 -> v4: * Added warning on CIDR parse failures Changes v2 -> v3: * Dropped validation of host-bits of new entries on creation * Entries are now again written verbatim to `/etc/pmg/mynetworks` * Host bits are now dropped when writing the postfix template Changes v1 -> v2: * Reverted unneeded loop iterator change * Display CIDRs in GUI as the user entered them src/PMG/Config.pm | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm index 9ba5c76..a0b1866 100755 --- a/src/PMG/Config.pm +++ b/src/PMG/Config.pm @@ -8,6 +8,7 @@ use Data::Dumper; use PVE::Tools; use PVE::JSONSchema qw(get_standard_option); use PVE::SectionConfig; +use PVE::Network; use base qw(PVE::SectionConfig); @@ -1011,6 +1012,7 @@ sub read_pmg_mynetworks { if ($line =~ m!^((?:$IPV4RE|$IPV6RE))/(\d+)\s*(?:#(.*)\s*)?$!) { my ($network, $prefix_size, $comment) = ($1, $2, $3); my $cidr = "$network/${prefix_size}"; + # FIXME: Drop unused `network_address` and `prefix_size` with PMG 8.0 $mynetworks->{$cidr} = { cidr => $cidr, network_address => $network, @@ -1337,10 +1339,15 @@ sub get_template_vars { my $netlist = PVE::INotify::read_file('mynetworks'); foreach my $cidr (keys %$netlist) { - if ($cidr =~ m/^($IPV6RE)\/(\d+)$/) { - $mynetworks->{"[$1]/$2"} = 1; + my $ip = PVE::Network::IP_from_cidr($cidr); + + if (!$ip) { + warn "failed to parse mynetworks entry '$cidr', ignoring\n"; + } elsif ($ip->version() == 4) { + $mynetworks->{$ip->prefix()} = 1; } else { - $mynetworks->{$cidr} = 1; + my $address = '[' . $ip->short() . ']/' . $ip->prefixlen(); + $mynetworks->{$address} = 1; } } -- 2.30.2 From s.ivanov at proxmox.com Thu Dec 29 17:54:16 2022 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Thu, 29 Dec 2022 17:54:16 +0100 Subject: [pmg-devel] applied: [PATCH pmg-api v4] fix #4410: Remove non-null host bits from CIDR when writing postfix config In-Reply-To: <20221229094515.1295216-1-c.heiss@proxmox.com> References: <20221229094515.1295216-1-c.heiss@proxmox.com> Message-ID: <20221229175416.67d85840@rosa.proxmox.com> Thanks for the patch! On Thu, 29 Dec 2022 10:45:17 +0100 Christoph Heiss wrote: > This will drop non-null host bits from `mynetworks` CIDRs when writing > the `main.cf` postfix template. > Backwards-compatibility with old entries in `/etc/pmg/mynetworks` is > thus also preserved. > > Add an additional comment to the mynetworks API, indicating that unused > fields can/should be dropped with the next PMG version. > > No GUI changes. The entries are written to `/etc/pmg/mynetworks` as the > user enters them. Suggested by Stoiko, see discussion in v2 thread [0]. > > [0] https://lists.proxmox.com/pipermail/pmg-devel/2022-December/002247.html > > Signed-off-by: Christoph Heiss > --- > > Changes v3 -> v4: > * Added warning on CIDR parse failures > > Changes v2 -> v3: > * Dropped validation of host-bits of new entries on creation > * Entries are now again written verbatim to `/etc/pmg/mynetworks` > * Host bits are now dropped when writing the postfix template > > Changes v1 -> v2: > * Reverted unneeded loop iterator change > * Display CIDRs in GUI as the user entered them > > src/PMG/Config.pm | 13 ++++++++++--- > 1 file changed, 10 insertions(+), 3 deletions(-) > > diff --git a/src/PMG/Config.pm b/src/PMG/Config.pm > index 9ba5c76..a0b1866 100755 > --- a/src/PMG/Config.pm > +++ b/src/PMG/Config.pm > @@ -8,6 +8,7 @@ use Data::Dumper; > use PVE::Tools; > use PVE::JSONSchema qw(get_standard_option); > use PVE::SectionConfig; > +use PVE::Network; > > use base qw(PVE::SectionConfig); > > @@ -1011,6 +1012,7 @@ sub read_pmg_mynetworks { > if ($line =~ m!^((?:$IPV4RE|$IPV6RE))/(\d+)\s*(?:#(.*)\s*)?$!) { > my ($network, $prefix_size, $comment) = ($1, $2, $3); > my $cidr = "$network/${prefix_size}"; > + # FIXME: Drop unused `network_address` and `prefix_size` with PMG 8.0 > $mynetworks->{$cidr} = { > cidr => $cidr, > network_address => $network, > @@ -1337,10 +1339,15 @@ sub get_template_vars { > > my $netlist = PVE::INotify::read_file('mynetworks'); > foreach my $cidr (keys %$netlist) { > - if ($cidr =~ m/^($IPV6RE)\/(\d+)$/) { > - $mynetworks->{"[$1]/$2"} = 1; > + my $ip = PVE::Network::IP_from_cidr($cidr); > + > + if (!$ip) { > + warn "failed to parse mynetworks entry '$cidr', ignoring\n"; > + } elsif ($ip->version() == 4) { > + $mynetworks->{$ip->prefix()} = 1; > } else { > - $mynetworks->{$cidr} = 1; > + my $address = '[' . $ip->short() . ']/' . $ip->prefixlen(); > + $mynetworks->{$address} = 1; > } > } > > -- > 2.30.2 > > > > _______________________________________________ > pmg-devel mailing list > pmg-devel at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pmg-devel > > From t.lamprecht at proxmox.com Fri Dec 30 15:18:48 2022 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Fri, 30 Dec 2022 15:18:48 +0100 Subject: [pmg-devel] applied: [PATCH pmg-api] pmgdb dump: encode ruledata before printing In-Reply-To: <20221130132129.66108-1-s.ivanov@proxmox.com> References: <20221130132129.66108-1-s.ivanov@proxmox.com> Message-ID: Am 30/11/2022 um 14:21 schrieb Stoiko Ivanov: > was overlooked with the utf-8 support for rules and objects > > this patch prevents a "Wide character in print at .." when dumping the > ruledata > > Reported-by: Dominik Csapak > Signed-off-by: Stoiko Ivanov > --- > src/PMG/CLI/pmgdb.pm | 28 ++++++++++++++++------------ > 1 file changed, 16 insertions(+), 12 deletions(-) > > for the record: this got already applied by Stoiko with Dominik's R-b and T-b trailers https://git.proxmox.com/?p=pmg-api.git;a=commit;h=99447a919dfc019eaa3a9e413fb9f60af5313711 From t.lamprecht at proxmox.com Fri Dec 30 15:25:28 2022 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Fri, 30 Dec 2022 15:25:28 +0100 Subject: [pmg-devel] applied: [PATCH pmg-docs] pmgqm: add example for systemd.timer edit In-Reply-To: <20221213114548.6504-1-s.ivanov@proxmox.com> References: <20221213114548.6504-1-s.ivanov@proxmox.com> Message-ID: <26d0b23d-7616-d90f-dafb-1775a3f5e5ec@proxmox.com> Am 13/12/2022 um 12:45 schrieb Stoiko Ivanov: > The resetting of OnCalendar can be a bit confusing for users, who are > not familiar with the intricacies of systemd overrides. > Add a simple example to address a common use-case, and point to the > relevant man pages for more details. > > Signed-off-by: Stoiko Ivanov > --- > pmgqm.adoc | 12 ++++++++++++ > 1 file changed, 12 insertions(+) > > applied, thanks!