[pmg-devel] [PATCH pmg-api/gui/docs] enhance greylist (configurable mask + ipv6 support)

[Stoiko Ivanov]

On Wed, 15 Apr 2020 12:53:13 +0200
Stoiko Ivanov <s.ivanov at proxmox.com> wrote:

> This patchset started out as adding support for configuring the netmask used
> for comparing triples of (ipnet,sender,receiver) while greylisting, because
> some cloud providers send out the same mail from different outbound ips
> (from a network which is larger then /19 - e.g. office365)
> 
> While looking through the code it seemed worthwhile to also add support for
> greylisting ipv6 addresses.
> 
> As a sideeffect the use_spf flag in pmgpolicy now also works for ipv6
> addresses.
> 
> One potential caveat compared to the current code is that it now could happen
> that we have 2 triples with the same sender+receiver but different ips added to
> the table in case those 2 ips send the mails to 2 different clusternodes
> within 2 minutes (clustersync intervall).
> 
> I tested the changes in my (limited, but clustered) environment:
> * sending from an ipv6 address not covered by the SPF record with hard fail
> * sending from an ipv6 address covered by the SPF record with hard fail
> * sending from different ipv6 addresses in the same configured network
> * syncing between an updated master and old node
> (all of the tests worked)
One thing I did not test (and which does not work sensibly):
* Downgrade pmg-api to a version without the database change:
  The unconditional `pmgdb init` call in the postinst script dies because
  cgreylist is not a regular table but a view (and even after manually
  'fixing' errors are logged, since the cgreylist view is not updateable).

The one fix that could work somewhat smoothly (inspired by an
off-list discussion with Dominik):
* keep the old cgreylist table as actual table and duplicate the inserts
  from greylist to cgreylist (if possible using a trigger). This would
  support syncing of nodes with the new and the old schema and downgrades
  (after downgrading the next upgrade might remove all data from greylist,
  since it would have expired by then). The only downside is that the
  information is stored twice (we could then get rid of 'cgreylist' with
  the upgrade to PMG 7


one alternative, if we don't mind that syncing between 2 nodes with
different versions is not working smoothly (as in the greylist entries
don't get synrchonized and you get an error every 2 minutes in the log -
mailprocessing is unaffected):
* simply drop the cgreylist table - then on downgrade it would be
  recreated (although empty) and the downgraded version would work.


would be grateful for your input on how to weigh clustersync vs.
downgrades (and the two options above). of course suggestions for
alternatives are also welcome!

in any case - this asks for a v2 (please don't merge yet)


> 
> additionally the first patch for pmg-api fixes a glitch in test_greylist.pl
> (not stopping the pmgpolicy server used for testing when an error is
> encountered)
> 
> pmg-api:
> Stoiko Ivanov (5):
>   test_greylist: exit pmgpolicy on failed test
>   Change greylisting table to use 'inet' for ips
>   pmgpolicy: add IPv6 support
>   greylist: make netmasks configurable
>   add tests for greylisting ipv6
> 
>  src/PMG/Cluster.pm         | 30 +++++++++----
>  src/PMG/Config.pm          | 24 ++++++++++-
>  src/PMG/DBTools.pm         | 86 ++++++++++++++++++++++++++++----------
>  src/bin/pmgpolicy          | 76 ++++++++++++++++++++-------------
>  src/tests/test_greylist.pl | 41 ++++++++++++++++--
>  5 files changed, 192 insertions(+), 65 deletions(-)
> 
> pmg-gui:
> Stoiko Ivanov (1):
>   MailProxyOptions: add greylist enhancements
> 
>  js/MailProxyOptions.js | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
> 
> pmg-docs:
> Stoiko Ivanov (1):
>   add new greylist params gen-pmg.conf.5.-opts.pl
> 
>  gen-pmg.conf.5-opts.pl | 3 +++
>  1 file changed, 3 insertions(+)
>