[pmg-devel] [PATCH pmg-api 0/5] ruledb - improve experience for non-ascii tests and mails

[Stoiko Ivanov]

this patchseries partially fixes #2465 and #2541, two quite often reported
issues, which are causing quite a disappointing experience for users
in non-ascii only environments

the main assumption of the patches are:
* envelope addresses are either ascii or utf-8 (latter only with smtputf8)
* thus we can unconditionally de-/encode envelope addresses for database
  results/lookups
* the matching in the rule-objects will see the relevant parts of the mail
  as properly encoded perl-strings (with multi-byte characters - e.g. the
  euro sign as \x{20ac} instead of \x{e2}\x{82}\x{ac})
(I did a bit of testing to verify them, by e.g. sending an ISO-8859-1
encoded mail and matching for an umlaut in the subject)

While going through the RuleDB classes I remembered, that we have a few
pieces of legacy objects (Attach, ReportSpam, Counter actions) there, and
went ahead with deprecating them (initially I simply deleted them, but
decided to be more cautious and just log the deprecation until 8.0, when
we can drop them explicitly). They cannot be instantiated currently (short
of a direct insert into the database) - but I don't know if they were ever
used in pre 5.0 times in their current form. - patch 2/5.

Out of scope of the series for now:
* utf-8 support in the LDAP subsystem (deployments with a configured LDAP
  profile still won't be able to process smtputf8 mails) - mostly until I
  get around to create test-environment with the appropriate schema for
  having non-ascii mail-addresses
* Domain/Email objects - did not find the time to consider how to store
  them most sensibly (puny-code, utf-8) and if the choice should be
  carried over to all of our 'email' formats (it probably shouldn't)

patches 1/5 and 4/5 address 2 small bugs I ran into while testing

Given that I quite often miss a few fine points or use-cases I'd be very
grateful for some more experimenting/testing!

Stoiko Ivanov (5):
  ruledb: modfield: properly encode field after variable substitution
  ruledb: add deprecation warnings for unused actions
  fix #2541 ruledb: encode relevant values as utf-8 in database
  ruledb: encode e-mail addresses for syslog
  partially fix #2465: handle smtputf8 addresses in the rule-system

 src/PMG/MailQueue.pm            | 10 +++++----
 src/PMG/RuleDB.pm               | 37 ++++++++++++++++++++++++++-------
 src/PMG/RuleDB/Accept.pm        |  2 +-
 src/PMG/RuleDB/Attach.pm        |  7 +++++++
 src/PMG/RuleDB/BCC.pm           | 19 ++++++++++++++---
 src/PMG/RuleDB/Block.pm         |  2 +-
 src/PMG/RuleDB/Counter.pm       |  5 +++++
 src/PMG/RuleDB/Disclaimer.pm    |  2 +-
 src/PMG/RuleDB/Group.pm         |  4 ++--
 src/PMG/RuleDB/MatchField.pm    |  6 +++++-
 src/PMG/RuleDB/MatchFilename.pm |  5 ++++-
 src/PMG/RuleDB/ModField.pm      | 10 ++++++---
 src/PMG/RuleDB/Notify.pm        | 19 ++++++++++++++---
 src/PMG/RuleDB/Quarantine.pm    | 19 ++++++++++++++---
 src/PMG/RuleDB/Remove.pm        | 20 ++++++++++++------
 src/PMG/RuleDB/ReportSpam.pm    |  5 +++++
 src/PMG/RuleDB/Rule.pm          |  2 +-
 src/PMG/RuleDB/Spam.pm          |  5 +++--
 src/PMG/RuleDB/WhoRegex.pm      |  5 ++++-
 src/PMG/Utils.pm                |  5 +++++
 src/bin/pmg-smtp-filter         |  3 ++-
 21 files changed, 150 insertions(+), 42 deletions(-)

-- 
2.30.2