[PVE-User] isolate node communication

[nada]

hi folks
Our task is to isolate cluster management from virtuals.
Nodes and virtuals (CT/QM) were at the same subnet before.
The following was already isolated by different subnet and VLAN before.
* 10.19.0.0/16 VLAN19 ... 2nd corosync ring
* 10.8.0.0/16 VLAN8 ... independent CEPH (not hyperconv)

===== ISOLATION
I used BRAIN from part 3.3.8 at
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration
MANY thanks for these admin guides !
Really helpful source of PROXMOX community knowledge !

status
* pve-manager/7.3-3/c3928077 (running kernel: 5.15.74-1-pve)
* nodes&virtuals at OLD subnet 10.0.0.0/16 gateway 10.0.0.1
* nodes at NEW subnet=10.34.1.91/16 gateway=10.34.0.1 VLAN=34 (vmbr0.34)
* virtuals will continue to run in OLD subnet 10.0.0.0/16 (vmbr0)

Following config files are from a TESTING cluster 'minimox'
with 3 nodes (mox91,mox92,mox93)

example of host node isolation with
OLD IP 10.0.1.93/16 gateway 10.0.0.1
NEW IP 10.34.1.93/16 gateway 10.34.0.1 VLAN34

before isolation I was able to ping&nmap cluster node from inside of 
CT/QM
after isolation it is NOT possible to ping&nmap cluster node from inside 
of CT/QM ;-)

everything appears to work well
BUT I see a lot of rejected packages at syslog
so is it good ???
or should I do it different way at PRODUCTION cluster ???
BTW when I restarted corosync at ALL isolated nodes once more
there are NO more messages about rejected packets at syslog
any comments are appreciated
Nada

===== INTERFACES

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr1
iface vmbr1 inet static
	address 10.8.3.93/16
	bridge-ports vlan8
	bridge-stp off
	bridge-fd 0
#ceph

auto vmbr2
iface vmbr2 inet static
	address 10.19.0.93/16
	bridge-ports vlan19
	bridge-stp off
	bridge-fd 0
#corosync

auto vmbr0
iface vmbr0 inet static
	address 10.0.1.93/16
	bridge-ports eno1
	bridge-stp off
	bridge-fd 0
	bridge-vlan-aware yes
	bridge-vids 2-4094
#LAN4virtuals

auto vmbr0.34
iface vmbr0.34 inet static
	address 10.34.1.93/16
	gateway 10.34.0.1
#LAN4management

auto vlan8
iface vlan8 inet manual
	vlan-raw-device eno1

auto vlan19
iface vlan19 inet manual
	vlan-raw-device eno1

==== COROSYNC
# cat /etc/pve/corosync.conf
logging {
   debug: off
   to_syslog: yes
}

nodelist {
   node {
     name: mox91
     nodeid: 1
     quorum_votes: 1
     ring0_addr: 10.34.1.91
     ring1_addr: 10.19.0.91
   }
   node {
     name: mox92
     nodeid: 2
     quorum_votes: 1
     ring0_addr: 10.34.1.92
     ring1_addr: 10.19.0.92
   }
   node {
     name: mox93
     nodeid: 3
     quorum_votes: 1
     ring0_addr: 10.34.1.93
     ring1_addr: 10.19.0.93
   }
}

quorum {
   provider: corosync_votequorum
}

totem {
   cluster_name: minimox
   config_version: 16
   interface {
     linknumber: 0
     knet_link_priority: 100
   }
   interface {
     linknumber: 1
     knet_link_priority: 25
   }
   ip_version: ipv4
   link_mode: passive
   secauth: on
   version: 2
}

===== REJECTED packets

# systemctl status corosync.service
● corosync.service - Corosync Cluster Engine
      Loaded: loaded (/lib/systemd/system/corosync.service; enabled; 
vendor preset: enabled)
      Active: active (running) since Mon 2023-06-19 19:14:58 CEST; 19h 
ago
        Docs: man:corosync
              man:corosync.conf
              man:corosync_overview
    Main PID: 1947 (corosync)
       Tasks: 9 (limit: 18927)
      Memory: 136.9M
         CPU: 10min 38.124s
      CGroup: /system.slice/corosync.service
              └─1947 /usr/sbin/corosync -f

Jun 20 14:45:25 mox93 corosync[1947]:   [KNET  ] rx: Packet rejected 
from 10.34.1.92:5405
Jun 20 14:45:25 mox93 corosync[1947]:   [KNET  ] rx: Packet rejected 
from 10.34.1.91:5405
Jun 20 14:45:26 mox93 corosync[1947]:   [KNET  ] rx: Packet rejected 
from 10.34.1.92:5405
Jun 20 14:45:26 mox93 corosync[1947]:   [KNET  ] rx: Packet rejected 
from 10.34.1.91:5405
Jun 20 14:45:27 mox93 corosync[1947]:   [KNET  ] rx: Packet rejected 
from 10.34.1.92:5405
Jun 20 14:45:27 mox93 corosync[1947]:   [KNET  ] rx: Packet rejected 
from 10.34.1.91:5405
Jun 20 14:45:28 mox93 corosync[1947]:   [KNET  ] rx: Packet rejected 
from 10.34.1.92:5405
Jun 20 14:45:28 mox93 corosync[1947]:   [KNET  ] rx: Packet rejected 
from 10.34.1.91:5405
Jun 20 14:45:29 mox93 corosync[1947]:   [KNET  ] rx: Packet rejected 
from 10.34.1.92:5405
Jun 20 14:45:29 mox93 corosync[1947]:   [KNET  ] rx: Packet rejected 
from 10.34.1.91:5405