[pve-devel] pve-firewall : -m conntrack --ctstate INVALID -j DROP is dropping connections on firewall restart

[Alexandre DERUMIER]

Also, another improvement,

it that instead simply dropping invalid, we could reject for tcp/udp

iptables -N INVALID
iptables -A INVALID -j NFLOG --nflog-prefix  "INVALID: "
iptables -A INVALID -p tcp -j REJECT --reject-with tcp-reset 
iptables -A INVALID -p udp -j REJECT 
iptables -A INVALID -j DROP 
iptables -A FORWARD -m conntrack --ctstate INVALID -j INVALID


That's way, my cephfs client get the RST, and reestablish connection. (same for ssh).



----- Mail original -----
De: "Alexandre Derumier" <aderumier at odiso.com>
À: "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Vendredi 15 Février 2019 08:22:31
Objet: Re: [pve-devel] pve-firewall : -m conntrack --ctstate INVALID -j DROP is dropping connections on firewall restart

>>By reverting the 2 rules order, it's working fine 
>> 
>>iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 
>>iptables -A FORWARD -m conntrack --ctstate INVALID -j DROP 

Sorry, the order don't change nothing. (as it's matching on cstate, and we can only have 1 state) 

It seem that I can workaround it, by flushing conntrack (conntrack -F), after flushing the rules, and before readd them 


here a log in invalid drop: 
0 6 - 15/Feb/2019:08:08:25 +0100 DROP: IN=vmbr1v95 OUT=vmbr1v95 PHYSIN=tap322i0 PHYSOUT=bond0.95 MAC=00:1b:17:00:0a:24:ea:d7:b1:1f:1f:8f:08:00 SRC=10.3.95.232 DST=10.5.0.89 LEN=61 TOS=0x00 PREC=0x00 TTL=64 ID=44720 DF PROTO=TCP SPT=58474 DPT=6789 SEQ=3872631874 ACK=733288034 WINDOW=551 ACK PSH 
0 6 - 15/Feb/2019:08:08:31 +0100 DROP: IN=vmbr1v95 OUT=vmbr1v95 PHYSIN=tap322i0 PHYSOUT=bond0.95 MAC=00:1b:17:00:0a:24:ea:d7:b1:1f:1f:8f:08:00 SRC=10.3.95.232 DST=10.5.0.89 LEN=61 TOS=0x00 PREC=0x00 TTL=64 ID=44721 DF PROTO=TCP SPT=58474 DPT=6789 SEQ=3872631883 ACK=733288034 WINDOW=551 ACK FIN 
0 6 - 15/Feb/2019:08:08:33 +0100 DROP: IN=vmbr1v95 OUT=vmbr1v95 PHYSIN=tap322i0 PHYSOUT=bond0.95 MAC=00:1b:17:00:0a:24:ea:d7:b1:1f:1f:8f:08:00 SRC=10.3.95.232 DST=10.5.0.88 LEN=61 TOS=0x00 PREC=0x00 TTL=64 ID=40022 DF PROTO=TCP SPT=59814 DPT=6800 SEQ=3239353171 ACK=1007274272 WINDOW=245 ACK PSH 

(it's an established conection from 10.3.95.232 to 10.5.0.88) 

Maybe because we have old entries in the conntrack ? 



I think than flushing conntrack after remove rules should be fine, 
if we reenable firewall, and vm have already an established connection, 
it should fill the conntrack again. 
(this is the same with vm live migration) 







----- Mail original ----- 
De: "Alexandre Derumier" <aderumier at odiso.com> 
À: "pve-devel" <pve-devel at pve.proxmox.com> 
Envoyé: Jeudi 14 Février 2019 19:47:11 
Objet: [pve-devel] pve-firewall : -m conntrack --ctstate INVALID -j DROP is dropping connections on firewall restart 

Hi, 

I have notice a bug with current firewall implementation, 

with a simple test like: 


1) 
host fw: 
-------- 
iptables -F 
iptables -X 
iptables -A FORWARD -m conntrack --ctstate INVALID -j DROP 
iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 

iptables-save > rules.dump 

2)start vm, an established a connection. (like ssh for example) 

3) iptables -F. (like pve-firewall stop) 

4) restore the rules (iptables-restore -n < rules.dump), (like pve-firewall start) 



Then, 
the connection is dropped with 
iptables -A FORWARD -m conntrack --ctstate INVALID -j DROP 

This is critical for some protocol like cephfs, where this hang the mds connection for 15min by default. (client think it's still connected,) 


By reverting the 2 rules order, it's working fine 

iptables -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 
iptables -A FORWARD -m conntrack --ctstate INVALID -j DROP 


I'm not expert enough with conntrack, but I have see a lot of documentation about this order (ufw firewall, archlinux wiki,....) 

Does somebody have an idea what should be the correct order ? 


(Note that running iptables INPUT inside the vm, is working fine with INVALID first, so maybe it's specific to forwarding) 





_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 

_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel