[PVE-User] Ceph and firewalling

[Alexandre DERUMIER]

Hi,

I had this problem with cephfs in the vm mainly, when firewall is stopped (rules are flushed - but existing connections still conntrack), then start again the firewall,

and conntrack put in invalid because it don't have tracked connection sequence when firewall was stopped.

This could happen with previous proxmox release, when /etc/pve/cluster.cfg couldn't be read during restart of pve-cluster I think. (this has been fixed in last pve-firewall).

But to really be sure to not have the problem anymore :

add in /etc/sysctl.conf

net.netfilter.nf_conntrack_tcp_be_liberal = 1


and to get it loaded at boot, add

/etc/modules-load.d/nf_conntrack.conf
nf_conntrack
nf_conntrack_ipv4
nf_conntrack_ipv6



----- Mail original -----
De: "Mark Schouten" <mark at tuxis.nl>
À: "proxmoxve" <pve-user at pve.proxmox.com>
Envoyé: Mercredi 8 Mai 2019 02:35:15
Objet: [PVE-User] Ceph and firewalling

Hi, 

While upgrading two clusters tonight, it seems that the Ceph-cluster gets confused by the updates of tonight. I think it has something to do with the firewall and connection tracking. A restart of ceph-mon on a node seems to work. 

I *think* the issue is that when pve-firewall is upgraded, the conntracktable is emptied, and existing connections are captured by the 'ctstate INVALID'-rule. But it is kinda hard to reproduce. 

If you ask me, the rules for the 'management' ipset should be applied before the conntrack-rules, or am I setting things up incorrectly? 


The following packages are updated in this run: 
root at proxmox01:~# grep upgrade /var/log/dpkg.log 
2019-05-08 02:09:46 upgrade base-files:amd64 9.9+deb9u8 9.9+deb9u9 
2019-05-08 02:09:46 upgrade ceph-mds:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:47 upgrade ceph-mgr:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:48 upgrade ceph-mon:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:49 upgrade ceph:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:49 upgrade ceph-osd:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:51 upgrade ceph-base:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:52 upgrade ceph-common:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:54 upgrade librbd1:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:54 upgrade python-rados:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:54 upgrade python-rbd:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:54 upgrade python-rgw:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:54 upgrade python-ceph:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:54 upgrade python-cephfs:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:54 upgrade libcephfs2:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:54 upgrade librgw2:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:55 upgrade libradosstriper1:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:55 upgrade librados2:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:55 upgrade ceph-fuse:amd64 12.2.11-pve1 12.2.12-pve1 
2019-05-08 02:09:56 upgrade libhttp-daemon-perl:all 6.01-1 6.01-2 
2019-05-08 02:09:56 upgrade libjs-jquery:all 3.1.1-2 3.1.1-2+deb9u1 
2019-05-08 02:09:56 upgrade libmariadbclient18:amd64 10.1.37-0+deb9u1 10.1.38-0+deb9u1 
2019-05-08 02:09:56 upgrade libpng16-16:amd64 1.6.28-1 1.6.28-1+deb9u1 
2019-05-08 02:09:56 upgrade libpq5:amd64 9.6.11-0+deb9u1 9.6.12-0+deb9u1 
2019-05-08 02:09:56 upgrade rsync:amd64 3.1.2-1+deb9u1 3.1.2-1+deb9u2 
2019-05-08 02:09:56 upgrade pve-cluster:amd64 5.0-33 5.0-36 
2019-05-08 02:09:56 upgrade libpve-storage-perl:all 5.0-39 5.0-41 
2019-05-08 02:09:57 upgrade pve-firewall:amd64 3.0-18 3.0-20 
2019-05-08 02:09:57 upgrade pve-ha-manager:amd64 2.0-8 2.0-9 
2019-05-08 02:09:57 upgrade pve-qemu-kvm:amd64 2.12.1-2 2.12.1-3 
2019-05-08 02:09:59 upgrade pve-edk2-firmware:all 1.20181023-1 1.20190312-1 
2019-05-08 02:10:00 upgrade qemu-server:amd64 5.0-47 5.0-50 
2019-05-08 02:10:00 upgrade libpve-common-perl:all 5.0-47 5.0-51 
2019-05-08 02:10:00 upgrade libpve-access-control:amd64 5.1-3 5.1-8 
2019-05-08 02:10:00 upgrade libpve-http-server-perl:all 2.0-12 2.0-13 
2019-05-08 02:10:00 upgrade libssh2-1:amd64 1.7.0-1 1.7.0-1+deb9u1 
2019-05-08 02:10:00 upgrade linux-libc-dev:amd64 4.9.144-3.1 4.9.168-1 
2019-05-08 02:10:08 upgrade pve-kernel-4.15:all 5.3-3 5.4-1 
2019-05-08 02:10:08 upgrade postfix-sqlite:amd64 3.1.9-0+deb9u2 3.1.12-0+deb9u1 
2019-05-08 02:10:08 upgrade postfix:amd64 3.1.9-0+deb9u2 3.1.12-0+deb9u1 
2019-05-08 02:10:10 upgrade proxmox-widget-toolkit:all 1.0-23 1.0-26 
2019-05-08 02:10:10 upgrade pve-container:all 2.0-35 2.0-37 
2019-05-08 02:10:10 upgrade pve-docs:all 5.3-3 5.4-2 
2019-05-08 02:10:11 upgrade pve-i18n:all 1.0-9 1.1-4 
2019-05-08 02:10:11 upgrade pve-xtermjs:amd64 3.10.1-2 3.12.0-1 
2019-05-08 02:10:11 upgrade pve-manager:amd64 5.3-11 5.4-5 
2019-05-08 02:10:11 upgrade proxmox-ve:all 5.3-1 5.4-1 
2019-05-08 02:10:11 upgrade pve-kernel-4.15.18-12-pve:amd64 4.15.18-35 4.15.18-36 
2019-05-08 02:10:19 upgrade python-cryptography:amd64 1.7.1-3 1.7.1-3+deb9u1 
2019-05-08 02:10:19 upgrade unzip:amd64 6.0-21 6.0-21+deb9u1 
2019-05-08 02:10:19 upgrade ruby2.3-dev:amd64 2.3.3-1+deb9u4 2.3.3-1+deb9u6 
2019-05-08 02:10:19 upgrade libruby2.3:amd64 2.3.3-1+deb9u4 2.3.3-1+deb9u6 
2019-05-08 02:10:20 upgrade publicsuffix:all 20181003.1334-0+deb9u1 20190415.1030-0+deb9u1 
2019-05-08 02:10:20 upgrade ruby2.3:amd64 2.3.3-1+deb9u4 2.3.3-1+deb9u6 


-- 
Mark Schouten <mark at tuxis.nl> 
Tuxis, Ede, https://www.tuxis.nl 
T: +31 318 200208 


_______________________________________________ 
pve-user mailing list 
pve-user at pve.proxmox.com 
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user