[pve-devel] [PATCH firewall] make nfct_catch non-blocking

[Alexandre DERUMIER]

>>Did you test it without modifying the patch or only with your syslog change?

I have only change my rsyslog config to drop pve-firewall log

if $programname == 'pve-firewall' then
        stop


----- Mail original -----
De: "David Limbeck" <d.limbeck at proxmox.com>
À: "Alexandre Derumier" <aderumier at odiso.com>, "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Lundi 4 Février 2019 14:45:14
Objet: Re: [pve-devel] [PATCH firewall] make nfct_catch non-blocking

Hi, 

Did you test it without modifying the patch or only with your syslog change? 

On 1/30/19 2:31 PM, Alexandre DERUMIER wrote: 
> Hi, 
> 
> I have done some tests, and can't reproduce it. 
> 
> I wonder if it could be related to syslog, the only thing I have change, is dropping pve-firewall log in rsyslog. 
> 
> 218 // also log to syslog 
> 219 
> 220 vsyslog(loglevel, fmt, ap2); 
> 
> 
> It's quite possible than /dev/log was overloaded with the rate, rsyslog was not able to spool it. (I also forward log to central syslog with tcp, could be related). 
> I known if /dev/log buffer is full, syslog call are blocking. 
> 
> don't known how vsyslog() is working in this case. 
> 
> Could it be possible to have an option to disable syslog logging ? (or maybe add an option to use udp to send mail). 
> 
> 
> Also, I have notice that we don't have timestamp in pve-firewall.log for conntrack log. 
> and maybe could we log them in a separate file ? (not sure how the gui will react if we need to filter a vm log, with the rate of new log coming) 
> 
> 
> 
> ----- Mail original ----- 
> De: "aderumier" <aderumier at odiso.com> 
> À: "David Limbeck" <d.limbeck at proxmox.com> 
> Cc: "pve-devel" <pve-devel at pve.proxmox.com> 
> Envoyé: Samedi 26 Janvier 2019 08:07:43 
> Objet: Re: [pve-devel] [PATCH firewall] make nfct_catch non-blocking 
> 
> Thanks ! 
> 
> I'll test it Monday. 
> 
> ----- Mail original ----- 
> De: "David Limbeck" <d.limbeck at proxmox.com> 
> À: "aderumier" <aderumier at odiso.com>, "Wolfgang Bumiller" <w.bumiller at proxmox.com> 
> Cc: "pve-devel" <pve-devel at pve.proxmox.com> 
> Envoyé: Vendredi 25 Janvier 2019 14:31:30 
> Objet: Re: [pve-devel] [PATCH firewall] make nfct_catch non-blocking 
> 
> Hi, 
> 
> A new commit was pushed that enables building of debug symbols for 
> pve-firewall. Please build and install it again with that commit 
> included and run it again. 
> 
> This might help narrow it down some more. 
> 
> On 1/14/19 11:42 AM, Alexandre DERUMIER wrote: 
>> Hi, 
>> 
>> I have able to reproduce, after 1hour. 
>> 
>> I have enable debug to get it run in foreground. 
>> 
>> This time, the process was not crashed, but was hanging. 
>> 
>> output was simply hanging, and no more write in /var/log/pve-firewall.log 
>> 
>> Also, memory was pretty huge, and still increasing during the hang (not sure if it's related to debug mode) 
>> 
>> 
>> ps -aux|grep logger 
>> root 19434 26.2 0.4 1770688 1679136 pts/1 Rl+ 10:44 11:27 ./pvefw-logger 
>> 
>> after some minutes 
>> 
>> root 19434 24.8 0.8 3625024 3533496 pts/1 Sl+ 10:44 12:20 ./pvefw-logger 
>> 
>> 
>> I was able to do a coredump with gdb 
>> http://odisoweb1.odiso.net/core.19434.gz 
>> 
>> Hope it's help. 
>> 
>> 
>> ----- Mail original ----- 
>> De: "Wolfgang Bumiller" <w.bumiller at proxmox.com> 
>> À: "aderumier" <aderumier at odiso.com> 
>> Cc: "David Limbeck" <d.limbeck at proxmox.com>, "pve-devel" <pve-devel at pve.proxmox.com> 
>> Envoyé: Lundi 14 Janvier 2019 08:01:54 
>> Objet: Re: [pve-devel] [PATCH firewall] make nfct_catch non-blocking 
>> 
>> On Fri, Jan 11, 2019 at 06:05:36PM +0100, Alexandre DERUMIER wrote: 
>>>>> Do you have any additional information as to why it stopped? 
>>> no sorry. 
>>> 
>>>>> Maybe we could increase the buffer size via nfnl_set_rcv_buffer_size by 
>>>>> default and continue to ignore ENOBUFS? 
>>> I'll try next week. maybe doing strace on the process to have some clues ? (I'ts crashing after 30min-1h) 
>> A coredump should work and produce less noise, perhaps? 
>> 
>> 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
> 
>