[pve-devel] BUG in vlan aware bridge

[Stoyan Marinov]

Alright, I said tomorrow, but I did a bit more fiddling with tcpdump and that's quite odd:
01:59:31.462866 d2:24:de:13:3d:6e > c6:29:5b:54:e3:b9, ethertype IPv4 (0x0800), length 1514: 10.3.4.111 > 192.168.0.220: ICMP echo reply, id 112, seq 29, length 1480
01:59:31.462866 d2:24:de:13:3d:6e > c6:29:5b:54:e3:b9, ethertype 802.1Q (0x8100), length 566: vlan 10, p 0, ethertype IPv4, 10.3.4.111 > 192.168.0.220: ip-proto-1
01:59:32.486719 d2:24:de:13:3d:6e > c6:29:5b:54:e3:b9, ethertype IPv4 (0x0800), length 1514: 10.3.4.111 > 192.168.0.220: ICMP echo reply, id 112, seq 30, length 1480
01:59:32.486719 d2:24:de:13:3d:6e > c6:29:5b:54:e3:b9, ethertype 802.1Q (0x8100), length 566: vlan 10, p 0, ethertype IPv4, 10.3.4.111 > 192.168.0.220: ip-proto-1

It seems like the first fragment arrives properly and the 2nd one is a vlan tagged ethernet frame. That's weird.

On top of all I noticed that if I run a tcpdump on proxmox host, listening on vmbr0 - it all works!


> On 13 Oct 2021, at 1:45 AM, Stoyan Marinov <stoyan at marinov.us> wrote:
> 
> OK, I have just verified it has nothing to do with bonds. I get the same behavior with vlan aware bridge, bridge-nf-call-iptables=1 with regular eth0 being part of the bridge. Packets arrive fragmented on tap, reassembled by netfilter and then re-injected in bridge assembled (full size).
> 
> I did have limited success by setting net.bridge.bridge-nf-filter-vlan-tagged to 1. Now packets seem to get fragmented on the way out and back in, but there are still issues:
> 
> 1. I'm testing with ping -s 2000 (1500 mtu everywhere) to an external box. I do see reply packets arrive on the vm nic, but ping doesn't see them. Haven't analyzed much further.
> 2. While watching with tcpdump (inside the vm) i notice "ip reassembly time exceeded" messages being generated from the vm.
> 
> I'll try to investigate a bit further tomorrow.
> 
>> On 12 Oct 2021, at 11:26 PM, Stoyan Marinov <stoyan at marinov.us> wrote:
>> 
>> That's an interesting observation. Now that I think about it, it could be caused by bonding and not the underlying device. When I tested this (about an year ago) I was using bonding on the mlx adapters and not using bonding on intel ones.
>> 
>>> On 12 Oct 2021, at 3:36 PM, VELARTIS Philipp Dürhammer <p.duerhammer at velartis.at> wrote:
>>> 
>>> HI,
>>> 
>>> we use HP Server with Intel Cards or the standard hp nic ( ithink also intel)
>>> 
>>> Also I see the I did a mistake:
>>> 
>>> Setup working:
>>> tapX (UNtagged) <- -> vmbr0 <- - > bond0
>>> 
>>> is correct. (before I had also tagged) 
>>> 
>>> it should be :
>>> 
>>> Setup not working:
>>> tapX (tagged) <- -> vmbr0 <- - > bond0
>>> 
>>> Setup working:
>>> tapX (untagged) <- -> vmbr0 <- - > bond0
>>> 
>>> Setup also working:
>>> tapX < - - > vmbr0v350 < -- > bond0.350 < -- > bond0
>>> 
>>> -----Ursprüngliche Nachricht-----
>>> Von: pve-devel <pve-devel-bounces at lists.proxmox.com> Im Auftrag von Stoyan Marinov
>>> Gesendet: Dienstag, 12. Oktober 2021 13:16
>>> An: Proxmox VE development discussion <pve-devel at lists.proxmox.com>
>>> Betreff: Re: [pve-devel] BUG in vlan aware bridge
>>> 
>>> I'm having the very same issue with Mellanox ethernet adapters. I don't see this behavior with Intel nics. What network cards do you have?
>>> 
>>>> On 12 Oct 2021, at 1:48 PM, VELARTIS Philipp Dürhammer <p.duerhammer at velartis.at> wrote:
>>>> 
>>>> HI,
>>>> 
>>>> i am playing around since days because we have strange packet losses.
>>>> Finally I can report following (Linux 5.11.22-4-pve, Proxmox 7, all devices MTU 1500):
>>>> 
>>>> Packet with sizes > 1500 without VLAN working well but at the moment they are Tagged they are dropped by the bond device.
>>>> Netfilter (set to 1) always reassembles the packets when they arrive a bridge. But they don't get fragmented again I they are VLAN tagged. So the bond device drops them. If the bridge is NOT Vlan aware they also get fragmented and it works well.
>>>> 
>>>> Setup not working:
>>>> 
>>>> tapX (tagged) <- -> vmbr0 <- - > bond0
>>>> 
>>>> Setup working:
>>>> 
>>>> tapX (tagged) <- -> vmbr0 <- - > bond0
>>>> 
>>>> Setup also working:
>>>> 
>>>> tapX < - - > vmbr0v350 < -- > bond0.350 < -- > bond0
>>>> 
>>>> Have you got any idea where to search? I don't understand who is in charge of fragmenting packages again if they get reassembled by netfilter. (and why it is not working with vlan aware bridges)
>>>> 
>>>> 
>>>> _______________________________________________
>>>> pve-devel mailing list
>>>> pve-devel at lists.proxmox.com
>>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>>>> 
>>> 
>>> 
>>> _______________________________________________
>>> pve-devel mailing list
>>> pve-devel at lists.proxmox.com
>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>>> _______________________________________________
>>> pve-devel mailing list
>>> pve-devel at lists.proxmox.com
>>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>> 
>> 
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel at lists.proxmox.com
>> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel