[PVE-User] Strange Behavior with multiple venet0 interfaces

[Marc Aymerich]

In openVz support forum user maratus has give the solution :) it is:

you may specify what ip address should be used as a source ip address
with help of "ip" utility inside VE just like it should be done on a
ordinary Linux machine,

# ip r add 10.0.0.0/24 dev venet0 src 10.0.0.155
# ip r add 77.178.25.0/24 dev venet0 src 77.178.25.145


On Wed, Jul 8, 2009 at 6:44 PM, Marc Aymerich<glicerinu at gmail.com> wrote:
> Hi all,
> I have a server with various network interfaces connected on different
> networks (eth0:77.178.25.0/24, eth1:10.0.0.0/24,eth2:10.0.10.0/24,
> eth3:10.10.0.0/24, and more...). Today I have been doing tests with
> an OpenVZ virtual machine with multiple venet0 interfaces  (vente0:0,
> venet0:1 and venet0:2), each interface belonging to a different network
> (venet0:0 77.178.25.145/24, venet0:2 10.10.0.155/24, venet0:1
> 10.0.0.155/24).
>
> I observe a strange behavior when I ping from OpenVZ VE to
> another machines on the network. The source IP address
> of the output packets for the VE is in all the cases 77.178.25.145,
> in spite of being 10.10.0.155 or 10.0.0.155
>
> In the worst case it causes packet lossing because some machines of
> the private network doesnt know how/to route packets to
> 77.178.25.145.
>
> Whats the reason of this behavior? is this a bug?  Is there any way to
> fix it? Perhaps a clue to solve the problem is that the interfaces are
> called venet0:0,venet0:1, venet0:2 instead of venet0, venet1 and venet2?
>
>
> I know this problem disappears when using the Veth interfaces instead
> of Venet0, but I like to use the Venet0 system because it offer more
> performance and security.
>
>
> I leave an example to ping 10.0.0.10 from the virtual machine.
>
> 1) Tcpdump show as on the hardware node packets leave by the correct
> interface (eth1: network 10.0.0.0/24) but with the source address
> changed, 77.178.25.145 instead of 10.0.0.155
>
> tcpdump -i eth1
> 14:20:13.939721 IP 77.178.25.145 > 10.0.0.10: ICMP echo request, id
> 63234, seq 2, length 64
> 14:20:14.939208 IP 77.178.25.145 > 10.0.0.10: ICMP echo request, id
> 63234, seq 3, length 64
>
> Then the answers returns for another interface :( (eth0: network
> 77.178.25.0/24)
>
> tcpdump -i eth0
>
> 14:22:46.075225 IP 10.0.0.10 > 77.178.25.145: ICMP echo reply, id
> 63490, seq 6564, length 64
> 14:22:46.075473 IP 10.0.0.10 > 77.178.25.145: ICMP echo reply, id
> 63490, seq 6565, length 64
>
> finally the packets are relayed to the virtual machine.
>
> tcpdump -i venet0
>
> 12:24:45.211486 IP 10.0.0.10 > 77.178.25.145: ICMP echo reply, id
> 10500, seq 21, length 64
> 12:24:46.211202 IP 77.178.25.145 > 10.0.0.10: ICMP echo request, id
> 10500, seq 22, length 64
>
>
>
> Configuration of the Virtual Machine - OpenVZ
>
> test:/# route -n
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
> 192.0.2.1       0.0.0.0         255.255.255.255 UH    0      0        0 venet0
> 0.0.0.0         192.0.2.1       0.0.0.0         UG    0      0        0 venet0
>
>
> test:/# ifconfig
> lo        Link encap:Local Loopback
>       inet addr:127.0.0.1  Mask:255.0.0.0
>       inet6 addr: ::1/128 Scope:Host
>       UP LOOPBACK RUNNING  MTU:16436  Metric:1
>       RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>       TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>       collisions:0 txqueuelen:0
>       RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>
> venet0    Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>       inet addr:127.0.0.1  P-t-P:127.0.0.1  Bcast:0.0.0.0
> Mask:255.255.255.255
>       UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
>       RX packets:177951 errors:0 dropped:0 overruns:0 frame:0
>       TX packets:179295 errors:0 dropped:0 overruns:0 carrier:0
>       collisions:0 txqueuelen:0
>       RX bytes:14947840 (14.2 MiB)  TX bytes:15060736 (14.3 MiB)
>
> venet0:0  Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>       inet addr:77.178.25.145  P-t-P:77.246.179.123  Bcast:0.0.0.0
>  Mask:255.255.255.255
>       UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
>
> venet0:1  Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>       inet addr:10.0.0.155  P-t-P:10.0.0.155  Bcast:0.0.0.0
> Mask:255.255.255.255
>       UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
>
> venet0:2  Link encap:UNSPEC  HWaddr
> 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
>       inet addr:10.10.0.155  P-t-P:10.10.0.155  Bcast:0.0.0.0
> Mask:255.255.255.255
>       UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
>
>
>
> ----------------------------------------------------------------------------
>
> Hardware Node configuration.
>
> backup:~# route -n
> Kernel IP routing table
> Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
> 77.178.25.145   0.0.0.0         255.255.255.255 UH    0      0        0 venet0
> 10.0.0.155      0.0.0.0         255.255.255.255 UH    0      0        0 venet0
> 10.10.0.155     0.0.0.0         255.255.255.255 UH    0      0        0 venet0
> 77.178.25.0     0.0.0.0         255.255.255.0   U     0      0        0 vmbr0
> 10.0.0.0        0.0.0.0         255.255.255.0   U     0      0        0 eth1
> 10.10.0.0       0.0.0.0         255.255.255.0   U     0      0        0 eth3
> 10.0.10.0       0.0.0.0         255.255.255.0   U     0      0        0 eth2
> 10.0.10.0       0.0.0.0         255.255.255.0   U     0      0        0 eth4
> 0.0.0.0         77.178.25.1     0.0.0.0         UG    0      0        0 vmbr0
>
>
> /etc/vz/conf/101.conf
>
> # CPU fair sheduler parameter
> CPUUNITS="1000"
> CPUS="1"
> VE_ROOT="/var/lib/vz/root/$VEID"
> VE_PRIVATE="/var/lib/vz/private/$VEID"
> OSTEMPLATE="debian-5.0-standard_5.0-1_i386"
> ORIGIN_SAMPLE="pve.auto"
> IP_ADDRESS="77.178.25.145 10.0.0.155 10.10.0.155"
> HOSTNAME="test.xxxx.org"
> DESCRIPTION="Debian 5.0 (standard)"
> NAMESERVER="208.67.222.222"
> SEARCHDOMAIN="xxxxx.org"
>
>
> Thanks very much!
> Marc
>