[pve-devel] proxmox 2018 : add support for "virtual" network and network plugins ?

[Alexandre DERUMIER]

>>I think we basically have two kinds of networks: 
>> 
>>1.) local networks: 
>> 
>>This is what we already have in /etc/network/interface. Access to local network 
>>is 
>>usually restricted to admins. 
>> 
>>2.) virtual networks: 
>> 
>>Basically a linux bridge where we can connect VM to. One can connect such 
>>virtual network to local network: 
>> 
>>- directly (this is what we currently use for the firewall) 
>>- vlan 
>>- vxlan 
>> 
>>Or we can connect that bridge to some SDN. 
>> 
>>We can also add additional service to such virtual network: 
>> 
>>- SNAT, DNAT 
>>- Firewall 
>>- DHCP 
>>- Routing, ... 

Yes, I totally agreed with you. 




For vxlan with linux bridge, I have found very good documentation here: 

https://vincent.bernat.im/fr/blog/2017-vxlan-linux 
https://vincent.bernat.im/fr/blog/2017-vxlan-bgp-evpn 

(In french, sorry). 

But basically, we can: 

create a simple bridge with vxlan interface (1 bridge by vxlan) 

Host1 (10.0.0.1) 
------- 
ip link add vxlan100 type vxlan \ 
id 100 \ 
dstport 4789 \ 
local 10.0.0.1 \ 
group ff05::100 \ 
dev eth0 \ 
ttl 5 

# brctl addbr vmbr100 
# brctl addif vmbr100 vxlan100 


ip link add vxlan200 type vxlan \ 
id 100 \ 
dstport 4789 \ 
local 10.0.0.1 \ 
group ff05::100 \ 
dev eth0 \ 
ttl 5 

# brctl addbr vmbr200 
# brctl addif vmbr200 vxlan200 


Host2 (10.0.0.2) 
------- 
ip link add vxlan100 type vxlan \ 
id 100 \ 
dstport 4789 \ 
local 10.0.0.2 \ 
group ff05::100 \ 
dev eth0 \ 
ttl 5 

# brctl addbr vmbr100 
# brctl addif vmbr100 vxlan100 


ip link add vxlan200 type vxlan \ 
id 100 \ 
dstport 4789 \ 
local 10.0.0.2 \ 
group ff05::100 \ 
dev eth0 \ 
ttl 5 

# brctl addbr vmbr200 
# brctl addif vmbr200 vxlan200 


This simple setup use multicast to send arp requests to all vni. 
Can work with layer2 lan, but not across internet. 

Anoter mode, is to use unicast instead multicast 
---------------------------------------------------------- 
host1 
------ 
ip link add vxlan100 type vxlan \ 
id 100 \ 
dstport 4789 \ 
local 10.0.0.1 \ 
group ff05::100 \ 
dev eth0 \ 
ttl 5 
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.0.2 
# brctl addbr vmbr100 
# brctl addif vmbr100 vxlan100 


ip link add vxlan200 type vxlan \ 
id 100 \ 
dstport 4789 \ 
local 10.0.0.1 \ 
group ff05::100 \ 
dev eth0 \ 
ttl 5 
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.0.2 
# brctl addbr vmbr200 
# brctl addif vmbr200 vxlan200 

host2 
------ 
ip link add vxlan100 type vxlan \ 
id 100 \ 
dstport 4789 \ 
local 10.0.0.2 \ 
group ff05::100 \ 
dev eth0 \ 
ttl 5 
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.0.1 
# brctl addbr vmbr100 
# brctl addif vmbr100 vxlan100 


ip link add vxlan200 type vxlan \ 
id 100 \ 
dstport 4789 \ 
local 10.0.0.2 \ 
group ff05::100 \ 
dev eth0 \ 
ttl 5 
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 10.0.0.1 
# brctl addbr vmbr200 
# brctl addif vmbr200 vxlan200 


This works fine for small setup, as arp will be replicate in unicast to all vni 


So to avoid arp (for big network), we can disable learning on vni , and use a bgp daemon (bgp-evpn protocol) to sync the fbd 
host1: 
------- 
ip link add vxlan100 type vxlan 
id 100 \ 
dstport 4789 \ 
local 10.0.0.1 \ 
nolearning 

host2 
------- 
ip link add vxlan100 type vxlan 
id 100 \ 
dstport 4789 \ 
local 10.0.0.2 \ 
nolearning 

then quagga/or frr local on each host, to peer with others hosts or through bgp routes reflector. (see the doc) 



They are also description of manual fbd setup (could be done by a proxmox daemon, as we known the mac address of vms, but this will work only for 1 proxmox cluster). 
They are examples in documentation with behaviour of docker libnetworkd and flannel. 


It could be great to have something easy to setup, without need to configure each host manually. 
for example, something like 
/etc/pve/network.conf: 

vxlanplugin: customer1 
vxlan 100 
underlay_network 10.0.0.0/8 

and in vm config: net0: virtio=....,network=customer1 

this will create the vmbr100 with vxlan100 interface and take the local ip of each host, do the unicast config if needed with all others hosts,.... 



De: "dietmar" <dietmar at proxmox.com> 
À: "aderumier" <aderumier at odiso.com>, "pve-devel" <pve-devel at pve.proxmox.com> 
Envoyé: Jeudi 4 Janvier 2018 09:30:52 
Objet: Re: [pve-devel] proxmox 2018 : add support for "virtual" network and network plugins ? 

I think we basically have two kinds of networks: 

1.) local networks: 

This is what we already have in /etc/network/interface. Access to local network 
is 
usually restricted to admins. 

2.) virtual networks: 

Basically a linux bridge where we can connect VM to. One can connect such 
virtual network to local network: 

- directly (this is what we currently use for the firewall) 
- vlan 
- vxlan 

Or we can connect that bridge to some SDN. 

We can also add additional service to such virtual network: 

- SNAT, DNAT 
- Firewall 
- DHCP 
- Routing, ... 


> On January 2, 2018 at 3:04 PM Alexandre DERUMIER <aderumier at odiso.com> wrote: 
> I think we have 2 kind of setup: 
> 
> - basic local vswitch (bridge, ovs, snabwitch,....) : can be easily setup with 
> systemd-network + some tap/eth plug/unplug scripts. 
> - bigger sdn setup, with external controllers. (which could manage networks 
> across multiple proxmox clusters too)