[pve-devel] proxmox 2018 : add support for "virtual" network and network plugins ?

[Alexandre DERUMIER]

about this

>>internet------>89.248.0.0/16---router-192.168.0.1--------192.168.0.2(mac1)--proxmox0(vmbrvxlan1)(89.248.1.1/24)----------vm1(89.248.1.10/24)
>>                                                                                    (vmbrvxlan2)(89.248.2.1/24)----------vm2(89.248.2.10/24)
>>  
>>                                                     ----192.168.0.3(mac2)--proxmox1(vmbrvxlan1)(89.248.1.1/24)----------vm3(89.248.1.11/24)
>>                                                                                    (vmbrvxlan2)(89.248.2.1/24)----------vm4(89.248.2.11/24)
>>
>>192.168.0.0/24 is an interconnect lan
>>

>>proxyarp is enable on proxmox out interface (192.168.0.2 and 192.168.0.3), and this interface is not enslaved to vmbrvxlan.
>>If vm is migrated, we need to send a garp to the router because the mac address will change.

proxyarp can't work, because it's not same network range


>>as alternative, if proxmox nodes and router use a routing protocol (bgp,ospf), we could announce vm ip on new host after migration.
>>If router (or intermediate switch), can support vxlan, we can connect it directly to vmbrxlan1/2 network, and vm migration is really transparent.

I verified this, no need to have vxlan on the router.


You can simply peer with bgp between your router and proxmox nodes, and ebpn-vpn routes from vmbrbxlan1/2 will be announce to the external router, and all is working out of the box.
on vm move, the bgp protocol detect that mac address is on a new vtep, and a bgp announce is send to the router.
(BTW, for anycast gateway and vxlan, bgp-evpn is mandatory. So if interconnect is needed with outside world, it's only 3/4 lines of configuration in the bgp daemon.)


----- Mail original -----
De: "Alexandre Derumier" <aderumier at odiso.com>
À: "dietmar" <dietmar at proxmox.com>
Cc: "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Dimanche 28 Janvier 2018 08:38:26
Objet: Re: [pve-devel] proxmox 2018 : add support for "virtual" network and network plugins ?

> 
> internet------>89.248.0.0/16---router--(89.248.1.1/24)-------proxmox0(vmbr0)----------vm1(89.248.1.10/24) 
> 
> -------proxmox1(vmbr0)----------vm2(89.248.1.10/24) 
> 
> ---router--(89.248.2.1/24)-------proxmox0(vmbr1)----------vm3(89.248.2.10/24) 
> 
> -------proxmox1(vmbr1)----------vm4(89.248.2.10/24) 
> 
> 
> you can have the vms with default gw to 89.248.x.1/24. 

>>This looks like a normal PVE setup. Or where is the virtual network (vxlan) 
>>here? 

maybe do you want something like 



internet------>89.248.0.0/16---router-192.168.0.1--------192.168.0.2(mac1)--proxmox0(vmbrvxlan1)(89.248.1.1/24)----------vm1(89.248.1.10/24) 
(vmbrvxlan2)(89.248.2.1/24)----------vm2(89.248.2.10/24) 

----192.168.0.3(mac2)--proxmox1(vmbrvxlan1)(89.248.1.1/24)----------vm3(89.248.1.11/24) 
(vmbrvxlan2)(89.248.2.1/24)----------vm4(89.248.2.11/24) 

192.168.0.0/24 is an interconnect lan 

proxyarp is enable on proxmox out interface (192.168.0.2 and 192.168.0.3), and this interface is not enslaved to vmbrvxlan. 


If vm is migrated, we need to send a garp to the router because the mac address will change. 
as alternative, if proxmox nodes and router use a routing protocol (bgp,ospf), we could announce vm ip on new host after migration. 

If router (or intermediate switch), can support vxlan, we can connect it directly to vmbrxlan1/2 network, and vm migration is really transparent. 



I'm not sure it's a really common pattern usage for cloud providers. 
For example google gcp, amazon ec2, rackspace : all are using vxlan/tenant on private address space + floating ip from 1 big public network 
because with ipv4 public address, it's difficult to reserve big range (or costly). user reserve 1 public ip once month, 1 another public ip another month. 

It's not uncommon to have for example 40vm, with 38vm with private ip and 2 vm only ip address. (can be a ssh bastion, or a vm proxy with public, and the other (web,sql,....) are all privates) 
It's even possible that 2 tenants of 2 customers have the same private ip address space. (it's possible with vrf). 

anycast gateway for private make sense here, because you can make snat,nat, and inter vxlan datacenter routing. 
public network is 1 network, the provider network, with the provider router is used to exit to internet. 

http://superuser.openstack.org/articles/tenant-networks-vs-provider-networks-in-the-private-cloud-context/ 



This can give use (without floating ip:ip, 2 interfaces by vm) 

internet------>89.248.0.0/16---router--(89.248.0.1/16)-------proxmox0(vmbr0 public)-------------------------------vm1(89.248.4.10/16) (default gw 89.248.0.1) 
-------------------------------vm2(89.248.4.11/16)(default gw 89.248.0.1) 

(vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm1(10.0.1.10/24) (route add 10.0.0.0/16 gw 10.0.1.1) 
(vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm2(10.0.2.10/24) (route add 10.0.0.0/16 gw 10.0.2.1) 

-------proxmox1(vmbr0 public)-------------------------------vm3(89.248.4.13/16)(default gw 89.248.0.1) 
-------------------------------vm4(89.248.4.14/16)(default gw 89.248.0.1) 

(vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm3(10.0.1.11/24) (route add 10.0.0.0/16 gw 10.0.1.1) 
(vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm4(10.0.2.12/24) (route add 10.0.0.0/16 gw 10.0.2.1) 

note that we can't manage dhcp on public from proxmox, and dhcp for private can't add the special private routes. 





or with floating:ip 


internet------>89.248.0.0/16---router--(89.248.0.1/16)-------proxmox0: eth0:1 : 89.248.4.10 (nat) 
eth0:2 : 89.248.4.11 (nat) 
(vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm1(10.0.1.10/24) (default gw 10.0.1.1/24) 
(vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm2(10.0.2.10/24) (default gw 10.0.2.1/24) 

-------proxmox1: eth0:1 : 89.248.4.13 (nat) 
eth0:2 : 89.248.4.14 (nat) 

(vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm3(10.0.1.11/24)(default gw 10.0.1.1/24) 
(vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm4(10.0.2.12/24)(default gw 10.0.2.1/24) 

All can be managed (dhcp for private, nat for public ip) 



----- Mail original ----- 
De: "dietmar" <dietmar at proxmox.com> 
À: "Alexandre Derumier" <aderumier at odiso.com> 
Cc: "pve-devel" <pve-devel at pve.proxmox.com> 
Envoyé: Vendredi 26 Janvier 2018 18:42:07 
Objet: Re: [pve-devel] proxmox 2018 : add support for "virtual" network and network plugins ? 

> On January 26, 2018 at 6:18 PM Alexandre DERUMIER <aderumier at odiso.com> wrote: 
> 
> 
> > After all, if we don't do s-nat, nat 1:1, why not just configure the gateway 
> > 
> > of the vm to use the external router directly 
> 
> >>Because there is no route to the external router? Please can you elaborate 
> >>on 
> >>that? 
> 
> I mean, for example 
> 
> 
> internet------>89.248.0.0/16---router--(89.248.1.1/24)-------proxmox0(vmbr0)----------vm1(89.248.1.10/24) 
> 
> -------proxmox1(vmbr0)----------vm2(89.248.1.10/24) 
> 
> ---router--(89.248.2.1/24)-------proxmox0(vmbr1)----------vm3(89.248.2.10/24) 
> 
> -------proxmox1(vmbr1)----------vm4(89.248.2.10/24) 
> 
> 
> you can have the vms with default gw to 89.248.x.1/24. 

This looks like a normal PVE setup. Or where is the virtual network (vxlan) 
here?