[pve-devel] proxmox 2018 : add support for "virtual" network and network plugins ?

Sun Jan 28 09:58:43 CET 2018

and for my specific use, I need to do something like (1 public vxlan, anycast default gw for vm, this vxlan exist on my router(s), and bgp is doing routing betwwen proxmox and differents routeurs

internet------>89.248.0.0/16---router(192.168.0.1)------(192.168.0.20)proxmox0(vmbrvxlan0 public -89.248.0.1/16 anycast)---------vm1(89.248.4.10/16) (default gw 89.248.0.1)
                               router(192.168.0.2)                                                                      ---------vm2(89.248.4.11/16)(default gw 89.248.0.1)
                               router(192.168.0.3)                            (vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm1(10.0.1.10/24) (route add 10.0.0.0/16 gw 10.0.1.1)
                               router(192.168.0.4)                            (vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm2(10.0.2.10/24) (route add 10.0.0.0/16 gw 10.0.2.1)                                                            

                                                  -------(192.168.0.30)proxmox1(vmbrvxlan0 public -89.248.0.1/16 anycast)---------vm3(89.248.4.13/16)(default gw 89.248.0.1)
                                                                                                                         ---------vm4(89.248.4.14/16)(default gw 89.248.0.1)

                                                                               (vmbrvxlan1 private- 10.0.1.1/24 anycast)----------vm3(10.0.1.11/24) (route add 10.0.0.0/16 gw 10.0.1.1)
                                                                               (vmbrvxlan2 private- 10.0.2.1/24 anycast)----------vm4(10.0.2.12/24) (route add 10.0.0.0/16 gw 10.0.2.1)

dhcp can be manage on public vmbrvxlan0
192.168.0.0/24 is the underlay network, full bgp between routers and proxmox nodes, across mutiple routers/rack/datacenters (layer3 everywhere).
vmbrxvlan0 is the overlay public network, like a virtual layer2 between proxmox nodes and routers.
vm use public vmbrvxlan0 as default gateway, and after that is going to correct router through vxlan and bgp routing.

----- Mail original -----
De: "Alexandre Derumier" <aderumier at odiso.com>
À: "dietmar" <dietmar at proxmox.com>
Cc: "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Dimanche 28 Janvier 2018 08:38:26
Objet: Re: [pve-devel] proxmox 2018 : add support for "virtual" network and network plugins ?

> 
> internet------>89.248.0.0/16---router--(89.248.1.1/24)-------proxmox0(vmbr0)----------vm1(89.248.1.10/24) 
> 
> -------proxmox1(vmbr0)----------vm2(89.248.1.10/24) 
> 
> ---router--(89.248.2.1/24)-------proxmox0(vmbr1)----------vm3(89.248.2.10/24) 
> 
> -------proxmox1(vmbr1)----------vm4(89.248.2.10/24) 
> 
> 
> you can have the vms with default gw to 89.248.x.1/24. 

>>This looks like a normal PVE setup. Or where is the virtual network (vxlan) 
>>here? 

maybe do you want something like 

internet------>89.248.0.0/16---router-192.168.0.1--------192.168.0.2(mac1)--proxmox0(vmbrvxlan1)(89.248.1.1/24)----------vm1(89.248.1.10/24) 
(vmbrvxlan2)(89.248.2.1/24)----------vm2(89.248.2.10/24) 

----192.168.0.3(mac2)--proxmox1(vmbrvxlan1)(89.248.1.1/24)----------vm3(89.248.1.11/24) 
(vmbrvxlan2)(89.248.2.1/24)----------vm4(89.248.2.11/24) 

192.168.0.0/24 is an interconnect lan 

proxyarp is enable on proxmox out interface (192.168.0.2 and 192.168.0.3), and this interface is not enslaved to vmbrvxlan. 

If vm is migrated, we need to send a garp to the router because the mac address will change. 
as alternative, if proxmox nodes and router use a routing protocol (bgp,ospf), we could announce vm ip on new host after migration. 

If router (or intermediate switch), can support vxlan, we can connect it directly to vmbrxlan1/2 network, and vm migration is really transparent. 

I'm not sure it's a really common pattern usage for cloud providers. 
For example google gcp, amazon ec2, rackspace : all are using vxlan/tenant on private address space + floating ip from 1 big public network 
because with ipv4 public address, it's difficult to reserve big range (or costly). user reserve 1 public ip once month, 1 another public ip another month. 

It's not uncommon to have for example 40vm, with 38vm with private ip and 2 vm only ip address. (can be a ssh bastion, or a vm proxy with public, and the other (web,sql,....) are all privates) 
It's even possible that 2 tenants of 2 customers have the same private ip address space. (it's possible with vrf). 

anycast gateway for private make sense here, because you can make snat,nat, and inter vxlan datacenter routing. 
public network is 1 network, the provider network, with the provider router is used to exit to internet. 

http://superuser.openstack.org/articles/tenant-networks-vs-provider-networks-in-the-private-cloud-context/ 

This can give use (without floating ip:ip, 2 interfaces by vm) 

internet------>89.248.0.0/16---router--(89.248.0.1/16)-------proxmox0(vmbr0 public)-------------------------------vm1(89.248.4.10/16) (default gw 89.248.0.1) 
-------------------------------vm2(89.248.4.11/16)(default gw 89.248.0.1) 

(vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm1(10.0.1.10/24) (route add 10.0.0.0/16 gw 10.0.1.1) 
(vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm2(10.0.2.10/24) (route add 10.0.0.0/16 gw 10.0.2.1) 

-------proxmox1(vmbr0 public)-------------------------------vm3(89.248.4.13/16)(default gw 89.248.0.1) 
-------------------------------vm4(89.248.4.14/16)(default gw 89.248.0.1) 

(vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm3(10.0.1.11/24) (route add 10.0.0.0/16 gw 10.0.1.1) 
(vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm4(10.0.2.12/24) (route add 10.0.0.0/16 gw 10.0.2.1) 

note that we can't manage dhcp on public from proxmox, and dhcp for private can't add the special private routes. 

or with floating:ip 

internet------>89.248.0.0/16---router--(89.248.0.1/16)-------proxmox0: eth0:1 : 89.248.4.10 (nat) 
eth0:2 : 89.248.4.11 (nat) 
(vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm1(10.0.1.10/24) (default gw 10.0.1.1/24) 
(vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm2(10.0.2.10/24) (default gw 10.0.2.1/24) 

-------proxmox1: eth0:1 : 89.248.4.13 (nat) 
eth0:2 : 89.248.4.14 (nat) 

(vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm3(10.0.1.11/24)(default gw 10.0.1.1/24) 
(vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm4(10.0.2.12/24)(default gw 10.0.2.1/24) 

All can be managed (dhcp for private, nat for public ip) 

----- Mail original ----- 
De: "dietmar" <dietmar at proxmox.com> 
À: "Alexandre Derumier" <aderumier at odiso.com> 
Cc: "pve-devel" <pve-devel at pve.proxmox.com> 
Envoyé: Vendredi 26 Janvier 2018 18:42:07 
Objet: Re: [pve-devel] proxmox 2018 : add support for "virtual" network and network plugins ? 

> On January 26, 2018 at 6:18 PM Alexandre DERUMIER <aderumier at odiso.com> wrote: 
> 
> 
> > After all, if we don't do s-nat, nat 1:1, why not just configure the gateway 
> > 
> > of the vm to use the external router directly 
> 
> >>Because there is no route to the external router? Please can you elaborate 
> >>on 
> >>that? 
> 
> I mean, for example 
> 
> 
> internet------>89.248.0.0/16---router--(89.248.1.1/24)-------proxmox0(vmbr0)----------vm1(89.248.1.10/24) 
> 
> -------proxmox1(vmbr0)----------vm2(89.248.1.10/24) 
> 
> ---router--(89.248.2.1/24)-------proxmox0(vmbr1)----------vm3(89.248.2.10/24) 
> 
> -------proxmox1(vmbr1)----------vm4(89.248.2.10/24) 
> 
> 
> you can have the vms with default gw to 89.248.x.1/24. 

This looks like a normal PVE setup. Or where is the virtual network (vxlan) 
here? 

_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 