[pve-devel] proxmox 2018 : add support for "virtual" network and network plugins ?

[Alexandre DERUMIER]

> 
> internet------>89.248.0.0/16---router--(89.248.1.1/24)-------proxmox0(vmbr0)----------vm1(89.248.1.10/24)
> 
>                                                      -------proxmox1(vmbr0)----------vm2(89.248.1.10/24)
> 
>                            ---router--(89.248.2.1/24)-------proxmox0(vmbr1)----------vm3(89.248.2.10/24)
> 
>                                                      -------proxmox1(vmbr1)----------vm4(89.248.2.10/24)
> 
>                        
> you can have the vms with default gw to 89.248.x.1/24. 

>>This looks like a normal PVE setup. Or where is the virtual network (vxlan)
>>here?

maybe do you want something like



internet------>89.248.0.0/16---router-192.168.0.1--------192.168.0.2(mac1)--proxmox0(vmbrvxlan1)(89.248.1.1/24)----------vm1(89.248.1.10/24)
                                                                                    (vmbrvxlan2)(89.248.2.1/24)----------vm2(89.248.2.10/24)
  
                                                     ----192.168.0.3(mac2)--proxmox1(vmbrvxlan1)(89.248.1.1/24)----------vm3(89.248.1.11/24)
                                                                                    (vmbrvxlan2)(89.248.2.1/24)----------vm4(89.248.2.11/24)

192.168.0.0/24 is an interconnect lan

proxyarp is enable on proxmox out interface (192.168.0.2 and 192.168.0.3), and this interface is not enslaved to vmbrvxlan.


If vm is migrated, we need to send a garp to the router because the mac address will change.
as alternative, if proxmox nodes and router use a routing protocol (bgp,ospf), we could announce vm ip on new host after migration.

If router (or intermediate switch), can support vxlan, we can connect it directly to vmbrxlan1/2 network, and vm migration is really transparent.



I'm not sure it's a really common pattern usage for cloud providers. 
For example google gcp, amazon ec2, rackspace : all are using vxlan/tenant on private address space + floating ip from 1 big public network
because with ipv4 public address, it's difficult to reserve big range (or costly). user reserve 1 public ip once month, 1 another public ip another month.

It's not uncommon to have for example 40vm, with 38vm with private ip and 2 vm only ip address. (can be a ssh bastion, or a vm proxy with public, and the other (web,sql,....) are all privates)
It's even possible that 2 tenants of 2 customers have the same private ip address space. (it's possible with vrf).

anycast gateway for private make sense here, because you can make snat,nat, and inter vxlan datacenter routing.
public network is 1 network, the provider network, with the provider router is used to exit to internet.

http://superuser.openstack.org/articles/tenant-networks-vs-provider-networks-in-the-private-cloud-context/



This can give use (without floating ip:ip, 2 interfaces by vm)

internet------>89.248.0.0/16---router--(89.248.0.1/16)-------proxmox0(vmbr0 public)-------------------------------vm1(89.248.4.10/16) (default gw 89.248.0.1)
                                                                                   -------------------------------vm2(89.248.4.11/16)(default gw 89.248.0.1)

                                                                     (vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm1(10.0.1.10/24) (route add 10.0.0.0/16 gw 10.0.1.1)
                                                                     (vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm2(10.0.2.10/24) (route add 10.0.0.0/16 gw 10.0.2.1)

                                                      -------proxmox1(vmbr0 public)-------------------------------vm3(89.248.4.13/16)(default gw 89.248.0.1)
                                                                                   -------------------------------vm4(89.248.4.14/16)(default gw 89.248.0.1)

                                                                     (vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm3(10.0.1.11/24) (route add 10.0.0.0/16 gw 10.0.1.1)
                                                                     (vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm4(10.0.2.12/24) (route add 10.0.0.0/16 gw 10.0.2.1)

note that we can't manage dhcp on public from proxmox, and dhcp for private can't add the special private routes.





or with floating:ip


internet------>89.248.0.0/16---router--(89.248.0.1/16)-------proxmox0: eth0:1 : 89.248.4.10 (nat)                                                                                
                                                                       eth0:2 : 89.248.4.11 (nat)
                                                                     (vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm1(10.0.1.10/24) (default gw 10.0.1.1/24)
                                                                     (vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm2(10.0.2.10/24) (default gw 10.0.2.1/24)

                                                      -------proxmox1:  eth0:1 : 89.248.4.13 (nat)   
                                                                        eth0:2 : 89.248.4.14 (nat)   

                                                                     (vmbrvxlan1 private- 10.0.1.1/24 anycast)---------vm3(10.0.1.11/24)(default gw 10.0.1.1/24)
                                                                     (vmbrvxlan2 private- 10.0.2.1/24 anycast)---------vm4(10.0.2.12/24)(default gw 10.0.2.1/24)

All can be managed (dhcp for private,  nat for public ip)

                          

----- Mail original -----
De: "dietmar" <dietmar at proxmox.com>
À: "Alexandre Derumier" <aderumier at odiso.com>
Cc: "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Vendredi 26 Janvier 2018 18:42:07
Objet: Re: [pve-devel] proxmox 2018 : add support for "virtual" network and network plugins ?

> On January 26, 2018 at 6:18 PM Alexandre DERUMIER <aderumier at odiso.com> wrote: 
> 
> 
> > After all, if we don't do s-nat, nat 1:1, why not just configure the gateway 
> > 
> > of the vm to use the external router directly 
> 
> >>Because there is no route to the external router? Please can you elaborate 
> >>on 
> >>that? 
> 
> I mean, for example 
> 
> 
> internet------>89.248.0.0/16---router--(89.248.1.1/24)-------proxmox0(vmbr0)----------vm1(89.248.1.10/24) 
> 
> -------proxmox1(vmbr0)----------vm2(89.248.1.10/24) 
> 
> ---router--(89.248.2.1/24)-------proxmox0(vmbr1)----------vm3(89.248.2.10/24) 
> 
> -------proxmox1(vmbr1)----------vm4(89.248.2.10/24) 
> 
> 
> you can have the vms with default gw to 89.248.x.1/24. 

This looks like a normal PVE setup. Or where is the virtual network (vxlan) 
here?