[PVE-User] networking adjustment | hope to get some feedback

[mj]

Hi,

So, I setup a test rig, with (only) two proxmox test-servers, with two 
NICs per server to test.

This /etc/network/interfaces seems to work well:

> iface eth1 inet manual
> 
> iface eth2 inet manual
> 
> auto bond0
> iface bond0 inet manual
> 	slaves eth1 eth2
> 	bond_miimon 100
> 	bond_mode 802.3ad
>       bond_xmit_hash_policy layer3+4
> 
> auto vmbr0
> iface vmbr0 inet static
> 	address  a.b.c.10
> 	netmask  255.255.255.0
> 	gateway  a.b.c.1
> 	bridge_ports bond0
> 	bridge_stp off
> 	bridge_fd 0
> 	up ip addr add 10.10.89.10/24 dev vmbr0 || true
> 	down ip addr del 10.10.89.10/24 dev vmbr0 || true

On the (procurve 5400) chassis, I configured the LACP like this:
> trunk D1-D2 Trk1 LACP
and
> trunk D3-D4 Trk2 LACP
resulting in this:
>  Procurve chassis(config)# show trunk
> 
> Load Balancing Method:  L3-based (default)
> 
>  Port | Name                             Type      | Group  Type    
>  ---- + -------------------------------- --------- + ------ --------
>  D1   | Link to pve001 - 1               10GbE-T   | Trk1   LACP    
>  D2   | Link to pve001 - 2               10GbE-T   | Trk1   LACP    
>  D3   | Link to pve002 - 1               10GbE-T   | Trk2   LACP    
>  D4   | Link to pve002 - 2               10GbE-T   | Trk2   LACP

The above config allows me to assign VLANs to lacp trunks ("Trk1", 
"Trk2") in the chassis webinterface like you would do with ports.

Then I did some reading on load balancing between the trunked ports, and 
figured that load balancing based on L4 would perhaps work better for 
us, so I changed it with
> trunk-load-balance L4

Since we are running the public and cluster network over the same wires, 
I don't think we can enable jumbo frames. Or would there be a way to 
make ceph traffic use a specific vlan, so we can enable jumbo frames on 
that vlan?

I realise that this is perhaps all very specific to our environment, but 
again: if there is anyone here with insights, tips, trics, please, 
feedback is welcome.

For example: with the dual 10G LACP connection to each server, we can 
only use mtu size 1500. Are we loosing much there..? Or would there be a 
way around this, somehow?

I tried also assigning the ceph cluster ip 10.10.89.10/11/12 to the 
bond0, instead of assigning it to vmbr0 as an alias. But in that setup I 
could never ping the other machine, so that somehow doesn't work. :-(

(thinking that with the ceph ip on the bond0, my VMs would not be able 
to see that traffic..?)

Again: all feedback welcome.

MJ



On 06/18/2018 11:56 AM, mj wrote:
> Hi all,
> 
> After having bought some new networking equipment, and gaining more 
> insight over the last two years, I am planning to make some adjustments 
> to our proxmox/ceph setup, and I would *greatly* appreciate some 
> feedback :-)
> 
> We are running a three-identical-server proxmox/ceph setup, with on each 
> server:
> 
> NIC1 Ceph cluster and monitors on 10.10.89.10/11/12 (10G ethernet)
> NIC2 clients and public ip a.b.c.10/11/12 (1G ethernet)
> 
> Since we bought new hardware, I can connect each server to our HP 
> chassis, over a dual 10G bonded LACP connection.
> 
> I obviously need to keep the (NIC1) public IP, but since the ceph 
> monitors ip is difficult to change, I'd like to keep the (NIC2) 
> 10.10.89.x as well.
> 
> I also need to keep the (tagged and untagged) VLAN's for proxmox and the 
> VMs running on it.
> 
> I realise that it used to be recommened to split cluster and client 
> traffic, but consensus nowadays on the ceph mailinglist seems to be: 
> keep it simple and don't split, unless specifically required. With this 
> in mind, I would also like to consolidate networking and run all traffic 
> over this dual lacp-bonded 10G connection to our HP chassis, including 
> the VLANs.
> 
> But how to achieve this..? :-) (and here come the questions...)
> 
> My idea is to first enable (active) LACP on our ProCurve 5400 chassis 
> ports, trunk type "LACP", but unsure about the "Trunk Group". Do I need 
> to select a different Truck Group (Trk1, Trk2 & Trk3) for each 
> dual-cable-connection to a server..?
> 
> And will the port-configured VLANs on the lacp-member-ports (both tagged 
> and untagged) continue to flow normally through this lacp bond..?
> 
> Then, about configuration on proxmox, would something like below do the 
> trick..?
> 
> auto bond0
> iface bond0 inet manual
>        slaves eth0 eth1
>        bond_miimon 100
>        bond_mode 802.3ad
>        bond_xmit_hash_policy layer2+3
> 
> auto vmbr0
> iface vmbr0 inet static
>        address  a.b.c.10/11/12 (public IPs)
>        netmask  255.255.255.0
>        gateway  a.b.c.1
>        bridge_ports bond0
>        bridge_stp off
>        bridge_fd 0
>        up ip addr add 10.10.89.10/11/12 dev vmbr0 || true (ceph mon IPs)
>        down ip addr del 10.100.222.1/24 dev vmbr0 || true
> 
> Any feedback on the above? As this is production, I'd like to be 
> reasonably sure that this would work, before trying.
> 
> Your comments will be very much appreciated!
> 
> MJ
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user