[PVE-User] networking adjustment | hope to get some feedback

Fri Jun 22 11:54:44 CEST 2018

Thank you Alexandre!

On 06/22/2018 02:54 AM, Alexandre DERUMIER wrote:
>>> For example: with the dual 10G LACP connection to each server, we can
>>> only use mtu size 1500. Are we loosing much there..? Or would there be a
>>> way around this, somehow?
> 
> you can setup mtu 9000 on your bridge and bond.
> if your vms have mtu 1500 (inside the vm), the packet will use 1500 mtu
> 
> 
>>> I tried also assigning the ceph cluster ip 10.10.89.10/11/12 to the
>>> bond0, instead of assigning it to vmbr0 as an alias. But in that setup I
>>> could never ping the other machine, so that somehow doesn't work. :-(
> 
> you can setup ip on interface which is plugged in a bridge.
> 
> but you can use vlan interface on the bond for example
> 
> 
> - management proxmox without vlan && ceph on dedicated vlan
> ------------------------------------------------------------
> auto bond0
> iface bond0 inet manual
> slaves eth1 eth2
> bond_miimon 100
> bond_mode 802.3ad
> bond_xmit_hash_policy layer3+4
> 
>   
> auto vmbr0
> iface vmbr0 inet static
> address a.b.c.10
> netmask 255.255.255.0
> gateway a.b.c.1
> bridge_ports bond0
> bridge_stp off
> bridge_fd 0
> 
> 
> #dedicated ceph vlan
> auto bond0.100
> iface bond0.100 inet static
> address ...
> netmask ....
> 
> 
> or with dedicated vlan for proxmox management && ceph
> ------------------------------------------------------
> 
> 
> 
> auto bond0
> iface bond0 inet manual
> slaves eth1 eth2
> bond_miimon 100
> bond_mode 802.3ad
> bond_xmit_hash_policy layer3+4
> 
> 
> auto vmbr0
> iface vmbr0 inet static
> bridge_ports bond0
> bridge_stp off
> bridge_fd 0
> 
> #dedicated proxmox vlan
> auto bond0.99
> iface bond0.99 inet static
> address a.b.c.10
> netmask 255.255.255.0
> gateway a.b.c.1
> 
> #dedicated ceph vlan
> auto bond0.100
> iface bond0.100 inet static
> address ...
> netmask ....
> 
> 
> 
> 
> 
> ----- Mail original -----
> De: "mj" <lists at merit.unu.edu>
> À: "proxmoxve" <pve-user at pve.proxmox.com>
> Envoyé: Jeudi 21 Juin 2018 14:31:28
> Objet: Re: [PVE-User] networking adjustment | hope to get some feedback
> 
> Hi,
> 
> So, I setup a test rig, with (only) two proxmox test-servers, with two
> NICs per server to test.
> 
> This /etc/network/interfaces seems to work well:
> 
>> iface eth1 inet manual
>>
>> iface eth2 inet manual
>>
>> auto bond0
>> iface bond0 inet manual
>> slaves eth1 eth2
>> bond_miimon 100
>> bond_mode 802.3ad
>> bond_xmit_hash_policy layer3+4
>>
>> auto vmbr0
>> iface vmbr0 inet static
>> address a.b.c.10
>> netmask 255.255.255.0
>> gateway a.b.c.1
>> bridge_ports bond0
>> bridge_stp off
>> bridge_fd 0
>> up ip addr add 10.10.89.10/24 dev vmbr0 || true
>> down ip addr del 10.10.89.10/24 dev vmbr0 || true
> 
> On the (procurve 5400) chassis, I configured the LACP like this:
>> trunk D1-D2 Trk1 LACP
> and
>> trunk D3-D4 Trk2 LACP
> resulting in this:
>> Procurve chassis(config)# show trunk
>>
>> Load Balancing Method: L3-based (default)
>>
>> Port | Name Type | Group Type
>> ---- + -------------------------------- --------- + ------ --------
>> D1 | Link to pve001 - 1 10GbE-T | Trk1 LACP
>> D2 | Link to pve001 - 2 10GbE-T | Trk1 LACP
>> D3 | Link to pve002 - 1 10GbE-T | Trk2 LACP
>> D4 | Link to pve002 - 2 10GbE-T | Trk2 LACP
> 
> The above config allows me to assign VLANs to lacp trunks ("Trk1",
> "Trk2") in the chassis webinterface like you would do with ports.
> 
> Then I did some reading on load balancing between the trunked ports, and
> figured that load balancing based on L4 would perhaps work better for
> us, so I changed it with
>> trunk-load-balance L4
> 
> Since we are running the public and cluster network over the same wires,
> I don't think we can enable jumbo frames. Or would there be a way to
> make ceph traffic use a specific vlan, so we can enable jumbo frames on
> that vlan?
> 
> I realise that this is perhaps all very specific to our environment, but
> again: if there is anyone here with insights, tips, trics, please,
> feedback is welcome.
> 
> For example: with the dual 10G LACP connection to each server, we can
> only use mtu size 1500. Are we loosing much there..? Or would there be a
> way around this, somehow?
> 
> I tried also assigning the ceph cluster ip 10.10.89.10/11/12 to the
> bond0, instead of assigning it to vmbr0 as an alias. But in that setup I
> could never ping the other machine, so that somehow doesn't work. :-(
> 
> (thinking that with the ceph ip on the bond0, my VMs would not be able
> to see that traffic..?)
> 
> Again: all feedback welcome.
> 
> MJ
> 
> 
> 
> On 06/18/2018 11:56 AM, mj wrote:
>> Hi all,
>>
>> After having bought some new networking equipment, and gaining more
>> insight over the last two years, I am planning to make some adjustments
>> to our proxmox/ceph setup, and I would *greatly* appreciate some
>> feedback :-)
>>
>> We are running a three-identical-server proxmox/ceph setup, with on each
>> server:
>>
>> NIC1 Ceph cluster and monitors on 10.10.89.10/11/12 (10G ethernet)
>> NIC2 clients and public ip a.b.c.10/11/12 (1G ethernet)
>>
>> Since we bought new hardware, I can connect each server to our HP
>> chassis, over a dual 10G bonded LACP connection.
>>
>> I obviously need to keep the (NIC1) public IP, but since the ceph
>> monitors ip is difficult to change, I'd like to keep the (NIC2)
>> 10.10.89.x as well.
>>
>> I also need to keep the (tagged and untagged) VLAN's for proxmox and the
>> VMs running on it.
>>
>> I realise that it used to be recommened to split cluster and client
>> traffic, but consensus nowadays on the ceph mailinglist seems to be:
>> keep it simple and don't split, unless specifically required. With this
>> in mind, I would also like to consolidate networking and run all traffic
>> over this dual lacp-bonded 10G connection to our HP chassis, including
>> the VLANs.
>>
>> But how to achieve this..? :-) (and here come the questions...)
>>
>> My idea is to first enable (active) LACP on our ProCurve 5400 chassis
>> ports, trunk type "LACP", but unsure about the "Trunk Group". Do I need
>> to select a different Truck Group (Trk1, Trk2 & Trk3) for each
>> dual-cable-connection to a server..?
>>
>> And will the port-configured VLANs on the lacp-member-ports (both tagged
>> and untagged) continue to flow normally through this lacp bond..?
>>
>> Then, about configuration on proxmox, would something like below do the
>> trick..?
>>
>> auto bond0
>> iface bond0 inet manual
>> slaves eth0 eth1
>> bond_miimon 100
>> bond_mode 802.3ad
>> bond_xmit_hash_policy layer2+3
>>
>> auto vmbr0
>> iface vmbr0 inet static
>> address a.b.c.10/11/12 (public IPs)
>> netmask 255.255.255.0
>> gateway a.b.c.1
>> bridge_ports bond0
>> bridge_stp off
>> bridge_fd 0
>> up ip addr add 10.10.89.10/11/12 dev vmbr0 || true (ceph mon IPs)
>> down ip addr del 10.100.222.1/24 dev vmbr0 || true
>>
>> Any feedback on the above? As this is production, I'd like to be
>> reasonably sure that this would work, before trying.
>>
>> Your comments will be very much appreciated!
>>
>> MJ
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 