[PVE-User] networking adjustment | hope to get some feedback
Alexandre DERUMIER
aderumier at odiso.com
Fri Jun 22 02:54:48 CEST 2018
>>For example: with the dual 10G LACP connection to each server, we can
>>only use mtu size 1500. Are we loosing much there..? Or would there be a
>>way around this, somehow?
you can setup mtu 9000 on your bridge and bond.
if your vms have mtu 1500 (inside the vm), the packet will use 1500 mtu
>>I tried also assigning the ceph cluster ip 10.10.89.10/11/12 to the
>>bond0, instead of assigning it to vmbr0 as an alias. But in that setup I
>>could never ping the other machine, so that somehow doesn't work. :-(
you can setup ip on interface which is plugged in a bridge.
but you can use vlan interface on the bond for example
- management proxmox without vlan && ceph on dedicated vlan
------------------------------------------------------------
auto bond0
iface bond0 inet manual
slaves eth1 eth2
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4
auto vmbr0
iface vmbr0 inet static
address a.b.c.10
netmask 255.255.255.0
gateway a.b.c.1
bridge_ports bond0
bridge_stp off
bridge_fd 0
#dedicated ceph vlan
auto bond0.100
iface bond0.100 inet static
address ...
netmask ....
or with dedicated vlan for proxmox management && ceph
------------------------------------------------------
auto bond0
iface bond0 inet manual
slaves eth1 eth2
bond_miimon 100
bond_mode 802.3ad
bond_xmit_hash_policy layer3+4
auto vmbr0
iface vmbr0 inet static
bridge_ports bond0
bridge_stp off
bridge_fd 0
#dedicated proxmox vlan
auto bond0.99
iface bond0.99 inet static
address a.b.c.10
netmask 255.255.255.0
gateway a.b.c.1
#dedicated ceph vlan
auto bond0.100
iface bond0.100 inet static
address ...
netmask ....
----- Mail original -----
De: "mj" <lists at merit.unu.edu>
À: "proxmoxve" <pve-user at pve.proxmox.com>
Envoyé: Jeudi 21 Juin 2018 14:31:28
Objet: Re: [PVE-User] networking adjustment | hope to get some feedback
Hi,
So, I setup a test rig, with (only) two proxmox test-servers, with two
NICs per server to test.
This /etc/network/interfaces seems to work well:
> iface eth1 inet manual
>
> iface eth2 inet manual
>
> auto bond0
> iface bond0 inet manual
> slaves eth1 eth2
> bond_miimon 100
> bond_mode 802.3ad
> bond_xmit_hash_policy layer3+4
>
> auto vmbr0
> iface vmbr0 inet static
> address a.b.c.10
> netmask 255.255.255.0
> gateway a.b.c.1
> bridge_ports bond0
> bridge_stp off
> bridge_fd 0
> up ip addr add 10.10.89.10/24 dev vmbr0 || true
> down ip addr del 10.10.89.10/24 dev vmbr0 || true
On the (procurve 5400) chassis, I configured the LACP like this:
> trunk D1-D2 Trk1 LACP
and
> trunk D3-D4 Trk2 LACP
resulting in this:
> Procurve chassis(config)# show trunk
>
> Load Balancing Method: L3-based (default)
>
> Port | Name Type | Group Type
> ---- + -------------------------------- --------- + ------ --------
> D1 | Link to pve001 - 1 10GbE-T | Trk1 LACP
> D2 | Link to pve001 - 2 10GbE-T | Trk1 LACP
> D3 | Link to pve002 - 1 10GbE-T | Trk2 LACP
> D4 | Link to pve002 - 2 10GbE-T | Trk2 LACP
The above config allows me to assign VLANs to lacp trunks ("Trk1",
"Trk2") in the chassis webinterface like you would do with ports.
Then I did some reading on load balancing between the trunked ports, and
figured that load balancing based on L4 would perhaps work better for
us, so I changed it with
> trunk-load-balance L4
Since we are running the public and cluster network over the same wires,
I don't think we can enable jumbo frames. Or would there be a way to
make ceph traffic use a specific vlan, so we can enable jumbo frames on
that vlan?
I realise that this is perhaps all very specific to our environment, but
again: if there is anyone here with insights, tips, trics, please,
feedback is welcome.
For example: with the dual 10G LACP connection to each server, we can
only use mtu size 1500. Are we loosing much there..? Or would there be a
way around this, somehow?
I tried also assigning the ceph cluster ip 10.10.89.10/11/12 to the
bond0, instead of assigning it to vmbr0 as an alias. But in that setup I
could never ping the other machine, so that somehow doesn't work. :-(
(thinking that with the ceph ip on the bond0, my VMs would not be able
to see that traffic..?)
Again: all feedback welcome.
MJ
On 06/18/2018 11:56 AM, mj wrote:
> Hi all,
>
> After having bought some new networking equipment, and gaining more
> insight over the last two years, I am planning to make some adjustments
> to our proxmox/ceph setup, and I would *greatly* appreciate some
> feedback :-)
>
> We are running a three-identical-server proxmox/ceph setup, with on each
> server:
>
> NIC1 Ceph cluster and monitors on 10.10.89.10/11/12 (10G ethernet)
> NIC2 clients and public ip a.b.c.10/11/12 (1G ethernet)
>
> Since we bought new hardware, I can connect each server to our HP
> chassis, over a dual 10G bonded LACP connection.
>
> I obviously need to keep the (NIC1) public IP, but since the ceph
> monitors ip is difficult to change, I'd like to keep the (NIC2)
> 10.10.89.x as well.
>
> I also need to keep the (tagged and untagged) VLAN's for proxmox and the
> VMs running on it.
>
> I realise that it used to be recommened to split cluster and client
> traffic, but consensus nowadays on the ceph mailinglist seems to be:
> keep it simple and don't split, unless specifically required. With this
> in mind, I would also like to consolidate networking and run all traffic
> over this dual lacp-bonded 10G connection to our HP chassis, including
> the VLANs.
>
> But how to achieve this..? :-) (and here come the questions...)
>
> My idea is to first enable (active) LACP on our ProCurve 5400 chassis
> ports, trunk type "LACP", but unsure about the "Trunk Group". Do I need
> to select a different Truck Group (Trk1, Trk2 & Trk3) for each
> dual-cable-connection to a server..?
>
> And will the port-configured VLANs on the lacp-member-ports (both tagged
> and untagged) continue to flow normally through this lacp bond..?
>
> Then, about configuration on proxmox, would something like below do the
> trick..?
>
> auto bond0
> iface bond0 inet manual
> slaves eth0 eth1
> bond_miimon 100
> bond_mode 802.3ad
> bond_xmit_hash_policy layer2+3
>
> auto vmbr0
> iface vmbr0 inet static
> address a.b.c.10/11/12 (public IPs)
> netmask 255.255.255.0
> gateway a.b.c.1
> bridge_ports bond0
> bridge_stp off
> bridge_fd 0
> up ip addr add 10.10.89.10/11/12 dev vmbr0 || true (ceph mon IPs)
> down ip addr del 10.100.222.1/24 dev vmbr0 || true
>
> Any feedback on the above? As this is production, I'd like to be
> reasonably sure that this would work, before trying.
>
> Your comments will be very much appreciated!
>
> MJ
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
_______________________________________________
pve-user mailing list
pve-user at pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
More information about the pve-user
mailing list