[PVE-User] 802.3ad Bonding Proxmox 1.4

Thu Nov 12 19:14:42 CET 2009

Excellent. You're welcome, Andrew!

By the way, the behavior of jumbo frames vs. non-jumbo is highly  
dependent on the switching platform, NICs and NIC driver behavior,  
etc. In a previous job I did a little comparison test of different SAN  
equipment, and at one point I tested raw transfer speed with and  
without jumbo frames on the iSCSI VLAN. At that time, with the servers  
and equipment I was using, it was worth about a 15 to 20% speed  
increase for essentially no work -- just make sure the Ethernet switch  
modules we were going to buy would support jumbo frames, and the  
server NICs already did, and we'd get better throughput "for free".

We ended up not buying that iSCSI SAN, so I didn't put this into  
practice at the time. But at my current job, we did purchase an iSCSI  
SAN, so remembering that lesson, I insisted that we implement it  
properly with jumbo frames. But the routing switch we happen to be  
using (Cisco Catalyst 4948) is extremely fast and can route and switch  
packets and frames all day long with equal speed, and with very low  
latency, regardless of jumbo or not. It turns out (for both iPerf raw  
testing and actually using the iSCSI SAN) that our servers can use  
almost the same bandwidth, jumbo or not! I think it was 940Mb/sec  
versus 985 or something. Sending the same amount of bandwidth without  
using jumbo requires about 6 times the number of packets and frames,  
but the routing switch can deliver those 6 times packets just fine.  
When we perform a backup-to-SAN-disk of a server that's not on a jumbo  
frame VLAN, the source-server-to-backup-server graph looks just like  
the backup-server-to-SAN graph when you look at bits/sec, but one  
looks 6 times as high as the other when you look at packets/sec.

So I guess my message is that jumbo frames are never a bad thing, but  
depending on other factors, it may not give you an immediate speed  
boost that you thought it might. Anyway, glad your Proxmox is working  
as expected. I still believe this is a terrific virtualization  
platform that is going to take over the world.

-- JeffS

On Nov 12, 2009, at 12:43 PM, Andrew Niemantsverdriet wrote:

> Ok, I have got it figured out. The jumbo frames on the other setup
> makes all the difference in the world for raw throughput. That being
> said I was able to get ~940Mbits/sec from several different hosts at
> the same time. Which means the network no longer chokes when a large
> OpenVZ machine is being transfered.
>
> So I guess everything is working correctly just not as I had  
> expected it to.
>
> Thanks Jeff for you explanation.
>
> _
> /-\ ndrew
>
> On Thu, Nov 12, 2009 at 5:58 AM, Andrew Niemantsverdriet
> <andrew at rocky.edu> wrote:
>> Jeff,
>>
>> I understand what you are saying kind of. I have a SAN network setup
>> and a similar configuration. I am getting 22,000Mbit/sec on the SAN
>> network, granted there are some differences like jumbo frames which
>> effect throughput. Again this was tested with iperf.
>>
>> So something is not right in my setup with Proxmox setup. I will do
>> some further investigation and see what I can come up with. Thanks  
>> for
>> your help.
>>
>>  _
>> /-\ ndrew
>>
>>
>>
>> On Wed, Nov 11, 2009 at 4:37 PM, Jeff Saxe <JSaxe at briworks.com>  
>> wrote:
>>> I'm not sure why 943Mb/sec "sucks" -- personally, I'd be pretty  
>>> pleased to
>>> get such bandwidth, if I asked for a virtual machine and you  
>>> hosted it for
>>> me on your Proxmox server.  :-)
>>> But seriously, you may not be taking into account precisely what  
>>> happens
>>> when you use a layer 2 Ethernet aggregate (EtherChannel, or port  
>>> channel).
>>> The accepted standards of layer 2 say that frames should not  
>>> arrive out of
>>> order from how they were transmitted, so the way a port-channeling  
>>> device
>>> treats each frame is to run it through some quick hash algorithm  
>>> (based on
>>> either source or destination MAC, IP, or layer 4 port numbers, or  
>>> some
>>> combination), and whatever the hash comes up with, it sends the  
>>> frame on
>>> that link out of the bundle. The result of this is that a single
>>> long-running conversation between two endpoints (for instance, one  
>>> long FTP
>>> transfer) is always going to choose the same Ethernet port over  
>>> and over for
>>> each frame, so even if you bond together 5 gig Ethernets, one file  
>>> transfer
>>> is going to go through only one of the five. So a speed of 943Mb/ 
>>> sec is not
>>> surprising -- likely, you are nearly saturating just one gig port  
>>> while the
>>> others remain idle.
>>> An Ethernet port channel gives you good redundancy, fast failover,  
>>> easy
>>> expansion, no need to use a routing protocol, no need to think  
>>> hard about
>>> spanning tree, safety from accidentally plugging into wrong ports  
>>> (when you
>>> use 802.3ad protocol), etc. But it does not automatically give you  
>>> high
>>> bandwidth for focused transmissions. It only gives you high average
>>> bandwidth in the larger case, where you have many hosts (or  
>>> several IP
>>> addresses on the same hosts, or many TCP conversations, again  
>>> depending on
>>> the frame distribution algorithm in use on each side of the  
>>> aggregate).
>>> Sorry if that messes up your plans.
>>> -- Jeff Saxe, Network Engineer
>>> Blue Ridge InternetWorks, Charlottesville, VA
>>> 434-817-0707 ext. 2024  /  JSaxe at briworks.com
>>>
>>>
>>>
>>> On Nov 11, 2009, at 5:13 PM, Andrew Niemantsverdriet wrote:
>>>
>>> I just went in and enabled STP the bridge is now and able to
>>> communicate. It is slow though. Still can't see more than 943Mbits/ 
>>> sec
>>> through the bond0 interface.
>>>
>>> # network interface settings
>>> auto lo
>>> iface lo inet loopback
>>>
>>> iface eth0 inet manual
>>>
>>> iface eth1 inet manual
>>>
>>> iface eth2 inet manual
>>>
>>> iface eth3 inet manual
>>>
>>> iface eth4 inet manual
>>>
>>> auto eth5
>>> iface eth5 inet static
>>> address  192.168.3.4
>>> netmask  255.255.255.0
>>>
>>> auto bond0
>>> iface bond0 inet manual
>>> slaves eth0 eth1 eth2 eth3 eth4
>>> bond_miimon 100
>>> bond_mode 802.3ad
>>>
>>> auto vmbr0
>>> iface vmbr0 inet static
>>> address  192.0.2.6
>>> netmask  255.255.255.0
>>> gateway 192.0.2.1
>>> bridge_ports bond0
>>> bridge_stp on
>>> bridge_fd 0
>>>
>>> The switch shows 802.3ad partners so that is working however the  
>>> speed
>>> sucks although that is better than not working.
>>>
>>> Any ideas?
>>>
>>>
>>>
>>
>>
>>
>> --
>>  _
>> /-\ ndrew Niemantsverdriet
>> Academic Computing
>> (406) 238-7360
>> Rocky Mountain College
>> 1511 Poly Dr.
>> Billings MT, 59102
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.proxmox.com/pipermail/pve-user/attachments/20091112/8056fa01/attachment.htm>