<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Excellent. You're welcome, Andrew!<div><br></div><div>By the way, the behavior of jumbo frames vs. non-jumbo is highly dependent on the switching platform, NICs and NIC driver behavior, etc. In a previous job I did a little comparison test of different SAN equipment, and at one point I tested raw transfer speed with and without jumbo frames on the iSCSI VLAN. At that time, with the servers and equipment I was using, it was worth about a 15 to 20% speed increase for essentially no work -- just make sure the Ethernet switch modules we were going to buy would support jumbo frames, and the server NICs already did, and we'd get better throughput "for free".</div><div><br></div><div>We ended up not buying that iSCSI SAN, so I didn't put this into practice at the time. But at my current job, we did purchase an iSCSI SAN, so remembering that lesson, I insisted that we implement it properly with jumbo frames. But the routing switch we happen to be using (Cisco Catalyst 4948) is extremely fast and can route and switch packets and frames all day long with equal speed, and with very low latency, regardless of jumbo or not. It turns out (for both iPerf raw testing and actually using the iSCSI SAN) that our servers can use almost the same bandwidth, jumbo or not! I think it was 940Mb/sec versus 985 or something. Sending the same amount of bandwidth without using jumbo requires about 6 times the number of packets and frames, but the routing switch can deliver those 6 times packets just fine. When we perform a backup-to-SAN-disk of a server that's not on a jumbo frame VLAN, the source-server-to-backup-server graph looks just like the backup-server-to-SAN graph when you look at bits/sec, but one looks 6 times as high as the other when you look at packets/sec.</div><div><br></div><div>So I guess my message is that jumbo frames are never a bad thing, but depending on other factors, it may not give you an immediate speed boost that you thought it might. Anyway, glad your Proxmox is working as expected. I still believe this is a <i>terrific</i> virtualization platform that is going to take over the world.<br><div> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><br class="Apple-interchange-newline">-- JeffS</span><div><br></div></div></span><br class="Apple-interchange-newline"> </div><br><div><div>On Nov 12, 2009, at 12:43 PM, Andrew Niemantsverdriet wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div>Ok, I have got it figured out. The jumbo frames on the other setup<br>makes all the difference in the world for raw throughput. That being<br>said I was able to get ~940Mbits/sec from several different hosts at<br>the same time. Which means the network no longer chokes when a large<br>OpenVZ machine is being transfered.<br><br>So I guess everything is working correctly just not as I had expected it to.<br><br>Thanks Jeff for you explanation.<br><br> _<br>/-\ ndrew<br><br>On Thu, Nov 12, 2009 at 5:58 AM, Andrew Niemantsverdriet<br><<a href="mailto:andrew@rocky.edu">andrew@rocky.edu</a>> wrote:<br><blockquote type="cite">Jeff,<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">I understand what you are saying kind of. I have a SAN network setup<br></blockquote><blockquote type="cite">and a similar configuration. I am getting 22,000Mbit/sec on the SAN<br></blockquote><blockquote type="cite">network, granted there are some differences like jumbo frames which<br></blockquote><blockquote type="cite">effect throughput. Again this was tested with iperf.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">So something is not right in my setup with Proxmox setup. I will do<br></blockquote><blockquote type="cite">some further investigation and see what I can come up with. Thanks for<br></blockquote><blockquote type="cite">your help.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"> _<br></blockquote><blockquote type="cite">/-\ ndrew<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">On Wed, Nov 11, 2009 at 4:37 PM, Jeff Saxe <<a href="mailto:JSaxe@briworks.com">JSaxe@briworks.com</a>> wrote:<br></blockquote><blockquote type="cite"><blockquote type="cite">I'm not sure why 943Mb/sec "sucks" -- personally, I'd be pretty pleased to<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">get such bandwidth, if I asked for a virtual machine and you hosted it for<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">me on your Proxmox server. :-)<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">But seriously, you may not be taking into account precisely what happens<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">when you use a layer 2 Ethernet aggregate (EtherChannel, or port channel).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">The accepted standards of layer 2 say that frames should not arrive out of<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">order from how they were transmitted, so the way a port-channeling device<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">treats each frame is to run it through some quick hash algorithm (based on<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">either source or destination MAC, IP, or layer 4 port numbers, or some<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">combination), and whatever the hash comes up with, it sends the frame on<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">that link out of the bundle. The result of this is that a single<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">long-running conversation between two endpoints (for instance, one long FTP<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">transfer) is always going to choose the same Ethernet port over and over for<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">each frame, so even if you bond together 5 gig Ethernets, one file transfer<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">is going to go through only one of the five. So a speed of 943Mb/sec is not<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">surprising -- likely, you are nearly saturating just one gig port while the<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">others remain idle.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">An Ethernet port channel gives you good redundancy, fast failover, easy<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">expansion, no need to use a routing protocol, no need to think hard about<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">spanning tree, safety from accidentally plugging into wrong ports (when you<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">use 802.3ad protocol), etc. But it does not automatically give you high<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">bandwidth for focused transmissions. It only gives you high average<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">bandwidth in the larger case, where you have many hosts (or several IP<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">addresses on the same hosts, or many TCP conversations, again depending on<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">the frame distribution algorithm in use on each side of the aggregate).<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Sorry if that messes up your plans.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">-- Jeff Saxe, Network Engineer<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Blue Ridge InternetWorks, Charlottesville, VA<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">434-817-0707 ext. 2024 / <a href="mailto:JSaxe@briworks.com">JSaxe@briworks.com</a><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">On Nov 11, 2009, at 5:13 PM, Andrew Niemantsverdriet wrote:<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">I just went in and enabled STP the bridge is now and able to<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">communicate. It is slow though. Still can't see more than 943Mbits/sec<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">through the bond0 interface.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"># network interface settings<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">auto lo<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">iface lo inet loopback<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">iface eth0 inet manual<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">iface eth1 inet manual<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">iface eth2 inet manual<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">iface eth3 inet manual<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">iface eth4 inet manual<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">auto eth5<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">iface eth5 inet static<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">address 192.168.3.4<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">netmask 255.255.255.0<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">auto bond0<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">iface bond0 inet manual<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">slaves eth0 eth1 eth2 eth3 eth4<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">bond_miimon 100<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">bond_mode 802.3ad<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">auto vmbr0<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">iface vmbr0 inet static<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">address 192.0.2.6<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">netmask 255.255.255.0<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">gateway 192.0.2.1<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">bridge_ports bond0<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">bridge_stp on<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">bridge_fd 0<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">The switch shows 802.3ad partners so that is working however the speed<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">sucks although that is better than not working.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">Any ideas?<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">--<br></blockquote><blockquote type="cite"> _<br></blockquote><blockquote type="cite">/-\ ndrew Niemantsverdriet<br></blockquote><blockquote type="cite">Academic Computing<br></blockquote><blockquote type="cite">(406) 238-7360<br></blockquote><blockquote type="cite">Rocky Mountain College<br></blockquote><blockquote type="cite">1511 Poly Dr.<br></blockquote><blockquote type="cite">Billings MT, 59102<br></blockquote><blockquote type="cite"><br></blockquote><br><br></div></blockquote></div><br></div></body></html>