<html><body><div style="font-family: Verdana; font-size: 10pt; color: #000000"><div><br></div><div><br></div><div data-marker="__SIG_PRE__"><table cellpadding="0" border="0" class="MsoNormalTable" style="font-family: 'Times New Roman';" data-mce-style="font-family: 'Times New Roman';"><tbody><tr><td style="padding: 0cm;" data-mce-style="padding: 0cm;"><p><span style="font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" data-mce-style="font-family: Calibri, sans-serif; color: #1f497d;"><img width="211" height="150" src="http://www.odiso.com/wp-content/uploads/2014/03/odiso.png" data-mce-src="http://www.odiso.com/wp-content/uploads/2014/03/odiso.png"></span></p></td><td width="100%" style="width: 1679px; padding: 0cm 0cm 0cm 6pt;" data-mce-style="width: 1679px; padding: 0cm 0cm 0cm 6pt;"><table cellspacing="4" cellpadding="0" border="0" class="MsoNormalTable"><tbody><tr><td style="padding: 3pt;" data-mce-style="padding: 3pt;"><p><b><span style="font-size: 12.5pt; font-family: Calibri, sans-serif; color: rgb(56, 139, 198);" data-mce-style="font-size: 12.5pt; font-family: Calibri, sans-serif; color: #388bc6;">Alexandre</span></b> <b><span style="font-size: 12.5pt; font-family: Calibri, sans-serif; color: rgb(66, 66, 66);" data-mce-style="font-size: 12.5pt; font-family: Calibri, sans-serif; color: #424242;">Derumier</span></b> <span style="font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" data-mce-style="font-family: Calibri, sans-serif; color: #1f497d;"><br></span><b><span style="font-size: 8pt; font-family: Calibri, sans-serif; color: rgb(66, 66, 66);" data-mce-style="font-size: 8pt; font-family: Calibri, sans-serif; color: #424242;">Ingénieur système et stockage</span></b></p></td></tr><tr><td style="padding: 3pt;" data-mce-style="padding: 3pt;"><p><b><span style="font-size: 8pt; font-family: Calibri, sans-serif; color: rgb(66, 66, 66);" data-mce-style="font-size: 8pt; font-family: Calibri, sans-serif; color: #424242;">Fixe :</span></b> <span style="font-size: 8pt; font-family: Calibri, sans-serif; color: rgb(99, 100, 102); text-align: right;" data-mce-style="font-size: 8pt; font-family: Calibri, sans-serif; color: #636466; text-align: right;">03 20 68 90 88</span> <span style="font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" data-mce-style="font-family: Calibri, sans-serif; color: #1f497d;"><br></span><b><span style="font-size: 8pt; font-family: Calibri, sans-serif; color: rgb(66, 66, 66);" data-mce-style="font-size: 8pt; font-family: Calibri, sans-serif; color: #424242;">Fax :</span></b> <span style="font-size: 8pt; font-family: Calibri, sans-serif; color: rgb(99, 100, 102); text-align: right;" data-mce-style="font-size: 8pt; font-family: Calibri, sans-serif; color: #636466; text-align: right;">03 20 68 90 81</span></p></td></tr><tr><td style="padding: 3pt;" data-mce-style="padding: 3pt;"><p><span style="font-size: 7.5pt; font-family: Calibri, sans-serif; color: gray;" data-mce-style="font-size: 7.5pt; font-family: Calibri, sans-serif; color: gray;">45 Bvd du Général Leclerc 59100 Roubaix</span> <span style="font-family: Calibri, sans-serif; color: rgb(31, 73, 125);" data-mce-style="font-family: Calibri, sans-serif; color: #1f497d;"><br></span><span style="font-size: 7.5pt; font-family: Calibri, sans-serif; color: gray;" data-mce-style="font-size: 7.5pt; font-family: Calibri, sans-serif; color: gray;">12 rue Marivaux 75002 Paris</span></p></td></tr><tr><td style="padding: 3pt; line-height: 20px;" data-mce-style="padding: 3pt; line-height: 20px;"><a target="_blank" href="https://twitter.com/OdisoHosting" data-mce-href="https://twitter.com/OdisoHosting"><img src="http://www.odiso.com/wp-content/uploads/2014/03/Twitter.png" data-mce-src="http://www.odiso.com/wp-content/uploads/2014/03/Twitter.png"></a> <a target="_blank" href="https://twitter.com/mindbaz" data-mce-href="https://twitter.com/mindbaz"><img src="http://www.odiso.com/wp-content/uploads/2014/03/Twitter.png" data-mce-src="http://www.odiso.com/wp-content/uploads/2014/03/Twitter.png"></a> <a target="_blank" href="https://www.linkedin.com/company/odiso" data-mce-href="https://www.linkedin.com/company/odiso"><img src="http://www.odiso.com/wp-content/uploads/2014/03/Linkedin.png" data-mce-src="http://www.odiso.com/wp-content/uploads/2014/03/Linkedin.png"></a> <a target="_blank" href="http://www.viadeo.com/fr/company/odiso" data-mce-href="http://www.viadeo.com/fr/company/odiso"><img src="http://www.odiso.com/wp-content/uploads/2014/03/Viadeo.png" data-mce-src="http://www.odiso.com/wp-content/uploads/2014/03/Viadeo.png"></a> <a target="_blank" href="https://www.facebook.com/monsiteestlent" data-mce-href="https://www.facebook.com/monsiteestlent"><img src="http://www.odiso.com/wp-content/uploads/2014/03/Facebook.png" data-mce-src="http://www.odiso.com/wp-content/uploads/2014/03/Facebook.png"></a></td></tr></tbody></table></td></tr></tbody></table><table style="font-family: 'Times New Roman';" data-mce-style="font-family: 'Times New Roman';"><tbody><tr><td><p style="font-style: italic;" data-mce-style="font-style: italic;"><a href="http://www.monsiteestlent.com/" style="text-decoration: none;" data-mce-href="http://www.monsiteestlent.com/" data-mce-style="text-decoration: none;"><span color="#388bc6" data-mce-style="color: #388bc6;" style="color: #388bc6;">MonSiteEstLent.com</span></a> - Blog dédié à la webperformance et la gestion de pics de trafic</p></td></tr></tbody></table></div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>De: </b>"Cesar Peschiera" <brain@click.com.py><br><b>À: </b>"aderumier" <aderumier@odiso.com><br><b>Cc: </b>"pve-devel" <pve-devel@pve.proxmox.com><br><b>Envoyé: </b>Samedi 3 Janvier 2015 03:41:20<br><b>Objet: </b>Re: [pve-devel] Quorum problems with NICs Intel of 10 Gb/s and VMsturns off<br></div><div><br></div><div data-marker="__QUOTED_TEXT__">Hi Alexandre<br><br>Many thanks for your reply, which is much appreciated.<br><br>Unfortunately, your suggestion does not work for me, so i will comment the<br>results.<br><br>Between some comments, also in this message i have 7 questions for you, and<br>i'll be very grateful if you can answer me.<br><br>Only for that be clear about of the version of the programs that i have<br>installed in the nodes that has a behaviour strange (2 of 6 PVE nodes):<br>shell> pveversion -v<br>proxmox-ve-2.6.32: 3.3-139 (running kernel: 3.10.0-5-pve)<br>pve-manager: 3.3-5 (running version: 3.3-5/bfebec03)<br>pve-kernel-3.10.0-5-pve: 3.10.0-19<br>pve-kernel-2.6.32-34-pve: 2.6.32-139<br>lvm2: 2.02.98-pve4<br>clvm: 2.02.98-pve4<br>corosync-pve: 1.4.7-1<br>openais-pve: 1.1.4-3<br>libqb0: 0.11.1-2<br>redhat-cluster-pve: 3.2.0-2<br>resource-agents-pve: 3.9.2-4<br>fence-agents-pve: 4.0.10-1<br>pve-cluster: 3.0-15<br>qemu-server: 3.3-5 <------especial patch created by Alexandre for me<br>pve-firmware: 1.1-3<br>libpve-common-perl: 3.0-19<br>libpve-access-control: 3.0-15<br>libpve-storage-perl: 3.0-25<br>pve-libspice-server1: 0.12.4-3<br>vncterm: 1.1-8<br>vzctl: 4.0-1pve6<br>vzprocps: 2.0.11-2<br>vzquota: 3.1-2<br>pve-qemu-kvm: 2.2-2 <------especial patch created by Alexandre for me<br>ksm-control-daemon: 1.1-1<br>glusterfs-client: 3.5.2-1<br><br>After a minute of apply on only a node (pve6), these commands, i lost the<br>quorum in two nodes (pve5 and pve6):<br>The commands executed on only a node (pve6):<br>echo 1 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping<br>echo 0 > /sys/class/net/vmbr0/bridge/multicast_querier<br><br>The error message in the node where i applied the commands (pve6) is this:<br>Message from syslogd@pve6 at Jan 2 20:58:32 ...<br>rgmanager[4912]: #1: Quorum Dissolved<br><br>And as collateral effect, as the pve5 node is configured with HA for a VM<br>with a "failover domain" between pve5 and pve6 (the nodes), also pve5 has<br>loss the quorum and the VM that is in HA turns off brutally.<br><br>These are the error messages in the screen of the pve5 node:<br>[ 61.246002] dlm: rgmanager: send_repeat_remove dir 6 rg="pvevm:112"<br>[119373.380111] dlm: closing connection to node 1"<br>[119373:300150] dlm: closing connection to node 2"<br>[119373:380182] dlm: closing connection to node 3"<br>[119373:300205] dlm: closing connection to node 4"<br>[119373:380229] dlm: closing connection to node 6"<br>[119373:300268] dlm: closing connection to node 7"<br>[119373:380319] dlm: closing connection to node 8"<br>[119545:042242] dlm: closing connection to node 3"<br>[119545:042264] dlm: closing connection to node 8"<br>[119545:042281] dlm: closing connection to node 7"<br>[119545:042300] dlm: closing connection to node 2"<br>[119545:042316] dlm: closing connection to node 1"<br>[119545:042331] dlm: closing connection to node 4"<br>[119545:042347] dlm: closing connection to node 5"<br>[119545:042891] dlm: dlm user daemon left 1 lockspaces<br><br>So i believe that pve has a bug and a great problem, but i am not sure of<br>that, but i know that if the pve6 node for some reason turns off brutally,<br>the pve5 node will lose quorum and his VM in HA also will turn off, and this<br>behaviour will give me several problems due that actually i don't know what<br>i must do for start the VM in the node that is alive?<br><br>So my questions are:<br>1) Why the pve5 node lost the quorum if i don't applied any change in this<br>node?<br>(this node always had the multicast snooping filter disabled)<br>2) Why the VM that is running on pve5 node and also is configured in HA<br>turns off brutally?<br>3) If it is a bug, can someone apply a patch to code?<br><br>Moreover, talking about of firewall enabled for the VMs:<br>I remember that +/- 1 month ago, i tried apply to the firewall a rule<br>restrictive of access of the IP address of cluster communication to the VMs<br>without successful, ie, with a policy of firewall by default of "allow",<br>each time that i enable this unique and restrictive rule to the VM, the VM<br>lose all network communication. Maybe i am wrong in something.<br><br>So i would like to ask you somethings:<br><br>4) Can you do a test, and then tell me the results?<br>5) If the results are positives, can you tell me how do it?<br>6) And if the results are negatives, can you apply a patch to code?<br><br>Moreover, the last question:<br>7) As each PVE node has his "firewall" tag in the PVE GUI, i guess that such<br>option is for apply firewall rules of in/out that affect only to this node,<br>right?, or for what exist such option?<br><br><br><br>----- Original Message ----- <br>From: "Alexandre DERUMIER" <aderumier@odiso.com><br>To: "Cesar Peschiera" <brain@click.com.py><br>Cc: "pve-devel" <pve-devel@pve.proxmox.com><br>Sent: Friday, January 02, 2015 5:40 AM<br>Subject: Re: [pve-devel] Quorum problems with NICs Intel of 10 Gb/s and<br>VMsturns off<br><br><br>Hi,<br><br>>>But as i need that the VMs and the PVE host can be accessed from any<br>>>workstation, the vlan option isn't a option useful for me.<br>Ok<br><br><br>>>And about of cluster communication and the VMs, as i don't want that the<br>>>multicast packages go to the VMs, i believe that i can cut it for the VMs<br>>>of<br>>>two modes:<br>>><br>>>a) Removing the option "post-up echo 0 ><br>>>/sys/devices/virtual/net/vmbr0/bridge/multicast_snooping " to my NIC<br>>>configuration of the PVE host if i will have a behaviour stable.<br><br>Yes, indeed you can enable snooping to filter multicast<br><br>>>b) By firewall will be very easy, since that i know the IP address of<br>>>origin<br>>>of cluster communication, but unfortunately the wiki of PVE don't show<br>>>clearly how can i apply it, ie, i see the "firewall" tag in datacenter,<br>>>PVE<br>>>hosts and in the network configuration of the VMs, and the wiki don't says<br>>>nothing about of this, for me, with a global configuration that affect to<br>>>all VMs of the cluster will be wonderfull using IPset or some other way<br>>>that<br>>>be simple of apply.<br><br>I think you can create a security group with a rule which block the<br>multicast adress of your pve cluster<br><br>#pvecm status|grep "Multicast addresses"<br><br>to get your cluster multicast address<br><br>Then add this security group to each vm.<br><br><br>(Currently datacenter rules apply only on hosts IN|OUT iptables rules, but<br>not in FORWARD iptables rules which is used by vms)<br><br><br><br><br><br><br>----- Mail original -----<br>De: "Cesar Peschiera" <brain@click.com.py><br>À: "aderumier" <aderumier@odiso.com><br>Cc: "pve-devel" <pve-devel@pve.proxmox.com><br>Envoyé: Vendredi 2 Janvier 2015 05:10:08<br>Objet: Re: [pve-devel] Quorum problems with NICs Intel of 10 Gb/s and<br>VMsturns off<br><br>Hi Alexandre.<br><br>Thanks for your reply.<br><br>But as i need that the VMs and the PVE host can be accessed from any<br>workstation, the vlan option isn't a option useful for me.<br><br>Anyway, i am testing with I/OAT DMA Engine enabled in the Bios Hardware,<br>that after some days with few activity, the CMAN cluster is stable, soon i<br>will prove with a lot of network activity .<br><br>And about of cluster communication and the VMs, as i don't want that the<br>multicast packages go to the VMs, i believe that i can cut it for the VMs of<br>two modes:<br><br>a) Removing the option "post-up echo 0 ><br>/sys/devices/virtual/net/vmbr0/bridge/multicast_snooping " to my NIC<br>configuration of the PVE host if i will have a behaviour stable.<br><br>b) By firewall will be very easy, since that i know the IP address of origin<br>of cluster communication, but unfortunately the wiki of PVE don't show<br>clearly how can i apply it, ie, i see the "firewall" tag in datacenter, PVE<br>hosts and in the network configuration of the VMs, and the wiki don't says<br>nothing about of this, for me, with a global configuration that affect to<br>all VMs of the cluster will be wonderfull using IPset or some other way that<br>be simple of apply.<br><br>Do you have some idea of how avoid that multicast packages go to the VMs in<br>a stable mode? and how apply it?<br><br>----- Original Message ----- <br>From: "Alexandre DERUMIER" <aderumier@odiso.com><br>To: "Cesar Peschiera" <brain@click.com.py><br>Cc: "pve-devel" <pve-devel@pve.proxmox.com><br>Sent: Wednesday, December 31, 2014 3:33 AM<br>Subject: Re: [pve-devel] Quorum problems with NICs Intel of 10 Gb/s and<br>VMsturns off<br><br><br>Hi Cesar,<br><br>I think I totaly forgot that we can't add an ip on an interface slave of a<br>bridge.<br><br>Myself I'm using a tagged vlan interface for the cluster communication<br><br>something like:<br><br>auto bond0<br>iface bond0 inet manual<br>slaves eth0 eth2<br>bond_miimon 100<br>bond_mode 802.3ad<br>bond_xmit_hash_policy layer2<br><br>auto bond0.100<br>iface bond0 inet static<br>address 192.100.100.50<br>netmask 255.255.255.0<br>gateway 192.100.100.4<br><br>auto vmbr0<br>iface vmbr0 inet manual<br>bridge_ports bond0<br>bridge_stp off<br>bridge_fd 0<br>post-up echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping<br><br>----- Mail original ----- <br>De: "Cesar Peschiera" <brain@click.com.py><br>À: "aderumier" <aderumier@odiso.com><br>Cc: "pve-devel" <pve-devel@pve.proxmox.com><br>Envoyé: Mercredi 31 Décembre 2014 05:01:37<br>Objet: Re: [pve-devel] Quorum problems with NICs Intel of 10 Gb/s and<br>VMsturns off<br><br>Hi Alexandre<br><br>Today, and after a week, again a node lost the cluster communication. So i<br>changed the configuration of the Bios Hardware to "I/OAT DMA enabled" (that<br>work very well in others nodes Dell R320 with NICs of 1 Gb/s).<br><br>Moreover, trying to follow your advice of to put 192.100.100.51 ip address<br>directly to bond0 and not in vmbr0, when i reboot the node, it is totally<br>isolated, and i see a message that says that vmbr0 missing a IP address.<br>Also the node is totally isolated when i apply this ip address to vmbr0:<br>0.0.0.0/255.255.255.255<br><br>In practical terms, can you tell me how can i add a IP address to bond0 and<br>also have a bridge for these same NICs?<br><br>- Now, this is my configuration:<br>auto bond0<br>iface bond0 inet manual<br>slaves eth0 eth2<br>bond_miimon 100<br>bond_mode 802.3ad<br>bond_xmit_hash_policy layer2<br><br>auto vmbr0<br>iface vmbr0 inet static<br>address 192.100.100.50<br>netmask 255.255.255.0<br>gateway 192.100.100.4<br>bridge_ports bond0<br>bridge_stp off<br>bridge_fd 0<br>post-up echo 0 ><br>/sys/devices/virtual/net/vmbr0/bridge/multicast_snooping<br><br><br>----- Original Message ----- <br>From: "Alexandre DERUMIER" <aderumier@odiso.com><br>To: "Cesar Peschiera" <brain@click.com.py><br>Cc: "pve-devel" <pve-devel@pve.proxmox.com><br>Sent: Friday, December 19, 2014 7:59 AM<br>Subject: Re: [pve-devel] Quorum problems with NICs Intel of 10 Gb/s and<br>VMsturns off<br><br><br>maybe can you try to put 192.100.100.51 ip address directly to bond0,<br><br>to avoid corosync traffic going through to vmbr0.<br><br>(I remember some old offloading bugs with 10gbe nic and linux bridge)<br><br><br>----- Mail original ----- <br>De: "Cesar Peschiera" <brain@click.com.py><br>À: "aderumier" <aderumier@odiso.com><br>Cc: "pve-devel" <pve-devel@pve.proxmox.com><br>Envoyé: Vendredi 19 Décembre 2014 11:08:33<br>Objet: Re: [pve-devel] Quorum problems with NICs Intel of 10 Gb/s and<br>VMsturns off<br><br>>can you post your /etc/network/interfaces of theses 10gb/s nodes ?<br><br>This is my configuration:<br>Note: The LAN use 192.100.100.0/24<br><br>#Network interfaces<br>auto lo<br>iface lo inet loopback<br><br>iface eth0 inet manual<br>iface eth1 inet manual<br>iface eth2 inet manual<br>iface eth3 inet manual<br>iface eth4 inet manual<br>iface eth5 inet manual<br>iface eth6 inet manual<br>iface eth7 inet manual<br>iface eth8 inet manual<br>iface eth9 inet manual<br>iface eth10 inet manual<br>iface eth11 inet manual<br><br>#PVE Cluster and VMs (NICs are of 10 Gb/s):<br>auto bond0<br>iface bond0 inet manual<br>slaves eth0 eth2<br>bond_miimon 100<br>bond_mode 802.3ad<br>bond_xmit_hash_policy layer2<br><br>#PVE Cluster and VMs:<br>auto vmbr0<br>iface vmbr0 inet static<br>address 192.100.100.51<br>netmask 255.255.255.0<br>gateway 192.100.100.4<br>bridge_ports bond0<br>bridge_stp off<br>bridge_fd 0<br>post-up echo 0 ><br>/sys/devices/virtual/net/vmbr0/bridge/multicast_snooping<br>post-up echo 1 > /sys/class/net/vmbr0/bridge/multicast_querier<br><br>#A link for DRBD (NICs are of 10 Gb/s):<br>auto bond401<br>iface bond401 inet static<br>address 10.1.1.51<br>netmask 255.255.255.0<br>slaves eth1 eth3<br>bond_miimon 100<br>bond_mode balance-rr<br>mtu 9000<br><br>#Other link for DRBD (NICs are of 10 Gb/s):<br>auto bond402<br>iface bond402 inet static<br>address 10.2.2.51<br>netmask 255.255.255.0<br>slaves eth4 eth6<br>bond_miimon 100<br>bond_mode balance-rr<br>mtu 9000<br><br>#Other link for DRBD (NICs are of 10 Gb/s):<br>auto bond403<br>iface bond403 inet static<br>address 10.3.3.51<br>netmask 255.255.255.0<br>slaves eth5 eth7<br>bond_miimon 100<br>bond_mode balance-rr<br>mtu 9000<br><br>#A link for the NFS-Backups (NICs are of 1 Gb/s):<br>auto bond10<br>iface bond10 inet static<br>address 10.100.100.51<br>netmask 255.255.255.0<br>slaves eth8 eth10<br>bond_miimon 100<br>bond_mode balance-rr<br>#bond_mode active-backup<br>mtu 9000<br></div></div></body></html>