[pve-devel] Error in PVE with win2008r2 and 256GB RAM

Thu Dec 18 08:45:53 CET 2014

>Do you mix 2.6.32 and 3.10 kernel in your cluster ?
Yes, but the problem is only on the nodes that has NICs of 10 Gb/s, the 
other PVE nodes that has NICs of 1 Gb/s and the 3.10 kernel never lost the 
cluster communication.

>What are your hardware switches ?
Dell N2024 (managed, and for the moment with "igmp snooping" disabled)

----- Original Message ----- 
From: "Alexandre DERUMIER" <aderumier at odiso.com>
To: "Cesar Peschiera" <brain at click.com.py>
Cc: "pve-devel" <pve-devel at pve.proxmox.com>
Sent: Thursday, December 18, 2014 4:11 AM
Subject: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM

>>Many thanks for your answer, but i am not sure if it is a good idea due to
>>that i don't understand the advantage since that i can disable the huge
>>pages of this mode:

They are 2 modes for hugepages,

the transparent hugepage mode, managed by the kernel

But also, old way, manual hugepages (aka hugetlbfs, mounted in 
/dev/hugepage..)

from:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-memory-transhuge.html

" However, transparent hugepage mode is not recommended for database 
workloads."

So, hugepages are always usefull for big memory vms, because it's reduce cpu 
usage on memory access.
But transparent hugepage sometimes don't work good with some workloads like 
database.

I think than manually defined hugepages can give good an extra boost vs 
disable TBL.

>>Moreover, i have a serious problem with the PVE cluster communication in 
>>two
>>of eight PVE nodes, and with a VM, if you can help me, i will be extremely
>>grateful, please see this link:
>>http://forum.proxmox.com/threads/20523-Quorum-problems-with-PVE-2-3-and-3-3?p=104995#post104995

Do you mix 2.6.32 and 3.10 kernel in your cluster ?
(I have had some strange problems with mixed kernel, never find the 
problem).

Also, this could be a multicast snooping problem.

What are your hardware switches ?
Myself, I disable snooping on linux vmbr, enable snooping on physical 
swiches + igmp querier on physical switches.

----- Mail original -----
De: "Cesar Peschiera" <brain at click.com.py>
À: "pve-devel" <pve-devel at pve.proxmox.com>, "aderumier" 
<aderumier at odiso.com>
Envoyé: Jeudi 18 Décembre 2014 07:09:11
Objet: Fw: [pve-devel] Error in PVE with win2008r2 and 256GB RAM

Hi Alexandre

Many thanks for your answer, but i am not sure if it is a good idea due to
that i don't understand the advantage since that i can disable the huge
pages of this mode:

shell> vim /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="...transparent_hugepage=never"

shell> update-grub

Moreover, i have a serious problem with the PVE cluster communication in two
of eight PVE nodes, and with a VM, if you can help me, i will be extremely
grateful, please see this link:
http://forum.proxmox.com/threads/20523-Quorum-problems-with-PVE-2-3-and-3-3?p=104995#post104995

Best regards
Cesar

> ----- Original Message ----- 
> From: "Alexandre DERUMIER" <aderumier at odiso.com>
> To: "Cesar Peschiera" <brain at click.com.py>
> Cc: "pve-devel" <pve-devel at pve.proxmox.com>
> Sent: Thursday, December 18, 2014 2:24 AM
> Subject: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>
>
> Note that currently,
>
> hugepages are managed with transparent hugepage mecanism.
>
> But it's seem that we can defined manually hugepages by numa nodes
>
> -object
> memory-backend-file,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,size=1024M,id=ram-node0
> -numa node,nodeid=0,cpus=0,memdev=ram-node0
> -object
> memory-backend-file,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,size=1024M,id=ram-node1
> -numa node,nodeid=1,cpus=1,memdev=ram-node1
>
>
> I'll try to make a patch if you want to test.
>
>
> ----- Mail original ----- 
> De: "aderumier" <aderumier at odiso.com>
> À: "Cesar Peschiera" <brain at click.com.py>
> Cc: "pve-devel" <pve-devel at pve.proxmox.com>
> Envoyé: Jeudi 18 Décembre 2014 06:15:44
> Objet: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>
>>>Moreover, i guess that i have problems of Hugapages.
>
> I have found interesting blog:
>
> http://developerblog.redhat.com/2014/03/10/examining-huge-pages-or-transparent-huge-pages-performance/
>
> It's explain how to see if hugepage impact performance or not.
>
>
>
> ----- Mail original ----- 
> De: "Cesar Peschiera" <brain at click.com.py>
> À: "aderumier" <aderumier at odiso.com>, "pve-devel"
> <pve-devel at pve.proxmox.com>
> Envoyé: Jeudi 18 Décembre 2014 03:44:33
> Objet: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>
> Hi Alexandre
>
> I have installed your patches and with some test of MS-SQL-Server, i see a
> better behavior in terms of speed (soon i will give the comparisons).
>
> Moreover, i guess that i have problems of Hugapages.
> Please see this link, and answer me if you can:
> http://forum.proxmox.com/threads/20449-Win2008R2-exaggeratedly-slow-with-256GB-RAM-and-strange-behaviours-in-PVE?p=104996#post104996
>
>
> ----- Original Message ----- 
> From: "Alexandre DERUMIER" <aderumier at odiso.com>
> To: "Cesar Peschiera" <brain at click.com.py>
> Cc: <pve-devel at pve.proxmox.com>
> Sent: Tuesday, December 02, 2014 9:50 AM
> Subject: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>
>
> Hi,
> can you test this:
>
> http://odisoweb1.odiso.net/pve-qemu-kvm_2.2-2_amd64.deb
> http://odisoweb1.odiso.net/qemu-server_3.3-5_amd64.deb
>
>
> then edit your vm config file:
>
>
> sockets: 2
> cores: 4
> memory: 262144
> numa0: memory=131072,policy=bind
> numa1: memory=131072,policy=bind
>
>
> (you need 1 numa by socket, total numa memory must be equal to vm memory).
>
> you can change cores number if you want.
>
>
> and start the vm ?
>
>
> ----- Mail original ----- 
>
> De: "Alexandre DERUMIER" <aderumier at odiso.com>
> À: "Cesar Peschiera" <brain at click.com.py>
> Cc: pve-devel at pve.proxmox.com
> Envoyé: Mardi 2 Décembre 2014 12:40:29
> Objet: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>
> Hi,
>
> some news.
>
> It's seem that current proxmox qemu build don't have numa support enable.
>
> So, previous command line don't work.
>
>
> I'll send a patch for pve-qemu-kvm and also to add numa options to vm
> config
> file.
>
>
>
> ----- Mail original ----- 
>
> De: "Alexandre DERUMIER" <aderumier at odiso.com>
> À: "Cesar Peschiera" <brain at click.com.py>
> Cc: pve-devel at pve.proxmox.com
> Envoyé: Mardi 2 Décembre 2014 07:05:47
> Objet: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>
>>>at i would like to ask you if you can give me your suggestions in
>>>practical terms, besides the brief theoretical explanation, this is due
>>>to
>>>that i am not a developer and i don't understand as apply it in my PVE.
>
> About the command line, each vm is a kvm process.
>
> So start your vm with current config, do a "ps -aux" , copy the big
> "kvm -id
> ... " command line for your vm,
>
> stop the vm.
>
> then,
>
> add my specials lines about numa,
>
> and paste the command line to start the vm !
>
>
> (kvm is so simple ;)
>
>
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 