[pve-devel] Error in PVE with win2008r2 and 256GB RAM

Thu Dec 18 09:26:20 CET 2014

Hi Alexandre

Until you correct the problem, and for avoid the loss of cluster 
communication in these two nodes, can be a temporal solution that I add to 
rc.local file this line:
/usr/bin/pvecm expected 1

As i don't have a fence device by Hardware, and i know that for apply HA 
with manual fence, first i should disconnect the electrical energy in the 
node that has the bad behaviour, maybe this solution can be util for get the 
bread and the cake.

----- Original Message ----- 
From: "Cesar Peschiera" <brain at click.com.py>
To: "Alexandre DERUMIER" <aderumier at odiso.com>
Cc: "pve-devel" <pve-devel at pve.proxmox.com>
Sent: Thursday, December 18, 2014 4:45 AM
Subject: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM

> >Do you mix 2.6.32 and 3.10 kernel in your cluster ?
> Yes, but the problem is only on the nodes that has NICs of 10 Gb/s, the 
> other PVE nodes that has NICs of 1 Gb/s and the 3.10 kernel never lost the 
> cluster communication.
>
>>What are your hardware switches ?
> Dell N2024 (managed, and for the moment with "igmp snooping" disabled)
>
> ----- Original Message ----- 
> From: "Alexandre DERUMIER" <aderumier at odiso.com>
> To: "Cesar Peschiera" <brain at click.com.py>
> Cc: "pve-devel" <pve-devel at pve.proxmox.com>
> Sent: Thursday, December 18, 2014 4:11 AM
> Subject: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>
>
>>>Many thanks for your answer, but i am not sure if it is a good idea due 
>>>to
>>>that i don't understand the advantage since that i can disable the huge
>>>pages of this mode:
>
> They are 2 modes for hugepages,
>
> the transparent hugepage mode, managed by the kernel
>
> But also, old way, manual hugepages (aka hugetlbfs, mounted in 
> /dev/hugepage..)
>
> from:
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-memory-transhuge.html
>
> " However, transparent hugepage mode is not recommended for database 
> workloads."
>
>
> So, hugepages are always usefull for big memory vms, because it's reduce 
> cpu usage on memory access.
> But transparent hugepage sometimes don't work good with some workloads 
> like database.
>
> I think than manually defined hugepages can give good an extra boost vs 
> disable TBL.
>
>
>
>>>Moreover, i have a serious problem with the PVE cluster communication in 
>>>two
>>>of eight PVE nodes, and with a VM, if you can help me, i will be 
>>>extremely
>>>grateful, please see this link:
>>>http://forum.proxmox.com/threads/20523-Quorum-problems-with-PVE-2-3-and-3-3?p=104995#post104995
>
> Do you mix 2.6.32 and 3.10 kernel in your cluster ?
> (I have had some strange problems with mixed kernel, never find the 
> problem).
>
> Also, this could be a multicast snooping problem.
>
> What are your hardware switches ?
> Myself, I disable snooping on linux vmbr, enable snooping on physical 
> swiches + igmp querier on physical switches.
>
>
>
> ----- Mail original -----
> De: "Cesar Peschiera" <brain at click.com.py>
> À: "pve-devel" <pve-devel at pve.proxmox.com>, "aderumier" 
> <aderumier at odiso.com>
> Envoyé: Jeudi 18 Décembre 2014 07:09:11
> Objet: Fw: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>
> Hi Alexandre
>
> Many thanks for your answer, but i am not sure if it is a good idea due to
> that i don't understand the advantage since that i can disable the huge
> pages of this mode:
>
> shell> vim /etc/default/grub:
> GRUB_CMDLINE_LINUX_DEFAULT="...transparent_hugepage=never"
>
> shell> update-grub
>
> Moreover, i have a serious problem with the PVE cluster communication in 
> two
> of eight PVE nodes, and with a VM, if you can help me, i will be extremely
> grateful, please see this link:
> http://forum.proxmox.com/threads/20523-Quorum-problems-with-PVE-2-3-and-3-3?p=104995#post104995
>
> Best regards
> Cesar
>
>> ----- Original Message ----- 
>> From: "Alexandre DERUMIER" <aderumier at odiso.com>
>> To: "Cesar Peschiera" <brain at click.com.py>
>> Cc: "pve-devel" <pve-devel at pve.proxmox.com>
>> Sent: Thursday, December 18, 2014 2:24 AM
>> Subject: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>>
>>
>> Note that currently,
>>
>> hugepages are managed with transparent hugepage mecanism.
>>
>> But it's seem that we can defined manually hugepages by numa nodes
>>
>> -object
>> memory-backend-file,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,size=1024M,id=ram-node0
>> -numa node,nodeid=0,cpus=0,memdev=ram-node0
>> -object
>> memory-backend-file,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu,size=1024M,id=ram-node1
>> -numa node,nodeid=1,cpus=1,memdev=ram-node1
>>
>>
>> I'll try to make a patch if you want to test.
>>
>>
>> ----- Mail original ----- 
>> De: "aderumier" <aderumier at odiso.com>
>> À: "Cesar Peschiera" <brain at click.com.py>
>> Cc: "pve-devel" <pve-devel at pve.proxmox.com>
>> Envoyé: Jeudi 18 Décembre 2014 06:15:44
>> Objet: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>>
>>>>Moreover, i guess that i have problems of Hugapages.
>>
>> I have found interesting blog:
>>
>> http://developerblog.redhat.com/2014/03/10/examining-huge-pages-or-transparent-huge-pages-performance/
>>
>> It's explain how to see if hugepage impact performance or not.
>>
>>
>>
>> ----- Mail original ----- 
>> De: "Cesar Peschiera" <brain at click.com.py>
>> À: "aderumier" <aderumier at odiso.com>, "pve-devel"
>> <pve-devel at pve.proxmox.com>
>> Envoyé: Jeudi 18 Décembre 2014 03:44:33
>> Objet: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>>
>> Hi Alexandre
>>
>> I have installed your patches and with some test of MS-SQL-Server, i see 
>> a
>> better behavior in terms of speed (soon i will give the comparisons).
>>
>> Moreover, i guess that i have problems of Hugapages.
>> Please see this link, and answer me if you can:
>> http://forum.proxmox.com/threads/20449-Win2008R2-exaggeratedly-slow-with-256GB-RAM-and-strange-behaviours-in-PVE?p=104996#post104996
>>
>>
>> ----- Original Message ----- 
>> From: "Alexandre DERUMIER" <aderumier at odiso.com>
>> To: "Cesar Peschiera" <brain at click.com.py>
>> Cc: <pve-devel at pve.proxmox.com>
>> Sent: Tuesday, December 02, 2014 9:50 AM
>> Subject: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>>
>>
>> Hi,
>> can you test this:
>>
>> http://odisoweb1.odiso.net/pve-qemu-kvm_2.2-2_amd64.deb
>> http://odisoweb1.odiso.net/qemu-server_3.3-5_amd64.deb
>>
>>
>> then edit your vm config file:
>>
>>
>> sockets: 2
>> cores: 4
>> memory: 262144
>> numa0: memory=131072,policy=bind
>> numa1: memory=131072,policy=bind
>>
>>
>> (you need 1 numa by socket, total numa memory must be equal to vm 
>> memory).
>>
>> you can change cores number if you want.
>>
>>
>> and start the vm ?
>>
>>
>> ----- Mail original ----- 
>>
>> De: "Alexandre DERUMIER" <aderumier at odiso.com>
>> À: "Cesar Peschiera" <brain at click.com.py>
>> Cc: pve-devel at pve.proxmox.com
>> Envoyé: Mardi 2 Décembre 2014 12:40:29
>> Objet: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>>
>> Hi,
>>
>> some news.
>>
>> It's seem that current proxmox qemu build don't have numa support enable.
>>
>> So, previous command line don't work.
>>
>>
>> I'll send a patch for pve-qemu-kvm and also to add numa options to vm
>> config
>> file.
>>
>>
>>
>> ----- Mail original ----- 
>>
>> De: "Alexandre DERUMIER" <aderumier at odiso.com>
>> À: "Cesar Peschiera" <brain at click.com.py>
>> Cc: pve-devel at pve.proxmox.com
>> Envoyé: Mardi 2 Décembre 2014 07:05:47
>> Objet: Re: [pve-devel] Error in PVE with win2008r2 and 256GB RAM
>>
>>>>at i would like to ask you if you can give me your suggestions in
>>>>practical terms, besides the brief theoretical explanation, this is due
>>>>to
>>>>that i am not a developer and i don't understand as apply it in my PVE.
>>
>> About the command line, each vm is a kvm process.
>>
>> So start your vm with current config, do a "ps -aux" , copy the big
>> "kvm -id
>> ... " command line for your vm,
>>
>> stop the vm.
>>
>> then,
>>
>> add my specials lines about numa,
>>
>> and paste the command line to start the vm !
>>
>>
>> (kvm is so simple ;)
>>
>>
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>>
> 