[PVE-User] Proxmox Ceph high memory usage

Ronny Aasen ronny+pve-user at aasen.cx
Wed Jan 16 13:42:13 CET 2019


the memory consumption of the machine is an aggregate of multiple 
consumers.

identify what is using memory try commands like
top -o VIRT and top -o RES


to reduce VM memory usage, you can move or stop virtual machines, 
reconfigure them with less memory, or try to use KSM if you nave many 
identical vm's  https://en.wikipedia.org/wiki/Kernel_same-page_merging

to reduce ceoh osd memory consumption you can tweak the bluestore memory 
cache 
http://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/#automatic-cache-sizing 
; with only 16GB i think you need to try to reduce cache (and hence 
performance) a bit here.

also ceph memory usage increase quite a bit when recovering and 
backfilling, so when planning resource requirements, plan for the 
recovery situation, and have some free overhead.

kind regards
Ronny Aasen

On 16.01.2019 13:28, Gilberto Nunes wrote:
> pve-ceph01:~# ceph status
>    cluster:
>      id:     e67534b4-0a66-48db-ad6f-aa0868e962d8
>      health: HEALTH_WARN
>              nobackfill,norebalance,norecover,nodeep-scrub flag(s) set
>              394106/2589186 objects misplaced (15.221%)
>              Degraded data redundancy: 124011/2589186 objects degraded
> (4.790%), 158 pgs degraded, 76 pgs undersized
> 
>    services:
>      mon: 5 daemons, quorum
> pve-ceph01,pve-ceph02,pve-ceph03,pve-ceph04,pve-ceph05
>      mgr: pve-ceph05(active), standbys: pve-ceph01, pve-ceph03, pve-ceph04,
> pve-ceph02
>      osd: 21 osds: 21 up, 21 in; 230 remapped pgs
>           flags nobackfill,norebalance,norecover,nodeep-scrub
> 
>    data:
>      pools:   1 pools, 512 pgs
>      objects: 863.06k objects, 3.17TiB
>      usage:   9.73TiB used, 53.0TiB / 62.8TiB avail
>      pgs:     124011/2589186 objects degraded (4.790%)
>               394106/2589186 objects misplaced (15.221%)
>               180 active+clean
>               76  active+remapped+backfill_wait
>               70  active+recovery_wait
>               63  active+undersized+degraded+remapped+backfill_wait
>               49  active+recovery_wait+degraded+remapped
>               32  active+recovery_wait+degraded
>               28  active+recovery_wait+remapped
>               12  active+recovery_wait+undersized+degraded+remapped
>               1   active+recovering+degraded+remapped
>               1   active+undersized+degraded+remapped+backfilling
> 
>    io:
>      client:   694KiB/s rd, 172KiB/s wr, 118op/s rd, 38op/s wr
>      recovery: 257KiB/s, 0objects/s
> ---
> Gilberto Nunes Ferreira
> 
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
> 
> Skype: gilberto.nunes36
> 
> 
> 
> 
> 
> Em qua, 16 de jan de 2019 às 10:23, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> escreveu:
> 
>> Hi...
>> I am using BlueStore.
>> pve-manager/5.3-7/e8ed1e22 (running kernel: 4.15.18-9-pve)
>>   ceph                                 12.2.10-pve1
>>
>> Thanks
>> ---
>> Gilberto Nunes Ferreira
>>
>> (47) 3025-5907
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>> Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>> Em qua, 16 de jan de 2019 às 10:17, Eneko Lacunza <elacunza at binovo.es>
>> escreveu:
>>
>>> Hi Gilberto,
>>>
>>> Are you using Blustore? What version of Ceph?
>>>
>>> El 16/1/19 a las 13:11, Gilberto Nunes escribió:
>>>> Hi there
>>>>
>>>> Anybody else experiment hight memory usage in Proxmox CEPH Storage
>>> Server?
>>>> I have a 6 node PVE CEPH and after upgrade, I have noticed this high
>>> memory
>>>> usage...
>>>> All server has 16GB of ram. I know this is not recomended, but that
>>> what I
>>>> have at the moment.
>>>> In fact, just 3 servers ran with about 90% of memory usage.
>>>> All server is IBM x3200 m2 with SATA disks...
>>>> Here's ceph osd tree
>>>> ceph osd tree
>>>> ID  CLASS WEIGHT   TYPE NAME           STATUS REWEIGHT PRI-AFF
>>>>    -1       38.50000 root default
>>>>    -3        4.00000     host pve-ceph01
>>>>     0   hdd  1.00000         osd.0           up  1.00000       0
>>>>     1   hdd  1.00000         osd.1           up  1.00000       0
>>>>     2   hdd  1.00000         osd.2           up  1.00000       0
>>>>     3   hdd  1.00000         osd.3           up  1.00000       0
>>>>    -5        8.00000     host pve-ceph02
>>>>     4   hdd  2.00000         osd.4           up  1.00000 1.00000
>>>>     5   hdd  2.00000         osd.5           up  1.00000 1.00000
>>>>     6   hdd  2.00000         osd.6           up  1.00000 1.00000
>>>>     7   hdd  2.00000         osd.7           up  1.00000 1.00000
>>>>    -7        9.00000     host pve-ceph03
>>>>     8   hdd  3.00000         osd.8           up  1.00000 1.00000
>>>>     9   hdd  3.00000         osd.9           up  1.00000 1.00000
>>>>    10   hdd  3.00000         osd.10          up  1.00000 1.00000
>>>>    -9       12.00000     host pve-ceph04
>>>>    11   hdd  3.00000         osd.11          up  1.00000 1.00000
>>>>    12   hdd  3.00000         osd.12          up  1.00000 1.00000
>>>>    13   hdd  3.00000         osd.13          up  1.00000 1.00000
>>>>    14   hdd  3.00000         osd.14          up  1.00000 1.00000
>>>> -11        1.00000     host pve-ceph05
>>>>    15   hdd  0.50000         osd.15          up  1.00000       0
>>>>    16   hdd  0.50000         osd.16          up  1.00000       0
>>>> -13        4.50000     host pve-ceph06
>>>>    17   hdd  1.00000         osd.17          up  1.00000       0
>>>>    18   hdd  1.00000         osd.18          up  1.00000       0
>>>>    20   hdd  1.00000         osd.20          up  1.00000       0
>>>>    21   hdd  1.50000         osd.21          up  1.00000 1.00000
>>>>
>>>> ---
>>>> Gilberto Nunes Ferreira
>>>>
>>>> (47) 3025-5907
>>>> (47) 99676-7530 - Whatsapp / Telegram
>>>>
>>>> Skype: gilberto.nunes36
>>>> _______________________________________________
>>>> pve-user mailing list
>>>> pve-user at pve.proxmox.com
>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>
>>>
>>> --
>>> Zuzendari Teknikoa / Director Técnico
>>> Binovo IT Human Project, S.L.
>>> Telf. 943569206
>>> Astigarraga bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa)
>>> www.binovo.es
>>>
>>> _______________________________________________
>>> pve-user mailing list
>>> pve-user at pve.proxmox.com
>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>
>>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 



More information about the pve-user mailing list