[PVE-User] proxmox hyperconverged pg calculations in pve 7.2, ceph pacific

[Eneko Lacunza]

Hi Rainer,

I haven't used erasure coded pools so I can't comment, but you may have 
better luck asking in ceph-user mailing list, as the question is quite 
generic and not Proxmox related:

https://lists.ceph.io/postorius/lists/ceph-users.ceph.io/

Cheers

El 1/12/22 a las 8:37, Rainer Krienke escribió:
> Hello,
>
> I run a a hyperconverged pve cluster (V7.2) with 11 nodes. Each node 
> has 8 4TB disks. pve and ceph are installed an running.
>
> Next I wanted to create some ceph-pools with each 512 pgs. Since I 
> want to use erasure coding (5+3) when creating a pool one rbd pool for 
> metadata and the data pool are created. I used pveceph pool this command:
>
> pveceph pool create px-e --erasure-coding k=5,m=3 --pg_autoscale_mode 
> off --pg_num 512 --pg_num_min 128
>
> I was able to create two pools in this way but the third pveceph call 
> threw this error:
>
> "got unexpected control message: TASK ERROR: error with 'osd pool 
> create': mon_command failed -  pg_num 512 size 8 would mean 22148 
> total pgs, which exceeds max 22000 (mon_max_pg_per_osd 250 * 
> num_in_osds 88)"
>
> What I do not understand now are the calculations behind the scenes 
> for the calculated total pg number of 22148. But how is this total 
> number "22148"  calculated?
>
> I already reduced the number of pgs for the metadata pool of each 
> ec-pool and so I was able to create 4 pools in this way. But just for 
> fun I now tried to create ec-pool number 5 and I see the message from 
> above.
>
> Here are the pools created by now (scraped from ceph osd pool 
> autoscale-status):
> Pool:                Size:   Bias:  PG_NUM:
> rbd                  4599    1.0      32
> px-a-data          528.2G    1.0     512
> px-a-metadata      838.1k    1.0     128
> px-b-data              0     1.0     512
> px-b-metadata         19     1.0     128
> px-c-data              0     1.0     512
> px-c-metadata         19     1.0     128
> px-d-data              0     1.0     512
> px-d-metadata          0     1.0     128
>
> So the total number of pgs for all pools is currently 2592 which is 
> far from 22148 pgs?
>
> Any ideas?
> Thanks Rainer

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 |https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/