[PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent

[Daniel Oliver]

The built-in Ceph balancer only balances based on PG numbers, which can vary wildly in size for several reasons.

I ended up disabling the built-in balancer and switching to https://github.com/TheJJ/ceph-balancer, which we now run daily with the following parameters:
placementoptimizer.py balance --ignore-ideal-pgcounts=all --osdused=delta --osdfrom fullest

This keeps things nicely balanced from a fullness perspective, with the most important bit being ignore-ideal-pgcounts, as it allows balancing decisions outside of what the built-in balancer would decide.

From: pve-user <pve-user-bounces at lists.proxmox.com> on behalf of Frank Thommen <f.thommen at dkfz-heidelberg.de>
Date: Wednesday, 11 September 2024 at 12:00
To: pve-user at lists.proxmox.com <pve-user at lists.proxmox.com>
Subject: Re: [PVE-User] [Extern] - Re: "nearfull" status in PVE Dashboard not consistent
The OSDs are of different size, because we have 4 TB and 2 TB disks in
the systems.

We might give the reweight a try.



On 10.09.24 20:31, David der Nederlanden | ITTY via pve-user wrote:
> Hi Frank,
>
> The images didn't work 🙂
>
> Pool and osd nearfull are closely related, when OSD's get full your pool
>   also gets nearfull as Ceph needs to be able to follow the crush rules,
> which it can't if one of the OSD's gets full, hence it warns when it
> gets nearfull.
>
> I see that you're mixing OSD sizes, deleting and recreating the OSD's
> one by one caused this, as the OSD's got new weights you should be OK
> when you reweight them.
> You can do this by hand or using reweight-by-utilization, what you
> prefer.
>
> Not quite sure about the pool sizes, but an RBD pool with a 2/3 min/max
> rule should never be above 80%, as this gives you a nearfull pool when
> it starts backfilling when you lose one node, or even a full pool worst
> case, rendering the pool read only.
>
> Kind regards,
> David

_______________________________________________
pve-user mailing list
pve-user at lists.proxmox.com
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

-- 
Unless expressly stated otherwise, the contents of this e-mail represent 
only the views of the sender as expressed only to the intended recipient. 
It does not commit ripjar limited to any course of action or legal 
responsibility. This e-mail and the information it contains may be 
privileged and/or confidential. It is for the intended addressee(s) only. 
The unauthorised use, disclosure or copying of this e-mail, or any 
information it contains, is prohibited and could, in certain circumstances, 
be a criminal offence. If you are not an intended recipient, please notify 
admin at ripjar.com <mailto:admin at ripjar.com> immediately.


RIPJAR LIMITED - 
Registered in England & Wales - Company No: 8217339 - Registered Address: 
Suite 404, Eagle Tower, Montpellier Drive, Cheltenham, Gloucestershire GL50 
1TA.