[PVE-User] ceph rebalance/ raw vs pool usage

[Alwin Antreich]

Hello Mark,

On Tue, May 07, 2019 at 11:26:17PM +0100, Mark Adams wrote:
> Hi All,
> 
> I would appreciate a little pointer or clarification on this.
> 
> My "ceph" vm pool is showing 84.80% used. But the %RAW usage is only 71.88%
> used. is this normal? there is nothing else on this ceph cluster apart from
> this one pool.
It is normal that the pools used-% is higher, then the raw-% usage of
the cluster, because for one the bluestore OSDs (DB+WAL) occupy by
default ~1.5GiB. And depending on the OSDs the pool resides on (class
based rules), the amount may even further diverge.

The general %-usage numbers of your cluster may not allow a recovery if
a node or multiple OSDs will fail. Consider to put in more disks or
reduce the data usage.

> 
> Also, I notice some of my OSD's are out of balance. I've done some
> reweighting using "ceph osd reweight-by-utilization" which has helped a
> bit, but I think it needs to be tweaked some more (still some OSDs are over
> 82% utilised, while some are around 65-70%).
This is something that should be done with keeping an eye on the general
cluster performance and how it fills the OSDs further. Could please post
a 'ceph osd df tree'? It seems to me the cluster is uneven balanced in
disk size and/or count.

> 
> Is it safe enough to keep tweaking this? (I believe I should run ceph osd
> reweight-by-utilization 101 0.05 15) Is there any gotchas I need to be
> aware of when doing this apart from the obvious load of reshuffling the
> data around? The cluster has 30 OSDs and it looks like it will reweight 13.
Your cluster may get more and more unbalanced. Eg. making a OSD
replacement a bigger challenge.

--
Cheers,
Alwin