[PVE-User] ceph rebalance/ raw vs pool usage

Wed May 8 12:53:32 CEST 2019

On Wed, 8 May 2019 at 11:34, Alwin Antreich <a.antreich at proxmox.com> wrote:

> On Wed, May 08, 2019 at 09:34:44AM +0100, Mark Adams wrote:
> > Thanks for getting back to me Alwin. See my response below.
> >
> >
> > I have the same size and count in each node, but I have had a disk
> failure
> > (has been replaced) and also had issues with osds dropping when that
> memory
> > allocation bug was around just before last christmas (Think it was when
> > they made some bluestore updates, then the next release they increased
> the
> > default memory allocation to rectify the issue) so that could have messed
> > up the balance.
> Ok, that can impact the distribution of PGs. Could you please post the
> crush tunables too? Maybe there could be something to tweak, besides the
> reweight-by-utilization.
>

  "choose_local_tries": 0,
    "choose_local_fallback_tries": 0,
    "choose_total_tries": 50,
    "chooseleaf_descend_once": 1,
    "chooseleaf_vary_r": 1,
    "chooseleaf_stable": 1,
    "straw_calc_version": 1,
    "allowed_bucket_algs": 54,
    "profile": "jewel",
    "optimal_tunables": 1,
    "legacy_tunables": 0,
    "minimum_required_version": "jewel",
    "require_feature_tunables": 1,
    "require_feature_tunables2": 1,
    "has_v2_rules": 0,
    "require_feature_tunables3": 1,
    "has_v3_rules": 0,
    "has_v4_buckets": 1,
    "require_feature_tunables5": 1,
    "has_v5_rules": 0

> >
> > ceph osd df tree:
> >
> > ID CLASS WEIGHT    REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS TYPE
> > NAME
> > -1       209.58572        -  210TiB  151TiB 58.8TiB 71.92 1.00   - root
> > default
> > -3        69.86191        - 69.9TiB 50.2TiB 19.6TiB 71.91 1.00   -
>  host
> > prod-pve1
> >  0   ssd   6.98619  0.90002 6.99TiB 5.70TiB 1.29TiB 81.54 1.13 116
> >  osd.0
> >  1   ssd   6.98619  1.00000 6.99TiB 5.49TiB 1.49TiB 78.65 1.09 112
> >  osd.1
> >  2   ssd   6.98619  1.00000 6.99TiB 4.95TiB 2.03TiB 70.88 0.99 101
> >  osd.2
> >  4   ssd   6.98619  1.00000 6.99TiB 4.90TiB 2.09TiB 70.11 0.97 100
> >  osd.4
> >  5   ssd   6.98619  1.00000 6.99TiB 4.52TiB 2.47TiB 64.67 0.90  92
> >  osd.5
> >  6   ssd   6.98619  1.00000 6.99TiB 5.34TiB 1.64TiB 76.50 1.06 109
> >  osd.6
> >  7   ssd   6.98619  1.00000 6.99TiB 4.56TiB 2.42TiB 65.31 0.91  93
> >  osd.7
> >  8   ssd   6.98619  1.00000 6.99TiB 4.91TiB 2.08TiB 70.21 0.98 100
> >  osd.8
> >  9   ssd   6.98619  1.00000 6.99TiB 4.66TiB 2.32TiB 66.76 0.93  95
> >  osd.9
> > 30   ssd   6.98619  1.00000 6.99TiB 5.20TiB 1.78TiB 74.49 1.04 106
> >  osd.30
> > -5        69.86191        - 69.9TiB 50.3TiB 19.6TiB 71.93 1.00   -
>  host
> > prod-pve2
> > 10   ssd   6.98619  1.00000 6.99TiB 4.47TiB 2.52TiB 63.92 0.89  91
> >  osd.10
> > 11   ssd   6.98619  1.00000 6.99TiB 4.86TiB 2.13TiB 69.53 0.97  99
> >  osd.11
> > 12   ssd   6.98619  1.00000 6.99TiB 4.46TiB 2.52TiB 63.91 0.89  91
> >  osd.12
> > 13   ssd   6.98619  1.00000 6.99TiB 4.71TiB 2.28TiB 67.43 0.94  96
> >  osd.13
> > 14   ssd   6.98619  1.00000 6.99TiB 5.50TiB 1.49TiB 78.68 1.09 112
> >  osd.14
> > 15   ssd   6.98619  1.00000 6.99TiB 5.20TiB 1.79TiB 74.38 1.03 106
> >  osd.15
> > 16   ssd   6.98619  1.00000 6.99TiB 4.66TiB 2.32TiB 66.74 0.93  95
> >  osd.16
> > 17   ssd   6.98619  1.00000 6.99TiB 5.51TiB 1.48TiB 78.84 1.10 112
> >  osd.17
> > 18   ssd   6.98619  1.00000 6.99TiB 5.40TiB 1.59TiB 77.24 1.07 110
> >  osd.18
> > 19   ssd   6.98619  1.00000 6.99TiB 5.50TiB 1.49TiB 78.66 1.09 112
> >  osd.19
> > -7        69.86191        - 69.9TiB 50.2TiB 19.6TiB 71.93 1.00   -
>  host
> > prod-pve3
> > 20   ssd   6.98619  1.00000 6.99TiB 4.22TiB 2.77TiB 60.40 0.84  86
> >  osd.20
> > 21   ssd   6.98619  1.00000 6.99TiB 4.43TiB 2.56TiB 63.35 0.88  90
> >  osd.21
> > 22   ssd   6.98619  0.95001 6.99TiB 5.69TiB 1.30TiB 81.45 1.13 116
> >  osd.22
> > 23   ssd   6.98619  1.00000 6.99TiB 4.67TiB 2.32TiB 66.79 0.93  95
> >  osd.23
> > 24   ssd   6.98619  0.95001 6.99TiB 5.74TiB 1.24TiB 82.20 1.14 117
> >  osd.24
> > 25   ssd   6.98619  1.00000 6.99TiB 4.51TiB 2.47TiB 64.59 0.90  92
> >  osd.25
> > 26   ssd   6.98619  1.00000 6.99TiB 4.90TiB 2.09TiB 70.15 0.98 100
> >  osd.26
> > 27   ssd   6.98619  1.00000 6.99TiB 5.39TiB 1.59TiB 77.21 1.07 110
> >  osd.27
> > 28   ssd   6.98619  1.00000 6.99TiB 5.69TiB 1.29TiB 81.47 1.13 116
> >  osd.28
> > 29   ssd   6.98619  1.00000 6.99TiB 5.00TiB 1.98TiB 71.63 1.00 102
> >  osd.29
> >                       TOTAL  210TiB  151TiB 58.8TiB 71.92
> >
> > MIN/MAX VAR: 0.84/1.14  STDDEV: 6.44
> How many placement groups do(es) your pool(s) have?
>
>
1024

Cheers!

> >
> >
> >
> > >
> > > >
> > > > Is it safe enough to keep tweaking this? (I believe I should run
> ceph osd
> > > > reweight-by-utilization 101 0.05 15) Is there any gotchas I need to
> be
> > > > aware of when doing this apart from the obvious load of reshuffling
> the
> > > > data around? The cluster has 30 OSDs and it looks like it will
> reweight
> > > 13.
> > > Your cluster may get more and more unbalanced. Eg. making a OSD
> > > replacement a bigger challenge.
> > >
> > >
> > It can make the balance worse? I thought the whole point was to get it
> back
> > in balance! :)
> Yes, but just meant, be carefull. ;) I have re-read the section in
> ceph's docs and the reweights are relative to eachother. So, it should
> not do much harm, but I faintly recall that I had issues with PG
> distribution afterwards. My old memory. ^^
>
> --
> Cheers,
> Alwin
>