[PVE-User] pveceph : Unable to add any OSD
Phil Schwarz
infolist at schwarz-fr.net
Mon Sep 18 21:12:04 CEST 2017
Thanks for your help,
Le 18/09/2017 à 12:37, Alwin Antreich a écrit :
> On Sun, Sep 17, 2017 at 11:18:51AM +0200, Phil Schwarz wrote:
>> Hi,
>> going on on the same problem (links [1] & [2] )
>>
>> [1] : https://pve.proxmox.com/pipermail/pve-user/2017-July/168578.html
>> [2] : https://pve.proxmox.com/pipermail/pve-user/2017-September/168775.html
>>
>> -Added a brand new node, updated to last ceph version (the proxmox team
>> recompiled one)
> Can you please post a 'ceph versions' and a 'ceph osd tree' to get some
> overview on your setup?
root at arya:~# ceph versionceph version 12.2.0
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc)
root at arya:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 10.06328 root default
-3 0 host daenerys
-5 1.81360 host jaime
5 hdd 1.81360 osd.5 up 1.00000 1.00000
-2 6.59999 host jon
1 hdd 4.20000 osd.1 up 1.00000 1.00000
3 hdd 2.39999 osd.3 up 1.00000 1.00000
-4 1.64969 host tyrion
2 hdd 0.44969 osd.2 up 1.00000 1.00000
4 hdd 1.20000 osd.4 up 1.00000 1.00000
0 0 osd.0 down 0 1.00000
>
>>
>> -plugged a new disk
>>
>> -used GUI (same result with pveceph createosd from cli) to create a new osd
>> (with bluestore feature).
> PVE GUI and CLI use the same API for managing ceph
>
>>
>> 1. The OSD doesn't appear on gui
>> 2. The OSD is seen as down and out of any node
>> 3. the /var/log/ceph/ceph-osd.admin.log logfile seems to figure a
>> mismatch between filestore and bluestore:
> Do you see any errors in the mon logs or ceph.log itself?
(Jaime is a mon& mgr)
root at jaime:~# tail -f /var/log/ceph/ceph-mon.1.log
2017-09-18 21:05:00.084847 7f8a1b4a8700 1 mon.1 at 0(leader).log v2152264
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 0 entries (version 2152264)
2017-09-18 21:05:09.963784 7f8a1868c700 0
mon.1 at 0(leader).data_health(2028) update_stats avail 90% total 58203 MB,
used 2743 MB, avail 52474 MB
2017-09-18 21:05:29.878648 7f8a15e87700 0 mon.1 at 0(leader) e4
handle_command mon_command({"prefix": "osd new", "uuid":
"89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"} v 0) v1
2017-09-18 21:05:29.878705 7f8a15e87700 0 log_channel(audit) log [INF]
: from='client.6392525 -' entity='client.bootstrap-osd' cmd=[{"prefix":
"osd new", "uuid": "89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"}]: dispatch
2017-09-18 21:05:29.927377 7f8a1b4a8700 1 mon.1 at 0(leader).osd e1141
e1141: 6 total, 5 up, 5 in
2017-09-18 21:05:29.932253 7f8a1b4a8700 0 log_channel(audit) log [INF]
: from='client.6392525 -' entity='client.bootstrap-osd' cmd='[{"prefix":
"osd new", "uuid": "89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"}]': finished
2017-09-18 21:05:29.932388 7f8a1b4a8700 0 log_channel(cluster) log
[DBG] : osdmap e1141: 6 total, 5 up, 5 in
2017-09-18 21:05:29.932983 7f8a15e87700 0 mon.1 at 0(leader) e4
handle_command mon_command({"prefix": "osd metadata", "id": 0} v 0) v1
2017-09-18 21:05:29.933040 7f8a15e87700 0 log_channel(audit) log [DBG]
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon'
cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch
2017-09-18 21:05:29.933337 7f8a15e87700 0 mon.1 at 0(leader) e4
handle_command mon_command({"prefix": "osd metadata", "id": 2} v 0) v1
2017-09-18 21:05:29.933383 7f8a15e87700 0 log_channel(audit) log [DBG]
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon'
cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch
2017-09-18 21:05:29.933674 7f8a15e87700 0 mon.1 at 0(leader) e4
handle_command mon_command({"prefix": "osd metadata", "id": 3} v 0) v1
2017-09-18 21:05:29.933692 7f8a15e87700 0 log_channel(audit) log [DBG]
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon'
cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch
2017-09-18 21:05:29.933880 7f8a15e87700 0 mon.1 at 0(leader) e4
handle_command mon_command({"prefix": "osd metadata", "id": 4} v 0) v1
2017-09-18 21:05:29.933897 7f8a15e87700 0 log_channel(audit) log [DBG]
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon'
cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch
2017-09-18 21:05:29.934062 7f8a15e87700 0 mon.1 at 0(leader) e4
handle_command mon_command({"prefix": "osd metadata", "id": 5} v 0) v1
2017-09-18 21:05:29.934089 7f8a15e87700 0 log_channel(audit) log [DBG]
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon'
cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch
2017-09-18 21:05:30.113007 7f8a1b4a8700 1 mon.1 at 0(leader).log v2152265
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 3 entries (version 2152265)
2017-09-18 21:05:31.154227 7f8a1b4a8700 1 mon.1 at 0(leader).log v2152266
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 0 entries (version 2152266)
2017-09-18 21:05:32.289428 7f8a1b4a8700 1 mon.1 at 0(leader).log v2152267
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 0 entries (version 2152267)
2017-09-18 21:05:36.782573 7f8a1b4a8700 1 mon.1 at 0(leader).log v2152268
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 0 entries (version 2152268)
2017-09-18 21:06:09.964314 7f8a1868c700 0
mon.1 at 0(leader).data_health(2028) update_stats avail 90% total 58203 MB,
used 2744 MB, avail 52473 MB
2017-09-18 21:06:20.040930 7f8a1b4a8700 1 mon.1 at 0(leader).log v2152269
check_sub sending message to client.5804116 10.250.0.23:0/4045099631
with 0 entries (version 2152269)
And ceph.log
root at jaime:~# tail -f /var/log/ceph/ceph.log
2017-09-18 12:00:00.000160 mon.1 mon.0 10.250.0.21:6789/0 38100 :
cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883
objects misplaced (3.653%); Degraded data redundancy: 73487/1415883
objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs
undersized; Degraded data redundancy (low space): 29 pgs
backfill_toofull; application not enabled on 2 pool(s)
2017-09-18 13:00:00.000160 mon.1 mon.0 10.250.0.21:6789/0 38101 :
cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883
objects misplaced (3.653%); Degraded data redundancy: 73487/1415883
objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs
undersized; Degraded data redundancy (low space): 29 pgs
backfill_toofull; application not enabled on 2 pool(s)
2017-09-18 14:00:00.000133 mon.1 mon.0 10.250.0.21:6789/0 38102 :
cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883
objects misplaced (3.653%); Degraded data redundancy: 73487/1415883
objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs
undersized; Degraded data redundancy (low space): 29 pgs
backfill_toofull; application not enabled on 2 pool(s)
201
Yes, the cluster is not really healthy, indeed ....
Thanks
>
>>
>> 2017-09-16 19:12:00.468481 7f6469cdde00 0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5624
>> 2017-09-16 19:12:00.470154 7f6469cdde00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:00.471408 7f6469cdde00 1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:00.471727 7f6469cdde00 1 journal close /dev/sdb2
>> 2017-09-16 19:12:00.471994 7f6469cdde00 0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:05.042622 7f000b944e00 0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5702
>> 2017-09-16 19:12:05.066343 7f000b944e00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:05.066549 7f000b944e00 1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:05.066717 7f000b944e00 1 journal close /dev/sdb2
>> 2017-09-16 19:12:05.066843 7f000b944e00 0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:08.198548 7f5740748e00 0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5767
>> 2017-09-16 19:12:08.223674 7f5740748e00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:08.223831 7f5740748e00 1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:08.224213 7f5740748e00 1 journal close /dev/sdb2
>> 2017-09-16 19:12:08.224342 7f5740748e00 0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:09.149622 7f7b06058e00 0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5800
>> 2017-09-16 19:12:09.173319 7f7b06058e00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:09.173402 7f7b06058e00 1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:09.173485 7f7b06058e00 1 journal close /dev/sdb2
>> 2017-09-16 19:12:09.173511 7f7b06058e00 0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:10.197944 7f7561d50e00 0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5828
>> 2017-09-16 19:12:10.222504 7f7561d50e00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:10.222723 7f7561d50e00 1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:10.222753 7f7561d50e00 1 journal close /dev/sdb2
>> 2017-09-16 19:12:10.222785 7f7561d50e00 0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:14.370797 7f9fecb7fe00 0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5964
>> 2017-09-16 19:12:14.371221 7f9fecb7fe00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:14.371350 7f9fecb7fe00 1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:14.371616 7f9fecb7fe00 1 journal close /dev/sdb2
>> 2017-09-16 19:12:14.371745 7f9fecb7fe00 0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:21.171036 7f5d7579be00 0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 6130
>> 2017-09-16 19:12:21.209441 7f5d7579be00 0 probe_block_device_fsid /dev/sdb2
>> is bluestore, 92a4a9eb-0a6a-405d-be83-11e4af42fa30
>>
>>
>>
>> Any hint ?
>>
>> Thanks by advance
>> Best regards
>>
> --
> Cheers,
> Alwin
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
More information about the pve-user
mailing list