[PVE-User] pveceph : Unable to add any OSD

Phil Schwarz infolist at schwarz-fr.net
Mon Sep 18 21:12:04 CEST 2017


Thanks for your help,

Le 18/09/2017 à 12:37, Alwin Antreich a écrit :
> On Sun, Sep 17, 2017 at 11:18:51AM +0200, Phil Schwarz wrote:
>> Hi,
>> going on on the same problem (links [1] & [2] )
>>
>> [1] : https://pve.proxmox.com/pipermail/pve-user/2017-July/168578.html
>> [2] : https://pve.proxmox.com/pipermail/pve-user/2017-September/168775.html
>>
>> -Added a brand new node, updated to last ceph version (the proxmox team
>> recompiled one)
> Can you please post a 'ceph versions' and a 'ceph osd tree' to get some
> overview on your setup?

root at arya:~# ceph versionceph version 12.2.0 
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc)


root at arya:~# ceph osd tree
ID CLASS WEIGHT   TYPE NAME         STATUS REWEIGHT PRI-AFF
-1       10.06328 root default
-3              0     host daenerys
-5        1.81360     host jaime
  5   hdd  1.81360         osd.5         up  1.00000 1.00000
-2        6.59999     host jon
  1   hdd  4.20000         osd.1         up  1.00000 1.00000
  3   hdd  2.39999         osd.3         up  1.00000 1.00000
-4        1.64969     host tyrion
  2   hdd  0.44969         osd.2         up  1.00000 1.00000
  4   hdd  1.20000         osd.4         up  1.00000 1.00000
  0              0 osd.0               down        0 1.00000


> 
>>
>> -plugged a new disk
>>
>> -used GUI (same result with pveceph createosd from cli) to create a new osd
>> (with bluestore feature).
> PVE GUI and CLI use the same API for managing ceph
> 
>>
>> 1. The OSD doesn't appear on gui
>> 2. The OSD is seen as down and out of any node
>> 3. the /var/log/ceph/ceph-osd.admin.log logfile seems to figure a
>> mismatch between filestore and bluestore:

> Do you see any errors in the mon logs or ceph.log itself?
(Jaime is a mon& mgr)
root at jaime:~# tail -f /var/log/ceph/ceph-mon.1.log

2017-09-18 21:05:00.084847 7f8a1b4a8700  1 mon.1 at 0(leader).log v2152264 
check_sub sending message to client.5804116 10.250.0.23:0/4045099631 
with 0 entries (version 2152264)
2017-09-18 21:05:09.963784 7f8a1868c700  0 
mon.1 at 0(leader).data_health(2028) update_stats avail 90% total 58203 MB, 
used 2743 MB, avail 52474 MB
2017-09-18 21:05:29.878648 7f8a15e87700  0 mon.1 at 0(leader) e4 
handle_command mon_command({"prefix": "osd new", "uuid": 
"89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"} v 0) v1
2017-09-18 21:05:29.878705 7f8a15e87700  0 log_channel(audit) log [INF] 
: from='client.6392525 -' entity='client.bootstrap-osd' cmd=[{"prefix": 
"osd new", "uuid": "89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"}]: dispatch
2017-09-18 21:05:29.927377 7f8a1b4a8700  1 mon.1 at 0(leader).osd e1141 
e1141: 6 total, 5 up, 5 in
2017-09-18 21:05:29.932253 7f8a1b4a8700  0 log_channel(audit) log [INF] 
: from='client.6392525 -' entity='client.bootstrap-osd' cmd='[{"prefix": 
"osd new", "uuid": "89fce23c-8535-48fa-bfc0-ae9a2a5d7cd6"}]': finished
2017-09-18 21:05:29.932388 7f8a1b4a8700  0 log_channel(cluster) log 
[DBG] : osdmap e1141: 6 total, 5 up, 5 in
2017-09-18 21:05:29.932983 7f8a15e87700  0 mon.1 at 0(leader) e4 
handle_command mon_command({"prefix": "osd metadata", "id": 0} v 0) v1
2017-09-18 21:05:29.933040 7f8a15e87700  0 log_channel(audit) log [DBG] 
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon' 
cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch
2017-09-18 21:05:29.933337 7f8a15e87700  0 mon.1 at 0(leader) e4 
handle_command mon_command({"prefix": "osd metadata", "id": 2} v 0) v1
2017-09-18 21:05:29.933383 7f8a15e87700  0 log_channel(audit) log [DBG] 
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon' 
cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch
2017-09-18 21:05:29.933674 7f8a15e87700  0 mon.1 at 0(leader) e4 
handle_command mon_command({"prefix": "osd metadata", "id": 3} v 0) v1
2017-09-18 21:05:29.933692 7f8a15e87700  0 log_channel(audit) log [DBG] 
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon' 
cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch
2017-09-18 21:05:29.933880 7f8a15e87700  0 mon.1 at 0(leader) e4 
handle_command mon_command({"prefix": "osd metadata", "id": 4} v 0) v1
2017-09-18 21:05:29.933897 7f8a15e87700  0 log_channel(audit) log [DBG] 
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon' 
cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch
2017-09-18 21:05:29.934062 7f8a15e87700  0 mon.1 at 0(leader) e4 
handle_command mon_command({"prefix": "osd metadata", "id": 5} v 0) v1
2017-09-18 21:05:29.934089 7f8a15e87700  0 log_channel(audit) log [DBG] 
: from='client.5804116 10.250.0.23:0/4045099631' entity='mgr.jon' 
cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch
2017-09-18 21:05:30.113007 7f8a1b4a8700  1 mon.1 at 0(leader).log v2152265 
check_sub sending message to client.5804116 10.250.0.23:0/4045099631 
with 3 entries (version 2152265)
2017-09-18 21:05:31.154227 7f8a1b4a8700  1 mon.1 at 0(leader).log v2152266 
check_sub sending message to client.5804116 10.250.0.23:0/4045099631 
with 0 entries (version 2152266)
2017-09-18 21:05:32.289428 7f8a1b4a8700  1 mon.1 at 0(leader).log v2152267 
check_sub sending message to client.5804116 10.250.0.23:0/4045099631 
with 0 entries (version 2152267)
2017-09-18 21:05:36.782573 7f8a1b4a8700  1 mon.1 at 0(leader).log v2152268 
check_sub sending message to client.5804116 10.250.0.23:0/4045099631 
with 0 entries (version 2152268)
2017-09-18 21:06:09.964314 7f8a1868c700  0 
mon.1 at 0(leader).data_health(2028) update_stats avail 90% total 58203 MB, 
used 2744 MB, avail 52473 MB
2017-09-18 21:06:20.040930 7f8a1b4a8700  1 mon.1 at 0(leader).log v2152269 
check_sub sending message to client.5804116 10.250.0.23:0/4045099631 
with 0 entries (version 2152269)


And ceph.log

root at jaime:~# tail -f /var/log/ceph/ceph.log
2017-09-18 12:00:00.000160 mon.1 mon.0 10.250.0.21:6789/0 38100 : 
cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883 
objects misplaced (3.653%); Degraded data redundancy: 73487/1415883 
objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs 
undersized; Degraded data redundancy (low space): 29 pgs 
backfill_toofull; application not enabled on 2 pool(s)
2017-09-18 13:00:00.000160 mon.1 mon.0 10.250.0.21:6789/0 38101 : 
cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883 
objects misplaced (3.653%); Degraded data redundancy: 73487/1415883 
objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs 
undersized; Degraded data redundancy (low space): 29 pgs 
backfill_toofull; application not enabled on 2 pool(s)
2017-09-18 14:00:00.000133 mon.1 mon.0 10.250.0.21:6789/0 38102 : 
cluster [ERR] overall HEALTH_ERR 3 backfillfull osd(s); 51727/1415883 
objects misplaced (3.653%); Degraded data redundancy: 73487/1415883 
objects degraded (5.190%), 30 pgs unclean, 21 pgs degraded, 21 pgs 
undersized; Degraded data redundancy (low space): 29 pgs 
backfill_toofull; application not enabled on 2 pool(s)
201

Yes, the cluster is not really healthy, indeed ....

Thanks

> 
>>
>> 2017-09-16 19:12:00.468481 7f6469cdde00  0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5624
>> 2017-09-16 19:12:00.470154 7f6469cdde00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:00.471408 7f6469cdde00  1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:00.471727 7f6469cdde00  1 journal close /dev/sdb2
>> 2017-09-16 19:12:00.471994 7f6469cdde00  0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:05.042622 7f000b944e00  0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5702
>> 2017-09-16 19:12:05.066343 7f000b944e00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:05.066549 7f000b944e00  1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:05.066717 7f000b944e00  1 journal close /dev/sdb2
>> 2017-09-16 19:12:05.066843 7f000b944e00  0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:08.198548 7f5740748e00  0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5767
>> 2017-09-16 19:12:08.223674 7f5740748e00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:08.223831 7f5740748e00  1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:08.224213 7f5740748e00  1 journal close /dev/sdb2
>> 2017-09-16 19:12:08.224342 7f5740748e00  0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:09.149622 7f7b06058e00  0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5800
>> 2017-09-16 19:12:09.173319 7f7b06058e00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:09.173402 7f7b06058e00  1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:09.173485 7f7b06058e00  1 journal close /dev/sdb2
>> 2017-09-16 19:12:09.173511 7f7b06058e00  0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:10.197944 7f7561d50e00  0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5828
>> 2017-09-16 19:12:10.222504 7f7561d50e00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:10.222723 7f7561d50e00  1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:10.222753 7f7561d50e00  1 journal close /dev/sdb2
>> 2017-09-16 19:12:10.222785 7f7561d50e00  0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:14.370797 7f9fecb7fe00  0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 5964
>> 2017-09-16 19:12:14.371221 7f9fecb7fe00 -1 bluestore(/dev/sdb2)
>> _read_bdev_label unable to decode label at offset 102:
>> buffer::malformed_input: void
>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode past
>> end of struct encoding
>> 2017-09-16 19:12:14.371350 7f9fecb7fe00  1 journal _open /dev/sdb2 fd 4:
>> 750050447360 bytes, block size 4096 bytes, directio = 0, aio = 0
>> 2017-09-16 19:12:14.371616 7f9fecb7fe00  1 journal close /dev/sdb2
>> 2017-09-16 19:12:14.371745 7f9fecb7fe00  0 probe_block_device_fsid /dev/sdb2
>> is filestore, 00000000-0000-0000-0000-000000000000
>> 2017-09-16 19:12:21.171036 7f5d7579be00  0 ceph version 12.2.0
>> (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process (unknown),
>> pid 6130
>> 2017-09-16 19:12:21.209441 7f5d7579be00  0 probe_block_device_fsid /dev/sdb2
>> is bluestore, 92a4a9eb-0a6a-405d-be83-11e4af42fa30
>>
>>
>>
>> Any hint ?
>>
>> Thanks by advance
>> Best regards
>>
> --
> Cheers,
> Alwin
> 
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> 



More information about the pve-user mailing list