[PVE-User] [ceph-users] OSD won't start, even created ??
Phil Schwarz
infolist at schwarz-fr.net
Sun Sep 10 18:54:00 CEST 2017
OK, a few more hints -for a potential helper ;-) :
- Did a complete cluster update (dist-upgrade)
- Installed a mgr for every of the 3 mons.
- plugged a new disk in the same server
- Using GUI, installed the disk as new OSD WITH Bluestore chosen.
The OSD isn't visible until
ceph osd crush add osd.0 0 host=d
And then, the OSD appears in the GUI BUT:
- it is seen as filestore one
- is marked as down
ceph-osd.admin.log ::
tail -f /var/log/ceph/ceph-osd.admin.log
2017-09-10 17:28:24.937583 7feace863e00 0 ceph version 12.2.0
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process
(unknown), pid 15859
2017-09-10 17:28:24.938042 7feace863e00 -1 bluestore(/dev/sdc2)
_read_bdev_label unable to decode label at offset 62:
buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) no longer
understand old encoding version 1 < struct_compat
2017-09-10 17:28:24.938140 7feace863e00 1 journal _open /dev/sdc2 fd 4:
319967006720 bytes, block size 4096 bytes, directio = 0, aio = 0
2017-09-10 17:28:24.938425 7feace863e00 -1 journal read_header error
decoding journal header
2017-09-10 17:28:24.938433 7feace863e00 1 journal close /dev/sdc2
2017-09-10 17:28:25.933794 7f00ad7d7e00 0 ceph version 12.2.0
(36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc), process
(unknown), pid 15938
2017-09-10 17:28:25.936225 7f00ad7d7e00 -1 bluestore(/dev/sdc2)
_read_bdev_label unable to decode label at offset 62:
buffer::malformed_input: void
bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) no longer
understand old encoding version 1 < struct_compat
2017-09-10 17:28:25.944991 7f00ad7d7e00 1 journal _open /dev/sdc2 fd 4:
319967006720 bytes, block size 4096 bytes, directio = 0, aio = 0
2017-09-10 17:28:25.945369 7f00ad7d7e00 -1 journal read_header error
decoding journal header
2017-09-10 17:28:25.945396 7f00ad7d7e00 1 journal close /dev/sdc2
sgdisk --print /dev/sdc
Disk /dev/sdc: 625142448 sectors, 298.1 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 093F0FB0-1F83-4E8F-825E-7666662DA535
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 625142414
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number Start (sector) End (sector) Size Code Name
1 2048 206847 100.0 MiB F800 ceph data
2 206848 625142414 298.0 GiB FFFF ceph block
My cluster is getting near full (In fact it is full; i have to manually
reweight some larger OSDs to decrease to Defcon 3...)
thanks by advance
Best regards
Le 09/09/2017 à 16:08, Phil Schwarz a écrit :
> Did a few more tests :
>
> Older Ceph server with a pveceph create osd command (
>
> (pveceph create osd /dev/sdb
>
> equivalent to
>
> ceph-disk prepare --zap-disk --fs-type xfs --cluster ceph --cluster-uuid
> a5c0cfed-...4bf939ed70 /dev/sdb )
>
> sgdisk --print /dev/sdd
>
> Disk /dev/sdd: 2930277168 sectors, 1.4 TiB
> Logical sector size: 512 bytes
> Disk identifier (GUID): 638646CF-......-62296C871132
> Partition table holds up to 128 entries
> First usable sector is 34, last usable sector is 2930277134
> Partitions will be aligned on 2048-sector boundaries
> Total free space is 2014 sectors (1007.0 KiB)
>
> Number Start (sector) End (sector) Size Code Name
> 1 10487808 2930277134 1.4 TiB F800 ceph data
> 2 2048 10487807 5.0 GiB F802 ceph journal
>
>
> On a newer ceph server ( dpkg -l : 12.2.0-pve1 version)
>
> sgdisk --print /dev/sdb
>
> Disk /dev/sdb: 1465149168 sectors, 698.6 GiB
> Logical sector size: 512 bytes
> Disk identifier (GUID): D63886B6-0.....26-BCBCD6FFCA3C
> Partition table holds up to 128 entries
> First usable sector is 34, last usable sector is 1465149134
> Partitions will be aligned on 2048-sector boundaries
> Total free space is 2014 sectors (1007.0 KiB)
>
> Number Start (sector) End (sector) Size Code Name
> 1 2048 206847 100.0 MiB F800 ceph data
> 2 206848 1465149134 698.5 GiB FFFF ceph block
>
>
> Related to the cep-osd.admin log , i think i used a osd creation process
> leading to a bluestore osd (instead of a filestore one).
> And seems that afterward the ceph server is unable to use the new
> bluestore :
>
> ( bluestore(/dev/sdb2) _read_bdev_label unable to decode label at offset
> 102: buffer::malformed_input: void
> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode
> past end of struct encoding
> )
> just before trying to use it as a filestore one :
>
> ( probe_block_device_fsid /dev/sdb2 is filestore )
>
>
> Tried to use the --bluestore 0 flag when creating the osd, but the flag
> is unknown.
>
> thanks by advance for any hint.
> Being ready to do a few more tests.
> Best regards.
>
> Le 08/09/2017 à 17:27, Phil Schwarz a écrit :
>> Hi,
>> any help would be really useful.
>> Does anyone got a clue with my issue ?
>>
>> Thanks by advance.
>> Best regards;
>>
>>
>> Le 05/09/2017 à 20:25, Phil Schwarz a écrit :
>>> Hi,
>>> I come back with same issue as seen in previous thread ( link given)
>>>
>>> trying to a 2TB SATA as OSD:
>>> Using proxmox GUI or CLI (command given) give the same (bad) result.
>>>
>>> Didn't want to use a direct 'ceph osd create', thus bypassing pxmfs
>>> redundant filesystem.
>>>
>>> I tried to build an OSD woth same disk on another machine (stronger one
>>> with Opteron QuadCore), failing at the same time.
>>>
>>>
>>> Sorry for crossposting, but i think, i fail against the pveceph wrapper.
>>>
>>>
>>> Any help or clue would be really useful..
>>>
>>> Thanks
>>> Best regards.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -- Link to previous thread (but same problem):
>>> https://www.mail-archive.com/ceph-users@lists.ceph.com/msg38897.html
>>>
>>>
>>> -- commands :
>>> fdisk /dev/sdc ( mklabel msdos, w, q)
>>> ceph-disk zap /dev/sdc
>>> pveceph createosd /dev/sdc
>>>
>>> -- dpkg -l
>>>
>>> dpkg -l |grep ceph
>>> ii ceph 12.1.2-pve1 amd64
>>> distributed storage and file system
>>> ii ceph-base 12.1.2-pve1 amd64 common
>>> ceph daemon libraries and management tools
>>> ii ceph-common 12.1.2-pve1 amd64 common
>>> utilities to mount and interact with a ceph storage cluster
>>> ii ceph-mgr 12.1.2-pve1 amd64
>>> manager for the ceph distributed storage system
>>> ii ceph-mon 12.1.2-pve1 amd64
>>> monitor server for the ceph storage system
>>> ii ceph-osd 12.1.2-pve1 amd64 OSD
>>> server for the ceph storage system
>>> ii libcephfs1 10.2.5-7.2 amd64 Ceph
>>> distributed file system client library
>>> ii libcephfs2 12.1.2-pve1 amd64 Ceph
>>> distributed file system client library
>>> ii python-cephfs 12.1.2-pve1 amd64 Python
>>> 2 libraries for the Ceph libcephfs library
>>>
>>> -- tail -f /var/log/ceph/ceph-osd.admin.log
>>>
>>> 2017-09-03 18:28:20.856641 7fad97e45e00 0 ceph version 12.1.2
>>> (cd7bc3b11cdbe6fa94324b7322fb2a4716a052a7) luminous (rc), process
>>> (unknown), pid 5493
>>> 2017-09-03 18:28:20.857104 7fad97e45e00 -1 bluestore(/dev/sdc2)
>>> _read_bdev_label unable to decode label at offset 102:
>>> buffer::malformed_input: void
>>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode
>>> past end of struct encoding
>>> 2017-09-03 18:28:20.857200 7fad97e45e00 1 journal _open /dev/sdc2 fd 4:
>>> 2000293007360 bytes, block size 4096 bytes, directio = 0, aio = 0
>>> 2017-09-03 18:28:20.857366 7fad97e45e00 1 journal close /dev/sdc2
>>> 2017-09-03 18:28:20.857431 7fad97e45e00 0 probe_block_device_fsid
>>> /dev/sdc2 is filestore, 00000000-0000-0000-0000-000000000000
>>> 2017-09-03 18:28:21.937285 7fa5766a5e00 0 ceph version 12.1.2
>>> (cd7bc3b11cdbe6fa94324b7322fb2a4716a052a7) luminous (rc), process
>>> (unknown), pid 5590
>>> 2017-09-03 18:28:21.944189 7fa5766a5e00 -1 bluestore(/dev/sdc2)
>>> _read_bdev_label unable to decode label at offset 102:
>>> buffer::malformed_input: void
>>> bluestore_bdev_label_t::decode(ceph::buffer::list::iterator&) decode
>>> past end of struct encoding
>>> 2017-09-03 18:28:21.944305 7fa5766a5e00 1 journal _open /dev/sdc2 fd 4:
>>> 2000293007360 bytes, block size 4096 bytes, directio = 0, aio = 0
>>> 2017-09-03 18:28:21.944527 7fa5766a5e00 1 journal close /dev/sdc2
>>> 2017-09-03 18:28:21.944588 7fa5766a5e00 0 probe_block_device_fsid
>>> /dev/sdc2 is filestore, 00000000-0000-0000-0000-000000000000
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users at lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
More information about the pve-user
mailing list