[PVE-User] osd init authentication failed: (1) Operation not permitted

[Alwin Antreich]

Hello Thomas,

On Fri, Jun 26, 2020 at 07:51:57AM +0000, Naumann, Thomas wrote:
> Hi,
> 
> in our production cluster (proxmox 5.4, ceph 12.2) there is an issue
> since yesterday. after an increase of a pool 5 OSDs do not start,
> status is "down/in", ceph health: HEALTH_WARN nodown,noout flag(s) set,
> 5 osds down, 128 osds: 123 up, 128 in.
> 
> last lines of OSD-logfile:
> 2020-06-26 08:40:26.240005 7f6d245fff80  1 freelist init
> 2020-06-26 08:40:26.243779 7f6d245fff80  1
> bluestore(/var/lib/ceph/osd/ceph-45) _open_alloc opening allocation
> metadata
> 2020-06-26 08:40:26.251501 7f6d245fff80  1
> bluestore(/var/lib/ceph/osd/ceph-45) _open_alloc loaded 3.47TiB in 1
> extents
> 2020-06-26 08:40:26.253058 7f6d245fff80  0 <cls>
> /mnt/big/pve/ceph/ceph-12.2.13/src/cls/cephfs/cls_cephfs.cc:197:
> loading cephfs
> 2020-06-26 08:40:26.253309 7f6d245fff80  0 _get_class not permitted to
> load sdk
> 2020-06-26 08:40:26.256486 7f6d245fff80  0 _get_class not permitted to
> load kvs
> 2020-06-26 08:40:26.256611 7f6d245fff80  0 <cls>
> /mnt/big/pve/ceph/ceph-12.2.13/src/cls/hello/cls_hello.cc:296: loading
> cls_hello
> 2020-06-26 08:40:26.258362 7f6d245fff80  0 _get_class not permitted to
> load lua
> 2020-06-26 08:40:26.259850 7f6d245fff80  0 osd.45 46770 crush map has
> features 288514051259236352, adjusting msgr requires for clients
> 2020-06-26 08:40:26.259859 7f6d245fff80  0 osd.45 46770 crush map has
> features 288514051259236352 was 8705, adjusting msgr requires for mons
> 2020-06-26 08:40:26.259863 7f6d245fff80  0 osd.45 46770 crush map has
> features 1009089991638532096, adjusting msgr requires for osds
> 2020-06-26 08:40:26.305880 7f6d245fff80  0 osd.45 46770 load_pgs
> 2020-06-26 08:40:28.024638 7f6d245fff80  0 osd.45 46770 load_pgs opened
> 129 pgs
> 2020-06-26 08:40:28.024803 7f6d245fff80  0 osd.45 46770 using
> weightedpriority op queue with priority op cut off at 64.
> 2020-06-26 08:40:28.025741 7f6d245fff80 -1 osd.45 46770 log_to_monitors
> {default=true}
> 2020-06-26 08:40:28.028397 7f6d245fff80 -1 osd.45 46770 init
> authentication failed: (1) Operation not permitted
> 
> Does anyone know how to fix this?
Are does OSDs on the same host? What is the current status of the
cluster?

--
Cheers,
Alwin