[PVE-User] pveceph createosd after destroyed osd

[Alwin Antreich]

On Tue, Jul 03, 2018 at 12:18:53PM +0100, Mark Adams wrote:
> Hi Alwin, please see my response below.
> 
> On 3 July 2018 at 10:07, Alwin Antreich <a.antreich at proxmox.com> wrote:
> 
> > On Tue, Jul 03, 2018 at 01:05:51AM +0100, Mark Adams wrote:
> > > Currently running the newest 5.2-1 version, I had a test cluster which
> > was
> > > working fine. I since added more disks, first stopping, then setting out,
> > > then destroying each osd so I could recreate it all from scratch.
> > >
> > > However, when adding a new osd (either via GUI or pveceph CLI) it seems
> > to
> > > show a successful create, however does not show in the gui as an osd
> > under
> > > the host.
> > >
> > > It's like the osd information is being stored by proxmox/ceph somewhere
> > > else and not being correctly removed and recreated?
> > >
> > > I can see that the newly created disk (after it being destroyed) is
> > > down/out.
> > >
> > > Is this by design? is there a way to force the disk back? shouldn't it
> > show
> > > in the gui once you create it again?
> > >
> > Did you zero your disks after removal? On the first ~200 MB of the disk,
> > there are leftovers that need to be zeroed before use. After that the
> > OSD should be added fine.
> >
> >
> I hadn't done this, no - it has helped with the majority of disks thanks
> and I can now re-add them (I also had to remove the folders from
> /var/lib/ceph/osd which had other osd names - not sure if the destroy
> process is supposed to remove them also?)
They will not interfere. ;)

> 
> However I have a strange problem on the 2nd host, where it will not make
> osd.12 ... I get no error output from the gui or pveceph createosd /dev/sda
> - it just doesn't appear as an osd.
> 
> It successfully partitions the disk, but doesn't create a folder in
> /var/lib/ceph/osd/ for the osd mount. I can see there is lock files in
> /var/lib/ceph/tmp/ ... which I would think should only be there whilst the
> creation is taking place?
>From the OSD creation or different lock? Shouldn't make troubles either.

> 
> journalctl -xe is showing me the problem I think, "command_with_stdin:
> Error EEXIST: entity osd.12 exists but key does not match"
> 
> Where is this key? how should I be clearing it out so it will create?
> 
'ceph auth list' will show you all keys in ceph, there will be an orphan
osd.12. Removal 'ceph auth del <osd.id>'.

--
Cheers,
Alwin