[PVE-User] pveceph createosd after destroyed osd

[Mark Adams]

Hi Anwin;

Thanks for that - It's all working now! Just to confirm though, shouldn't
the destroy button handle some of these actions? or is it left out on
purpose?

Regards,
Mark

On 3 July 2018 at 16:16, Alwin Antreich <a.antreich at proxmox.com> wrote:

> On Tue, Jul 03, 2018 at 12:18:53PM +0100, Mark Adams wrote:
> > Hi Alwin, please see my response below.
> >
> > On 3 July 2018 at 10:07, Alwin Antreich <a.antreich at proxmox.com> wrote:
> >
> > > On Tue, Jul 03, 2018 at 01:05:51AM +0100, Mark Adams wrote:
> > > > Currently running the newest 5.2-1 version, I had a test cluster
> which
> > > was
> > > > working fine. I since added more disks, first stopping, then setting
> out,
> > > > then destroying each osd so I could recreate it all from scratch.
> > > >
> > > > However, when adding a new osd (either via GUI or pveceph CLI) it
> seems
> > > to
> > > > show a successful create, however does not show in the gui as an osd
> > > under
> > > > the host.
> > > >
> > > > It's like the osd information is being stored by proxmox/ceph
> somewhere
> > > > else and not being correctly removed and recreated?
> > > >
> > > > I can see that the newly created disk (after it being destroyed) is
> > > > down/out.
> > > >
> > > > Is this by design? is there a way to force the disk back? shouldn't
> it
> > > show
> > > > in the gui once you create it again?
> > > >
> > > Did you zero your disks after removal? On the first ~200 MB of the
> disk,
> > > there are leftovers that need to be zeroed before use. After that the
> > > OSD should be added fine.
> > >
> > >
> > I hadn't done this, no - it has helped with the majority of disks thanks
> > and I can now re-add them (I also had to remove the folders from
> > /var/lib/ceph/osd which had other osd names - not sure if the destroy
> > process is supposed to remove them also?)
> They will not interfere. ;)
>
> >
> > However I have a strange problem on the 2nd host, where it will not make
> > osd.12 ... I get no error output from the gui or pveceph createosd
> /dev/sda
> > - it just doesn't appear as an osd.
> >
> > It successfully partitions the disk, but doesn't create a folder in
> > /var/lib/ceph/osd/ for the osd mount. I can see there is lock files in
> > /var/lib/ceph/tmp/ ... which I would think should only be there whilst
> the
> > creation is taking place?
> From the OSD creation or different lock? Shouldn't make troubles either.
>
> >
> > journalctl -xe is showing me the problem I think, "command_with_stdin:
> > Error EEXIST: entity osd.12 exists but key does not match"
> >
> > Where is this key? how should I be clearing it out so it will create?
> >
> 'ceph auth list' will show you all keys in ceph, there will be an orphan
> osd.12. Removal 'ceph auth del <osd.id>'.
>
>