[PVE-User] pveceph createosd after destroyed osd

[Mark Adams]

Hi Alwin, please see my response below.

On 3 July 2018 at 10:07, Alwin Antreich <a.antreich at proxmox.com> wrote:

> On Tue, Jul 03, 2018 at 01:05:51AM +0100, Mark Adams wrote:
> > Currently running the newest 5.2-1 version, I had a test cluster which
> was
> > working fine. I since added more disks, first stopping, then setting out,
> > then destroying each osd so I could recreate it all from scratch.
> >
> > However, when adding a new osd (either via GUI or pveceph CLI) it seems
> to
> > show a successful create, however does not show in the gui as an osd
> under
> > the host.
> >
> > It's like the osd information is being stored by proxmox/ceph somewhere
> > else and not being correctly removed and recreated?
> >
> > I can see that the newly created disk (after it being destroyed) is
> > down/out.
> >
> > Is this by design? is there a way to force the disk back? shouldn't it
> show
> > in the gui once you create it again?
> >
> Did you zero your disks after removal? On the first ~200 MB of the disk,
> there are leftovers that need to be zeroed before use. After that the
> OSD should be added fine.
>
>
I hadn't done this, no - it has helped with the majority of disks thanks
and I can now re-add them (I also had to remove the folders from
/var/lib/ceph/osd which had other osd names - not sure if the destroy
process is supposed to remove them also?)

However I have a strange problem on the 2nd host, where it will not make
osd.12 ... I get no error output from the gui or pveceph createosd /dev/sda
- it just doesn't appear as an osd.

It successfully partitions the disk, but doesn't create a folder in
/var/lib/ceph/osd/ for the osd mount. I can see there is lock files in
/var/lib/ceph/tmp/ ... which I would think should only be there whilst the
creation is taking place?

journalctl -xe is showing me the problem I think, "command_with_stdin:
Error EEXIST: entity osd.12 exists but key does not match"

Where is this key? how should I be clearing it out so it will create?

Thanks,
Mark


--
> Cheers,
> Alwin
>
>