[PVE-User] Cephfs starting 2nd MDS

Tue Aug 7 12:13:11 CEST 2018

Dear list,

I'm trying to bring up a second mds with no luck.

This is what my ceph.conf looks like:

[global]

           auth client required = cephx
           auth cluster required = cephx
           auth service required = cephx
           cluster network = 10.10.144.0/24
           filestore xattr use omap = true
           fsid = 5349724e-fa96-4fd6-8e44-8da2a39253f7
           keyring = /etc/pve/priv/$cluster.$name.keyring
           osd journal size = 5120
           osd pool default min size = 1
           public network = 172.18.144.0/24
           mon allow pool delete = true

[osd]
           keyring = /var/lib/ceph/osd/ceph-$id/keyring

[mon.2]
           host = scvirt03
           mon addr = 172.18.144.243:6789

[mon.0]
           host = scvirt01
           mon addr = 172.18.144.241:6789
[mon.1]
           host = scvirt02
           mon addr = 172.18.144.242:6789

[mds.0]
          host = scvirt02
[mds.1]
          host = scvirt03

I did the following to set up the service:

apt install ceph-mds

mkdir /var/lib/ceph/mds

mkdir /var/lib/ceph/mds/ceph-$(hostname -s)

chown -R ceph:ceph /var/lib/ceph/mds

chmod -R 0750 /var/lib/ceph/mds

ceph auth get-or-create mds.$(hostname -s) mon 'allow profile mds' mgr 
'allow profile mds' osd 'allow rwx' mds 'allow' > 
/var/lib/ceph/mds/ceph-$(hostname -s)/keyring

chmod -R 0600 /var/lib/ceph/mds/ceph-$(hostname -s)/keyring

systemctl enable ceph-mds@$(hostname -s).service

systemctl start ceph-mds@$(hostname -s).service

The service will not start. I also did the same procedure with the first 
mds which is running with no problems.

1st mds:

root at scvirt02:/home/urzadmin# systemctl status -l ceph-mds@$(hostname 
-s).service
● ceph-mds at scvirt02.service - Ceph metadata server daemon
     Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled; 
vendor preset: enabled)
    Drop-In: /lib/systemd/system/ceph-mds at .service.d
             └─ceph-after-pve-cluster.conf
     Active: active (running) since Thu 2018-06-07 13:08:58 CEST; 2 
months 0 days ago
   Main PID: 612704 (ceph-mds)
     CGroup: 
/system.slice/system-ceph\x2dmds.slice/ceph-mds at scvirt02.service
             └─612704 /usr/bin/ceph-mds -f --cluster ceph --id scvirt02 
--setuser ceph --setgroup ceph

Jul 29 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-29 06:25:01.792601 
7f6e4bae0700 -1 received  signal: Hangup from  PID: 3831071 task name: 
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0
Jul 30 06:25:02 scvirt02 ceph-mds[612704]: 2018-07-30 06:25:02.081591 
7f6e4bae0700 -1 received  signal: Hangup from  PID: 184355 task name: 
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0
Jul 31 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-31 06:25:01.448571 
7f6e4bae0700 -1 received  signal: Hangup from  PID: 731440 task name: 
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0
Aug 01 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-01 06:25:01.274541 
7f6e4bae0700 -1 received  signal: Hangup from  PID: 1278492 task name: 
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0
Aug 02 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-02 06:25:02.009054 
7f6e4bae0700 -1 received  signal: Hangup from  PID: 1825500 task name: 
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0
Aug 03 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-03 06:25:02.042845 
7f6e4bae0700 -1 received  signal: Hangup from  PID: 2372815 task name: 
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0
Aug 04 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-04 06:25:01.404619 
7f6e4bae0700 -1 received  signal: Hangup from  PID: 2919837 task name: 
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0
Aug 05 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-05 06:25:01.214749 
7f6e4bae0700 -1 received  signal: Hangup from  PID: 3467000 task name: 
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0
Aug 06 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-06 06:25:01.149512 
7f6e4bae0700 -1 received  signal: Hangup from  PID: 4014197 task name: 
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0
Aug 07 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-07 06:25:01.863104 
7f6e4bae0700 -1 received  signal: Hangup from  PID: 367698 task name: 
killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw  UID: 0

2nd mds:

root at scvirt03:/home/urzadmin# systemctl status -l ceph-mds@$(hostname 
-s).service
● ceph-mds at scvirt03.service - Ceph metadata server daemon
     Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled; 
vendor preset: enabled)
    Drop-In: /lib/systemd/system/ceph-mds at .service.d
             └─ceph-after-pve-cluster.conf
     Active: inactive (dead) since Tue 2018-08-07 10:27:18 CEST; 1h 
38min ago
    Process: 3620063 ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER} 
--id scvirt03 --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS)
   Main PID: 3620063 (code=exited, status=0/SUCCESS)

Aug 07 10:27:17 scvirt03 systemd[1]: Started Ceph metadata server daemon.
Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: starting mds.scvirt03 at -
Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008338 
7f6be03816c0 -1 auth: unable to find a keyring on 
/etc/pve/priv/ceph.mds.scvirt03.keyring: (13) Permission denied
Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008351 
7f6be03816c0 -1 mds.scvirt03 ERROR: failed to get monmap: (13) 
Permission denied

content of /etc/pve/priv

root at scvirt03:/home/urzadmin# ls -la /etc/pve/priv/
total 5
drwx------ 2 root www-data    0 Apr 15  2017 .
drwxr-xr-x 2 root www-data    0 Jan  1  1970 ..
-rw------- 1 root www-data 1675 Apr 15  2017 authkey.key
-rw------- 1 root www-data 1976 Jul  6 15:41 authorized_keys
drwx------ 2 root www-data    0 Apr 16  2017 ceph
-rw------- 1 root www-data   63 Apr 15  2017 ceph.client.admin.keyring
-rw------- 1 root www-data  214 Apr 15  2017 ceph.mon.keyring
-rw------- 1 root www-data 4224 Jul  6 15:41 known_hosts
drwx------ 2 root www-data    0 Apr 15  2017 lock
-rw------- 1 root www-data 3243 Apr 15  2017 pve-root-ca.key
-rw------- 1 root www-data    3 Jul  6 15:41 pve-root-ca.srl
-rw------- 1 root www-data   36 May 23 13:03 urzbackup.cred

What could be the reason this failure?

Cheers,

Vadim

-- 
Vadim Bulst

Universität Leipzig / URZ
04109  Leipzig, Augustusplatz 10

phone: ++49-341-97-33380
mail:    vadim.bulst at uni-leipzig.de