[PVE-User] Cephfs starting 2nd MDS
Vadim Bulst
vadim.bulst at bbz.uni-leipzig.de
Wed Aug 8 07:54:45 CEST 2018
Hi Alwin,
thanks for your advise. But no success. Still same error.
mds-section:
[mds.1]
host = scvirt03
keyring = /var/lib/ceph/mds/ceph-scvirt03/keyring
Vadim
On 07.08.2018 15:30, Alwin Antreich wrote:
> Hello Vadim,
>
> On Tue, Aug 7, 2018, 12:13 Vadim Bulst <vadim.bulst at bbz.uni-leipzig.de>
> wrote:
>
>> Dear list,
>>
>> I'm trying to bring up a second mds with no luck.
>>
>> This is what my ceph.conf looks like:
>>
>> [global]
>>
>> auth client required = cephx
>> auth cluster required = cephx
>> auth service required = cephx
>> cluster network = 10.10.144.0/24
>> filestore xattr use omap = true
>> fsid = 5349724e-fa96-4fd6-8e44-8da2a39253f7
>> keyring = /etc/pve/priv/$cluster.$name.keyring
>> osd journal size = 5120
>> osd pool default min size = 1
>> public network = 172.18.144.0/24
>> mon allow pool delete = true
>>
>> [osd]
>> keyring = /var/lib/ceph/osd/ceph-$id/keyring
>>
>> [mon.2]
>> host = scvirt03
>> mon addr = 172.18.144.243:6789
>>
>> [mon.0]
>> host = scvirt01
>> mon addr = 172.18.144.241:6789
>> [mon.1]
>> host = scvirt02
>> mon addr = 172.18.144.242:6789
>>
>> [mds.0]
>> host = scvirt02
>> [mds.1]
>> host = scvirt03
>>
>>
>> I did the following to set up the service:
>>
>> apt install ceph-mds
>>
>> mkdir /var/lib/ceph/mds
>>
>> mkdir /var/lib/ceph/mds/ceph-$(hostname -s)
>>
>> chown -R ceph:ceph /var/lib/ceph/mds
>>
>> chmod -R 0750 /var/lib/ceph/mds
>>
>> ceph auth get-or-create mds.$(hostname -s) mon 'allow profile mds' mgr
>> 'allow profile mds' osd 'allow rwx' mds 'allow' >
>> /var/lib/ceph/mds/ceph-$(hostname -s)/keyring
>>
>> chmod -R 0600 /var/lib/ceph/mds/ceph-$(hostname -s)/keyring
>>
>> systemctl enable ceph-mds@$(hostname -s).service
>>
>> systemctl start ceph-mds@$(hostname -s).service
>>
>>
>> The service will not start. I also did the same procedure with the first
>> mds which is running with no problems.
>>
>> 1st mds:
>>
>> root at scvirt02:/home/urzadmin# systemctl status -l ceph-mds@$(hostname
>> -s).service
>> ● ceph-mds at scvirt02.service - Ceph metadata server daemon
>> Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled;
>> vendor preset: enabled)
>> Drop-In: /lib/systemd/system/ceph-mds at .service.d
>> └─ceph-after-pve-cluster.conf
>> Active: active (running) since Thu 2018-06-07 13:08:58 CEST; 2
>> months 0 days ago
>> Main PID: 612704 (ceph-mds)
>> CGroup:
>> /system.slice/system-ceph\x2dmds.slice/ceph-mds at scvirt02.service
>> └─612704 /usr/bin/ceph-mds -f --cluster ceph --id scvirt02
>> --setuser ceph --setgroup ceph
>>
>> Jul 29 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-29 06:25:01.792601
>> 7f6e4bae0700 -1 received signal: Hangup from PID: 3831071 task name:
>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0
>> Jul 30 06:25:02 scvirt02 ceph-mds[612704]: 2018-07-30 06:25:02.081591
>> 7f6e4bae0700 -1 received signal: Hangup from PID: 184355 task name:
>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0
>> Jul 31 06:25:01 scvirt02 ceph-mds[612704]: 2018-07-31 06:25:01.448571
>> 7f6e4bae0700 -1 received signal: Hangup from PID: 731440 task name:
>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0
>> Aug 01 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-01 06:25:01.274541
>> 7f6e4bae0700 -1 received signal: Hangup from PID: 1278492 task name:
>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0
>> Aug 02 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-02 06:25:02.009054
>> 7f6e4bae0700 -1 received signal: Hangup from PID: 1825500 task name:
>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0
>> Aug 03 06:25:02 scvirt02 ceph-mds[612704]: 2018-08-03 06:25:02.042845
>> 7f6e4bae0700 -1 received signal: Hangup from PID: 2372815 task name:
>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0
>> Aug 04 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-04 06:25:01.404619
>> 7f6e4bae0700 -1 received signal: Hangup from PID: 2919837 task name:
>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0
>> Aug 05 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-05 06:25:01.214749
>> 7f6e4bae0700 -1 received signal: Hangup from PID: 3467000 task name:
>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0
>> Aug 06 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-06 06:25:01.149512
>> 7f6e4bae0700 -1 received signal: Hangup from PID: 4014197 task name:
>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0
>> Aug 07 06:25:01 scvirt02 ceph-mds[612704]: 2018-08-07 06:25:01.863104
>> 7f6e4bae0700 -1 received signal: Hangup from PID: 367698 task name:
>> killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0
>>
>> 2nd mds:
>>
>> root at scvirt03:/home/urzadmin# systemctl status -l ceph-mds@$(hostname
>> -s).service
>> ● ceph-mds at scvirt03.service - Ceph metadata server daemon
>> Loaded: loaded (/lib/systemd/system/ceph-mds at .service; enabled;
>> vendor preset: enabled)
>> Drop-In: /lib/systemd/system/ceph-mds at .service.d
>> └─ceph-after-pve-cluster.conf
>> Active: inactive (dead) since Tue 2018-08-07 10:27:18 CEST; 1h
>> 38min ago
>> Process: 3620063 ExecStart=/usr/bin/ceph-mds -f --cluster ${CLUSTER}
>> --id scvirt03 --setuser ceph --setgroup ceph (code=exited,
>> status=0/SUCCESS)
>> Main PID: 3620063 (code=exited, status=0/SUCCESS)
>>
>> Aug 07 10:27:17 scvirt03 systemd[1]: Started Ceph metadata server daemon.
>> Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: starting mds.scvirt03 at -
>> Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008338
>> 7f6be03816c0 -1 auth: unable to find a keyring on
>> /etc/pve/priv/ceph.mds.scvirt03.keyring: (13) Permission denied
>> Aug 07 10:27:18 scvirt03 ceph-mds[3620063]: 2018-08-07 10:27:18.008351
>> 7f6be03816c0 -1 mds.scvirt03 ERROR: failed to get monmap: (13)
>> Permission denied
>>
>>
>> content of /etc/pve/priv
>>
>> root at scvirt03:/home/urzadmin# ls -la /etc/pve/priv/
>> total 5
>> drwx------ 2 root www-data 0 Apr 15 2017 .
>> drwxr-xr-x 2 root www-data 0 Jan 1 1970 ..
>> -rw------- 1 root www-data 1675 Apr 15 2017 authkey.key
>> -rw------- 1 root www-data 1976 Jul 6 15:41 authorized_keys
>> drwx------ 2 root www-data 0 Apr 16 2017 ceph
>> -rw------- 1 root www-data 63 Apr 15 2017 ceph.client.admin.keyring
>> -rw------- 1 root www-data 214 Apr 15 2017 ceph.mon.keyring
>> -rw------- 1 root www-data 4224 Jul 6 15:41 known_hosts
>> drwx------ 2 root www-data 0 Apr 15 2017 lock
>> -rw------- 1 root www-data 3243 Apr 15 2017 pve-root-ca.key
>> -rw------- 1 root www-data 3 Jul 6 15:41 pve-root-ca.srl
>> -rw------- 1 root www-data 36 May 23 13:03 urzbackup.cred
>>
>>
>> What could be the reason this failure?
>>
> The ceph user has no permissions to access the the keyring under /etc/pve.
> Add a section for [mds] into the ceph.conf pointing to the keyring, similar
> to the OSD one. This way the MDS will find the key in It's working
> directory.
>
> Cheers,
> Alwin
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
More information about the pve-user
mailing list