[PVE-User] ceph

[Eneko Lacunza]

You should change your public_network to 192.168.255.0/24 .

Then, one by one, remove a monitor and recreate it, check values for new 
monitor are on correct network.

Finally restart one by one OSD services and check their listening IPs 
(they listen on public and private Ceph networks).

Cheers

El 10/1/23 a las 14:29, Piviul escribió:
> On 1/10/23 09:04, Eneko Lacunza via pve-user wrote:
>>
>> I think you may have a wrong Ceph network definition in 
>> /etc/pve/ceph.conf, check for "public_network". 
> # cat /etc/pve/ceph.conf
> [global]
>      auth_client_required = cephx
>      auth_cluster_required = cephx
>      auth_service_required = cephx
>      cluster_network = 192.168.254.3/24
>      fsid = 332d7723-d16f-443f-947d-a5ab160e4fac
>      mon_allow_pool_delete = true
>      mon_host = 192.168.70.34 192.168.70.30 192.168.70.32
>      osd_pool_default_min_size = 2
>      osd_pool_default_size = 3
>      public_network = 192.168.70.34/20
>
> [client]
>      keyring = /etc/pve/priv/$cluster.$name.keyring
>
> [mds]
>      keyring = /var/lib/ceph/mds/ceph-$id/keyring
>
> [mds.pve01]
>      host = pve01
>      mds standby for name = pve
>
> [mds.pve02]
>      host = pve02
>      mds_standby_for_name = pve
>
> [mds.pve03]
>      host = pve03
>      mds_standby_for_name = pve
>
> [mon.pve01]
>      public_addr = 192.168.70.30
>
> [mon.pve02]
>      public_addr = 192.168.70.32
>
> [mon.pve03]
>      public_addr = 192.168.70.34
>
>
>> Also check what IP are listening on OSD daemons.
> # netstat -tunlp | grep ceph-
> tcp        0      0 192.168.254.1:6806      0.0.0.0:* LISTEN 
> 1760/ceph-osd
> tcp        0      0 192.168.70.30:6806      0.0.0.0:* LISTEN 
> 1760/ceph-osd
> tcp        0      0 192.168.254.1:6807      0.0.0.0:* LISTEN 
> 1760/ceph-osd
> tcp        0      0 192.168.70.30:6807      0.0.0.0:* LISTEN 
> 1761/ceph-osd
> tcp        0      0 192.168.254.1:6808      0.0.0.0:* LISTEN 
> 1762/ceph-osd
> tcp        0      0 192.168.70.30:6808      0.0.0.0:* LISTEN 
> 1760/ceph-osd
> tcp        0      0 192.168.254.1:6809      0.0.0.0:* LISTEN 
> 1762/ceph-osd
> tcp        0      0 192.168.70.30:6809      0.0.0.0:* LISTEN 
> 1760/ceph-osd
> tcp        0      0 192.168.254.1:6810      0.0.0.0:* LISTEN 
> 1762/ceph-osd
> tcp        0      0 192.168.70.30:6810      0.0.0.0:* LISTEN 
> 1762/ceph-osd
> tcp        0      0 192.168.254.1:6811      0.0.0.0:* LISTEN 
> 1762/ceph-osd
> tcp        0      0 192.168.70.30:6811      0.0.0.0:* LISTEN 
> 1762/ceph-osd
> tcp        0      0 192.168.254.1:6812      0.0.0.0:* LISTEN 
> 68106/ceph-osd
> tcp        0      0 192.168.70.30:6812      0.0.0.0:* LISTEN 
> 1762/ceph-osd
> tcp        0      0 192.168.254.1:6813      0.0.0.0:* LISTEN 
> 68106/ceph-osd
> tcp        0      0 192.168.70.30:6813      0.0.0.0:* LISTEN 
> 1762/ceph-osd
> tcp        0      0 192.168.254.1:6814      0.0.0.0:* LISTEN 
> 68106/ceph-osd
> tcp        0      0 192.168.70.30:6814      0.0.0.0:* LISTEN 
> 68106/ceph-osd
> tcp        0      0 192.168.254.1:6815      0.0.0.0:* LISTEN 
> 68106/ceph-osd
> tcp        0      0 192.168.70.30:6815      0.0.0.0:* LISTEN 
> 68106/ceph-osd
> tcp        0      0 192.168.70.30:6816      0.0.0.0:* LISTEN 
> 68106/ceph-osd
> tcp        0      0 192.168.70.30:6817      0.0.0.0:* LISTEN 
> 68106/ceph-osd
> tcp        0      0 192.168.70.30:3300      0.0.0.0:* LISTEN 
> 1753/ceph-mon
> tcp        0      0 192.168.70.30:6789      0.0.0.0:* LISTEN 
> 1753/ceph-mon
> tcp        0      0 192.168.254.1:6800      0.0.0.0:* LISTEN 
> 1761/ceph-osd
> tcp        0      0 192.168.70.30:6800      0.0.0.0:* LISTEN 
> 1740/ceph-mds
> tcp        0      0 192.168.254.1:6801      0.0.0.0:* LISTEN 
> 1761/ceph-osd
> tcp        0      0 192.168.70.30:6801      0.0.0.0:* LISTEN 
> 1740/ceph-mds
> tcp        0      0 192.168.254.1:6802      0.0.0.0:* LISTEN 
> 1760/ceph-osd
> tcp        0      0 192.168.70.30:6802      0.0.0.0:* LISTEN 
> 1761/ceph-osd
> tcp        0      0 192.168.254.1:6803      0.0.0.0:* LISTEN 
> 1760/ceph-osd
> tcp        0      0 192.168.70.30:6803      0.0.0.0:* LISTEN 
> 1761/ceph-osd
> tcp        0      0 192.168.254.1:6804      0.0.0.0:* LISTEN 
> 1761/ceph-osd
> tcp        0      0 192.168.70.30:6804      0.0.0.0:* LISTEN 
> 1760/ceph-osd
> tcp        0      0 192.168.254.1:6805      0.0.0.0:* LISTEN 
> 1761/ceph-osd
> tcp        0      0 192.168.70.30:6805      0.0.0.0:* LISTEN 
> 1761/ceph-osd
>
> I wrong something on CEPH configuration?
>
> Piviul
>
>
>
> _______________________________________________
> pve-user mailing list
> pve-user at lists.proxmox.com
> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Eneko Lacunza
Zuzendari teknikoa | Director técnico
Binovo IT Human Project

Tel. +34 943 569 206 |https://www.binovo.es
Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun

https://www.youtube.com/user/CANALBINOVO
https://www.linkedin.com/company/37269706/