[PVE-User] [ceph-users] Re: Ceph Usage web and terminal.
Сергей Цаболов
tsabolov at t8.ru
Wed Dec 29 15:06:16 CET 2021
Ok, I understand the case.
29.12.2021 16:13, Uwe Sauter пишет:
> Am 29.12.21 um 13:51 schrieb Сергей Цаболов:
>> Hi, Uwe
>>
>> 29.12.2021 14:16, Uwe Sauter пишет:
>>> Just a feeling but I'd say that the imbalance in OSDs (one host having many more disks than the
>>> rest) is your problem.
>> Yes, last node in cluster have more disk then the rest, but
>>
>> one disk is 12TB and all others 9 HD is 1TB
>>
>>> Assuming that your configuration keeps 3 copies of each VM image then the imbalance probably means
>>> that 2 of these 3 copies reside on pve-3111 and if this host is unavailable, all VM images with 2
>>> copies on that host become unresponsive, too.
>> In Proxmox web ceph pool I set the Size: 2 , Min.Size: 2
>>
> So this means that you want to have 2 copies in the regular case (size) and also 2 copies in the
> failure case (min size) so that the VMs stay available.
Yes I think before like you answer, but is not so worked.
>
> So you might solve your problem by decreasing min size to 1 (dangerous!!) or by increasing size to
> 3, which means that in the regular case you will have 3 copies but if only 2 are available, it will
> still work and re-sync the 3rd copy once it comes online again.
I understand if decreasing min.size to 1 is very (dangerous!!!)
If I increasing to 3 min.size keep 2 is default .
But I'm afraid if set the 3/2 (good choice) MAX AVAIL in pool is will
decrease in two or more space, or am I wrong?
For now I have with all disk :
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd `106 TiB 99 TiB 7.7 TiB 7.7 TiB 7.26
TOTAL 106 TiB 99 TiB 7.7 TiB 7.7 TiB 7.26
--- POOLS ---
POOL ID PGS STORED OBJECTS
USED %USED MAX AVAIL
device_health_metrics 1 1 8.3 MiB 22 17 MiB
0 44 TiB
vm.pool 2 1024 3.0 TiB 864.55k
6.0 TiB 6.39 44 TiB ( terminal 44 TiB = 48.37 ) in web I
see 51.50 TB
cephfs_data 3 32 874 GiB 223.76k 1.7
TiB 1.91 44 TiB
cephfs_metadata 4 32 25 MiB 27 51 MiB
0 44 TiB
Am I right in my reasoning ?
Thank you!
>
>> With : ceph osd map vm.pool object-name (vm ID) I see some of vm object one copy is on osd.12,
>> example :
>>
>> osdmap e14321 pool 'vm.pool' (2) object '114' -> pg 2.10486407 (2.7) -> up ([12,8], p12) acting
>> ([12,8], p12)
>>
>> But this example :
>>
>> osdmap e14321 pool 'vm.pool' (2) object '113' -> pg 2.8bd09f6d (2.36d) -> up ([10,7], p10) acting
>> ([10,7], p10)
>>
>> osd.10 and osd.7
>>
>>> Check your failure domain for Ceph and possibly change it from OSD to host. This should prevent that
>>> one host holds multiple copies of a VM image.
>> I didn 't understand a little what to check ?
>>
>> Can you explain me with example?
>>
> I don't have an example but you can read about the concept at:
>
> https://docs.ceph.com/en/latest/rados/operations/crush-map/#crush-maps
>
>
> Regards,
>
> Uwe
>
>
>
>>>
>>> Regards,
>>>
>>> Uwe
>>>
>>> Am 29.12.21 um 09:36 schrieb Сергей Цаболов:
>>>> Hello to all.
>>>>
>>>> In my case I have the 7 node cluster Proxmox and working Ceph (ceph version 15.2.15 octopus
>>>> (stable)": 7)
>>>>
>>>> Ceph HEALTH_OK
>>>>
>>>> ceph -s
>>>> cluster:
>>>> id: 9662e3fa-4ce6-41df-8d74-5deaa41a8dde
>>>> health: HEALTH_OK
>>>>
>>>> services:
>>>> mon: 7 daemons, quorum pve-3105,pve-3107,pve-3108,pve-3103,pve-3101,pve-3111,pve-3109 (age 17h)
>>>> mgr: pve-3107(active, since 41h), standbys: pve-3109, pve-3103, pve-3105, pve-3101, pve-3111,
>>>> pve-3108
>>>> mds: cephfs:1 {0=pve-3105=up:active} 6 up:standby
>>>> osd: 22 osds: 22 up (since 17h), 22 in (since 17h)
>>>>
>>>> task status:
>>>>
>>>> data:
>>>> pools: 4 pools, 1089 pgs
>>>> objects: 1.09M objects, 4.1 TiB
>>>> usage: 7.7 TiB used, 99 TiB / 106 TiB avail
>>>> pgs: 1089 active+clean
>>>>
>>>> ---------------------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>> ceph osd tree
>>>>
>>>> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
>>>> -1 106.43005 root default
>>>> -13 14.55478 host pve-3101
>>>> 10 hdd 7.27739 osd.10 up 1.00000 1.00000
>>>> 11 hdd 7.27739 osd.11 up 1.00000 1.00000
>>>> -11 14.55478 host pve-3103
>>>> 8 hdd 7.27739 osd.8 up 1.00000 1.00000
>>>> 9 hdd 7.27739 osd.9 up 1.00000 1.00000
>>>> -3 14.55478 host pve-3105
>>>> 0 hdd 7.27739 osd.0 up 1.00000 1.00000
>>>> 1 hdd 7.27739 osd.1 up 1.00000 1.00000
>>>> -5 14.55478 host pve-3107
>>>> 2 hdd 7.27739 osd.2 up 1.00000 1.00000
>>>> 3 hdd 7.27739 osd.3 up 1.00000 1.00000
>>>> -9 14.55478 host pve-3108
>>>> 6 hdd 7.27739 osd.6 up 1.00000 1.00000
>>>> 7 hdd 7.27739 osd.7 up 1.00000 1.00000
>>>> -7 14.55478 host pve-3109
>>>> 4 hdd 7.27739 osd.4 up 1.00000 1.00000
>>>> 5 hdd 7.27739 osd.5 up 1.00000 1.00000
>>>> -15 19.10138 host pve-3111
>>>> 12 hdd 10.91409 osd.12 up 1.00000 1.00000
>>>> 13 hdd 0.90970 osd.13 up 1.00000 1.00000
>>>> 14 hdd 0.90970 osd.14 up 1.00000 1.00000
>>>> 15 hdd 0.90970 osd.15 up 1.00000 1.00000
>>>> 16 hdd 0.90970 osd.16 up 1.00000 1.00000
>>>> 17 hdd 0.90970 osd.17 up 1.00000 1.00000
>>>> 18 hdd 0.90970 osd.18 up 1.00000 1.00000
>>>> 19 hdd 0.90970 osd.19 up 1.00000 1.00000
>>>> 20 hdd 0.90970 osd.20 up 1.00000 1.00000
>>>> 21 hdd 0.90970 osd.21 up 1.00000 1.00000
>>>>
>>>> ---------------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>> POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
>>>> vm.pool 2 1024 3.0 TiB 863.31k 6.0 TiB 6.38 44 TiB (this pool
>>>> have the all VM disk)
>>>>
>>>> ---------------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>> ceph osd map vm.pool vm.pool.object
>>>> osdmap e14319 pool 'vm.pool' (2) object 'vm.pool.object' -> pg 2.196f68d5 (2.d5) -> up ([2,4], p2)
>>>> acting ([2,4], p2)
>>>>
>>>> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> pveversion -v
>>>> proxmox-ve: 6.4-1 (running kernel: 5.4.143-1-pve)
>>>> pve-manager: 6.4-13 (running version: 6.4-13/9f411e79)
>>>> pve-kernel-helper: 6.4-8
>>>> pve-kernel-5.4: 6.4-7
>>>> pve-kernel-5.4.143-1-pve: 5.4.143-1
>>>> pve-kernel-5.4.106-1-pve: 5.4.106-1
>>>> ceph: 15.2.15-pve1~bpo10
>>>> ceph-fuse: 15.2.15-pve1~bpo10
>>>> corosync: 3.1.2-pve1
>>>> criu: 3.11-3
>>>> glusterfs-client: 5.5-3
>>>> ifupdown: residual config
>>>> ifupdown2: 3.0.0-1+pve4~bpo10
>>>> ksm-control-daemon: 1.3-1
>>>> libjs-extjs: 6.0.1-10
>>>> libknet1: 1.22-pve1~bpo10+1
>>>> libproxmox-acme-perl: 1.1.0
>>>> libproxmox-backup-qemu0: 1.1.0-1
>>>> libpve-access-control: 6.4-3
>>>> libpve-apiclient-perl: 3.1-3
>>>> libpve-common-perl: 6.4-4
>>>> libpve-guest-common-perl: 3.1-5
>>>> libpve-http-server-perl: 3.2-3
>>>> libpve-storage-perl: 6.4-1
>>>> libqb0: 1.0.5-1
>>>> libspice-server1: 0.14.2-4~pve6+1
>>>> lvm2: 2.03.02-pve4
>>>> lxc-pve: 4.0.6-2
>>>> lxcfs: 4.0.6-pve1
>>>> novnc-pve: 1.1.0-1
>>>> proxmox-backup-client: 1.1.13-2
>>>> proxmox-mini-journalreader: 1.1-1
>>>> proxmox-widget-toolkit: 2.6-1
>>>> pve-cluster: 6.4-1
>>>> pve-container: 3.3-6
>>>> pve-docs: 6.4-2
>>>> pve-edk2-firmware: 2.20200531-1
>>>> pve-firewall: 4.1-4
>>>> pve-firmware: 3.3-2
>>>> pve-ha-manager: 3.1-1
>>>> pve-i18n: 2.3-1
>>>> pve-qemu-kvm: 5.2.0-6
>>>> pve-xtermjs: 4.7.0-3
>>>> qemu-server: 6.4-2
>>>> smartmontools: 7.2-pve2
>>>> spiceterm: 3.1-1
>>>> vncterm: 1.6-2
>>>> zfsutils-linux: 2.0.6-pve1~bpo10+1
>>>>
>>>> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>> And now my problem:
>>>>
>>>> For all VM I have one pool for VM disks
>>>>
>>>> When node/host pve-3111 is shutdown in many of other nodes/hosts pve-3107, pve-3105 VM not
>>>> shutdown but not available in network.
>>>>
>>>> After the node/host is up Ceph back to HEALTH_OK and the all VM back to access in Network (without
>>>> reboot).
>>>>
>>>> Can some one to suggest me what I can to check in Ceph ?
>>>>
>>>> Thanks.
>>>>
>
--
-------------------------
С уважением
Сергей Цаболов,
Системный администратор
ООО "Т8"
Тел.: +74992716161,
Моб: +79850334875
tsabolov at t8.ru
ООО «Т8», 107076, г. Москва, Краснобогатырская ул., д. 44, стр.1
www.t8.ru
More information about the pve-user
mailing list