[PVE-User] Ceph doesn't show all available space !

[Ronny Aasen]

On 25.08.2018 00:48, Gilberto Nunes wrote:
> Ok guys
>
> I have success deploy a 3 node ceph server.
> In this server I have:
>
> Node1
>
> OSD.0 = 4 TB
> OSD.1 = 4 TB
> OSD.2 = 4 TB
> OSD.3 = 4 TB
>
> Node2
>
> OSD.4 = 3 TB
> OSD.5 = 3 TB
> OSD.6 = 3 TB
> OSD.7 = 3 TB
>
> Node3
>
> OSD.8 = 3 TB
> OSD.9 = 3 TB
> OSD.10 = 2 TB
>
> In the Ceph cluster, the total among of data storage is:
> 11.17 GiB of 32.75 TiB
>
> I create a pool, with default value.
> But when I attach the ceph cluster storage to my proxmox, I just see 10 TB!
> Why I can see the 32 TB or at least 21.58 ( 32.75 - 11.17) ????
>
> So odd!
>

ceph does show you the RAW total space available, since you can 
configure different pools. ie 3x replication

Since you have 3 nodes, your data will be replicated on all 3 nodes with 
the default size=3 pools.
so you should in theory only be able to use 3+3+2= 8TB  TB of data since 
once those are full you can not have 3 copies of objects any more.  IOW 
you get the space of the smalles node, and the rest of the space is 
"wasted" once you add a 4th node, or more disks on the small one.  ceph 
can use the remaining storage on the larger nodes as well.

for small clusters, you should try to make the storage homogeneous, 
since it makes balancing easier..

try to make the machines more even, if you want more usable space.

kind regards
Ronny Aasen