[PVE-User] Ceph or Gluster

[Lindsay Mathieson]

Also, what sort of iowait percentages are you seeing?

On 23/04/2016 7:50 AM, Brian :: wrote:
> Hi Lindsay,
>
> With NVME journals on a 3 node 4 OSD cluster if I do a quick dd of a
> 1GB file on a VM I can see 2.34Gbps on the storage network straight
> away so if I was only using 1Gbps here the network would be a
> bottlekneck. If I perform the same in 2 VMs traffic hits 4.19Gbps on
> the storage network.
>
> The throughput in the VM is 1073741824 bytes (1.1 GB) copied, 3.43556
> s, 313 MB/s (R=3)
>
> Would be very interested in hearing more about your gluster setup.. I
> don't know anything about it - how many nodes are involved?
>
>
>
>
>
> On Fri, Apr 22, 2016 at 3:02 PM, Lindsay Mathieson
> <lindsay.mathieson at gmail.com> wrote:
>> On 22/04/2016 11:31 PM, Brian :: wrote:
>>> 10Gbps or faster at a minimum or you will have pain. Even using 4
>>> nodes with 4 spinner disks in each node and you will be maxing out
>>> 1Gbps network.
>>
>>
>> Can't say I saw that on our cluster.
>>
>> - 3 Nodes
>> - 3 OSD's per Node
>> - SSD journals for each OSD.
>> - 2*1G Eth in LACP Bond dedicated to Ceph
>> - 1G Admin Net
>> - Size = 3 (Replica 3)
>>
>> Never came close to maxing out 1 1Gbps connection, write throughput and IOPS
>> are terrible. Read is pretty good though.
>>
>> Currently trialing Gluster 3.7.11 on ZFS Bricks, Replica 3 also. Triple the
>> throughput ans IOPS I was getting with Ceph, maxes out the 2*1G connection,
>> also seems to deal with VM I/O spikes better too, not letting other VM's be
>> stalled.
>>
>> Not convinced its as robust as Ceph yet, give it a few more weeks. It does
>> cope very well with failover and brick heals (using 64MB shards).
>>
>> --
>> Lindsay Mathieson
>>
>>
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user


-- 
Lindsay Mathieson