[PVE-User] Proxmox CEPH 6 servers failures!

Gilberto Nunes gilberto.nunes32 at gmail.com
Fri Oct 5 15:08:24 CEST 2018 

Ok! Now I get it!
pvecm show me
pve-ceph01:/etc/pve# pvecm status
Quorum information
------------------
Date:             Fri Oct  5 10:04:57 2018
Quorum provider:  corosync_votequorum
Nodes:            6
Node ID:          0x00000001
Ring ID:          1/32764
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   6
Highest expected: 6
Total votes:      6
Quorum:           4
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.10.10.100 (local)
0x00000002          1 10.10.10.110
0x00000003          1 10.10.10.120
0x00000004          1 10.10.10.130
0x00000005          1 10.10.10.140
0x00000006          1 10.10.10.150

*Quorum:          4*
So I need 4 server online, at least!
Now when I loose 3 of 6, I remain, of course, just with 3 and not with 4,
which is required...
I will request new server to make quorum. Thanks for clarify this situation!
---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em sex, 5 de out de 2018 às 09:53, Gilberto Nunes <
gilberto.nunes32 at gmail.com> escreveu:

> Folks...
>
> I CEPH servers are in the same network: 10.10.10.0/24...
> There is a optic channel between the builds: buildA and buildB, just to
> identified!
> When I create the cluster in first time, 3 servers going down in buildB,
> and the remain ceph servers continued to worked properly...
> I do not understand why now this cant happens anymore!
> Sorry if I sound like a newbie! I still learn about it!
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
>
>
>
>
>
> Em sex, 5 de out de 2018 às 09:44, Marcus Haarmann <
> marcus.haarmann at midoco.de> escreveu:
>
>> Gilberto,
>>
>> the underlying problem is a ceph problem and not related to VMs or
>> Proxmox.
>> The ceph system requires a mayority of monitor nodes to be active.
>> Your setup seems to have 3 mon nodes, which results in a loss of quorum
>> when two of these servers are gone.
>> Check "ceph -s" on each side if you see any reaction of ceph.
>> If not, probably not enough mons are present.
>>
>> Also, when one side is down you should see a non-presence of some OSD
>> instances.
>> In this case, ceph might be up but your VMs which are spread over the OSD
>> disks,
>> might block because of the non-accessibility of the primary storage.
>> The distribution of data over the OSD instances is steered by the crush
>> map.
>> You should make sure to have enough copies configured and the crush map
>> set up in a way
>> that on each side of your cluster is minimum one copy.
>> In case the crush map is mis-configured, all copies of your data may be
>> on the wrong side,
>> esulting in proxmox not being able to access the VM data.
>>
>> Marcus Haarmann
>>
>>
>> Von: "Gilberto Nunes" <gilberto.nunes32 at gmail.com>
>> An: "pve-user" <pve-user at pve.proxmox.com>
>> Gesendet: Freitag, 5. Oktober 2018 14:31:20
>> Betreff: Re: [PVE-User] Proxmox CEPH 6 servers failures!
>>
>> Nice.. Perhaps if I create a VM in Proxmox01 and Proxmox02, and join this
>> VM into Cluster Ceph, can I solve to quorum problem?
>> ---
>> Gilberto Nunes Ferreira
>>
>> (47) 3025-5907
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>> Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>> Em sex, 5 de out de 2018 às 09:23, dorsy <dorsyka at yahoo.com> escreveu:
>>
>> > Your question has already been answered. You need majority to have
>> quorum.
>> >
>> > On 2018. 10. 05. 14:10, Gilberto Nunes wrote:
>> > > Hi
>> > > Perhaps this can help:
>> > >
>> > > https://imageshack.com/a/img921/6208/X7ha8R.png
>> > >
>> > > I was thing about it, and perhaps if I deploy a VM in both side, with
>> > > Proxmox and add this VM to the CEPH cluster, maybe this can help!
>> > >
>> > > thanks
>> > > ---
>> > > Gilberto Nunes Ferreira
>> > >
>> > > (47) 3025-5907
>> > > (47) 99676-7530 - Whatsapp / Telegram
>> > >
>> > > Skype: gilberto.nunes36
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Em sex, 5 de out de 2018 às 03:55, Alexandre DERUMIER <
>> > aderumier at odiso.com>
>> > > escreveu:
>> > >
>> > >> Hi,
>> > >>
>> > >> Can you resend your schema, because it's impossible to read.
>> > >>
>> > >>
>> > >> but you need to have to quorum on monitor to have the cluster
>> working.
>> > >>
>> > >>
>> > >> ----- Mail original -----
>> > >> De: "Gilberto Nunes" <gilberto.nunes32 at gmail.com>
>> > >> À: "proxmoxve" <pve-user at pve.proxmox.com>
>> > >> Envoyé: Jeudi 4 Octobre 2018 22:05:16
>> > >> Objet: [PVE-User] Proxmox CEPH 6 servers failures!
>> > >>
>> > >> Hi there
>> > >>
>> > >> I have something like this:
>> > >>
>> > >> CEPH01 ----|
>> > >> |----- CEPH04
>> > >> |
>> > >> |
>> > >> CEPH02
>> ----|-----------------------------------------------------|----
>> > >> CEPH05
>> > >> | Optic Fiber
>> > >> |
>> > >> CEPH03 ----|
>> > >> |--- CEPH06
>> > >>
>> > >> Sometime, when Optic Fiber not work, and just CEPH01, CEPH02 and
>> CEPH03
>> > >> remains, the entire cluster fail!
>> > >> I find out the cause!
>> > >>
>> > >> ceph.conf
>> > >>
>> > >> [global] auth client required = cephx auth cluster required = cephx
>> auth
>> > >> service required = cephx cluster network = 10.10.10.0/24 fsid =
>> > >> e67534b4-0a66-48db-ad6f-aa0868e962d8 keyring =
>> > >> /etc/pve/priv/$cluster.$name.keyring mon allow pool delete = true
>> osd
>> > >> journal size = 5120 osd pool default min size = 2 osd pool default
>> size
>> > =
>> > >> 3
>> > >> public network = 10.10.10.0/24 [osd] keyring =
>> > >> /var/lib/ceph/osd/ceph-$id/keyring [mon.pve-ceph01] host =
>> pve-ceph01
>> > mon
>> > >> addr = 10.10.10.100:6789 mon osd allow primary affinity = true
>> > >> [mon.pve-ceph02] host = pve-ceph02 mon addr = 10.10.10.110:6789 mon
>> osd
>> > >> allow primary affinity = true [mon.pve-ceph03] host = pve-ceph03 mon
>> > addr
>> > >> =
>> > >> 10.10.10.120:6789 mon osd allow primary affinity = true
>> > [mon.pve-ceph04]
>> > >> host = pve-ceph04 mon addr = 10.10.10.130:6789 mon osd allow
>> primary
>> > >> affinity = true [mon.pve-ceph05] host = pve-ceph05 mon addr =
>> > >> 10.10.10.140:6789 mon osd allow primary affinity = true
>> > [mon.pve-ceph06]
>> > >> host = pve-ceph06 mon addr = 10.10.10.150:6789 mon osd allow
>> primary
>> > >> affinity = true
>> > >>
>> > >> Any help will be welcome!
>> > >>
>> > >> ---
>> > >> Gilberto Nunes Ferreira
>> > >>
>> > >> (47) 3025-5907
>> > >> (47) 99676-7530 - Whatsapp / Telegram
>> > >>
>> > >> Skype: gilberto.nunes36
>> > >> _______________________________________________
>> > >> pve-user mailing list
>> > >> pve-user at pve.proxmox.com
>> > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>> > >>
>> > >> _______________________________________________
>> > >> pve-user mailing list
>> > >> pve-user at pve.proxmox.com
>> > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>> > >>
>> > > _______________________________________________
>> > > pve-user mailing list
>> > > pve-user at pve.proxmox.com
>> > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>> > _______________________________________________
>> > pve-user mailing list
>> > pve-user at pve.proxmox.com
>> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>> >
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>

More information about the pve-user mailing list