[PVE-User] Proxmox CEPH 6 servers failures!
Gilberto Nunes
gilberto.nunes32 at gmail.com
Fri Oct 5 15:37:31 CEST 2018
And what if the same hardware that have Proxmox with running VM make a part
of ceph cluster?? it could work?
---
Gilberto Nunes Ferreira
(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram
Skype: gilberto.nunes36
Em sex, 5 de out de 2018 às 10:08, Gilberto Nunes <
gilberto.nunes32 at gmail.com> escreveu:
> Ok! Now I get it!
> pvecm show me
> pve-ceph01:/etc/pve# pvecm status
> Quorum information
> ------------------
> Date: Fri Oct 5 10:04:57 2018
> Quorum provider: corosync_votequorum
> Nodes: 6
> Node ID: 0x00000001
> Ring ID: 1/32764
> Quorate: Yes
>
> Votequorum information
> ----------------------
> Expected votes: 6
> Highest expected: 6
> Total votes: 6
> Quorum: 4
> Flags: Quorate
>
> Membership information
> ----------------------
> Nodeid Votes Name
> 0x00000001 1 10.10.10.100 (local)
> 0x00000002 1 10.10.10.110
> 0x00000003 1 10.10.10.120
> 0x00000004 1 10.10.10.130
> 0x00000005 1 10.10.10.140
> 0x00000006 1 10.10.10.150
>
> *Quorum: 4*
> So I need 4 server online, at least!
> Now when I loose 3 of 6, I remain, of course, just with 3 and not with 4,
> which is required...
> I will request new server to make quorum. Thanks for clarify this
> situation!
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
>
>
>
>
>
> Em sex, 5 de out de 2018 às 09:53, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> escreveu:
>
>> Folks...
>>
>> I CEPH servers are in the same network: 10.10.10.0/24...
>> There is a optic channel between the builds: buildA and buildB, just to
>> identified!
>> When I create the cluster in first time, 3 servers going down in buildB,
>> and the remain ceph servers continued to worked properly...
>> I do not understand why now this cant happens anymore!
>> Sorry if I sound like a newbie! I still learn about it!
>> ---
>> Gilberto Nunes Ferreira
>>
>> (47) 3025-5907
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>> Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>> Em sex, 5 de out de 2018 às 09:44, Marcus Haarmann <
>> marcus.haarmann at midoco.de> escreveu:
>>
>>> Gilberto,
>>>
>>> the underlying problem is a ceph problem and not related to VMs or
>>> Proxmox.
>>> The ceph system requires a mayority of monitor nodes to be active.
>>> Your setup seems to have 3 mon nodes, which results in a loss of quorum
>>> when two of these servers are gone.
>>> Check "ceph -s" on each side if you see any reaction of ceph.
>>> If not, probably not enough mons are present.
>>>
>>> Also, when one side is down you should see a non-presence of some OSD
>>> instances.
>>> In this case, ceph might be up but your VMs which are spread over the
>>> OSD disks,
>>> might block because of the non-accessibility of the primary storage.
>>> The distribution of data over the OSD instances is steered by the crush
>>> map.
>>> You should make sure to have enough copies configured and the crush map
>>> set up in a way
>>> that on each side of your cluster is minimum one copy.
>>> In case the crush map is mis-configured, all copies of your data may be
>>> on the wrong side,
>>> esulting in proxmox not being able to access the VM data.
>>>
>>> Marcus Haarmann
>>>
>>>
>>> Von: "Gilberto Nunes" <gilberto.nunes32 at gmail.com>
>>> An: "pve-user" <pve-user at pve.proxmox.com>
>>> Gesendet: Freitag, 5. Oktober 2018 14:31:20
>>> Betreff: Re: [PVE-User] Proxmox CEPH 6 servers failures!
>>>
>>> Nice.. Perhaps if I create a VM in Proxmox01 and Proxmox02, and join
>>> this
>>> VM into Cluster Ceph, can I solve to quorum problem?
>>> ---
>>> Gilberto Nunes Ferreira
>>>
>>> (47) 3025-5907
>>> (47) 99676-7530 - Whatsapp / Telegram
>>>
>>> Skype: gilberto.nunes36
>>>
>>>
>>>
>>>
>>>
>>> Em sex, 5 de out de 2018 às 09:23, dorsy <dorsyka at yahoo.com> escreveu:
>>>
>>> > Your question has already been answered. You need majority to have
>>> quorum.
>>> >
>>> > On 2018. 10. 05. 14:10, Gilberto Nunes wrote:
>>> > > Hi
>>> > > Perhaps this can help:
>>> > >
>>> > > https://imageshack.com/a/img921/6208/X7ha8R.png
>>> > >
>>> > > I was thing about it, and perhaps if I deploy a VM in both side,
>>> with
>>> > > Proxmox and add this VM to the CEPH cluster, maybe this can help!
>>> > >
>>> > > thanks
>>> > > ---
>>> > > Gilberto Nunes Ferreira
>>> > >
>>> > > (47) 3025-5907
>>> > > (47) 99676-7530 - Whatsapp / Telegram
>>> > >
>>> > > Skype: gilberto.nunes36
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > Em sex, 5 de out de 2018 às 03:55, Alexandre DERUMIER <
>>> > aderumier at odiso.com>
>>> > > escreveu:
>>> > >
>>> > >> Hi,
>>> > >>
>>> > >> Can you resend your schema, because it's impossible to read.
>>> > >>
>>> > >>
>>> > >> but you need to have to quorum on monitor to have the cluster
>>> working.
>>> > >>
>>> > >>
>>> > >> ----- Mail original -----
>>> > >> De: "Gilberto Nunes" <gilberto.nunes32 at gmail.com>
>>> > >> À: "proxmoxve" <pve-user at pve.proxmox.com>
>>> > >> Envoyé: Jeudi 4 Octobre 2018 22:05:16
>>> > >> Objet: [PVE-User] Proxmox CEPH 6 servers failures!
>>> > >>
>>> > >> Hi there
>>> > >>
>>> > >> I have something like this:
>>> > >>
>>> > >> CEPH01 ----|
>>> > >> |----- CEPH04
>>> > >> |
>>> > >> |
>>> > >> CEPH02
>>> ----|-----------------------------------------------------|----
>>> > >> CEPH05
>>> > >> | Optic Fiber
>>> > >> |
>>> > >> CEPH03 ----|
>>> > >> |--- CEPH06
>>> > >>
>>> > >> Sometime, when Optic Fiber not work, and just CEPH01, CEPH02 and
>>> CEPH03
>>> > >> remains, the entire cluster fail!
>>> > >> I find out the cause!
>>> > >>
>>> > >> ceph.conf
>>> > >>
>>> > >> [global] auth client required = cephx auth cluster required = cephx
>>> auth
>>> > >> service required = cephx cluster network = 10.10.10.0/24 fsid =
>>> > >> e67534b4-0a66-48db-ad6f-aa0868e962d8 keyring =
>>> > >> /etc/pve/priv/$cluster.$name.keyring mon allow pool delete = true
>>> osd
>>> > >> journal size = 5120 osd pool default min size = 2 osd pool default
>>> size
>>> > =
>>> > >> 3
>>> > >> public network = 10.10.10.0/24 [osd] keyring =
>>> > >> /var/lib/ceph/osd/ceph-$id/keyring [mon.pve-ceph01] host =
>>> pve-ceph01
>>> > mon
>>> > >> addr = 10.10.10.100:6789 mon osd allow primary affinity = true
>>> > >> [mon.pve-ceph02] host = pve-ceph02 mon addr = 10.10.10.110:6789
>>> mon osd
>>> > >> allow primary affinity = true [mon.pve-ceph03] host = pve-ceph03
>>> mon
>>> > addr
>>> > >> =
>>> > >> 10.10.10.120:6789 mon osd allow primary affinity = true
>>> > [mon.pve-ceph04]
>>> > >> host = pve-ceph04 mon addr = 10.10.10.130:6789 mon osd allow
>>> primary
>>> > >> affinity = true [mon.pve-ceph05] host = pve-ceph05 mon addr =
>>> > >> 10.10.10.140:6789 mon osd allow primary affinity = true
>>> > [mon.pve-ceph06]
>>> > >> host = pve-ceph06 mon addr = 10.10.10.150:6789 mon osd allow
>>> primary
>>> > >> affinity = true
>>> > >>
>>> > >> Any help will be welcome!
>>> > >>
>>> > >> ---
>>> > >> Gilberto Nunes Ferreira
>>> > >>
>>> > >> (47) 3025-5907
>>> > >> (47) 99676-7530 - Whatsapp / Telegram
>>> > >>
>>> > >> Skype: gilberto.nunes36
>>> > >> _______________________________________________
>>> > >> pve-user mailing list
>>> > >> pve-user at pve.proxmox.com
>>> > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>> > >>
>>> > >> _______________________________________________
>>> > >> pve-user mailing list
>>> > >> pve-user at pve.proxmox.com
>>> > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>> > >>
>>> > > _______________________________________________
>>> > > pve-user mailing list
>>> > > pve-user at pve.proxmox.com
>>> > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>> > _______________________________________________
>>> > pve-user mailing list
>>> > pve-user at pve.proxmox.com
>>> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>> >
>>> _______________________________________________
>>> pve-user mailing list
>>> pve-user at pve.proxmox.com
>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>> _______________________________________________
>>> pve-user mailing list
>>> pve-user at pve.proxmox.com
>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>
>>
More information about the pve-user
mailing list