[PVE-User] Cluster network via directly connected interfaces?

[Uwe Sauter]

And one other thing.

I don't think that multicast traffic is possible in this solution so you need to configure corosync to do unicast UDP. 
Make this change after creating the cluster on the first node but before joining any other nodes. Easiest point in time 
for that change.

/etc/pve/corosync.conf

totem {
[…]
   config_version: +=1 ######### means: increment by one fore every change
   transport: udpu
}

And, as you already mentioned, having such a setup won't scale. Three nodes is the only size where this is sensible to do.


Do you plan to use Ceph?





Am 22.11.18 um 19:55 schrieb Frank Thommen:
> Good point.  Thanks a lot
> frank
> 
> 
> On 11/22/2018 07:51 PM, Uwe Sauter wrote:
>> FYI:
>>
>> I had such a thing working. What you need to keep in mind is that you should configure both interfaces per host on the 
>> same (software) bridge and keep STP on… that way when you loose the link from node A to node B the traffic will be 
>> going through node C.
>>
>> +--------------------+
>> |                    |
>> | Node A   br0       |
>> |         /   \      |
>> |       eth0   eth1  |
>> +------/-----------\-+
>>        /             \
>> +----/------+  +-----\----+
>> |  eth1     |  |    eth0  |
>> |  /        |  |       \  |
>> | br0--eth0-----eth1--br0 |
>> |   Node B  |  |  Node C  |
>> +-----------+  +----------+
>>
>>
>>
>>
>> Am 22.11.18 um 19:42 schrieb Frank Thommen:
>>> What I /really/ meant was "but the throughput would /not/ be higher when using a switch"...
>>>
>>>
>>> On 11/22/2018 07:37 PM, Frank Thommen wrote:
>>>> But the throughput would be higher when using a switch, would it? It's still just 1Gbit
>>>>
>>>> frank
>>>>
>>>>
>>>> On 11/22/2018 07:34 PM, Mark Schouten wrote:
>>>>> Other than limited throughput, I can’t think of a problem. But limited throughput might cause unforeseen situations.
>>>>>
>>>>> Mark Schouten
>>>>>
>>>>>> Op 22 nov. 2018 om 19:30 heeft Frank Thommen <frank.thommen at uni-heidelberg.de> het volgende geschreven:
>>>>>>
>>>>>> Please excuse, if this is too basic, but after reading https://pve.proxmox.com/wiki/Cluster_Manager I wondered, if 
>>>>>> the cluster/corosync network could be built by directly connected network interfaces.  I.e not like this:
>>>>>>
>>>>>> +-------+
>>>>>> | pve01 |----------+
>>>>>> +-------+          |
>>>>>>                     |
>>>>>> +-------+     +----------------+
>>>>>> | pve02 |-----| network switch |
>>>>>> +-------+     +----------------+
>>>>>>                     |
>>>>>> +-------+          |
>>>>>> | pve03 |----------+
>>>>>> +-------+
>>>>>>
>>>>>>
>>>>>> but like this:
>>>>>>
>>>>>> +-------+
>>>>>> | pve01 |---+
>>>>>> +-------+   |
>>>>>>      |       |
>>>>>> +-------+   |
>>>>>> | pve02 |   |
>>>>>> +-------+   |
>>>>>>      |       |
>>>>>> +-------+   |
>>>>>> | pve03 |---+
>>>>>> +-------+
>>>>>>
>>>>>> (all connections 1Gbit, there are currently not plans to extend over three nodes)
>>>>>>
>>>>>> I can't see any drawback in that solution.  It would remove one layer of hardware dependency and potential spof 
>>>>>> (the switch).  If we don't trust the interfaces, we might be able to configure a second network with the three 
>>>>>> remaining interfaces.
>>>>>>
>>>>>> Is such a "direct-connection" topology feasible?  Recommended? Strictly not recommended?
>>>>>>
>>>>>> I am currently just planning and thinking and there is no cluster (or even a PROXMOX server) in place.
>>>>>>
>>>>>> Cheers
>>>>>> frank
>>>>>> _______________________________________________
>>>>>> pve-user mailing list
>>>>>> pve-user at pve.proxmox.com
>>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>>> _______________________________________________
>>>>> pve-user mailing list
>>>>> pve-user at pve.proxmox.com
>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>>>>
>>>>
>>>
>> _______________________________________________
>> pve-user mailing list
>> pve-user at pve.proxmox.com
>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user