[PVE-User] Proxmox (2 node) cluster questions

Sat Aug 29 15:31:00 CEST 2020

Hi,

the main problem with 2 nodes cluster, is that if 1 node is done, you lost quorum,
so /etc/pve is read only.

if that occur, you can manually said to proxmox that you want only 1 node in the quorum,
with "pvecm expected 1" command. then you'll be able to write again in  /etc/pve.
(But do it only when you are sure that the other node is down)

>>So i hope you don't mind me asking a couple of questions since i am tempted
>>to try again:

>>   -  Can a proxmox cluster (with failover possibility) work with local
>>   storage. (Or do i need distributed storage from a NAS / NFS via ceph?)

I'm  not sure, but maybe with zfs it's possible. (but the replication is async, so you'll lost data since the last sync)

>>   - Can i use failover possiblity in a 2 nodes cluster?
manually yes. (but not with HA).
you can use "pvecm expected 1",  then on node1 "mv /etc/pve/nodes/node2/qemu-server/* /etc/pve/nodes/node1/qemu-server" to move vm config.
then if the storage is available on node1 (shared storage or maybe zfs local), you'll be able to start vm

>>   - Can i use vm migration in a 2 nodes cluster?
yes sure

>>   - Does it matter if i have the storage 'mounted' in Datacenter rather
>>   than on the server in a directory (gui wise). (Datacenter > storage >
>>   'mounts')
This is the same, but for network storage(nfs,cifs), it's better to use datacenter option, as it's monitoring server, and if a network timeout occur,
the pvestatd daemon will not hang when try to get stats

>>   - Is it better to rename the mounts to vmdata rather than vmdata <number>

for the failover, you only need to have same "storage name" define for each node.
so yes, local mountpoint should be same on each node, as you can define 1 storage name only once at datacenter level.

----- Mail original -----
De: "Rutger Verhoeven" <rutger.verhoeven at gmail.com>
À: "proxmoxve" <pve-user at pve.proxmox.com>
Envoyé: Vendredi 28 Août 2020 18:41:45
Objet: [PVE-User] Proxmox (2 node) cluster questions

Hello all, 

Awhile ago i attempted to join 2 nodes into a cluster. I did this on 
proxmox 6.0 and had the following setup: 
- 2x 2TB SSD (1x 250GB partition to install proxmox on) (1x 1750GB 
partition to store VM data on) I mounted these in /var/lib/vmdata5 on 
server5 (and vmdata4 on server4)). 
- 2x 5TB 5400RPM for extra storage. This is called vmstorage5 or vmstorage4 
and is also mounted in /var/lib 

underneath is an example of my storage.cfg: 
---- 

*cat /etc/pve/storage.cfgdir: local path /var/lib/vz content 
iso,vztmpl,backuplvmthin: local-lvm thinpool data vgname pve content 
rootdir,imagesdir: vmdata4 path /var/lib/vmdata4 content 
vztmpl,iso,snippets,backup,images,rootdir maxfiles 2 nodes server4 shared 
1dir: vmstorage4 path /var/lib/vmstorage4 content 
vztmpl,iso,snippets,backup,rootdir,images maxfiles 2 nodes server4 shared 
1nfs: VM export /VM path /mnt/pve/VM server qnap.domain.local content 
rootdir,vztmpl,iso,snippets,backup,images maxfiles 1 nodes server4* 
* options vers=4* 
---- 

Also output of 'df -h': 
------ 

*df -hFilesystem Size Used Avail Use% Mounted onudev 
48G 0 48G 0% /devtmpfs 
9.5G 9.7M 9.5G 1% /run/dev/mapper/pve-root 30G 2.9G 
25G 11% /tmpfs 48G 43M 48G 1% /dev/shmtmpfs 
5.0M 0 5.0M 0% /run/locktmpfs 
48G 0 48G 0% /sys/fs/cgroup/dev/sda2 
511M 312K 511M 1% /boot/efi/dev/mapper/vg_vmdata4-lvv4 1.7T 2.1G 
1.6T 1% /var/lib/vmdata4/dev/mapper/vg_storage4-lvs4 4.6T 1.8T 2.6T 
42% /var/lib/vmstorage4/dev/fuse 30M 20K 30M 1% 
/etc/pvetmpfs 9.5G 0 9.5G 0% /run/user/0* 
*nas.domain.local:/VM 7.1T 2.8T 4.3T 40% /mnt/pve/VM* 
---- 

These machines are connected with the following network setup: 
----- 

*cat /etc/network/interfacesauto lo* 
*iface lo inet loopback* 

*### !!! do not touch ilo nic !!! #####auto eno1#iface eno1 inet 
static######################################## this nic is used for proxmox 
clustering ###auto eno2iface eno2 inet 
manual###############################################iface ens1f0 inet 
manualiface ens1f1 inet manualauto bond0iface bond0 inet manual bond-slaves 
ens1f0 ens1f1 bond-miimon 100 bond-mode 802.3ad bond-xmit-hash-policy 
layer2+3auto vmbr0iface vmbr0 inet static address 192.168.xx.xx netmask 
255.255.252.0 gateway 192.168.xx.xx bridge_ports bond0 bridge_stp off 
bridge_fd 0* 
--- 
The servers both have dual 10GB/s SFP+ and are connected via a unifi 10GBPS 
SFP+ switch. 
I haven't been able to create a seperate vlan yet for management interface 
/ seperate cluster purposes. 

For some reason it backfired, made the files in /etc/pve/qemu-server 
readonly and i could not change anything in this files (also wasnt able via 
the gui). I ended up reinstalling both nodes and solved everything with 
manual migration (via bash script). 

Since then i'm not so eager to try and join these servers in a cluster 
anymore because server5 is a production machine with all kinds of 
applications on it. 

sidenote: im using ansible to create VM's on these machines, but i must 
admit that mostly i work via the gui. 

*My goals:* 
- One host to manage them all (also for ansible) 
- easy vm migration between the servers. 

In the Netherlands we have a saying: "the blood crawls where it cannot go" . 

So i hope you don't mind me asking a couple of questions since i am tempted 
to try again: 

- Can a proxmox cluster (with failover possibility) work with local 
storage. (Or do i need distributed storage from a NAS / NFS via ceph?) 
- Can i use failover possiblity in a 2 nodes cluster? 
- Can i use vm migration in a 2 nodes cluster? 
- Does it matter if i have the storage 'mounted' in Datacenter rather 
than on the server in a directory (gui wise). (Datacenter > storage > 
'mounts') 
- Is it better to rename the mounts to vmdata rather than vmdata <number> 

Any tips regarding this are appreciated. Thank you all in advance. 
_______________________________________________ 
pve-user mailing list 
pve-user at lists.proxmox.com 
https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user 