From w.bumiller at proxmox.com Mon Dec 3 13:38:36 2018 From: w.bumiller at proxmox.com (Wolfgang Bumiller) Date: Mon, 3 Dec 2018 13:38:36 +0100 (CET) Subject: [PVE-User] lxc hang situation In-Reply-To: <20181130155150.zpwsfn3v7mz5o2yz@daruma.hachimitsu.nl> References: <20181130155150.zpwsfn3v7mz5o2yz@daruma.hachimitsu.nl> Message-ID: <1608038347.86.1543840716252@webmail.proxmox.com> > On November 30, 2018 at 4:51 PM Stephan Leemburg wrote: > (...) > After some more searching, we found with > > grep copy_net_ns /proc/[0-9]*/stack > > that there where 2 more processes also blocked on copy_net_ns. These where > two ionclean processes in other containers. Killing them (with -9) showed > that restarted ionclean processes immediatly blocked again on copy_net_ns. Next time, can you post the full /proc/$pid/stack of all tasks hanging in that function? I was never able to reproduce this, and never got the info I needed. > (...) > it says that this problem should be solved in kernel 4.17. The locking was completely refactored and should work better now. But race conditions are hard to test properly. > (...) > As the kernel is ubuntu based would it be possible to start using the ubuntu > 18.10 kernel which is 4.18 to get around this problem? If you otherwise keep running into this issue that's definitely worth a try... From sir_Misiek1 at o2.pl Tue Dec 4 10:27:42 2018 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Tue, 4 Dec 2018 10:27:42 +0100 Subject: [PVE-User] Container problem Message-ID: root at hayne:~# systemctl start pve-container at 108 Job for pve-container at 108.service failed because the control process exited with error code. See "systemctl status pve-container at 108.service" and "journalctl -xe" for details. root at hayne:~# systemctl status pve-container at 108.service ? pve-container at 108.service - PVE LXC Container: 108 ?? Loaded: loaded (/lib/systemd/system/pve-container at .service; static; vendor preset: enabled) ?? Active: failed (Result: exit-code) since Tue 2018-12-04 10:25:45 CET; 12s ago ???? Docs: man:lxc-start ?????????? man:lxc ?????????? man:pct ? Process: 9268 ExecStart=/usr/bin/lxc-start -n 108 (code=exited, status=1/FAILURE) Dec 04 10:25:44 hayne systemd[1]: Starting PVE LXC Container: 108... Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Control process exited, code=exited status=1 Dec 04 10:25:45 hayne systemd[1]: Failed to start PVE LXC Container: 108. Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Unit entered failed state. Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Failed with result 'exit-code'. From t.lamprecht at proxmox.com Tue Dec 4 10:41:26 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Tue, 4 Dec 2018 10:41:26 +0100 Subject: [PVE-User] Container problem In-Reply-To: References: Message-ID: <6eca3a4a-bbfe-eb58-41d1-e50324cf0018@proxmox.com> On 12/4/18 10:27 AM, lord_Niedzwiedz wrote: > root at hayne:~# systemctl start pve-container at 108 > Job for pve-container at 108.service failed because the control process exited with error code. > See "systemctl status pve-container at 108.service" and "journalctl -xe" for details. > > root at hayne:~# systemctl status pve-container at 108.service > ? pve-container at 108.service - PVE LXC Container: 108 > Loaded: loaded (/lib/systemd/system/pve-container at .service; static; vendor preset: enabled) > Active: failed (Result: exit-code) since Tue 2018-12-04 10:25:45 CET; 12s ago > Docs: man:lxc-start > man:lxc > man:pct > Process: 9268 ExecStart=/usr/bin/lxc-start -n 108 (code=exited, status=1/FAILURE) > > Dec 04 10:25:44 hayne systemd[1]: Starting PVE LXC Container: 108... > Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Control process exited, code=exited status=1 > Dec 04 10:25:45 hayne systemd[1]: Failed to start PVE LXC Container: 108. > Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Unit entered failed state. > Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Failed with result 'exit-code'. > How about at least some minimal context and more telling logs? ^^ # lxc-start -n 108 -l DEBUG -o ct108-start.log optionally add the "-F" flag to start the CT in foreground.. cheers, Thomas From martin at proxmox.com Tue Dec 4 11:24:40 2018 From: martin at proxmox.com (Martin Maurer) Date: Tue, 4 Dec 2018 11:24:40 +0100 Subject: [PVE-User] Proxmox VE 5.3 released! Message-ID: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> Hi all! We are very excited to announce the general availability of Proxmox VE 5.3! Proxmox VE now integrates CephFS, a distributed, POSIX-compliant file system which serves as an interface to the Ceph storage (like the RBD). You can store backupfiles, ISO images, and container templates. CephFS can be created and configured easily with its Metadata server (MDS) in the GUI. We improved disk management and you can now add ZFS raid volumes, LVM, and LVMthin pools as well as additional disks with a traditional file system. The existing ZFS over iSCSI storage plug-in can now access LIO target in the Linux kernel. Other new features are nesting for LXC containers so you can use LXC and LXD inside containers or access NFS or CIFS. If you are adventurous, you can configure PCI passthrough and vGPUs via the GUI. Countless bugfixes and more smaller improvements are listed in the release notes. Release notes https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_5.3 Video tutorial Watch our short introduction video What's new in Proxmox VE 5.3? https://www.proxmox.com/en/training/video-tutorials Download https://www.proxmox.com/en/downloads Alternate ISO download: http://download.proxmox.com/iso/ Source Code https://git.proxmox.com Bugtracker https://bugzilla.proxmox.com FAQ Q: Can I install Proxmox VE 5.3 on top of Debian Stretch? A: Yes, see https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch Q: Can I upgrade Proxmox VE 5.x to 5.3 with apt? A: Yes, just via GUI or via CLI with apt update && apt dist-upgrade Q: Can I upgrade Proxmox VE 4.x to 5.3 with apt dist-upgrade? A: Yes, see https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0. If you Ceph on V4.x please also check https://pve.proxmox.com/wiki/Ceph_Jewel_to_Luminous. Please note, Proxmox VE 4.x is already end of support since June 2018, see Proxmox VE Support Lifecycle https://forum.proxmox.com/threads/proxmox-ve-support-lifecycle.35755/ Many THANKS to our active community for all your feedback, testing, bug reporting and patch submitting! -- Best Regards, Martin Maurer Proxmox VE project leader martin at proxmox.com https://www.proxmox.com From gilberto.nunes32 at gmail.com Tue Dec 4 11:41:56 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Tue, 4 Dec 2018 08:41:56 -0200 Subject: [PVE-User] [pve-devel] Proxmox VE 5.3 released! In-Reply-To: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> References: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> Message-ID: Hi there! I would like to say a big THANK YOU! Proxmox has evolve a lot since previosly versions... Indeed, I knew this wonderful VE in version 1.x and compare it today, seems a lot of improvements.... Thanks a lot. --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em ter, 4 de dez de 2018 ?s 08:24, Martin Maurer escreveu: > Hi all! > > We are very excited to announce the general availability of Proxmox VE 5.3! > > Proxmox VE now integrates CephFS, a distributed, POSIX-compliant file > system which serves as an interface to the Ceph storage (like the RBD). > You can store backupfiles, ISO images, and container templates. CephFS > can be created and configured easily with its Metadata server (MDS) in > the GUI. > > We improved disk management and you can now add ZFS raid volumes, LVM, > and LVMthin pools as well as additional disks with a traditional file > system. The existing ZFS over iSCSI storage plug-in can now access LIO > target in the Linux kernel. Other new features are nesting for LXC > containers so you can use LXC and LXD inside containers or access NFS or > CIFS. If you are adventurous, you can configure PCI passthrough and > vGPUs via the GUI. > > Countless bugfixes and more smaller improvements are listed in the > release notes. > > Release notes > https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_5.3 > > Video tutorial > Watch our short introduction video What's new in Proxmox VE 5.3? > https://www.proxmox.com/en/training/video-tutorials > > Download > https://www.proxmox.com/en/downloads > Alternate ISO download: > http://download.proxmox.com/iso/ > > Source Code > https://git.proxmox.com > > Bugtracker > https://bugzilla.proxmox.com > > FAQ > Q: Can I install Proxmox VE 5.3 on top of Debian Stretch? > A: Yes, see > https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch > > Q: Can I upgrade Proxmox VE 5.x to 5.3 with apt? > A: Yes, just via GUI or via CLI with apt update && apt dist-upgrade > > Q: Can I upgrade Proxmox VE 4.x to 5.3 with apt dist-upgrade? > A: Yes, see https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0. If you > Ceph on V4.x please also check > https://pve.proxmox.com/wiki/Ceph_Jewel_to_Luminous. Please note, > Proxmox VE 4.x is already end of support since June 2018, see Proxmox VE > Support Lifecycle > https://forum.proxmox.com/threads/proxmox-ve-support-lifecycle.35755/ > > Many THANKS to our active community for all your feedback, testing, bug > reporting and patch submitting! > > -- > Best Regards, > > Martin Maurer > Proxmox VE project leader > > martin at proxmox.com > https://www.proxmox.com > > > _______________________________________________ > pve-devel mailing list > pve-devel at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > From sir_Misiek1 at o2.pl Tue Dec 4 12:49:02 2018 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Tue, 4 Dec 2018 12:49:02 +0100 Subject: [PVE-User] Container problem In-Reply-To: <6eca3a4a-bbfe-eb58-41d1-e50324cf0018@proxmox.com> References: <6eca3a4a-bbfe-eb58-41d1-e50324cf0018@proxmox.com> Message-ID: W dniu 04.12.2018 o?10:41, Thomas Lamprecht pisze: > On 12/4/18 10:27 AM, lord_Niedzwiedz wrote: >> root at hayne:~# systemctl start pve-container at 108 >> Job for pve-container at 108.service failed because the control process exited with error code. >> See "systemctl status pve-container at 108.service" and "journalctl -xe" for details. >> >> root at hayne:~# systemctl status pve-container at 108.service >> ? pve-container at 108.service - PVE LXC Container: 108 >> Loaded: loaded (/lib/systemd/system/pve-container at .service; static; vendor preset: enabled) >> Active: failed (Result: exit-code) since Tue 2018-12-04 10:25:45 CET; 12s ago >> Docs: man:lxc-start >> man:lxc >> man:pct >> Process: 9268 ExecStart=/usr/bin/lxc-start -n 108 (code=exited, status=1/FAILURE) >> >> Dec 04 10:25:44 hayne systemd[1]: Starting PVE LXC Container: 108... >> Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Control process exited, code=exited status=1 >> Dec 04 10:25:45 hayne systemd[1]: Failed to start PVE LXC Container: 108. >> Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Unit entered failed state. >> Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Failed with result 'exit-code'. >> > > How about at least some minimal context and more telling logs? ^^ > > # lxc-start -n 108 -l DEBUG -o ct108-start.log > > optionally add the "-F" flag to start the CT in foreground.. > > cheers, > Thomas Ok, I brought the container back. I was spoiled by nfs. But now I can not restore the disk; / root @ hayne: / Working # echo sync; qm importdisk 108 vm-108-disk-0.raw local-lvm; sync sync Configuration file 'nodes / hayne / qemu-server / 108.conf' does not exist From sir_Misiek1 at o2.pl Tue Dec 4 12:58:37 2018 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Tue, 4 Dec 2018 12:58:37 +0100 Subject: [PVE-User] Container problem In-Reply-To: References: <6eca3a4a-bbfe-eb58-41d1-e50324cf0018@proxmox.com> Message-ID: >>> root at hayne:~# systemctl start pve-container at 108 >>> Job for pve-container at 108.service failed because the control process >>> exited with error code. >>> See "systemctl status pve-container at 108.service" and "journalctl >>> -xe" for details. >>> >>> root at hayne:~# systemctl status pve-container at 108.service >>> ? pve-container at 108.service - PVE LXC Container: 108 >>> ??? Loaded: loaded (/lib/systemd/system/pve-container at .service; >>> static; vendor preset: enabled) >>> ??? Active: failed (Result: exit-code) since Tue 2018-12-04 10:25:45 >>> CET; 12s ago >>> ????? Docs: man:lxc-start >>> ??????????? man:lxc >>> ??????????? man:pct >>> ?? Process: 9268 ExecStart=/usr/bin/lxc-start -n 108 (code=exited, >>> status=1/FAILURE) >>> >>> Dec 04 10:25:44 hayne systemd[1]: Starting PVE LXC Container: 108... >>> Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Control >>> process exited, code=exited status=1 >>> Dec 04 10:25:45 hayne systemd[1]: Failed to start PVE LXC Container: >>> 108. >>> Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Unit >>> entered failed state. >>> Dec 04 10:25:45 hayne systemd[1]: pve-container at 108.service: Failed >>> with result 'exit-code'. >>> >> >> How about at least some minimal context and more telling logs? ^^ >> >> # lxc-start -n 108 -l DEBUG -o ct108-start.log >> >> optionally add the "-F" flag to start the CT in foreground.. >> >> cheers, >> Thomas > Ok, I brought the container back. > I was spoiled by nfs. > > But now I can not restore the disk; / > > root @ hayne: / Working # echo sync; qm importdisk 108 > vm-108-disk-0.raw local-lvm; sync > sync > > Configuration file 'nodes / hayne / qemu-server / 108.conf' does not > exist Ok. I see. Sory (no coffe) Ok, I see, this is not a virtual machine but a container. How to import a disk into a container (replace)? From matias.ruggieri at hotmail.com Tue Dec 4 13:48:45 2018 From: matias.ruggieri at hotmail.com (Matias Ruggieri) Date: Tue, 4 Dec 2018 12:48:45 +0000 Subject: [PVE-User] Proxmox VE 5.3 released! In-Reply-To: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> References: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> Message-ID: what? a way to close the year !!!! Many congratulations to the entire Proxmox team! On 12/4/18 7:24 AM, Martin Maurer wrote: > Hi all! > > We are very excited to announce the general availability of Proxmox VE > 5.3! > > Proxmox VE now integrates CephFS, a distributed, POSIX-compliant file > system which serves as an interface to the Ceph storage (like the > RBD). You can store backupfiles, ISO images, and container templates. > CephFS can be created and configured easily with its Metadata server > (MDS) in the GUI. > > We improved disk management and you can now add ZFS raid volumes, LVM, > and LVMthin pools as well as additional disks with a traditional file > system. The existing ZFS over iSCSI storage plug-in can now access LIO > target in the Linux kernel. Other new features are nesting for LXC > containers so you can use LXC and LXD inside containers or access NFS > or CIFS. If you are adventurous, you can configure PCI passthrough and > vGPUs via the GUI. > > Countless bugfixes and more smaller improvements are listed in the > release notes. > > Release notes > https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_5.3 > > Video tutorial > Watch our short introduction video What's new in Proxmox VE 5.3? > https://www.proxmox.com/en/training/video-tutorials > > Download > https://www.proxmox.com/en/downloads > Alternate ISO download: > http://download.proxmox.com/iso/ > > Source Code > https://git.proxmox.com > > Bugtracker > https://bugzilla.proxmox.com > > FAQ > Q: Can I install Proxmox VE 5.3 on top of Debian Stretch? > A: Yes, see > https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch > > Q: Can I upgrade Proxmox VE 5.x to 5.3 with apt? > A: Yes, just via GUI or via CLI with apt update && apt dist-upgrade > > Q: Can I upgrade Proxmox VE 4.x to 5.3 with apt dist-upgrade? > A: Yes, see https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0. If > you Ceph on V4.x please also check > https://pve.proxmox.com/wiki/Ceph_Jewel_to_Luminous. Please note, > Proxmox VE 4.x is already end of support since June 2018, see Proxmox > VE Support Lifecycle > https://forum.proxmox.com/threads/proxmox-ve-support-lifecycle.35755/ > > Many THANKS to our active community for all your feedback, testing, > bug reporting and patch submitting! > From lindsay.mathieson at gmail.com Tue Dec 4 14:59:41 2018 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Tue, 4 Dec 2018 23:59:41 +1000 Subject: [PVE-User] Proxmox VE 5.3 released! In-Reply-To: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> References: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> Message-ID: One server has upgraded clean so far, but the 2nd one wants to remove pve :( apt-get dist-upgrade The following packages were automatically installed and are no longer required: ? apparmor ceph-fuse criu ebtables fonts-font-awesome hdparm ipset libapparmor-perl libboost-iostreams1.55.0 libboost-program-options1.55.0 libboost-random1.55.0 ? libboost-regex1.55.0 libboost-system1.55.0 libboost-thread1.55.0 libgnutlsxx28 libgoogle-perftools4 libicu52 libipset3 libjs-extjs libjson-c3 liblttng-ust-ctl2 liblttng-ust0 ? libnet1 libnetfilter-log1 libprotobuf-c1 libprotobuf10 libpve-access-control libpve-apiclient-perl libpve-guest-common-perl libpve-http-server-perl libpve-storage-perl ? librados2-perl libtcmalloc-minimal4 libtemplate-perl libunwind8 liburcu4 libuuid-perl lxc-pve lxcfs novnc-pve proxmox-widget-toolkit pve-cluster pve-container pve-docs ? pve-edk2-firmware pve-firewall pve-ha-manager pve-i18n pve-manager pve-xtermjs python-ipaddr python-protobuf qemu-server spiceterm uidmap Use 'apt autoremove' to remove them. The following packages will be REMOVED: ? grub-common grub-pc grub-pc-bin grub2-common libnvpair1 libzfs2 proxmox-ve pve-kernel-4.15 pve-kernel-4.15.17-2-pve pve-kernel-4.15.17-3-pve pve-kernel-4.15.18-4-pve ? pve-kernel-4.15.18-5-pve The following NEW packages will be installed: ? libboost-program-options1.62.0 libboost-random1.62.0 libboost-regex1.62.0 libfdt1 libjson-c3 libopus0 The following packages will be upgraded: ? ceph-common libcephfs1 libnvpair1linux libpve-common-perl libpve-storage-perl librados2 libradosstriper1 librbd1 librgw2 libzfs2linux libzpool2linux pve-container ? pve-libspice-server1 pve-manager pve-qemu-kvm python-cephfs python-rados python-rbd qemu-server zfs-initramfs zfs-zed zfsutils-linux I've done a plain apt-get update + upgrade and rebooted. Current versions: pveversion -v proxmox-ve: 5.3-1 (running kernel: 4.15.18-5-pve) pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9) pve-kernel-4.15: 5.2-8 pve-kernel-4.15.18-5-pve: 4.15.18-24 pve-kernel-4.15.18-4-pve: 4.15.18-23 pve-kernel-4.15.17-3-pve: 4.15.17-14 pve-kernel-4.15.17-2-pve: 4.15.17-10 corosync: 2.4.4-pve1 criu: 2.11.1-1~bpo90 glusterfs-client: 3.10.5-1 ksm-control-daemon: 1.2-2 libjs-extjs: 6.0.1-2 libpve-access-control: 5.1-3 libpve-apiclient-perl: 2.0-5 libpve-common-perl: 5.0-40 libpve-guest-common-perl: 2.0-18 libpve-http-server-perl: 2.0-11 libpve-storage-perl: 5.0-29 libqb0: 1.0.3-1~bpo9 lvm2: 2.02.168-pve6 lxc-pve: 3.0.2+pve1-5 lxcfs: 3.0.2-2 novnc-pve: 1.0.0-2 openvswitch-switch: 2.7.0-3 proxmox-widget-toolkit: 1.0-22 pve-cluster: 5.0-31 pve-container: 2.0-27 pve-docs: 5.3-1 pve-firewall: 3.0-16 pve-firmware: 2.0-6 pve-ha-manager: 2.0-5 pve-i18n: 1.0-9 pve-libspice-server1: 0.12.8-3 pve-qemu-kvm: 2.11.2-1 pve-sheepdog: 1.0.2-1 pve-xtermjs: 1.0-5 qemu-server: 5.0-36 smartmontools: 6.5+svn4324-1 spiceterm: 3.0-5 vncterm: 1.5-3 zfsutils-linux: 0.7.11-pve1~bpo1 This is probably my oldest server, started off as Proxmox 3.x On 4/12/2018 8:24 pm, Martin Maurer wrote: > Hi all! > > We are very excited to announce the general availability of Proxmox VE > 5.3! > > Proxmox VE now integrates CephFS, a distributed, POSIX-compliant file > system which serves as an interface to the Ceph storage (like the > RBD). You can store backupfiles, ISO images, and container templates. > CephFS can be created and configured easily with its Metadata server > (MDS) in the GUI. > > We improved disk management and you can now add ZFS raid volumes, LVM, > and LVMthin pools as well as additional disks with a traditional file > system. The existing ZFS over iSCSI storage plug-in can now access LIO > target in the Linux kernel. Other new features are nesting for LXC > containers so you can use LXC and LXD inside containers or access NFS > or CIFS. If you are adventurous, you can configure PCI passthrough and > vGPUs via the GUI. > > Countless bugfixes and more smaller improvements are listed in the > release notes. > > Release notes > https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_5.3 > > Video tutorial > Watch our short introduction video What's new in Proxmox VE 5.3? > https://www.proxmox.com/en/training/video-tutorials > > Download > https://www.proxmox.com/en/downloads > Alternate ISO download: > http://download.proxmox.com/iso/ > > Source Code > https://git.proxmox.com > > Bugtracker > https://bugzilla.proxmox.com > > FAQ > Q: Can I install Proxmox VE 5.3 on top of Debian Stretch? > A: Yes, see > https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch > > Q: Can I upgrade Proxmox VE 5.x to 5.3 with apt? > A: Yes, just via GUI or via CLI with apt update && apt dist-upgrade > > Q: Can I upgrade Proxmox VE 4.x to 5.3 with apt dist-upgrade? > A: Yes, see https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0. If > you Ceph on V4.x please also check > https://pve.proxmox.com/wiki/Ceph_Jewel_to_Luminous. Please note, > Proxmox VE 4.x is already end of support since June 2018, see Proxmox > VE Support Lifecycle > https://forum.proxmox.com/threads/proxmox-ve-support-lifecycle.35755/ > > Many THANKS to our active community for all your feedback, testing, > bug reporting and patch submitting! > -- Lindsay From s.ivanov at proxmox.com Tue Dec 4 15:17:03 2018 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Tue, 4 Dec 2018 15:17:03 +0100 Subject: [PVE-User] Proxmox VE 5.3 released! In-Reply-To: References: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> Message-ID: <20181204151703.640d5127@rosa.proxmox.com> Hi, to me the most strage thing is the removal of grub (which in turn probably leads to the removal of proxmox-ve) could you please post the complete output of `apt update`? do you have any other repositories configured (/etc/apt/sources.list, /etc/apt/sources.list.d/*)? Thanks, stoiko On Tue, 4 Dec 2018 23:59:41 +1000 Lindsay Mathieson wrote: > One server has upgraded clean so far, but the 2nd one wants to remove > pve :( > > apt-get dist-upgrade > The following packages were automatically installed and are no longer > required: > ? apparmor ceph-fuse criu ebtables fonts-font-awesome hdparm ipset > libapparmor-perl libboost-iostreams1.55.0 > libboost-program-options1.55.0 libboost-random1.55.0 > ? libboost-regex1.55.0 libboost-system1.55.0 libboost-thread1.55.0 > libgnutlsxx28 libgoogle-perftools4 libicu52 libipset3 libjs-extjs > libjson-c3 liblttng-ust-ctl2 liblttng-ust0 > ? libnet1 libnetfilter-log1 libprotobuf-c1 libprotobuf10 > libpve-access-control libpve-apiclient-perl libpve-guest-common-perl > libpve-http-server-perl libpve-storage-perl > ? librados2-perl libtcmalloc-minimal4 libtemplate-perl libunwind8 > liburcu4 libuuid-perl lxc-pve lxcfs novnc-pve proxmox-widget-toolkit > pve-cluster pve-container pve-docs > ? pve-edk2-firmware pve-firewall pve-ha-manager pve-i18n pve-manager > pve-xtermjs python-ipaddr python-protobuf qemu-server spiceterm uidmap > Use 'apt autoremove' to remove them. > The following packages will be REMOVED: > ? grub-common grub-pc grub-pc-bin grub2-common libnvpair1 libzfs2 > proxmox-ve pve-kernel-4.15 pve-kernel-4.15.17-2-pve > pve-kernel-4.15.17-3-pve pve-kernel-4.15.18-4-pve > ? pve-kernel-4.15.18-5-pve > The following NEW packages will be installed: > ? libboost-program-options1.62.0 libboost-random1.62.0 > libboost-regex1.62.0 libfdt1 libjson-c3 libopus0 > The following packages will be upgraded: > ? ceph-common libcephfs1 libnvpair1linux libpve-common-perl > libpve-storage-perl librados2 libradosstriper1 librbd1 librgw2 > libzfs2linux libzpool2linux pve-container > ? pve-libspice-server1 pve-manager pve-qemu-kvm python-cephfs > python-rados python-rbd qemu-server zfs-initramfs zfs-zed > zfsutils-linux > > > I've done a plain apt-get update + upgrade and rebooted. > > Current versions: > > pveversion -v > proxmox-ve: 5.3-1 (running kernel: 4.15.18-5-pve) > pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9) > pve-kernel-4.15: 5.2-8 > pve-kernel-4.15.18-5-pve: 4.15.18-24 > pve-kernel-4.15.18-4-pve: 4.15.18-23 > pve-kernel-4.15.17-3-pve: 4.15.17-14 > pve-kernel-4.15.17-2-pve: 4.15.17-10 > corosync: 2.4.4-pve1 > criu: 2.11.1-1~bpo90 > glusterfs-client: 3.10.5-1 > ksm-control-daemon: 1.2-2 > libjs-extjs: 6.0.1-2 > libpve-access-control: 5.1-3 > libpve-apiclient-perl: 2.0-5 > libpve-common-perl: 5.0-40 > libpve-guest-common-perl: 2.0-18 > libpve-http-server-perl: 2.0-11 > libpve-storage-perl: 5.0-29 > libqb0: 1.0.3-1~bpo9 > lvm2: 2.02.168-pve6 > lxc-pve: 3.0.2+pve1-5 > lxcfs: 3.0.2-2 > novnc-pve: 1.0.0-2 > openvswitch-switch: 2.7.0-3 > proxmox-widget-toolkit: 1.0-22 > pve-cluster: 5.0-31 > pve-container: 2.0-27 > pve-docs: 5.3-1 > pve-firewall: 3.0-16 > pve-firmware: 2.0-6 > pve-ha-manager: 2.0-5 > pve-i18n: 1.0-9 > pve-libspice-server1: 0.12.8-3 > pve-qemu-kvm: 2.11.2-1 > pve-sheepdog: 1.0.2-1 > pve-xtermjs: 1.0-5 > qemu-server: 5.0-36 > smartmontools: 6.5+svn4324-1 > spiceterm: 3.0-5 > vncterm: 1.5-3 > zfsutils-linux: 0.7.11-pve1~bpo1 > > > > This is probably my oldest server, started off as Proxmox 3.x > > On 4/12/2018 8:24 pm, Martin Maurer wrote: > > Hi all! > > > > We are very excited to announce the general availability of Proxmox > > VE 5.3! > > > > Proxmox VE now integrates CephFS, a distributed, POSIX-compliant > > file system which serves as an interface to the Ceph storage (like > > the RBD). You can store backupfiles, ISO images, and container > > templates. CephFS can be created and configured easily with its > > Metadata server (MDS) in the GUI. > > > > We improved disk management and you can now add ZFS raid volumes, > > LVM, and LVMthin pools as well as additional disks with a > > traditional file system. The existing ZFS over iSCSI storage > > plug-in can now access LIO target in the Linux kernel. Other new > > features are nesting for LXC containers so you can use LXC and LXD > > inside containers or access NFS or CIFS. If you are adventurous, > > you can configure PCI passthrough and vGPUs via the GUI. > > > > Countless bugfixes and more smaller improvements are listed in the > > release notes. > > > > Release notes > > https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_5.3 > > > > Video tutorial > > Watch our short introduction video What's new in Proxmox VE 5.3? > > https://www.proxmox.com/en/training/video-tutorials > > > > Download > > https://www.proxmox.com/en/downloads > > Alternate ISO download: > > http://download.proxmox.com/iso/ > > > > Source Code > > https://git.proxmox.com > > > > Bugtracker > > https://bugzilla.proxmox.com > > > > FAQ > > Q: Can I install Proxmox VE 5.3 on top of Debian Stretch? > > A: Yes, see > > https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch > > > > Q: Can I upgrade Proxmox VE 5.x to 5.3 with apt? > > A: Yes, just via GUI or via CLI with apt update && apt dist-upgrade > > > > Q: Can I upgrade Proxmox VE 4.x to 5.3 with apt dist-upgrade? > > A: Yes, see https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0. > > If you Ceph on V4.x please also check > > https://pve.proxmox.com/wiki/Ceph_Jewel_to_Luminous. Please note, > > Proxmox VE 4.x is already end of support since June 2018, see > > Proxmox VE Support Lifecycle > > https://forum.proxmox.com/threads/proxmox-ve-support-lifecycle.35755/ > > > > Many THANKS to our active community for all your feedback, testing, > > bug reporting and patch submitting! > > > From a.antreich at proxmox.com Tue Dec 4 15:23:05 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Tue, 4 Dec 2018 15:23:05 +0100 Subject: [PVE-User] Proxmox VE 5.3 released! In-Reply-To: References: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> Message-ID: <20181204142305.lm65vpyi33svbad6@dona.proxmox.com> Hi Lindsay, On Tue, Dec 04, 2018 at 11:59:41PM +1000, Lindsay Mathieson wrote: > One server has upgraded clean so far, but the 2nd one wants to remove pve :( > > apt-get dist-upgrade > The following packages were automatically installed and are no longer > required: > ? apparmor ceph-fuse criu ebtables fonts-font-awesome hdparm ipset > libapparmor-perl libboost-iostreams1.55.0 libboost-program-options1.55.0 > libboost-random1.55.0 > ? libboost-regex1.55.0 libboost-system1.55.0 libboost-thread1.55.0 > libgnutlsxx28 libgoogle-perftools4 libicu52 libipset3 libjs-extjs libjson-c3 > liblttng-ust-ctl2 liblttng-ust0 > ? libnet1 libnetfilter-log1 libprotobuf-c1 libprotobuf10 > libpve-access-control libpve-apiclient-perl libpve-guest-common-perl > libpve-http-server-perl libpve-storage-perl > ? librados2-perl libtcmalloc-minimal4 libtemplate-perl libunwind8 liburcu4 > libuuid-perl lxc-pve lxcfs novnc-pve proxmox-widget-toolkit pve-cluster > pve-container pve-docs > ? pve-edk2-firmware pve-firewall pve-ha-manager pve-i18n pve-manager > pve-xtermjs python-ipaddr python-protobuf qemu-server spiceterm uidmap > Use 'apt autoremove' to remove them. > The following packages will be REMOVED: > ? grub-common grub-pc grub-pc-bin grub2-common libnvpair1 libzfs2 proxmox-ve > pve-kernel-4.15 pve-kernel-4.15.17-2-pve pve-kernel-4.15.17-3-pve > pve-kernel-4.15.18-4-pve > ? pve-kernel-4.15.18-5-pve > The following NEW packages will be installed: > ? libboost-program-options1.62.0 libboost-random1.62.0 libboost-regex1.62.0 > libfdt1 libjson-c3 libopus0 > The following packages will be upgraded: > ? ceph-common libcephfs1 libnvpair1linux libpve-common-perl > libpve-storage-perl librados2 libradosstriper1 librbd1 librgw2 libzfs2linux > libzpool2linux pve-container > ? pve-libspice-server1 pve-manager pve-qemu-kvm python-cephfs python-rados > python-rbd qemu-server zfs-initramfs zfs-zed zfsutils-linux > > > I've done a plain apt-get update + upgrade and rebooted. > > Current versions: > > pveversion -v > proxmox-ve: 5.3-1 (running kernel: 4.15.18-5-pve) > pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9) > pve-kernel-4.15: 5.2-8 > pve-kernel-4.15.18-5-pve: 4.15.18-24 > pve-kernel-4.15.18-4-pve: 4.15.18-23 > pve-kernel-4.15.17-3-pve: 4.15.17-14 > pve-kernel-4.15.17-2-pve: 4.15.17-10 > corosync: 2.4.4-pve1 > criu: 2.11.1-1~bpo90 > glusterfs-client: 3.10.5-1 > ksm-control-daemon: 1.2-2 > libjs-extjs: 6.0.1-2 > libpve-access-control: 5.1-3 > libpve-apiclient-perl: 2.0-5 > libpve-common-perl: 5.0-40 > libpve-guest-common-perl: 2.0-18 > libpve-http-server-perl: 2.0-11 > libpve-storage-perl: 5.0-29 > libqb0: 1.0.3-1~bpo9 > lvm2: 2.02.168-pve6 > lxc-pve: 3.0.2+pve1-5 > lxcfs: 3.0.2-2 > novnc-pve: 1.0.0-2 > openvswitch-switch: 2.7.0-3 > proxmox-widget-toolkit: 1.0-22 > pve-cluster: 5.0-31 > pve-container: 2.0-27 > pve-docs: 5.3-1 > pve-firewall: 3.0-16 > pve-firmware: 2.0-6 > pve-ha-manager: 2.0-5 > pve-i18n: 1.0-9 > pve-libspice-server1: 0.12.8-3 > pve-qemu-kvm: 2.11.2-1 > pve-sheepdog: 1.0.2-1 > pve-xtermjs: 1.0-5 > qemu-server: 5.0-36 > smartmontools: 6.5+svn4324-1 > spiceterm: 3.0-5 > vncterm: 1.5-3 > zfsutils-linux: 0.7.11-pve1~bpo1 > > > > This is probably my oldest server, started off as Proxmox 3.x Please check your repository entries in '/etc/apt/source.list' & '/etc/apt/source.list.d/', are they pointing to the right repositories? -- Cheers, Alwin From lindsay.mathieson at gmail.com Tue Dec 4 15:25:36 2018 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Wed, 5 Dec 2018 00:25:36 +1000 Subject: [PVE-User] Proxmox VE 5.3 released! In-Reply-To: <20181204151703.640d5127@rosa.proxmox.com> References: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> <20181204151703.640d5127@rosa.proxmox.com> Message-ID: <9a64e26b-66b5-bafe-dc09-eb9d3a714632@gmail.com> On 5/12/2018 12:17 am, Stoiko Ivanov wrote: > Hi, > > to me the most strage thing is the removal of grub (which in turn > probably leads to the removal of proxmox-ve) Get:1 file:/var/cache/apt-build/repository apt-build InRelease Ign:1 file:/var/cache/apt-build/repository apt-build InRelease Get:2 file:/var/cache/apt-build/repository apt-build Release [1,530 B] Get:2 file:/var/cache/apt-build/repository apt-build Release [1,530 B] Get:3 file:/var/cache/apt-build/repository apt-build Release.gpg Ign:3 file:/var/cache/apt-build/repository apt-build Release.gpg Ign:4 http://ftp.au.debian.org/debian stretch InRelease Hit:5 http://ftp.au.debian.org/debian stretch Release Hit:7 http://security.debian.org stretch/updates InRelease Hit:8 http://packages.lizardfs.com/debian/jessie jessie InRelease Hit:9 http://download.proxmox.com/debian stretch InRelease It was using the Australian internode mirror, I've since copied the sources.list from a clean server and done a apt-get clean, but still getting the grub & pve removal :( > > could you please post the complete output of `apt update`? > do you have any other repositories configured > (/etc/apt/sources.list, /etc/apt/sources.list.d/*)? > > Thanks, > stoiko > > > > On Tue, 4 Dec 2018 23:59:41 +1000 > Lindsay Mathieson wrote: > >> One server has upgraded clean so far, but the 2nd one wants to remove >> pve :( >> >> apt-get dist-upgrade >> The following packages were automatically installed and are no longer >> required: >> ? apparmor ceph-fuse criu ebtables fonts-font-awesome hdparm ipset >> libapparmor-perl libboost-iostreams1.55.0 >> libboost-program-options1.55.0 libboost-random1.55.0 >> ? libboost-regex1.55.0 libboost-system1.55.0 libboost-thread1.55.0 >> libgnutlsxx28 libgoogle-perftools4 libicu52 libipset3 libjs-extjs >> libjson-c3 liblttng-ust-ctl2 liblttng-ust0 >> ? libnet1 libnetfilter-log1 libprotobuf-c1 libprotobuf10 >> libpve-access-control libpve-apiclient-perl libpve-guest-common-perl >> libpve-http-server-perl libpve-storage-perl >> ? librados2-perl libtcmalloc-minimal4 libtemplate-perl libunwind8 >> liburcu4 libuuid-perl lxc-pve lxcfs novnc-pve proxmox-widget-toolkit >> pve-cluster pve-container pve-docs >> ? pve-edk2-firmware pve-firewall pve-ha-manager pve-i18n pve-manager >> pve-xtermjs python-ipaddr python-protobuf qemu-server spiceterm uidmap >> Use 'apt autoremove' to remove them. >> The following packages will be REMOVED: >> ? grub-common grub-pc grub-pc-bin grub2-common libnvpair1 libzfs2 >> proxmox-ve pve-kernel-4.15 pve-kernel-4.15.17-2-pve >> pve-kernel-4.15.17-3-pve pve-kernel-4.15.18-4-pve >> ? pve-kernel-4.15.18-5-pve >> The following NEW packages will be installed: >> ? libboost-program-options1.62.0 libboost-random1.62.0 >> libboost-regex1.62.0 libfdt1 libjson-c3 libopus0 >> The following packages will be upgraded: >> ? ceph-common libcephfs1 libnvpair1linux libpve-common-perl >> libpve-storage-perl librados2 libradosstriper1 librbd1 librgw2 >> libzfs2linux libzpool2linux pve-container >> ? pve-libspice-server1 pve-manager pve-qemu-kvm python-cephfs >> python-rados python-rbd qemu-server zfs-initramfs zfs-zed >> zfsutils-linux >> >> >> I've done a plain apt-get update + upgrade and rebooted. >> >> Current versions: >> >> pveversion -v >> proxmox-ve: 5.3-1 (running kernel: 4.15.18-5-pve) >> pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9) >> pve-kernel-4.15: 5.2-8 >> pve-kernel-4.15.18-5-pve: 4.15.18-24 >> pve-kernel-4.15.18-4-pve: 4.15.18-23 >> pve-kernel-4.15.17-3-pve: 4.15.17-14 >> pve-kernel-4.15.17-2-pve: 4.15.17-10 >> corosync: 2.4.4-pve1 >> criu: 2.11.1-1~bpo90 >> glusterfs-client: 3.10.5-1 >> ksm-control-daemon: 1.2-2 >> libjs-extjs: 6.0.1-2 >> libpve-access-control: 5.1-3 >> libpve-apiclient-perl: 2.0-5 >> libpve-common-perl: 5.0-40 >> libpve-guest-common-perl: 2.0-18 >> libpve-http-server-perl: 2.0-11 >> libpve-storage-perl: 5.0-29 >> libqb0: 1.0.3-1~bpo9 >> lvm2: 2.02.168-pve6 >> lxc-pve: 3.0.2+pve1-5 >> lxcfs: 3.0.2-2 >> novnc-pve: 1.0.0-2 >> openvswitch-switch: 2.7.0-3 >> proxmox-widget-toolkit: 1.0-22 >> pve-cluster: 5.0-31 >> pve-container: 2.0-27 >> pve-docs: 5.3-1 >> pve-firewall: 3.0-16 >> pve-firmware: 2.0-6 >> pve-ha-manager: 2.0-5 >> pve-i18n: 1.0-9 >> pve-libspice-server1: 0.12.8-3 >> pve-qemu-kvm: 2.11.2-1 >> pve-sheepdog: 1.0.2-1 >> pve-xtermjs: 1.0-5 >> qemu-server: 5.0-36 >> smartmontools: 6.5+svn4324-1 >> spiceterm: 3.0-5 >> vncterm: 1.5-3 >> zfsutils-linux: 0.7.11-pve1~bpo1 >> >> >> >> This is probably my oldest server, started off as Proxmox 3.x >> >> On 4/12/2018 8:24 pm, Martin Maurer wrote: >>> Hi all! >>> >>> We are very excited to announce the general availability of Proxmox >>> VE 5.3! >>> >>> Proxmox VE now integrates CephFS, a distributed, POSIX-compliant >>> file system which serves as an interface to the Ceph storage (like >>> the RBD). You can store backupfiles, ISO images, and container >>> templates. CephFS can be created and configured easily with its >>> Metadata server (MDS) in the GUI. >>> >>> We improved disk management and you can now add ZFS raid volumes, >>> LVM, and LVMthin pools as well as additional disks with a >>> traditional file system. The existing ZFS over iSCSI storage >>> plug-in can now access LIO target in the Linux kernel. Other new >>> features are nesting for LXC containers so you can use LXC and LXD >>> inside containers or access NFS or CIFS. If you are adventurous, >>> you can configure PCI passthrough and vGPUs via the GUI. >>> >>> Countless bugfixes and more smaller improvements are listed in the >>> release notes. >>> >>> Release notes >>> https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_5.3 >>> >>> Video tutorial >>> Watch our short introduction video What's new in Proxmox VE 5.3? >>> https://www.proxmox.com/en/training/video-tutorials >>> >>> Download >>> https://www.proxmox.com/en/downloads >>> Alternate ISO download: >>> http://download.proxmox.com/iso/ >>> >>> Source Code >>> https://git.proxmox.com >>> >>> Bugtracker >>> https://bugzilla.proxmox.com >>> >>> FAQ >>> Q: Can I install Proxmox VE 5.3 on top of Debian Stretch? >>> A: Yes, see >>> https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch >>> >>> Q: Can I upgrade Proxmox VE 5.x to 5.3 with apt? >>> A: Yes, just via GUI or via CLI with apt update && apt dist-upgrade >>> >>> Q: Can I upgrade Proxmox VE 4.x to 5.3 with apt dist-upgrade? >>> A: Yes, see https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0. >>> If you Ceph on V4.x please also check >>> https://pve.proxmox.com/wiki/Ceph_Jewel_to_Luminous. Please note, >>> Proxmox VE 4.x is already end of support since June 2018, see >>> Proxmox VE Support Lifecycle >>> https://forum.proxmox.com/threads/proxmox-ve-support-lifecycle.35755/ >>> >>> Many THANKS to our active community for all your feedback, testing, >>> bug reporting and patch submitting! >>> > -- Lindsay From lindsay.mathieson at gmail.com Tue Dec 4 15:27:18 2018 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Wed, 5 Dec 2018 00:27:18 +1000 Subject: [PVE-User] Proxmox VE 5.3 released! In-Reply-To: <20181204142305.lm65vpyi33svbad6@dona.proxmox.com> References: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> <20181204142305.lm65vpyi33svbad6@dona.proxmox.com> Message-ID: <9ddcf900-739f-61de-b722-15cb0317808f@gmail.com> On 5/12/2018 12:23 am, Alwin Antreich wrote: > Please check your repository entries in '/etc/apt/source.list' & > '/etc/apt/source.list.d/', are they pointing to the right repositories? sources.list should be ok: deb http://ftp.au.debian.org/debian stretch main contrib # PVE pve-no-subscription repository provided by proxmox.com, NOT recommended for production use deb http://download.proxmox.com/debian stretch pve-no-subscription # security updates deb http://security.debian.org stretch/updates main contrib I'll check the others in the morning - getting late here :( Thanks. -- Lindsay From s.ivanov at proxmox.com Tue Dec 4 15:39:04 2018 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Tue, 4 Dec 2018 15:39:04 +0100 Subject: [PVE-User] Proxmox VE 5.3 released! In-Reply-To: <9a64e26b-66b5-bafe-dc09-eb9d3a714632@gmail.com> References: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> <20181204151703.640d5127@rosa.proxmox.com> <9a64e26b-66b5-bafe-dc09-eb9d3a714632@gmail.com> Message-ID: <20181204153904.4a449190@rosa.proxmox.com> Hi, On Wed, 5 Dec 2018 00:25:36 +1000 Lindsay Mathieson wrote: > On 5/12/2018 12:17 am, Stoiko Ivanov wrote: > > Hi, > > > > to me the most strage thing is the removal of grub (which in turn > > probably leads to the removal of proxmox-ve) > > > Get:1 file:/var/cache/apt-build/repository apt-build InRelease > Ign:1 file:/var/cache/apt-build/repository apt-build InRelease > Get:2 file:/var/cache/apt-build/repository apt-build Release [1,530 B] > Get:2 file:/var/cache/apt-build/repository apt-build Release [1,530 B] > Get:3 file:/var/cache/apt-build/repository apt-build Release.gpg > Ign:3 file:/var/cache/apt-build/repository apt-build Release.gpg > Ign:4 http://ftp.au.debian.org/debian stretch InRelease > Hit:5 http://ftp.au.debian.org/debian stretch Release > Hit:7 http://security.debian.org stretch/updates InRelease > Hit:8 http://packages.lizardfs.com/debian/jessie jessie InRelease > Hit:9 http://download.proxmox.com/debian stretch InRelease The /var/cache/apt-build/ lines look like you might have some leftover installation of apt-build? [0] Unless you know that you need it, remove the corresponding .list file (probably in /etc/apt/sources.list.d/apt-build.list). The lizardfs line still references jessie (the debian release PVE 4.x was based on) - PVE 5.x is based on stretch and lizardfs seems to provide a package-mirror for stretch as well [1], but lizardfs has also made it into the official debian repositories and backports [2] (although with a slightly older version). Fix the lizardfs.list to point to a version suitable for stretch. Hope that helps! stoiko [0] https://packages.debian.org/stretch/apt-build [1] http://packages.lizardfs.com/debian/stretch/ [2] https://packages.debian.org/stretch/lizardfs-master > > > It was using the Australian internode mirror, I've since copied the > sources.list from a clean server and done a apt-get clean, but still > getting the grub & pve removal :( > > > > > > could you please post the complete output of `apt update`? > > do you have any other repositories configured > > (/etc/apt/sources.list, /etc/apt/sources.list.d/*)? > > > > Thanks, > > stoiko > > > > > > > > On Tue, 4 Dec 2018 23:59:41 +1000 > > Lindsay Mathieson wrote: > > > >> One server has upgraded clean so far, but the 2nd one wants to > >> remove pve :( > >> > >> apt-get dist-upgrade > >> The following packages were automatically installed and are no > >> longer required: > >> ? apparmor ceph-fuse criu ebtables fonts-font-awesome hdparm > >> ipset libapparmor-perl libboost-iostreams1.55.0 > >> libboost-program-options1.55.0 libboost-random1.55.0 > >> ? libboost-regex1.55.0 libboost-system1.55.0 > >> libboost-thread1.55.0 libgnutlsxx28 libgoogle-perftools4 libicu52 > >> libipset3 libjs-extjs libjson-c3 liblttng-ust-ctl2 liblttng-ust0 > >> ? libnet1 libnetfilter-log1 libprotobuf-c1 libprotobuf10 > >> libpve-access-control libpve-apiclient-perl > >> libpve-guest-common-perl libpve-http-server-perl > >> libpve-storage-perl librados2-perl libtcmalloc-minimal4 > >> libtemplate-perl libunwind8 liburcu4 libuuid-perl lxc-pve lxcfs > >> novnc-pve proxmox-widget-toolkit pve-cluster pve-container pve-docs > >> ? pve-edk2-firmware pve-firewall pve-ha-manager pve-i18n > >> pve-manager pve-xtermjs python-ipaddr python-protobuf qemu-server > >> spiceterm uidmap Use 'apt autoremove' to remove them. > >> The following packages will be REMOVED: > >> ? grub-common grub-pc grub-pc-bin grub2-common libnvpair1 libzfs2 > >> proxmox-ve pve-kernel-4.15 pve-kernel-4.15.17-2-pve > >> pve-kernel-4.15.17-3-pve pve-kernel-4.15.18-4-pve > >> ? pve-kernel-4.15.18-5-pve > >> The following NEW packages will be installed: > >> ? libboost-program-options1.62.0 libboost-random1.62.0 > >> libboost-regex1.62.0 libfdt1 libjson-c3 libopus0 > >> The following packages will be upgraded: > >> ? ceph-common libcephfs1 libnvpair1linux libpve-common-perl > >> libpve-storage-perl librados2 libradosstriper1 librbd1 librgw2 > >> libzfs2linux libzpool2linux pve-container > >> ? pve-libspice-server1 pve-manager pve-qemu-kvm python-cephfs > >> python-rados python-rbd qemu-server zfs-initramfs zfs-zed > >> zfsutils-linux > >> > >> > >> I've done a plain apt-get update + upgrade and rebooted. > >> > >> Current versions: > >> > >> pveversion -v > >> proxmox-ve: 5.3-1 (running kernel: 4.15.18-5-pve) > >> pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9) > >> pve-kernel-4.15: 5.2-8 > >> pve-kernel-4.15.18-5-pve: 4.15.18-24 > >> pve-kernel-4.15.18-4-pve: 4.15.18-23 > >> pve-kernel-4.15.17-3-pve: 4.15.17-14 > >> pve-kernel-4.15.17-2-pve: 4.15.17-10 > >> corosync: 2.4.4-pve1 > >> criu: 2.11.1-1~bpo90 > >> glusterfs-client: 3.10.5-1 > >> ksm-control-daemon: 1.2-2 > >> libjs-extjs: 6.0.1-2 > >> libpve-access-control: 5.1-3 > >> libpve-apiclient-perl: 2.0-5 > >> libpve-common-perl: 5.0-40 > >> libpve-guest-common-perl: 2.0-18 > >> libpve-http-server-perl: 2.0-11 > >> libpve-storage-perl: 5.0-29 > >> libqb0: 1.0.3-1~bpo9 > >> lvm2: 2.02.168-pve6 > >> lxc-pve: 3.0.2+pve1-5 > >> lxcfs: 3.0.2-2 > >> novnc-pve: 1.0.0-2 > >> openvswitch-switch: 2.7.0-3 > >> proxmox-widget-toolkit: 1.0-22 > >> pve-cluster: 5.0-31 > >> pve-container: 2.0-27 > >> pve-docs: 5.3-1 > >> pve-firewall: 3.0-16 > >> pve-firmware: 2.0-6 > >> pve-ha-manager: 2.0-5 > >> pve-i18n: 1.0-9 > >> pve-libspice-server1: 0.12.8-3 > >> pve-qemu-kvm: 2.11.2-1 > >> pve-sheepdog: 1.0.2-1 > >> pve-xtermjs: 1.0-5 > >> qemu-server: 5.0-36 > >> smartmontools: 6.5+svn4324-1 > >> spiceterm: 3.0-5 > >> vncterm: 1.5-3 > >> zfsutils-linux: 0.7.11-pve1~bpo1 > >> > >> > >> > >> This is probably my oldest server, started off as Proxmox 3.x > >> > >> On 4/12/2018 8:24 pm, Martin Maurer wrote: > >>> Hi all! > >>> > >>> We are very excited to announce the general availability of > >>> Proxmox VE 5.3! > >>> > >>> Proxmox VE now integrates CephFS, a distributed, POSIX-compliant > >>> file system which serves as an interface to the Ceph storage (like > >>> the RBD). You can store backupfiles, ISO images, and container > >>> templates. CephFS can be created and configured easily with its > >>> Metadata server (MDS) in the GUI. > >>> > >>> We improved disk management and you can now add ZFS raid volumes, > >>> LVM, and LVMthin pools as well as additional disks with a > >>> traditional file system. The existing ZFS over iSCSI storage > >>> plug-in can now access LIO target in the Linux kernel. Other new > >>> features are nesting for LXC containers so you can use LXC and LXD > >>> inside containers or access NFS or CIFS. If you are adventurous, > >>> you can configure PCI passthrough and vGPUs via the GUI. > >>> > >>> Countless bugfixes and more smaller improvements are listed in the > >>> release notes. > >>> > >>> Release notes > >>> https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_5.3 > >>> > >>> Video tutorial > >>> Watch our short introduction video What's new in Proxmox VE 5.3? > >>> https://www.proxmox.com/en/training/video-tutorials > >>> > >>> Download > >>> https://www.proxmox.com/en/downloads > >>> Alternate ISO download: > >>> http://download.proxmox.com/iso/ > >>> > >>> Source Code > >>> https://git.proxmox.com > >>> > >>> Bugtracker > >>> https://bugzilla.proxmox.com > >>> > >>> FAQ > >>> Q: Can I install Proxmox VE 5.3 on top of Debian Stretch? > >>> A: Yes, see > >>> https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Stretch > >>> > >>> Q: Can I upgrade Proxmox VE 5.x to 5.3 with apt? > >>> A: Yes, just via GUI or via CLI with apt update && apt > >>> dist-upgrade > >>> > >>> Q: Can I upgrade Proxmox VE 4.x to 5.3 with apt dist-upgrade? > >>> A: Yes, see https://pve.proxmox.com/wiki/Upgrade_from_4.x_to_5.0. > >>> If you Ceph on V4.x please also check > >>> https://pve.proxmox.com/wiki/Ceph_Jewel_to_Luminous. Please note, > >>> Proxmox VE 4.x is already end of support since June 2018, see > >>> Proxmox VE Support Lifecycle > >>> https://forum.proxmox.com/threads/proxmox-ve-support-lifecycle.35755/ > >>> > >>> Many THANKS to our active community for all your feedback, > >>> testing, bug reporting and patch submitting! > >>> > > > From elacunza at binovo.es Tue Dec 4 15:57:10 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Tue, 4 Dec 2018 15:57:10 +0100 Subject: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? Message-ID: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> Hi all, We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the three Proxmox nodes is new too :) Generally all was good and VMs are working? well. :-) BUT, we have some problems with the cluster; promxox1 node joins and then after about 4 minutes drops from the cluster. All multicast tests https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast run fine except the last one: *** proxmox1: root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox3 : waiting for response msg proxmox4 : waiting for response msg proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging proxmox3 : given amount of query messages was sent proxmox4 : given amount of query messages was sent proxmox3 :?? unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.073/0.184/0.390/0.061 proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.092/0.207/0.421/0.068 proxmox4 :?? unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.049/0.167/0.369/0.059 proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.063/0.185/0.386/0.064 *** proxmox3: root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox1 : waiting for response msg proxmox4 : waiting for response msg proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : waiting for response msg proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging proxmox4 : given amount of query messages was sent proxmox1 : given amount of query messages was sent proxmox1 :?? unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.083/0.193/1.030/0.055 proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.102/0.209/1.050/0.054 proxmox4 :?? unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.041/0.108/0.172/0.026 proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.123/0.190/0.030 *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox1 : waiting for response msg proxmox3 : waiting for response msg proxmox1 : waiting for response msg proxmox3 : waiting for response msg proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : given amount of query messages was sent proxmox3 : given amount of query messages was sent proxmox1 :?? unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.085/0.188/0.356/0.040 proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.114/0.208/0.377/0.041 proxmox3 :?? unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.117/0.289/0.023 proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.064/0.134/0.290/0.026 Ok, so it seems we have a network problem on proxmox1 node. Network cards are as follows: - proxmox1: Intel X540 (10Gtek) - proxmox3: Intel X710 (Intel) - proxmox4: Intel X710 (Intel) Switch is Dell N1224T-ON. Does anyone have experience with Intel X540 chip network cards or Linux ixgbe network driver or 10Gtek manufacturer? If we change corosync communication to 1 Gbit network cards (broadcom) connected to an old HPE 1800-24G switch, cluster is stable... We also have a running cluster with Dell n1224T-ON switch and X710 network cards without issues. Thanks a lot Eneko -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From marcus.haarmann at midoco.de Tue Dec 4 16:09:35 2018 From: marcus.haarmann at midoco.de (Marcus Haarmann) Date: Tue, 4 Dec 2018 16:09:35 +0100 (CET) Subject: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? In-Reply-To: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> References: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> Message-ID: <2086835097.116238.1543936175736.JavaMail.zimbra@midoco.de> Hi, you did not provide details about your configuration. How is the network card set up ? Bonding ? Send your /etc/network/interfaces details. If bonding is active, check if the mode is correct in /proc/net/bonding. We encountered differences between /etc/network/interfaces setup and resulting mode. Also, check your switch configuration, VLAN setup, MTU etc. Marcus Haarmann Von: "Eneko Lacunza" An: "pve-user" Gesendet: Dienstag, 4. Dezember 2018 15:57:10 Betreff: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? Hi all, We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the three Proxmox nodes is new too :) Generally all was good and VMs are working well. :-) BUT, we have some problems with the cluster; promxox1 node joins and then after about 4 minutes drops from the cluster. All multicast tests https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast run fine except the last one: *** proxmox1: root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox3 : waiting for response msg proxmox4 : waiting for response msg proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging proxmox3 : given amount of query messages was sent proxmox4 : given amount of query messages was sent proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.073/0.184/0.390/0.061 proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.092/0.207/0.421/0.068 proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.049/0.167/0.369/0.059 proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.063/0.185/0.386/0.064 *** proxmox3: root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox1 : waiting for response msg proxmox4 : waiting for response msg proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : waiting for response msg proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging proxmox4 : given amount of query messages was sent proxmox1 : given amount of query messages was sent proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.083/0.193/1.030/0.055 proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.102/0.209/1.050/0.054 proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.041/0.108/0.172/0.026 proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.123/0.190/0.030 *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox1 : waiting for response msg proxmox3 : waiting for response msg proxmox1 : waiting for response msg proxmox3 : waiting for response msg proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : given amount of query messages was sent proxmox3 : given amount of query messages was sent proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.085/0.188/0.356/0.040 proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.114/0.208/0.377/0.041 proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.117/0.289/0.023 proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.064/0.134/0.290/0.026 Ok, so it seems we have a network problem on proxmox1 node. Network cards are as follows: - proxmox1: Intel X540 (10Gtek) - proxmox3: Intel X710 (Intel) - proxmox4: Intel X710 (Intel) Switch is Dell N1224T-ON. Does anyone have experience with Intel X540 chip network cards or Linux ixgbe network driver or 10Gtek manufacturer? If we change corosync communication to 1 Gbit network cards (broadcom) connected to an old HPE 1800-24G switch, cluster is stable... We also have a running cluster with Dell n1224T-ON switch and X710 network cards without issues. Thanks a lot Eneko -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From elacunza at binovo.es Tue Dec 4 16:18:25 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Tue, 4 Dec 2018 16:18:25 +0100 Subject: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? In-Reply-To: <2086835097.116238.1543936175736.JavaMail.zimbra@midoco.de> References: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> <2086835097.116238.1543936175736.JavaMail.zimbra@midoco.de> Message-ID: <6a097b94-ed00-16cd-b8ef-ea7a74696503@binovo.es> hi Marcus, El 4/12/18 a las 16:09, Marcus Haarmann escribi?: > Hi, > > you did not provide details about your configuration. > How is the network card set up ? Bonding ? > Send your /etc/network/interfaces details. > If bonding is active, check if the mode is correct in /proc/net/bonding. > We encountered differences between /etc/network/interfaces setup and resulting mode. > Also, check your switch configuration, VLAN setup, MTU etc. Yes, sorry about that. I have double checked the switch and all 3 node SFP+ port have the same configuration. /etc/network/interfaces? in proxmox1 node: auto lo iface lo inet loopback iface eth0 inet manual iface eth1 inet manual iface eth2 inet manual iface eth3 inet manual iface eth4 inet manual iface eth5 inet manual auto vmbr10 iface vmbr10 inet static ??? address? 192.168.10.201 ??? netmask? 255.255.255.0 ??? bridge_ports eth3 ??? bridge_stp off ??? bridge_fd 0 auto vmbr0 iface vmbr0 inet static ??? address? 192.168.0.201 ??? netmask? 255.255.255.0 ??? gateway? 192.168.0.100 ??? bridge_ports eth4 ??? bridge_stp off ??? bridge_fd 0 auto eth4.100 iface eth4.100 inet static ??? address 10.0.2.1 ??? netmask 255.255.255.0 ??? up ip addr add 10.0.3.1/24 dev eth4.100 Cluster is running on vmbr0 network (192.168.0.0/24) Cheers > > Marcus Haarmann > > > Von: "Eneko Lacunza" > An: "pve-user" > Gesendet: Dienstag, 4. Dezember 2018 15:57:10 > Betreff: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? > > Hi all, > > We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph > hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the > three Proxmox nodes is new too :) > > Generally all was good and VMs are working well. :-) > > BUT, we have some problems with the cluster; promxox1 node joins and > then after about 4 minutes drops from the cluster. > > All multicast tests > https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast > run fine except the last one: > > *** proxmox1: > > root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 > > proxmox3 : waiting for response msg > > proxmox4 : waiting for response msg > > proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox3 : given amount of query messages was sent > > proxmox4 : given amount of query messages was sent > > proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.073/0.184/0.390/0.061 > > proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.092/0.207/0.421/0.068 > > proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.049/0.167/0.369/0.059 > > proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.063/0.185/0.386/0.064 > > > *** proxmox3: > > root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 > > proxmox1 : waiting for response msg > > proxmox4 : waiting for response msg > > proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox1 : waiting for response msg > > proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox4 : given amount of query messages was sent > > proxmox1 : given amount of query messages was sent > > proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.083/0.193/1.030/0.055 > > proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.102/0.209/1.050/0.054 > > proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.041/0.108/0.172/0.026 > > proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.123/0.190/0.030 > > > *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 > > proxmox1 : waiting for response msg > > proxmox3 : waiting for response msg > > proxmox1 : waiting for response msg > > proxmox3 : waiting for response msg > > proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox1 : given amount of query messages was sent > > proxmox3 : given amount of query messages was sent > > proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.085/0.188/0.356/0.040 > > proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.114/0.208/0.377/0.041 > > proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.117/0.289/0.023 > > proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.064/0.134/0.290/0.026 > > > Ok, so it seems we have a network problem on proxmox1 node. Network > cards are as follows: > > - proxmox1: Intel X540 (10Gtek) > - proxmox3: Intel X710 (Intel) > - proxmox4: Intel X710 (Intel) > > Switch is Dell N1224T-ON. > > Does anyone have experience with Intel X540 chip network cards or Linux > ixgbe network driver or 10Gtek manufacturer? > > If we change corosync communication to 1 Gbit network cards (broadcom) > connected to an old HPE 1800-24G switch, cluster is stable... > > We also have a running cluster with Dell n1224T-ON switch and X710 > network cards without issues. > > Thanks a lot > Eneko > > -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From elacunza at binovo.es Tue Dec 4 17:54:04 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Tue, 4 Dec 2018 17:54:04 +0100 Subject: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? In-Reply-To: <6a097b94-ed00-16cd-b8ef-ea7a74696503@binovo.es> References: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> <2086835097.116238.1543936175736.JavaMail.zimbra@midoco.de> <6a097b94-ed00-16cd-b8ef-ea7a74696503@binovo.es> Message-ID: Hi all, Seems I found the solution. eth3 on proxmox1 is a broadcom 1gbit card connected to HPE switch; it is VLAN 10 untagged on the switch end. I changed the vmbr10 bridge to use eth4.10 on the X540 card, and after ifdown/ifup and corosync and pve-cluster restart, now everything seems good; cluster is stable and omping is happy too after 10 minutes :) It is strange because multicast is on VLAN 1 network... Cheers and thanks a lot Eneko El 4/12/18 a las 16:18, Eneko Lacunza escribi?: > > hi Marcus, > > El 4/12/18 a las 16:09, Marcus Haarmann escribi?: >> Hi, >> >> you did not provide details about your configuration. >> How is the network card set up ? Bonding ? >> Send your /etc/network/interfaces details. >> If bonding is active, check if the mode is correct in /proc/net/bonding. >> We encountered differences between /etc/network/interfaces setup and >> resulting mode. >> Also, check your switch configuration, VLAN setup, MTU etc. > Yes, sorry about that. I have double checked the switch and all 3 node > SFP+ port have the same configuration. > > /etc/network/interfaces? in proxmox1 node: > auto lo > iface lo inet loopback > iface eth0 inet manual > iface eth1 inet manual > iface eth2 inet manual > iface eth3 inet manual > iface eth4 inet manual > iface eth5 inet manual > > auto vmbr10 > iface vmbr10 inet static > ??? address? 192.168.10.201 > ??? netmask? 255.255.255.0 > ??? bridge_ports eth3 > ??? bridge_stp off > ??? bridge_fd 0 > > auto vmbr0 > iface vmbr0 inet static > ??? address? 192.168.0.201 > ??? netmask? 255.255.255.0 > ??? gateway? 192.168.0.100 > ??? bridge_ports eth4 > ??? bridge_stp off > ??? bridge_fd 0 > > auto eth4.100 > iface eth4.100 inet static > ??? address 10.0.2.1 > ??? netmask 255.255.255.0 > ??? up ip addr add 10.0.3.1/24 dev eth4.100 > > Cluster is running on vmbr0 network (192.168.0.0/24) > > Cheers > >> >> Marcus Haarmann >> >> >> Von: "Eneko Lacunza" >> An: "pve-user" >> Gesendet: Dienstag, 4. Dezember 2018 15:57:10 >> Betreff: [PVE-User] Multicast problems with Intel X540 - 10Gtek >> network card? >> >> Hi all, >> >> We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph >> hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the >> three Proxmox nodes is new too :) >> >> Generally all was good and VMs are working well. :-) >> >> BUT, we have some problems with the cluster; promxox1 node joins and >> then after about 4 minutes drops from the cluster. >> >> All multicast tests >> https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast >> >> run fine except the last one: >> >> *** proxmox1: >> >> root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 >> >> proxmox3 : waiting for response msg >> >> proxmox4 : waiting for response msg >> >> proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging >> >> proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging >> >> proxmox3 : given amount of query messages was sent >> >> proxmox4 : given amount of query messages was sent >> >> proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = >> 0.073/0.184/0.390/0.061 >> >> proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, >> min/avg/max/std-dev = 0.092/0.207/0.421/0.068 >> >> proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = >> 0.049/0.167/0.369/0.059 >> >> proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, >> min/avg/max/std-dev = 0.063/0.185/0.386/0.064 >> >> >> *** proxmox3: >> >> root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 >> >> proxmox1 : waiting for response msg >> >> proxmox4 : waiting for response msg >> >> proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging >> >> proxmox1 : waiting for response msg >> >> proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging >> >> proxmox4 : given amount of query messages was sent >> >> proxmox1 : given amount of query messages was sent >> >> proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = >> 0.083/0.193/1.030/0.055 >> >> proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >> = 0.102/0.209/1.050/0.054 >> >> proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = >> 0.041/0.108/0.172/0.026 >> >> proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >> = 0.048/0.123/0.190/0.030 >> >> >> *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 >> >> proxmox1 : waiting for response msg >> >> proxmox3 : waiting for response msg >> >> proxmox1 : waiting for response msg >> >> proxmox3 : waiting for response msg >> >> proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging >> >> proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging >> >> proxmox1 : given amount of query messages was sent >> >> proxmox3 : given amount of query messages was sent >> >> proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = >> 0.085/0.188/0.356/0.040 >> >> proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >> = 0.114/0.208/0.377/0.041 >> >> proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = >> 0.048/0.117/0.289/0.023 >> >> proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >> = 0.064/0.134/0.290/0.026 >> >> >> Ok, so it seems we have a network problem on proxmox1 node. Network >> cards are as follows: >> >> - proxmox1: Intel X540 (10Gtek) >> - proxmox3: Intel X710 (Intel) >> - proxmox4: Intel X710 (Intel) >> >> Switch is Dell N1224T-ON. >> >> Does anyone have experience with Intel X540 chip network cards or Linux >> ixgbe network driver or 10Gtek manufacturer? >> >> If we change corosync communication to 1 Gbit network cards (broadcom) >> connected to an old HPE 1800-24G switch, cluster is stable... >> >> We also have a running cluster with Dell n1224T-ON switch and X710 >> network cards without issues. >> >> Thanks a lot >> Eneko >> >> > > -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From ronny+pve-user at aasen.cx Tue Dec 4 20:03:17 2018 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Tue, 4 Dec 2018 20:03:17 +0100 Subject: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? In-Reply-To: References: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> <2086835097.116238.1543936175736.JavaMail.zimbra@midoco.de> <6a097b94-ed00-16cd-b8ef-ea7a74696503@binovo.es> Message-ID: <5e092d6e-ec2a-3a34-b895-1b3896b04c75@aasen.cx> vmbr10 is a bridge (or as switch by another name) if you want the switch to work reliably with multicast you probably need to enable multicast querier. |echo 1 > /sys/devices/virtual/net/vmbr0/bridge/multicast_querier| or you can disable snooping, so that it treats multicast as broadcast. | echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping| this problem with multicast traffic also may lead to unreliable ipv6 nd? and nd-ra usage. https://pve.proxmox.com/wiki/Multicast_notes have some more notes and exampes around mulicast_querier kind regards Ronny Aasen On 04.12.2018 17:54, Eneko Lacunza wrote: > Hi all, > > Seems I found the solution. > > eth3 on proxmox1 is a broadcom 1gbit card connected to HPE switch; it > is VLAN 10 untagged on the switch end. > > I changed the vmbr10 bridge to use eth4.10 on the X540 card, and after > ifdown/ifup and corosync and pve-cluster restart, now everything seems > good; cluster is stable and omping is happy too after 10 minutes :) > > It is strange because multicast is on VLAN 1 network... > > Cheers and thanks a lot > Eneko > > El 4/12/18 a las 16:18, Eneko Lacunza escribi?: >> >> hi Marcus, >> >> El 4/12/18 a las 16:09, Marcus Haarmann escribi?: >>> Hi, >>> >>> you did not provide details about your configuration. >>> How is the network card set up ? Bonding ? >>> Send your /etc/network/interfaces details. >>> If bonding is active, check if the mode is correct in >>> /proc/net/bonding. >>> We encountered differences between /etc/network/interfaces setup and >>> resulting mode. >>> Also, check your switch configuration, VLAN setup, MTU etc. >> Yes, sorry about that. I have double checked the switch and all 3 >> node SFP+ port have the same configuration. >> >> /etc/network/interfaces? in proxmox1 node: >> auto lo >> iface lo inet loopback >> iface eth0 inet manual >> iface eth1 inet manual >> iface eth2 inet manual >> iface eth3 inet manual >> iface eth4 inet manual >> iface eth5 inet manual >> >> auto vmbr10 >> iface vmbr10 inet static >> ??? address? 192.168.10.201 >> ??? netmask? 255.255.255.0 >> ??? bridge_ports eth3 >> ??? bridge_stp off >> ??? bridge_fd 0 >> >> auto vmbr0 >> iface vmbr0 inet static >> ??? address? 192.168.0.201 >> ??? netmask? 255.255.255.0 >> ??? gateway? 192.168.0.100 >> ??? bridge_ports eth4 >> ??? bridge_stp off >> ??? bridge_fd 0 >> >> auto eth4.100 >> iface eth4.100 inet static >> ??? address 10.0.2.1 >> ??? netmask 255.255.255.0 >> ??? up ip addr add 10.0.3.1/24 dev eth4.100 >> >> Cluster is running on vmbr0 network (192.168.0.0/24) >> >> Cheers >> >>> >>> Marcus Haarmann >>> >>> >>> Von: "Eneko Lacunza" >>> An: "pve-user" >>> Gesendet: Dienstag, 4. Dezember 2018 15:57:10 >>> Betreff: [PVE-User] Multicast problems with Intel X540 - 10Gtek >>> network card? >>> >>> Hi all, >>> >>> We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph >>> hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the >>> three Proxmox nodes is new too :) >>> >>> Generally all was good and VMs are working well. :-) >>> >>> BUT, we have some problems with the cluster; promxox1 node joins and >>> then after about 4 minutes drops from the cluster. >>> >>> All multicast tests >>> https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast >>> >>> run fine except the last one: >>> >>> *** proxmox1: >>> >>> root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 >>> >>> proxmox3 : waiting for response msg >>> >>> proxmox4 : waiting for response msg >>> >>> proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging >>> >>> proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging >>> >>> proxmox3 : given amount of query messages was sent >>> >>> proxmox4 : given amount of query messages was sent >>> >>> proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>> = 0.073/0.184/0.390/0.061 >>> >>> proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, >>> min/avg/max/std-dev = 0.092/0.207/0.421/0.068 >>> >>> proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>> = 0.049/0.167/0.369/0.059 >>> >>> proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, >>> min/avg/max/std-dev = 0.063/0.185/0.386/0.064 >>> >>> >>> *** proxmox3: >>> >>> root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 >>> >>> proxmox1 : waiting for response msg >>> >>> proxmox4 : waiting for response msg >>> >>> proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging >>> >>> proxmox1 : waiting for response msg >>> >>> proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging >>> >>> proxmox4 : given amount of query messages was sent >>> >>> proxmox1 : given amount of query messages was sent >>> >>> proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>> = 0.083/0.193/1.030/0.055 >>> >>> proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, >>> min/avg/max/std-dev = 0.102/0.209/1.050/0.054 >>> >>> proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>> = 0.041/0.108/0.172/0.026 >>> >>> proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, >>> min/avg/max/std-dev = 0.048/0.123/0.190/0.030 >>> >>> >>> *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 >>> proxmox4 >>> >>> proxmox1 : waiting for response msg >>> >>> proxmox3 : waiting for response msg >>> >>> proxmox1 : waiting for response msg >>> >>> proxmox3 : waiting for response msg >>> >>> proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging >>> >>> proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging >>> >>> proxmox1 : given amount of query messages was sent >>> >>> proxmox3 : given amount of query messages was sent >>> >>> proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>> = 0.085/0.188/0.356/0.040 >>> >>> proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, >>> min/avg/max/std-dev = 0.114/0.208/0.377/0.041 >>> >>> proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>> = 0.048/0.117/0.289/0.023 >>> >>> proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, >>> min/avg/max/std-dev = 0.064/0.134/0.290/0.026 >>> >>> >>> Ok, so it seems we have a network problem on proxmox1 node. Network >>> cards are as follows: >>> >>> - proxmox1: Intel X540 (10Gtek) >>> - proxmox3: Intel X710 (Intel) >>> - proxmox4: Intel X710 (Intel) >>> >>> Switch is Dell N1224T-ON. >>> >>> Does anyone have experience with Intel X540 chip network cards or Linux >>> ixgbe network driver or 10Gtek manufacturer? >>> >>> If we change corosync communication to 1 Gbit network cards (broadcom) >>> connected to an old HPE 1800-24G switch, cluster is stable... >>> >>> We also have a running cluster with Dell n1224T-ON switch and X710 >>> network cards without issues. >>> >>> Thanks a lot >>> Eneko >>> >>> >> >> > > From smr at kmi.com Tue Dec 4 23:50:41 2018 From: smr at kmi.com (Stefan M. Radman) Date: Tue, 4 Dec 2018 22:50:41 +0000 Subject: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? In-Reply-To: <5e092d6e-ec2a-3a34-b895-1b3896b04c75@aasen.cx> References: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> <2086835097.116238.1543936175736.JavaMail.zimbra@midoco.de> <6a097b94-ed00-16cd-b8ef-ea7a74696503@binovo.es> <5e092d6e-ec2a-3a34-b895-1b3896b04c75@aasen.cx> Message-ID: <98FCEDEC-60AB-43D0-8AA6-7DE78BBE7C85@kmi.com> Don't put your corosync traffic on bridges. Dedicate an untagged interface on each node for corosync. All you need for your cluster network is this: auto eth3 iface eth3 inet static address 192.168.10.201 netmask 255.255.255.0 #corosync ring0 Put that interface into an isolated VLAN with IGMP snooping enabled. Prune that VLAN from all trunks to limit its extent and your troubles. Stefan On Dec 4, 2018, at 8:03 PM, Ronny Aasen > wrote: vmbr10 is a bridge (or as switch by another name) if you want the switch to work reliably with multicast you probably need to enable multicast querier. |echo 1 > /sys/devices/virtual/net/vmbr0/bridge/multicast_querier| or you can disable snooping, so that it treats multicast as broadcast. | echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping| this problem with multicast traffic also may lead to unreliable ipv6 nd and nd-ra usage. https://pve.proxmox.com/wiki/Multicast_notes have some more notes and exampes around mulicast_querier kind regards Ronny Aasen On 04.12.2018 17:54, Eneko Lacunza wrote: Hi all, Seems I found the solution. eth3 on proxmox1 is a broadcom 1gbit card connected to HPE switch; it is VLAN 10 untagged on the switch end. I changed the vmbr10 bridge to use eth4.10 on the X540 card, and after ifdown/ifup and corosync and pve-cluster restart, now everything seems good; cluster is stable and omping is happy too after 10 minutes :) It is strange because multicast is on VLAN 1 network... Cheers and thanks a lot Eneko El 4/12/18 a las 16:18, Eneko Lacunza escribi?: hi Marcus, El 4/12/18 a las 16:09, Marcus Haarmann escribi?: Hi, you did not provide details about your configuration. How is the network card set up ? Bonding ? Send your /etc/network/interfaces details. If bonding is active, check if the mode is correct in /proc/net/bonding. We encountered differences between /etc/network/interfaces setup and resulting mode. Also, check your switch configuration, VLAN setup, MTU etc. Yes, sorry about that. I have double checked the switch and all 3 node SFP+ port have the same configuration. /etc/network/interfaces in proxmox1 node: auto lo iface lo inet loopback iface eth0 inet manual iface eth1 inet manual iface eth2 inet manual iface eth3 inet manual iface eth4 inet manual iface eth5 inet manual auto vmbr10 iface vmbr10 inet static address 192.168.10.201 netmask 255.255.255.0 bridge_ports eth3 bridge_stp off bridge_fd 0 auto vmbr0 iface vmbr0 inet static address 192.168.0.201 netmask 255.255.255.0 gateway 192.168.0.100 bridge_ports eth4 bridge_stp off bridge_fd 0 auto eth4.100 iface eth4.100 inet static address 10.0.2.1 netmask 255.255.255.0 up ip addr add 10.0.3.1/24 dev eth4.100 Cluster is running on vmbr0 network (192.168.0.0/24) Cheers Marcus Haarmann Von: "Eneko Lacunza" > An: "pve-user" > Gesendet: Dienstag, 4. Dezember 2018 15:57:10 Betreff: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? Hi all, We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the three Proxmox nodes is new too :) Generally all was good and VMs are working well. :-) BUT, we have some problems with the cluster; promxox1 node joins and then after about 4 minutes drops from the cluster. All multicast tests https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast run fine except the last one: *** proxmox1: root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox3 : waiting for response msg proxmox4 : waiting for response msg proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging proxmox3 : given amount of query messages was sent proxmox4 : given amount of query messages was sent proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.073/0.184/0.390/0.061 proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.092/0.207/0.421/0.068 proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.049/0.167/0.369/0.059 proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.063/0.185/0.386/0.064 *** proxmox3: root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox1 : waiting for response msg proxmox4 : waiting for response msg proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : waiting for response msg proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging proxmox4 : given amount of query messages was sent proxmox1 : given amount of query messages was sent proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.083/0.193/1.030/0.055 proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.102/0.209/1.050/0.054 proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.041/0.108/0.172/0.026 proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.123/0.190/0.030 *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox1 : waiting for response msg proxmox3 : waiting for response msg proxmox1 : waiting for response msg proxmox3 : waiting for response msg proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : given amount of query messages was sent proxmox3 : given amount of query messages was sent proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.085/0.188/0.356/0.040 proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.114/0.208/0.377/0.041 proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.117/0.289/0.023 proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.064/0.134/0.290/0.026 Ok, so it seems we have a network problem on proxmox1 node. Network cards are as follows: - proxmox1: Intel X540 (10Gtek) - proxmox3: Intel X710 (Intel) - proxmox4: Intel X710 (Intel) Switch is Dell N1224T-ON. Does anyone have experience with Intel X540 chip network cards or Linux ixgbe network driver or 10Gtek manufacturer? If we change corosync communication to 1 Gbit network cards (broadcom) connected to an old HPE 1800-24G switch, cluster is stable... We also have a running cluster with Dell n1224T-ON switch and X710 network cards without issues. Thanks a lot Eneko _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From lindsay.mathieson at gmail.com Wed Dec 5 05:50:02 2018 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Wed, 5 Dec 2018 14:50:02 +1000 Subject: [PVE-User] Proxmox VE 5.3 released! In-Reply-To: <20181204153904.4a449190@rosa.proxmox.com> References: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> <20181204151703.640d5127@rosa.proxmox.com> <9a64e26b-66b5-bafe-dc09-eb9d3a714632@gmail.com> <20181204153904.4a449190@rosa.proxmox.com> Message-ID: On 5/12/2018 12:39 am, Stoiko Ivanov wrote: > The/var/cache/apt-build/ lines look like you might have some leftover > installation of apt-build? [0] > Unless you know that you need it, remove the corresponding .list file > (probably in /etc/apt/sources.list.d/apt-build.list). > > > The lizardfs line still references jessie (the debian release PVE 4.x > was based on) - PVE 5.x is based on stretch and lizardfs seems to > provide a package-mirror for stretch as well [1], but lizardfs has also > made it into the official debian repositories and backports [2] > (although with a slightly older version). > Fix the lizardfs.list to point to a version suitable for stretch. Turned out to be my zabbix.list in the end, it was referencing jessie. Once I updated it to stretch, all was good. nb. The lizardfs jessie repo works fine with stretch, though I am actually using the stretch binaries anyway. Thanks all. -- Lindsay From lindsay.mathieson at gmail.com Wed Dec 5 07:03:57 2018 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Wed, 5 Dec 2018 16:03:57 +1000 Subject: [PVE-User] Proxmox VE 5.3 released! In-Reply-To: References: <824b2ad6-33f9-38bb-4e0d-90df874b70a9@proxmox.com> <20181204151703.640d5127@rosa.proxmox.com> <9a64e26b-66b5-bafe-dc09-eb9d3a714632@gmail.com> <20181204153904.4a449190@rosa.proxmox.com> Message-ID: <20246bc0-2705-6560-a70c-cd4ed83cca0c@gmail.com> On 5/12/2018 2:50 pm, Lindsay Mathieson wrote: > Turned out to be my zabbix.list in the end, it was referencing jessie. > Once I updated it to stretch, all was good. > And .... turns out I was mistaken, still had the problem. It was my zfs setup - I used to have zfsonlinux installed, must have been a hangup from that - was blocking on zfsutils. In the end I had to let it uninstall grub, and pve, then reinstall them. Seems to be ok now. -- Lindsay From elacunza at binovo.es Wed Dec 5 11:08:33 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Wed, 5 Dec 2018 11:08:33 +0100 Subject: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? In-Reply-To: <5e092d6e-ec2a-3a34-b895-1b3896b04c75@aasen.cx> References: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> <2086835097.116238.1543936175736.JavaMail.zimbra@midoco.de> <6a097b94-ed00-16cd-b8ef-ea7a74696503@binovo.es> <5e092d6e-ec2a-3a34-b895-1b3896b04c75@aasen.cx> Message-ID: Hi Ronny, El 4/12/18 a las 20:03, Ronny Aasen escribi?: > vmbr10 is a bridge (or as switch by another name) It is a bridge in proxmox1 host. > if you want the switch to work reliably with multicast you probably > need to enable multicast querier. > |echo 1 > /sys/devices/virtual/net/vmbr0/bridge/multicast_querier| I tried enabling the querier on the Dell switch, but didn't fix the issue... > > or you can disable snooping, so that it treats multicast as broadcast. | > echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping| > > this problem with multicast traffic also may lead to unreliable ipv6 > nd? and nd-ra usage. > https://pve.proxmox.com/wiki/Multicast_notes have some more notes and > exampes around mulicast_querier > Yes, I read that page, but the issue didn't make sense to me (doesn't make yet... ;)? ) Seems that using a network port on another switch, although not used for multicast, was confusing someone... Thanks a lot Eneko > kind regards > Ronny Aasen > > > > On 04.12.2018 17:54, Eneko Lacunza wrote: >> Hi all, >> >> Seems I found the solution. >> >> eth3 on proxmox1 is a broadcom 1gbit card connected to HPE switch; it >> is VLAN 10 untagged on the switch end. >> >> I changed the vmbr10 bridge to use eth4.10 on the X540 card, and >> after ifdown/ifup and corosync and pve-cluster restart, now >> everything seems good; cluster is stable and omping is happy too >> after 10 minutes :) >> >> It is strange because multicast is on VLAN 1 network... >> >> Cheers and thanks a lot >> Eneko >> >> El 4/12/18 a las 16:18, Eneko Lacunza escribi?: >>> >>> hi Marcus, >>> >>> El 4/12/18 a las 16:09, Marcus Haarmann escribi?: >>>> Hi, >>>> >>>> you did not provide details about your configuration. >>>> How is the network card set up ? Bonding ? >>>> Send your /etc/network/interfaces details. >>>> If bonding is active, check if the mode is correct in >>>> /proc/net/bonding. >>>> We encountered differences between /etc/network/interfaces setup >>>> and resulting mode. >>>> Also, check your switch configuration, VLAN setup, MTU etc. >>> Yes, sorry about that. I have double checked the switch and all 3 >>> node SFP+ port have the same configuration. >>> >>> /etc/network/interfaces? in proxmox1 node: >>> auto lo >>> iface lo inet loopback >>> iface eth0 inet manual >>> iface eth1 inet manual >>> iface eth2 inet manual >>> iface eth3 inet manual >>> iface eth4 inet manual >>> iface eth5 inet manual >>> >>> auto vmbr10 >>> iface vmbr10 inet static >>> ??? address? 192.168.10.201 >>> ??? netmask? 255.255.255.0 >>> ??? bridge_ports eth3 >>> ??? bridge_stp off >>> ??? bridge_fd 0 >>> >>> auto vmbr0 >>> iface vmbr0 inet static >>> ??? address? 192.168.0.201 >>> ??? netmask? 255.255.255.0 >>> ??? gateway? 192.168.0.100 >>> ??? bridge_ports eth4 >>> ??? bridge_stp off >>> ??? bridge_fd 0 >>> >>> auto eth4.100 >>> iface eth4.100 inet static >>> ??? address 10.0.2.1 >>> ??? netmask 255.255.255.0 >>> ??? up ip addr add 10.0.3.1/24 dev eth4.100 >>> >>> Cluster is running on vmbr0 network (192.168.0.0/24) >>> >>> Cheers >>> >>>> >>>> Marcus Haarmann >>>> >>>> >>>> Von: "Eneko Lacunza" >>>> An: "pve-user" >>>> Gesendet: Dienstag, 4. Dezember 2018 15:57:10 >>>> Betreff: [PVE-User] Multicast problems with Intel X540 - 10Gtek >>>> network card? >>>> >>>> Hi all, >>>> >>>> We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph >>>> hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the >>>> three Proxmox nodes is new too :) >>>> >>>> Generally all was good and VMs are working well. :-) >>>> >>>> BUT, we have some problems with the cluster; promxox1 node joins and >>>> then after about 4 minutes drops from the cluster. >>>> >>>> All multicast tests >>>> https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast >>>> >>>> run fine except the last one: >>>> >>>> *** proxmox1: >>>> >>>> root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 >>>> >>>> proxmox3 : waiting for response msg >>>> >>>> proxmox4 : waiting for response msg >>>> >>>> proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging >>>> >>>> proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging >>>> >>>> proxmox3 : given amount of query messages was sent >>>> >>>> proxmox4 : given amount of query messages was sent >>>> >>>> proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>>> = 0.073/0.184/0.390/0.061 >>>> >>>> proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, >>>> min/avg/max/std-dev = 0.092/0.207/0.421/0.068 >>>> >>>> proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>>> = 0.049/0.167/0.369/0.059 >>>> >>>> proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, >>>> min/avg/max/std-dev = 0.063/0.185/0.386/0.064 >>>> >>>> >>>> *** proxmox3: >>>> >>>> root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 >>>> proxmox4 >>>> >>>> proxmox1 : waiting for response msg >>>> >>>> proxmox4 : waiting for response msg >>>> >>>> proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging >>>> >>>> proxmox1 : waiting for response msg >>>> >>>> proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging >>>> >>>> proxmox4 : given amount of query messages was sent >>>> >>>> proxmox1 : given amount of query messages was sent >>>> >>>> proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>>> = 0.083/0.193/1.030/0.055 >>>> >>>> proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, >>>> min/avg/max/std-dev = 0.102/0.209/1.050/0.054 >>>> >>>> proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>>> = 0.041/0.108/0.172/0.026 >>>> >>>> proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, >>>> min/avg/max/std-dev = 0.048/0.123/0.190/0.030 >>>> >>>> >>>> *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 >>>> proxmox4 >>>> >>>> proxmox1 : waiting for response msg >>>> >>>> proxmox3 : waiting for response msg >>>> >>>> proxmox1 : waiting for response msg >>>> >>>> proxmox3 : waiting for response msg >>>> >>>> proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging >>>> >>>> proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging >>>> >>>> proxmox1 : given amount of query messages was sent >>>> >>>> proxmox3 : given amount of query messages was sent >>>> >>>> proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>>> = 0.085/0.188/0.356/0.040 >>>> >>>> proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, >>>> min/avg/max/std-dev = 0.114/0.208/0.377/0.041 >>>> >>>> proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev >>>> = 0.048/0.117/0.289/0.023 >>>> >>>> proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, >>>> min/avg/max/std-dev = 0.064/0.134/0.290/0.026 >>>> >>>> >>>> Ok, so it seems we have a network problem on proxmox1 node. Network >>>> cards are as follows: >>>> >>>> - proxmox1: Intel X540 (10Gtek) >>>> - proxmox3: Intel X710 (Intel) >>>> - proxmox4: Intel X710 (Intel) >>>> >>>> Switch is Dell N1224T-ON. >>>> >>>> Does anyone have experience with Intel X540 chip network cards or >>>> Linux >>>> ixgbe network driver or 10Gtek manufacturer? >>>> >>>> If we change corosync communication to 1 Gbit network cards (broadcom) >>>> connected to an old HPE 1800-24G switch, cluster is stable... >>>> >>>> We also have a running cluster with Dell n1224T-ON switch and X710 >>>> network cards without issues. >>>> >>>> Thanks a lot >>>> Eneko >>>> >>>> >>> >>> >> >> > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From elacunza at binovo.es Wed Dec 5 13:34:59 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Wed, 5 Dec 2018 13:34:59 +0100 Subject: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? In-Reply-To: <98FCEDEC-60AB-43D0-8AA6-7DE78BBE7C85@kmi.com> References: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> <2086835097.116238.1543936175736.JavaMail.zimbra@midoco.de> <6a097b94-ed00-16cd-b8ef-ea7a74696503@binovo.es> <5e092d6e-ec2a-3a34-b895-1b3896b04c75@aasen.cx> <98FCEDEC-60AB-43D0-8AA6-7DE78BBE7C85@kmi.com> Message-ID: <4cd4cc42-9e6f-c0a4-a27d-2a796865d995@binovo.es> Hi Stefan, El 4/12/18 a las 23:50, Stefan M. Radman escribi?: > Don't put your corosync traffic on bridges. > Dedicate an untagged interface on each node for corosync. This is surprising, because Proxmox does this by default! :) > All you need for your cluster network is this: > > auto eth3 > iface eth3 inet static > address 192.168.10.201 > netmask 255.255.255.0 > #corosync ring0 > > Put that interface into an isolated VLAN with IGMP snooping enabled. > Prune that VLAN from all trunks to limit its extent and your troubles. But we're no using eth3/vmbr10 for corosync. We're using vmbr0 (default Proxmox config) por corosync. Thanks a lot Eneko > > Stefan > > > On Dec 4, 2018, at 8:03 PM, Ronny Aasen > wrote: > > vmbr10 is a bridge (or as switch by another name) > if you want the switch to work reliably with multicast you probably need to enable multicast querier. > |echo 1 > /sys/devices/virtual/net/vmbr0/bridge/multicast_querier| > > or you can disable snooping, so that it treats multicast as broadcast. | > echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping| > > this problem with multicast traffic also may lead to unreliable ipv6 nd and nd-ra usage. > https://pve.proxmox.com/wiki/Multicast_notes have some more notes and exampes around mulicast_querier > > kind regards > Ronny Aasen > > > > On 04.12.2018 17:54, Eneko Lacunza wrote: > Hi all, > > Seems I found the solution. > > eth3 on proxmox1 is a broadcom 1gbit card connected to HPE switch; it is VLAN 10 untagged on the switch end. > > I changed the vmbr10 bridge to use eth4.10 on the X540 card, and after ifdown/ifup and corosync and pve-cluster restart, now everything seems good; cluster is stable and omping is happy too after 10 minutes :) > > It is strange because multicast is on VLAN 1 network... > > Cheers and thanks a lot > Eneko > > El 4/12/18 a las 16:18, Eneko Lacunza escribi?: > > hi Marcus, > > El 4/12/18 a las 16:09, Marcus Haarmann escribi?: > Hi, > > you did not provide details about your configuration. > How is the network card set up ? Bonding ? > Send your /etc/network/interfaces details. > If bonding is active, check if the mode is correct in /proc/net/bonding. > We encountered differences between /etc/network/interfaces setup and resulting mode. > Also, check your switch configuration, VLAN setup, MTU etc. > Yes, sorry about that. I have double checked the switch and all 3 node SFP+ port have the same configuration. > > /etc/network/interfaces in proxmox1 node: > auto lo > iface lo inet loopback > iface eth0 inet manual > iface eth1 inet manual > iface eth2 inet manual > iface eth3 inet manual > iface eth4 inet manual > iface eth5 inet manual > > auto vmbr10 > iface vmbr10 inet static > address 192.168.10.201 > netmask 255.255.255.0 > bridge_ports eth3 > bridge_stp off > bridge_fd 0 > > auto vmbr0 > iface vmbr0 inet static > address 192.168.0.201 > netmask 255.255.255.0 > gateway 192.168.0.100 > bridge_ports eth4 > bridge_stp off > bridge_fd 0 > > auto eth4.100 > iface eth4.100 inet static > address 10.0.2.1 > netmask 255.255.255.0 > up ip addr add 10.0.3.1/24 dev eth4.100 > > Cluster is running on vmbr0 network (192.168.0.0/24) > > Cheers > > > Marcus Haarmann > > > Von: "Eneko Lacunza" > > An: "pve-user" > > Gesendet: Dienstag, 4. Dezember 2018 15:57:10 > Betreff: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? > > Hi all, > > We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph > hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the > three Proxmox nodes is new too :) > > Generally all was good and VMs are working well. :-) > > BUT, we have some problems with the cluster; promxox1 node joins and > then after about 4 minutes drops from the cluster. > > All multicast tests > https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast > run fine except the last one: > > *** proxmox1: > > root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 > > proxmox3 : waiting for response msg > > proxmox4 : waiting for response msg > > proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox3 : given amount of query messages was sent > > proxmox4 : given amount of query messages was sent > > proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.073/0.184/0.390/0.061 > > proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.092/0.207/0.421/0.068 > > proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.049/0.167/0.369/0.059 > > proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.063/0.185/0.386/0.064 > > > *** proxmox3: > > root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 > > proxmox1 : waiting for response msg > > proxmox4 : waiting for response msg > > proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox1 : waiting for response msg > > proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox4 : given amount of query messages was sent > > proxmox1 : given amount of query messages was sent > > proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.083/0.193/1.030/0.055 > > proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.102/0.209/1.050/0.054 > > proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.041/0.108/0.172/0.026 > > proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.123/0.190/0.030 > > > *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 > > proxmox1 : waiting for response msg > > proxmox3 : waiting for response msg > > proxmox1 : waiting for response msg > > proxmox3 : waiting for response msg > > proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox1 : given amount of query messages was sent > > proxmox3 : given amount of query messages was sent > > proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.085/0.188/0.356/0.040 > > proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.114/0.208/0.377/0.041 > > proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.117/0.289/0.023 > > proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.064/0.134/0.290/0.026 > > > Ok, so it seems we have a network problem on proxmox1 node. Network > cards are as follows: > > - proxmox1: Intel X540 (10Gtek) > - proxmox3: Intel X710 (Intel) > - proxmox4: Intel X710 (Intel) > > Switch is Dell N1224T-ON. > > Does anyone have experience with Intel X540 chip network cards or Linux > ixgbe network driver or 10Gtek manufacturer? > > If we change corosync communication to 1 Gbit network cards (broadcom) > connected to an old HPE 1800-24G switch, cluster is stable... > > We also have a running cluster with Dell n1224T-ON switch and X710 > network cards without issues. > > Thanks a lot > Eneko > > > > > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From smr at kmi.com Wed Dec 5 13:52:23 2018 From: smr at kmi.com (Stefan M. Radman) Date: Wed, 5 Dec 2018 12:52:23 +0000 Subject: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? In-Reply-To: <4cd4cc42-9e6f-c0a4-a27d-2a796865d995@binovo.es> References: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> <2086835097.116238.1543936175736.JavaMail.zimbra@midoco.de> <6a097b94-ed00-16cd-b8ef-ea7a74696503@binovo.es> <5e092d6e-ec2a-3a34-b895-1b3896b04c75@aasen.cx> <98FCEDEC-60AB-43D0-8AA6-7DE78BBE7C85@kmi.com> <4cd4cc42-9e6f-c0a4-a27d-2a796865d995@binovo.es> Message-ID: <64DCA5F1-0DA1-4419-BCB8-988E43B7F17B@kmi.com> Hi Eneko Even if PVE does it by default it does not mean that it is the best thing to do. The default configuration of PVE is a starting point. https://pve.proxmox.com/wiki/Separate_Cluster_Network Separate Cluster Network - Introduction It is good practice to use a separate network for corosync, which handles the cluster communication in Proxmox VE. It is one of the most important part in an fault tolerant (HA) system and other network traffic may disturb corosync. I'd recommend a thorough reading of the document quoted above. Don't use vmbr0 for cluster traffic. Don't use any vmbr for cluster traffic. Stefan On Dec 5, 2018, at 13:34, Eneko Lacunza > wrote: Hi Stefan, El 4/12/18 a las 23:50, Stefan M. Radman escribi?: Don't put your corosync traffic on bridges. Dedicate an untagged interface on each node for corosync. This is surprising, because Proxmox does this by default! :) All you need for your cluster network is this: auto eth3 iface eth3 inet static address 192.168.10.201 netmask 255.255.255.0 #corosync ring0 Put that interface into an isolated VLAN with IGMP snooping enabled. Prune that VLAN from all trunks to limit its extent and your troubles. But we're no using eth3/vmbr10 for corosync. We're using vmbr0 (default Proxmox config) por corosync. Thanks a lot Eneko Stefan On Dec 4, 2018, at 8:03 PM, Ronny Aasen > wrote: vmbr10 is a bridge (or as switch by another name) if you want the switch to work reliably with multicast you probably need to enable multicast querier. |echo 1 > /sys/devices/virtual/net/vmbr0/bridge/multicast_querier| or you can disable snooping, so that it treats multicast as broadcast. | echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping| this problem with multicast traffic also may lead to unreliable ipv6 nd and nd-ra usage. https://pve.proxmox.com/wiki/Multicast_notes have some more notes and exampes around mulicast_querier kind regards Ronny Aasen On 04.12.2018 17:54, Eneko Lacunza wrote: Hi all, Seems I found the solution. eth3 on proxmox1 is a broadcom 1gbit card connected to HPE switch; it is VLAN 10 untagged on the switch end. I changed the vmbr10 bridge to use eth4.10 on the X540 card, and after ifdown/ifup and corosync and pve-cluster restart, now everything seems good; cluster is stable and omping is happy too after 10 minutes :) It is strange because multicast is on VLAN 1 network... Cheers and thanks a lot Eneko El 4/12/18 a las 16:18, Eneko Lacunza escribi?: hi Marcus, El 4/12/18 a las 16:09, Marcus Haarmann escribi?: Hi, you did not provide details about your configuration. How is the network card set up ? Bonding ? Send your /etc/network/interfaces details. If bonding is active, check if the mode is correct in /proc/net/bonding. We encountered differences between /etc/network/interfaces setup and resulting mode. Also, check your switch configuration, VLAN setup, MTU etc. Yes, sorry about that. I have double checked the switch and all 3 node SFP+ port have the same configuration. /etc/network/interfaces in proxmox1 node: auto lo iface lo inet loopback iface eth0 inet manual iface eth1 inet manual iface eth2 inet manual iface eth3 inet manual iface eth4 inet manual iface eth5 inet manual auto vmbr10 iface vmbr10 inet static address 192.168.10.201 netmask 255.255.255.0 bridge_ports eth3 bridge_stp off bridge_fd 0 auto vmbr0 iface vmbr0 inet static address 192.168.0.201 netmask 255.255.255.0 gateway 192.168.0.100 bridge_ports eth4 bridge_stp off bridge_fd 0 auto eth4.100 iface eth4.100 inet static address 10.0.2.1 netmask 255.255.255.0 up ip addr add 10.0.3.1/24 dev eth4.100 Cluster is running on vmbr0 network (192.168.0.0/24) Cheers Marcus Haarmann Von: "Eneko Lacunza" > An: "pve-user" > Gesendet: Dienstag, 4. Dezember 2018 15:57:10 Betreff: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? Hi all, We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the three Proxmox nodes is new too :) Generally all was good and VMs are working well. :-) BUT, we have some problems with the cluster; promxox1 node joins and then after about 4 minutes drops from the cluster. All multicast tests https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast run fine except the last one: *** proxmox1: root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox3 : waiting for response msg proxmox4 : waiting for response msg proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging proxmox3 : given amount of query messages was sent proxmox4 : given amount of query messages was sent proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.073/0.184/0.390/0.061 proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.092/0.207/0.421/0.068 proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.049/0.167/0.369/0.059 proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.063/0.185/0.386/0.064 *** proxmox3: root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox1 : waiting for response msg proxmox4 : waiting for response msg proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : waiting for response msg proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging proxmox4 : given amount of query messages was sent proxmox1 : given amount of query messages was sent proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.083/0.193/1.030/0.055 proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.102/0.209/1.050/0.054 proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.041/0.108/0.172/0.026 proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.123/0.190/0.030 *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox1 : waiting for response msg proxmox3 : waiting for response msg proxmox1 : waiting for response msg proxmox3 : waiting for response msg proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : given amount of query messages was sent proxmox3 : given amount of query messages was sent proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.085/0.188/0.356/0.040 proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.114/0.208/0.377/0.041 proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.117/0.289/0.023 proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.064/0.134/0.290/0.026 Ok, so it seems we have a network problem on proxmox1 node. Network cards are as follows: - proxmox1: Intel X540 (10Gtek) - proxmox3: Intel X710 (Intel) - proxmox4: Intel X710 (Intel) Switch is Dell N1224T-ON. Does anyone have experience with Intel X540 chip network cards or Linux ixgbe network driver or 10Gtek manufacturer? If we change corosync communication to 1 Gbit network cards (broadcom) connected to an old HPE 1800-24G switch, cluster is stable... We also have a running cluster with Dell n1224T-ON switch and X710 network cards without issues. Thanks a lot Eneko _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From elacunza at binovo.es Wed Dec 5 13:55:05 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Wed, 5 Dec 2018 13:55:05 +0100 Subject: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? In-Reply-To: <64DCA5F1-0DA1-4419-BCB8-988E43B7F17B@kmi.com> References: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> <2086835097.116238.1543936175736.JavaMail.zimbra@midoco.de> <6a097b94-ed00-16cd-b8ef-ea7a74696503@binovo.es> <5e092d6e-ec2a-3a34-b895-1b3896b04c75@aasen.cx> <98FCEDEC-60AB-43D0-8AA6-7DE78BBE7C85@kmi.com> <4cd4cc42-9e6f-c0a4-a27d-2a796865d995@binovo.es> <64DCA5F1-0DA1-4419-BCB8-988E43B7F17B@kmi.com> Message-ID: <215268ab-b120-4d10-0c2c-8ef5ad14d962@binovo.es> Hi Stefan, Thanks a lot for your suggestions! Cheers Eneko El 5/12/18 a las 13:52, Stefan M. Radman escribi?: > Hi Eneko > > Even if PVE does it by default it does not mean that it is the best thing to do. > The default configuration of PVE is a starting point. > > https://pve.proxmox.com/wiki/Separate_Cluster_Network > Separate Cluster Network - Introduction > It is good practice to use a separate network for corosync, which handles the cluster communication in Proxmox VE. > It is one of the most important part in an fault tolerant (HA) system and other network traffic may disturb corosync. > > I'd recommend a thorough reading of the document quoted above. > > Don't use vmbr0 for cluster traffic. > Don't use any vmbr for cluster traffic. > > Stefan > > On Dec 5, 2018, at 13:34, Eneko Lacunza > wrote: > > Hi Stefan, > > El 4/12/18 a las 23:50, Stefan M. Radman escribi?: > Don't put your corosync traffic on bridges. > Dedicate an untagged interface on each node for corosync. > This is surprising, because Proxmox does this by default! :) > > All you need for your cluster network is this: > > auto eth3 > iface eth3 inet static > address 192.168.10.201 > netmask 255.255.255.0 > #corosync ring0 > > Put that interface into an isolated VLAN with IGMP snooping enabled. > Prune that VLAN from all trunks to limit its extent and your troubles. > > But we're no using eth3/vmbr10 for corosync. We're using vmbr0 (default Proxmox config) por corosync. > > Thanks a lot > Eneko > > Stefan > > > On Dec 4, 2018, at 8:03 PM, Ronny Aasen > wrote: > > vmbr10 is a bridge (or as switch by another name) > if you want the switch to work reliably with multicast you probably need to enable multicast querier. > |echo 1 > /sys/devices/virtual/net/vmbr0/bridge/multicast_querier| > > or you can disable snooping, so that it treats multicast as broadcast. | > echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping| > > this problem with multicast traffic also may lead to unreliable ipv6 nd and nd-ra usage. > https://pve.proxmox.com/wiki/Multicast_notes have some more notes and exampes around mulicast_querier > > kind regards > Ronny Aasen > > > > On 04.12.2018 17:54, Eneko Lacunza wrote: > Hi all, > > Seems I found the solution. > > eth3 on proxmox1 is a broadcom 1gbit card connected to HPE switch; it is VLAN 10 untagged on the switch end. > > I changed the vmbr10 bridge to use eth4.10 on the X540 card, and after ifdown/ifup and corosync and pve-cluster restart, now everything seems good; cluster is stable and omping is happy too after 10 minutes :) > > It is strange because multicast is on VLAN 1 network... > > Cheers and thanks a lot > Eneko > > El 4/12/18 a las 16:18, Eneko Lacunza escribi?: > > hi Marcus, > > El 4/12/18 a las 16:09, Marcus Haarmann escribi?: > Hi, > > you did not provide details about your configuration. > How is the network card set up ? Bonding ? > Send your /etc/network/interfaces details. > If bonding is active, check if the mode is correct in /proc/net/bonding. > We encountered differences between /etc/network/interfaces setup and resulting mode. > Also, check your switch configuration, VLAN setup, MTU etc. > Yes, sorry about that. I have double checked the switch and all 3 node SFP+ port have the same configuration. > > /etc/network/interfaces in proxmox1 node: > auto lo > iface lo inet loopback > iface eth0 inet manual > iface eth1 inet manual > iface eth2 inet manual > iface eth3 inet manual > iface eth4 inet manual > iface eth5 inet manual > > auto vmbr10 > iface vmbr10 inet static > address 192.168.10.201 > netmask 255.255.255.0 > bridge_ports eth3 > bridge_stp off > bridge_fd 0 > > auto vmbr0 > iface vmbr0 inet static > address 192.168.0.201 > netmask 255.255.255.0 > gateway 192.168.0.100 > bridge_ports eth4 > bridge_stp off > bridge_fd 0 > > auto eth4.100 > iface eth4.100 inet static > address 10.0.2.1 > netmask 255.255.255.0 > up ip addr add 10.0.3.1/24 dev eth4.100 > > Cluster is running on vmbr0 network (192.168.0.0/24) > > Cheers > > > Marcus Haarmann > > > Von: "Eneko Lacunza" > > An: "pve-user" > > Gesendet: Dienstag, 4. Dezember 2018 15:57:10 > Betreff: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? > > Hi all, > > We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph > hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the > three Proxmox nodes is new too :) > > Generally all was good and VMs are working well. :-) > > BUT, we have some problems with the cluster; promxox1 node joins and > then after about 4 minutes drops from the cluster. > > All multicast tests > https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast > run fine except the last one: > > *** proxmox1: > > root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 > > proxmox3 : waiting for response msg > > proxmox4 : waiting for response msg > > proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox3 : given amount of query messages was sent > > proxmox4 : given amount of query messages was sent > > proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.073/0.184/0.390/0.061 > > proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.092/0.207/0.421/0.068 > > proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.049/0.167/0.369/0.059 > > proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.063/0.185/0.386/0.064 > > > *** proxmox3: > > root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 > > proxmox1 : waiting for response msg > > proxmox4 : waiting for response msg > > proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox1 : waiting for response msg > > proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox4 : given amount of query messages was sent > > proxmox1 : given amount of query messages was sent > > proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.083/0.193/1.030/0.055 > > proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.102/0.209/1.050/0.054 > > proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.041/0.108/0.172/0.026 > > proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.123/0.190/0.030 > > > *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 > > proxmox1 : waiting for response msg > > proxmox3 : waiting for response msg > > proxmox1 : waiting for response msg > > proxmox3 : waiting for response msg > > proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging > > proxmox1 : given amount of query messages was sent > > proxmox3 : given amount of query messages was sent > > proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.085/0.188/0.356/0.040 > > proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.114/0.208/0.377/0.041 > > proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.117/0.289/0.023 > > proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.064/0.134/0.290/0.026 > > > Ok, so it seems we have a network problem on proxmox1 node. Network > cards are as follows: > > - proxmox1: Intel X540 (10Gtek) > - proxmox3: Intel X710 (Intel) > - proxmox4: Intel X710 (Intel) > > Switch is Dell N1224T-ON. > > Does anyone have experience with Intel X540 chip network cards or Linux > ixgbe network driver or 10Gtek manufacturer? > > If we change corosync communication to 1 Gbit network cards (broadcom) > connected to an old HPE 1800-24G switch, cluster is stable... > > We also have a running cluster with Dell n1224T-ON switch and X710 > network cards without issues. > > Thanks a lot > Eneko > > > > > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > -- > Zuzendari Teknikoa / Director T?cnico > Binovo IT Human Project, S.L. > Telf. 943569206 > Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) > www.binovo.es > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From gilberto.nunes32 at gmail.com Fri Dec 7 18:34:42 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Fri, 7 Dec 2018 15:34:42 -0200 Subject: [PVE-User] Proxmox Ceph workload issue.... Message-ID: Hi there I have a 6 nodes Ceph cluster make with Proxmox. In order to recude the workload rebalance, I activate some features, like this: ceph tell osd.* injectargs '--osd-max-backfills 1' ceph tell osd.* injectargs '--osd-max-recovery-threads 1' ceph tell osd.* injectargs '--osd-recovery-op-priority 1' ceph tell osd.* injectargs '--osd-client-op-priority 63' ceph tell osd.* injectargs '--osd-recovery-max-active 1' ceph tell osd.* injectargs '--osd-max-scrubs 1' ceph tell osd.* injectargs '--osd-scrub-max-interval 4838400' ceph tell osd.* injectargs '--osd-scrub-min-interval 2419200' ceph tell osd.* injectargs '--osd-deep-scrub-interval 2419200' ceph tell osd.* injectargs '--osd-scrub-interval-randomize-ratio 1.0' ceph tell osd.* injectargs '--osd-disk-thread-ioprio-class idle' ceph tell osd.* injectargs '--osd-disk-thread-ioprio-priority 0' ceph tell osd.* injectargs '--osd-scrub-chunk-max 1' ceph tell osd.* injectargs '--osd-scrub-chunk-min 1' ceph tell osd.* injectargs '--osd-deep-scrub-stride 1048576' ceph tell osd.* injectargs '--osd-scrub-load-threshold 5.0' ceph tell osd.* injectargs '--osd-scrub-sleep 0.1' ceph osd set nodeep-scrub Is there any troble if I leave this feature like above? Thanks a lot --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From chibi at gol.com Mon Dec 10 04:45:27 2018 From: chibi at gol.com (Christian Balzer) Date: Mon, 10 Dec 2018 12:45:27 +0900 Subject: [PVE-User] HA scalability and predictability Message-ID: <20181210124527.5e777e1b@batzmaru.gol.ad.jp> Hello, still investigating PVE as a large ganeti cluster replacement. Some years ago we did our owh HA VM cluster based on Pacemaker, libvirt (KVM) and DRBD. While this worked well it also showed the limitations in Pacemaker and LRM in particular. Things got pretty sluggish with 60VMs and a total of 120 resources. This cluster will have about 800VMs, has anybody done this number of HA VMs with PVE and what's their experience? Secondly, it is an absolute requirement that a node failure will result in a predictable and restricted failover. I.e. the cluster will have a n+1 (or n+2) redundancy with at least one node being essentially a hot spare. Failovers should only go to the spare(s), never another compute node. I presume a "node1:2,node8:1" and "restricted 1" should do the trick here. Regards, Christian -- Christian Balzer Network/Systems Engineer chibi at gol.com Rakuten Communications From aderumier at odiso.com Mon Dec 10 07:39:59 2018 From: aderumier at odiso.com (Alexandre DERUMIER) Date: Mon, 10 Dec 2018 07:39:59 +0100 (CET) Subject: [PVE-User] HA scalability and predictability In-Reply-To: <20181210124527.5e777e1b@batzmaru.gol.ad.jp> References: <20181210124527.5e777e1b@batzmaru.gol.ad.jp> Message-ID: <1229869665.1606705.1544423999722.JavaMail.zimbra@oxygem.tv> >>I presume a "node1:2,node8:1" and "restricted 1" should do the trick here. restricted is only used, if both both node1 && node8 are down, the vm don't go to another node. But the weight indeed, should do the trick. for example, n nodes with :2 , and spare(s) node(s) with :1 ----- Mail original ----- De: "Christian Balzer" ?: "proxmoxve" Envoy?: Lundi 10 D?cembre 2018 04:45:27 Objet: [PVE-User] HA scalability and predictability Hello, still investigating PVE as a large ganeti cluster replacement. Some years ago we did our owh HA VM cluster based on Pacemaker, libvirt (KVM) and DRBD. While this worked well it also showed the limitations in Pacemaker and LRM in particular. Things got pretty sluggish with 60VMs and a total of 120 resources. This cluster will have about 800VMs, has anybody done this number of HA VMs with PVE and what's their experience? Secondly, it is an absolute requirement that a node failure will result in a predictable and restricted failover. I.e. the cluster will have a n+1 (or n+2) redundancy with at least one node being essentially a hot spare. Failovers should only go to the spare(s), never another compute node. I presume a "node1:2,node8:1" and "restricted 1" should do the trick here. Regards, Christian -- Christian Balzer Network/Systems Engineer chibi at gol.com Rakuten Communications _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From chibi at gol.com Mon Dec 10 08:18:46 2018 From: chibi at gol.com (Christian Balzer) Date: Mon, 10 Dec 2018 16:18:46 +0900 Subject: [PVE-User] HA scalability and predictability In-Reply-To: <1229869665.1606705.1544423999722.JavaMail.zimbra@oxygem.tv> References: <20181210124527.5e777e1b@batzmaru.gol.ad.jp> <1229869665.1606705.1544423999722.JavaMail.zimbra@oxygem.tv> Message-ID: <20181210161846.66ff474e@batzmaru.gol.ad.jp> On Mon, 10 Dec 2018 07:39:59 +0100 (CET) Alexandre DERUMIER wrote: > >>I presume a "node1:2,node8:1" and "restricted 1" should do the trick here. > > restricted is only used, if both both node1 && node8 are down, the vm don't go to another node. > Yes, that's exactly what I want here. Any other (non-listed) node would be already at capacity and migrating the VMs there would be potentially fatal. A double node failure is something that will need to be dealt with manually. > But the weight indeed, should do the trick. > > for example, n nodes with :2 , and spare(s) node(s) with :1 > > ----- Mail original ----- > De: "Christian Balzer" > ?: "proxmoxve" > Envoy?: Lundi 10 D?cembre 2018 04:45:27 > Objet: [PVE-User] HA scalability and predictability > > Hello, > > still investigating PVE as a large ganeti cluster replacement. > > Some years ago we did our owh HA VM cluster based on Pacemaker, libvirt > (KVM) and DRBD. > While this worked well it also showed the limitations in Pacemaker and LRM > in particular. Things got pretty sluggish with 60VMs and a total of 120 > resources. > This cluster will have about 800VMs, has anybody done this number of HA > VMs with PVE and what's their experience? > > Secondly, it is an absolute requirement that a node failure will result in > a predictable and restricted failover. > I.e. the cluster will have a n+1 (or n+2) redundancy with at least one > node being essentially a hot spare. > Failovers should only go to the spare(s), never another compute node. > > I presume a "node1:2,node8:1" and "restricted 1" should do the trick here. > > Regards, > > Christian -- Christian Balzer Network/Systems Engineer chibi at gol.com Rakuten Communications From a.antreich at proxmox.com Mon Dec 10 10:33:32 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Mon, 10 Dec 2018 10:33:32 +0100 Subject: [PVE-User] Proxmox Ceph workload issue.... In-Reply-To: References: Message-ID: <20181210093332.h4vmcrohkvi526q3@dona.proxmox.com> Hello Gilberto, On Fri, Dec 07, 2018 at 03:34:42PM -0200, Gilberto Nunes wrote: > Hi there > > I have a 6 nodes Ceph cluster make with Proxmox. > In order to recude the workload rebalance, I activate some features, like > this: > > ceph tell osd.* injectargs '--osd-max-backfills 1' > ceph tell osd.* injectargs '--osd-max-recovery-threads 1' > ceph tell osd.* injectargs '--osd-recovery-op-priority 1' > ceph tell osd.* injectargs '--osd-client-op-priority 63' > ceph tell osd.* injectargs '--osd-recovery-max-active 1' > ceph tell osd.* injectargs '--osd-max-scrubs 1' > ceph tell osd.* injectargs '--osd-scrub-max-interval 4838400' > ceph tell osd.* injectargs '--osd-scrub-min-interval 2419200' > ceph tell osd.* injectargs '--osd-deep-scrub-interval 2419200' > ceph tell osd.* injectargs '--osd-scrub-interval-randomize-ratio 1.0' > ceph tell osd.* injectargs '--osd-disk-thread-ioprio-class idle' > ceph tell osd.* injectargs '--osd-disk-thread-ioprio-priority 0' > ceph tell osd.* injectargs '--osd-scrub-chunk-max 1' > ceph tell osd.* injectargs '--osd-scrub-chunk-min 1' > ceph tell osd.* injectargs '--osd-deep-scrub-stride 1048576' > ceph tell osd.* injectargs '--osd-scrub-load-threshold 5.0' > ceph tell osd.* injectargs '--osd-scrub-sleep 0.1' > ceph osd set nodeep-scrub > > Is there any troble if I leave this feature like above? Well, you need to monitor your cluster and see if it helps or does any harm. Also check with these settings, if your cluster will ever do scrubbing (or finish). -- Cheers, Alwin From gilberto.nunes32 at gmail.com Mon Dec 10 10:51:03 2018 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Mon, 10 Dec 2018 07:51:03 -0200 Subject: [PVE-User] Proxmox Ceph workload issue.... In-Reply-To: <20181210093332.h4vmcrohkvi526q3@dona.proxmox.com> References: <20181210093332.h4vmcrohkvi526q3@dona.proxmox.com> Message-ID: Right now, the cluster itself are rebalancing for about 4 days... But since the third day, no performance impact in the environment, because I ran the command mentioned in the previously mail. Better slow rebalance then poor performance... --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em seg, 10 de dez de 2018 ?s 07:33, Alwin Antreich escreveu: > Hello Gilberto, > > On Fri, Dec 07, 2018 at 03:34:42PM -0200, Gilberto Nunes wrote: > > Hi there > > > > I have a 6 nodes Ceph cluster make with Proxmox. > > In order to recude the workload rebalance, I activate some features, like > > this: > > > > ceph tell osd.* injectargs '--osd-max-backfills 1' > > ceph tell osd.* injectargs '--osd-max-recovery-threads 1' > > ceph tell osd.* injectargs '--osd-recovery-op-priority 1' > > ceph tell osd.* injectargs '--osd-client-op-priority 63' > > ceph tell osd.* injectargs '--osd-recovery-max-active 1' > > ceph tell osd.* injectargs '--osd-max-scrubs 1' > > ceph tell osd.* injectargs '--osd-scrub-max-interval 4838400' > > ceph tell osd.* injectargs '--osd-scrub-min-interval 2419200' > > ceph tell osd.* injectargs '--osd-deep-scrub-interval 2419200' > > ceph tell osd.* injectargs '--osd-scrub-interval-randomize-ratio 1.0' > > ceph tell osd.* injectargs '--osd-disk-thread-ioprio-class idle' > > ceph tell osd.* injectargs '--osd-disk-thread-ioprio-priority 0' > > ceph tell osd.* injectargs '--osd-scrub-chunk-max 1' > > ceph tell osd.* injectargs '--osd-scrub-chunk-min 1' > > ceph tell osd.* injectargs '--osd-deep-scrub-stride 1048576' > > ceph tell osd.* injectargs '--osd-scrub-load-threshold 5.0' > > ceph tell osd.* injectargs '--osd-scrub-sleep 0.1' > > ceph osd set nodeep-scrub > > > > Is there any troble if I leave this feature like above? > Well, you need to monitor your cluster and see if it helps or does any > harm. Also check with these settings, if your cluster will ever do > scrubbing (or finish). > > -- > Cheers, > Alwin > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gaio at sv.lnf.it Mon Dec 10 17:04:37 2018 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Mon, 10 Dec 2018 17:04:37 +0100 Subject: [PVE-User] Remove a MON from a cluster... Message-ID: <20181210160437.GT3315@sv.lnf.it> Seems sufficiently clear, but i prefere to ask here. I need to remove a node from a PVE 4.4 cluster; the node was only a MON and OSDs node. I have to: 0) remove every single OSD, remove the host from the crushmap (done). 1) remove the node from the MONs (via web interface, or via: pveceph destroymon ) 2) edit by hand (seems there's no way in 4.4) /etc/pve/storage.cfg, adding/removing in 'monhost'; Q: some 'restart' are needed? 3) shut down the node, and apply: https://pve.proxmox.com/wiki/Proxmox_VE_4.x_Cluster#Remove_a_cluster_node Right? -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From miguel_3_gonzalez at yahoo.es Sat Dec 15 13:02:26 2018 From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=) Date: Sat, 15 Dec 2018 13:02:26 +0100 Subject: [PVE-User] =?utf-8?q?Can=C2=B4t_ping_to_the_outside_-_OVH_Proxmox?= =?utf-8?q?_5=2E3?= Message-ID: Hi, ?? I am migrating some VMs from a Soyoustart (OVH) Proxmox 5.1 to a brand new Proxmox 5.3 server (again Soyoustart). ?? I have followed the instructions from OVH and Proxmox and I can ping from the VM to the host and the gateway and I can ping from the host to the VM. But I can?t ping the DNS server or anything outside the host machine (i.e.: the legacy Proxmox host). ? Some people suggest to enable ip forwarding, but I don?t have enabled on the legacy server. ? But I enable it anyway echo 1 > /proc/sys/net/ipv4/ip_forward ?? ? and nothing happens. ? iptables seems to be turned off both in host and vm: ?? iptables -L Chain INPUT (policy ACCEPT) target???? prot opt source?????????????? destination???????? Chain FORWARD (policy ACCEPT) target???? prot opt source?????????????? destination???????? Chain OUTPUT (policy ACCEPT) target???? prot opt source?????????????? destination???????? So I?m out of ideas here Any suggestion? Miguel --- This email has been checked for viruses by AVG. https://www.avg.com From f.thommen at dkfz-heidelberg.de Sun Dec 16 14:28:19 2018 From: f.thommen at dkfz-heidelberg.de (Frank Thommen) Date: Sun, 16 Dec 2018 14:28:19 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration Message-ID: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> Hi, I understand that with the new PVE release PVE hosts (hypervisors) can be used as Ceph servers. But it's not clear to me if (or when) that makes sense. Do I really want to have Ceph MDS/OSD on the same hardware as my hypervisors? Doesn't that a) accumulate multiple POFs on the same hardware and b) occupy computing resources (CPU, RAM), that I'd rather use for my VMs and containers? Wouldn't I rather want to have a separate Ceph cluster? Or didn't I get the point of the Ceph integration? Cheers Frank From alwin at antreich.com Sun Dec 16 15:39:05 2018 From: alwin at antreich.com (Alwin Antreich) Date: Sun, 16 Dec 2018 15:39:05 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> Message-ID: <20181216142629.npzu6ykxzk4hue2v@ipnerd.net> Hello Frank, On Sun, Dec 16, 2018 at 02:28:19PM +0100, Frank Thommen wrote: > Hi, > > I understand that with the new PVE release PVE hosts (hypervisors) can be > used as Ceph servers. But it's not clear to me if (or when) that makes > sense. Do I really want to have Ceph MDS/OSD on the same hardware as my > hypervisors? Doesn't that a) accumulate multiple POFs on the same hardware > and b) occupy computing resources (CPU, RAM), that I'd rather use for my VMs > and containers? Wouldn't I rather want to have a separate Ceph cluster? The integration of Ceph services in PVE started with Proxmox VE 3.0. With PVE 5.3 (current) we added CephFS services to the PVE. So you can run a hyper-converged Ceph with RBD/CephFS on the same servers as your VM/CT. a) can you please be more specific in what you see as multiple point of failures? b) depends on the workload of your nodes. Modern server hardware has enough power to be able to run multiple services. It all comes down to have enough resources for each domain (eg. Ceph, KVM, CT, host). I recommend to use a simple calculation for the start, just to get a direction. In principle: ==CPU== core='CPU with HT on' * reserve a core for each Ceph daemon (preferable on the same NUMA as the network; higher frequency is better) * one core for the network card (higher frequency = lower latency) * rest of the cores for OS (incl. monitoring, backup, ...), KVM/CT usage * don't overcommit ==Memory== * 1 GB per TB of used disk space on an OSD (more on recovery) * enough memory for KVM/CT * free memory for OS, backup, monitoring, live migration * don't overcommit ==Disk== * one OSD daemon per disk, even disk sizes throughout the cluster * more disks, more hosts, better distribution ==Network== * at least 10 GbE for storage traffic (more the better), see our benchmark paper https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/ * separate networks, cluster, storage, client traffic, additional for separate the migration network from any other * use to physical networks for corosync (ring0 & ring1) This list doesn't cover every aspect (eg. how much failure is allowed), but I think it is a good start. With the above points for the sizing of your cluster, the question of a separation of a hyper-converged service might be a little easier. -- Cheers, Alwin From yannis.milios at gmail.com Sun Dec 16 15:41:48 2018 From: yannis.milios at gmail.com (Yannis Milios) Date: Sun, 16 Dec 2018 14:41:48 +0000 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> Message-ID: That's up to you to decide, PVE supports both hyper-converged setups (where compute and storage nodes share the same hardware) and scenarios where compute/storage nodes are separate. You can choose for example to have 3 nodes in a PVE cluster, acting as compute nodes and 3 separate nodes for the Ceph cluster, acting as storage nodes. The main difference is that the costs involved in this scenario are much higher compared to the hyper-converged setup. It's also possible to manage Ceph clusters by using PVE gui (i.e use the webgui for the Ceph tasks only). Clearly in a hyper-converged setup, as the host(s) resources will be shared between the compute and storage layer, proper measures in the design and capacity planning must be taken in order to avoid downgraded performance. For example ZFS is well known to consume high amounts of RAM.In a hyper-converged setup, if improperly configured, it may consume half of host RAM, potentially leaving VMs under pressure .. Yannis On Sun, 16 Dec 2018 at 13:28, Frank Thommen wrote: > Hi, > > I understand that with the new PVE release PVE hosts (hypervisors) can > be used as Ceph servers. But it's not clear to me if (or when) that > makes sense. Do I really want to have Ceph MDS/OSD on the same hardware > as my hypervisors? Doesn't that a) accumulate multiple POFs on the same > hardware and b) occupy computing resources (CPU, RAM), that I'd rather > use for my VMs and containers? Wouldn't I rather want to have a > separate Ceph cluster? > > Or didn't I get the point of the Ceph integration? > > Cheers > Frank > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From jeremy at jcarnus.fr Sun Dec 16 15:42:49 2018 From: jeremy at jcarnus.fr (=?UTF-8?Q?J=C3=A9r=C3=A9my_Carnus?=) Date: Sun, 16 Dec 2018 09:42:49 -0500 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration Message-ID: J?r?my Garnis eek00opppp ________________________________ De: Yannis Milios Envoy?: dimanche 16 d?cembre 2018 09:41 ?: PVE User List Objet: Re: [PVE-User] (Very) basic question regarding PVE Ceph integration That's up to you to decide, PVE supports both hyper-converged setups (where compute and storage nodes share the same hardware) and scenarios where compute/storage nodes are separate. You can choose for example to have 3 nodes in a PVE cluster, acting as compute nodes and 3 separate nodes for the Ceph cluster, acting as storage nodes. The main difference is that the costs involved in this scenario are much higher compared to the hyper-converged setup. It's also possible to manage Ceph clusters by using PVE gui (i.e use the webgui for the Ceph tasks only). Clearly in a hyper-converged setup, as the host(s) resources will be shared between the compute and storage layer, proper measures in the design and capacity planning must be taken in order to avoid downgraded performance. For example ZFS is well known to consume high amounts of RAM.In a hyper-converged setup, if improperly configured, it may consume half of host RAM, potentially leaving VMs under pressure .. Yannis On Sun, 16 Dec 2018 at 13:28, Frank Thommen wrote: > Hi, > > I understand that with the new PVE release PVE hosts (hypervisors) can > be used as Ceph servers.? But it's not clear to me if (or when) that > makes sense.? Do I really want to have Ceph MDS/OSD on the same hardware > as my hypervisors?? Doesn't that a) accumulate multiple POFs on the same > hardware and b) occupy computing resources (CPU, RAM), that I'd rather > use for my VMs and containers?? Wouldn't I rather want to have a > separate Ceph cluster? > > Or didn't I get the point of the Ceph integration? > > Cheers > Frank > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From f.thommen at dkfz-heidelberg.de Sun Dec 16 17:16:50 2018 From: f.thommen at dkfz-heidelberg.de (Frank Thommen) Date: Sun, 16 Dec 2018 17:16:50 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: <20181216142629.npzu6ykxzk4hue2v@ipnerd.net> References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> <20181216142629.npzu6ykxzk4hue2v@ipnerd.net> Message-ID: Hi Alwin, On 16/12/18 15:39, Alwin Antreich wrote: > Hello Frank, > > On Sun, Dec 16, 2018 at 02:28:19PM +0100, Frank Thommen wrote: >> Hi, >> >> I understand that with the new PVE release PVE hosts (hypervisors) can be >> used as Ceph servers. But it's not clear to me if (or when) that makes >> sense. Do I really want to have Ceph MDS/OSD on the same hardware as my >> hypervisors? Doesn't that a) accumulate multiple POFs on the same hardware >> and b) occupy computing resources (CPU, RAM), that I'd rather use for my VMs >> and containers? Wouldn't I rather want to have a separate Ceph cluster? > The integration of Ceph services in PVE started with Proxmox VE 3.0. > With PVE 5.3 (current) we added CephFS services to the PVE. So you can > run a hyper-converged Ceph with RBD/CephFS on the same servers as your > VM/CT. > > a) can you please be more specific in what you see as multiple point of > failures? not only I run the hypervisor which controls containers and virtual machines on the server, but also the fileservice which is used to store the VM and container images. > b) depends on the workload of your nodes. Modern server hardware has > enough power to be able to run multiple services. It all comes down to > have enough resources for each domain (eg. Ceph, KVM, CT, host). > > I recommend to use a simple calculation for the start, just to get a > direction. > > In principle: > > ==CPU== > core='CPU with HT on' > > * reserve a core for each Ceph daemon > (preferable on the same NUMA as the network; higher frequency is > better) > * one core for the network card (higher frequency = lower latency) > * rest of the cores for OS (incl. monitoring, backup, ...), KVM/CT usage > * don't overcommit > > ==Memory== > * 1 GB per TB of used disk space on an OSD (more on recovery) > * enough memory for KVM/CT > * free memory for OS, backup, monitoring, live migration > * don't overcommit > > ==Disk== > * one OSD daemon per disk, even disk sizes throughout the cluster > * more disks, more hosts, better distribution > > ==Network== > * at least 10 GbE for storage traffic (more the better), > see our benchmark paper > https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/ > * separate networks, cluster, storage, client traffic, > additional for separate the migration network from any other > * use to physical networks for corosync (ring0 & ring1) > > This list doesn't cover every aspect (eg. how much failure is allowed), > but I think it is a good start. With the above points for the sizing of > your cluster, the question of a separation of a hyper-converged service > might be a little easier. Thanks a lot. This sure helps in our planning. frank > > -- > Cheers, > Alwin > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From miguel_3_gonzalez at yahoo.es Sun Dec 16 18:30:49 2018 From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=) Date: Sun, 16 Dec 2018 18:30:49 +0100 Subject: [PVE-User] =?utf-8?q?Can=C2=B4t_ping_to_the_outside_-_OVH_Proxmo?= =?utf-8?q?x_5=2E3?= In-Reply-To: References: <33e2f0b7-216f-ff88-278a-de7d0cc9ee27@it-functions.nl> <99031130-08dc-e84f-e38f-01b209110941@yahoo.es> Message-ID: <1f3e91f2-e083-af11-1380-7c97ff6e666e@yahoo.es> Hi Stephan, ? I use public failover IP addresses. I ask about your gateway configuration, you use: ? 91.121.183.137 ? and as far as I know, the gateway must be the public IP address of the host ending with .254. That?s what OVH says in their docs. ? Thanks! On 12/15/18 2:43 PM, Stephan Leemburg wrote: > OVH Requires you to route traffic from VM's via the IP address of your > hardware. > > So 137 is the ip address of the hardware. > > Do you use any public ip addresses on your soyoustart system? > > Or just private range and then send them out via NAT? > > Met vriendelijke groet, > Stephan Leemburg > IT Functions > > e: sleemburg at it-functions.nl > p: +31 (0)71 889 23 33 > m: +31(0)6 83 22 30 69 > kvk: 27313647 > > On 15-12-18 14:39, Miguel Gonz?lez wrote: >> There must be something wrong with the configuration since I have tested >> another server and seems to be fine. >> >> Why do you use 137? In the proxmox docs they say the gateway is >> xxx.xxx.xxx.254 >> >> Thanks! >> >> >> On 12/15/18 2:16 PM, Stephan Leemburg wrote: >>> Did you setup routing correctly within the containers / vm's? >>> OVH/SoYouStart has awkward network routing requirements. >>> >>> I have 2 servers at soyoustart and they do fine with the correct >>> network configuration. >>> >>> Below is an example from one of my containers. >>> >>> Also, it is a good idea to setup a firewall and put your containers on >>> vmbr devices connected to the lan side of your firewall. >>> >>> Then on the lan side you have 'normal' network configurations. >>> >>> The pve has ip address 91.121.183.137 >>> >>> I have a subnet 54.37.62.224/28 on which containers and vm's live. >>> >>> # cat /etc/network/interfaces >>> >>> auto lo >>> iface lo inet loopback >>> >>> auto eth0 >>> iface eth0 inet static >>> ???? address 54.37.62.232 >>> ???? netmask 255.255.255.255 >>> ???? pre-up /etc/network/firewall $IFACE up >>> ???? post-up ip route add 91.121.183.137 dev $IFACE >>> ???? post-up ip route add 54.37.62.224/28 dev $IFACE >>> ???? post-up ip route add default via 91.121.183.137 dev $IFACE >>> ???? post-down ip route del default via 91.121.183.137 dev $IFACE >>> ???? post-down ip route del 54.37.62.224/28 dev $IFACE >>> ???? post-down ip route del 91.121.183.137 dev $IFACE >>> ???? post-down /etc/network/firewall $IFACE down' >>> >>> >>> Met vriendelijke groet, >>> Stephan Leemburg >>> >>> On 15-12-18 13:02, Miguel Gonz?lez wrote: >>>> Hi, >>>> >>>> ???? I am migrating some VMs from a Soyoustart (OVH) Proxmox 5.1 to a >>>> brand new Proxmox 5.3 server (again Soyoustart). >>>> >>>> ???? I have followed the instructions from OVH and Proxmox and I >>>> can ping >>>> from the VM to the host and the gateway and I can ping from the >>>> host to >>>> the VM. But I can?t ping the DNS server or anything outside the host >>>> machine (i.e.: the legacy Proxmox host). >>>> >>>> ??? Some people suggest to enable ip forwarding, but I don?t have >>>> enabled >>>> on the legacy server. >>>> >>>> ??? But I enable it anyway echo 1 > /proc/sys/net/ipv4/ip_forward >>>> ?? ??? and nothing happens. >>>> >>>> ??? iptables seems to be turned off both in host and vm: >>>> >>>> ???? iptables -L >>>> Chain INPUT (policy ACCEPT) >>>> target???? prot opt source?????????????? destination >>>> >>>> Chain FORWARD (policy ACCEPT) >>>> target???? prot opt source?????????????? destination >>>> >>>> Chain OUTPUT (policy ACCEPT) >>>> target???? prot opt source?????????????? destination >>>> >>>> So I?m out of ideas here >>>> >>>> Any suggestion? >>>> >>>> Miguel >>>> >>>> >>>> >>>> --- >>>> This email has been checked for viruses by AVG. >>>> https://www.avg.com >>>> _______________________________________________ >>>> pve-user mailing list >>>> pve-user at pve.proxmox.com >>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From alwin at antreich.com Sun Dec 16 19:47:40 2018 From: alwin at antreich.com (Alwin Antreich) Date: Sun, 16 Dec 2018 19:47:40 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> <20181216142629.npzu6ykxzk4hue2v@ipnerd.net> Message-ID: <20181216184740.uli56kockq6nwuvo@ipnerd.net> On Sun, Dec 16, 2018 at 05:16:50PM +0100, Frank Thommen wrote: > Hi Alwin, > > On 16/12/18 15:39, Alwin Antreich wrote: > > Hello Frank, > > > > On Sun, Dec 16, 2018 at 02:28:19PM +0100, Frank Thommen wrote: > > > Hi, > > > > > > I understand that with the new PVE release PVE hosts (hypervisors) can be > > > used as Ceph servers. But it's not clear to me if (or when) that makes > > > sense. Do I really want to have Ceph MDS/OSD on the same hardware as my > > > hypervisors? Doesn't that a) accumulate multiple POFs on the same hardware > > > and b) occupy computing resources (CPU, RAM), that I'd rather use for my VMs > > > and containers? Wouldn't I rather want to have a separate Ceph cluster? > > The integration of Ceph services in PVE started with Proxmox VE 3.0. > > With PVE 5.3 (current) we added CephFS services to the PVE. So you can > > run a hyper-converged Ceph with RBD/CephFS on the same servers as your > > VM/CT. > > > > a) can you please be more specific in what you see as multiple point of > > failures? > > not only I run the hypervisor which controls containers and virtual machines > on the server, but also the fileservice which is used to store the VM and > container images. Sorry, I am still not quite sure, what your question/concern is. Failure tolerance needs to be planned into the system design, irrespective of service distribution. Proxmox VE has a HA stack that restarts all services from a failed node (if configured) on a other node. https://pve.proxmox.com/pve-docs/chapter-ha-manager.html Ceph does selfhealing (if enough nodes are available) or still works in a degraded state. http://docs.ceph.com/docs/luminous/start/intro/ > > > > b) depends on the workload of your nodes. Modern server hardware has > > enough power to be able to run multiple services. It all comes down to > > have enough resources for each domain (eg. Ceph, KVM, CT, host). > > > > I recommend to use a simple calculation for the start, just to get a > > direction. > > > > In principle: > > > > ==CPU== > > core='CPU with HT on' > > > > * reserve a core for each Ceph daemon > > (preferable on the same NUMA as the network; higher frequency is > > better) > > * one core for the network card (higher frequency = lower latency) > > * rest of the cores for OS (incl. monitoring, backup, ...), KVM/CT usage > > * don't overcommit > > > > ==Memory== > > * 1 GB per TB of used disk space on an OSD (more on recovery) > > * enough memory for KVM/CT > > * free memory for OS, backup, monitoring, live migration > > * don't overcommit > > > > ==Disk== > > * one OSD daemon per disk, even disk sizes throughout the cluster > > * more disks, more hosts, better distribution > > > > ==Network== > > * at least 10 GbE for storage traffic (more the better), > > see our benchmark paper > > https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/ > > * separate networks, cluster, storage, client traffic, > > additional for separate the migration network from any other > > * use to physical networks for corosync (ring0 & ring1) > > > > This list doesn't cover every aspect (eg. how much failure is allowed), > > but I think it is a good start. With the above points for the sizing of > > your cluster, the question of a separation of a hyper-converged service > > might be a little easier. > > Thanks a lot. This sure helps in our planning. > > frank You're welcome. -- Cheers, Alwin From miguel_3_gonzalez at yahoo.es Sun Dec 16 20:23:28 2018 From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=) Date: Sun, 16 Dec 2018 20:23:28 +0100 Subject: [PVE-User] =?utf-8?q?Can=C2=B4t_ping_to_the_outside_-_OVH_Proxmo?= =?utf-8?q?x_5=2E3?= In-Reply-To: References: <33e2f0b7-216f-ff88-278a-de7d0cc9ee27@it-functions.nl> <99031130-08dc-e84f-e38f-01b209110941@yahoo.es> <1f3e91f2-e083-af11-1380-7c97ff6e666e@yahoo.es> Message-ID: Sorry Stephan, I?ve been working with this setup for about 2 years. I am just wondering if in the case of an PVE host IP address like 111.222.333.74 (this last 74 is real) should have my VM?s? gateway with 111.222.333.254 or 111.222.333.137. That?s what I am asking. Right now my legacy server has exactly the same IP address and the new one except that the last .74 is .220. The gateways configured in all VMs running perfectly on that legacy server has also .254 as gateway in their configuration. That?s what confuses me. So summarizing: legacy dedicated server IP: 111.222.333.220 --> All VMs have 111.222.333.254 as gateway new dedicated server IP: 111.222.333.74 --> Configuring 111.222.333.254 in the VM makes reachable the public IP address of the new server and the gateway but I can?t ping to the outside world. I hope this clarifies the situation :) Miguel On 12/16/18 8:03 PM, Stephan Leemburg wrote: > Hi Miguel, > > Yes, on the pve host the OVH gateway is the .254 > > But your containers and vm's on the pve host must use the ip address > of the pve as their default gateway. > > Also you need to assign mac addresses from the ovh control panel if > you are using the public failover ip addresses. > > Kind regards, > Stephan > > On 16-12-18 18:30, Miguel Gonz?lez wrote: >> Hi Stephan, >> >> ?? I use public failover IP addresses. I ask about your gateway >> configuration, you use: >> >> ?? 91.121.183.137 >> >> ?? and as far as I know, the gateway must be the public IP address of >> the >> host ending with .254. That?s what OVH says in their docs. >> >> ?? Thanks! >> >> On 12/15/18 2:43 PM, Stephan Leemburg wrote: >>> OVH Requires you to route traffic from VM's via the IP address of your >>> hardware. >>> >>> So 137 is the ip address of the hardware. >>> >>> Do you use any public ip addresses on your soyoustart system? >>> >>> Or just private range and then send them out via NAT? >>> >>> Met vriendelijke groet, >>> Stephan Leemburg >>> IT Functions >>> >>> e: sleemburg at it-functions.nl >>> p: +31 (0)71 889 23 33 >>> m: +31(0)6 83 22 30 69 >>> kvk: 27313647 >>> >>> On 15-12-18 14:39, Miguel Gonz?lez wrote: >>>> There must be something wrong with the configuration since I have >>>> tested >>>> another server and seems to be fine. >>>> >>>> Why do you use 137? In the proxmox docs they say the gateway is >>>> xxx.xxx.xxx.254 >>>> >>>> Thanks! >>>> >>>> >>>> On 12/15/18 2:16 PM, Stephan Leemburg wrote: >>>>> Did you setup routing correctly within the containers / vm's? >>>>> OVH/SoYouStart has awkward network routing requirements. >>>>> >>>>> I have 2 servers at soyoustart and they do fine with the correct >>>>> network configuration. >>>>> >>>>> Below is an example from one of my containers. >>>>> >>>>> Also, it is a good idea to setup a firewall and put your >>>>> containers on >>>>> vmbr devices connected to the lan side of your firewall. >>>>> >>>>> Then on the lan side you have 'normal' network configurations. >>>>> >>>>> The pve has ip address 91.121.183.137 >>>>> >>>>> I have a subnet 54.37.62.224/28 on which containers and vm's live. >>>>> >>>>> # cat /etc/network/interfaces >>>>> >>>>> auto lo >>>>> iface lo inet loopback >>>>> >>>>> auto eth0 >>>>> iface eth0 inet static >>>>> ????? address 54.37.62.232 >>>>> ????? netmask 255.255.255.255 >>>>> ????? pre-up /etc/network/firewall $IFACE up >>>>> ????? post-up ip route add 91.121.183.137 dev $IFACE >>>>> ????? post-up ip route add 54.37.62.224/28 dev $IFACE >>>>> ????? post-up ip route add default via 91.121.183.137 dev $IFACE >>>>> ????? post-down ip route del default via 91.121.183.137 dev $IFACE >>>>> ????? post-down ip route del 54.37.62.224/28 dev $IFACE >>>>> ????? post-down ip route del 91.121.183.137 dev $IFACE >>>>> ????? post-down /etc/network/firewall $IFACE down' >>>>> >>>>> >>>>> Met vriendelijke groet, >>>>> Stephan Leemburg >>>>> >>>>> On 15-12-18 13:02, Miguel Gonz?lez wrote: >>>>>> Hi, >>>>>> >>>>>> ????? I am migrating some VMs from a Soyoustart (OVH) Proxmox 5.1 >>>>>> to a >>>>>> brand new Proxmox 5.3 server (again Soyoustart). >>>>>> >>>>>> ????? I have followed the instructions from OVH and Proxmox and I >>>>>> can ping >>>>>> from the VM to the host and the gateway and I can ping from the >>>>>> host to >>>>>> the VM. But I can?t ping the DNS server or anything outside the host >>>>>> machine (i.e.: the legacy Proxmox host). >>>>>> >>>>>> ???? Some people suggest to enable ip forwarding, but I don?t have >>>>>> enabled >>>>>> on the legacy server. >>>>>> >>>>>> ???? But I enable it anyway echo 1 > /proc/sys/net/ipv4/ip_forward >>>>>> ??? ??? and nothing happens. >>>>>> >>>>>> ???? iptables seems to be turned off both in host and vm: >>>>>> >>>>>> ????? iptables -L >>>>>> Chain INPUT (policy ACCEPT) >>>>>> target???? prot opt source?????????????? destination >>>>>> >>>>>> Chain FORWARD (policy ACCEPT) >>>>>> target???? prot opt source?????????????? destination >>>>>> >>>>>> Chain OUTPUT (policy ACCEPT) >>>>>> target???? prot opt source?????????????? destination >>>>>> >>>>>> So I?m out of ideas here >>>>>> >>>>>> Any suggestion? >>>>>> >>>>>> Miguel >>>>>> >>>>>> >>>>>> >>>>>> --- >>>>>> This email has been checked for viruses by AVG. >>>>>> https://www.avg.com >>>>>> _______________________________________________ >>>>>> pve-user mailing list >>>>>> pve-user at pve.proxmox.com >>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From sleemburg at it-functions.nl Sun Dec 16 20:36:54 2018 From: sleemburg at it-functions.nl (Stephan Leemburg) Date: Sun, 16 Dec 2018 20:36:54 +0100 Subject: [PVE-User] =?utf-8?q?Can=C2=B4t_ping_to_the_outside_-_OVH_Proxmo?= =?utf-8?q?x_5=2E3?= In-Reply-To: References: <33e2f0b7-216f-ff88-278a-de7d0cc9ee27@it-functions.nl> <99031130-08dc-e84f-e38f-01b209110941@yahoo.es> <1f3e91f2-e083-af11-1380-7c97ff6e666e@yahoo.es> Message-ID: <59fc97e1-dc5a-0fc7-ce2b-da007f117158@it-functions.nl> Hi Miguel, Your new PVE host 111.222.333.74 needs to have 111.222.333.254 as it's default gateway. The VM's need 111.222.333.74 as their default gateway. This is what OVH/SoYouStart requires. Also if you assign public failover ip addresses to your VM's, then you need to generate a mac address for them in the management console of soyoustart and assign that mac address to the public interface of the VM. So consider if you have 16 failover ip addresses in the range 1.2.3.1/28 Then the /etc/network/interfaces of one such vm should be: auto lo iface lo inet loopback auto eth0 iface eth0 inet static ??? address 1.2.3.1 ??? netmask 255.255.255.255 ??? pre-up /etc/network/firewall $IFACE up ??? post-up ip route add 111.222.333.74 dev $IFACE ??? post-up ip route add 1.2.3.1/28 dev $IFACE ??? post-up ip route add default via 91.121.183.137 dev $IFACE ??? post-down ip route del default via 91.121.183.137 dev $IFACE ??? post-down ip route del 1.2.3.1/28 dev $IFACE ??? post-down ip route del 111.222.333.74 dev $IFACE ??? post-down /etc/network/firewall $IFACE down Best regards, Stephan On 16-12-18 20:23, Miguel Gonz?lez wrote: > Sorry Stephan, I?ve been working with this setup for about 2 years. > > I am just wondering if in the case of an PVE host IP address like > > 111.222.333.74 (this last 74 is real) should have my VM?s? gateway with > 111.222.333.254 or 111.222.333.137. > > That?s what I am asking. > > Right now my legacy server has exactly the same IP address and the new > one except that the last .74 is .220. The gateways configured in all VMs > running perfectly on that legacy server has also .254 as gateway in > their configuration. That?s what confuses me. > > So summarizing: > > legacy dedicated server IP: 111.222.333.220 > > --> All VMs have 111.222.333.254 as gateway > > new dedicated server IP: 111.222.333.74 > > --> Configuring 111.222.333.254 in the VM makes reachable the public IP > address of the new server and the gateway but I can?t ping to the > outside world. > > I hope this clarifies the situation :) > > Miguel > > > On 12/16/18 8:03 PM, Stephan Leemburg wrote: >> Hi Miguel, >> >> Yes, on the pve host the OVH gateway is the .254 >> >> But your containers and vm's on the pve host must use the ip address >> of the pve as their default gateway. >> >> Also you need to assign mac addresses from the ovh control panel if >> you are using the public failover ip addresses. >> >> Kind regards, >> Stephan >> >> On 16-12-18 18:30, Miguel Gonz?lez wrote: >>> Hi Stephan, >>> >>> ?? I use public failover IP addresses. I ask about your gateway >>> configuration, you use: >>> >>> ?? 91.121.183.137 >>> >>> ?? and as far as I know, the gateway must be the public IP address of >>> the >>> host ending with .254. That?s what OVH says in their docs. >>> >>> ?? Thanks! >>> >>> On 12/15/18 2:43 PM, Stephan Leemburg wrote: >>>> OVH Requires you to route traffic from VM's via the IP address of your >>>> hardware. >>>> >>>> So 137 is the ip address of the hardware. >>>> >>>> Do you use any public ip addresses on your soyoustart system? >>>> >>>> Or just private range and then send them out via NAT? >>>> >>>> Met vriendelijke groet, >>>> Stephan Leemburg >>>> IT Functions >>>> >>>> e: sleemburg at it-functions.nl >>>> p: +31 (0)71 889 23 33 >>>> m: +31(0)6 83 22 30 69 >>>> kvk: 27313647 >>>> >>>> On 15-12-18 14:39, Miguel Gonz?lez wrote: >>>>> There must be something wrong with the configuration since I have >>>>> tested >>>>> another server and seems to be fine. >>>>> >>>>> Why do you use 137? In the proxmox docs they say the gateway is >>>>> xxx.xxx.xxx.254 >>>>> >>>>> Thanks! >>>>> >>>>> >>>>> On 12/15/18 2:16 PM, Stephan Leemburg wrote: >>>>>> Did you setup routing correctly within the containers / vm's? >>>>>> OVH/SoYouStart has awkward network routing requirements. >>>>>> >>>>>> I have 2 servers at soyoustart and they do fine with the correct >>>>>> network configuration. >>>>>> >>>>>> Below is an example from one of my containers. >>>>>> >>>>>> Also, it is a good idea to setup a firewall and put your >>>>>> containers on >>>>>> vmbr devices connected to the lan side of your firewall. >>>>>> >>>>>> Then on the lan side you have 'normal' network configurations. >>>>>> >>>>>> The pve has ip address 91.121.183.137 >>>>>> >>>>>> I have a subnet 54.37.62.224/28 on which containers and vm's live. >>>>>> >>>>>> # cat /etc/network/interfaces >>>>>> >>>>>> auto lo >>>>>> iface lo inet loopback >>>>>> >>>>>> auto eth0 >>>>>> iface eth0 inet static >>>>>> ????? address 54.37.62.232 >>>>>> ????? netmask 255.255.255.255 >>>>>> ????? pre-up /etc/network/firewall $IFACE up >>>>>> ????? post-up ip route add 91.121.183.137 dev $IFACE >>>>>> ????? post-up ip route add 54.37.62.224/28 dev $IFACE >>>>>> ????? post-up ip route add default via 91.121.183.137 dev $IFACE >>>>>> ????? post-down ip route del default via 91.121.183.137 dev $IFACE >>>>>> ????? post-down ip route del 54.37.62.224/28 dev $IFACE >>>>>> ????? post-down ip route del 91.121.183.137 dev $IFACE >>>>>> ????? post-down /etc/network/firewall $IFACE down' >>>>>> >>>>>> >>>>>> Met vriendelijke groet, >>>>>> Stephan Leemburg >>>>>> >>>>>> On 15-12-18 13:02, Miguel Gonz?lez wrote: >>>>>>> Hi, >>>>>>> >>>>>>> ????? I am migrating some VMs from a Soyoustart (OVH) Proxmox 5.1 >>>>>>> to a >>>>>>> brand new Proxmox 5.3 server (again Soyoustart). >>>>>>> >>>>>>> ????? I have followed the instructions from OVH and Proxmox and I >>>>>>> can ping >>>>>>> from the VM to the host and the gateway and I can ping from the >>>>>>> host to >>>>>>> the VM. But I can?t ping the DNS server or anything outside the host >>>>>>> machine (i.e.: the legacy Proxmox host). >>>>>>> >>>>>>> ???? Some people suggest to enable ip forwarding, but I don?t have >>>>>>> enabled >>>>>>> on the legacy server. >>>>>>> >>>>>>> ???? But I enable it anyway echo 1 > /proc/sys/net/ipv4/ip_forward >>>>>>> ??? ??? and nothing happens. >>>>>>> >>>>>>> ???? iptables seems to be turned off both in host and vm: >>>>>>> >>>>>>> ????? iptables -L >>>>>>> Chain INPUT (policy ACCEPT) >>>>>>> target???? prot opt source?????????????? destination >>>>>>> >>>>>>> Chain FORWARD (policy ACCEPT) >>>>>>> target???? prot opt source?????????????? destination >>>>>>> >>>>>>> Chain OUTPUT (policy ACCEPT) >>>>>>> target???? prot opt source?????????????? destination >>>>>>> >>>>>>> So I?m out of ideas here >>>>>>> >>>>>>> Any suggestion? >>>>>>> >>>>>>> Miguel >>>>>>> >>>>>>> >>>>>>> >>>>>>> --- >>>>>>> This email has been checked for viruses by AVG. >>>>>>> https://www.avg.com >>>>>>> _______________________________________________ >>>>>>> pve-user mailing list >>>>>>> pve-user at pve.proxmox.com >>>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From dorsyka at yahoo.com Mon Dec 17 01:33:02 2018 From: dorsyka at yahoo.com (dORSY) Date: Mon, 17 Dec 2018 00:33:02 +0000 (UTC) Subject: [PVE-User] =?utf-8?q?Can=C2=B4t_ping_to_the_outside_-_OVH_Proxmo?= =?utf-8?q?x_5=2E3?= In-Reply-To: <59fc97e1-dc5a-0fc7-ce2b-da007f117158@it-functions.nl> References: <33e2f0b7-216f-ff88-278a-de7d0cc9ee27@it-functions.nl> <99031130-08dc-e84f-e38f-01b209110941@yahoo.es> <1f3e91f2-e083-af11-1380-7c97ff6e666e@yahoo.es> <59fc97e1-dc5a-0fc7-ce2b-da007f117158@it-functions.nl> Message-ID: <404865576.5259566.1545006782663@mail.yahoo.com> I use a much simpler solution: 2: ens18: mtu 1500 qdisc fq_codel state UP group default qlen 1000 ??? link/ether my:ma:ca:dd:re:ss brd ff:ff:ff:ff:ff:ff ??? inet xx.yy.zz.235/32 brd?xx.yy.zz.235/32 scope global ens18 ?????? valid_lft forever preferred_lft forever # ip ro default via xx.yy.zz.235 dev ens18 So basically your own /32 IP is your GW. Then the soyoustart switch does the job if you registered the MAC for your VM. Also works for windows. On Sunday, 16 December 2018, 20:37:05 CET, Stephan Leemburg wrote: Hi Miguel, Your new PVE host 111.222.333.74 needs to have 111.222.333.254 as it's default gateway. The VM's need 111.222.333.74 as their default gateway. This is what OVH/SoYouStart requires. Also if you assign public failover ip addresses to your VM's, then you need to generate a mac address for them in the management console of soyoustart and assign that mac address to the public interface of the VM. So consider if you have 16 failover ip addresses in the range 1.2.3.1/28 Then the /etc/network/interfaces of one such vm should be: auto lo iface lo inet loopback auto eth0 iface eth0 inet static ??? address 1.2.3.1 ??? netmask 255.255.255.255 ??? pre-up /etc/network/firewall $IFACE up ??? post-up ip route add 111.222.333.74 dev $IFACE ??? post-up ip route add 1.2.3.1/28 dev $IFACE ??? post-up ip route add default via 91.121.183.137 dev $IFACE ??? post-down ip route del default via 91.121.183.137 dev $IFACE ??? post-down ip route del 1.2.3.1/28 dev $IFACE ??? post-down ip route del 111.222.333.74 dev $IFACE ??? post-down /etc/network/firewall $IFACE down Best regards, Stephan On 16-12-18 20:23, Miguel Gonz?lez wrote: > Sorry Stephan, I?ve been working with this setup for about 2 years. > > I am just wondering if in the case of an PVE host IP address like > > 111.222.333.74 (this last 74 is real) should have my VM?s? gateway with > 111.222.333.254 or 111.222.333.137. > > That?s what I am asking. > > Right now my legacy server has exactly the same IP address and the new > one except that the last .74 is .220. The gateways configured in all VMs > running perfectly on that legacy server has also .254 as gateway in > their configuration. That?s what confuses me. > > So summarizing: > > legacy dedicated server IP: 111.222.333.220 > > --> All VMs have 111.222.333.254 as gateway > > new dedicated server IP: 111.222.333.74 > > --> Configuring 111.222.333.254 in the VM makes reachable the public IP > address of the new server and the gateway but I can?t ping to the > outside world. > > I hope this clarifies the situation :) > > Miguel > > > On 12/16/18 8:03 PM, Stephan Leemburg wrote: >> Hi Miguel, >> >> Yes, on the pve host the OVH gateway is the .254 >> >> But your containers and vm's on the pve host must use the ip address >> of the pve as their default gateway. >> >> Also you need to assign mac addresses from the ovh control panel if >> you are using the public failover ip addresses. >> >> Kind regards, >> Stephan >> >> On 16-12-18 18:30, Miguel Gonz?lez wrote: >>> Hi Stephan, >>> >>>? ?? I use public failover IP addresses. I ask about your gateway >>> configuration, you use: >>> >>>? ?? 91.121.183.137 >>> >>>? ?? and as far as I know, the gateway must be the public IP address of >>> the >>> host ending with .254. That?s what OVH says in their docs. >>> >>>? ?? Thanks! >>> >>> On 12/15/18 2:43 PM, Stephan Leemburg wrote: >>>> OVH Requires you to route traffic from VM's via the IP address of your >>>> hardware. >>>> >>>> So 137 is the ip address of the hardware. >>>> >>>> Do you use any public ip addresses on your soyoustart system? >>>> >>>> Or just private range and then send them out via NAT? >>>> >>>> Met vriendelijke groet, >>>> Stephan Leemburg >>>> IT Functions >>>> >>>> e: sleemburg at it-functions.nl >>>> p: +31 (0)71 889 23 33 >>>> m: +31(0)6 83 22 30 69 >>>> kvk: 27313647 >>>> >>>> On 15-12-18 14:39, Miguel Gonz?lez wrote: >>>>> There must be something wrong with the configuration since I have >>>>> tested >>>>> another server and seems to be fine. >>>>> >>>>> Why do you use 137? In the proxmox docs they say the gateway is >>>>> xxx.xxx.xxx.254 >>>>> >>>>> Thanks! >>>>> >>>>> >>>>> On 12/15/18 2:16 PM, Stephan Leemburg wrote: >>>>>> Did you setup routing correctly within the containers / vm's? >>>>>> OVH/SoYouStart has awkward network routing requirements. >>>>>> >>>>>> I have 2 servers at soyoustart and they do fine with the correct >>>>>> network configuration. >>>>>> >>>>>> Below is an example from one of my containers. >>>>>> >>>>>> Also, it is a good idea to setup a firewall and put your >>>>>> containers on >>>>>> vmbr devices connected to the lan side of your firewall. >>>>>> >>>>>> Then on the lan side you have 'normal' network configurations. >>>>>> >>>>>> The pve has ip address 91.121.183.137 >>>>>> >>>>>> I have a subnet 54.37.62.224/28 on which containers and vm's live. >>>>>> >>>>>> # cat /etc/network/interfaces >>>>>> >>>>>> auto lo >>>>>> iface lo inet loopback >>>>>> >>>>>> auto eth0 >>>>>> iface eth0 inet static >>>>>>? ????? address 54.37.62.232 >>>>>>? ????? netmask 255.255.255.255 >>>>>>? ????? pre-up /etc/network/firewall $IFACE up >>>>>>? ????? post-up ip route add 91.121.183.137 dev $IFACE >>>>>>? ????? post-up ip route add 54.37.62.224/28 dev $IFACE >>>>>>? ????? post-up ip route add default via 91.121.183.137 dev $IFACE >>>>>>? ????? post-down ip route del default via 91.121.183.137 dev $IFACE >>>>>>? ????? post-down ip route del 54.37.62.224/28 dev $IFACE >>>>>>? ????? post-down ip route del 91.121.183.137 dev $IFACE >>>>>>? ????? post-down /etc/network/firewall $IFACE down' >>>>>> >>>>>> >>>>>> Met vriendelijke groet, >>>>>> Stephan Leemburg >>>>>> >>>>>> On 15-12-18 13:02, Miguel Gonz?lez wrote: >>>>>>> Hi, >>>>>>> >>>>>>>? ????? I am migrating some VMs from a Soyoustart (OVH) Proxmox 5.1 >>>>>>> to a >>>>>>> brand new Proxmox 5.3 server (again Soyoustart). >>>>>>> >>>>>>>? ????? I have followed the instructions from OVH and Proxmox and I >>>>>>> can ping >>>>>>> from the VM to the host and the gateway and I can ping from the >>>>>>> host to >>>>>>> the VM. But I can?t ping the DNS server or anything outside the host >>>>>>> machine (i.e.: the legacy Proxmox host). >>>>>>> >>>>>>>? ???? Some people suggest to enable ip forwarding, but I don?t have >>>>>>> enabled >>>>>>> on the legacy server. >>>>>>> >>>>>>>? ???? But I enable it anyway echo 1 > /proc/sys/net/ipv4/ip_forward >>>>>>>? ??? ??? and nothing happens. >>>>>>> >>>>>>>? ???? iptables seems to be turned off both in host and vm: >>>>>>> >>>>>>>? ????? iptables -L >>>>>>> Chain INPUT (policy ACCEPT) >>>>>>> target???? prot opt source?????????????? destination >>>>>>> >>>>>>> Chain FORWARD (policy ACCEPT) >>>>>>> target???? prot opt source?????????????? destination >>>>>>> >>>>>>> Chain OUTPUT (policy ACCEPT) >>>>>>> target???? prot opt source?????????????? destination >>>>>>> >>>>>>> So I?m out of ideas here >>>>>>> >>>>>>> Any suggestion? >>>>>>> >>>>>>> Miguel >>>>>>> >>>>>>> >>>>>>> >>>>>>> --- >>>>>>> This email has been checked for viruses by AVG. >>>>>>> https://www.avg.com >>>>>>> _______________________________________________ >>>>>>> pve-user mailing list >>>>>>> pve-user at pve.proxmox.com >>>>>>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From elacunza at binovo.es Mon Dec 17 09:23:36 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Mon, 17 Dec 2018 09:23:36 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> <20181216142629.npzu6ykxzk4hue2v@ipnerd.net> Message-ID: Hi, El 16/12/18 a las 17:16, Frank Thommen escribi?: >>> I understand that with the new PVE release PVE hosts (hypervisors) >>> can be >>> used as Ceph servers.? But it's not clear to me if (or when) that makes >>> sense.? Do I really want to have Ceph MDS/OSD on the same hardware >>> as my >>> hypervisors?? Doesn't that a) accumulate multiple POFs on the same >>> hardware >>> and b) occupy computing resources (CPU, RAM), that I'd rather use >>> for my VMs >>> and containers?? Wouldn't I rather want to have a separate Ceph >>> cluster? >> The integration of Ceph services in PVE started with Proxmox VE 3.0. >> With PVE 5.3 (current) we added CephFS services to the PVE. So you can >> run a hyper-converged Ceph with RBD/CephFS on the same servers as your >> VM/CT. >> >> a) can you please be more specific in what you see as multiple point of >> failures? > > not only I run the hypervisor which controls containers and virtual > machines on the server, but also the fileservice which is used to > store the VM and container images. I think you have less points of failure :-) because you'll have 3 points (nodes) of failure in an hyperconverged scenario and 6 in a separate virtualization/storage cluster scenario...? it depends how you look at it. > >> b) depends on the workload of your nodes. Modern server hardware has >> enough power to be able to run multiple services. It all comes down to >> have enough resources for each domain (eg. Ceph, KVM, CT, host). >> >> I recommend to use a simple calculation for the start, just to get a >> direction. >> >> In principle: >> >> ==CPU== >> core='CPU with HT on' >> >> * reserve a core for each Ceph daemon >> ?? (preferable on the same NUMA as the network; higher frequency is >> ?? better) >> * one core for the network card (higher frequency = lower latency) >> * rest of the cores for OS (incl. monitoring, backup, ...), KVM/CT usage >> * don't overcommit >> >> ==Memory== >> * 1 GB per TB of used disk space on an OSD (more on recovery) Note this is not true anymore with Bluestore, because you have to add cache space into account (1GB for HDD and 3GB for SSD OSDs if I recall correctly.), and also currently OSD processes aren't that good with RAM use accounting... :) >> * enough memory for KVM/CT >> * free memory for OS, backup, monitoring, live migration >> * don't overcommit >> >> ==Disk== >> * one OSD daemon per disk, even disk sizes throughout the cluster >> * more disks, more hosts, better distribution >> >> ==Network== >> * at least 10 GbE for storage traffic (more the better), >> ?? see our benchmark paper >> https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/ 10Gbit helps a lot with latency; small clusters can work perfectly with 2x1Gbit if they aren't latency-sensitive (we have been running a handfull of those for some years now). Cheers Eneko -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From damart.vidin at gmail.com Mon Dec 17 10:16:19 2018 From: damart.vidin at gmail.com (David Martin) Date: Mon, 17 Dec 2018 10:16:19 +0100 Subject: [PVE-User] Export / import VM Message-ID: <09063E75-0453-4CDC-8BCC-9EB5FACB30BE@gmail.com> Hi, How to import or export VM ? I receiving diff?rent files format (ova,ovf, vmdk) but i don t find function in web interface. David From a.antreich at proxmox.com Mon Dec 17 10:22:18 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Mon, 17 Dec 2018 10:22:18 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> <20181216142629.npzu6ykxzk4hue2v@ipnerd.net> Message-ID: <20181217092218.a5aqwilatd5dmx4b@dona.proxmox.com> Hello Eneko, On Mon, Dec 17, 2018 at 09:23:36AM +0100, Eneko Lacunza wrote: > Hi, > > El 16/12/18 a las 17:16, Frank Thommen escribi?: > > > > I understand that with the new PVE release PVE hosts > > > > (hypervisors) can be > > > > used as Ceph servers.? But it's not clear to me if (or when) that makes > > > > sense.? Do I really want to have Ceph MDS/OSD on the same > > > > hardware as my > > > > hypervisors?? Doesn't that a) accumulate multiple POFs on the > > > > same hardware > > > > and b) occupy computing resources (CPU, RAM), that I'd rather > > > > use for my VMs > > > > and containers?? Wouldn't I rather want to have a separate Ceph > > > > cluster? > > > The integration of Ceph services in PVE started with Proxmox VE 3.0. > > > With PVE 5.3 (current) we added CephFS services to the PVE. So you can > > > run a hyper-converged Ceph with RBD/CephFS on the same servers as your > > > VM/CT. > > > > > > a) can you please be more specific in what you see as multiple point of > > > failures? > > > > not only I run the hypervisor which controls containers and virtual > > machines on the server, but also the fileservice which is used to store > > the VM and container images. > I think you have less points of failure :-) because you'll have 3 points > (nodes) of failure in an hyperconverged scenario and 6 in a separate > virtualization/storage cluster scenario...? it depends how you look at it. > > > > > b) depends on the workload of your nodes. Modern server hardware has > > > enough power to be able to run multiple services. It all comes down to > > > have enough resources for each domain (eg. Ceph, KVM, CT, host). > > > > > > I recommend to use a simple calculation for the start, just to get a > > > direction. > > > > > > In principle: > > > > > > ==CPU== > > > core='CPU with HT on' > > > > > > * reserve a core for each Ceph daemon > > > ?? (preferable on the same NUMA as the network; higher frequency is > > > ?? better) > > > * one core for the network card (higher frequency = lower latency) > > > * rest of the cores for OS (incl. monitoring, backup, ...), KVM/CT usage > > > * don't overcommit > > > > > > ==Memory== > > > * 1 GB per TB of used disk space on an OSD (more on recovery) > Note this is not true anymore with Bluestore, because you have to add cache > space into account (1GB for HDD and 3GB for SSD OSDs if I recall > correctly.), and also currently OSD processes aren't that good with RAM use > accounting... :) I want to add, that the recommendations for luminous still state that. Also it is more a rule of thumb, not what the usage actually will be. http://docs.ceph.com/docs/luminous/start/hardware-recommendations/ With 12.2.10 the bluestore_cache_* settings have been replaced by osd_memory_target and are set to 4GB by default. http://docs.ceph.com/docs/master/releases/luminous/ > > > * enough memory for KVM/CT > > > * free memory for OS, backup, monitoring, live migration > > > * don't overcommit > > > > > > ==Disk== > > > * one OSD daemon per disk, even disk sizes throughout the cluster > > > * more disks, more hosts, better distribution > > > > > > ==Network== > > > * at least 10 GbE for storage traffic (more the better), > > > ?? see our benchmark paper > > > https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/ > 10Gbit helps a lot with latency; small clusters can work perfectly with > 2x1Gbit if they aren't latency-sensitive (we have been running a handfull of > those for some years now). > > Cheers > Eneko -- Cheers, Alwin From elacunza at binovo.es Mon Dec 17 11:30:11 2018 From: elacunza at binovo.es (Eneko Lacunza) Date: Mon, 17 Dec 2018 11:30:11 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: <20181217092218.a5aqwilatd5dmx4b@dona.proxmox.com> References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> <20181216142629.npzu6ykxzk4hue2v@ipnerd.net> <20181217092218.a5aqwilatd5dmx4b@dona.proxmox.com> Message-ID: <5ec7c77f-db05-f983-5cbe-a6fbd053f15d@binovo.es> Hi Alwin, El 17/12/18 a las 10:22, Alwin Antreich escribi?: > >>>> b) depends on the workload of your nodes. Modern server hardware has >>>> enough power to be able to run multiple services. It all comes down to >>>> have enough resources for each domain (eg. Ceph, KVM, CT, host). >>>> >>>> I recommend to use a simple calculation for the start, just to get a >>>> direction. >>>> >>>> In principle: >>>> >>>> ==CPU== >>>> core='CPU with HT on' >>>> >>>> * reserve a core for each Ceph daemon >>>> ?? (preferable on the same NUMA as the network; higher frequency is >>>> ?? better) >>>> * one core for the network card (higher frequency = lower latency) >>>> * rest of the cores for OS (incl. monitoring, backup, ...), KVM/CT usage >>>> * don't overcommit >>>> >>>> ==Memory== >>>> * 1 GB per TB of used disk space on an OSD (more on recovery) >> Note this is not true anymore with Bluestore, because you have to add cache >> space into account (1GB for HDD and 3GB for SSD OSDs if I recall >> correctly.), and also currently OSD processes aren't that good with RAM use >> accounting... :) > I want to add, that the recommendations for luminous still state that. > Also it is more a rule of thumb, not what the usage actually will be. > http://docs.ceph.com/docs/luminous/start/hardware-recommendations/ > > With 12.2.10 the bluestore_cache_* settings have been replaced by > osd_memory_target and are set to 4GB by default. > http://docs.ceph.com/docs/master/releases/luminous/ You're right, it seems that docs aren't updated to reflect Bluestore RAM use. The problem here is that Bluestore can't use Linux cache, so has an in-process cache of his own. Cheers! Eneko -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From frank.thommen at uni-heidelberg.de Mon Dec 17 11:52:26 2018 From: frank.thommen at uni-heidelberg.de (Frank Thommen) Date: Mon, 17 Dec 2018 11:52:26 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: <20181216184740.uli56kockq6nwuvo@ipnerd.net> References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> <20181216142629.npzu6ykxzk4hue2v@ipnerd.net> <20181216184740.uli56kockq6nwuvo@ipnerd.net> Message-ID: <61ac961c-78c0-6633-83b6-0e183ec8a373@uni-heidelberg.de> Hi Alwin, On 12/16/18 7:47 PM, Alwin Antreich wrote: > On Sun, Dec 16, 2018 at 05:16:50PM +0100, Frank Thommen wrote: >> Hi Alwin, >> >> On 16/12/18 15:39, Alwin Antreich wrote: >>> Hello Frank, >>> >>> On Sun, Dec 16, 2018 at 02:28:19PM +0100, Frank Thommen wrote: >>>> Hi, >>>> >>>> I understand that with the new PVE release PVE hosts (hypervisors) can be >>>> used as Ceph servers. But it's not clear to me if (or when) that makes >>>> sense. Do I really want to have Ceph MDS/OSD on the same hardware as my >>>> hypervisors? Doesn't that a) accumulate multiple POFs on the same hardware >>>> and b) occupy computing resources (CPU, RAM), that I'd rather use for my VMs >>>> and containers? Wouldn't I rather want to have a separate Ceph cluster? >>> The integration of Ceph services in PVE started with Proxmox VE 3.0. >>> With PVE 5.3 (current) we added CephFS services to the PVE. So you can >>> run a hyper-converged Ceph with RBD/CephFS on the same servers as your >>> VM/CT. >>> >>> a) can you please be more specific in what you see as multiple point of >>> failures? >> >> not only I run the hypervisor which controls containers and virtual machines >> on the server, but also the fileservice which is used to store the VM and >> container images. > Sorry, I am still not quite sure, what your question/concern is. > Failure tolerance needs to be planned into the system design, irrespective > of service distribution. > > Proxmox VE has a HA stack that restarts all services from a failed node > (if configured) on a other node. > https://pve.proxmox.com/pve-docs/chapter-ha-manager.html > > Ceph does selfhealing (if enough nodes > are available) or still works in a degraded state. > http://docs.ceph.com/docs/luminous/start/intro/ Yes, I am aware of PVE and Ceph failover/healing capabilities. But I always liked to separate basic and central services on the hardware level. This way if one server "explodes", only one service is affected. With PVE+Ceph on one node, such an outage would affect two basic services at once. I don't say they wouldn't continue to run productively, but they would run in degraded and non-failure-safe mode - assumed we had three such nodes in the cluster - until the broken node can be restored. But that's probably just my old-fashioned conservative approach. That's why I wanted to ask the list members for their assessment ;-) > [...] Cheers frank From ronny+pve-user at aasen.cx Mon Dec 17 11:58:13 2018 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Mon, 17 Dec 2018 11:58:13 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: <61ac961c-78c0-6633-83b6-0e183ec8a373@uni-heidelberg.de> References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> <20181216142629.npzu6ykxzk4hue2v@ipnerd.net> <20181216184740.uli56kockq6nwuvo@ipnerd.net> <61ac961c-78c0-6633-83b6-0e183ec8a373@uni-heidelberg.de> Message-ID: <59355a4f-1bf3-3d93-55ba-baa9f37532e1@aasen.cx> On 17.12.2018 11:52, Frank Thommen wrote: > Hi Alwin, > > On 12/16/18 7:47 PM, Alwin Antreich wrote: >> On Sun, Dec 16, 2018 at 05:16:50PM +0100, Frank Thommen wrote: >>> Hi Alwin, >>> >>> On 16/12/18 15:39, Alwin Antreich wrote: >>>> Hello Frank, >>>> >>>> On Sun, Dec 16, 2018 at 02:28:19PM +0100, Frank Thommen wrote: >>>>> Hi, >>>>> >>>>> I understand that with the new PVE release PVE hosts (hypervisors) >>>>> can be >>>>> used as Ceph servers.? But it's not clear to me if (or when) that >>>>> makes >>>>> sense.? Do I really want to have Ceph MDS/OSD on the same hardware >>>>> as my >>>>> hypervisors?? Doesn't that a) accumulate multiple POFs on the same >>>>> hardware >>>>> and b) occupy computing resources (CPU, RAM), that I'd rather use >>>>> for my VMs >>>>> and containers?? Wouldn't I rather want to have a separate Ceph >>>>> cluster? >>>> The integration of Ceph services in PVE started with Proxmox VE 3.0. >>>> With PVE 5.3 (current) we added CephFS services to the PVE. So you can >>>> run a hyper-converged Ceph with RBD/CephFS on the same servers as your >>>> VM/CT. >>>> >>>> a) can you please be more specific in what you see as multiple point of >>>> failures? >>> >>> not only I run the hypervisor which controls containers and virtual >>> machines >>> on the server, but also the fileservice which is used to store the VM >>> and >>> container images. >> Sorry, I am still not quite sure, what your question/concern is. >> Failure tolerance needs to be planned into the system design, >> irrespective >> of service distribution. >> >> Proxmox VE has a HA stack that restarts all services from a failed node >> (if configured) on a other node. >> https://pve.proxmox.com/pve-docs/chapter-ha-manager.html >> >> Ceph does selfhealing (if enough nodes >> are available) or still works in a degraded state. >> http://docs.ceph.com/docs/luminous/start/intro/ > > Yes, I am aware of PVE and Ceph failover/healing capabilities.? But I > always liked to separate basic and central services on the hardware > level.? This way if one server "explodes", only one service is affected. > ?With PVE+Ceph on one node, such an outage would affect two basic > services at once.? I don't say they wouldn't continue to run > productively, but they would run in degraded and non-failure-safe mode - > assumed we had three such nodes in the cluster - until the broken node > can be restored. with ceph i feel the very minimum osd node size is 4. with 25% freespace so you can loose one node and have it rebuilt to a healthy cluster. I only consider HCI if the performance of the cluster is way above the requirements. and once the need grow and you add some nodes, you often end up growing osd nodes out into separate machines. kind regards Ronny Aasen From frank.thommen at uni-heidelberg.de Mon Dec 17 13:20:50 2018 From: frank.thommen at uni-heidelberg.de (Frank Thommen) Date: Mon, 17 Dec 2018 13:20:50 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> <20181216142629.npzu6ykxzk4hue2v@ipnerd.net> Message-ID: On 12/17/18 9:23 AM, Eneko Lacunza wrote: > Hi, > > El 16/12/18 a las 17:16, Frank Thommen escribi?: >>>> I understand that with the new PVE release PVE hosts (hypervisors) >>>> can be >>>> used as Ceph servers.? But it's not clear to me if (or when) that makes >>>> sense.? Do I really want to have Ceph MDS/OSD on the same hardware >>>> as my >>>> hypervisors?? Doesn't that a) accumulate multiple POFs on the same >>>> hardware >>>> and b) occupy computing resources (CPU, RAM), that I'd rather use >>>> for my VMs >>>> and containers?? Wouldn't I rather want to have a separate Ceph >>>> cluster? >>> The integration of Ceph services in PVE started with Proxmox VE 3.0. >>> With PVE 5.3 (current) we added CephFS services to the PVE. So you can >>> run a hyper-converged Ceph with RBD/CephFS on the same servers as your >>> VM/CT. >>> >>> a) can you please be more specific in what you see as multiple point of >>> failures? >> >> not only I run the hypervisor which controls containers and virtual >> machines on the server, but also the fileservice which is used to >> store the VM and container images. > I think you have less points of failure :-) because you'll have 3 points > (nodes) of failure in an hyperconverged scenario and 6 in a separate > virtualization/storage cluster scenario...? it depends how you look at it. Right, but I look at it from the service side: one hardware failure -> one service affected vs. one hardware failure -> two service affected. >>> b) depends on the workload of your nodes. Modern server hardware has >>> enough power to be able to run multiple services. It all comes down to >>> have enough resources for each domain (eg. Ceph, KVM, CT, host). >>> >>> I recommend to use a simple calculation for the start, just to get a >>> direction. >>> >>> In principle: >>> >>> ==CPU== >>> core='CPU with HT on' >>> >>> * reserve a core for each Ceph daemon >>> ?? (preferable on the same NUMA as the network; higher frequency is >>> ?? better) >>> * one core for the network card (higher frequency = lower latency) >>> * rest of the cores for OS (incl. monitoring, backup, ...), KVM/CT usage >>> * don't overcommit >>> >>> ==Memory== >>> * 1 GB per TB of used disk space on an OSD (more on recovery) > Note this is not true anymore with Bluestore, because you have to add > cache space into account (1GB for HDD and 3GB for SSD OSDs if I recall > correctly.), and also currently OSD processes aren't that good with RAM > use accounting... :) >>> * enough memory for KVM/CT >>> * free memory for OS, backup, monitoring, live migration >>> * don't overcommit >>> >>> ==Disk== >>> * one OSD daemon per disk, even disk sizes throughout the cluster >>> * more disks, more hosts, better distribution >>> >>> ==Network== >>> * at least 10 GbE for storage traffic (more the better), >>> ?? see our benchmark paper >>> https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2018-02.41761/ >>> > 10Gbit helps a lot with latency; small clusters can work perfectly with > 2x1Gbit if they aren't latency-sensitive (we have been running a > handfull of those for some years now). I will keep the two points in mind. Thank you. frank From aderumier at odiso.com Mon Dec 17 13:50:48 2018 From: aderumier at odiso.com (Alexandre DERUMIER) Date: Mon, 17 Dec 2018 13:50:48 +0100 (CET) Subject: [PVE-User] Export / import VM In-Reply-To: <09063E75-0453-4CDC-8BCC-9EB5FACB30BE@gmail.com> References: <09063E75-0453-4CDC-8BCC-9EB5FACB30BE@gmail.com> Message-ID: <1669387857.2135164.1545051048028.JavaMail.zimbra@oxygem.tv> for import, you can do it with command line only. for an ovf: qm importovf [OPTIONS] for only 1disk: qm importdisk [OPTIONS] For export, they are no command line currently. (but you can use "qemu-img convert ..." to convert disk to vmdk or other format.) ----- Mail original ----- De: "David Martin" ?: "proxmoxve" Envoy?: Lundi 17 D?cembre 2018 10:16:19 Objet: [PVE-User] Export / import VM Hi, How to import or export VM ? I receiving diff?rent files format (ova,ovf, vmdk) but i don t find function in web interface. David _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From t.lamprecht at proxmox.com Mon Dec 17 13:59:23 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 17 Dec 2018 13:59:23 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> Message-ID: On 12/16/18 2:28 PM, Frank Thommen wrote: > I understand that with the new PVE release PVE hosts (hypervisors) can be used as Ceph servers. But it's not clear to me if (or when) that makes sense. Do I really want to have Ceph MDS/OSD on the same hardware as my hypervisors? Doesn't that a) accumulate multiple POFs on the same hardware and b) occupy computing resources (CPU, RAM), that I'd rather use for my VMs and containers? Wouldn't I rather want to have a separate Ceph cluster? Some users also run a bit of a mixed approach. For example 6 nodes all Proxmox VE, 3 are used for compute (VM/CT) and 3 are used for ceph, you get full management through one stack but still have some separation. You also could do a bit less separation and and have nodes which mostly host Ceph but also some low resource CTs/VMs, and some which mostly do Compute. In smaller setups, e.g., with 3 nodes you often can have less POFs easier. E.g., in this case it's easy to use 10G or 40G NICs with two ports each and setup a full mesh for the ceph storage traffic, you get full performance but save a switch, thus cost and a bit of complexity (one POF less). It often depends on the use case and the desired reliability/availability. IMO, it's often a balance between single node resources and total node count. Because you need to have a few nodes to ensure that one or two can die or get temporarily rendered unusable, but do not want to have to many small nodes as (maintenance) overhead and complexity increases here. E.g., from a availability POV,it would be better to have 5 normal but powerful nodes than three 4-socket, >1TB ram monster nodes. But, while then a high node count with small nodes would increase the capacity to handle node losses, you counterpart this with added complexity and also fixed cost per node. Further, each nodes reserve to handle fail over resources also gets smaller, in absolute terms. Oh and for your 3 PVE + 3 Ceph example you mentioned somewhere in this thread: Here each instance can only loose one node, so while theoretically you could afford two node losses, they need to be the "right" nodes, i.e., cannot be from the same instance. If you to mixed 5 node cluster you can loose two nodes too, but here it does not matter which two are affected. This is IMO a win for such a setup, it handles more scenarios with even one node less. Anyway, so much ramblings from my side.. cheers, Thomas From a.antreich at proxmox.com Mon Dec 17 14:09:56 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Mon, 17 Dec 2018 14:09:56 +0100 Subject: [PVE-User] (Very) basic question regarding PVE Ceph integration In-Reply-To: <61ac961c-78c0-6633-83b6-0e183ec8a373@uni-heidelberg.de> References: <42878d3a-6ee3-611a-9b22-4497d3a0a68c@dkfz-heidelberg.de> <20181216142629.npzu6ykxzk4hue2v@ipnerd.net> <20181216184740.uli56kockq6nwuvo@ipnerd.net> <61ac961c-78c0-6633-83b6-0e183ec8a373@uni-heidelberg.de> Message-ID: <20181217130956.kurmsu54p64izfvt@dona.proxmox.com> Hello Frank, On Mon, Dec 17, 2018 at 11:52:26AM +0100, Frank Thommen wrote: > Yes, I am aware of PVE and Ceph failover/healing capabilities. But I always > liked to separate basic and central services on the hardware level. This > way if one server "explodes", only one service is affected. With PVE+Ceph > on one node, such an outage would affect two basic services at once. I > don't say they wouldn't continue to run productively, but they would run in > degraded and non-failure-safe mode - assumed we had three such nodes in the > cluster - until the broken node can be restored. > > But that's probably just my old-fashioned conservative approach. That's why > I wanted to ask the list members for their assessment ;-) In the small constellation of 3x hyper-converged vs 3x PVE + 3x Ceph, the result will be the same (<50% for quorum). 3x hyper-converged = can sustain one node failure in total 6x (3+3) separated nodes = one failed node (2x total) in each domain Some thoughts to the separation. While it has its benefits to separate Ceph from PVE, in that case one node in each domain can fail and they are not interfering with the other domain. It might also give better latency (no VM/CT), but this is more depend on the hardware used then the separation. On the other hand, when you already consider to use 6x nodes (from above), then the hyper-converged setup might be more beneficial. As Ceph's performance grows with the count of nodes (and OSDs). Also the VM/CT will be more distributed throughout the cluster. A node failure will have less impact on the count of VM/CT than a failure in a separated setup. So each setup has it's pro/con and needs to be weight (distribution vs separation). -- Cheers, Alwin From smr at kmi.com Mon Dec 17 18:22:56 2018 From: smr at kmi.com (Stefan M. Radman) Date: Mon, 17 Dec 2018 17:22:56 +0000 Subject: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? In-Reply-To: <215268ab-b120-4d10-0c2c-8ef5ad14d962@binovo.es> References: <951493d1-27a9-1126-48fd-12e519c2bdf9@binovo.es> <2086835097.116238.1543936175736.JavaMail.zimbra@midoco.de> <6a097b94-ed00-16cd-b8ef-ea7a74696503@binovo.es> <5e092d6e-ec2a-3a34-b895-1b3896b04c75@aasen.cx> <98FCEDEC-60AB-43D0-8AA6-7DE78BBE7C85@kmi.com> <4cd4cc42-9e6f-c0a4-a27d-2a796865d995@binovo.es> <64DCA5F1-0DA1-4419-BCB8-988E43B7F17B@kmi.com> <215268ab-b120-4d10-0c2c-8ef5ad14d962@binovo.es> Message-ID: <35CB45A2-FAA9-421A-830B-A683002FBADA@kmi.com> For a production cluster I would actually recommend two physically separated networks for corosync. https://pve.proxmox.com/wiki/Separate_Cluster_Network#Redundant_Ring_Protocol Cheers Stefan On Dec 5, 2018, at 13:55, Eneko Lacunza > wrote: Hi Stefan, Thanks a lot for your suggestions! Cheers Eneko El 5/12/18 a las 13:52, Stefan M. Radman escribi?: Hi Eneko Even if PVE does it by default it does not mean that it is the best thing to do. The default configuration of PVE is a starting point. https://pve.proxmox.com/wiki/Separate_Cluster_Network Separate Cluster Network - Introduction It is good practice to use a separate network for corosync, which handles the cluster communication in Proxmox VE. It is one of the most important part in an fault tolerant (HA) system and other network traffic may disturb corosync. I'd recommend a thorough reading of the document quoted above. Don't use vmbr0 for cluster traffic. Don't use any vmbr for cluster traffic. Stefan On Dec 5, 2018, at 13:34, Eneko Lacunza > wrote: Hi Stefan, El 4/12/18 a las 23:50, Stefan M. Radman escribi?: Don't put your corosync traffic on bridges. Dedicate an untagged interface on each node for corosync. This is surprising, because Proxmox does this by default! :) All you need for your cluster network is this: auto eth3 iface eth3 inet static address 192.168.10.201 netmask 255.255.255.0 #corosync ring0 Put that interface into an isolated VLAN with IGMP snooping enabled. Prune that VLAN from all trunks to limit its extent and your troubles. But we're no using eth3/vmbr10 for corosync. We're using vmbr0 (default Proxmox config) por corosync. Thanks a lot Eneko Stefan On Dec 4, 2018, at 8:03 PM, Ronny Aasen > wrote: vmbr10 is a bridge (or as switch by another name) if you want the switch to work reliably with multicast you probably need to enable multicast querier. |echo 1 > /sys/devices/virtual/net/vmbr0/bridge/multicast_querier| or you can disable snooping, so that it treats multicast as broadcast. | echo 0 > /sys/devices/virtual/net/vmbr0/bridge/multicast_snooping| this problem with multicast traffic also may lead to unreliable ipv6 nd and nd-ra usage. https://pve.proxmox.com/wiki/Multicast_notes have some more notes and exampes around mulicast_querier kind regards Ronny Aasen On 04.12.2018 17:54, Eneko Lacunza wrote: Hi all, Seems I found the solution. eth3 on proxmox1 is a broadcom 1gbit card connected to HPE switch; it is VLAN 10 untagged on the switch end. I changed the vmbr10 bridge to use eth4.10 on the X540 card, and after ifdown/ifup and corosync and pve-cluster restart, now everything seems good; cluster is stable and omping is happy too after 10 minutes :) It is strange because multicast is on VLAN 1 network... Cheers and thanks a lot Eneko El 4/12/18 a las 16:18, Eneko Lacunza escribi?: hi Marcus, El 4/12/18 a las 16:09, Marcus Haarmann escribi?: Hi, you did not provide details about your configuration. How is the network card set up ? Bonding ? Send your /etc/network/interfaces details. If bonding is active, check if the mode is correct in /proc/net/bonding. We encountered differences between /etc/network/interfaces setup and resulting mode. Also, check your switch configuration, VLAN setup, MTU etc. Yes, sorry about that. I have double checked the switch and all 3 node SFP+ port have the same configuration. /etc/network/interfaces in proxmox1 node: auto lo iface lo inet loopback iface eth0 inet manual iface eth1 inet manual iface eth2 inet manual iface eth3 inet manual iface eth4 inet manual iface eth5 inet manual auto vmbr10 iface vmbr10 inet static address 192.168.10.201 netmask 255.255.255.0 bridge_ports eth3 bridge_stp off bridge_fd 0 auto vmbr0 iface vmbr0 inet static address 192.168.0.201 netmask 255.255.255.0 gateway 192.168.0.100 bridge_ports eth4 bridge_stp off bridge_fd 0 auto eth4.100 iface eth4.100 inet static address 10.0.2.1 netmask 255.255.255.0 up ip addr add 10.0.3.1/24 dev eth4.100 Cluster is running on vmbr0 network (192.168.0.0/24) Cheers Marcus Haarmann Von: "Eneko Lacunza" > An: "pve-user" > Gesendet: Dienstag, 4. Dezember 2018 15:57:10 Betreff: [PVE-User] Multicast problems with Intel X540 - 10Gtek network card? Hi all, We have just updated a 3-node Proxmox cluster from 3.4 to 5.2, Ceph hammer to Luminous and the network from 1 Gbit to 10Gbit... one of the three Proxmox nodes is new too :) Generally all was good and VMs are working well. :-) BUT, we have some problems with the cluster; promxox1 node joins and then after about 4 minutes drops from the cluster. All multicast tests https://pve.proxmox.com/wiki/Multicast_notes#Using_omping_to_test_multicast run fine except the last one: *** proxmox1: root at proxmox1:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox3 : waiting for response msg proxmox4 : waiting for response msg proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging proxmox3 : given amount of query messages was sent proxmox4 : given amount of query messages was sent proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.073/0.184/0.390/0.061 proxmox3 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.092/0.207/0.421/0.068 proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.049/0.167/0.369/0.059 proxmox4 : multicast, xmt/rcv/%loss = 600/262/56%, min/avg/max/std-dev = 0.063/0.185/0.386/0.064 *** proxmox3: root at proxmox3:/etc# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox1 : waiting for response msg proxmox4 : waiting for response msg proxmox4 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : waiting for response msg proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging proxmox4 : given amount of query messages was sent proxmox1 : given amount of query messages was sent proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.083/0.193/1.030/0.055 proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.102/0.209/1.050/0.054 proxmox4 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.041/0.108/0.172/0.026 proxmox4 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.123/0.190/0.030 *** root at proxmox4:~# omping -c 600 -i 1 -F -q proxmox1 proxmox3 proxmox4 proxmox1 : waiting for response msg proxmox3 : waiting for response msg proxmox1 : waiting for response msg proxmox3 : waiting for response msg proxmox3 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : joined (S,G) = (*, 232.43.211.234), pinging proxmox1 : given amount of query messages was sent proxmox3 : given amount of query messages was sent proxmox1 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.085/0.188/0.356/0.040 proxmox1 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.114/0.208/0.377/0.041 proxmox3 : unicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.048/0.117/0.289/0.023 proxmox3 : multicast, xmt/rcv/%loss = 600/600/0%, min/avg/max/std-dev = 0.064/0.134/0.290/0.026 Ok, so it seems we have a network problem on proxmox1 node. Network cards are as follows: - proxmox1: Intel X540 (10Gtek) - proxmox3: Intel X710 (Intel) - proxmox4: Intel X710 (Intel) Switch is Dell N1224T-ON. Does anyone have experience with Intel X540 chip network cards or Linux ixgbe network driver or 10Gtek manufacturer? If we change corosync communication to 1 Gbit network cards (broadcom) connected to an old HPE 1800-24G switch, cluster is stable... We also have a running cluster with Dell n1224T-ON switch and X710 network cards without issues. Thanks a lot Eneko _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From in_touch_uk at yahoo.co.uk Tue Dec 18 10:54:23 2018 From: in_touch_uk at yahoo.co.uk (Mark Kaye) Date: Tue, 18 Dec 2018 09:54:23 +0000 (UTC) Subject: [PVE-User] Proxmox Suricata Setup References: <1248560438.9167177.1545126863400.ref@mail.yahoo.com> Message-ID: <1248560438.9167177.1545126863400@mail.yahoo.com> Hi, I've followed the instructions for setting up Suricata on my Proxmox server as detailed at:https://pve.proxmox.com/wiki/Firewall However, no traffic is currently being filtered by Suricata to or?from the VMs or containers (dns, http, fast logs show nothing). My /etc/default/suricata is: RUN=yesSURCONF=/etc/suricata/suricata.yaml LISTENMODE=nfqueue IFACE=eth0 NFQUEUE=2TCMALLOC="YES"PIDFILE=/var/run/suricata.pid I have a OVS bond (bond0) setup with an OVS bridge (vmbr0) for the public interface. I've had to override the default Suricata systemd configuration in order to run the Suricata init script & thus the /etc/default/suricata configuration using the following: [Service]ExecStart=ExecStart=/etc/init.d/suricata startExecStop=ExecStop=/etc/init.d/suricata stop IPTABLES--------------?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? ? 0? ? ?0 PVEFW-reject? tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? tcp dpt:4394086? ?13M PVEFW-DropBroadcast? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? 0? ? ?0 ACCEPT? ? ?icmp --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? icmptype 3 code 4? ? 0? ? ?0 ACCEPT? ? ?icmp --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? icmptype 11? ? 0? ? ?0 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ctstate INVALID? ? 0? ? ?0 DROP? ? ? ?udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? multiport dports 135,445? ? 3? ?234 DROP? ? ? ?udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? udp dpts:137:139? ? 0? ? ?0 DROP? ? ? ?udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? udp spt:137 dpts:1024:65535? ? 0? ? ?0 DROP? ? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? multiport dports 135,139,445? ? 0? ? ?0 DROP? ? ? ?udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? udp dpt:1900? ?44? 4694 DROP? ? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? tcp flags:!0x17/0x02? ? 0? ? ?0 DROP? ? ? ?udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? udp spt:53?1617 90572? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:WDy2wbFe7jNYEyoO3QhUELZ4mIQ */ Chain PVEFW-DropBroadcast (2 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination?2034? 202K DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ADDRTYPE match dst-type BROADCAST90388? ?13M DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ADDRTYPE match dst-type MULTICAST? ? 0? ? ?0 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ADDRTYPE match dst-type ANYCAST? ? 0? ? ?0 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 224.0.0.0/4?1664 95500? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w */ Chain PVEFW-FORWARD (1 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? 197 91271 PVEFW-IPS? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ctstate RELATED,ESTABLISHED? ? 0? ? ?0 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ctstate INVALID? ?76? 5238 ACCEPT? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ctstate RELATED,ESTABLISHED14773 1962K PVEFW-FWBR-IN? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? PHYSDEV match --physdev-in fwln+ --physdev-is-bridged? ?75? 4358 PVEFW-FWBR-OUT? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? PHYSDEV match --physdev-out fwln+ --physdev-is-bridged? ?75? 4358? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:SObnoWmKDADnyocGfaX95tEgIRE */ Chain PVEFW-FWBR-IN (1 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination94086? ?13M PVEFW-smurfs? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ctstate INVALID,NEW94086? ?13M veth100i0-IN? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? PHYSDEV match --physdev-out veth100i0 --physdev-is-bridged? ? 0? ? ?0? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:4OE5DGCM8EKNLmQbkV3LIx5w1QM */ Chain PVEFW-FWBR-OUT (1 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? 170? 9994 veth100i0-OUT? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? PHYSDEV match --physdev-in veth100i0 --physdev-is-bridged? 170? 9994? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:lXaefvpIDNAYTJwiaBF5f1+faEw */ Chain PVEFW-HOST-IN (1 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination3551K 7022M ACCEPT? ? ?all? --? lo? ? ?*? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0?1289 51560 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ctstate INVALID5599K 7971M ACCEPT? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ctstate RELATED,ESTABLISHED3494K? 606M PVEFW-smurfs? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ctstate INVALID,NEW? ? 0? ? ?0 RETURN? ? ?2? ? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0?8672? 451K RETURN? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? match-set PVEFW-0-management-v4 src tcp dpt:8006? ? 0? ? ?0 RETURN? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? match-set PVEFW-0-management-v4 src tcp dpts:5900:5999? ? 0? ? ?0 RETURN? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? match-set PVEFW-0-management-v4 src tcp dpt:3128? ?21? 4420 RETURN? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? match-set PVEFW-0-management-v4 src tcp dpt:22? ? 0? ? ?0 RETURN? ? ?udp? --? *? ? ? *? ? ? ?192.168.1.0/24? ? ? ?192.168.1.0/24? ? ? ?udp dpts:5404:5405? ? 0? ? ?0 RETURN? ? ?udp? --? *? ? ? *? ? ? ?192.168.1.0/24? ? ? ?0.0.0.0/0? ? ? ? ? ? ADDRTYPE match dst-type MULTICAST udp dpts:5404:54053486K? 606M RETURN? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? 0? ? ?0? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:ALOh8WDdOmO5Ptu5R2nVTkfCuQE */ Chain PVEFW-HOST-OUT (1 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination3551K 7022M ACCEPT? ? ?all? --? *? ? ? lo? ? ? 0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? 7? ?280 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ctstate INVALID5686K 5074M ACCEPT? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ctstate RELATED,ESTABLISHED? ?20? ?800 RETURN? ? ?2? ? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? 0? ? ?0 RETURN? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 192.168.1.0/24? ? ? ?tcp dpt:8006? ? 0? ? ?0 RETURN? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 192.168.1.0/24? ? ? ?tcp dpt:22? ? 0? ? ?0 RETURN? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 192.168.1.0/24? ? ? ?tcp dpts:5900:5999? ? 0? ? ?0 RETURN? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 192.168.1.0/24? ? ? ?tcp dpt:3128? ? 0? ? ?0 RETURN? ? ?udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 192.168.1.0/24? ? ? ?udp dpts:5404:5405?435K? 168M RETURN? ? ?udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ADDRTYPE match dst-type MULTICAST udp dpts:5404:54053303K? 487M RETURN? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? 0? ? ?0? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:RXafES/zd/ydTJpoGNqsst+Y1Ws */ Chain PVEFW-INPUT (1 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination2177K 2663M PVEFW-HOST-IN? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0?591K? 102M? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk */ Chain PVEFW-IPS (1 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? 121 86033 NFQUEUE? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? PHYSDEV match --physdev-out veth100i0 --physdev-is-bridged NFQUEUE num 0 bypass? ?76? 5238? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:M9aJUBIEQiOQag1Lup5QtAbon7c */ Chain PVEFW-OUTPUT (1 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? 13M? ?13G PVEFW-HOST-OUT? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/03738K? 655M? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0 */ Chain PVEFW-Reject (0 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? ? 0? ? ?0 PVEFW-reject? tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? tcp dpt:43? ? 0? ? ?0 PVEFW-DropBroadcast? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? 0? ? ?0 ACCEPT? ? ?icmp --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? icmptype 3 code 4? ? 0? ? ?0 ACCEPT? ? ?icmp --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? icmptype 11? ? 0? ? ?0 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ctstate INVALID? ? 0? ? ?0 PVEFW-reject? udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? multiport dports 135,445? ? 0? ? ?0 PVEFW-reject? udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? udp dpts:137:139? ? 0? ? ?0 PVEFW-reject? udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? udp spt:137 dpts:1024:65535? ? 0? ? ?0 PVEFW-reject? tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? multiport dports 135,139,445? ? 0? ? ?0 DROP? ? ? ?udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? udp dpt:1900? ? 0? ? ?0 DROP? ? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? tcp flags:!0x17/0x02? ? 0? ? ?0 DROP? ? ? ?udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? udp spt:53? ? 0? ? ?0? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo */ Chain PVEFW-SET-ACCEPT-MARK (2 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? 170? 9994 MARK? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? MARK or 0x80000000? 170? 9994? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY */ Chain PVEFW-logflags (5 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? ? 0? ? ?0 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? 0? ? ?0? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A */ Chain PVEFW-reject (6 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? ? 0? ? ?0 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ADDRTYPE match dst-type BROADCAST? ? 0? ? ?0 DROP? ? ? ?all? --? *? ? ? *? ? ? ?224.0.0.0/4? ? ? ? ? 0.0.0.0/0? ? 0? ? ?0 DROP? ? ? ?icmp --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? 0? ? ?0 REJECT? ? ?tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? reject-with tcp-reset? ? 0? ? ?0 REJECT? ? ?udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? reject-with icmp-port-unreachable? ? 0? ? ?0 REJECT? ? ?icmp --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? reject-with icmp-host-unreachable? ? 0? ? ?0 REJECT? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? reject-with icmp-host-prohibited? ? 0? ? ?0? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc */ Chain PVEFW-smurflog (2 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? ? 0? ? ?0 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? 0? ? ?0? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk */ Chain PVEFW-smurfs (2 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? 170 58821 RETURN? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0? ? ? ? ? ? ? 0.0.0.0/0? ? 0? ? ?0 PVEFW-smurflog? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ?[goto]? ADDRTYPE match src-type BROADCAST? ? 0? ? ?0 PVEFW-smurflog? all? --? *? ? ? *? ? ? ?224.0.0.0/4? ? ? ? ? 0.0.0.0/0? ? ? ? ? ?[goto]3588K? 619M? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:HssVe5QCBXd5mc9kC88749+7fag */ Chain PVEFW-tcpflags (0 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? ? 0? ? ?0 PVEFW-logflags? tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ?[goto]? tcp flags:0x3F/0x29? ? 0? ? ?0 PVEFW-logflags? tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ?[goto]? tcp flags:0x3F/0x00? ? 0? ? ?0 PVEFW-logflags? tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ?[goto]? tcp flags:0x06/0x06? ? 0? ? ?0 PVEFW-logflags? tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ?[goto]? tcp flags:0x03/0x03? ? 0? ? ?0 PVEFW-logflags? tcp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ?[goto]? tcp spt:0 flags:0x17/0x02? ? 0? ? ?0? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo */ Chain veth100i0-IN (1 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? ? 0? ? ?0 ACCEPT? ? ?udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? udp spt:67 dpt:6894086? ?13M PVEFW-Drop? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0?1617 90572 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? 0? ? ?0? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:6d+rQEp2SN9HQCSdz1apldUWciw */ Chain veth100i0-OUT (1 references)?pkts bytes target? ? ?prot opt in? ? ?out? ? ?source? ? ? ? ? ? ? ?destination? ? 0? ? ?0 PVEFW-SET-ACCEPT-MARK? udp? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ?[goto]? udp spt:68 dpt:67? ? 0? ? ?0 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? MAC ! XX:XX:XX:XX:XX:XX? ? 0? ? ?0 DROP? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? ! match-set PVEFW-100-ipfilter-net0-v4 src? 120? 7057 MARK? ? ? ?all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? MARK and 0x7fffffff? 120? 7057 PVEFW-SET-ACCEPT-MARK? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ?[goto]? ? 0? ? ?0? ? ? ? ? ? all? --? *? ? ? *? ? ? ?0.0.0.0/0? ? ? ? ? ? 0.0.0.0/0? ? ? ? ? ? /* PVESIG:6E+MZF3R6zzuBwUPvrw3GrGj9GQ */ Anybody got this working or have any advice? Cheers,Mark? From aderumier at odiso.com Tue Dec 18 14:44:48 2018 From: aderumier at odiso.com (Alexandre DERUMIER) Date: Tue, 18 Dec 2018 14:44:48 +0100 (CET) Subject: [PVE-User] Proxmox Suricata Setup In-Reply-To: <1248560438.9167177.1545126863400@mail.yahoo.com> References: <1248560438.9167177.1545126863400.ref@mail.yahoo.com> <1248560438.9167177.1545126863400@mail.yahoo.com> Message-ID: <1808245057.2178239.1545140688320.JavaMail.zimbra@oxygem.tv> as you have configured nfqueue=2, do you have setup ips_queues: 2 ? ----- Mail original ----- De: "Mark Kaye" ?: "proxmoxve" Envoy?: Mardi 18 D?cembre 2018 10:54:23 Objet: [PVE-User] Proxmox Suricata Setup Hi, I've followed the instructions for setting up Suricata on my Proxmox server as detailed at:https://pve.proxmox.com/wiki/Firewall However, no traffic is currently being filtered by Suricata to or from the VMs or containers (dns, http, fast logs show nothing). My /etc/default/suricata is: RUN=yesSURCONF=/etc/suricata/suricata.yaml LISTENMODE=nfqueue IFACE=eth0 NFQUEUE=2TCMALLOC="YES"PIDFILE=/var/run/suricata.pid I have a OVS bond (bond0) setup with an OVS bridge (vmbr0) for the public interface. I've had to override the default Suricata systemd configuration in order to run the Suricata init script & thus the /etc/default/suricata configuration using the following: [Service]ExecStart=ExecStart=/etc/init.d/suricata startExecStop=ExecStop=/etc/init.d/suricata stop IPTABLES-------------- pkts bytes target prot opt in out source destination 0 0 PVEFW-reject tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:4394086 13M PVEFW-DropBroadcast all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 3 code 4 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 11 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID 0 0 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 135,445 3 234 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpts:137:139 0 0 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:137 dpts:1024:65535 0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 135,139,445 0 0 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:1900 44 4694 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:!0x17/0x02 0 0 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:53 1617 90572 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:WDy2wbFe7jNYEyoO3QhUELZ4mIQ */ Chain PVEFW-DropBroadcast (2 references) pkts bytes target prot opt in out source destination 2034 202K DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type BROADCAST90388 13M DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type MULTICAST 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type ANYCAST 0 0 DROP all -- * * 0.0.0.0/0 224.0.0.0/4 1664 95500 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:NyjHNAtFbkH7WGLamPpdVnxHy4w */ Chain PVEFW-FORWARD (1 references) pkts bytes target prot opt in out source destination 197 91271 PVEFW-IPS all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID 76 5238 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED14773 1962K PVEFW-FWBR-IN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in fwln+ --physdev-is-bridged 75 4358 PVEFW-FWBR-OUT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out fwln+ --physdev-is-bridged 75 4358 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:SObnoWmKDADnyocGfaX95tEgIRE */ Chain PVEFW-FWBR-IN (1 references) pkts bytes target prot opt in out source destination94086 13M PVEFW-smurfs all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID,NEW94086 13M veth100i0-IN all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out veth100i0 --physdev-is-bridged 0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:4OE5DGCM8EKNLmQbkV3LIx5w1QM */ Chain PVEFW-FWBR-OUT (1 references) pkts bytes target prot opt in out source destination 170 9994 veth100i0-OUT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in veth100i0 --physdev-is-bridged 170 9994 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:lXaefvpIDNAYTJwiaBF5f1+faEw */ Chain PVEFW-HOST-IN (1 references) pkts bytes target prot opt in out source destination3551K 7022M ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 1289 51560 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID5599K 7971M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED3494K 606M PVEFW-smurfs all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID,NEW 0 0 RETURN 2 -- * * 0.0.0.0/0 0.0.0.0/0 8672 451K RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 match-set PVEFW-0-management-v4 src tcp dpt:8006 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 match-set PVEFW-0-management-v4 src tcp dpts:5900:5999 0 0 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 match-set PVEFW-0-management-v4 src tcp dpt:3128 21 4420 RETURN tcp -- * * 0.0.0.0/0 0.0.0.0/0 match-set PVEFW-0-management-v4 src tcp dpt:22 0 0 RETURN udp -- * * 192.168.1.0/24 192.168.1.0/24 udp dpts:5404:5405 0 0 RETURN udp -- * * 192.168.1.0/24 0.0.0.0/0 ADDRTYPE match dst-type MULTICAST udp dpts:5404:54053486K 606M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:ALOh8WDdOmO5Ptu5R2nVTkfCuQE */ Chain PVEFW-HOST-OUT (1 references) pkts bytes target prot opt in out source destination3551K 7022M ACCEPT all -- * lo 0.0.0.0/0 0.0.0.0/0 7 280 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID5686K 5074M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 20 800 RETURN 2 -- * * 0.0.0.0/0 0.0.0.0/0 0 0 RETURN tcp -- * * 0.0.0.0/0 192.168.1.0/24 tcp dpt:8006 0 0 RETURN tcp -- * * 0.0.0.0/0 192.168.1.0/24 tcp dpt:22 0 0 RETURN tcp -- * * 0.0.0.0/0 192.168.1.0/24 tcp dpts:5900:5999 0 0 RETURN tcp -- * * 0.0.0.0/0 192.168.1.0/24 tcp dpt:3128 0 0 RETURN udp -- * * 0.0.0.0/0 192.168.1.0/24 udp dpts:5404:5405 435K 168M RETURN udp -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type MULTICAST udp dpts:5404:54053303K 487M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:RXafES/zd/ydTJpoGNqsst+Y1Ws */ Chain PVEFW-INPUT (1 references) pkts bytes target prot opt in out source destination2177K 2663M PVEFW-HOST-IN all -- * * 0.0.0.0/0 0.0.0.0/0 591K 102M all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:+5iMmLaxKXynOB/+5xibfx7WhFk */ Chain PVEFW-IPS (1 references) pkts bytes target prot opt in out source destination 121 86033 NFQUEUE all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out veth100i0 --physdev-is-bridged NFQUEUE num 0 bypass 76 5238 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:M9aJUBIEQiOQag1Lup5QtAbon7c */ Chain PVEFW-OUTPUT (1 references) pkts bytes target prot opt in out source destination 13M 13G PVEFW-HOST-OUT all -- * * 0.0.0.0/0 0.0.0.0/03738K 655M all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:LjHoZeSSiWAG3+2ZAyL/xuEehd0 */ Chain PVEFW-Reject (0 references) pkts bytes target prot opt in out source destination 0 0 PVEFW-reject tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:43 0 0 PVEFW-DropBroadcast all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 3 code 4 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 11 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID 0 0 PVEFW-reject udp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 135,445 0 0 PVEFW-reject udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpts:137:139 0 0 PVEFW-reject udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:137 dpts:1024:65535 0 0 PVEFW-reject tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 135,139,445 0 0 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:1900 0 0 DROP tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:!0x17/0x02 0 0 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:53 0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:CZJnIN6rAdpu+ej59QPr9+laMUo */ Chain PVEFW-SET-ACCEPT-MARK (2 references) pkts bytes target prot opt in out source destination 170 9994 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x80000000 170 9994 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:Hg/OIgIwJChBUcWU8Xnjhdd2jUY */ Chain PVEFW-logflags (5 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:MN4PH1oPZeABMuWr64RrygPfW7A */ Chain PVEFW-reject (6 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type BROADCAST 0 0 DROP all -- * * 224.0.0.0/4 0.0.0.0/0 0 0 DROP icmp -- * * 0.0.0.0/0 0.0.0.0/0 0 0 REJECT tcp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with tcp-reset 0 0 REJECT udp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 REJECT icmp -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-unreachable 0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited 0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:Jlkrtle1mDdtxDeI9QaDSL++Npc */ Chain PVEFW-smurflog (2 references) pkts bytes target prot opt in out source destination 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:2gfT1VMkfr0JL6OccRXTGXo+1qk */ Chain PVEFW-smurfs (2 references) pkts bytes target prot opt in out source destination 170 58821 RETURN all -- * * 0.0.0.0 0.0.0.0/0 0 0 PVEFW-smurflog all -- * * 0.0.0.0/0 0.0.0.0/0 [goto] ADDRTYPE match src-type BROADCAST 0 0 PVEFW-smurflog all -- * * 224.0.0.0/4 0.0.0.0/0 [goto]3588K 619M all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:HssVe5QCBXd5mc9kC88749+7fag */ Chain PVEFW-tcpflags (0 references) pkts bytes target prot opt in out source destination 0 0 PVEFW-logflags tcp -- * * 0.0.0.0/0 0.0.0.0/0 [goto] tcp flags:0x3F/0x29 0 0 PVEFW-logflags tcp -- * * 0.0.0.0/0 0.0.0.0/0 [goto] tcp flags:0x3F/0x00 0 0 PVEFW-logflags tcp -- * * 0.0.0.0/0 0.0.0.0/0 [goto] tcp flags:0x06/0x06 0 0 PVEFW-logflags tcp -- * * 0.0.0.0/0 0.0.0.0/0 [goto] tcp flags:0x03/0x03 0 0 PVEFW-logflags tcp -- * * 0.0.0.0/0 0.0.0.0/0 [goto] tcp spt:0 flags:0x17/0x02 0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:CMFojwNPqllyqD67NeI5m+bP5mo */ Chain veth100i0-IN (1 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp spt:67 dpt:6894086 13M PVEFW-Drop all -- * * 0.0.0.0/0 0.0.0.0/0 1617 90572 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:6d+rQEp2SN9HQCSdz1apldUWciw */ Chain veth100i0-OUT (1 references) pkts bytes target prot opt in out source destination 0 0 PVEFW-SET-ACCEPT-MARK udp -- * * 0.0.0.0/0 0.0.0.0/0 [goto] udp spt:68 dpt:67 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 MAC ! XX:XX:XX:XX:XX:XX 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ! match-set PVEFW-100-ipfilter-net0-v4 src 120 7057 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK and 0x7fffffff 120 7057 PVEFW-SET-ACCEPT-MARK all -- * * 0.0.0.0/0 0.0.0.0/0 [goto] 0 0 all -- * * 0.0.0.0/0 0.0.0.0/0 /* PVESIG:6E+MZF3R6zzuBwUPvrw3GrGj9GQ */ Anybody got this working or have any advice? Cheers,Mark _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From miguel_3_gonzalez at yahoo.es Sat Dec 22 12:03:32 2018 From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=) Date: Sat, 22 Dec 2018 12:03:32 +0100 Subject: [PVE-User] =?utf-8?q?can=C2=B4t_login_in_PVE_with_Linux_realm_in_?= =?utf-8?b?NS4z?= Message-ID: <5ddb4b6d-a70d-706b-61ca-48cd14d80989@yahoo.es> Dear all, ?? I have been playing with this new Proxmox 5.3 server for more than a week before I migrate all VMs from a 5.1 server and I have been able to log in the GUI at 8006 port with no issue. Suddenly this morning after being able to log in with no issue I keep on getting Connection error 401: permission denied - invalid PVE ticket. ? If I check the logs I see the auth works: Dec 22 11:53:52 node1 pvedaemon[10764]: successful auth for user 'root at pam' ?I made sure that system time is right too. I am not using ntp (as I am not doing with the legacy server) ?google search is not helping either. ?I have tried to recreate the certs with pvecm updatecerts -f ?Any suggestions? ? ? ? --- This email has been checked for viruses by AVG. https://www.avg.com From miguel_3_gonzalez at yahoo.es Sat Dec 22 12:25:32 2018 From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=) Date: Sat, 22 Dec 2018 12:25:32 +0100 Subject: [PVE-User] =?utf-8?q?can=C2=B4t_login_in_PVE_with_Linux_realm_in_?= =?utf-8?b?NS4z?= Message-ID: <5e36bf9f-724b-edd8-8203-ebbf52d9d44a@yahoo.es> Dear all, ?? I have been playing with this new Proxmox 5.3 server for more than a week before I migrate all VMs from a 5.1 server and I have been able to log in the GUI at 8006 port with no issue. Suddenly this morning after being able to log in with no issue I keep on getting Connection error 401: permission denied - invalid PVE ticket. ? If I check the logs I see the auth works: Dec 22 11:53:52 node1 pvedaemon[10764]: successful auth for user 'root at pam' ?I made sure that system time is right too. I am not using ntp (as I am not doing with the legacy server) ?google search is not helping either. ?I have tried to recreate the certs with pvecm updatecerts -f ?Any suggestions? ? ? ? --- This email has been checked for viruses by AVG. https://www.avg.com From jfranco at maila.net.br Sat Dec 22 19:28:59 2018 From: jfranco at maila.net.br (Jean R. Franco) Date: Sat, 22 Dec 2018 16:28:59 -0200 (BRST) Subject: [PVE-User] =?utf-8?q?can=C2=B4t_login_in_PVE_with_Linux_realm_in?= =?utf-8?q?_5=2E3?= In-Reply-To: <5e36bf9f-724b-edd8-8203-ebbf52d9d44a@yahoo.es> References: <5e36bf9f-724b-edd8-8203-ebbf52d9d44a@yahoo.es> Message-ID: <578963047.184262.1545503339889.JavaMail.zimbra@maila.net.br> Hi Miguel, This is something that happens even if you reboot the host? If you cannot reboot, make sure pveproxy is running: /etc/init.d/pveproxy status ? pveproxy.service - PVE API Proxy Server Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2018-11-22 23:52:56 -02; 4 weeks 1 days ago Hth, ----- Mensagem original ----- De: "Miguel Gonz?lez" Para: "PVE User List" Enviadas: S?bado, 22 de dezembro de 2018 9:25:32 Assunto: [PVE-User] can?t login in PVE with Linux realm in 5.3 Dear all, ?? I have been playing with this new Proxmox 5.3 server for more than a week before I migrate all VMs from a 5.1 server and I have been able to log in the GUI at 8006 port with no issue. Suddenly this morning after being able to log in with no issue I keep on getting Connection error 401: permission denied - invalid PVE ticket. ? If I check the logs I see the auth works: Dec 22 11:53:52 node1 pvedaemon[10764]: successful auth for user 'root at pam' ?I made sure that system time is right too. I am not using ntp (as I am not doing with the legacy server) ?google search is not helping either. ?I have tried to recreate the certs with pvecm updatecerts -f ?Any suggestions? ? ? ? --- This email has been checked for viruses by AVG. https://www.avg.com _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From miguel_3_gonzalez at yahoo.es Sat Dec 22 20:41:57 2018 From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=) Date: Sat, 22 Dec 2018 20:41:57 +0100 Subject: [PVE-User] =?utf-8?q?can=C2=B4t_login_in_PVE_with_Linux_realm_in?= =?utf-8?q?_5=2E3?= In-Reply-To: <5e36bf9f-724b-edd8-8203-ebbf52d9d44a@yahoo.es> References: <5e36bf9f-724b-edd8-8203-ebbf52d9d44a@yahoo.es> Message-ID: Just in case someone else has this issue too. Apparently recreating the authkey.key as mentioned here and recreate the certs worked out: https://forum.proxmox.com/threads/connecting-error-401-unrecognized-key-format.32162/#post-159151 mv /etc/pve/priv/authkey.key /etc/pve/priv/authkey.key.tmp pvecm updatecerts --force systemctl restart pveproxy pvedaemon Regards, Miguel On 12/22/18 12:25 PM, Miguel Gonz?lez wrote: > Dear all, > > ?? I have been playing with this new Proxmox 5.3 server for more than a > week before I migrate all VMs from a 5.1 server and I have been able to > log in the GUI at 8006 port with no issue. Suddenly this morning after > being able to log in with no issue I keep on getting Connection error > 401: permission denied - invalid PVE ticket. > > ? If I check the logs I see the auth works: > > Dec 22 11:53:52 node1 pvedaemon[10764]: successful auth for > user 'root at pam' > > ?I made sure that system time is right too. I am not using ntp (as I am > not doing with the legacy server) > > ?google search is not helping either. > > ?I have tried to recreate the certs with pvecm updatecerts -f > > ?Any suggestions? > > ? > > ? > > ? > > > --- This email has been checked for viruses by AVG. https://www.avg.com From dewanggaba at xtremenitro.org Mon Dec 24 15:59:16 2018 From: dewanggaba at xtremenitro.org (Dewangga Alam) Date: Mon, 24 Dec 2018 21:59:16 +0700 Subject: [PVE-User] Unexpected behaviour at pmxcfs pve5.0 + pve5.2 Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hello! Currently I have mixed cluster pve5.0 and pve5.2, total nodes is 17, 6 nodes using pve5.0 and the rest is pve5.2. The issue was addressed to pmxcfs, really slow when I test using `touch a`, the result was : root at pve50:/etc/pve# time touch a real 0m13.099s user 0m0.000s sys 0m0.000s root at pve52:/etc/pve# time touch b real 0m14.216s user 0m0.001s sys 0m0.000s I ran debug logs by invoking `echo "1" >/etc/pve/.debug` command and I got this. https://paste.fedoraproject.org/paste/F-OW-VVNdL3ZZ8iv6SzNsA/raw I didn't found any strange behavior, but, somehow the pmxcfs really slow. Any helps are appreciated. :) Regards, Dewangga -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEZpxiw/Jg6pEte5xQ5X/SIKAozXAFAlwg9EIACgkQ5X/SIKAo zXDgBA//UM3lc41d8x6bGEk30NPMzUUEtt57A/rhYNuLhKrtdbdsrrcju7rD373h hUOFmjvDZmM4hLx/kIxOBpoBelmQKne5+riRllfes9ADQf7/t/afNpuuH23Wuh/o TI13G+xp9QA1+YCv4IgPgqbR69WT55SnluUb0ARW4oANa8+p38HiXCRwvIbt8gcO /LNF0C0klhuHg8rD8jlekwxOKLN43aPnnbKDg75zORrdB3pAfFv8rDQnplkjsM35 RPtaVX6Pz8mXxRvy8ToBtoboIH4o4VlFWF2nGjKfG/Y9byriimNRp39kjgfMJUCp fckUDPnB0aoxeIq56ytoEWGDXR0y2z5KiZDfSfovGaijLeql7khL+33fBPEKO8An 04y/Zp3G89BCuZQzr7l/3ZfEMo8NUYZf4/PSTL7URI3rO4aIIQOU1XSLCXnaGIMk ouZfeh0awsde4MisD5j1AwSH4w8DKrTxSTrtA6oEdRKwHEizAv4zLpgG+6m0kkU9 jKac26UMMTOlbb8TTM3qkUO0ewWqVd1hHdLDVnkKzorvbUiXLmsiv05SMXfpme5Q GpsdYLlo6oulHoUbbPGCmAv7DKer3DV/s9bYmFpam5F0E3ZqZ3ec7pYr/rrgTdCO xTA2Jlye3KYryPWshntwhEHdrBdF7L/mOnQ0+A4wHab/IOTX1oY= =Rntz -----END PGP SIGNATURE----- From gbr at majentis.com Thu Dec 27 16:04:04 2018 From: gbr at majentis.com (Gerald Brandt) Date: Thu, 27 Dec 2018 09:04:04 -0600 Subject: [PVE-User] Upgrade from 3.4, or reinstall? Message-ID: <4591bcc1-0afb-49f9-46ba-958a42e0c8b3@majentis.com> Hi, I have an old 3.4 box. Is it worth upgrading, or should I just backup and reinstall? Gerald From brians at iptel.co Thu Dec 27 17:15:15 2018 From: brians at iptel.co (Brian :) Date: Thu, 27 Dec 2018 16:15:15 +0000 Subject: [PVE-User] Upgrade from 3.4, or reinstall? In-Reply-To: <4591bcc1-0afb-49f9-46ba-958a42e0c8b3@majentis.com> References: <4591bcc1-0afb-49f9-46ba-958a42e0c8b3@majentis.com> Message-ID: You will have to go to latest 3.x then to 4.x then to 5.x - upgrade should be fine. But if its quicker to backup VMS and restore and some downtime is acceptable ( downtime with upgrade on single box anyway ) then that maybe easier option for you. On Thu, Dec 27, 2018 at 3:04 PM Gerald Brandt wrote: > > Hi, > > I have an old 3.4 box. Is it worth upgrading, or should I just backup > and reinstall? > > Gerald > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From gaio at sv.lnf.it Fri Dec 28 15:17:39 2018 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Fri, 28 Dec 2018 15:17:39 +0100 Subject: [PVE-User] PVE 4 -> 5, multipath differences? Message-ID: <20181228141739.GB4570@sv.lnf.it> I've just upgraded my cluster frm 4.4 to latest 5. Before that, i've also do a firmware upgrade of the SAN (HP MSA 1040), but seems not related. In old PVE 4.4/jessie i got: dixie:~# multipath -ll mpath0 (3600c0ff00026ed11a7cb565701000000) dm-1 HP,MSA 1040 SAN size=1.4T features='1 queue_if_no_path' hwhandler='0' wp=rw `-+- policy='round-robin 0' prio=1 status=active |- 2:0:0:0 sda 8:0 active ready running |- 3:0:0:0 sdb 8:16 active ready running |- 4:0:0:0 sdc 8:32 active ready running `- 5:0:0:0 sdd 8:48 active ready running in new PVE 5/stretch i got: root at ashpool:~# multipath -ll mpath2 (3600c0ff00026ed11475d215a01000000) dm-0 HP,MSA 1040 SAN size=1.8T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='1 alua' wp=rw |-+- policy='round-robin 0' prio=50 status=active | |- 12:0:0:4 sdg 8:96 active ready running | `- 15:0:0:4 sdr 65:16 active ready running `-+- policy='round-robin 0' prio=10 status=enabled |- 13:0:0:4 sdj 8:144 active ready running `- 14:0:0:4 sds 65:32 active ready running config file is the same. Reading around seems that my SAN is a dual controller Active/Passive one, so probably it is correct to have two path active and two ready, but... i seek some feedback. ;-) Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From nick-liste at posteo.eu Sat Dec 29 11:32:01 2018 From: nick-liste at posteo.eu (Nicola Ferrari (#554252)) Date: Sat, 29 Dec 2018 11:32:01 +0100 Subject: [PVE-User] WebUI Asking login when changing node Message-ID: <4ca227cc-aee5-ec0a-1dc0-f914ff109c57@posteo.eu> Hi list! We're experiencing a little problem on a 3 machines cluster, let's call them pve1, pve2, pve3 accessing webui pointing to https://pve1:8006 shows entire cluster correctly.. ssh on all nodes is working and accessing other nodes doesn't require credentials (so keys are working) live migration between nodes is working... but... clicking on pve2 (or any resource on pve2) if viewing interface at pve1's ip keeps asking to re-enter login credentials! so to manage pve2 resources we need to point to pve2's ip and everything is fine (but obviewsly clicking on pve1 from pve2 re-asks login...) This happens after cluster upgrade from pve4 to pve5 in last july.. pveversion -v output: proxmox-ve: 5.2-2 (running kernel: 4.15.18-1-pve) pve-manager: 5.2-6 (running version: 5.2-6/bcd5f008) pve-kernel-4.15: 5.2-4 pve-kernel-4.15.18-1-pve: 4.15.18-17 pve-kernel-4.13.16-2-pve: 4.13.16-48 corosync: 2.4.2-pve5 criu: 2.11.1-1~bpo90 glusterfs-client: 3.8.8-1 ksm-control-daemon: 1.2-2 libjs-extjs: 6.0.1-2 libpve-access-control: 5.0-8 libpve-apiclient-perl: 2.0-5 libpve-common-perl: 5.0-37 libpve-guest-common-perl: 2.0-17 libpve-http-server-perl: 2.0-9 libpve-storage-perl: 5.0-24 libqb0: 1.0.1-1 lvm2: 2.02.168-pve6 lxc-pve: 3.0.0-3 lxcfs: 3.0.0-1 novnc-pve: 1.0.0-2 proxmox-widget-toolkit: 1.0-19 pve-cluster: 5.0-29 pve-container: 2.0-24 pve-docs: 5.2-5 pve-firewall: 3.0-13 pve-firmware: 2.0-5 pve-ha-manager: 2.0-5 pve-i18n: 1.0-6 pve-libspice-server1: 0.12.8-3 pve-qemu-kvm: 2.11.2-1 pve-xtermjs: 1.0-5 qemu-server: 5.0-30 smartmontools: 6.5+svn4324-1 spiceterm: 3.0-5 vncterm: 1.5-3 Any hints from you experts? :) Thanks to everybody! Nick -- +---------------------+ | Linux User #554252 | +---------------------+ From yannis.milios at gmail.com Sat Dec 29 15:13:10 2018 From: yannis.milios at gmail.com (Yannis Milios) Date: Sat, 29 Dec 2018 15:13:10 +0100 Subject: [PVE-User] WebUI Asking login when changing node In-Reply-To: <4ca227cc-aee5-ec0a-1dc0-f914ff109c57@posteo.eu> References: <4ca227cc-aee5-ec0a-1dc0-f914ff109c57@posteo.eu> Message-ID: A few things I would try are ... - Clear browser cache. - Check if installed package versions are the same on *all* nodes (pveversion -v). - Restart pveproxy service on pve2 (or any other related service). - Check the logs of pve2 when you try accessing it for any clues. Do you get the same problem when accessing pve2 from pve3? Yannis On Sat, 29 Dec 2018 at 11:32, Nicola Ferrari (#554252) wrote: > Hi list! > > We're experiencing a little problem on a 3 machines cluster, let's call > them pve1, pve2, pve3 > > accessing webui pointing to https://pve1:8006 shows entire cluster > correctly.. > ssh on all nodes is working and accessing other nodes doesn't require > credentials (so keys are working) > live migration between nodes is working... but... > > clicking on pve2 (or any resource on pve2) if viewing interface at > pve1's ip keeps asking to re-enter login credentials! > so to manage pve2 resources we need to point to pve2's ip and everything > is fine (but obviewsly clicking on pve1 from pve2 re-asks login...) > > This happens after cluster upgrade from pve4 to pve5 in last july.. > > pveversion -v output: > proxmox-ve: 5.2-2 (running kernel: 4.15.18-1-pve) > pve-manager: 5.2-6 (running version: 5.2-6/bcd5f008) > pve-kernel-4.15: 5.2-4 > pve-kernel-4.15.18-1-pve: 4.15.18-17 > pve-kernel-4.13.16-2-pve: 4.13.16-48 > corosync: 2.4.2-pve5 > criu: 2.11.1-1~bpo90 > glusterfs-client: 3.8.8-1 > ksm-control-daemon: 1.2-2 > libjs-extjs: 6.0.1-2 > libpve-access-control: 5.0-8 > libpve-apiclient-perl: 2.0-5 > libpve-common-perl: 5.0-37 > libpve-guest-common-perl: 2.0-17 > libpve-http-server-perl: 2.0-9 > libpve-storage-perl: 5.0-24 > libqb0: 1.0.1-1 > lvm2: 2.02.168-pve6 > lxc-pve: 3.0.0-3 > lxcfs: 3.0.0-1 > novnc-pve: 1.0.0-2 > proxmox-widget-toolkit: 1.0-19 > pve-cluster: 5.0-29 > pve-container: 2.0-24 > pve-docs: 5.2-5 > pve-firewall: 3.0-13 > pve-firmware: 2.0-5 > pve-ha-manager: 2.0-5 > pve-i18n: 1.0-6 > pve-libspice-server1: 0.12.8-3 > pve-qemu-kvm: 2.11.2-1 > pve-xtermjs: 1.0-5 > qemu-server: 5.0-30 > smartmontools: 6.5+svn4324-1 > spiceterm: 3.0-5 > vncterm: 1.5-3 > > > Any hints from you experts? :) > Thanks to everybody! > > Nick > > > -- > +---------------------+ > | Linux User #554252 | > +---------------------+ > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Sent from Gmail Mobile From alfiomunoz at gmail.com Sat Dec 29 15:36:26 2018 From: alfiomunoz at gmail.com (Alfio munoz) Date: Sat, 29 Dec 2018 10:36:26 -0400 Subject: [PVE-User] WebUI Asking login when changing node In-Reply-To: <4ca227cc-aee5-ec0a-1dc0-f914ff109c57@posteo.eu> References: <4ca227cc-aee5-ec0a-1dc0-f914ff109c57@posteo.eu> Message-ID: Hi, you can also check the date and the time of the three servers. El s?b., 29 dic. 2018 6:32 a. m., Nicola Ferrari (#554252) < nick-liste at posteo.eu> escribi?: > Hi list! > > We're experiencing a little problem on a 3 machines cluster, let's call > them pve1, pve2, pve3 > > accessing webui pointing to https://pve1:8006 shows entire cluster > correctly.. > ssh on all nodes is working and accessing other nodes doesn't require > credentials (so keys are working) > live migration between nodes is working... but... > > clicking on pve2 (or any resource on pve2) if viewing interface at > pve1's ip keeps asking to re-enter login credentials! > so to manage pve2 resources we need to point to pve2's ip and everything > is fine (but obviewsly clicking on pve1 from pve2 re-asks login...) > > This happens after cluster upgrade from pve4 to pve5 in last july.. > > pveversion -v output: > proxmox-ve: 5.2-2 (running kernel: 4.15.18-1-pve) > pve-manager: 5.2-6 (running version: 5.2-6/bcd5f008) > pve-kernel-4.15: 5.2-4 > pve-kernel-4.15.18-1-pve: 4.15.18-17 > pve-kernel-4.13.16-2-pve: 4.13.16-48 > corosync: 2.4.2-pve5 > criu: 2.11.1-1~bpo90 > glusterfs-client: 3.8.8-1 > ksm-control-daemon: 1.2-2 > libjs-extjs: 6.0.1-2 > libpve-access-control: 5.0-8 > libpve-apiclient-perl: 2.0-5 > libpve-common-perl: 5.0-37 > libpve-guest-common-perl: 2.0-17 > libpve-http-server-perl: 2.0-9 > libpve-storage-perl: 5.0-24 > libqb0: 1.0.1-1 > lvm2: 2.02.168-pve6 > lxc-pve: 3.0.0-3 > lxcfs: 3.0.0-1 > novnc-pve: 1.0.0-2 > proxmox-widget-toolkit: 1.0-19 > pve-cluster: 5.0-29 > pve-container: 2.0-24 > pve-docs: 5.2-5 > pve-firewall: 3.0-13 > pve-firmware: 2.0-5 > pve-ha-manager: 2.0-5 > pve-i18n: 1.0-6 > pve-libspice-server1: 0.12.8-3 > pve-qemu-kvm: 2.11.2-1 > pve-xtermjs: 1.0-5 > qemu-server: 5.0-30 > smartmontools: 6.5+svn4324-1 > spiceterm: 3.0-5 > vncterm: 1.5-3 > > > Any hints from you experts? :) > Thanks to everybody! > > Nick > > > -- > +---------------------+ > | Linux User #554252 | > +---------------------+ > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From andrew.mcrory at sayso.net Sun Dec 30 03:06:13 2018 From: andrew.mcrory at sayso.net (Andrew McRory / CTO) Date: Sat, 29 Dec 2018 21:06:13 -0500 Subject: [PVE-User] WebUI Asking login when changing node In-Reply-To: <4ca227cc-aee5-ec0a-1dc0-f914ff109c57@posteo.eu> References: <4ca227cc-aee5-ec0a-1dc0-f914ff109c57@posteo.eu> Message-ID: <6912892ba155d0085fc836f8e39edd5f@sayso.net> Renew your certificates pvecm updatecerts --force and let us know if that helps --- Andrew McRory Sayso Communications, Inc. On 2018-12-29 05:32, Nicola Ferrari (#554252) wrote: > Hi list! > > We're experiencing a little problem on a 3 machines cluster, let's call > them pve1, pve2, pve3 > > accessing webui pointing to https://pve1:8006 shows entire cluster > correctly.. > ssh on all nodes is working and accessing other nodes doesn't require > credentials (so keys are working) > live migration between nodes is working... but... > > clicking on pve2 (or any resource on pve2) if viewing interface at > pve1's ip keeps asking to re-enter login credentials! > so to manage pve2 resources we need to point to pve2's ip and > everything > is fine (but obviewsly clicking on pve1 from pve2 re-asks login...) > > This happens after cluster upgrade from pve4 to pve5 in last july.. > > pveversion -v output: > proxmox-ve: 5.2-2 (running kernel: 4.15.18-1-pve) > pve-manager: 5.2-6 (running version: 5.2-6/bcd5f008) > pve-kernel-4.15: 5.2-4 > pve-kernel-4.15.18-1-pve: 4.15.18-17 > pve-kernel-4.13.16-2-pve: 4.13.16-48 > corosync: 2.4.2-pve5 > criu: 2.11.1-1~bpo90 > glusterfs-client: 3.8.8-1 > ksm-control-daemon: 1.2-2 > libjs-extjs: 6.0.1-2 > libpve-access-control: 5.0-8 > libpve-apiclient-perl: 2.0-5 > libpve-common-perl: 5.0-37 > libpve-guest-common-perl: 2.0-17 > libpve-http-server-perl: 2.0-9 > libpve-storage-perl: 5.0-24 > libqb0: 1.0.1-1 > lvm2: 2.02.168-pve6 > lxc-pve: 3.0.0-3 > lxcfs: 3.0.0-1 > novnc-pve: 1.0.0-2 > proxmox-widget-toolkit: 1.0-19 > pve-cluster: 5.0-29 > pve-container: 2.0-24 > pve-docs: 5.2-5 > pve-firewall: 3.0-13 > pve-firmware: 2.0-5 > pve-ha-manager: 2.0-5 > pve-i18n: 1.0-6 > pve-libspice-server1: 0.12.8-3 > pve-qemu-kvm: 2.11.2-1 > pve-xtermjs: 1.0-5 > qemu-server: 5.0-30 > smartmontools: 6.5+svn4324-1 > spiceterm: 3.0-5 > vncterm: 1.5-3 > > > Any hints from you experts? :) > Thanks to everybody! > > Nick