From t.lamprecht at proxmox.com Sun Nov 1 13:48:09 2020 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Sun, 1 Nov 2020 13:48:09 +0100 Subject: [PVE-User] Backup broken? In-Reply-To: <1bea39e7-d656-d809-0774-baa935479f53@bdick.de> References: <1bea39e7-d656-d809-0774-baa935479f53@bdick.de> Message-ID: <7d0ea9da-3b9e-d9da-a2f6-1c7f5eca11f8@proxmox.com> Hi, On 31.10.20 15:11, Bernhard Dick wrote: > I've a weekly backup job on my proxmox (community version 6.2) infrastructure. Yesterday I've updated to the latest package versions available and in the night the backup job ran. But he decided to first create a backup and then delete _all_ backups being on the backup location which is now empty besides some log files. > Here is an excerpt of one of the mailed logs: > > 101: 2020-10-31 02:04:31 INFO: creating vzdump archive '/mnt/pve/backup-fsn0/dump/vzdump-lxc-101-2020_10_31-02_04_30.tar.lzo' > 101: 2020-10-31 02:05:15 INFO: Total bytes written: 838041600 (800MiB, 19MiB/s) > 101: 2020-10-31 02:05:16 INFO: archive file size: 383MB > 101: 2020-10-31 02:05:16 INFO: removing backup 'backup-fsn0:backup/vzdump-lxc-101-2020_10_14-00_57_00.tar.zst' > 101: 2020-10-31 02:05:16 INFO: removing backup 'backup-fsn0:backup/vzdump-lxc-101-2020_10_16-19_20_41.tar.lzo' > 101: 2020-10-31 02:05:16 INFO: removing backup 'backup-fsn0:backup/vzdump-lxc-101-2020_10_17-02_10_31.tar.lzo' > 101: 2020-10-31 02:05:17 INFO: removing backup 'backup-fsn0:backup/vzdump-lxc-101-2020_10_24-02_03_29.tar.lzo' > 101: 2020-10-31 02:05:25 INFO: removing backup 'backup-fsn0:backup/vzdump-lxc-101-2020_10_31-02_04_30.tar.lzo' > 101: 2020-10-31 02:05:25 INFO: remove vzdump snapshot > 101: 2020-10-31 02:05:26 INFO: Finished Backup of VM 101 (00:00:56) > It'd be great if you could provided the full task log, not only an excerpt. > Is it possible that I've triggert a very evil bug here? > Can you please post the storage config and the backup job one? I.e., # cat /etc/pve/storage.cfg # cat /etc/pve/vzdump.cron thanks! - Thomas From bernhard at bdick.de Sun Nov 1 14:14:48 2020 From: bernhard at bdick.de (Bernhard Dick) Date: Sun, 1 Nov 2020 14:14:48 +0100 Subject: [PVE-User] Backup broken? In-Reply-To: <7d0ea9da-3b9e-d9da-a2f6-1c7f5eca11f8@proxmox.com> References: <1bea39e7-d656-d809-0774-baa935479f53@bdick.de> <7d0ea9da-3b9e-d9da-a2f6-1c7f5eca11f8@proxmox.com> Message-ID: <746ccce3-578f-c75d-8958-f3776b7f1166@bdick.de> Hi, Am 01.11.2020 um 13:48 schrieb Thomas Lamprecht: > Hi, > > On 31.10.20 15:11, Bernhard Dick wrote: >> I've a weekly backup job on my proxmox (community version 6.2) infrastructure. Yesterday I've updated to the latest package versions available and in the night the backup job ran. But he decided to first create a backup and then delete _all_ backups being on the backup location which is now empty besides some log files. >> Here is an excerpt of one of the mailed logs: >> >> [..] > > It'd be great if you could provided the full task log, not only an excerpt. Here is a full log of two Systems (one container, one VM) of from the job: vzdump --quiet 1 --mailnotification always --mailto bernhard at bdick.de --compress lzo --mode snapshot --storage backup-fsn0 --node he2 --all 1 101: 2020-10-31 20:43:26 INFO: Starting Backup of VM 101 (lxc) 101: 2020-10-31 20:43:26 INFO: status = running 101: 2020-10-31 20:43:26 INFO: CT Name: ns0.bdick.de 101: 2020-10-31 20:43:26 INFO: including mount point rootfs ('/') in backup 101: 2020-10-31 20:43:26 INFO: backup mode: snapshot 101: 2020-10-31 20:43:26 INFO: ionice priority: 7 101: 2020-10-31 20:43:26 INFO: create storage snapshot 'vzdump' 101: 2020-10-31 20:43:27 INFO: creating vzdump archive '/mnt/pve/backup-fsn0/dump/vzdump-lxc-101-2020_10_31-20_43_26.tar.lzo' 101: 2020-10-31 20:44:05 INFO: Total bytes written: 838082560 (800MiB, 21MiB/s) 101: 2020-10-31 20:44:06 INFO: archive file size: 383MB 101: 2020-10-31 20:44:06 INFO: removing backup 'backup-fsn0:backup/vzdump-lxc-101-2020_10_31-20_43_26.tar.lzo' 101: 2020-10-31 20:44:06 INFO: remove vzdump snapshot 101: 2020-10-31 20:44:06 INFO: Finished Backup of VM 101 (00:00:40) 102: 2020-10-31 20:44:06 INFO: Starting Backup of VM 102 (qemu) 102: 2020-10-31 20:44:06 INFO: status = running 102: 2020-10-31 20:44:06 INFO: VM Name: db.bdick.de 102: 2020-10-31 20:44:06 INFO: include disk 'sata0' 'local-zfs:vm-102-disk-0' 8G 102: 2020-10-31 20:44:06 INFO: backup mode: snapshot 102: 2020-10-31 20:44:06 INFO: ionice priority: 7 102: 2020-10-31 20:44:07 INFO: creating vzdump archive '/mnt/pve/backup-fsn0/dump/vzdump-qemu-102-2020_10_31-20_44_06.vma.lzo' 102: 2020-10-31 20:44:07 INFO: started backup task 'edde0ee1-047d-49a4-8b2d-22da79c5b540' 102: 2020-10-31 20:44:07 INFO: resuming VM again 102: 2020-10-31 20:44:10 INFO: 9% (738.9 MiB of 8.0 GiB) in 3s, read: 246.3 MiB/s, write: 45.4 MiB/s 102: 2020-10-31 20:44:13 INFO: 12% (1.0 GiB of 8.0 GiB) in 6s, read: 102.4 MiB/s, write: 89.2 MiB/s 102: 2020-10-31 20:44:16 INFO: 16% (1.3 GiB of 8.0 GiB) in 9s, read: 96.8 MiB/s, write: 82.3 MiB/s 102: 2020-10-31 20:44:19 INFO: 17% (1.4 GiB of 8.0 GiB) in 12s, read: 35.8 MiB/s, write: 26.7 MiB/s 102: 2020-10-31 20:44:22 INFO: 21% (1.7 GiB of 8.0 GiB) in 15s, read: 115.6 MiB/s, write: 77.3 MiB/s 102: 2020-10-31 20:44:25 INFO: 24% (2.0 GiB of 8.0 GiB) in 18s, read: 72.1 MiB/s, write: 71.7 MiB/s 102: 2020-10-31 20:44:28 INFO: 26% (2.1 GiB of 8.0 GiB) in 21s, read: 53.6 MiB/s, write: 53.6 MiB/s 102: 2020-10-31 20:44:31 INFO: 27% (2.2 GiB of 8.0 GiB) in 24s, read: 16.1 MiB/s, write: 16.1 MiB/s 102: 2020-10-31 20:44:34 INFO: 29% (2.4 GiB of 8.0 GiB) in 27s, read: 78.6 MiB/s, write: 73.4 MiB/s 102: 2020-10-31 20:44:37 INFO: 33% (2.7 GiB of 8.0 GiB) in 30s, read: 102.6 MiB/s, write: 71.5 MiB/s 102: 2020-10-31 20:44:40 INFO: 36% (2.9 GiB of 8.0 GiB) in 33s, read: 80.7 MiB/s, write: 80.6 MiB/s 102: 2020-10-31 20:44:43 INFO: 39% (3.1 GiB of 8.0 GiB) in 36s, read: 71.3 MiB/s, write: 71.3 MiB/s 102: 2020-10-31 20:44:46 INFO: 41% (3.3 GiB of 8.0 GiB) in 39s, read: 66.5 MiB/s, write: 66.5 MiB/s 102: 2020-10-31 20:44:49 INFO: 44% (3.5 GiB of 8.0 GiB) in 42s, read: 72.2 MiB/s, write: 72.2 MiB/s 102: 2020-10-31 20:44:52 INFO: 47% (3.8 GiB of 8.0 GiB) in 45s, read: 97.6 MiB/s, write: 97.3 MiB/s 102: 2020-10-31 20:44:55 INFO: 52% (4.2 GiB of 8.0 GiB) in 48s, read: 116.8 MiB/s, write: 55.6 MiB/s 102: 2020-10-31 20:44:58 INFO: 89% (7.2 GiB of 8.0 GiB) in 51s, read: 1023.2 MiB/s, write: 0 B/s 102: 2020-10-31 20:44:59 INFO: 100% (8.0 GiB of 8.0 GiB) in 52s, read: 846.9 MiB/s, write: 0 B/s 102: 2020-10-31 20:44:59 INFO: backup is sparse: 4.92 GiB (61%) total zero data 102: 2020-10-31 20:44:59 INFO: transferred 8.00 GiB in 52 seconds (157.5 MiB/s) 102: 2020-10-31 20:44:59 INFO: archive file size: 1.40GB 102: 2020-10-31 20:44:59 INFO: removing backup 'backup-fsn0:backup/vzdump-qemu-102-2020_10_31-20_44_06.vma.lzo' 102: 2020-10-31 20:44:59 INFO: Finished Backup of VM 102 (00:00:53) > >> Is it possible that I've triggert a very evil bug here? >> > > Can you please post the storage config and the backup job one? I.e., > > # cat /etc/pve/storage.cfg dir: local path /var/lib/vz content backup,iso,vztmpl zfspool: local-zfs pool rpool/data content rootdir,images sparse 1 cifs: backup-fsn0 path /mnt/pve/backup-fsn0 server uXXXXXX-sub1.your-storagebox.de share uXXXXXX-sub1 content backup maxfiles 0 username uXXXXXX-sub1 > > # cat /etc/pve/vzdump.cron # cluster wide vzdump cron schedule # Automatically generated file - do not edit PATH="/usr/sbin:/usr/bin:/sbin:/bin" 0 2 * * 6 root vzdump --mailto bernhard at bdick.de --all 1 --quiet 1 --compress lzo --mode snapshot --mailnotification always --storage backup-fsn0 I can even reproduce this behaviour by triggering the global Backup job from the web console. If I backup single VMs/Containers from the Host part of the web console it runs fine, however the global job removes also those backups when it is running. Regards, Bernhard > > thanks! > - Thomas > > From t.lamprecht at proxmox.com Sun Nov 1 20:48:41 2020 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Sun, 1 Nov 2020 20:48:41 +0100 Subject: [PVE-User] Backup broken? In-Reply-To: <746ccce3-578f-c75d-8958-f3776b7f1166@bdick.de> References: <1bea39e7-d656-d809-0774-baa935479f53@bdick.de> <7d0ea9da-3b9e-d9da-a2f6-1c7f5eca11f8@proxmox.com> <746ccce3-578f-c75d-8958-f3776b7f1166@bdick.de> Message-ID: <9c16c66e-6a83-4da9-84e8-28efa0ff7c3a@proxmox.com> On 01.11.20 14:14, Bernhard Dick wrote: > I can even reproduce this behaviour by triggering the global Backup job from the web console. If I backup single VMs/Containers from the Host part of the web console it runs fine, however the global job removes also those backups when it is running. Yes, there was a regression with this when adopting the newer prune "keep-daily", "keep-weekly", ... logic. It acts quite different internally, but the storage special case for maxfiles==0 was handled rather implicit, thus this did not rang any alarm bells. I transformed it to a more explicit logic and we'll add some more extensive test for this special case, so that it won't happen again. The fix is packaged in pve-manager version 6.2-15, currently available on pvetest. You can either add the pvetest repository[0], do `apt update && apt install pve-manager`, then drop the test repo again, or manually download and install it, with using `apt install` this still checks if the package is valid (i.e., signed by a trusted key): # wget http://download.proxmox.com/debian/pve/dists/buster/pvetest/binary-amd64/pve-manager_6.2-15_amd64.deb # apt install ./pve-manager_6.2-15_amd64.deb thanks for your report! regards, Thomas [0]: https://pve.proxmox.com/wiki/Package_Repositories From t.lamprecht at proxmox.com Sun Nov 1 21:17:57 2020 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Sun, 1 Nov 2020 21:17:57 +0100 Subject: [PVE-User] Backup broken? In-Reply-To: <9c16c66e-6a83-4da9-84e8-28efa0ff7c3a@proxmox.com> References: <1bea39e7-d656-d809-0774-baa935479f53@bdick.de> <7d0ea9da-3b9e-d9da-a2f6-1c7f5eca11f8@proxmox.com> <746ccce3-578f-c75d-8958-f3776b7f1166@bdick.de> <9c16c66e-6a83-4da9-84e8-28efa0ff7c3a@proxmox.com> Message-ID: <64aa6821-c21f-fba2-091b-89536d5d6b5f@proxmox.com> On 01.11.20 20:48, Thomas Lamprecht wrote: > with using > `apt install` this still checks if the package is valid (i.e., signed by a trusted > key): > > # wget http://download.proxmox.com/debian/pve/dists/buster/pvetest/binary-amd64/pve-manager_6.2-15_amd64.deb > # apt install ./pve-manager_6.2-15_amd64.deb Albeit, I have some feeling I was confused here, it's late Sunday after all, so be safe check the SHA256 sum. # sha256sum pve-manager_6.2-15_amd64.deb 3b5a7377406ae9d85757a5ac31d9e2688a87acf878d6bc0ccde2e5475cffa651 pve-manager_6.2-15_amd64.deb cheers, Thomas From bernhard at bdick.de Mon Nov 2 10:17:10 2020 From: bernhard at bdick.de (Bernhard Dick) Date: Mon, 2 Nov 2020 10:17:10 +0100 Subject: [PVE-User] Backup broken? In-Reply-To: <9c16c66e-6a83-4da9-84e8-28efa0ff7c3a@proxmox.com> References: <1bea39e7-d656-d809-0774-baa935479f53@bdick.de> <7d0ea9da-3b9e-d9da-a2f6-1c7f5eca11f8@proxmox.com> <746ccce3-578f-c75d-8958-f3776b7f1166@bdick.de> <9c16c66e-6a83-4da9-84e8-28efa0ff7c3a@proxmox.com> Message-ID: <761c2c57-6f30-b784-2233-284ed20dc2cb@bdick.de> Hi, Am 01.11.2020 um 20:48 schrieb Thomas Lamprecht: > On 01.11.20 14:14, Bernhard Dick wrote: >> I can even reproduce this behaviour by triggering the global Backup job from the web console. If I backup single VMs/Containers from the Host part of the web console it runs fine, however the global job removes also those backups when it is running. > > Yes, there was a regression with this when adopting the newer prune "keep-daily", > "keep-weekly", ... logic. It acts quite different internally, but the storage > special case for maxfiles==0 was handled rather implicit, thus this did not > rang any alarm bells. I transformed it to a more explicit logic and we'll > add some more extensive test for this special case, so that it won't happen again. > > The fix is packaged in pve-manager version 6.2-15, currently available on pvetest. > You can either add the pvetest repository[0], do `apt update && apt install pve-manager`, > then drop the test repo again, or manually download and install it, with using > `apt install` this still checks if the package is valid (i.e., signed by a trusted > key): > > # wget http://download.proxmox.com/debian/pve/dists/buster/pvetest/binary-amd64/pve-manager_6.2-15_amd64.deb > # apt install ./pve-manager_6.2-15_amd64.deb > I tried it now and it works as expected. So thanks for fixing this fast. Regards Bernhard > thanks for your report! > > regards, > Thomas > > > [0]: https://pve.proxmox.com/wiki/Package_Repositories > > From devzero at web.de Tue Nov 3 11:43:25 2020 From: devzero at web.de (Roland) Date: Tue, 3 Nov 2020 11:43:25 +0100 Subject: [PVE-User] pbs datastore view weird behaviour Message-ID: <4f14e929-2118-11fa-6959-b4ae620fb7d6@web.de> hello, i'm sure that i could click on "datastore" root node in the pbs gui to get a complete view on all datastore configuration. i found this useful. furthermore, all other root-nodes containing sub-nodes (configuration, administration) can be expanded/collapsed - datastore root node behaves different. is this changed intentionally in one of the last updates (i.e. feature change) or is this a bug ? regard roland From f.gruenbichler at proxmox.com Tue Nov 3 14:03:46 2020 From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=) Date: Tue, 03 Nov 2020 14:03:46 +0100 Subject: [PVE-User] pbs datastore view weird behaviour In-Reply-To: <4f14e929-2118-11fa-6959-b4ae620fb7d6@web.de> References: <4f14e929-2118-11fa-6959-b4ae620fb7d6@web.de> Message-ID: <1604408555.h89cyy4u6g.astroid@nora.none> On November 3, 2020 11:43 am, Roland wrote: > hello, > > i'm sure that i could click on "datastore" root node in the pbs gui to > get a complete view on all datastore configuration. i found this useful. this was possible, yes > furthermore, all other root-nodes containing sub-nodes (configuration, > administration) can be expanded/collapsed - datastore root node behaves > different. > > is this changed intentionally in one of the last updates (i.e. feature > change) or is this a bug ? most of the content of that panel was moved to the revamped new datastore view (which now has tabs to view content, setup job schedules, etc.). adding a new datastore was moved to the tree. From devzero at web.de Tue Nov 3 14:55:21 2020 From: devzero at web.de (Roland) Date: Tue, 3 Nov 2020 14:55:21 +0100 Subject: [PVE-User] pbs datastore view weird behaviour In-Reply-To: <1604408555.h89cyy4u6g.astroid@nora.none> References: <4f14e929-2118-11fa-6959-b4ae620fb7d6@web.de> <1604408555.h89cyy4u6g.astroid@nora.none> Message-ID: <7fa36132-db01-5093-f17f-528574829ce3@web.de> thanks for letting us know Am 03.11.20 um 14:03 schrieb Fabian Gr?nbichler: > On November 3, 2020 11:43 am, Roland wrote: >> hello, >> >> i'm sure that i could click on "datastore" root node in the pbs gui to >> get a complete view on all datastore configuration. i found this useful. > this was possible, yes > >> furthermore, all other root-nodes containing sub-nodes (configuration, >> administration) can be expanded/collapsed - datastore root node behaves >> different. >> >> is this changed intentionally in one of the last updates (i.e. feature >> change) or is this a bug ? > most of the content of that panel was moved to the revamped new > datastore view (which now has tabs to view content, setup job schedules, > etc.). adding a new datastore was moved to the tree. > > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gaio at sv.lnf.it Wed Nov 4 18:21:41 2020 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Wed, 4 Nov 2020 18:21:41 +0100 Subject: [PVE-User] ZFS 'zpool trim'? Message-ID: <20201104172141.GH3229@sv.lnf.it> I've tried a 'zpool trim' on a SSD pool in one of my server, after searching about and finding: https://openzfs.org/wiki/Features#TRIM_Support Seems to work as expected (at least: print in logs: Nov 4 18:12:38 ino zed: eid=68 class=trim_start pool_guid=0x76DD565E9F8A86A8 vdev_path=/dev/disk/by-id/ata-KINGSTON_SEDC500M480G_50026B7683586349-part3 vdev_state=ONLINE Nov 4 18:12:38 ino zed: eid=69 class=history_event pool_guid=0x76DD565E9F8A86A8 Nov 4 18:13:37 ino zed: eid=70 class=trim_finish pool_guid=0x76DD565E9F8A86A8 vdev_path=/dev/disk/by-id/ata-KINGSTON_SEDC500M480G_50026B7683586349-part3 vdev_state=ONLINE Nov 4 18:13:37 ino zed: eid=71 class=history_event pool_guid=0x76DD565E9F8A86A8 and don't destroy my disk. ;-) But i've tried to look into PVE wiki, or in PVE ZFS packages for some docs, or scripts, and found nothing. There's some drawbacks of trimming ZFS pools in PVE, and so it is not enabled by default? Or simply there's no way to determine a ZFS pool composed of SSD disks, and so it is up to the sysadmin to setup a script for trimming? Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From mark at openvs.co.uk Wed Nov 4 18:28:06 2020 From: mark at openvs.co.uk (Mark Adams) Date: Wed, 4 Nov 2020 17:28:06 +0000 Subject: [PVE-User] ZFS 'zpool trim'? In-Reply-To: <20201104172141.GH3229@sv.lnf.it> References: <20201104172141.GH3229@sv.lnf.it> Message-ID: Can't comment on why it's not enabled by default (or a checkbox maybe when creating through the gui?) but just enable autotrim on your ssd pools. zpool set autotrim=on POOLNAME On Wed, 4 Nov 2020 at 17:22, Marco Gaiarin wrote: > > I've tried a 'zpool trim' on a SSD pool in one of my server, after > searching about and finding: > > https://openzfs.org/wiki/Features#TRIM_Support > > Seems to work as expected (at least: print in logs: > > Nov 4 18:12:38 ino zed: eid=68 class=trim_start > pool_guid=0x76DD565E9F8A86A8 > vdev_path=/dev/disk/by-id/ata-KINGSTON_SEDC500M480G_50026B7683586349-part3 > vdev_state=ONLINE > Nov 4 18:12:38 ino zed: eid=69 class=history_event > pool_guid=0x76DD565E9F8A86A8 > Nov 4 18:13:37 ino zed: eid=70 class=trim_finish > pool_guid=0x76DD565E9F8A86A8 > vdev_path=/dev/disk/by-id/ata-KINGSTON_SEDC500M480G_50026B7683586349-part3 > vdev_state=ONLINE > Nov 4 18:13:37 ino zed: eid=71 class=history_event > pool_guid=0x76DD565E9F8A86A8 > > and don't destroy my disk. ;-) > > > But i've tried to look into PVE wiki, or in PVE ZFS packages for some > docs, or scripts, and found nothing. > > > There's some drawbacks of trimming ZFS pools in PVE, and so it is not > enabled by default? > Or simply there's no way to determine a ZFS pool composed of SSD disks, > and so it is up to the sysadmin to setup a script for trimming? > > > Thanks. > > -- > dott. Marco Gaiarin GNUPG Key ID: > 240A3D66 > Associazione ``La Nostra Famiglia'' > http://www.lanostrafamiglia.it/ > Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento > (PN) > marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f > +39-0434-842797 > > Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! > http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 > (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From mark at tuxis.nl Wed Nov 4 23:45:06 2020 From: mark at tuxis.nl (Mark Schouten) Date: Wed, 4 Nov 2020 23:45:06 +0100 Subject: [PVE-User] ZFS 'zpool trim'? In-Reply-To: References: <20201104172141.GH3229@sv.lnf.it> Message-ID: <20201104224506.ux552gi7ahvooolb@shell.tuxis.net> On Wed, Nov 04, 2020 at 05:28:06PM +0000, Mark Adams via pve-user wrote: > Can't comment on why it's not enabled by default (or a checkbox maybe when > creating through the gui?) but just enable autotrim on your ssd pools. > > zpool set autotrim=on POOLNAME I did this, at first. But then disabled it becuase it slowed down my storage quite a bit. Might be specific to my setup, but the trim-kernel-thread took quite some IO according to iotop. -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info at tuxis.nl From gaio at sv.lnf.it Fri Nov 6 12:43:11 2020 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Fri, 6 Nov 2020 12:43:11 +0100 Subject: [PVE-User] ZFS 'zpool trim'? In-Reply-To: <20201104224506.ux552gi7ahvooolb@shell.tuxis.net> References: <20201104172141.GH3229@sv.lnf.it> <20201104224506.ux552gi7ahvooolb@shell.tuxis.net> Message-ID: <20201106114311.GF4137@sv.lnf.it> Mandi! Mark Schouten In chel di` si favelave... > > zpool set autotrim=on POOLNAME > I did this, at first. But then disabled it becuase it slowed down my > storage quite a bit. Might be specific to my setup, but the > trim-kernel-thread took quite some IO according to iotop. Exactly, i've read also that, that also 'match' with what 'normally' i know on trim: apart some specific usage case, doing an automatic trim could be a slowdown, and is preferable to schedule a daily/weekly/monthly task, based on usage profiles, to do a trim. It seems strane to me that debian/Proxmox does not have a 'framework' (apart /etc/cron.d/zfsutils-linux; i meant something like a predefined systemd unit) for that. A crontab will suffices. ;-) -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From gaio at sv.lnf.it Fri Nov 6 17:30:03 2020 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Fri, 6 Nov 2020 17:30:03 +0100 Subject: [PVE-User] Network hardware tuning and bond... Message-ID: <20201106163003.GN4137@sv.lnf.it> [Probably it is all but a specific PVE question, but...] I need to setup a PVE standalone node that have to host some VMs, one of that have to be a 'firewall'; one or more NICs are dedicated to WAN link. Following the 'bufferbloat' recommendation: https://www.bufferbloat.net/projects/bloat/wiki/Tests_for_Bufferbloat/ i've added to a typical bridge: iface enp2s0f1 inet manual ethernet-autoneg on link-speed 100 link-duplex full hardware-dma-ring-tx 18 offload-tso off offload-gso off offload-gro off auto vmbr1 iface vmbr1 inet static address 172.17.44.50 netmask 255.255.255.0 gateway 172.17.44.1 bridge_ports enp2s0f1 bridge_waitport 30 bridge_stp off bridge_fd 0 and seems to work, eg link speed is 100Mbit/s and offload parameters are off. But i've tried to do the same thing with an LACP bond, adding this parameters both to single interfaces and also on the bond interface, but seems get ignored. It is not possible to set hardware parameters in bond and/or in bond members?! Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From elacunza at binovo.es Tue Nov 10 09:03:02 2020 From: elacunza at binovo.es (Eneko Lacunza) Date: Tue, 10 Nov 2020 09:03:02 +0100 Subject: Matching WUI VM hardware disks to Linux guest disks Message-ID: <030bb7dd-6d29-0302-6225-9e2f213c32ca@binovo.es> Hi all, I have hit a simple problem. Let be a VM with 3 disks, with .conf extract: scsi0: ceph-proxmox:vm-100-disk-1,cache=writeback,size=6G scsi1: ceph-proxmox:vm-100-disk-0,cache=writeback,size=400G scsi2: ceph-proxmox:vm-100-disk-3,cache=writeback,size=400G We have two virtual disks with identical size (400G). How can I be sure what device on Linux guest is each? # lsblk NAME????????????????? MAJ:MIN RM? SIZE RO TYPE? MOUNTPOINT sda???????????????????? 8:0??? 0??? 6G? 0 disk ??sda1????????????????? 8:1??? 0? 5.7G? 0 part? / ??sda2????????????????? 8:2??? 0??? 1K? 0 part ??sda5????????????????? 8:5??? 0? 283M? 0 part? [SWAP] sdb???????????????????? 8:16?? 0? 400G? 0 disk? /mnt sdc???????????????????? 8:32?? 0? 400G? 0 disk sr0??????????????????? 11:0??? 1 1024M? 0 rom What disk is sdb and what sdc? :-) In this case, I know sdc is scsi1, and sdb is scsi2; which is counter-intuitive and WUI doesn't seem to offer any help for this. (I just added 1G to a disk and then checked sizes on guest). Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza at binovo.es Zuzendari teknikoa | https://www.binovo.es Director t?cnico | Astigarragako Bidea, 2 - 2? izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun From elacunza at binovo.es Tue Nov 10 09:07:42 2020 From: elacunza at binovo.es (Eneko Lacunza) Date: Tue, 10 Nov 2020 09:07:42 +0100 Subject: Matching WUI VM hardware disks to Linux guest disks In-Reply-To: <030bb7dd-6d29-0302-6225-9e2f213c32ca@binovo.es> References: <030bb7dd-6d29-0302-6225-9e2f213c32ca@binovo.es> Message-ID: <9d1d4b23-14dd-0578-698f-1d621ef7a227@binovo.es> Hi again, Ok, just clicking the "send mail" button revealed the solution ;) # lsscsi [1:0:0:0]??? cd/dvd? QEMU???? QEMU DVD-ROM???? 2.5+? /dev/sr0 [2:0:0:0]??? disk??? QEMU???? QEMU HARDDISK??? 2.5+? /dev/sda [2:0:0:1]??? disk??? QEMU???? QEMU HARDDISK??? 2.5+? /dev/sdc [2:0:0:2]??? disk??? QEMU???? QEMU HARDDISK??? 2.5+? /dev/sdb SCSI IDs match WUI scsiX numbers. No sure wether that can be improved somehow? :-) Thanks and sorry for the noise. El 10/11/20 a las 9:03, Eneko Lacunza escribi?: > Hi all, > > I have hit a simple problem. Let be a VM with 3 disks, with .conf > extract: > > scsi0: ceph-proxmox:vm-100-disk-1,cache=writeback,size=6G > scsi1: ceph-proxmox:vm-100-disk-0,cache=writeback,size=400G > scsi2: ceph-proxmox:vm-100-disk-3,cache=writeback,size=400G > > We have two virtual disks with identical size (400G). > > How can I be sure what device on Linux guest is each? > > # lsblk > NAME????????????????? MAJ:MIN RM? SIZE RO TYPE? MOUNTPOINT > sda???????????????????? 8:0??? 0??? 6G? 0 disk > ??sda1????????????????? 8:1??? 0? 5.7G? 0 part? / > ??sda2????????????????? 8:2??? 0??? 1K? 0 part > ??sda5????????????????? 8:5??? 0? 283M? 0 part? [SWAP] > sdb???????????????????? 8:16?? 0? 400G? 0 disk? /mnt > sdc???????????????????? 8:32?? 0? 400G? 0 disk > sr0??????????????????? 11:0??? 1 1024M? 0 rom > > What disk is sdb and what sdc? :-) > > In this case, I know sdc is scsi1, and sdb is scsi2; which is > counter-intuitive and WUI doesn't seem to offer any help for this. > > (I just added 1G to a disk and then checked sizes on guest). > > Cheers > -- Eneko Lacunza | +34 943 569 206 | elacunza at binovo.es Zuzendari teknikoa | https://www.binovo.es Director t?cnico | Astigarragako Bidea, 2 - 2? izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun From leesteken at protonmail.ch Tue Nov 10 09:12:25 2020 From: leesteken at protonmail.ch (Arjen) Date: Tue, 10 Nov 2020 08:12:25 +0000 Subject: [PVE-User] Matching WUI VM hardware disks to Linux guest disks In-Reply-To: References: Message-ID: On Tuesday, November 10, 2020 9:03 AM, Eneko Lacunza via pve-user wrote: > Hi all, > > I have hit a simple problem. Let be a VM with 3 disks, with .conf extract: > > scsi0: ceph-proxmox:vm-100-disk-1,cache=writeback,size=6G > scsi1: ceph-proxmox:vm-100-disk-0,cache=writeback,size=400G > scsi2: ceph-proxmox:vm-100-disk-3,cache=writeback,size=400G > > We have two virtual disks with identical size (400G). > > How can I be sure what device on Linux guest is each? > > lsblk > > ====== > > NAME????????????????? MAJ:MIN RM? SIZE RO TYPE? MOUNTPOINT > sda???????????????????? 8:0??? 0??? 6G? 0 disk > ??sda1????????????????? 8:1??? 0? 5.7G? 0 part? / > ??sda2????????????????? 8:2??? 0??? 1K? 0 part > ??sda5????????????????? 8:5??? 0? 283M? 0 part? [SWAP] > sdb???????????????????? 8:16?? 0? 400G? 0 disk? /mnt > sdc???????????????????? 8:32?? 0? 400G? 0 disk > sr0??????????????????? 11:0??? 1 1024M? 0 rom > > What disk is sdb and what sdc? :-) > > In this case, I know sdc is scsi1, and sdb is scsi2; which is > counter-intuitive and WUI doesn't seem to offer any help for this. > > (I just added 1G to a disk and then checked sizes on guest). I think you can deduce the SCSI number from the MINor device number on Linux: 0/16 = 0: scsi0 16/16 = 1: scsi1 32/16 = 2: scsi2 See for more details: https://tldp.org/HOWTO/SCSI-2.4-HOWTO/dnames.html Best regards From elacunza at binovo.es Tue Nov 10 09:21:45 2020 From: elacunza at binovo.es (Eneko Lacunza) Date: Tue, 10 Nov 2020 09:21:45 +0100 Subject: [PVE-User] Matching WUI VM hardware disks to Linux guest disks In-Reply-To: References: Message-ID: <6231215d-deeb-c475-50e0-1e0ce86e69a4@binovo.es> Hi Arjen, El 10/11/20 a las 9:12, Arjen via pve-user escribi?: >> NAME????????????????? MAJ:MIN RM? SIZE RO TYPE? MOUNTPOINT >> sda???????????????????? 8:0??? 0??? 6G? 0 disk >> ??sda1????????????????? 8:1??? 0? 5.7G? 0 part? / >> ??sda2????????????????? 8:2??? 0??? 1K? 0 part >> ??sda5????????????????? 8:5??? 0? 283M? 0 part? [SWAP] >> sdb???????????????????? 8:16?? 0? 400G? 0 disk? /mnt >> sdc???????????????????? 8:32?? 0? 400G? 0 disk >> sr0??????????????????? 11:0??? 1 1024M? 0 rom >> >> What disk is sdb and what sdc? :-) >> >> In this case, I know sdc is scsi1, and sdb is scsi2; which is >> counter-intuitive and WUI doesn't seem to offer any help for this. >> >> (I just added 1G to a disk and then checked sizes on guest). > I think you can deduce the SCSI number from the MINor device number on Linux: > 0/16 = 0: scsi0 > 16/16 = 1: scsi1 > 32/16 = 2: scsi2 > > See for more details:https://tldp.org/HOWTO/SCSI-2.4-HOWTO/dnames.html I'm afraid this doesn't work on this case... scsi1 is sdc, 32/16 = 2... Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza at binovo.es Zuzendari teknikoa | https://www.binovo.es Director t?cnico | Astigarragako Bidea, 2 - 2? izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun From d.csapak at proxmox.com Tue Nov 10 10:26:00 2020 From: d.csapak at proxmox.com (Dominik Csapak) Date: Tue, 10 Nov 2020 10:26:00 +0100 Subject: [PVE-User] Matching WUI VM hardware disks to Linux guest disks In-Reply-To: References: Message-ID: <41c2acee-e0c9-0944-66f5-a41b8706122a@proxmox.com> hi, you can check out `lsblk -o +serial` should output something like: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT SERIAL sda 8:0 0 50G 0 disk drive-scsi0 ??sda1 8:1 0 50G 0 part / sdb 8:16 0 100G 0 disk drive-scsi9 ??sdb1 8:17 0 100G 0 part /mnt/foo sdc 8:32 0 2G 0 disk drive-scsi5 ??sdc1 8:33 0 2G 0 part ??sdc9 8:41 0 8M 0 part sdd 8:48 0 2G 0 disk drive-scsi4 ??sdd1 8:49 0 2G 0 part ??sdd9 8:57 0 8M 0 part sde 8:64 0 2G 0 disk drive-scsi3 ??sde1 8:65 0 2G 0 part ??sde9 8:73 0 8M 0 part sdf 8:80 0 50G 0 disk drive-scsi2 ??sdf1 8:81 0 50G 0 part /mnt/bar sdg 8:96 0 200G 0 disk drive-scsi1 ??sdg1 8:97 0 200G 0 part /mnt/baz sr0 11:0 1 1024M 0 rom QM00003 From elacunza at binovo.es Tue Nov 10 10:42:48 2020 From: elacunza at binovo.es (Eneko Lacunza) Date: Tue, 10 Nov 2020 10:42:48 +0100 Subject: [PVE-User] Matching WUI VM hardware disks to Linux guest disks In-Reply-To: <41c2acee-e0c9-0944-66f5-a41b8706122a@proxmox.com> References: <41c2acee-e0c9-0944-66f5-a41b8706122a@proxmox.com> Message-ID: <4a093a20-da51-a532-f8bd-127caed8aac1@binovo.es> Hi Dominik, El 10/11/20 a las 10:26, Dominik Csapak escribi?: > hi, > > you can check out > `lsblk -o +serial` > > should output something like: > > NAME?? MAJ:MIN RM? SIZE RO TYPE MOUNTPOINT????????????? SERIAL > sda????? 8:0??? 0?? 50G? 0 disk drive-scsi0 > ??sda1?? 8:1??? 0?? 50G? 0 part / > sdb????? 8:16?? 0? 100G? 0 disk drive-scsi9 > ??sdb1?? 8:17?? 0? 100G? 0 part /mnt/foo > sdc????? 8:32?? 0??? 2G? 0 disk drive-scsi5 > ??sdc1?? 8:33?? 0??? 2G? 0 part > ??sdc9?? 8:41?? 0??? 8M? 0 part > sdd????? 8:48?? 0??? 2G? 0 disk drive-scsi4 > ??sdd1?? 8:49?? 0??? 2G? 0 part > ??sdd9?? 8:57?? 0??? 8M? 0 part > sde????? 8:64?? 0??? 2G? 0 disk drive-scsi3 > ??sde1?? 8:65?? 0??? 2G? 0 part > ??sde9?? 8:73?? 0??? 8M? 0 part > sdf????? 8:80?? 0?? 50G? 0 disk drive-scsi2 > ??sdf1?? 8:81?? 0?? 50G? 0 part /mnt/bar > sdg????? 8:96?? 0? 200G? 0 disk drive-scsi1 > ??sdg1?? 8:97?? 0? 200G? 0 part /mnt/baz > sr0???? 11:0??? 1 1024M? 0 rom????????????????????????? QM00003 > Yes, this is straightforward, just one more flag to remember. Thanks! -- Eneko Lacunza | +34 943 569 206 | elacunza at binovo.es Zuzendari teknikoa | https://www.binovo.es Director t?cnico | Astigarragako Bidea, 2 - 2? izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun From chris.hofstaedtler at deduktiva.com Tue Nov 10 10:53:31 2020 From: chris.hofstaedtler at deduktiva.com (Chris Hofstaedtler | Deduktiva) Date: Tue, 10 Nov 2020 10:53:31 +0100 Subject: [PVE-User] Matching WUI VM hardware disks to Linux guest disks In-Reply-To: References: Message-ID: <20201110095331.rq2nwbvk26izj6r2@zeha.at> Hi, * Eneko Lacunza via pve-user [201110 09:03]: > I have hit a simple problem. Let be a VM with 3 disks, with .conf extract: > > scsi0: ceph-proxmox:vm-100-disk-1,cache=writeback,size=6G > scsi1: ceph-proxmox:vm-100-disk-0,cache=writeback,size=400G > scsi2: ceph-proxmox:vm-100-disk-3,cache=writeback,size=400G > > We have two virtual disks with identical size (400G). > > How can I be sure what device on Linux guest is each? You can also check - and use in /etc/fstab - the /dev/disk/by-* symlinks. In a VM, maybe the most relevant "id" is the actual path. /dev/disk/by-path has these links (in my case): pci-0000:06:05.0-scsi-0:0:0:0 -> sda pci-0000:00:05.0-scsi-0:0:0:1 -> sdb If your sdb/sdc are swapped, the SCSI IDs in the path should still be correct. If you don't like the pci path in there, /dev/disk/by-id has: scsi-0QEMU_QEMU_HARDDISK_drive-scsi0 -> sda scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 -> sdb But you'll have to check if those match with the VM settings (I'd expect them to). As you've discovered and others have said, lsscsi, or lsblk -S can be used to see the SCSI IDs, too. The same info is also availabe from udevadm: udevadm info /dev/sda If you dig around in /sys, it's also there ;-) HTH, Chris -- Chris Hofstaedtler / Deduktiva GmbH (FN 418592 b, HG Wien) www.deduktiva.com / +43 1 353 1707 From elacunza at binovo.es Tue Nov 10 15:01:33 2020 From: elacunza at binovo.es (Eneko Lacunza) Date: Tue, 10 Nov 2020 15:01:33 +0100 Subject: [PVE-User] Matching WUI VM hardware disks to Linux guest disks In-Reply-To: <20201110095331.rq2nwbvk26izj6r2@zeha.at> References: <20201110095331.rq2nwbvk26izj6r2@zeha.at> Message-ID: Hi Chris, El 10/11/20 a las 10:53, Chris Hofstaedtler | Deduktiva escribi?: > Hi, > > * Eneko Lacunza via pve-user [201110 09:03]: >> I have hit a simple problem. Let be a VM with 3 disks, with .conf extract: >> >> scsi0: ceph-proxmox:vm-100-disk-1,cache=writeback,size=6G >> scsi1: ceph-proxmox:vm-100-disk-0,cache=writeback,size=400G >> scsi2: ceph-proxmox:vm-100-disk-3,cache=writeback,size=400G >> >> We have two virtual disks with identical size (400G). >> >> How can I be sure what device on Linux guest is each? > You can also check - and use in /etc/fstab - the /dev/disk/by-* > symlinks. > > In a VM, maybe the most relevant "id" is the actual path. > /dev/disk/by-path has these links (in my case): > > pci-0000:06:05.0-scsi-0:0:0:0 -> sda > pci-0000:00:05.0-scsi-0:0:0:1 -> sdb > > If your sdb/sdc are swapped, the SCSI IDs in the path should still > be correct. lrwxrwxrwx 1 root root? 9 Oct 29 15:24 pci-0000:00:05.0-scsi-0:0:0:0 -> ../../sda lrwxrwxrwx 1 root root 10 Oct 29 15:24 pci-0000:00:05.0-scsi-0:0:0:0-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Oct 29 15:24 pci-0000:00:05.0-scsi-0:0:0:0-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Oct 29 15:24 pci-0000:00:05.0-scsi-0:0:0:0-part5 -> ../../sda5 lrwxrwxrwx 1 root root? 9 Nov 10 09:08 pci-0000:00:05.0-scsi-0:0:0:1 -> ../../sdc lrwxrwxrwx 1 root root? 9 Oct 29 15:24 pci-0000:00:05.0-scsi-0:0:0:2 -> ../../sdb Yes, they are. > If you don't like the pci path in there, /dev/disk/by-id has: > scsi-0QEMU_QEMU_HARDDISK_drive-scsi0 -> sda > scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 -> sdb > > But you'll have to check if those match with the VM settings (I'd > expect them to). lrwxrwxrwx 1 root root? 9 Oct 29 15:24 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0 -> ../../sda lrwxrwxrwx 1 root root 10 Oct 29 15:24 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Oct 29 15:24 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Oct 29 15:24 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-part5 -> ../../sda5 lrwxrwxrwx 1 root root? 9 Nov 10 09:08 scsi-0QEMU_QEMU_HARDDISK_drive-scsi1 -> ../../sdc lrwxrwxrwx 1 root root? 9 Oct 29 15:24 scsi-0QEMU_QEMU_HARDDISK_drive-scsi2 -> ../../sdb Those match too. > As you've discovered and others have said, lsscsi, or lsblk -S can > be used to see the SCSI IDs, too. The same info is also availabe > from udevadm: udevadm info /dev/sda > If you dig around in /sys, it's also there ;-) Thanks for detailing more check options! :) Cheers -- Eneko Lacunza | +34 943 569 206 | elacunza at binovo.es Zuzendari teknikoa | https://www.binovo.es Director t?cnico | Astigarragako Bidea, 2 - 2? izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun From abonilla at suse.com Tue Nov 10 16:10:57 2020 From: abonilla at suse.com (Alejandro Bonilla) Date: Tue, 10 Nov 2020 15:10:57 +0000 Subject: [PVE-User] qm cli Message-ID: Hello, Whenever I run commands from within one of the nodes, they appear to be targeted to the local system only. root at r620-1:~# for i in {123..128}; do qm snapshot $i ready ; done 2020-11-10 09:15:21.882 7f81d2ffd700 -1 set_mon_vals failed to set cluster_network = 10.0.0.0/24: Configuration option 'cluster_network' may not be modified at runtime ... Configuration file 'nodes/r620-1/qemu-server/124.conf' does not exist Configuration file 'nodes/r620-1/qemu-server/125.conf' does not exist ? ... Configuration file 'nodes/r620-1/qemu-server/127.conf' does not exist Configuration file 'nodes/r620-1/qemu-server/128.conf' does not exist Is there a way to ?centrally? run commands? Thanks From dietmar at proxmox.com Tue Nov 10 16:49:21 2020 From: dietmar at proxmox.com (Dietmar Maurer) Date: Tue, 10 Nov 2020 16:49:21 +0100 (CET) Subject: [PVE-User] qm cli In-Reply-To: References: Message-ID: <1573062207.1043.1605023362305@webmail.proxmox.com> > Whenever I run commands from within one of the nodes, they appear to be targeted to the local system only. Yes, this is how it works. From dietmar at proxmox.com Tue Nov 10 16:49:21 2020 From: dietmar at proxmox.com (Dietmar Maurer) Date: Tue, 10 Nov 2020 16:49:21 +0100 (CET) Subject: [PVE-User] qm cli In-Reply-To: References: Message-ID: <1573062207.1043.1605023362305@webmail.proxmox.com> > Whenever I run commands from within one of the nodes, they appear to be targeted to the local system only. Yes, this is how it works. From s.schwebel at uvensys.de Tue Nov 10 18:12:22 2020 From: s.schwebel at uvensys.de (Steffen Schwebel) Date: Tue, 10 Nov 2020 18:12:22 +0100 Subject: [PVE-User] qm cli In-Reply-To: References: Message-ID: Hello, have you seen the proxmox API documentation? https://pve.proxmox.com/wiki/Proxmox_VE_API On 11/10/20 4:10 PM, Alejandro Bonilla wrote: > Hello, > > Whenever I run commands from within one of the nodes, they appear to be targeted to the local system only. > > root at r620-1:~# for i in {123..128}; do qm snapshot $i ready ; done > 2020-11-10 09:15:21.882 7f81d2ffd700 -1 set_mon_vals failed to set cluster_network = 10.0.0.0/24: Configuration option 'cluster_network' may not be modified at runtime > ... > Configuration file 'nodes/r620-1/qemu-server/124.conf' does not exist > Configuration file 'nodes/r620-1/qemu-server/125.conf' does not exist > ? > ... > Configuration file 'nodes/r620-1/qemu-server/127.conf' does not exist > Configuration file 'nodes/r620-1/qemu-server/128.conf' does not exist > > Is there a way to ?centrally? run commands? > > Thanks > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Steffen Schwebel Mail: s.schwebel at uvensys.de uvensys GmbH Firmensitz und Sitz der Gesellschaft: uvensys GmbH Schorbachstra?e 11 35510 Butzbach HRB: AG Friedberg, 7780 USt-Id: DE282879294 Gesch?ftsf?hrer: Dr. Thomas Licht, t.licht at uvensys.de Volker Lieder, v.lieder at uvensys.de Mail: info at uvensys.de Internet: www.uvensys.de Durchwahl: 06403 789 36 22 Hotline: 06403 789 36 88 Zentrale: 06403 789 36 00 Fax: 06403 789 36 99 ========================================================== Jegliche Stellungnahmen und Meinungen dieser E-Mail sind alleine die des Autors und nicht notwendigerweise die der Firma. Falls erforderlich, k?nnen Sie eine gesonderte schriftliche Best?tigung anfordern. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. If verification is required please request a hard-copy version. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: From mark at tuxis.nl Thu Nov 12 09:54:52 2020 From: mark at tuxis.nl (Mark Schouten) Date: Thu, 12 Nov 2020 09:54:52 +0100 Subject: [PVE-User] qm cli In-Reply-To: <1573062207.1043.1605023362305@webmail.proxmox.com> References: <1573062207.1043.1605023362305@webmail.proxmox.com> Message-ID: <20201112085452.nz56wdqddt4qirio@shell.tuxis.net> On Tue, Nov 10, 2020 at 04:49:21PM +0100, Dietmar Maurer wrote: > > > Whenever I run commands from within one of the nodes, they appear to be targeted to the local system only. > > Yes, this is how it works. I for one would really love to use qm on a cluster base, instead of nodes. I currently first need to find out which node a VM runs on, then log on to that node, and then run the qm commands. Is it usefull to create a feature-request for this in Bugzilla? -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info at tuxis.nl From mark at tuxis.nl Thu Nov 12 09:54:52 2020 From: mark at tuxis.nl (Mark Schouten) Date: Thu, 12 Nov 2020 09:54:52 +0100 Subject: [PVE-User] qm cli In-Reply-To: <1573062207.1043.1605023362305@webmail.proxmox.com> References: <1573062207.1043.1605023362305@webmail.proxmox.com> Message-ID: <20201112085452.nz56wdqddt4qirio@shell.tuxis.net> On Tue, Nov 10, 2020 at 04:49:21PM +0100, Dietmar Maurer wrote: > > > Whenever I run commands from within one of the nodes, they appear to be targeted to the local system only. > > Yes, this is how it works. I for one would really love to use qm on a cluster base, instead of nodes. I currently first need to find out which node a VM runs on, then log on to that node, and then run the qm commands. Is it usefull to create a feature-request for this in Bugzilla? -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info at tuxis.nl From d.jaeger at proxmox.com Thu Nov 12 10:24:13 2020 From: d.jaeger at proxmox.com (Dominic =?iso-8859-1?Q?J=E4ger?=) Date: Thu, 12 Nov 2020 10:24:13 +0100 Subject: [PVE-User] qm cli In-Reply-To: <20201112085452.nz56wdqddt4qirio@shell.tuxis.net> References: <1573062207.1043.1605023362305@webmail.proxmox.com> <20201112085452.nz56wdqddt4qirio@shell.tuxis.net> Message-ID: <20201112092413.GA1174479@mala.proxmox.com> On Thu, Nov 12, 2020 at 09:54:52AM +0100, Mark Schouten wrote: > I for one would really love to use qm on a cluster base, instead of > nodes. I currently first need to find out which node a VM runs on, then > log on to that node, and then run the qm commands. For qm, yes. For pvesh, no. Instead of the initially mentioned command > for i in {123..128}; do qm snapshot $i ready ; done just do something like > for i in 100 101; do pvesh create nodes/dev/qemu/$i/snapshot --snapname ready; done where you replace dev with a node name of your choice. This API path is documented in [0]. qm snapshot is actually one of those commands that is mapped pretty directly to the API [1]. [0] https://pve.proxmox.com/pve-docs/api-viewer/index.html#/nodes/{node}/qemu/{vmid}/snapshot [1] https://git.proxmox.com/?p=qemu-server.git;a=blob;f=PVE/CLI/qm.pm;h=b3b9251c0ad17598beafc8763b082c6e8ab1e95b;hb=HEAD#l931 From hermann at qwer.tk Mon Nov 16 18:21:24 2020 From: hermann at qwer.tk (Hermann Himmelbauer) Date: Mon, 16 Nov 2020 18:21:24 +0100 Subject: [PVE-User] Server freezing randomly with Proxmox 6.2-4 on AMD Ryzen system In-Reply-To: <0e58d1d5-384b-55d2-9042-ae8c1e2ade6c@qwer.tk> References: <0e58d1d5-384b-55d2-9042-ae8c1e2ade6c@qwer.tk> Message-ID: <23e655af-534e-3066-e3cd-514421730aa2@qwer.tk> Hi, In case someone is interested, the problem is now solved, the system seems to be rock solid after ~ 2 month testing: I changed the AMD Ryzen 3 3200G to a AMD Ryzen 5 3600 on one node and to a AMD Ryzen 3 3100 on the two other nodes, now the problem is gone. I don't really know why, I can think of two reasons: 1) The 3200G did not support ECC but I use ECC RAM. Maybe this leads to errors (although intensive memory testing with memtest86 did not report anything). 2) The new CPUs do not have integrated graphic capabilities. I noticed that the two onboard 10GBit-Ethernet adapters now have other PCI addresses with the new CPU. And with the old CPUs there were problem with malfunctioning of these 10G adapters. Many thanks for input + your help. The ASRock Rack X470D4U2-2T is definitly stable now. Best Regards, Hermann Am 04.09.20 um 16:45 schrieb Hermann Himmelbauer: > Dear Proxmox users, > > I'm trying to install a 3-node cluster (latest proxmox/ceph) and > experience random freezes. The node can either be completely frozen (no > blinking cursor on console, no ping) or can get somewhat blocked / slow etc. > > This happens most often on node 2 (approx. 3-4 times / day), node 3 > never got stuck within 14 days runtime, node 1 once. > > Unfortunately I did not find any way to trigger this behaviour, however, > I *think* that this happens most often if I stress the machine in some > way (performance test within a virtual machine) and then idling the machine. > > When the machine freezes completely, there is no logfile. However, if it > is partially frozen, some info can be aquired via dmesg. (See attached > file). ("device=2b:00.0" is an intel 10GBit ethernet adapter (X550T). So > perhaps there is some driver issue regarding this ethernet adapter?) > > The system consists of the following components: > > - AMD Ryzen 3 3200G, 4x 3.60GHz, boxed (YD3200C5FHBOX) > - ASRock Rack X470D4U2-2T (Mainboard) > - Samsung SSD 970 EVO Plus 250GB, M.2 (MZ-V7S250BW) (builtin SSD for OS) > - 2 * Kingston Server Premier DIMM 16GB, DDR4-2666, CL19-19-19, ECC (BOM > Number: 9965745-002.A00G, Part Number: KSM26ED8/16ME) > - be quiet! Pure Power 11 CM 400W ATX 2.4 (BN296) (Power supply) > - 2 * Micron 5300 PRO - Read Intensive 960GB, SATA > (MTFDDAK960TDS-1AW1Z6) (SSD for Ceph) > - LogiLink PC0075, 2x RJ-45, PCIe 2.0 x1 (second NIC with two ports) > > The system is Linux Debian 10.4 (Proxmox 6.2-4) with kernel 5.4.34-1-pve > #1 SMP PVE 5.4.34-2 (Thu, 07 May 2020 10:02:02 +0200) x86_64 GNU/Linux. > > What I did so far (without success): > > - Disabled C6 as I read that this CPU-state can lead to unstable systems > (via "python zenstates.py --c6-disable" -> still errors). > - Updated my Bios to the latest version (3.30) > - Checked that the CPU + RAM are compatible to the mainboard (they are > listed as compatible on the ASRock website) > - Checked logs in IPMI (undervoltage, temperature etc., nothing is logged) > - Memory test (memtest86, no errors) > > Do you have any clue what could be the reason for these freezes? Should > I think of some hardware error? Or is this some known Linux bug that can > be fixed? > > Best Regards, > Hermann > > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From elacunza at binovo.es Tue Nov 17 09:33:37 2020 From: elacunza at binovo.es (Eneko Lacunza) Date: Tue, 17 Nov 2020 09:33:37 +0100 Subject: [PVE-User] Server freezing randomly with Proxmox 6.2-4 on AMD Ryzen system In-Reply-To: <23e655af-534e-3066-e3cd-514421730aa2@qwer.tk> References: <0e58d1d5-384b-55d2-9042-ae8c1e2ade6c@qwer.tk> <23e655af-534e-3066-e3cd-514421730aa2@qwer.tk> Message-ID: Hi Hermann, Glad to read this. My Ryzen 2200G home desktop hangs from time-to-time with desktop (non-gaming) use, so it's quite clear there's some problem with the integrated graphics. It's nice to know the main board is stable, though :) Cheers El 16/11/20 a las 18:21, Hermann Himmelbauer escribi?: > Hi, > In case someone is interested, the problem is now solved, the system > seems to be rock solid after ~ 2 month testing: > > I changed the AMD Ryzen 3 3200G to a AMD Ryzen 5 3600 on one node and > to a AMD Ryzen 3 3100 on the two other nodes, now the problem is gone. > > I don't really know why, I can think of two reasons: > > 1) The 3200G did not support ECC but I use ECC RAM. Maybe this leads > to errors (although intensive memory testing with memtest86 did not > report anything). > 2) The new CPUs do not have integrated graphic capabilities. I noticed > that the two onboard 10GBit-Ethernet adapters now have other PCI > addresses with the new CPU. And with the old CPUs there were problem > with malfunctioning of these 10G adapters. > > Many thanks for input + your help. > > The ASRock Rack X470D4U2-2T is definitly stable now. > > Best Regards, > Hermann > > Am 04.09.20 um 16:45 schrieb Hermann Himmelbauer: >> Dear Proxmox users, >> >> I'm trying to install a 3-node cluster (latest proxmox/ceph) and >> experience random freezes. The node can either be completely frozen (no >> blinking cursor on console, no ping) or can get somewhat blocked / >> slow etc. >> >> This happens most often on node 2 (approx. 3-4 times / day), node 3 >> never got stuck within 14 days runtime, node 1 once. >> >> Unfortunately I did not find any way to trigger this behaviour, however, >> I *think* that this happens most often if I stress the machine in some >> way (performance test within a virtual machine) and then idling the >> machine. >> >> When the machine freezes completely, there is no logfile. However, if it >> is partially frozen, some info can be aquired via dmesg. (See attached >> file). ("device=2b:00.0" is an intel 10GBit ethernet adapter (X550T). So >> perhaps there is some driver issue regarding this ethernet adapter?) >> >> The system consists of the following components: >> >> - AMD Ryzen 3 3200G, 4x 3.60GHz, boxed (YD3200C5FHBOX) >> - ASRock Rack X470D4U2-2T (Mainboard) >> - Samsung SSD 970 EVO Plus 250GB, M.2 (MZ-V7S250BW) (builtin SSD for OS) >> - 2 * Kingston Server Premier DIMM 16GB, DDR4-2666, CL19-19-19, ECC (BOM >> Number: 9965745-002.A00G, Part Number: KSM26ED8/16ME) >> - be quiet! Pure Power 11 CM 400W ATX 2.4 (BN296) (Power supply) >> - 2 * Micron 5300 PRO - Read Intensive 960GB, SATA >> (MTFDDAK960TDS-1AW1Z6) (SSD for Ceph) >> - LogiLink PC0075, 2x RJ-45, PCIe 2.0 x1 (second NIC with two ports) >> >> The system is Linux Debian 10.4 (Proxmox 6.2-4) with kernel 5.4.34-1-pve >> #1 SMP PVE 5.4.34-2 (Thu, 07 May 2020 10:02:02 +0200) x86_64 GNU/Linux. >> >> What I did so far (without success): >> >> - Disabled C6 as I read that this CPU-state can lead to unstable systems >> (via "python zenstates.py --c6-disable" -> still errors). >> - Updated my Bios to the latest version (3.30) >> - Checked that the CPU + RAM are compatible to the mainboard (they are >> listed as compatible on the ASRock website) >> - Checked logs in IPMI (undervoltage, temperature etc., nothing is >> logged) >> - Memory test (memtest86, no errors) >> >> Do you have any clue what could be the reason for these freezes? Should >> I think of some hardware error? Or is this some known Linux bug that can >> be fixed? >> >> Best Regards, >> Hermann >> >> >> _______________________________________________ >> pve-user mailing list >> pve-user at lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Eneko Lacunza | +34 943 569 206 | elacunza at binovo.es Zuzendari teknikoa | https://www.binovo.es Director t?cnico | Astigarragako Bidea, 2 - 2? izda. BINOVO IT HUMAN PROJECT S.L | oficina 10-11, 20180 Oiartzun From thomas.naumann at ovgu.de Mon Nov 23 08:59:06 2020 From: thomas.naumann at ovgu.de (Naumann, Thomas) Date: Mon, 23 Nov 2020 07:59:06 +0000 Subject: [PVE-User] Snapshots not visible via Webgui Message-ID: <03a279d5ec9b258c30ddba01c14772ecb7e53cda.camel@ovgu.de> Hi at all, we run Proxmox-/Cephcluster on latest version (6.2-15 / 14.2.11) from nosubscription-repo. In the morning snapshots of 3 VMs were taken (VMs shutdown/off) but only one snapshot of 1 VM is visible via webgui. All snapshots do exist and are visible via pvesh and/or rbd snap list. There are although entrys about snapshots in VM-config-files. All VMs run on same physical host, log files do not show any special about that topic. All physical hosts of the cluster were rebooted after snapshots were taking - nothing changed. Any hints about that? best regards -- Thomas Naumann Abteilung Netze und Kommunikation Otto-von-Guericke Universit?t Magdeburg Universit?tsrechenzentrum Universit?tsplatz 2 39106 Magdeburg fon: +49 391 67-58563 email: thomas.naumann at ovgu.de From lists at merit.unu.edu Wed Nov 25 09:18:42 2020 From: lists at merit.unu.edu (mj) Date: Wed, 25 Nov 2020 09:18:42 +0100 Subject: [PVE-User] confirmation on osd replacement Message-ID: Hi, I would just like to verify/confirm something here, as we are going to replace our spinning OSDs with SSDs. We have 8 OSDs per server, and two empty front driveslots available. The proxmox boot disk is internal, and currently known as /dev/sdk Suppose I insert two new SSDs in the (two empty) front drive bays, I expect the internal boot disk to shift from /dev/sdk to /dev/sdm The questions: - should we expect boot problems or other side-effects of doing that? (of course I will test on the first server, I'd just like to know what to expect) And then I am going to first add per server two new bluestore SSDs, making a (temporarily) total of 10 OSDs per server. And then I want to replace the 8 remaining filestore spinning OSDs with 6 bluestore SSDs. Making again a total of 8 OSDs per server. The idea is: first add two SSDs to increase IO capacity for the rest of the procedure, while at the same time reducing stress on our filestore journal SSD (wear level=75%) Any comments? MJ From abonilla at suse.com Wed Nov 25 15:03:45 2020 From: abonilla at suse.com (Alejandro Bonilla) Date: Wed, 25 Nov 2020 14:03:45 +0000 Subject: [PVE-User] confirmation on osd replacement In-Reply-To: References: Message-ID: Hi MJ > On Nov 25, 2020, at 3:18 AM, mj wrote: > > Hi, > > I would just like to verify/confirm something here, as we are going to replace our spinning OSDs with SSDs. > > We have 8 OSDs per server, and two empty front driveslots available. > > The proxmox boot disk is internal, and currently known as /dev/sdk > > Suppose I insert two new SSDs in the (two empty) front drive bays, I expect the internal boot disk to shift from /dev/sdk to /dev/sdm > > The questions: > - should we expect boot problems or other side-effects of doing that? > (of course I will test on the first server, I'd just like to know what to expect) Have a look at /etc/fstab for any disk path mounts - since I think Proxmox uses lvm mostly, you shouldn?t see a problem. > > And then I am going to first add per server two new bluestore SSDs, making a (temporarily) total of 10 OSDs per server. > > And then I want to replace the 8 remaining filestore spinning OSDs with 6 bluestore SSDs. Making again a total of 8 OSDs per server. > > The idea is: first add two SSDs to increase IO capacity for the rest of the procedure, while at the same time reducing stress on our filestore journal SSD (wear level=75%) > > Any comments? What is the pool replication configuration or ec-profile? How many nodes in the cluster? Are you planning to remove all disks per server at once or disk by disk? Will all new drives equal or increase the disk capacity of the cluster? > > MJ > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From lists at merit.unu.edu Thu Nov 26 08:54:10 2020 From: lists at merit.unu.edu (mj) Date: Thu, 26 Nov 2020 08:54:10 +0100 Subject: [PVE-User] confirmation on osd replacement In-Reply-To: References: Message-ID: <5f9fa038-d250-65a8-d762-11ad1fd74955@merit.unu.edu> Hi, Yes, perhaps I should have given more details :-) On 11/25/20 3:03 PM, Alejandro Bonilla wrote: > Have a look at /etc/fstab for any disk path mounts - since I think Proxmox uses lvm mostly, you shouldn?t see a problem. I will, thanks! > What is the pool replication configuration or ec-profile? How many nodes in the cluster? We're 3/2 replication, no ec. It's a three-node (small) cluster, 8 filestore OSDs per node, with an SSD journal (wear evel 75%) We will be using samsung PM833 bluestore of the same size (4TB spinners to 3.83GB PM833 SSDs) > Are you planning to remove all disks per server at once or disk by disk? I was planning to: - first add two SSDs to each server, and gradually increase their weight - then, disk-by-disk, replace the 8 (old) spinners with the 6 remaining SSDs > Will all new drives equal or increase the disk capacity of the cluster? Approx equal yes. The aim in not to increase space. MJ From martin at proxmox.com Thu Nov 26 13:17:21 2020 From: martin at proxmox.com (Martin Maurer) Date: Thu, 26 Nov 2020 13:17:21 +0100 Subject: [PVE-User] Proxmox VE 6.3 available Message-ID: <7f76f464-825e-41c7-ca3f-ca86d1be0c76@proxmox.com> Hi all! We are really excited to announce the general availability of our virtualization management platform Proxmox VE 6.3. The most notable new feature is the integration of the stable version 1.0 of our new Proxmox Backup Server. We have strong encryption on the client-side and we have made creating and managing encryption keys for you very simple, providing multiple ways to store keys. VM backups are blazing fast with QEMU dirty-bitmaps. This release also brings stable integration of Ceph Octopus 15.2.6, and you can now select your preferred Ceph version during the installation process. Many new Ceph-specific management features have been added to the GUI like for example displaying the recovery progress in the Ceph status panel or setting the placement groups (PGs) auto-scaling mode for each Ceph pool in the storage cluster. In general, we have added even more functionality and usability enhancements to the GUI. Proxmox VE 6.3 is built on Debian Buster 10.6 but uses the latest long-term support Linux kernel (5.4), QEMU 5.1, LXC 4.0, Ceph 15.2, and ZFS 0.85. Countless bug fixes and smaller improvements are included as well, see the full release notes. Release notes https://pve.proxmox.com/wiki/Roadmap Video tutorial https://www.proxmox.com/en/training/video-tutorials/ Download https://www.proxmox.com/en/downloads Alternate ISO download: http://download.proxmox.com/iso Documentation https://pve.proxmox.com/pve-docs Community Forum https://forum.proxmox.com Source Code https://git.proxmox.com Bugtracker https://bugzilla.proxmox.com FAQ Q: Can I dist-upgrade Proxmox VE 6.x to 6.3 with apt? A: Yes, just via GUI or via CLI with apt update && apt dist-upgrade Q: Can I install Proxmox VE 6.3 on top of Debian Buster? A: Yes, see https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Buster Q: Can I upgrade my Proxmox VE 5.4 cluster with Ceph Luminous to 6.x and higher with Ceph Nautilus and even Ceph Octopus? A: This is a three step process. First, you have to upgrade Proxmox VE from 5.4 to 6.3, and afterwards upgrade Ceph from Luminous to Nautilus. There are a lot of improvements and changes, please follow exactly the upgrade documentation. https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus Finally, do the upgrade to Ceph Octopus - https://pve.proxmox.com/wiki/Ceph_Nautilus_to_Octopus Q: Where can I get more information about future feature updates? A: Check our roadmap, forum, mailing lists, and subscribe to our newsletter. A big THANK YOU to our active community for all your feedback, testing, bug reporting and patch submitting! -- Best Regards, Martin Maurer martin at proxmox.com https://www.proxmox.com From uwe.sauter.de at gmail.com Thu Nov 26 13:18:00 2020 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Thu, 26 Nov 2020 13:18:00 +0100 Subject: [PVE-User] Caution: ceph-mon service does not start after today's updates Message-ID: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> Hi all, this is a warning for all that are eager to apply today's updates. In my case the ceph-mon@ service did not start after a reboot which caused a hanging Ceph once the second monitoring service went offline. Make sure to check ceph-mon@.service is running on freshly started hosts before rebooting the next host with a monitoring service. Regards, Uwe From lindsay.mathieson at gmail.com Thu Nov 26 15:10:14 2020 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Fri, 27 Nov 2020 00:10:14 +1000 Subject: [PVE-User] Caution: ceph-mon service does not start after today's updates In-Reply-To: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> Message-ID: <15f5893d-83cf-ee8c-11ff-0a67e376dc41@gmail.com> On 26/11/2020 10:18 pm, Uwe Sauter wrote: > this is a warning for all that are eager to apply today's updates. > In my case the ceph-mon@ service did not start after a reboot which caused a hanging Ceph once the second monitoring service > went offline. I ran into that, also the node failed to rejoin the cluster quorum. syslog had errors relating to the pem-ssl key. Manually start the pve cluster service and a 2nd reboot solved both issues. -- Lindsay From uwe.sauter.de at gmail.com Thu Nov 26 15:15:00 2020 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Thu, 26 Nov 2020 15:15:00 +0100 Subject: [PVE-User] Caution: ceph-mon service does not start after today's updates In-Reply-To: <15f5893d-83cf-ee8c-11ff-0a67e376dc41@gmail.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <15f5893d-83cf-ee8c-11ff-0a67e376dc41@gmail.com> Message-ID: <9c0ceb9e-ff96-b12f-a4ed-1e1ac250fbb6@gmail.com> Am 26.11.20 um 15:10 schrieb Lindsay Mathieson: > On 26/11/2020 10:18 pm, Uwe Sauter wrote: >> this is a warning for all that are eager to apply today's updates. >> In my case the ceph-mon@ service did not start after a reboot which caused a hanging Ceph once the second monitoring service >> went offline. > > I ran into that, also the node failed to rejoin the cluster quorum. syslog had errors relating to the pem-ssl key. > > Manually start the pve cluster service and a 2nd reboot solved both issues. > Yes, rebooting might help, but not reliably. I had nodes that needed several reboots until pvestatd did not fail. I also had failed ceph-mgr@ services (with Nautilus). My current suspicion is that my network takes too long to become available. Regards, Uwe From t.lamprecht at proxmox.com Thu Nov 26 15:46:00 2020 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Thu, 26 Nov 2020 15:46:00 +0100 Subject: [PVE-User] Caution: ceph-mon service does not start after today's updates In-Reply-To: <9c0ceb9e-ff96-b12f-a4ed-1e1ac250fbb6@gmail.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <15f5893d-83cf-ee8c-11ff-0a67e376dc41@gmail.com> <9c0ceb9e-ff96-b12f-a4ed-1e1ac250fbb6@gmail.com> Message-ID: <11d60eb0-9b17-53ba-cc21-79728e1296b0@proxmox.com> On 26.11.20 15:15, Uwe Sauter wrote: > Am 26.11.20 um 15:10 schrieb Lindsay Mathieson: >> On 26/11/2020 10:18 pm, Uwe Sauter wrote: >>> this is a warning for all that are eager to apply today's updates. >>> In my case the ceph-mon@ service did not start after a reboot which caused a hanging Ceph once the second monitoring service >>> went offline. >> >> I ran into that, also the node failed to rejoin the cluster quorum. syslog had errors relating to the pem-ssl key. >> >> Manually start the pve cluster service and a 2nd reboot solved both issues. >> > > Yes, rebooting might help, but not reliably. I had nodes that needed several reboots until pvestatd did not fail. > > I also had failed ceph-mgr@ services (with Nautilus). > > My current suspicion is that my network takes too long to become available. > Note, it's always good idea to check if all services are running OK again before continuing with upgrading the next host, not just on this update :-) Also, ceph monitors can be nicely restarted over the web interface, there's a visible status about which services run outdated versions/need a restart. Anyway, do you have any logs which could give more details for possible issues? regards, Thomas From lindsay.mathieson at gmail.com Thu Nov 26 16:03:10 2020 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Fri, 27 Nov 2020 01:03:10 +1000 Subject: [PVE-User] Caution: ceph-mon service does not start after today's updates In-Reply-To: <11d60eb0-9b17-53ba-cc21-79728e1296b0@proxmox.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <15f5893d-83cf-ee8c-11ff-0a67e376dc41@gmail.com> <9c0ceb9e-ff96-b12f-a4ed-1e1ac250fbb6@gmail.com> <11d60eb0-9b17-53ba-cc21-79728e1296b0@proxmox.com> Message-ID: <269aba60-412c-578b-9757-6a0567d270e5@gmail.com> On 27/11/2020 12:46 am, Thomas Lamprecht wrote: > Note, it's always good idea to check if all services are running OK again before > continuing with upgrading the next host, not just on this update:-) > > Also, ceph monitors can be nicely restarted over the web interface, there's a > visible status about which services run outdated versions/need a restart. > > > Anyway, do you have any logs which could give more details for possible issues? I have a node that is just failing to rejoin the cluster and the ceph mon & mgr fail to start. Seeing this repeated in syslog Nov 27 00:58:23 vnh pveproxy[2903]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. Nov 27 00:58:23 vnh pveproxy[2904]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. Nov 27 00:58:23 vnh pveproxy[2905]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.378 7fb182935700 -1 mon.vnh at 0(probing) e9 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.390 7fb17d92b700 -1 mon.vnh at 0(probing) e9 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.526 7fb183136700 -1 mon.vnh at 0(probing) e9 handle_auth_bad_method hmm, they didn't like 2 result (95) Operation not supported Nov 27 00:58:27 vnh ceph-mon[2073]: 2020-11-27 00:58:27.702 7fb182935700 -1 mon.vnh at 0(probing) e9 handle_auth_request no AuthAuthorizeHandler found for auth method 1 The following gets the node back on the cluster: systemctl start pve-cluster.service systemctl restart pvestatd.service But I can't get the mon, mgr or osd services to start. -- Lindsay From abonilla at suse.com Thu Nov 26 16:39:05 2020 From: abonilla at suse.com (Alejandro Bonilla) Date: Thu, 26 Nov 2020 15:39:05 +0000 Subject: [PVE-User] confirmation on osd replacement In-Reply-To: <5f9fa038-d250-65a8-d762-11ad1fd74955@merit.unu.edu> References: <5f9fa038-d250-65a8-d762-11ad1fd74955@merit.unu.edu> Message-ID: <7B860047-76E2-44B5-8F66-D04FFA216C07@suse.com> > On Nov 26, 2020, at 2:54 AM, mj wrote: > > Hi, > > Yes, perhaps I should have given more details :-) > > On 11/25/20 3:03 PM, Alejandro Bonilla wrote: > >> Have a look at /etc/fstab for any disk path mounts - since I think Proxmox uses lvm mostly, you shouldn?t see a problem. > I will, thanks! > >> What is the pool replication configuration or ec-profile? How many nodes in the cluster? > We're 3/2 replication, no ec. It's a three-node (small) cluster, 8 filestore OSDs per node, with an SSD journal (wear evel 75%) If it?s 3 replicas, min 2, then you should be able to clear all drives from a system at once and replace them all to minimize the amount of times the cluster will end up rebalancing. > > We will be using samsung PM833 bluestore of the same size > (4TB spinners to 3.83GB PM833 SSDs) > >> Are you planning to remove all disks per server at once or disk by disk? > I was planning to: > - first add two SSDs to each server, and gradually increase their weight Two per server to ensure the disk replacement will work as expected is a good idea - I don?t think you?ll gain anything with a gradual re-weight. > - then, disk-by-disk, replace the 8 (old) spinners with the 6 remaining SSDs IF you have two other replicas, then a full system disk replacement should be no trouble - especially after two other SSDs were added and most data was shuffled around. > >> Will all new drives equal or increase the disk capacity of the cluster? > Approx equal yes. > The aim in not to increase space. There are other reasons why I ask, specifically based on PG count and balancing of the cluster. > > MJ > From t.lamprecht at proxmox.com Thu Nov 26 17:14:40 2020 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Thu, 26 Nov 2020 17:14:40 +0100 Subject: [PVE-User] Caution: ceph-mon service does not start after today's updates In-Reply-To: <269aba60-412c-578b-9757-6a0567d270e5@gmail.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <15f5893d-83cf-ee8c-11ff-0a67e376dc41@gmail.com> <9c0ceb9e-ff96-b12f-a4ed-1e1ac250fbb6@gmail.com> <11d60eb0-9b17-53ba-cc21-79728e1296b0@proxmox.com> <269aba60-412c-578b-9757-6a0567d270e5@gmail.com> Message-ID: <72fbd5d5-1c9b-5d59-5256-4cf1dd1f79cb@proxmox.com> On 26.11.20 16:03, Lindsay Mathieson wrote: > On 27/11/2020 12:46 am, Thomas Lamprecht wrote: >> Note, it's always good idea to check if all services are running OK again before >> continuing with upgrading the next host, not just on this update:-) >> >> Also, ceph monitors can be nicely restarted over the web interface, there's a >> visible status about which services run outdated versions/need a restart. >> >> >> Anyway, do you have any logs which could give more details for possible issues? > > I have a node that is just failing to rejoin the cluster and the ceph mon & mgr fail to start. > > > Seeing this repeated in syslog > > ?? Nov 27 00:58:23 vnh pveproxy[2903]: /etc/pve/local/pve-ssl.key: > ?? failed to load local private key (key_file or key) at > ?? /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. > ?? Nov 27 00:58:23 vnh pveproxy[2904]: /etc/pve/local/pve-ssl.key: > ?? failed to load local private key (key_file or key) at > ?? /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. > ?? Nov 27 00:58:23 vnh pveproxy[2905]: /etc/pve/local/pve-ssl.key: > ?? failed to load local private key (key_file or key) at > ?? /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1737. > ?? Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.378 > ?? 7fb182935700 -1 mon.vnh at 0(probing) e9 handle_auth_bad_method hmm, > ?? they didn't like 2 result (95) Operation not supported > ?? Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.390 > ?? 7fb17d92b700 -1 mon.vnh at 0(probing) e9 handle_auth_bad_method hmm, > ?? they didn't like 2 result (95) Operation not supported > ?? Nov 27 00:58:26 vnh ceph-mon[2073]: 2020-11-27 00:58:26.526 > ?? 7fb183136700 -1 mon.vnh at 0(probing) e9 handle_auth_bad_method hmm, > ?? they didn't like 2 result (95) Operation not supported > ?? Nov 27 00:58:27 vnh ceph-mon[2073]: 2020-11-27 00:58:27.702 > ?? 7fb182935700 -1 mon.vnh at 0(probing) e9 handle_auth_request no > ?? AuthAuthorizeHandler found for auth method 1 > the errors seems like being the result of pve-cluster not coming up, which seems the actual problem. > > The following gets the node back on the cluster: > > systemctl start pve-cluster.service Anything of pve-cluster service in the log? What does: # systemd-analyze verify default.target outputs? cheers, Thomas From leandro at tecnetmza.com.ar Thu Nov 26 17:18:38 2020 From: leandro at tecnetmza.com.ar (Leandro Roggerone) Date: Thu, 26 Nov 2020 13:18:38 -0300 Subject: [PVE-User] moving a vmware machine into proxmox Message-ID: Hi guys , I successfully moved a vm from my ESXi server to my proxmox. This is what I did: I created a backup with guettoVCB tool. SCP the vmdk disk images into my proxmox local partition. Created a VM convert those images into qcow2 doing: qm importdisk 117 ESXi_1-000003-flat.vmdk local-lvm -format qcow2 Then I added this new disk image into my vm 117. After turning this machine on wich contains a windows server , following happened. Machine is ok, I can login and so. but There is sql server wich is not running. After trying to open sql server configuration manager, im getting the "sql server can not connect to wmi provider" error message. I know Im providing not enough information but this is perhaps some well know issue. I already confirmed that after rebooting original machine on ESXi server , there is no manual action required to have all process running. So , this can be perhaps something related with licence. Any feedback / feelling about this ? I need to move this machine without outage nor customer action. Regards. Leandro. Libre de virus. www.avast.com <#m_9061933542358666280_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> From alex at calicolabs.com Thu Nov 26 17:26:19 2020 From: alex at calicolabs.com (Alex Chekholko) Date: Thu, 26 Nov 2020 08:26:19 -0800 Subject: [PVE-User] moving a vmware machine into proxmox In-Reply-To: References: Message-ID: >From the Windows point of view, the hardware changed, check in device manager that all devices are there? In the service manager, make sure the relevant services started up? Also, just to double-check, did the original VM have only one disk? If it had more than one, you just need to copy the vmdk and import it the same way and attach it to the new VM. On Thu, Nov 26, 2020 at 8:19 AM Leandro Roggerone wrote: > Hi guys , I successfully moved a vm from my ESXi server to my proxmox. > This is what I did: > I created a backup with guettoVCB tool. > SCP the vmdk disk images into my proxmox local partition. > Created a VM > convert those images into qcow2 doing: > qm importdisk 117 ESXi_1-000003-flat.vmdk local-lvm -format qcow2 > Then I added this new disk image into my vm 117. > > After turning this machine on wich contains a windows server , following > happened. > Machine is ok, I can login and so. > but > There is sql server wich is not running. > After trying to open sql server configuration manager, im getting the "sql > server can not connect to wmi provider" error message. > > I know Im providing not enough information but this is perhaps some well > know issue. > I already confirmed that after rebooting original machine on ESXi server , > there is no manual action required to have all process running. > So , this can be perhaps something related with licence. > Any feedback / feelling about this ? > I need to move this machine without outage nor customer action. > Regards. > Leandro. > > > > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > Libre > de virus. www.avast.com > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > <#m_9061933542358666280_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From alex at calicolabs.com Thu Nov 26 17:26:19 2020 From: alex at calicolabs.com (Alex Chekholko) Date: Thu, 26 Nov 2020 08:26:19 -0800 Subject: [PVE-User] moving a vmware machine into proxmox In-Reply-To: References: Message-ID: >From the Windows point of view, the hardware changed, check in device manager that all devices are there? In the service manager, make sure the relevant services started up? Also, just to double-check, did the original VM have only one disk? If it had more than one, you just need to copy the vmdk and import it the same way and attach it to the new VM. On Thu, Nov 26, 2020 at 8:19 AM Leandro Roggerone wrote: > Hi guys , I successfully moved a vm from my ESXi server to my proxmox. > This is what I did: > I created a backup with guettoVCB tool. > SCP the vmdk disk images into my proxmox local partition. > Created a VM > convert those images into qcow2 doing: > qm importdisk 117 ESXi_1-000003-flat.vmdk local-lvm -format qcow2 > Then I added this new disk image into my vm 117. > > After turning this machine on wich contains a windows server , following > happened. > Machine is ok, I can login and so. > but > There is sql server wich is not running. > After trying to open sql server configuration manager, im getting the "sql > server can not connect to wmi provider" error message. > > I know Im providing not enough information but this is perhaps some well > know issue. > I already confirmed that after rebooting original machine on ESXi server , > there is no manual action required to have all process running. > So , this can be perhaps something related with licence. > Any feedback / feelling about this ? > I need to move this machine without outage nor customer action. > Regards. > Leandro. > > > > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > Libre > de virus. www.avast.com > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > <#m_9061933542358666280_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From t.lamprecht at proxmox.com Thu Nov 26 17:35:42 2020 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Thu, 26 Nov 2020 17:35:42 +0100 Subject: [PVE-User] Caution: ceph-mon service does not start after today's updates In-Reply-To: <72fbd5d5-1c9b-5d59-5256-4cf1dd1f79cb@proxmox.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <15f5893d-83cf-ee8c-11ff-0a67e376dc41@gmail.com> <9c0ceb9e-ff96-b12f-a4ed-1e1ac250fbb6@gmail.com> <11d60eb0-9b17-53ba-cc21-79728e1296b0@proxmox.com> <269aba60-412c-578b-9757-6a0567d270e5@gmail.com> <72fbd5d5-1c9b-5d59-5256-4cf1dd1f79cb@proxmox.com> Message-ID: <73072f32-7b87-d7a4-4ec0-96a1b1967cbb@proxmox.com> On 26.11.20 17:14, Thomas Lamprecht wrote: > What does: > # systemd-analyze verify default.target > > outputs? May have found an issue with systemd service ordering, seems there's a cycle which gets broken up arbitrary, working sometimes, but sometimes not. From abonilla at suse.com Thu Nov 26 17:40:04 2020 From: abonilla at suse.com (Alejandro Bonilla) Date: Thu, 26 Nov 2020 16:40:04 +0000 Subject: [PVE-User] Proxmox VE 6.3 available In-Reply-To: <7f76f464-825e-41c7-ca3f-ca86d1be0c76@proxmox.com> References: <7f76f464-825e-41c7-ca3f-ca86d1be0c76@proxmox.com> Message-ID: <02C5B172-E026-448D-BEC8-B46E42ACBFCD@suse.com> > On Nov 26, 2020, at 7:17 AM, Martin Maurer wrote: > > Hi all! > > We are really excited to announce the general availability of our virtualization management platform Proxmox VE 6.3. The most notable new feature is the integration of the stable version 1.0 of our new Proxmox Backup Server. We have strong encryption on the client-side and we have made creating and managing encryption keys for you very simple, providing multiple ways to store keys. VM backups are blazing fast with QEMU dirty-bitmaps. > > > This release also brings stable integration of Ceph Octopus 15.2.6, and you can now select your preferred Ceph version during the installation process. Many new Ceph- Is there any documentation on upgrading from N to O? Do I just modify /etc/apt/sources.list.d/ceph.list deb http://download.proxmox.com/debian/ceph-luminous buster main To /etc/apt/sources.list.d/ceph.list deb http://download.proxmox.com/debian/ceph-octopus buster main I?m guessing the version selector is for a first-time configuration wizard. Besides waiting on the above, my upgrade to 6.3-2 was uneventful. Thanks for a great release. > specific management features have been added to the GUI like for example displaying the recovery progress in the Ceph status panel or setting the placement groups (PGs) auto-scaling mode for each Ceph pool in the storage cluster. In general, we have added even more functionality and usability enhancements to the GUI. > > > Proxmox VE 6.3 is built on Debian Buster 10.6 but uses the latest long-term support Linux kernel (5.4), QEMU 5.1, LXC 4.0, Ceph 15.2, and ZFS 0.85. > > > Countless bug fixes and smaller improvements are included as well, see the full release notes. > > > Release notes > https://pve.proxmox.com/wiki/Roadmap > > > Video tutorial > https://www.proxmox.com/en/training/video-tutorials/ > > > Download > https://www.proxmox.com/en/downloads > > Alternate ISO download: > http://download.proxmox.com/iso > > Documentation > https://pve.proxmox.com/pve-docs > > Community Forum > https://forum.proxmox.com > > Source Code > https://git.proxmox.com > > Bugtracker > https://bugzilla.proxmox.com > > > FAQ > Q: Can I dist-upgrade Proxmox VE 6.x to 6.3 with apt? > A: Yes, just via GUI or via CLI with apt update && apt dist-upgrade > > Q: Can I install Proxmox VE 6.3 on top of Debian Buster? > A: Yes, see https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Buster > > Q: Can I upgrade my Proxmox VE 5.4 cluster with Ceph Luminous to 6.x and higher with Ceph Nautilus and even Ceph Octopus? > A: This is a three step process. First, you have to upgrade Proxmox VE from 5.4 to 6.3, and afterwards upgrade Ceph from Luminous to Nautilus. There are a lot of improvements and changes, please follow exactly the upgrade documentation. > https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 > https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus > > Finally, do the upgrade to Ceph Octopus - https://pve.proxmox.com/wiki/Ceph_Nautilus_to_Octopus > > Q: Where can I get more information about future feature updates? > A: Check our roadmap, forum, mailing lists, and subscribe to our newsletter. > > A big THANK YOU to our active community for all your feedback, testing, bug reporting and patch submitting! > > -- > Best Regards, > > Martin Maurer > > martin at proxmox.com > https://www.proxmox.com > > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From rumors at web.de Thu Nov 26 18:07:07 2020 From: rumors at web.de (Peter Simon) Date: Thu, 26 Nov 2020 18:07:07 +0100 Subject: [PVE-User] Proxmox VE 6.3 available In-Reply-To: References: <7f76f464-825e-41c7-ca3f-ca86d1be0c76@proxmox.com> Message-ID: <9fd80d46-3e12-0513-5e76-1a65980bfb28@web.de> Am 26.11.20 um 17:40 schrieb Alejandro Bonilla via pve-user: > Is there any documentation on upgrading from N to O? Do I just modify https://pve.proxmox.com/wiki/Ceph_Nautilus_to_Octopus > > /etc/apt/sources.list.d/ceph.list > debhttp://download.proxmox.com/debian/ceph-luminous buster main > > To > > /etc/apt/sources.list.d/ceph.list > debhttp://download.proxmox.com/debian/ceph-octopus buster main * Note*: It is*not*possible to upgrade from Ceph Luminous to Octopus directly. > > I?m guessing the version selector is for a first-time configuration wizard. > > Besides waiting on the above, my upgrade to 6.3-2 was uneventful. > > Thanks for a great release. so you missed the step Nautilus ... Cheers Peter From abonilla at suse.com Thu Nov 26 18:35:05 2020 From: abonilla at suse.com (Alejandro Bonilla) Date: Thu, 26 Nov 2020 17:35:05 +0000 Subject: [PVE-User] Proxmox VE 6.3 available In-Reply-To: <9fd80d46-3e12-0513-5e76-1a65980bfb28@web.de> References: <7f76f464-825e-41c7-ca3f-ca86d1be0c76@proxmox.com> <9fd80d46-3e12-0513-5e76-1a65980bfb28@web.de> Message-ID: <1866BF05-C785-489F-8428-CC8C9F24ECDE@suse.com> > On Nov 26, 2020, at 12:07 PM, Peter Simon wrote: > > > Am 26.11.20 um 17:40 schrieb Alejandro Bonilla via pve-user: >> Is there any documentation on upgrading from N to O? Do I just modify > https://pve.proxmox.com/wiki/Ceph_Nautilus_to_Octopus >> >> /etc/apt/sources.list.d/ceph.list >> debhttp://download.proxmox.com/debian/ceph-luminous buster main >> >> To >> >> /etc/apt/sources.list.d/ceph.list >> debhttp://download.proxmox.com/debian/ceph-octopus buster main > * > Note*: It is*not*possible to upgrade from Ceph Luminous to Octopus > directly. Thanks - I don?t know why that ceph repo had L (and I didn?t see it) but my systems have N installed - (perhaps a certain version is included in the proxmox media?) I did update to a newer version of Nautilus and will then do the same for O. Thanks for the hint! > >> >> I?m guessing the version selector is for a first-time configuration wizard. >> >> Besides waiting on the above, my upgrade to 6.3-2 was uneventful. >> >> Thanks for a great release. > > so you missed the step Nautilus ... > > Cheers > > Peter > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From t.lamprecht at proxmox.com Thu Nov 26 19:56:44 2020 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Thu, 26 Nov 2020 19:56:44 +0100 Subject: [PVE-User] Caution: ceph-mon service does not start after today's updates In-Reply-To: <269aba60-412c-578b-9757-6a0567d270e5@gmail.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <15f5893d-83cf-ee8c-11ff-0a67e376dc41@gmail.com> <9c0ceb9e-ff96-b12f-a4ed-1e1ac250fbb6@gmail.com> <11d60eb0-9b17-53ba-cc21-79728e1296b0@proxmox.com> <269aba60-412c-578b-9757-6a0567d270e5@gmail.com> Message-ID: <00c86fc4-6d3a-8a64-db84-27d5b5558617@proxmox.com> Some news. There are a few things at play, it boils down to two things: * a update of various service orderings in ceph with 14.2.12 (released a bit ago), they introduced pretty much everywhere a `Before=remote-fs-pre.target` order enforcement. * rrdcached, a service used by pve-cluster.service (pmxcfs), this has no native systemd service file, so systemd auto generates one, with an `Before=remote-pre.target` order enforcement which then has ordering for the aforementioned `Before=remote-fs-pre.target` Thus you get the cycle (-> means an after odering, all befores where transformed to after by reversing them (systemd does that too)): .> pve-cluster -> rrdcached -> remote-pre -> remote-fs-pre -> ceph-mgr@ -. | | `------------------------------------------------------------------------' We're building a new ceph version with the Before=remote-fs-pre.target removed, it is bogus for the ceph mgr, mds, mon, .. services as is. As you probably guessed, one can also fix this by adapting rrdcached, and as a work around you can do so: 1. copy over the generated ephemeral service file from /run to /etc, which has higher priority. # cp /run/systemd/generator.late/rrdcached.service /etc/systemd/system/ 2. Drop the after ordering for remote-fs.target # sed -i '/^After=remote-fs.target/d' /etc/systemd/system/rrdcached.service 3. reboot A ceph 14.2.15-pve2 package will soon be available, we'll also see if we can improve the rrdcached situation in the future, it has no fault on its own naturally, the systemd auto generators heuristic is to blame, but maybe we can see if upstream or Debian has interest in adding an hand crafted systemd unit file, avoiding auto-generation. Otionally we could maintain it for PVE, or do like in Proxmox Backup Server - use our own rust based RRD implementation regards, Thomas From t.lamprecht at proxmox.com Thu Nov 26 20:35:06 2020 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Thu, 26 Nov 2020 20:35:06 +0100 Subject: [PVE-User] [update available] Re: Caution: ceph-mon service does not start after today's updates In-Reply-To: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> Message-ID: <32d89f84-29a7-90c4-deed-1a3638c99098@proxmox.com> Hi all, Hi Uwe and Lindsay, ceph 14.2.15-pve2 is now available, it includes fixes for the ordering restraints of the various ceph service unit files[0], besides that there where zero actual code changes. Due to that and Stoiko's tests (much thanks for all the help!) confirming the positive result of mine, we uploaded it for general availability. Thanks also to you two for reporting! regards, Thomas [0]: https://git.proxmox.com/?p=ceph.git;a=blob;f=patches/0009-fix-service-ordering-avoid-Before-remote-fs-pre.targ.patch;h=8fe5a35c385f7d4007b4965d297fc0daa9091be3;hb=99b3812832d5d5a8ac48a2e3b084c0ecf1fd087c From uwe.sauter.de at gmail.com Thu Nov 26 20:54:59 2020 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Thu, 26 Nov 2020 20:54:59 +0100 Subject: [PVE-User] [update available] Re: Caution: ceph-mon service does not start after today's updates In-Reply-To: <32d89f84-29a7-90c4-deed-1a3638c99098@proxmox.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <32d89f84-29a7-90c4-deed-1a3638c99098@proxmox.com> Message-ID: <961e6e45-6345-793d-74e2-e16db96a4a70@gmail.com> Thomas, thank you and your team for that prompt response. I'll try out tomorrow first thing in the morning. Regards, Uwe Am 26.11.20 um 20:35 schrieb Thomas Lamprecht: > Hi all, > Hi Uwe and Lindsay, > > ceph 14.2.15-pve2 is now available, it includes fixes for the ordering > restraints of the various ceph service unit files[0], besides that there > where zero actual code changes. > > Due to that and Stoiko's tests (much thanks for all the help!) confirming > the positive result of mine, we uploaded it for general availability. > > Thanks also to you two for reporting! > > regards, > Thomas > > [0]: https://git.proxmox.com/?p=ceph.git;a=blob;f=patches/0009-fix-service-ordering-avoid-Before-remote-fs-pre.targ.patch;h=8fe5a35c385f7d4007b4965d297fc0daa9091be3;hb=99b3812832d5d5a8ac48a2e3b084c0ecf1fd087c > > From lists at merit.unu.edu Thu Nov 26 21:31:40 2020 From: lists at merit.unu.edu (mj) Date: Thu, 26 Nov 2020 21:31:40 +0100 Subject: [PVE-User] confirmation on osd replacement In-Reply-To: <7B860047-76E2-44B5-8F66-D04FFA216C07@suse.com> References: <5f9fa038-d250-65a8-d762-11ad1fd74955@merit.unu.edu> <7B860047-76E2-44B5-8F66-D04FFA216C07@suse.com> Message-ID: Hi Alejandro, Thanks for your feedback, much appreciated! Enjoy your weekend! MJ On 11/26/20 4:39 PM, Alejandro Bonilla wrote: > > >> On Nov 26, 2020, at 2:54 AM, mj wrote: >> >> Hi, >> >> Yes, perhaps I should have given more details :-) >> >> On 11/25/20 3:03 PM, Alejandro Bonilla wrote: >> >>> Have a look at /etc/fstab for any disk path mounts - since I think Proxmox uses lvm mostly, you shouldn?t see a problem. >> I will, thanks! >> >>> What is the pool replication configuration or ec-profile? How many nodes in the cluster? >> We're 3/2 replication, no ec. It's a three-node (small) cluster, 8 filestore OSDs per node, with an SSD journal (wear evel 75%) > > If it?s 3 replicas, min 2, then you should be able to clear all drives from a system at once and replace them all to minimize the amount of times the cluster will end up rebalancing. > >> >> We will be using samsung PM833 bluestore of the same size >> (4TB spinners to 3.83GB PM833 SSDs) >> >>> Are you planning to remove all disks per server at once or disk by disk? >> I was planning to: >> - first add two SSDs to each server, and gradually increase their weight > > Two per server to ensure the disk replacement will work as expected is a good idea - I don?t think you?ll gain anything with a gradual re-weight. > >> - then, disk-by-disk, replace the 8 (old) spinners with the 6 remaining SSDs > > IF you have two other replicas, then a full system disk replacement should be no trouble - especially after two other SSDs were added and most data was shuffled around. > >> >>> Will all new drives equal or increase the disk capacity of the cluster? >> Approx equal yes. >> The aim in not to increase space. > > There are other reasons why I ask, specifically based on PG count and balancing of the cluster. > >> >> MJ >> > From lindsay.mathieson at gmail.com Fri Nov 27 01:53:28 2020 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Fri, 27 Nov 2020 10:53:28 +1000 Subject: [PVE-User] [update available] Re: Caution: ceph-mon service does not start after today's updates In-Reply-To: <32d89f84-29a7-90c4-deed-1a3638c99098@proxmox.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <32d89f84-29a7-90c4-deed-1a3638c99098@proxmox.com> Message-ID: <87cdd210-4605-9427-c863-aed6e6e5ca38@gmail.com> On 27/11/2020 5:35 am, Thomas Lamprecht wrote: > Hi Uwe and Lindsay, > > ceph 14.2.15-pve2 is now available, it includes fixes for the ordering > restraints of the various ceph service unit files[0], besides that there > where zero actual code changes. > > Due to that and Stoiko's tests (much thanks for all the help!) confirming > the positive result of mine, we uploaded it for general availability. > > Thanks also to you two for reporting! Thanks for the really quick turnaround Thomas, and sorry for not getting back to you before, but it was 3am my time and once I got the servers back up, I really needed to crash :) Will test tonight, cheers. -- Lindsay From uwe.sauter.de at gmail.com Fri Nov 27 09:19:45 2020 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Fri, 27 Nov 2020 09:19:45 +0100 Subject: [PVE-User] [update available] Re: Caution: ceph-mon service does not start after today's updates In-Reply-To: <32d89f84-29a7-90c4-deed-1a3638c99098@proxmox.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <32d89f84-29a7-90c4-deed-1a3638c99098@proxmox.com> Message-ID: Good morning Thomas, the updated packages do the trick. I updated my ten hosts and not one needed a second reboot. Next will be the update from Nautilus to Octupus? but the instructions in the wiki are well written so I don't expect any issues there. Thanks again, Uwe Am 26.11.20 um 20:35 schrieb Thomas Lamprecht: > Hi all, > Hi Uwe and Lindsay, > > ceph 14.2.15-pve2 is now available, it includes fixes for the ordering > restraints of the various ceph service unit files[0], besides that there > where zero actual code changes. > > Due to that and Stoiko's tests (much thanks for all the help!) confirming > the positive result of mine, we uploaded it for general availability. > > Thanks also to you two for reporting! > > regards, > Thomas > > [0]: https://git.proxmox.com/?p=ceph.git;a=blob;f=patches/0009-fix-service-ordering-avoid-Before-remote-fs-pre.targ.patch;h=8fe5a35c385f7d4007b4965d297fc0daa9091be3;hb=99b3812832d5d5a8ac48a2e3b084c0ecf1fd087c > > From jean-luc.oms at lirmm.fr Fri Nov 27 12:47:37 2020 From: jean-luc.oms at lirmm.fr (Jean-Luc Oms) Date: Fri, 27 Nov 2020 12:47:37 +0100 Subject: [PVE-User] Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 Message-ID: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> Bonjour, Upgrading to last Proxmox VE / Ceph nautilus from the last 6.2 proxmox VE seems to introduce a python 2/3 version problem, dashboard healt stops working. root at ceph1:/usr/bin# ceph health HEALTH_ERR Module 'dashboard' has failed: ('invalid syntax', ('/usr/share/ceph/mgr/dashboard/controllers/orchestrator.py', 34, 11, '??? result: dict = {}\n')) This syntax was introduced in 3.6, and using strace it seems python 2.7 is used. Any option to resolve this ? everything was ok in 6.2-15. Thanks -- Jean-Luc Oms /STI-ReseauX - LIRMM - CNRS/UM/ +33 4 67 41 85 93 / +33 6 32 01 04 17 From lindsay.mathieson at gmail.com Fri Nov 27 14:07:46 2020 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Fri, 27 Nov 2020 23:07:46 +1000 Subject: [PVE-User] Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 In-Reply-To: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> References: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> Message-ID: <3e7c42b6-1a1f-d296-e240-fb98cf87a6ca@gmail.com> On 27/11/2020 9:47 pm, Jean-Luc Oms wrote: > Upgrading to last Proxmox VE / Ceph nautilus from the last 6.2 proxmox > VE seems to introduce a python 2/3 version problem, dashboard healt > stops working. Was just about to report that :) Same here. -- Lindsay From lindsay.mathieson at gmail.com Fri Nov 27 14:18:13 2020 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Fri, 27 Nov 2020 23:18:13 +1000 Subject: [PVE-User] [update available] Re: Caution: ceph-mon service does not start after today's updates In-Reply-To: <32d89f84-29a7-90c4-deed-1a3638c99098@proxmox.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <32d89f84-29a7-90c4-deed-1a3638c99098@proxmox.com> Message-ID: <791fa674-2701-e451-dcbe-b358f5b5298b@gmail.com> On 27/11/2020 5:35 am, Thomas Lamprecht wrote: > Hi Uwe and Lindsay, > > ceph 14.2.15-pve2 is now available, it includes fixes for the ordering > restraints of the various ceph service unit files[0], besides that there > where zero actual code changes. Thanks! Just updated and did a rolling reboot on our cluster. All servers/services came up perfectly, no issues. Gonna test that Octopus upgrade next :) CXheers, -- Lindsay From jean-luc.oms at lirmm.fr Fri Nov 27 14:29:42 2020 From: jean-luc.oms at lirmm.fr (Jean-Luc Oms) Date: Fri, 27 Nov 2020 14:29:42 +0100 Subject: [PVE-User] [update available] Re: Caution: ceph-mon service does not start after today's updates In-Reply-To: <791fa674-2701-e451-dcbe-b358f5b5298b@gmail.com> References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <32d89f84-29a7-90c4-deed-1a3638c99098@proxmox.com> <791fa674-2701-e451-dcbe-b358f5b5298b@gmail.com> Message-ID: Le 27/11/2020 ? 14:18, Lindsay Mathieson a ?crit?: > On 27/11/2020 5:35 am, Thomas Lamprecht wrote: >> Hi Uwe and Lindsay, >> >> ceph 14.2.15-pve2 is now available, it includes fixes for the ordering >> restraints of the various ceph service unit files[0], besides that there >> where zero actual code changes. > > > Thanks! Just updated and did a rolling reboot on our cluster. All > servers/services came up perfectly, no issues. You have cleared the ceph health error? ? > > > Gonna test that Octopus upgrade next :) > > > CXheers, > -- Jean-Luc Oms /STI-ReseauX - LIRMM - CNRS/UM/ +33 4 67 41 85 93 / +33 6 32 01 04 17 From lindsay.mathieson at gmail.com Fri Nov 27 14:31:19 2020 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Fri, 27 Nov 2020 23:31:19 +1000 Subject: [PVE-User] [update available] Re: Caution: ceph-mon service does not start after today's updates In-Reply-To: References: <4545c7f5-9b50-2f0a-952a-cc532ad01b13@gmail.com> <32d89f84-29a7-90c4-deed-1a3638c99098@proxmox.com> <791fa674-2701-e451-dcbe-b358f5b5298b@gmail.com> Message-ID: <7afdad4b-6d03-2cbe-46fd-aaf957bed591@gmail.com> On 27/11/2020 11:29 pm, Jean-Luc Oms wrote: > You have cleared the ceph health error? ? That dashboard one? no. This was in reference to some cluster & ceph service issues. -- Lindsay From jean-luc.oms at lirmm.fr Fri Nov 27 16:15:33 2020 From: jean-luc.oms at lirmm.fr (Jean-Luc Oms) Date: Fri, 27 Nov 2020 16:15:33 +0100 Subject: [PVE-User] Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 In-Reply-To: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> References: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> Message-ID: <9e7d2743-250e-43f9-7fd4-228382e95a29@lirmm.fr> Next step .... I've a small 'preprod' cluster for testing ... but without ceph. If I install ceph on oe node of this cluster this package is not installed: ceph-mgr-dashboard If I remove this package from my prod cluster, tested on the node running active manager, no dependencies and after manager restart, health is Ok ... Now i have installed : root at ceph1:~# dpkg -l | grep ceph ii? ceph???????????????????????????????? 14.2.15-pve2???????????????? amd64??????? distributed storage and file system ii? ceph-base??????????????????????????? 14.2.15-pve2???????????????? amd64??????? common ceph daemon libraries and management tools ii? ceph-common????????????????????????? 14.2.15-pve2???????????????? amd64??????? common utilities to mount and interact with a ceph storage cluster ii? ceph-fuse??????????????????????????? 14.2.15-pve2???????????????? amd64??????? FUSE-based client for the Ceph distributed file system ii? ceph-mds???????????????????????????? 14.2.15-pve2???????????????? amd64??????? metadata server for the ceph distributed file system ii? ceph-mgr???????????????????????????? 14.2.15-pve2???????????????? amd64??????? manager for the ceph distributed storage system ii? ceph-mon???????????????????????????? 14.2.15-pve2???????????????? amd64??????? monitor server for the ceph storage system ii? ceph-osd???????????????????????????? 14.2.15-pve2???????????????? amd64??????? OSD server for the ceph storage system ii? libcephfs2?????????????????????????? 14.2.15-pve2???????????????? amd64??????? Ceph distributed file system client library ii? python-ceph-argparse???????????????? 14.2.15-pve2???????????????? all????????? Python 2 utility libraries for Ceph CLI ii? python-cephfs??????????????????????? 14.2.15-pve2???????????????? amd64??????? Python 2 libraries for the Ceph libcephfs library Is this ok ? Is ceph-mgr-dashboard? needed ? Thanks Le 27/11/2020 ? 12:47, Jean-Luc Oms a ?crit?: > Bonjour, > > Upgrading to last Proxmox VE / Ceph nautilus from the last 6.2 proxmox > VE seems to introduce a python 2/3 version problem, dashboard healt > stops working. > > root at ceph1:/usr/bin# ceph health > HEALTH_ERR Module 'dashboard' has failed: ('invalid syntax', > ('/usr/share/ceph/mgr/dashboard/controllers/orchestrator.py', 34, 11, > '??? result: dict = {}\n')) > > This syntax was introduced in 3.6, and using strace it seems python 2.7 > is used. > > Any option to resolve this ? everything was ok in 6.2-15. > > Thanks > > -- Jean-Luc Oms /STI-ReseauX - LIRMM - CNRS/UM/ +33 4 67 41 85 93 / +33 6 32 01 04 17 From Jean-Daniel.Tissot at univ-fcomte.fr Fri Nov 27 17:29:53 2020 From: Jean-Daniel.Tissot at univ-fcomte.fr (Jean-Daniel TISSOT) Date: Fri, 27 Nov 2020 17:29:53 +0100 Subject: [PVE-User] Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 In-Reply-To: <3e7c42b6-1a1f-d296-e240-fb98cf87a6ca@gmail.com> References: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> <3e7c42b6-1a1f-d296-e240-fb98cf87a6ca@gmail.com> Message-ID: <47b5a337-b2ca-ce6d-37c5-e904db8d6e03@univ-fcomte.fr> Hi, I have another problem root at dmz-pve1:~ # ceph health HEALTH_WARN 1 pools have too many placement groups root at dmz-pve1:~ # pveceph pool ls ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ? Name????????????????? ? Size ? Min Size ? PG Num ? PG Autoscale Mode ? Crush Rule Name ??????????????? %-Used ????????? Used ? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ? device_health_metrics ???? 3 ???????? 2 ?????? 1 ? on??????????????? ? replicated_rule ? 4.19273845864154e-07 ?????? 4534827 ? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ? rbd?????????????????? ???? 3 ???????? 2 ???? 128 ? warn????????????? ? replicated_rule ??? 0.0116069903597236 ? 127014329075 ? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? In the GUI : 1 pools have too many placement groups Pool rbd has 128 placement groups, should have 32 I don't find any way to reduce placement groups tu 32 Any help welcome. Best regards, Le 27/11/2020 ? 14:07, Lindsay Mathieson a ?crit?: > On 27/11/2020 9:47 pm, Jean-Luc Oms wrote: >> Upgrading to last Proxmox VE / Ceph nautilus from the last 6.2 proxmox >> VE seems to introduce a python 2/3 version problem, dashboard healt >> stops working. > > Was just about to report that :) > > > Same here. > -- Bien cordialement, Jean-Daniel Tissot - IE CNRS http://chrono-environnement.univ-fcomte.fr UMR 6249 - Laboratoire Chrono-environnement UMR CNRS-UFC Universit? de Franche-Comt?, 16 route de Gray, 25030 Besan?on Cedex, FRANCE Jean-Daniel.Tissot at univ-fcomte.fr tel:+33 3 81 666 440 From marcomgabriel at gmail.com Fri Nov 27 17:45:47 2020 From: marcomgabriel at gmail.com (Marco M. Gabriel) Date: Fri, 27 Nov 2020 17:45:47 +0100 Subject: [PVE-User] Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 In-Reply-To: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> References: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> Message-ID: Same problem here after upgrading from 6.2.15 to 6.3 on a test cluster. But the problem disappeared suddenly when I upgraded Ceph from Nautilus to Octopus as well. Not sure if this is the reason why it disappeared and I wouldn't recommend doing this on a production cluster while ceph is in HEALTH_ERR. It would be fine if anyone could test and confirm that an upgrade to Ceph Octopus resolves the issue. Kind regards, Marco Am Fr., 27. Nov. 2020 um 12:54 Uhr schrieb Jean-Luc Oms : > > Bonjour, > > Upgrading to last Proxmox VE / Ceph nautilus from the last 6.2 proxmox > VE seems to introduce a python 2/3 version problem, dashboard healt > stops working. > > root at ceph1:/usr/bin# ceph health > HEALTH_ERR Module 'dashboard' has failed: ('invalid syntax', > ('/usr/share/ceph/mgr/dashboard/controllers/orchestrator.py', 34, 11, > ' result: dict = {}\n')) > > This syntax was introduced in 3.6, and using strace it seems python 2.7 > is used. > > Any option to resolve this ? everything was ok in 6.2-15. > > Thanks > > > -- > Jean-Luc Oms > /STI-ReseauX - LIRMM - CNRS/UM/ > +33 4 67 41 85 93 / +33 6 32 01 04 17 > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user From aderumier at odiso.com Fri Nov 27 17:59:02 2020 From: aderumier at odiso.com (alexandre derumier) Date: Fri, 27 Nov 2020 17:59:02 +0100 Subject: [PVE-User] Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 In-Reply-To: <47b5a337-b2ca-ce6d-37c5-e904db8d6e03@univ-fcomte.fr> References: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> <3e7c42b6-1a1f-d296-e240-fb98cf87a6ca@gmail.com> <47b5a337-b2ca-ce6d-37c5-e904db8d6e03@univ-fcomte.fr> Message-ID: <843b2e4d-ff82-bbdd-6906-8e6ff90bcc96@odiso.com> >>1 pools have too many placement groups Pool rbd has 128 placement groups, should have 32 >> >>I don't find any way to reduce placement groups tu 32 >> >>Any help welcome. you can't reduce PG on nautilus,? only since octopus.? (and ceph is able to do it automataaly with new pg autoscaler) I think it's a warning introduced in last nautilus update. If I remember, they are an option to disable this warning, (but I don't remember it) On 27/11/2020 17:29, Jean-Daniel TISSOT wrote: > Hi, > > I have another problem > > root at dmz-pve1:~ # ceph health HEALTH_WARN 1 pools have too many > placement groups root at dmz-pve1:~ # pveceph pool ls > ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > ? Name????????????????? ? Size ? Min Size ? PG Num ? PG Autoscale Mode > ? Crush Rule Name ??????????????? %-Used ????????? Used ? > ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > ? device_health_metrics ???? 3 ???????? 2 ?????? 1 ? on??????????????? > ? replicated_rule ? 4.19273845864154e-07 ? 4534827 ? > ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > ? rbd?????????????????? ???? 3 ???????? 2 ???? 128 ? warn????????????? > ? replicated_rule ??? 0.0116069903597236 ? 127014329075 ? > ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > > > In the GUI : > > 1 pools have too many placement groups Pool rbd has 128 placement > groups, should have 32 > > I don't find any way to reduce placement groups tu 32 > > Any help welcome. > > Best regards, > > Le 27/11/2020 ? 14:07, Lindsay Mathieson a ?crit?: >> On 27/11/2020 9:47 pm, Jean-Luc Oms wrote: >>> Upgrading to last Proxmox VE / Ceph nautilus from the last 6.2 proxmox >>> VE seems to introduce a python 2/3 version problem, dashboard healt >>> stops working. >> >> Was just about to report that :) >> >> >> Same here. >> From marcomgabriel at gmail.com Fri Nov 27 18:06:14 2020 From: marcomgabriel at gmail.com (Marco M. Gabriel) Date: Fri, 27 Nov 2020 18:06:14 +0100 Subject: [PVE-User] Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 In-Reply-To: <47b5a337-b2ca-ce6d-37c5-e904db8d6e03@univ-fcomte.fr> References: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> <3e7c42b6-1a1f-d296-e240-fb98cf87a6ca@gmail.com> <47b5a337-b2ca-ce6d-37c5-e904db8d6e03@univ-fcomte.fr> Message-ID: You can enable the pg-autoscaler. It does the work for you and optimizes the number of placement groups on a given pool. The pg-autoscaler was introduced with nautilus and we ran it for a while without any problems. Here is an explanation of how to enable and how to use the autoscaler: https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/ Best regards, Marco Am Fr., 27. Nov. 2020 um 17:37 Uhr schrieb Jean-Daniel TISSOT : > > Hi, > > I have another problem > > root at dmz-pve1:~ # ceph health HEALTH_WARN 1 pools have too many > placement groups root at dmz-pve1:~ # pveceph pool ls > ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > ? Name ? Size ? Min Size ? PG Num ? PG Autoscale Mode ? > Crush Rule Name ? %-Used ? Used ? > ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > ? device_health_metrics ? 3 ? 2 ? 1 ? on ? > replicated_rule ? 4.19273845864154e-07 ? 4534827 ? > ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > ? rbd ? 3 ? 2 ? 128 ? warn ? > replicated_rule ? 0.0116069903597236 ? 127014329075 ? > ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? > > > In the GUI : > > 1 pools have too many placement groups Pool rbd has 128 placement > groups, should have 32 > > I don't find any way to reduce placement groups tu 32 > > Any help welcome. > > Best regards, > > Le 27/11/2020 ? 14:07, Lindsay Mathieson a ?crit : > > On 27/11/2020 9:47 pm, Jean-Luc Oms wrote: > >> Upgrading to last Proxmox VE / Ceph nautilus from the last 6.2 proxmox > >> VE seems to introduce a python 2/3 version problem, dashboard healt > >> stops working. > > > > Was just about to report that :) > > > > > > Same here. > > > -- > Bien cordialement, > Jean-Daniel Tissot - IE CNRS http://chrono-environnement.univ-fcomte.fr > UMR 6249 - Laboratoire Chrono-environnement UMR CNRS-UFC > Universit? de Franche-Comt?, 16 route de Gray, 25030 Besan?on Cedex, FRANCE > Jean-Daniel.Tissot at univ-fcomte.fr tel:+33 3 81 666 440 > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user From aderumier at odiso.com Fri Nov 27 18:08:21 2020 From: aderumier at odiso.com (alexandre derumier) Date: Fri, 27 Nov 2020 18:08:21 +0100 Subject: [PVE-User] Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 In-Reply-To: References: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> <3e7c42b6-1a1f-d296-e240-fb98cf87a6ca@gmail.com> <47b5a337-b2ca-ce6d-37c5-e904db8d6e03@univ-fcomte.fr> Message-ID: <784dcedb-ad50-a568-806a-aa01f851c7d6@odiso.com> On 27/11/2020 18:06, Marco M. Gabriel wrote: > You can enable the pg-autoscaler. It does the work for you and > optimizes the number of placement groups on a given pool. Oh, yes indeed, I thinked it was introduced in Octopus, but it's indeed already avaible in Nautilus :) From Jean-Daniel.Tissot at univ-fcomte.fr Fri Nov 27 18:12:11 2020 From: Jean-Daniel.Tissot at univ-fcomte.fr (Jean-Daniel TISSOT) Date: Fri, 27 Nov 2020 18:12:11 +0100 Subject: [PVE-User] ProxmoxVE6.3/CEPH Octopus 15.2.6 was Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 In-Reply-To: <843b2e4d-ff82-bbdd-6906-8e6ff90bcc96@odiso.com> References: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> <3e7c42b6-1a1f-d296-e240-fb98cf87a6ca@gmail.com> <47b5a337-b2ca-ce6d-37c5-e904db8d6e03@univ-fcomte.fr> <843b2e4d-ff82-bbdd-6906-8e6ff90bcc96@odiso.com> Message-ID: <327d2503-b906-5e09-d81a-20fd2f64a638@univ-fcomte.fr> Sorry to steal the thread. In fact before upgrading to Octopus, I don't remember any warning. I upgrade Proxmox an follow the wiki to upgrade CEPH. All things seam working (I don't test HA by migrate a VM work perfectly. I have just the warning on pool rdb (1 pools have too many placement groups Pool rbd has 128 placement groups, should have 32) Le 27/11/2020 ? 17:59, alexandre derumier a ?crit?: > >>1 pools have too many placement groups Pool rbd has 128 placement > groups, should have 32 > >> > >>I don't find any way to reduce placement groups tu 32 > >> > >>Any help welcome. > > you can't reduce PG on nautilus,? only since octopus.? (and ceph is > able to do it automataaly with new pg autoscaler) > > I think it's a warning introduced in last nautilus update. > > If I remember, they are an option to disable this warning, (but I > don't remember it) > > > On 27/11/2020 17:29, Jean-Daniel TISSOT wrote: >> Hi, >> >> I have another problem >> >> root at dmz-pve1:~ # ceph health HEALTH_WARN 1 pools have too many >> placement groups root at dmz-pve1:~ # pveceph pool ls >> ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? >> ? Name????????????????? ? Size ? Min Size ? PG Num ? PG Autoscale >> Mode ? Crush Rule Name ??????????????? %-Used ????????? Used ? >> ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? >> ? device_health_metrics ???? 3 ???????? 2 ?????? 1 ? >> on??????????????? ? replicated_rule ? 4.19273845864154e-07 ? 4534827 >> ? >> ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? >> ? rbd?????????????????? ???? 3 ???????? 2 ???? 128 ? >> warn????????????? ? replicated_rule ??? 0.0116069903597236 ? >> 127014329075 ? >> ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? >> >> >> In the GUI : >> >> 1 pools have too many placement groups Pool rbd has 128 placement >> groups, should have 32 >> >> I don't find any way to reduce placement groups tu 32 >> >> Any help welcome. >> >> Best regards, >> >> Le 27/11/2020 ? 14:07, Lindsay Mathieson a ?crit?: >>> On 27/11/2020 9:47 pm, Jean-Luc Oms wrote: >>>> Upgrading to last Proxmox VE / Ceph nautilus from the last 6.2 proxmox >>>> VE seems to introduce a python 2/3 version problem, dashboard healt >>>> stops working. >>> >>> Was just about to report that :) >>> >>> >>> Same here. >>> > > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Bien cordialement, Jean-Daniel Tissot - IE CNRS http://chrono-environnement.univ-fcomte.fr UMR 6249 - Laboratoire Chrono-environnement UMR CNRS-UFC Universit? de Franche-Comt?, 16 route de Gray, 25030 Besan?on Cedex, FRANCE Jean-Daniel.Tissot at univ-fcomte.fr tel:+33 3 81 666 440 Alabama, Mississippi, Minnesota, South Carolina, Oregon... not so sweet home Black Panther Party, rena?t de tes cendres et reviens les aider https://www.youtube.com/watch?v=ZvilFSMVHTs From Jean-Daniel.Tissot at univ-fcomte.fr Fri Nov 27 18:18:57 2020 From: Jean-Daniel.Tissot at univ-fcomte.fr (Jean-Daniel TISSOT) Date: Fri, 27 Nov 2020 18:18:57 +0100 Subject: [PVE-User] Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 In-Reply-To: References: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> <3e7c42b6-1a1f-d296-e240-fb98cf87a6ca@gmail.com> <47b5a337-b2ca-ce6d-37c5-e904db8d6e03@univ-fcomte.fr> Message-ID: Many thanks. Work like a charm. No more warning. Again many thanks Marco Best regards, Jean-Daniel Le 27/11/2020 ? 18:06, Marco M. Gabriel a ?crit?: > You can enable the pg-autoscaler. It does the work for you and > optimizes the number of placement groups on a given pool. > > The pg-autoscaler was introduced with nautilus and we ran it for a > while without any problems. > > Here is an explanation of how to enable and how to use the autoscaler: > https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/ > > Best regards, > Marco > > > Am Fr., 27. Nov. 2020 um 17:37 Uhr schrieb Jean-Daniel TISSOT > : >> Hi, >> >> I have another problem >> >> root at dmz-pve1:~ # ceph health HEALTH_WARN 1 pools have too many >> placement groups root at dmz-pve1:~ # pveceph pool ls >> ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? >> ? Name ? Size ? Min Size ? PG Num ? PG Autoscale Mode ? >> Crush Rule Name ? %-Used ? Used ? >> ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? >> ? device_health_metrics ? 3 ? 2 ? 1 ? on ? >> replicated_rule ? 4.19273845864154e-07 ? 4534827 ? >> ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? >> ? rbd ? 3 ? 2 ? 128 ? warn ? >> replicated_rule ? 0.0116069903597236 ? 127014329075 ? >> ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? >> >> >> In the GUI : >> >> 1 pools have too many placement groups Pool rbd has 128 placement >> groups, should have 32 >> >> I don't find any way to reduce placement groups tu 32 >> >> Any help welcome. >> >> Best regards, >> >> Le 27/11/2020 ? 14:07, Lindsay Mathieson a ?crit : >>> On 27/11/2020 9:47 pm, Jean-Luc Oms wrote: >>>> Upgrading to last Proxmox VE / Ceph nautilus from the last 6.2 proxmox >>>> VE seems to introduce a python 2/3 version problem, dashboard healt >>>> stops working. >>> Was just about to report that :) >>> >>> >>> Same here. >>> >> -- >> Bien cordialement, >> Jean-Daniel Tissot - IE CNRS http://chrono-environnement.univ-fcomte.fr >> UMR 6249 - Laboratoire Chrono-environnement UMR CNRS-UFC >> Universit? de Franche-Comt?, 16 route de Gray, 25030 Besan?on Cedex, FRANCE >> Jean-Daniel.Tissot at univ-fcomte.fr tel:+33 3 81 666 440 >> >> _______________________________________________ >> pve-user mailing list >> pve-user at lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Bien cordialement, Jean-Daniel Tissot - IE CNRS http://chrono-environnement.univ-fcomte.fr UMR 6249 - Laboratoire Chrono-environnement UMR CNRS-UFC Universit? de Franche-Comt?, 16 route de Gray, 25030 Besan?on Cedex, FRANCE Jean-Daniel.Tissot at univ-fcomte.fr tel:+33 3 81 666 440 From lindsay.mathieson at gmail.com Fri Nov 27 22:17:56 2020 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Sat, 28 Nov 2020 07:17:56 +1000 Subject: [PVE-User] Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 In-Reply-To: <843b2e4d-ff82-bbdd-6906-8e6ff90bcc96@odiso.com> References: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> <3e7c42b6-1a1f-d296-e240-fb98cf87a6ca@gmail.com> <47b5a337-b2ca-ce6d-37c5-e904db8d6e03@univ-fcomte.fr> <843b2e4d-ff82-bbdd-6906-8e6ff90bcc96@odiso.com> Message-ID: <4f70ac8b-ac80-c852-c5d5-9a9254e184bd@gmail.com> On 28/11/2020 2:59 am, alexandre derumier wrote: > you can't reduce PG on nautilus,? only since octopus.? (and ceph is > able to do it automataaly with new pg autoscaler) Actually you can, has been the case since Nautilus. https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/ -- Lindsay From lindsay.mathieson at gmail.com Sat Nov 28 04:17:02 2020 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Sat, 28 Nov 2020 13:17:02 +1000 Subject: [PVE-User] Proxmox 6.3 - Upgrade Ceph Nautilus to Octopus Message-ID: Just performed this as per the instructions: https://pve.proxmox.com/wiki/Ceph_Nautilus_to_Octopus Went flawlessly, no problems. Small 5 node cluster, took about 30 min with me obsessively checking ever step. I did shut down all non-essential VM's first. Because I like to live on the edge, I did it remotely :) With the router and VPN (pfSense) virtualised on the promox cluster I was upgrading no less :) I also enabled the autoscaller - had proiblems with it in Nautilus, so it was set to warn. Now its slowly decreasing my pool from 512 pgs to 128. Not having any problems with this so far. -- Lindsay From lindsay.mathieson at gmail.com Sat Nov 28 04:23:21 2020 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Sat, 28 Nov 2020 13:23:21 +1000 Subject: [PVE-User] Python problem with upgrade to proxmoxVE6.3/ CEPH Nautilus 14.2.15 In-Reply-To: References: <930e1177-6137-aa49-718c-04f4fa74e975@lirmm.fr> Message-ID: On 28/11/2020 2:45 am, Marco M. Gabriel wrote: > It would be fine if anyone could test and confirm that an upgrade to > Ceph Octopus resolves the issue. Regards the dashboard not working on Proxmox 6.3 - upgrading to ceph Octopus fixed it for me. Also the dashboard looks a lot more swish :) -- Lindsay From lindsay.mathieson at gmail.com Mon Nov 30 00:42:43 2020 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Mon, 30 Nov 2020 09:42:43 +1000 Subject: [PVE-User] PX 6.3 - VM Shutdown hanging Message-ID: <08538f1f-8d16-4080-c875-ea44d5ef8ffb@gmail.com> This started happening when I updated to 6.3, but I also updated to Nautilus 14.2.15, so could be related to that. Also still happening with Octopus. When I shut down a windows VM it never reaches a a stopped state - hangs on the running status. * VM Console is just black. * The context menu for the VM only has start enabled, stop is disabled, I have to use "qm stop " to terminate it. * The VM icon is light gray with no green running arrow, but not fully grayed. * CPU cycles around 0% - 0.2% * Windows 10 VM's fully up[dated * VIRTIO SCSI bus * qemu tools etc installed This happens whether I shut the VM down using its windows menu or invoke it externally -- Lindsay From lindsay.mathieson at gmail.com Mon Nov 30 00:53:59 2020 From: lindsay.mathieson at gmail.com (Lindsay Mathieson) Date: Mon, 30 Nov 2020 09:53:59 +1000 Subject: [PVE-User] PX 6.3 - VM Shutdown hanging In-Reply-To: <08538f1f-8d16-4080-c875-ea44d5ef8ffb@gmail.com> References: <08538f1f-8d16-4080-c875-ea44d5ef8ffb@gmail.com> Message-ID: On 30/11/2020 9:42 am, Lindsay Mathieson wrote: > The context menu for the VM only has start enabled Correction, the Context menu start item is not enabled. -- Lindsay From leandro at tecnetmza.com.ar Mon Nov 30 12:13:57 2020 From: leandro at tecnetmza.com.ar (Leandro Roggerone) Date: Mon, 30 Nov 2020 08:13:57 -0300 Subject: [PVE-User] moving a vmware machine into proxmox In-Reply-To: References: Message-ID: Dear Alex: Thanks for your words. Let me clarify some points: a VM uses two disks , I moved and converted both. b Hardware changed ?... yes it has, (I wanted to share a picture but my mail is dismissed with a "no content allowed" message). Both platforms offer different hardware to guests so hardware, naturally will change when moving between platforms. c Regarding relevant services ... First I noticed , is that XAMP platform was not running. After cheking I confirmed that new guest didnt recognized network card, I think it was due to drivers difference. (I had to install new driver). Since some services are attached to listen to some interfaces / ips , those services did not start at startup. So.... After looking at network drive options I can confirm that can not set exactly same driver. Conclusion: hard disk and network card are some of the devices should be same in order to have a transparent migration. Since (I think) it is not possible , it will be hard to smoothly complete this migration. Hope to hear your comments about this. Leandro. Libre de virus. www.avast.com <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> El jue, 26 nov 2020 a las 13:27, Alex Chekholko via pve-user (< pve-user at lists.proxmox.com>) escribi?: > > > > ---------- Forwarded message ---------- > From: Alex Chekholko > To: Proxmox VE user list > Cc: PVE User List > Bcc: > Date: Thu, 26 Nov 2020 08:26:19 -0800 > Subject: Re: [PVE-User] moving a vmware machine into proxmox > From the Windows point of view, the hardware changed, check in device > manager that all devices are there? In the service manager, make sure the > relevant services started up? > > Also, just to double-check, did the original VM have only one disk? If it > had more than one, you just need to copy the vmdk and import it the same > way and attach it to the new VM. > > > On Thu, Nov 26, 2020 at 8:19 AM Leandro Roggerone < > leandro at tecnetmza.com.ar> > wrote: > > > Hi guys , I successfully moved a vm from my ESXi server to my proxmox. > > This is what I did: > > I created a backup with guettoVCB tool. > > SCP the vmdk disk images into my proxmox local partition. > > Created a VM > > convert those images into qcow2 doing: > > qm importdisk 117 ESXi_1-000003-flat.vmdk local-lvm -format qcow2 > > Then I added this new disk image into my vm 117. > > > > After turning this machine on wich contains a windows server , following > > happened. > > Machine is ok, I can login and so. > > but > > There is sql server wich is not running. > > After trying to open sql server configuration manager, im getting the > "sql > > server can not connect to wmi provider" error message. > > > > I know Im providing not enough information but this is perhaps some well > > know issue. > > I already confirmed that after rebooting original machine on ESXi server > , > > there is no manual action required to have all process running. > > So , this can be perhaps something related with licence. > > Any feedback / feelling about this ? > > I need to move this machine without outage nor customer action. > > Regards. > > Leandro. > > > > > > > > < > > > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > > > Libre > > de virus. www.avast.com > > < > > > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail > > > > > <#m_9061933542358666280_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > _______________________________________________ > > pve-user mailing list > > pve-user at lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > > > ---------- Forwarded message ---------- > From: Alex Chekholko via pve-user > To: Proxmox VE user list > Cc: Alex Chekholko , PVE User List < > pve-user at pve.proxmox.com> > Bcc: > Date: Thu, 26 Nov 2020 08:26:19 -0800 > Subject: Re: [PVE-User] moving a vmware machine into proxmox > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From ronan.dily at rouxel.fr Mon Nov 30 14:23:31 2020 From: ronan.dily at rouxel.fr (Ronan Dily) Date: Mon, 30 Nov 2020 13:23:31 +0000 Subject: [PVE-User] Convert linked clone to template Message-ID: Hi there, I've made a template and create a linked clone from this. Since I've done many change on the clone, I'd like to convert this clone to a template. Can I simply convert this VM into a template and delete the old template ? Thank you all, Ronan Dily Responsable Informatique T?l : 02 97 47 17 37 Mob : 06 25 01 92 64 ronan.dily at rouxel.fr From leandro at tecnetmza.com.ar Mon Nov 30 17:21:32 2020 From: leandro at tecnetmza.com.ar (Leandro Roggerone) Date: Mon, 30 Nov 2020 13:21:32 -0300 Subject: [PVE-User] moving machines among proxmoxs servers. Message-ID: Hi guys. Just wondering if is it possible to move machines without outage ? What do I need to achieve this ? Currently have only one box ... Thanks. Leandro. Libre de virus. www.avast.com <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> From abonilla at suse.com Mon Nov 30 17:45:29 2020 From: abonilla at suse.com (Alejandro Bonilla) Date: Mon, 30 Nov 2020 16:45:29 +0000 Subject: [PVE-User] moving machines among proxmoxs servers. In-Reply-To: References: Message-ID: <3A549993-732C-4F4E-87C0-BE4B8E23533D@suse.com> > On Nov 30, 2020, at 11:21 AM, Leandro Roggerone wrote: > > Hi guys. > Just wondering if is it possible to move machines without outage ? I thought at first you were referring to a live-migration which is easy to achieve 64 bytes from 10.0.0.111: icmp_seq=25 ttl=64 time=0.363 ms 64 bytes from 10.0.0.111: icmp_seq=26 ttl=64 time=0.397 ms 64 bytes from 10.0.0.111: icmp_seq=27 ttl=64 time=0.502 ms Request timeout for icmp_seq 28 64 bytes from 10.0.0.111: icmp_seq=29 ttl=64 time=0.366 ms 64 bytes from 10.0.0.111: icmp_seq=30 ttl=64 time=0.562 ms 64 bytes from 10.0.0.111: icmp_seq=31 ttl=64 time=0.469 ms And certainly happens with little to no outage. > What do I need to achieve this ? More than one node (cluster), storage, then perform a migration? using storage like Ceph will make the migration way faster. > Currently have only one box ... And then I got confused. Are you trying to migrate from another hypervisor or you are just asking if it?s possible at all and then would add another box? > Thanks. > Leandro. > From leandro at tecnetmza.com.ar Mon Nov 30 18:10:03 2020 From: leandro at tecnetmza.com.ar (Leandro Roggerone) Date: Mon, 30 Nov 2020 14:10:03 -0300 Subject: [PVE-User] moving machines among proxmoxs servers. In-Reply-To: References: Message-ID: Alejandro , thanks for your words. Let me explain: About live migration ... yes I think this is what I need to achieve. So basically you can "drag and drop" VMs from one node to another ? What do I need to achieve this ? / Only have one node. My current pve bos is in production with very important machines running on it. I will add a second pve server machine soon. But , I dont have any network storage , so the question would be: Having two pve machines (already running and fresh one) is it possible to perform live migrations? or is it mandatory to have an intermediate hardware or something like that? Regards, Leandro. Libre de virus. www.avast.com <#m_-8318222231522076233_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> El lun, 30 nov 2020 a las 13:45, Alejandro Bonilla via pve-user (< pve-user at lists.proxmox.com>) escribi?: > > > > ---------- Forwarded message ---------- > From: Alejandro Bonilla > To: Proxmox VE user list > Cc: > Bcc: > Date: Mon, 30 Nov 2020 16:45:29 +0000 > Subject: Re: [PVE-User] moving machines among proxmoxs servers. > > > > On Nov 30, 2020, at 11:21 AM, Leandro Roggerone < > leandro at tecnetmza.com.ar> wrote: > > > > Hi guys. > > Just wondering if is it possible to move machines without outage ? > > I thought at first you were referring to a live-migration which is easy to > achieve > > 64 bytes from 10.0.0.111: icmp_seq=25 ttl=64 time=0.363 ms > 64 bytes from 10.0.0.111: icmp_seq=26 ttl=64 time=0.397 ms > 64 bytes from 10.0.0.111: icmp_seq=27 ttl=64 time=0.502 ms > Request timeout for icmp_seq 28 > 64 bytes from 10.0.0.111: icmp_seq=29 ttl=64 time=0.366 ms > 64 bytes from 10.0.0.111: icmp_seq=30 ttl=64 time=0.562 ms > 64 bytes from 10.0.0.111: icmp_seq=31 ttl=64 time=0.469 ms > > And certainly happens with little to no outage. > > > What do I need to achieve this ? > > More than one node (cluster), storage, then perform a migration? using > storage like Ceph will make the migration way faster. > > > Currently have only one box ... > > And then I got confused. Are you trying to migrate from another hypervisor > or you are just asking if it?s possible at all and then would add another > box? > > > Thanks. > > Leandro. > > > > > > > ---------- Forwarded message ---------- > From: Alejandro Bonilla via pve-user > To: Proxmox VE user list > Cc: Alejandro Bonilla > Bcc: > Date: Mon, 30 Nov 2020 16:45:29 +0000 > Subject: Re: [PVE-User] moving machines among proxmoxs servers. > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > Libre de virus. www.avast.com <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> From gilberto.nunes32 at gmail.com Mon Nov 30 18:26:31 2020 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Mon, 30 Nov 2020 14:26:31 -0300 Subject: [PVE-User] moving machines among proxmoxs servers. In-Reply-To: References: Message-ID: Yes! Using two machines and GlusterFS for instance, is an easier way to achieve this. ( First of all you need create a cluster over Proxmox: https://pve.proxmox.com/wiki/Cluster_Manager) Just make two folders, like /DATA, in each server. Make sure this folder is separated into HDD, rather into just one HDD! Then, follow the instructions here to install and upgrade glusterfs: https://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/ (make sure you choose buster!) Install gluster server: apt install glusterfs-server. Now, make sure that you have a separated NIC to make all the network traffic run over this NIC. Use some private network address. After installing gluster server do this in the first node: gluster peer probe server1 Then use this command to create a replica 2 glusterfs server: gluster vol create VMS replica 2 server1:/DATA/vms server2:/DATA/vms (make sure server1 and server2 are in /etc/hosts with correspondent private IP) Make the vol VMS up: gluster vol start VMS Then add this do the /etc/fstab in server1: server1:VMS /vms glusterfs defaults,_netdev,x-systemd.automount,backupvolfile-server=server2 0 0 And add this do the /etc/fstab in server2: server2:VMS /vms glusterfs defaults,_netdev,x-systemd.automount,backupvolfile-server=server1 0 0 Add this in each server: file: /etc/systemd/system/glusterfsmounts.service: [Unit] Description=Glustermounting Requires=glusterd.service [Service] Type=simple RemainAfterExit=true ExecStartPre=/usr/sbin/gluster volume list ExecStart=/bin/mount -a -t glusterfs Restart=on-failure RestartSec=3 [Install] WantedBy=multi-user.target then you would: systemctl daemon-reload systemctl enable glusterfsmounts This will make sure the system mounts the /vms directory, after a reboot. You need apply some tricks here: gluster vol set VMS cluster.heal-timeout 5 gluster volume heal VMS enable gluster vol set VMS cluster.quorum-reads false gluster vol set VMS cluster.quorum-count 1 gluster vol set VMS network.ping-timeout 2 gluster volume set VMS cluster.favorite-child-policy mtime gluster volume heal VMS granular-entry-heal enable gluster volume set VMS cluster.data-self-heal-algorithm full After all that, go to Datacenter -> Storage -> Directory and add the /vms as a directory storage in your PROXMOX. Remember to mark as shared storage. I have used this set up for many months now and so far no issues. But the most clever way is to keep backup, right? Cheers --- Gilberto Nunes Ferreira Em seg., 30 de nov. de 2020 ?s 14:10, Leandro Roggerone escreveu: > > Alejandro , thanks for your words. > Let me explain: > About live migration ... yes I think this is what I need to achieve. > So basically you can "drag and drop" VMs from one node to another ? > > What do I need to achieve this ? / Only have one node. > My current pve bos is in production with very important machines running on > it. > I will add a second pve server machine soon. > But , I dont have any network storage , so the question would be: > Having two pve machines (already running and fresh one) is it possible to > perform live migrations? > or is it mandatory to have an intermediate hardware or something like that? > > Regards, > Leandro. > > > > > > Libre > de virus. www.avast.com > > <#m_-8318222231522076233_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > El lun, 30 nov 2020 a las 13:45, Alejandro Bonilla via pve-user (< > pve-user at lists.proxmox.com>) escribi?: > > > > > > > > > ---------- Forwarded message ---------- > > From: Alejandro Bonilla > > To: Proxmox VE user list > > Cc: > > Bcc: > > Date: Mon, 30 Nov 2020 16:45:29 +0000 > > Subject: Re: [PVE-User] moving machines among proxmoxs servers. > > > > > > > On Nov 30, 2020, at 11:21 AM, Leandro Roggerone < > > leandro at tecnetmza.com.ar> wrote: > > > > > > Hi guys. > > > Just wondering if is it possible to move machines without outage ? > > > > I thought at first you were referring to a live-migration which is easy to > > achieve > > > > 64 bytes from 10.0.0.111: icmp_seq=25 ttl=64 time=0.363 ms > > 64 bytes from 10.0.0.111: icmp_seq=26 ttl=64 time=0.397 ms > > 64 bytes from 10.0.0.111: icmp_seq=27 ttl=64 time=0.502 ms > > Request timeout for icmp_seq 28 > > 64 bytes from 10.0.0.111: icmp_seq=29 ttl=64 time=0.366 ms > > 64 bytes from 10.0.0.111: icmp_seq=30 ttl=64 time=0.562 ms > > 64 bytes from 10.0.0.111: icmp_seq=31 ttl=64 time=0.469 ms > > > > And certainly happens with little to no outage. > > > > > What do I need to achieve this ? > > > > More than one node (cluster), storage, then perform a migration? using > > storage like Ceph will make the migration way faster. > > > > > Currently have only one box ... > > > > And then I got confused. Are you trying to migrate from another hypervisor > > or you are just asking if it?s possible at all and then would add another > > box? > > > > > Thanks. > > > Leandro. > > > > > > > > > > > > > ---------- Forwarded message ---------- > > From: Alejandro Bonilla via pve-user > > To: Proxmox VE user list > > Cc: Alejandro Bonilla > > Bcc: > > Date: Mon, 30 Nov 2020 16:45:29 +0000 > > Subject: Re: [PVE-User] moving machines among proxmoxs servers. > > _______________________________________________ > > pve-user mailing list > > pve-user at lists.proxmox.com > > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > Libre > de virus. www.avast.com > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user From abonilla at suse.com Mon Nov 30 18:37:15 2020 From: abonilla at suse.com (Alejandro Bonilla) Date: Mon, 30 Nov 2020 17:37:15 +0000 Subject: [PVE-User] moving machines among proxmoxs servers. In-Reply-To: References: Message-ID: <74B151CA-6729-4EAD-8158-3830B949886F@suse.com> On Nov 30, 2020, at 12:10 PM, Leandro Roggerone > wrote: Alejandro , thanks for your words. Let me explain: About live migration ... yes I think this is what I need to achieve. So basically you can "drag and drop" VMs from one node to another ? Right click the VM -> Migrate -> Choose target node What do I need to achieve this ? / Only have one node. You can?t with one node - where are you going to migrate? My current pve bos is in production with very important machines running on it. I will add a second pve server machine soon. But , I dont have any network storage , so the question would be: Having two pve machines (already running and fresh one) is it possible to perform live migrations? or is it mandatory to have an intermediate hardware or something like that? Have a look at https://YourProxmoxServer:8006/pve-docs/chapter-qm.html#qm_migration The Help/Docs in your Proxmox node has the information you look for. Yes you can You need more than one node, a Proxmox Cluster An Online/Live migration will take longer if using local storage as it will copy the entire VM across nodes. Go into the Replication section within your VM and click Help to learn how to reduce the migration times. -> pve-docs/chapter-pvesr.html#chapter_pvesr HTH Regards, Leandro. Libre de virus. www.avast.com <#m_-8318222231522076233_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> El lun, 30 nov 2020 a las 13:45, Alejandro Bonilla via pve-user (< pve-user at lists.proxmox.com>) escribi?: ---------- Forwarded message ---------- From: Alejandro Bonilla > To: Proxmox VE user list > Cc: Bcc: Date: Mon, 30 Nov 2020 16:45:29 +0000 Subject: Re: [PVE-User] moving machines among proxmoxs servers. On Nov 30, 2020, at 11:21 AM, Leandro Roggerone < leandro at tecnetmza.com.ar> wrote: Hi guys. Just wondering if is it possible to move machines without outage ? I thought at first you were referring to a live-migration which is easy to achieve 64 bytes from 10.0.0.111: icmp_seq=25 ttl=64 time=0.363 ms 64 bytes from 10.0.0.111: icmp_seq=26 ttl=64 time=0.397 ms 64 bytes from 10.0.0.111: icmp_seq=27 ttl=64 time=0.502 ms Request timeout for icmp_seq 28 64 bytes from 10.0.0.111: icmp_seq=29 ttl=64 time=0.366 ms 64 bytes from 10.0.0.111: icmp_seq=30 ttl=64 time=0.562 ms 64 bytes from 10.0.0.111: icmp_seq=31 ttl=64 time=0.469 ms And certainly happens with little to no outage. What do I need to achieve this ? More than one node (cluster), storage, then perform a migration? using storage like Ceph will make the migration way faster. Currently have only one box ... And then I got confused. Are you trying to migrate from another hypervisor or you are just asking if it?s possible at all and then would add another box? Thanks. Leandro. ---------- Forwarded message ---------- From: Alejandro Bonilla via pve-user > To: Proxmox VE user list > Cc: Alejandro Bonilla > Bcc: Date: Mon, 30 Nov 2020 16:45:29 +0000 Subject: Re: [PVE-User] moving machines among proxmoxs servers. _______________________________________________ pve-user mailing list pve-user at lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user Libre de virus. www.avast.com <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> _______________________________________________ pve-user mailing list pve-user at lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user From jm at ginernet.com Mon Nov 30 18:28:32 2020 From: jm at ginernet.com (=?UTF-8?Q?Jos=c3=a9_Manuel_Giner?=) Date: Mon, 30 Nov 2020 18:28:32 +0100 Subject: [PVE-User] moving machines among proxmoxs servers. In-Reply-To: References: Message-ID: Hi, unfortunately Proxmox does not have a feature like vzmigrate that allows you to move a VPS from one Proxmox to another on different networks. You need a cluster. And if you want to do a live migration, the cluster must include a shared datastore with the VMs hosted on it. On 30/11/2020 18:10, Leandro Roggerone wrote: > Alejandro , thanks for your words. > Let me explain: > About live migration ... yes I think this is what I need to achieve. > So basically you can "drag and drop" VMs from one node to another ? > > What do I need to achieve this ? / Only have one node. > My current pve bos is in production with very important machines running on > it. > I will add a second pve server machine soon. > But , I dont have any network storage , so the question would be: > Having two pve machines (already running and fresh one) is it possible to > perform live migrations? > or is it mandatory to have an intermediate hardware or something like that? > > Regards, > Leandro. > > > > > > Libre > de virus. www.avast.com > > <#m_-8318222231522076233_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > El lun, 30 nov 2020 a las 13:45, Alejandro Bonilla via pve-user (< > pve-user at lists.proxmox.com>) escribi?: > >> >> >> >> ---------- Forwarded message ---------- >> From: Alejandro Bonilla >> To: Proxmox VE user list >> Cc: >> Bcc: >> Date: Mon, 30 Nov 2020 16:45:29 +0000 >> Subject: Re: [PVE-User] moving machines among proxmoxs servers. >> >> >>> On Nov 30, 2020, at 11:21 AM, Leandro Roggerone < >> leandro at tecnetmza.com.ar> wrote: >>> >>> Hi guys. >>> Just wondering if is it possible to move machines without outage ? >> >> I thought at first you were referring to a live-migration which is easy to >> achieve >> >> 64 bytes from 10.0.0.111: icmp_seq=25 ttl=64 time=0.363 ms >> 64 bytes from 10.0.0.111: icmp_seq=26 ttl=64 time=0.397 ms >> 64 bytes from 10.0.0.111: icmp_seq=27 ttl=64 time=0.502 ms >> Request timeout for icmp_seq 28 >> 64 bytes from 10.0.0.111: icmp_seq=29 ttl=64 time=0.366 ms >> 64 bytes from 10.0.0.111: icmp_seq=30 ttl=64 time=0.562 ms >> 64 bytes from 10.0.0.111: icmp_seq=31 ttl=64 time=0.469 ms >> >> And certainly happens with little to no outage. >> >>> What do I need to achieve this ? >> >> More than one node (cluster), storage, then perform a migration? using >> storage like Ceph will make the migration way faster. >> >>> Currently have only one box ... >> >> And then I got confused. Are you trying to migrate from another hypervisor >> or you are just asking if it?s possible at all and then would add another >> box? >> >>> Thanks. >>> Leandro. >>> >> >> >> >> >> ---------- Forwarded message ---------- >> From: Alejandro Bonilla via pve-user >> To: Proxmox VE user list >> Cc: Alejandro Bonilla >> Bcc: >> Date: Mon, 30 Nov 2020 16:45:29 +0000 >> Subject: Re: [PVE-User] moving machines among proxmoxs servers. >> _______________________________________________ >> pve-user mailing list >> pve-user at lists.proxmox.com >> https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > > Libre > de virus. www.avast.com > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Jos? Manuel Giner https://ginernet.com