From proxmox at iancoetzee.za.net Mon Jul 1 09:10:31 2019 From: proxmox at iancoetzee.za.net (Ian Coetzee) Date: Mon, 1 Jul 2019 09:10:31 +0200 Subject: [PVE-User] ceph-osd not starting after network related issues Message-ID: Hi All, This morning I have a bit of a big boo-boo on our production system. After a very sudden network outage somewhere during the night, one of my ceph-osd's is no longer starting up. If I try and start it manually, I get a very spectacular failure, see link. https://www.jacklin.co.za/zerobin/?04e2dcd13ab8dfc8#zKCISUvAm4o/6mnLmyu+8fSS1VumC65XaETt/dD7rn0= As near as I can tell, it seems to be asserting whether a file exsists, I have yet to determine which file that would be. Any pointers are welcome, as well as any other ideas to get the osd back. For some reason there is data on the osd that was not replicated to my other osd's, as such I can not just re-init this osd as some of the posts I could find suggests I am also going to head to the ceph ML in a bit (after I have registered) Kind regards From gaio at sv.lnf.it Mon Jul 1 11:00:48 2019 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Mon, 1 Jul 2019 11:00:48 +0200 Subject: [PVE-User] Again trouble, but this time with ext4/trim... Message-ID: <20190701090048.GA3471@sv.lnf.it> Trying to fix my trouble with Win2016, last week i've reboot completely my PVE 5 cluster. Cluster use a SAN (HP MSA 1040), thin storage mode, used as LVM in PVE, eg: lvm: VM0 vgname VM0 content images,rootdir shared 1 Because the underlying storage are thin, i've do some test in the past and enable 'discard' on PVE volumes, verifying that storage space get effectively freed. After that reboot, at every execution of 'fstrim' (via systemd timer) in two different VMs i got: Jul 1 00:00:03 vdmsv2 kernel: [542165.662171] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 647, block bitmap and bg descriptor inconsistent: 21279 vs 32768 free clusters Jul 1 00:00:03 vdmsv2 kernel: [542165.695216] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 648, block bitmap and bg descriptor inconsistent: 12011 vs 32768 free clusters Jul 1 00:00:03 vdmsv2 kernel: [542165.705122] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 649, block bitmap and bg descriptor inconsistent: 14971 vs 32768 free clusters Jul 1 00:00:03 vdmsv2 kernel: [542165.711123] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 650, block bitmap and bg descriptor inconsistent: 12194 vs 32768 free clusters Jul 1 00:00:06 vdmsv2 kernel: [542169.243447] JBD2: Spotted dirty metadata buffer (dev = sda6, blocknr = 0). There's a risk of filesystem corruption in case of system crash. Jul 1 00:00:06 vdmsv2 kernel: [542169.244995] JBD2: Spotted dirty metadata buffer (dev = sda6, blocknr = 0). There's a risk of filesystem corruption in case of system crash. (sda6 is /var) Jul 1 00:01:05 vdmsv1 fstrim[3806]: fstrim: /var/spool/squid: FITRIM ioctl failed: Messaggio errato Jul 1 00:01:05 vdmsv1 kernel: [9519779.612188] EXT4-fs error (device sdb1): ext4_validate_block_bitmap:386: comm fstrim: bg 506: bad block bitmap checksum Jul 1 00:01:05 vdmsv1 kernel: [9519779.636314] EXT4-fs warning (device sdb1): ext4_trim_all_free:5130: Error -74 loading buddy information for 506 Jul 1 00:01:08 vdmsv1 kernel: [9519782.441786] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 181, block bitmap and bg descriptor inconsistent: 32767 vs 32768 free clusters Jul 1 00:01:08 vdmsv1 kernel: [9519782.448571] JBD2: Spotted dirty metadata buffer (dev = sda6, blocknr = 0). There's a risk of filesystem corruption in case of system crash. Jul 1 00:01:09 vdmsv1 kernel: [9519782.973496] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 278, block bitmap and bg descriptor inconsistent: 32749 vs 32768 free clusters Jul 1 00:01:09 vdmsv1 kernel: [9519783.034358] EXT4-fs error (device sda6): ext4_mb_generate_buddy:759: group 325, block bitmap and bg descriptor inconsistent: 32740 vs 32768 free clusters Jul 1 00:01:09 vdmsv1 kernel: [9519783.238781] JBD2: Spotted dirty metadata buffer (dev = sda6, blocknr = 0). There's a risk of filesystem corruption in case of system crash. (sdb1 is /var/spool/squid and sda6 is /var) In that servers i've also some other FS, but the ext4 ones are low varying, mounted RO or noatime, but also some FS in XFS fs, that seems does not suffer. Every time i run 'fstrim -av' by hand i've never catch an error, so seems to me that trouble happen whene the system are under high IO (as when run many weekly tasks). For now, i disable fstrim timer. But seeking some help... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From jmr.richardson at gmail.com Tue Jul 2 04:19:55 2019 From: jmr.richardson at gmail.com (JR Richardson) Date: Mon, 1 Jul 2019 21:19:55 -0500 Subject: [PVE-User] Cluster Host Node Rebooted and VMs Did not restart? Message-ID: <000601d5307c$a417aa70$ec46ff50$@gmail.com> Hey Folks, Ran into an interesting problem today. Topology is a 4-Node Cluster on v.5.2, been up and running without issues for 10 months or so. Several VMs running on node 1, 2 & 3, maybe 30% loaded. I keep node 4 empty for HA and new build VM testing. All VMs are set for HA and auto start on node reboot. Node 1 rebooted, no indication why, nothing in the logs identifying cause of the node rebooting. When node 1 came back up it was 1 hour ahead of time and none of the VMs would start and HA did not start up the VMs on empty node 4. Syslog: Jul 1 14:42:00 vmpve01 systemd[1]: Starting Proxmox VE replication runner... Jul 1 14:42:01 vmpve01 systemd[1]: Started Proxmox VE replication runner. Jul 1 14:42:20 vmpve01 pvedaemon[18577]: successful auth for user 'NOC at pve' Jul 1 14:43:00 vmpve01 systemd[1]: Starting Proxmox VE replication runner... Jul 1 14:43:00 vmpve01 pvedaemon[18577]: successful auth for user 'NOC at pve' Jul 1 14:43:01 vmpve01 systemd[1]: Started Proxmox VE replication runner. Jul 1 14:43:40 vmpve01 pvedaemon[32611]: successful auth for user 'NOC at pve' Jul 1 14:43:50 vmpve01 pveproxy[9369]: worker exit Jul 1 14:43:50 vmpve01 pveproxy[2395]: worker 9369 finished Jul 1 14:43:50 vmpve01 pveproxy[2395]: starting 1 worker(s) Jul 1 14:43:50 vmpve01 pveproxy[2395]: worker 21373 started Jul 1 14:44:00 vmpve01 systemd[1]: Starting Proxmox VE replication runner... Jul 1 14:44:01 vmpve01 systemd[1]: Started Proxmox VE replication runner. Jul 1 14:44:03 vmpve01 systemd[1]: Time has been changed Jul 1 14:44:03 vmpve01 systemd[1]: apt-daily-upgrade.timer: Adding 51min 27.103008s random time. Jul 1 14:44:03 vmpve01 systemd[1]: pve-daily-update.timer: Adding 49min 46.847880s random time. Jul 1 14:44:03 vmpve01 systemd[1]: apt-daily.timer: Adding 8h 36min 11.274821s random time. Jul 1 14:44:13 vmpve01 pvedaemon[14464]: successful auth for user 'NOC at pve' Jul 1 14:44:54 vmpve01 pvedaemon[32611]: successful auth for user 'NOC at pve' Jul 1 14:45:00 vmpve01 systemd[1]: Starting Proxmox VE replication runner... Jul 1 14:45:01 vmpve01 systemd[1]: Started Proxmox VE replication runner. ****NODE REBOOTED**** Jul 1 15:45:38 vmpve01 systemd[1]: Started Load/Save Random Seed. Jul 1 15:45:38 vmpve01 systemd[1]: Started Flush Journal to Persistent Storage. Jul 1 15:45:38 vmpve01 systemd-modules-load[408]: Inserted module 'iscsi_tcp' Jul 1 15:45:38 vmpve01 systemd[1]: Mounted RPC Pipe File System. Jul 1 15:45:38 vmpve01 systemd[1]: Started udev Coldplug all Devices. ...... *****All normal logging cluster starting up then I start getting these re-occuring****** Jul 1 15:03:19 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vm pve01/local-vmstor01: -1 Jul 1 15:03:19 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vm pve01/local: -1 Jul 1 15:03:19 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vm pve01/local-lvm: -1 Jul 1 15:03:23 vmpve01 rrdcached[2114]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-node/vmpv e01) failed with status -1. (/var/lib/rrdcached/db/pve2-node/vmpve01: illegal attempt to update using time 156 2010251 when last update time is 1562011399 (minimum one second step)) Jul 1 15:03:23 vmpve01 systemd[1]: Starting Cleanup of Temporary Directories... Jul 1 15:03:23 vmpve01 systemd[1]: Started Cleanup of Temporary Directories. Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/vmpve 01: -1 Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/105: -1 Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/104: -1 Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/106: -1 Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/102: -1 Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/103: -1 Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/107: -1 Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1 Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vmpve01/local-lvm: -1 Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vmpve01/local: -1 Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vmpve01/local-vmstor01: -1 Jul 1 15:03:37 vmpve01 pvedaemon[2366]: starting task UPID:vmpve01:00001600:00016749:5D1A6719:hastart:107:root at pam: Jul 1 15:03:37 vmpve01 pmxcfs[2185]: [main] notice: ignore duplicate Jul 1 15:03:38 vmpve01 pvedaemon[2366]: end task UPID:vmpve01:00001600:00016749:5D1A6719:hastart:107:root at pam: OK Jul 1 15:03:38 vmpve01 pmxcfs[2185]: [main] notice: ignore duplicate Jul 1 15:03:39 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/vmpve01: -1 After system time caught up with actual current time, the RRDC errors when away. Unless I missed it, there was no log indication of attempting to restart the VMs. I manually tried to start the VMs but none on node 1 would start up. After system time synchronized with hardware clock, I rebooted the node again and could start all the VMs on node 1. After manual reboot, I waited about 10 minutes, time was in sync OK, but Cluster still did not auto start or HA start any of node 1 VMs on node 4. Cluster is normal at this time: root at vmpve01:/var/log# pvec pveceph pvecm root at vmpve01:/var/log# pvecm status Quorum information ------------------ Date: Mon Jul 1 21:12:29 2019 Quorum provider: corosync_votequorum Nodes: 4 Node ID: 0x00000001 Ring ID: 1/80 Quorate: Yes Votequorum information ---------------------- Expected votes: 4 Highest expected: 4 Total votes: 4 Quorum: 3 Flags: Quorate Membership information ---------------------- Nodeid Votes Name 0x00000001 1 10.99.96.191 (local) 0x00000002 1 10.99.96.192 0x00000003 1 10.99.96.193 0x00000004 1 10.99.96.194 I was just about to roll into proxmox updates this week too. Any suggestions on diagnosing root cause, not of node reboot, but of why VMs would not HA start when the node came up an hour in the future? Thanks. JR JR Richardson Engineering for the Masses Chasing the Azeotrope From herve.ballans at ias.u-psud.fr Tue Jul 2 14:43:55 2019 From: herve.ballans at ias.u-psud.fr (=?UTF-8?Q?Herv=c3=a9_Ballans?=) Date: Tue, 2 Jul 2019 14:43:55 +0200 Subject: [PVE-User] Shared same rbd disk on 2 Vms Message-ID: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> Dear list, Sorry if the question has already been asked... Context: ProxmoxVE 5.4-6 / Ceph Luminous 12.2.12 On a Debian VM, I have two disks : a rbd system disk and an additional rbd data disk. Both disks are formated on ext4. I would like to know if I can use the data disk on another VM (with rw rights) ? I'm pretty sure that Ceph RBD is compliant with this scenario but I have a doubt with the usage of ext4 partition ? Thanks in advance for your feedback, Herv? From elacunza at binovo.es Tue Jul 2 14:46:43 2019 From: elacunza at binovo.es (Eneko Lacunza) Date: Tue, 2 Jul 2019 14:46:43 +0200 Subject: [PVE-User] Shared same rbd disk on 2 Vms In-Reply-To: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> Message-ID: <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> You need a cluster file system to be able to do this (gfs for example). ext4 can't be mounted by two systems at the same time. https://en.wikipedia.org/wiki/GFS2 Maybe you can consider using NFS instead... Cheers El 2/7/19 a las 14:43, Herv? Ballans escribi?: > Dear list, > > Sorry if the question has already been asked... > > Context: ProxmoxVE 5.4-6 / Ceph Luminous 12.2.12 > > On a Debian VM, I have two disks : a rbd system disk and an additional > rbd data disk. Both disks are formated on ext4. > > I would like to know if I can use the data disk on another VM (with rw > rights) ? > > I'm pretty sure that Ceph RBD is compliant with this scenario but I > have a doubt with the usage of ext4 partition ? > > Thanks in advance for your feedback, > > Herv? > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From herve.ballans at ias.u-psud.fr Tue Jul 2 15:26:44 2019 From: herve.ballans at ias.u-psud.fr (=?UTF-8?Q?Herv=c3=a9_Ballans?=) Date: Tue, 2 Jul 2019 15:26:44 +0200 Subject: [PVE-User] Shared same rbd disk on 2 Vms In-Reply-To: <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> Message-ID: Thanks for your reply Eneko. Indeed, it makes sense that ext4 is not suitable fot this purpose. I think NFS is a good alternative! Cheers, Herv? Le 02/07/2019 ? 14:46, Eneko Lacunza a ?crit?: > You need a cluster file system to be able to do this (gfs for > example). ext4 can't be mounted by two systems at the same time. > > https://en.wikipedia.org/wiki/GFS2 > > Maybe you can consider using NFS instead... > > Cheers > > El 2/7/19 a las 14:43, Herv? Ballans escribi?: >> Dear list, >> >> Sorry if the question has already been asked... >> >> Context: ProxmoxVE 5.4-6 / Ceph Luminous 12.2.12 >> >> On a Debian VM, I have two disks : a rbd system disk and an >> additional rbd data disk. Both disks are formated on ext4. >> >> I would like to know if I can use the data disk on another VM (with >> rw rights) ? >> >> I'm pretty sure that Ceph RBD is compliant with this scenario but I >> have a doubt with the usage of ext4 partition ? >> >> Thanks in advance for your feedback, >> >> Herv? >> >> >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From tkropf at posteo.de Tue Jul 2 15:50:04 2019 From: tkropf at posteo.de (Tobias Kropf) Date: Tue, 2 Jul 2019 15:50:04 +0200 Subject: [PVE-User] Shared same rbd disk on 2 Vms In-Reply-To: References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> Message-ID: With any Network connection between the vms ... you can use DRBD on top of the rbd volumes? On 7/2/19 3:26 PM, Herv? Ballans wrote: > Thanks for your reply Eneko. Indeed, it makes sense that ext4 is not > suitable fot this purpose. > I think NFS is a good alternative! > > Cheers, > Herv? > > Le 02/07/2019 ? 14:46, Eneko Lacunza a ?crit?: >> You need a cluster file system to be able to do this (gfs for >> example). ext4 can't be mounted by two systems at the same time. >> >> https://en.wikipedia.org/wiki/GFS2 >> >> Maybe you can consider using NFS instead... >> >> Cheers >> >> El 2/7/19 a las 14:43, Herv? Ballans escribi?: >>> Dear list, >>> >>> Sorry if the question has already been asked... >>> >>> Context: ProxmoxVE 5.4-6 / Ceph Luminous 12.2.12 >>> >>> On a Debian VM, I have two disks : a rbd system disk and an >>> additional rbd data disk. Both disks are formated on ext4. >>> >>> I would like to know if I can use the data disk on another VM (with >>> rw rights) ? >>> >>> I'm pretty sure that Ceph RBD is compliant with this scenario but I >>> have a doubt with the usage of ext4 partition ? >>> >>> Thanks in advance for your feedback, >>> >>> Herv? >>> >>> >>> >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Mit freundlichen Gruessen Tobias Kropf From trojahn+proxmox at pluspol.info Tue Jul 2 19:13:38 2019 From: trojahn+proxmox at pluspol.info (Falko Trojahn) Date: Tue, 2 Jul 2019 19:13:38 +0200 Subject: [PVE-User] Shared same rbd disk on 2 Vms In-Reply-To: References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> Message-ID: Am 02.07.19 um 15:26 schrieb Herv? Ballans: > Thanks for your reply Eneko. Indeed, it makes sense that ext4 is not > suitable fot this purpose. > I think NFS is a good alternative! Or, while you have Ceph - what about using CephFS from within the VM instead of rbd disk image? Best regards, Falko From proxmox at iancoetzee.za.net Wed Jul 3 08:35:01 2019 From: proxmox at iancoetzee.za.net (Ian Coetzee) Date: Wed, 3 Jul 2019 08:35:01 +0200 Subject: [PVE-User] ceph-osd not starting after network related issues In-Reply-To: References: Message-ID: Hi All, Some feedback on my end. I managed to recover the "lost data" from one of the other OSDs. Seems like my initial summary was a bit off, in that the PG's was replicated, CEPH just wanted to confirm that the objects were still relevant. For future reference, I basically marked the OSD as lost > ceph osd lost Then the PGs went into an incomplete state After that I temporarily set an option on the OSDs to ignore the history (osd_find_best_info_ignore_history_les). Got the info from http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-March/017270.html After that CEPH was happy and started to rebalance the cluster, pheew, crisis averted. This failure did however convince me to increase our cluster size from 2:1 to 3:2. Sacrificing usable space for reliability. Now I need to give feedback on what happened, this is what I am still not sure about as SMART does not show any sector errors. I might as well start a badblocks and see if I detect anything in there. As always, I am open to other suggestion as to where to look for other clues on what went wrong. Kind regards On Mon, 1 Jul 2019 at 09:10, Ian Coetzee wrote: > Hi All, > > This morning I have a bit of a big boo-boo on our production system. > > After a very sudden network outage somewhere during the night, one of my > ceph-osd's is no longer starting up. > > If I try and start it manually, I get a very spectacular failure, see link. > > https://www.jacklin.co.za/zerobin/?04e2dcd13ab8dfc8#zKCISUvAm4o/6mnLmyu+8fSS1VumC65XaETt/dD7rn0= > > As near as I can tell, it seems to be asserting whether a file exsists, I > have yet to determine which file that would be. Any pointers are welcome, > as well as any other ideas to get the osd back. For some reason there is > data on the osd that was not replicated to my other osd's, as such I can > not just re-init this osd as some of the posts I could find suggests > > I am also going to head to the ceph ML in a bit (after I have registered) > > Kind regards > > From b.laessig at pengutronix.de Wed Jul 3 11:13:42 2019 From: b.laessig at pengutronix.de (Bjoern Laessig) Date: Wed, 03 Jul 2019 11:13:42 +0200 Subject: [PVE-User] Disable netboot on kvm-machines Message-ID: <1562145222.20320.19.camel@pengutronix.de> Hi, i am migrating some vms from a libvirt cluster to our shiny new proxmox cluster. I have a lot of virtual machines with a couple of network interfaces. Only one interface per virtual machine should be able to boot via pxelinux, the others not. How do i do that? At the moment my network interfaces are defined like this: net0: virtio=1a:1d:21:00:00:65,bridge=vmbr0,tag=2100 net1: virtio=1a:1d:21:01:00:65,bridge=vmbr0,tag=2304 net9: virtio=1a:1d:21:09:00:65,bridge=vmbr0,tag=2202 Only net9 should be able to boot from network. regards, Bj?rn L?ssig From herve.ballans at ias.u-psud.fr Wed Jul 3 14:20:09 2019 From: herve.ballans at ias.u-psud.fr (=?UTF-8?Q?Herv=c3=a9_Ballans?=) Date: Wed, 3 Jul 2019 14:20:09 +0200 Subject: [PVE-User] Shared same rbd disk on 2 Vms In-Reply-To: References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> Message-ID: <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> Le 02/07/2019 ? 15:50, Tobias Kropf a ?crit?: > With any Network connection between the vms ... you can use DRBD on top of the rbd volumes? Thanks Tobias for your reply. From what I understood in the past on drbd, it requires 2 disks to be synchronized between them (sort of network RAID 1 ?). Now I want exactly the opposite, a single disk that I can use on 2 machines... rv > > On 7/2/19 3:26 PM, Herv? Ballans wrote: >> Thanks for your reply Eneko. Indeed, it makes sense that ext4 is not >> suitable fot this purpose. >> I think NFS is a good alternative! >> >> Cheers, >> Herv? >> >> Le 02/07/2019 ? 14:46, Eneko Lacunza a ?crit?: >>> You need a cluster file system to be able to do this (gfs for >>> example). ext4 can't be mounted by two systems at the same time. >>> >>> https://en.wikipedia.org/wiki/GFS2 >>> >>> Maybe you can consider using NFS instead... >>> >>> Cheers >>> >>> El 2/7/19 a las 14:43, Herv? Ballans escribi?: >>>> Dear list, >>>> >>>> Sorry if the question has already been asked... >>>> >>>> Context: ProxmoxVE 5.4-6 / Ceph Luminous 12.2.12 >>>> >>>> On a Debian VM, I have two disks : a rbd system disk and an >>>> additional rbd data disk. Both disks are formated on ext4. >>>> >>>> I would like to know if I can use the data disk on another VM (with >>>> rw rights) ? >>>> >>>> I'm pretty sure that Ceph RBD is compliant with this scenario but I >>>> have a doubt with the usage of ext4 partition ? >>>> >>>> Thanks in advance for your feedback, >>>> >>>> Herv? From herve.ballans at ias.u-psud.fr Wed Jul 3 14:24:35 2019 From: herve.ballans at ias.u-psud.fr (=?UTF-8?Q?Herv=c3=a9_Ballans?=) Date: Wed, 3 Jul 2019 14:24:35 +0200 Subject: [PVE-User] Shared same rbd disk on 2 Vms In-Reply-To: References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> Message-ID: Thanks Falko. I was thinking about CephFS. But I don't want to add MDS on my ProxmoxVE cluster as this can add complexity to a very stable configuration at this time with only Ceph in block mode ! rv Le 02/07/2019 ? 19:13, Falko Trojahn a ?crit?: > Am 02.07.19 um 15:26 schrieb Herv? Ballans: >> Thanks for your reply Eneko. Indeed, it makes sense that ext4 is not >> suitable fot this purpose. >> I think NFS is a good alternative! > Or, while you have Ceph - what about using CephFS from within the VM > instead of rbd disk image? > > > Best regards, > Falko From gaio at sv.lnf.it Thu Jul 4 10:58:01 2019 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Thu, 4 Jul 2019 10:58:01 +0200 Subject: [PVE-User] Trouble with Win2016 server... In-Reply-To: <20190628130527.GG5544@sv.lnf.it> References: <20190620100814.GM6406@sv.lnf.it> <20190620130958.GT6406@sv.lnf.it> <20190621084531.GD2711@sv.lnf.it> <20190628130527.GG5544@sv.lnf.it> Message-ID: <20190704085801.GC2657@sv.lnf.it> > but nothing seems have changed, still i got errors on %windir%\security\database\secedit.sdb. Ok, still this is a bit mysterious, but at least have some more info. I've simply disabled 'discard' on the bootdisk, now the VM are: agent: 1 boot: dc bootdisk: scsi0 cores: 2 ide0: none,media=cdrom memory: 8192 name: vwssv1 net0: virtio=DA:CA:FA:50:8C:01,bridge=vmbr0 numa: 0 ostype: win10 scsi0: VM1:vm-124-disk-0,size=100G scsi1: DATA2:vm-124-disk-0,backup=0,discard=on,size=500G scsihw: virtio-scsi-pci smbios1: uuid=169d7e42-8a69-4f96-aeb1-df604af44bd1 sockets: 1 and windows recreate correctly '%windir%\security\database\secedit.sdb' file and stop complain at all. Still i had some more DB corrupted (for 'Administrator' user) but i've removed the profile and let windows recreate it from scratch, and clearly also this error deasppeard. So, seems there's some trouble with 'discard', at least in my setup... Still i've the 'scratch' VM i used for test, so if can be useful, i can provide some more info... Say me... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From gaio at sv.lnf.it Thu Jul 4 12:35:44 2019 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Thu, 4 Jul 2019 12:35:44 +0200 Subject: [PVE-User] Cluster does not start, corosync timeout... Message-ID: <20190704103544.GF2657@sv.lnf.it> We had a major power outgage here, and our cluster have some trouble on restart. The worster was: Jul 3 19:58:40 pvecn1 corosync[3443]: [MAIN ] Corosync Cluster Engine ('2.4.4-dirty'): started and ready to provide service. Jul 3 19:58:40 pvecn1 corosync[3443]: [MAIN ] Corosync built-in features: dbus rdma monitoring watchdog systemd xmlconf qdevices qnetd snmp pie relro bindnow Jul 3 19:58:40 pvecn1 corosync[3443]: notice [MAIN ] Corosync Cluster Engine ('2.4.4-dirty'): started and ready to provide service. Jul 3 19:58:40 pvecn1 corosync[3443]: info [MAIN ] Corosync built-in features: dbus rdma monitoring watchdog systemd xmlconf qdevices qnetd snmp pie relro bindnow Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Start operation timed out. Terminating. Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Unit entered failed state. Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Failed with result 'timeout'. after fiddling a bit, we solved all the stuff and cluster goes back as normal. But... some host in the cluster missed from /etc/hosts: this suffices to have corosync not to start correctly? Looking at docs (https://pve.proxmox.com/pve-docs/pve-admin-guide.html): While it?s often common use to reference all other nodenames in /etc/hosts with their IP this is not strictly necessary for a cluster, which normally uses multicast, to work. It maybe useful as you then can connect from one node to the other with SSH through the easier to remember node name. this mean i've not multicast correctly working? I was sure i had... Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From t.lamprecht at proxmox.com Thu Jul 4 13:19:34 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Thu, 4 Jul 2019 13:19:34 +0200 Subject: [PVE-User] Cluster does not start, corosync timeout... In-Reply-To: <20190704103544.GF2657@sv.lnf.it> References: <20190704103544.GF2657@sv.lnf.it> Message-ID: <6624aad9-c46d-f125-eb70-c6d4fd9e7c8f@proxmox.com> On 7/4/19 12:35 PM, Marco Gaiarin wrote: > We had a major power outgage here, and our cluster have some trouble on > restart. The worster was: > > Jul 3 19:58:40 pvecn1 corosync[3443]: [MAIN ] Corosync Cluster Engine ('2.4.4-dirty'): started and ready to provide service. > Jul 3 19:58:40 pvecn1 corosync[3443]: [MAIN ] Corosync built-in features: dbus rdma monitoring watchdog systemd xmlconf qdevices qnetd snmp pie relro bindnow > Jul 3 19:58:40 pvecn1 corosync[3443]: notice [MAIN ] Corosync Cluster Engine ('2.4.4-dirty'): started and ready to provide service. > Jul 3 19:58:40 pvecn1 corosync[3443]: info [MAIN ] Corosync built-in features: dbus rdma monitoring watchdog systemd xmlconf qdevices qnetd snmp pie relro bindnow > Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Start operation timed out. Terminating. > Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Unit entered failed state. Hmm, that's strange, do you have the full log between "19:58:40" and "20:00:09", as normally there should be some more info, at least for corosync and pve-cluster, e.g., the following output would be great: journalctl -u corosync -u pve-cluster --since "2019-07-03 19:58:40" --until "2019-07-03 20:00:09" > > But... some host in the cluster missed from /etc/hosts: this suffices > to have corosync not to start correctly? > depends on the config, as you stated yourself with multicast it normally won't be an issue, but maybe the switch had some issues with multicast initially after the power outage, as a guess. > > Looking at docs (https://pve.proxmox.com/pve-docs/pve-admin-guide.html): > > While it?s often common use to reference all other nodenames in /etc/hosts with their IP this is not strictly necessary for a cluster, which normally uses multicast, to work. It maybe useful as you then can connect from one node to the other with SSH through the easier to remember node name. > > this mean i've not multicast correctly working? I was sure i had... can you please post your corosync.conf ? From gaio at sv.lnf.it Thu Jul 4 14:23:45 2019 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Thu, 4 Jul 2019 14:23:45 +0200 Subject: [PVE-User] Cluster does not start, corosync timeout... In-Reply-To: <6624aad9-c46d-f125-eb70-c6d4fd9e7c8f@proxmox.com> References: <20190704103544.GF2657@sv.lnf.it> <6624aad9-c46d-f125-eb70-c6d4fd9e7c8f@proxmox.com> Message-ID: <20190704122345.GH2657@sv.lnf.it> Mandi! Thomas Lamprecht In chel di` si favelave... > Hmm, that's strange, do you have the full log between "19:58:40" and > "20:00:09", as normally there should be some more info, at least for > corosync and pve-cluster, e.g., the following output would be great: > journalctl -u corosync -u pve-cluster --since "2019-07-03 19:58:40" --until "2019-07-03 20:00:09" Just rotated: root at pvecn1:~# journalctl -u corosync -u pve-cluster --since "2019-07-03 19:58:40" --until "2019-07-03 20:00:09" -- Logs begin at Wed 2019-07-03 21:03:31 CEST, end at Thu 2019-07-04 14:12:38 CEST. -- looking at syslog.1: Jul 3 19:58:40 pvecn1 corosync[3443]: [MAIN ] Corosync Cluster Engine ('2.4.4-dirty'): started and ready to provide service. Jul 3 19:58:40 pvecn1 corosync[3443]: [MAIN ] Corosync built-in features: dbus rdma monitoring watchdog systemd xmlconf qdevices qnetd snmp pie relro bindnow Jul 3 19:58:40 pvecn1 corosync[3443]: notice [MAIN ] Corosync Cluster Engine ('2.4.4-dirty'): started and ready to provide service. Jul 3 19:58:40 pvecn1 corosync[3443]: info [MAIN ] Corosync built-in features: dbus rdma monitoring watchdog systemd xmlconf qdevices qnetd snmp pie relro bindnow Jul 3 19:58:41 pvecn1 pve-firewall[3491]: starting server Jul 3 19:58:41 pvecn1 pvestatd[3503]: starting server Jul 3 19:58:41 pvecn1 systemd[1]: Started Proxmox VE firewall. Jul 3 19:58:41 pvecn1 systemd[1]: Started PVE Status Daemon. Jul 3 19:58:41 pvecn1 kernel: [ 36.327130] ip6_tables: (C) 2000-2006 Netfilter Core Team Jul 3 19:58:41 pvecn1 kernel: [ 36.464756] ip_set: protocol 6 Jul 3 19:58:42 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:58:42 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:58:42 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:58:42 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:58:44 pvecn1 hpasmlited[1740]: hpDeferSPDThread: Starting thread to collect DIMM SPD Data. Jul 3 19:58:44 pvecn1 hpasmlited[1740]: Initialize data structures successful Jul 3 19:58:48 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:58:48 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:58:48 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:58:48 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:58:49 pvecn1 multipathd[913]: zd0: triggering change event to reinitialize Jul 3 19:58:49 pvecn1 multipathd[913]: zd0: add path (uevent) Jul 3 19:58:49 pvecn1 multipathd[913]: zd0: spurious uevent, path already in pathvec Jul 3 19:58:49 pvecn1 multipathd[913]: zd0: failed to get udev uid: Invalid argument Jul 3 19:58:49 pvecn1 multipathd[913]: zd0: failed to get sysfs uid: Invalid argument Jul 3 19:58:49 pvecn1 multipathd[913]: zd0: failed to get sgio uid: Inappropriate ioctl for device Jul 3 19:58:49 pvecn1 multipathd[913]: zd0: failed to get path uid Jul 3 19:58:49 pvecn1 multipathd[913]: uevent trigger error Jul 3 19:58:52 pvecn1 pvestatd[3503]: storage 'Backup' is not online Jul 3 19:58:54 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:58:54 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:58:54 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:58:54 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:59:00 pvecn1 systemd[1]: Starting Proxmox VE replication runner... Jul 3 19:59:00 pvecn1 pvesr[3641]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 19:59:00 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:59:00 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:59:00 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:59:00 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:59:01 pvecn1 pvestatd[3503]: storage 'Backup' is not online Jul 3 19:59:01 pvecn1 pvesr[3641]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 19:59:02 pvecn1 pvesr[3641]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 19:59:03 pvecn1 hpasmlited[1740]: hpDeferSPDThread: End of Collecting DIMM SPD data. Jul 3 19:59:03 pvecn1 pvesr[3641]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 19:59:04 pvecn1 pvesr[3641]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 19:59:05 pvecn1 pvesr[3641]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 19:59:06 pvecn1 pvesr[3641]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 19:59:06 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:59:06 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:59:06 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:59:06 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:59:07 pvecn1 pvesr[3641]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 19:59:08 pvecn1 pvesr[3641]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 19:59:09 pvecn1 pvesr[3641]: error with cfs lock 'file-replication_cfg': no quorum! Jul 3 19:59:09 pvecn1 systemd[1]: pvesr.service: Main process exited, code=exited, status=13/n/a Jul 3 19:59:09 pvecn1 systemd[1]: Failed to start Proxmox VE replication runner. Jul 3 19:59:09 pvecn1 systemd[1]: pvesr.service: Unit entered failed state. Jul 3 19:59:09 pvecn1 systemd[1]: pvesr.service: Failed with result 'exit-code'. Jul 3 19:59:11 pvecn1 pvestatd[3503]: storage 'Backup' is not online Jul 3 19:59:12 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:59:12 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:59:12 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:59:12 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:59:18 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:59:18 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:59:18 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:59:18 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:59:21 pvecn1 pvestatd[3503]: storage 'Backup' is not online Jul 3 19:59:24 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:59:24 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:59:24 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:59:24 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:59:30 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:59:30 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:59:30 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:59:30 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:59:31 pvecn1 pvestatd[3503]: storage 'Backup' is not online Jul 3 19:59:36 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:59:36 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:59:36 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:59:36 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:59:41 pvecn1 pvestatd[3503]: storage 'Backup' is not online Jul 3 19:59:42 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:59:42 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:59:42 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:59:42 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:59:48 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:59:48 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:59:48 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:59:48 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 19:59:51 pvecn1 pvestatd[3503]: storage 'Backup' is not online Jul 3 19:59:54 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 19:59:54 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 19:59:54 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 19:59:54 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 20:00:00 pvecn1 systemd[1]: Starting Proxmox VE replication runner... Jul 3 20:00:00 pvecn1 pvesr[4068]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 20:00:00 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 20:00:00 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 20:00:00 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 20:00:00 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 20:00:01 pvecn1 pvesr[4068]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 20:00:01 pvecn1 pvestatd[3503]: storage 'Backup' is not online Jul 3 20:00:02 pvecn1 pvesr[4068]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 20:00:03 pvecn1 pvesr[4068]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 20:00:04 pvecn1 pvesr[4068]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 20:00:05 pvecn1 pvesr[4068]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 20:00:06 pvecn1 pvesr[4068]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 20:00:06 pvecn1 pmxcfs[3322]: [quorum] crit: quorum_initialize failed: 2 Jul 3 20:00:06 pvecn1 pmxcfs[3322]: [confdb] crit: cmap_initialize failed: 2 Jul 3 20:00:06 pvecn1 pmxcfs[3322]: [dcdb] crit: cpg_initialize failed: 2 Jul 3 20:00:06 pvecn1 pmxcfs[3322]: [status] crit: cpg_initialize failed: 2 Jul 3 20:00:07 pvecn1 pvesr[4068]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 20:00:08 pvecn1 pvesr[4068]: trying to acquire cfs lock 'file-replication_cfg' ... Jul 3 20:00:09 pvecn1 pvesr[4068]: error with cfs lock 'file-replication_cfg': no quorum! Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Start operation timed out. Terminating. Jul 3 20:00:09 pvecn1 systemd[1]: pvesr.service: Main process exited, code=exited, status=13/n/a Jul 3 20:00:09 pvecn1 systemd[1]: Failed to start Proxmox VE replication runner. Jul 3 20:00:09 pvecn1 systemd[1]: pvesr.service: Unit entered failed state. Jul 3 20:00:09 pvecn1 systemd[1]: pvesr.service: Failed with result 'exit-code'. Jul 3 20:00:09 pvecn1 systemd[1]: Failed to start Corosync Cluster Engine. Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Unit entered failed state. Jul 3 20:00:09 pvecn1 systemd[1]: corosync.service: Failed with result 'timeout'. Jul 3 20:00:09 pvecn1 systemd[1]: Starting PVE API Daemon... Note that i'm not using pvesr, so all the warning about it can be safely ignored. Also, 'Backup' storage is a NFS storage in one of the node, probably still booting... > > But... some host in the cluster missed from /etc/hosts: this suffices > > to have corosync not to start correctly? > depends on the config, as you stated yourself with multicast it normally > won't be an issue, but maybe the switch had some issues with multicast initially > after the power outage, as a guess. I've tried to check multipath with 'omping' now (and i'm sure i've had checked setting up the claster), and works. So i'm not sure how multicast can 'not work initially', and subsequent start to work... > can you please post your corosync.conf ? Sure! root at pvecn1:~# cat /etc/pve/corosync.conf logging { debug: off to_syslog: yes } nodelist { node { name: pvecn2 nodeid: 2 quorum_votes: 1 ring0_addr: pvecn2 } node { name: pvecn1 nodeid: 1 quorum_votes: 1 ring0_addr: pvecn1 } node { name: pvecn3 nodeid: 3 quorum_votes: 1 ring0_addr: pvecn3 } } quorum { provider: corosync_votequorum } totem { cluster_name: CONEGLIANO config_version: 3 ip_version: ipv4 secauth: on version: 2 interface { bindnetaddr: 10.10.1.50 ringnumber: 0 } } Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From cj0nes at live.com Thu Jul 4 20:45:13 2019 From: cj0nes at live.com (Craig Jones) Date: Thu, 4 Jul 2019 18:45:13 +0000 Subject: [PVE-User] Host Rebooting when using PCIe Passthrough Message-ID: Hello, I have a VM that I'm passing a GPU through to. The passthrough itself works great. The issue is that whenever this VM is powered on, the host will reboot without any interaction from me. The reboot happens anywhere from 3 - 15 minutes after the VM has been powered on. I have many other VMs that don't cause this. The only difference between them and this one is the passthrough GPU. Attached are some potentially helpful outputs. The syslogs have been truncated from when the VM had been powered on to the last entry right before the host rebooted. Thanks, Craig --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cat etc_pve_qemu-server_101.conf.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: syslog.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cat etc_default_grub.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cat etc_modprobe.d_blacklist.conf.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: cat etc_modprobe.d_vfio.conf.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: find sys_kernel_iommu_groups -type l.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: lspci.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: lspci -n -s 01_00.txt URL: From martin at proxmox.com Thu Jul 4 21:06:25 2019 From: martin at proxmox.com (Martin Maurer) Date: Thu, 4 Jul 2019 21:06:25 +0200 Subject: [PVE-User] Proxmox VE 6.0 beta released! Message-ID: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> Hi all! We're happy to announce the first beta release for the Proxmox VE 6.x family! It's based on the great Debian Buster (Debian 10) and a 5.0 kernel, QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and countless improvements and bugfixes. The new installer supports ZFS root via UEFI, for example you can boot a ZFS mirror on NVMe SSDs (using systemd-boot instead of grub). The full release notes will be available together with the final release announcement. For more details, see: https://forum.proxmox.com/threads/proxmox-ve-6-0-beta-released.55670/ -- Best Regards, Martin Maurer martin at proxmox.com https://www.proxmox.com From jmr.richardson at gmail.com Thu Jul 4 21:37:36 2019 From: jmr.richardson at gmail.com (JR Richardson) Date: Thu, 4 Jul 2019 14:37:36 -0500 Subject: [PVE-User] Cluster Host Node Rebooted and VMs Did not restart? In-Reply-To: <000601d5307c$a417aa70$ec46ff50$@gmail.com> References: <000601d5307c$a417aa70$ec46ff50$@gmail.com> Message-ID: <5AD6B8FB-C520-4236-A3A0-577104A85AD7@gmail.com> Sorry for posting twice, not sure I used the correct mail list. > > Hey Folks, > > Ran into an interesting problem today. Topology is a 4-Node Cluster on v.5.2, been up and running without issues for 10 months or so. Several VMs running on node 1, 2 & 3, maybe 30% loaded. I keep node 4 empty for HA and new build VM testing. All VMs are set for HA and auto start on node reboot. > > Node 1 rebooted, no indication why, nothing in the logs identifying cause of the node rebooting. When node 1 came back up it was 1 hour ahead of time and none of the VMs would start and HA did not start up the VMs on empty node 4. > > Syslog: > Jul 1 14:42:00 vmpve01 systemd[1]: Starting Proxmox VE replication runner... > Jul 1 14:42:01 vmpve01 systemd[1]: Started Proxmox VE replication runner. > Jul 1 14:42:20 vmpve01 pvedaemon[18577]: successful auth for user 'NOC at pve' > Jul 1 14:43:00 vmpve01 systemd[1]: Starting Proxmox VE replication runner... > Jul 1 14:43:00 vmpve01 pvedaemon[18577]: successful auth for user 'NOC at pve' > Jul 1 14:43:01 vmpve01 systemd[1]: Started Proxmox VE replication runner. > Jul 1 14:43:40 vmpve01 pvedaemon[32611]: successful auth for user 'NOC at pve' > Jul 1 14:43:50 vmpve01 pveproxy[9369]: worker exit > Jul 1 14:43:50 vmpve01 pveproxy[2395]: worker 9369 finished > Jul 1 14:43:50 vmpve01 pveproxy[2395]: starting 1 worker(s) > Jul 1 14:43:50 vmpve01 pveproxy[2395]: worker 21373 started > Jul 1 14:44:00 vmpve01 systemd[1]: Starting Proxmox VE replication runner... > Jul 1 14:44:01 vmpve01 systemd[1]: Started Proxmox VE replication runner. > Jul 1 14:44:03 vmpve01 systemd[1]: Time has been changed > Jul 1 14:44:03 vmpve01 systemd[1]: apt-daily-upgrade.timer: Adding 51min 27.103008s random time. > Jul 1 14:44:03 vmpve01 systemd[1]: pve-daily-update.timer: Adding 49min 46.847880s random time. > Jul 1 14:44:03 vmpve01 systemd[1]: apt-daily.timer: Adding 8h 36min 11.274821s random time. > Jul 1 14:44:13 vmpve01 pvedaemon[14464]: successful auth for user 'NOC at pve' > Jul 1 14:44:54 vmpve01 pvedaemon[32611]: successful auth for user 'NOC at pve' > Jul 1 14:45:00 vmpve01 systemd[1]: Starting Proxmox VE replication runner... > Jul 1 14:45:01 vmpve01 systemd[1]: Started Proxmox VE replication runner. > ****NODE REBOOTED**** > Jul 1 15:45:38 vmpve01 systemd[1]: Started Load/Save Random Seed. > Jul 1 15:45:38 vmpve01 systemd[1]: Started Flush Journal to Persistent Storage. > Jul 1 15:45:38 vmpve01 systemd-modules-load[408]: Inserted module 'iscsi_tcp' > Jul 1 15:45:38 vmpve01 systemd[1]: Mounted RPC Pipe File System. > Jul 1 15:45:38 vmpve01 systemd[1]: Started udev Coldplug all Devices. > ????.. > *****All normal logging cluster starting up then I start getting these re-occuring****** > Jul 1 15:03:19 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vm > pve01/local-vmstor01: -1 > Jul 1 15:03:19 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vm > pve01/local: -1 > Jul 1 15:03:19 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vm > pve01/local-lvm: -1 > Jul 1 15:03:23 vmpve01 rrdcached[2114]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-node/vmpv > e01) failed with status -1. (/var/lib/rrdcached/db/pve2-node/vmpve01: illegal attempt to update using time 156 > 2010251 when last update time is 1562011399 (minimum one second step)) > Jul 1 15:03:23 vmpve01 systemd[1]: Starting Cleanup of Temporary Directories... > Jul 1 15:03:23 vmpve01 systemd[1]: Started Cleanup of Temporary Directories. > Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/vmpve > 01: -1 > Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/105: -1 > Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/104: -1 > Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/106: -1 > Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/102: -1 > Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/103: -1 > Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/107: -1 > Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/101: -1 > Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vmpve01/local-lvm: -1 > Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vmpve01/local: -1 > Jul 1 15:03:29 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/vmpve01/local-vmstor01: -1 > Jul 1 15:03:37 vmpve01 pvedaemon[2366]: starting task UPID:vmpve01:00001600:00016749:5D1A6719:hastart:107:root at pam: > Jul 1 15:03:37 vmpve01 pmxcfs[2185]: [main] notice: ignore duplicate > Jul 1 15:03:38 vmpve01 pvedaemon[2366]: end task UPID:vmpve01:00001600:00016749:5D1A6719:hastart:107:root at pam: OK > Jul 1 15:03:38 vmpve01 pmxcfs[2185]: [main] notice: ignore duplicate > Jul 1 15:03:39 vmpve01 pmxcfs[2185]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/vmpve01: -1 > > After system time caught up with actual current time, the RRDC errors when away. Unless I missed it, there was no log indication of attempting to restart the VMs. I manually tried to start the VMs but none on node 1 would start up. After system time synchronized with hardware clock, I rebooted the node again and could start all the VMs on node 1. After manual reboot, I waited about 10 minutes, time was in sync OK, but Cluster still did not auto start or HA start any of node 1 VMs on node 4. > > Cluster is normal at this time: > > root at vmpve01:/var/log# pvec > pveceph pvecm > root at vmpve01:/var/log# pvecm status > Quorum information > ------------------ > Date: Mon Jul 1 21:12:29 2019 > Quorum provider: corosync_votequorum > Nodes: 4 > Node ID: 0x00000001 > Ring ID: 1/80 > Quorate: Yes > > Votequorum information > ---------------------- > Expected votes: 4 > Highest expected: 4 > Total votes: 4 > Quorum: 3 > Flags: Quorate > > Membership information > ---------------------- > Nodeid Votes Name > 0x00000001 1 10.99.96.191 (local) > 0x00000002 1 10.99.96.192 > 0x00000003 1 10.99.96.193 > 0x00000004 1 10.99.96.194 > > I was just about to roll into proxmox updates this week too. Any suggestions on diagnosing root cause, not of node reboot, but of why VMs would not HA start when the node came up an hour in the future? > > Thanks. > > JR > > JR Richardson > Engineering for the Masses > Chasing the Azeotrope > From gilberto.nunes32 at gmail.com Thu Jul 4 21:39:06 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Thu, 4 Jul 2019 16:39:06 -0300 Subject: [PVE-User] [pve-devel] Proxmox VE 6.0 beta released! In-Reply-To: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> Message-ID: Good job! Is there something about qemu mc?? This kind of thing has been add to qemu mainstream tree or devel tree?? --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qui, 4 de jul de 2019 ?s 16:06, Martin Maurer escreveu: > Hi all! > > We're happy to announce the first beta release for the Proxmox VE 6.x > family! It's based on the great Debian Buster (Debian 10) and a 5.0 kernel, > QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and countless improvements > and bugfixes. The new installer supports ZFS root via UEFI, for example you > can boot a ZFS mirror on NVMe SSDs (using systemd-boot instead of grub). > The full release notes will be available together with the final release > announcement. > > For more details, see: > https://forum.proxmox.com/threads/proxmox-ve-6-0-beta-released.55670/ > > -- > Best Regards, > > Martin Maurer > > martin at proxmox.com > https://www.proxmox.com > > _______________________________________________ > pve-devel mailing list > pve-devel at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel > From proxmox at iancoetzee.za.net Fri Jul 5 08:08:27 2019 From: proxmox at iancoetzee.za.net (Ian Coetzee) Date: Fri, 5 Jul 2019 08:08:27 +0200 Subject: [PVE-User] Proxmox VE 6.0 beta released! In-Reply-To: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> Message-ID: Thank you for the hard work. Such an exciting project. On Thu, 4 Jul 2019 at 21:06, Martin Maurer wrote: > Hi all! > > We're happy to announce the first beta release for the Proxmox VE 6.x > family! It's based on the great Debian Buster (Debian 10) and a 5.0 kernel, > QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and countless improvements > and bugfixes. The new installer supports ZFS root via UEFI, for example you > can boot a ZFS mirror on NVMe SSDs (using systemd-boot instead of grub). > The full release notes will be available together with the final release > announcement. > > For more details, see: > https://forum.proxmox.com/threads/proxmox-ve-6-0-beta-released.55670/ > > -- > Best Regards, > > Martin Maurer > > martin at proxmox.com > https://www.proxmox.com > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From d.csapak at proxmox.com Fri Jul 5 08:23:15 2019 From: d.csapak at proxmox.com (Dominik Csapak) Date: Fri, 5 Jul 2019 08:23:15 +0200 Subject: [PVE-User] Host Rebooting when using PCIe Passthrough In-Reply-To: References: Message-ID: <70c519ac-d73d-1dbd-9a3c-1c3ae5fc89e4@proxmox.com> On 7/4/19 8:45 PM, Craig Jones wrote: > Hello, > > I have a VM that I'm passing a GPU through to. The passthrough itself > works great. The issue is that whenever this VM is powered on, the host > will reboot without any interaction from me. The reboot happens anywhere > from 3 - 15 minutes after the VM has been powered on. I have many other > VMs that don't cause this. The only difference between them and this one > is the passthrough GPU. Attached are some potentially helpful outputs. > The syslogs have been truncated from when the VM had been powered on to > the last entry right before the host rebooted. > > Thanks, > Craig > > one thing you could do is setup kernel crash logging (kdump) to see if the kernel crashes and why aside from that the only thing i see is that your gpu is not in an isolated iommu group: ----8<---- /sys/kernel/iommu_groups/1/devices/0000:00:01.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.1 /sys/kernel/iommu_groups/1/devices/0000:01:00.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.1 /sys/kernel/iommu_groups/1/devices/0000:02:00.0 ---->8---- 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV770 [Radeon HD 4870] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07) it seems it is together in a group with your nic this can be the cause for the crashes... hope this helps From alain.pean at c2n.upsaclay.fr Fri Jul 5 09:10:09 2019 From: alain.pean at c2n.upsaclay.fr (=?UTF-8?Q?Alain_P=c3=a9an?=) Date: Fri, 5 Jul 2019 09:10:09 +0200 Subject: [PVE-User] Proxmox VE 6.0 beta released! In-Reply-To: References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> Message-ID: It seems that the upgrade from 5.4 to 6.0 will be a hard way. New version of Debian, and major version of corosync, 3.0... Alain Le 05/07/2019 ? 08:08, Ian Coetzee a ?crit?: > Thank you for the hard work. > > Such an exciting project. > > On Thu, 4 Jul 2019 at 21:06, Martin Maurer wrote: > >> Hi all! >> >> We're happy to announce the first beta release for the Proxmox VE 6.x >> family! It's based on the great Debian Buster (Debian 10) and a 5.0 kernel, >> QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and countless improvements >> and bugfixes. The new installer supports ZFS root via UEFI, for example you >> can boot a ZFS mirror on NVMe SSDs (using systemd-boot instead of grub). >> The full release notes will be available together with the final release >> announcement. >> >> For more details, see: >> https://forum.proxmox.com/threads/proxmox-ve-6-0-beta-released.55670/ >> >> -- >> Best Regards, >> >> Martin Maurer >> >> martin at proxmox.com >> https://www.proxmox.com >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Administrateur Syst?me/R?seau C2N Centre de Nanosciences et Nanotechnologies (UMR 9001) Boulevard Thomas Gobert (ex Avenue de La Vauve), 91920 Palaiseau Tel : 01-70-27-06-88 Bureau A255 From elacunza at binovo.es Fri Jul 5 09:27:39 2019 From: elacunza at binovo.es (Eneko Lacunza) Date: Fri, 5 Jul 2019 09:27:39 +0200 Subject: [PVE-User] [pve-devel] Proxmox VE 6.0 beta released! In-Reply-To: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> Message-ID: <97129d49-718b-ab98-9955-a40cf428fc87@binovo.es> Hi Martin, Thanks a lot for your hard work, Maurer-ITans and the rest of developers... It seems that in PVE 6.0, with corosync 3.0, multicast won't be used by default? I think it could be interesting to have a PVE_6.x cluster wiki page to explain a bit the new cluster, max nodes, ... Also, thanks for taking the time to develop, test and describe a way for in-place upgrade without having to re-create the cluster, I think it would make the life easier for a lot of us... Cheers!! El 4/7/19 a las 21:06, Martin Maurer escribi?: > Hi all! > > We're happy to announce the first beta release for the Proxmox VE 6.x > family! It's based on the great Debian Buster (Debian 10) and a 5.0 > kernel, QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and countless > improvements and bugfixes. The new installer supports ZFS root via > UEFI, for example you can boot a ZFS mirror on NVMe SSDs (using > systemd-boot instead of grub). The full release notes will be > available together with the final release announcement. > > For more details, see: > https://forum.proxmox.com/threads/proxmox-ve-6-0-beta-released.55670/ > -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From lists at merit.unu.edu Fri Jul 5 09:32:53 2019 From: lists at merit.unu.edu (mj) Date: Fri, 5 Jul 2019 09:32:53 +0200 Subject: [PVE-User] [pve-devel] Proxmox VE 6.0 beta released! In-Reply-To: <97129d49-718b-ab98-9955-a40cf428fc87@binovo.es> References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> <97129d49-718b-ab98-9955-a40cf428fc87@binovo.es> Message-ID: <9657eace-9524-9481-0802-7f026b69ca77@merit.unu.edu> Hi, Looks like a great new release! Does corosync 3.0 mean that the notes on [https://pve.proxmox.com/wiki/Multicast_notes] are no longer relevant? Anything else/new to consider/check to make sure that kronosnet will work nicely? MJ On 7/5/19 9:27 AM, Eneko Lacunza wrote: > Hi Martin, > > Thanks a lot for your hard work, Maurer-ITans and the rest of developers... > > It seems that in PVE 6.0, with corosync 3.0, multicast won't be used by > default? I think it could be interesting to have a PVE_6.x cluster wiki > page to explain a bit the new cluster, max nodes, ... > > Also, thanks for taking the time to develop, test and describe a way for > in-place upgrade without having to re-create the cluster, I think it > would make the life easier for a lot of us... > > Cheers!! > > El 4/7/19 a las 21:06, Martin Maurer escribi?: >> Hi all! >> >> We're happy to announce the first beta release for the Proxmox VE 6.x >> family! It's based on the great Debian Buster (Debian 10) and a 5.0 >> kernel, QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and countless >> improvements and bugfixes. The new installer supports ZFS root via >> UEFI, for example you can boot a ZFS mirror on NVMe SSDs (using >> systemd-boot instead of grub). The full release notes will be >> available together with the final release announcement. >> >> For more details, see: >> https://forum.proxmox.com/threads/proxmox-ve-6-0-beta-released.55670/ >> > > From f.gruenbichler at proxmox.com Fri Jul 5 09:57:35 2019 From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?Q?Gr=FCnbichler?=) Date: Fri, 5 Jul 2019 09:57:35 +0200 Subject: [PVE-User] Proxmox VE 6.0 beta released! In-Reply-To: References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> Message-ID: <20190705075735.bfpc7mujyhhd3itu@nora.maurer-it.com> On Fri, Jul 05, 2019 at 09:10:09AM +0200, Alain P?an wrote: > It seems that the upgrade from 5.4 to 6.0 will be a hard way. New version of > Debian, and major version of corosync, 3.0... new (major) versions of PVE are always based on new Debian releases ;) the Corosync upgrade should be pretty pain-free, as long as you follow the instructions given in the upgrade Guide. From t.lamprecht at proxmox.com Fri Jul 5 10:34:08 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Fri, 5 Jul 2019 10:34:08 +0200 Subject: [PVE-User] [pve-devel] Proxmox VE 6.0 beta released! In-Reply-To: <9657eace-9524-9481-0802-7f026b69ca77@merit.unu.edu> References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> <97129d49-718b-ab98-9955-a40cf428fc87@binovo.es> <9657eace-9524-9481-0802-7f026b69ca77@merit.unu.edu> Message-ID: Hi, On 7/5/19 9:32 AM, mj wrote: > Looks like a great new release! > > Does corosync 3.0 mean that the notes on [https://pve.proxmox.com/wiki/Multicast_notes] are no longer relevant? We will update the documentation and wiki articles regarding this in the following days, until the final PVE 6 release it should be ready. > > Anything else/new to consider/check to make sure that kronosnet will work nicely? Ports stayed the same, communication is udp unicast per default, our firewall has now better default allow rules for the cluster networks used, so no, normally not to much special handling should be needed. Note: multicast may not be gone forever, kronosnet has some plans to add support for it, but that may need quite a bit of time, and even then we will try to keep support for the unicast kronosnet transport, if possible. cheers, Thomas > > MJ > > On 7/5/19 9:27 AM, Eneko Lacunza wrote: >> Hi Martin, >> >> Thanks a lot for your hard work, Maurer-ITans and the rest of developers... >> >> It seems that in PVE 6.0, with corosync 3.0, multicast won't be used by default? I think it could be interesting to have a PVE_6.x cluster wiki page to explain a bit the new cluster, max nodes, ... >> >> Also, thanks for taking the time to develop, test and describe a way for in-place upgrade without having to re-create the cluster, I think it would make the life easier for a lot of us... >> >> Cheers!! >> >> El 4/7/19 a las 21:06, Martin Maurer escribi?: >>> Hi all! >>> >>> We're happy to announce the first beta release for the Proxmox VE 6.x family! It's based on the great Debian Buster (Debian 10) and a 5.0 kernel, QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and countless improvements and bugfixes. The new installer supports ZFS root via UEFI, for example you can boot a ZFS mirror on NVMe SSDs (using systemd-boot instead of grub). The full release notes will be available together with the final release announcement. >>> >>> For more details, see: >>> https://forum.proxmox.com/threads/proxmox-ve-6-0-beta-released.55670/ From alain.pean at c2n.upsaclay.fr Fri Jul 5 10:42:15 2019 From: alain.pean at c2n.upsaclay.fr (=?UTF-8?Q?Alain_P=c3=a9an?=) Date: Fri, 5 Jul 2019 10:42:15 +0200 Subject: [PVE-User] Proxmox VE 6.0 beta released! In-Reply-To: <20190705075735.bfpc7mujyhhd3itu@nora.maurer-it.com> References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> <20190705075735.bfpc7mujyhhd3itu@nora.maurer-it.com> Message-ID: <645bc663-a93d-284b-dabc-bc915d054784@c2n.upsaclay.fr> Le 05/07/2019 ? 09:57, Fabian Gr?nbichler a ?crit?: > the Corosync upgrade should be pretty pain-free, as long as you follow > the instructions given in the upgrade Guide. I hope so, I remember the last upgrade from major release of corosync (3.x to 4.0 for PVE and 1.x to 2.0 for corosync if I remember correctly), was a little bit stressful, everything dissapearing, and only at the end you recover the management interface and the VMs. Fortunately, I had test it before, on a test cluster... Thanks for your answer (and your work) ! Alain -- Administrateur Syst?me/R?seau C2N Centre de Nanosciences et Nanotechnologies (UMR 9001) Boulevard Thomas Gobert (ex Avenue de La Vauve), 91920 Palaiseau Tel : 01-70-27-06-88 Bureau A255 From contact+dev at gilouweb.com Fri Jul 5 11:15:33 2019 From: contact+dev at gilouweb.com (Gilou) Date: Fri, 5 Jul 2019 11:15:33 +0200 Subject: [PVE-User] Proxmox VE 6.0 beta released! In-Reply-To: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> Message-ID: Le 04/07/2019 ? 21:06, Martin Maurer a ?crit?: > Hi all! > > We're happy to announce the first beta release for the Proxmox VE 6.x > family! It's based on the great Debian Buster (Debian 10) and a 5.0 > kernel, QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and countless > improvements and bugfixes. The new installer supports ZFS root via UEFI, > for example you can boot a ZFS mirror on NVMe SSDs (using systemd-boot > instead of grub). The full release notes will be available together with > the final release announcement. > > For more details, see: > https://forum.proxmox.com/threads/proxmox-ve-6-0-beta-released.55670/ > Awesome work, we'll be testing it for sure. Thanks for the nice doc and beta work ahead of Debian 10, it feels really good to see you following them closely. Cheers, Gilou From gilberto.nunes32 at gmail.com Fri Jul 5 15:19:59 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Fri, 5 Jul 2019 10:19:59 -0300 Subject: [PVE-User] Proxmox VE 6.0 beta released! In-Reply-To: References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> Message-ID: Hi there! Correct me if I wrong, but after make changes in network still need reboot the server? --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em sex, 5 de jul de 2019 ?s 06:16, Gilou escreveu: > Le 04/07/2019 ? 21:06, Martin Maurer a ?crit : > > Hi all! > > > > We're happy to announce the first beta release for the Proxmox VE 6.x > > family! It's based on the great Debian Buster (Debian 10) and a 5.0 > > kernel, QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and countless > > improvements and bugfixes. The new installer supports ZFS root via UEFI, > > for example you can boot a ZFS mirror on NVMe SSDs (using systemd-boot > > instead of grub). The full release notes will be available together with > > the final release announcement. > > > > For more details, see: > > https://forum.proxmox.com/threads/proxmox-ve-6-0-beta-released.55670/ > > > > Awesome work, we'll be testing it for sure. Thanks for the nice doc and > beta work ahead of Debian 10, it feels really good to see you following > them closely. > > Cheers, > > Gilou > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From s.ivanov at proxmox.com Fri Jul 5 15:32:53 2019 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Fri, 5 Jul 2019 15:32:53 +0200 Subject: [PVE-User] Proxmox VE 6.0 beta released! In-Reply-To: References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> Message-ID: <20190705153253.3abd07f4@rosa.proxmox.com> Hi, On Fri, 5 Jul 2019 10:19:59 -0300 Gilberto Nunes wrote: > Hi there! Correct me if I wrong, but after make changes in network > still need reboot the server? In the default setup this is still correct. However PVE has the `ifupdown2` package in the repository (since quite some while - 5.2 IIRC), which is (mostly) a drop-in replacement for `ifupdown` with increased functionality. `ifupdown2` ships the `ifreload` binary, which can handle a network reload without the need of rebooting the node to ensure a consistent state. Take it with a grain of salt, because we've seen certain edge-cases where it does not work as expected (which is one of the reasons we have not switched over to it in general). Hope this helps! stoiko > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em sex, 5 de jul de 2019 ?s 06:16, Gilou > escreveu: > > > Le 04/07/2019 ? 21:06, Martin Maurer a ?crit : > > > Hi all! > > > > > > We're happy to announce the first beta release for the Proxmox VE > > > 6.x family! It's based on the great Debian Buster (Debian 10) and > > > a 5.0 kernel, QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and > > > countless improvements and bugfixes. The new installer supports > > > ZFS root via UEFI, for example you can boot a ZFS mirror on NVMe > > > SSDs (using systemd-boot instead of grub). The full release notes > > > will be available together with the final release announcement. > > > > > > For more details, see: > > > https://forum.proxmox.com/threads/proxmox-ve-6-0-beta-released.55670/ > > > > > > > Awesome work, we'll be testing it for sure. Thanks for the nice doc > > and beta work ahead of Debian 10, it feels really good to see you > > following them closely. > > > > Cheers, > > > > Gilou > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From dor at volz.ua Fri Jul 5 18:34:46 2019 From: dor at volz.ua (Dmytro O. Redchuk) Date: Fri, 5 Jul 2019 19:34:46 +0300 Subject: [PVE-User] Proxmox VE 6.0 beta released! In-Reply-To: References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> Message-ID: <20190705163446.GB16870@volz.ua> ? ??., 05-?? ???. 2019, ? 10:19 Gilberto Nunes wrote: > Hi there! Correct me if I wrong, but after make changes in network still > need reboot the server? If you know what you're doing, you can cat interfaces.new > interfaces and systemctl restart networking or something like this. -- Dmytro O. Redchuk From jmr.richardson at gmail.com Fri Jul 5 19:39:08 2019 From: jmr.richardson at gmail.com (JR Richardson) Date: Fri, 5 Jul 2019 12:39:08 -0500 Subject: [PVE-User] Cluster Host Node Rebooted and VMs Did not restart? UPDATE In-Reply-To: <000601d5307c$a417aa70$ec46ff50$@gmail.com> References: <000601d5307c$a417aa70$ec46ff50$@gmail.com> Message-ID: > Ran into an interesting problem today. Topology is a 4-Node Cluster on v.5.2, been up and running without issues for 10 months or so. Several VMs running on node 1, 2 & 3, maybe 30% loaded. I keep node 4 empty for HA and new build VM testing. All VMs are set for HA and auto start on node reboot. > > Node 1 rebooted, no indication why, nothing in the logs identifying cause of the node rebooting. When node 1 came back up it was 1 hour ahead of time and none of the VMs would start and HA did not start up the VMs on empty node 4. > I tested this scenario in the lab running PVE 5.4-7 and cannot reproduce the original symptoms above. I manually set 1 node hardware clock +1 hour, rebooted and when it came back up, the system time and hardware clock synchronized to correct time within a minute or so and VMs started up OK. So my question, is there a startup routine in PVE 5.4-7 that synchronizes system and hardware clock shortly after boot up that was not present in PVE 5.2? Thanks. JR -- JR Richardson Engineering for the Masses Chasing the Azeotrope From arjenvanweelden at gmail.com Sat Jul 6 07:57:57 2019 From: arjenvanweelden at gmail.com (arjenvanweelden at gmail.com) Date: Sat, 06 Jul 2019 07:57:57 +0200 Subject: [PVE-User] Proxmox VE 6.0 beta released; missing documentation ZFS on UEFI In-Reply-To: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> Message-ID: <7580d7566fe3214b438145726be5742093dc9722.camel@gmail.com> On Thu, 2019-07-04 at 21:06 +0200, Martin Maurer wrote: > Hi all! > > We're happy to announce the first beta release for the Proxmox VE 6.x > family! It's based on the great Debian Buster (Debian 10) and a 5.0 > kernel, QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and countless Thank you for all the hard work you are putting into the new release. > improvements and bugfixes. The new installer supports ZFS root via > UEFI, for example you can boot a ZFS mirror on NVMe SSDs (using > systemd-boot instead of grub). The full release notes will be > available together with the final release announcement. I just installed the beta using a ZFS root with UEFI on a single drive, which boots fine but I want to mirror that first drive. Making a RAID1 out of a RAID0 ZFS vdev is the same as before, but I wonder about duplicating the EFI System Partition. I just wanted to let you know that the Help in the beta (/pve-docs/chapter-sysadmin.html#chapter_zfs) still claims "It is not possible to use ZFS as root file system with UEFI boot". Is there any new documentation of shall I wait for the final release? kind regards, Arjen PS: I'm also getting some locale warnings on this beta when using apt- get dist-upgrade: perl: warning: Setting locale failed. perl: warning: Please check that your locale settings: LANGUAGE = (unset), LC_ALL = (unset), LC_TIME = "nl_NL.UTF-8", LC_MONETARY = "nl_NL.UTF-8", LC_ADDRESS = "nl_NL.UTF-8", LC_TELEPHONE = "nl_NL.UTF-8", LC_NAME = "nl_NL.UTF-8", LC_MEASUREMENT = "nl_NL.UTF-8", LC_IDENTIFICATION = "nl_NL.UTF-8", LC_NUMERIC = "nl_NL.UTF-8", LC_PAPER = "nl_NL.UTF-8", LANG = "en_US.UTF-8" are supported and installed on your system. perl: warning: Falling back to a fallback locale ("en_US.UTF-8"). apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! From chris.hofstaedtler at deduktiva.com Mon Jul 8 02:16:34 2019 From: chris.hofstaedtler at deduktiva.com (Chris Hofstaedtler | Deduktiva) Date: Mon, 8 Jul 2019 02:16:34 +0200 Subject: [PVE-User] Debian buster inside PVE KVM Message-ID: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> Hello, while doing some test upgrades I ran into the buster RNG problem [1], where the newer kernel and systemd use a lot more randomness during boot, causing startup delays. Very clearly noticable in dmesg: [ 1.500056] random: fast init done [ 191.700840] random: crng init done [ 191.701445] random: 7 urandom warning(s) missed due to ratelimiting I couldn't find a supported way of enabling virtio_rng [2] in PVE 5.4 or the 6.0 beta. As a test, I've set "args: -device virtio-rng-pci" and that appears to work - the VM auto-loads the virtio_rng kmod and "crng init done" happens at ~4s after poweron. Are there any recommendations at this time or plans for adding virtio_rng? [1] https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#entropy-starvation [2] https://wiki.qemu.org/Features/VirtIORNG Thanks, -- Chris Hofstaedtler / Deduktiva GmbH (FN 418592 b, HG Wien) www.deduktiva.com / +43 1 353 1707 From f.gruenbichler at proxmox.com Mon Jul 8 07:56:05 2019 From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?Q?Gr=FCnbichler?=) Date: Mon, 8 Jul 2019 07:56:05 +0200 Subject: [PVE-User] Proxmox VE 6.0 beta released; missing documentation ZFS on UEFI In-Reply-To: <7580d7566fe3214b438145726be5742093dc9722.camel@gmail.com> References: <50efaacd-d15a-848c-584e-8697fb3a1d97@proxmox.com> <7580d7566fe3214b438145726be5742093dc9722.camel@gmail.com> Message-ID: <20190708055605.z6z2hkv3mbi4vjsf@nora.maurer-it.com> On Sat, Jul 06, 2019 at 07:57:57AM +0200, arjenvanweelden at gmail.com wrote: > On Thu, 2019-07-04 at 21:06 +0200, Martin Maurer wrote: > > Hi all! > > > > We're happy to announce the first beta release for the Proxmox VE 6.x > > family! It's based on the great Debian Buster (Debian 10) and a 5.0 > > kernel, QEMU 4.0, ZFS 0.8.1, Ceph 14.2.1, Corosync 3.0 and countless > > Thank you for all the hard work you are putting into the new release. > > > improvements and bugfixes. The new installer supports ZFS root via > > UEFI, for example you can boot a ZFS mirror on NVMe SSDs (using > > systemd-boot instead of grub). The full release notes will be > > available together with the final release announcement. > > I just installed the beta using a ZFS root with UEFI on a single drive, > which boots fine but I want to mirror that first drive. > Making a RAID1 out of a RAID0 ZFS vdev is the same as before, but I > wonder about duplicating the EFI System Partition. > > I just wanted to let you know that the Help in the beta ( host>/pve-docs/chapter-sysadmin.html#chapter_zfs) still claims "It is > not possible to use ZFS as root file system with UEFI boot". > > Is there any new documentation of shall I wait for the final release? docs will be updated later this week, and will include information on how to initialize an ESP to be picked up by our sync scripts/hooks. From f.gruenbichler at proxmox.com Mon Jul 8 08:05:40 2019 From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?Q?Gr=FCnbichler?=) Date: Mon, 8 Jul 2019 08:05:40 +0200 Subject: [PVE-User] Debian buster inside PVE KVM In-Reply-To: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> References: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> Message-ID: <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> On Mon, Jul 08, 2019 at 02:16:34AM +0200, Chris Hofstaedtler | Deduktiva wrote: > Hello, > > while doing some test upgrades I ran into the buster RNG problem [1], > where the newer kernel and systemd use a lot more randomness during > boot, causing startup delays. > > Very clearly noticable in dmesg: > [ 1.500056] random: fast init done > [ 191.700840] random: crng init done > [ 191.701445] random: 7 urandom warning(s) missed due to ratelimiting > > I couldn't find a supported way of enabling virtio_rng [2] in PVE > 5.4 or the 6.0 beta. As a test, I've set "args: -device > virtio-rng-pci" and that appears to work - the VM auto-loads the > virtio_rng kmod and "crng init done" happens at ~4s after poweron. yes, that's the way to go for now. > Are there any recommendations at this time or plans for adding > virtio_rng? filed [1] to keep track of adding proper support, as it sounds like a simple enough but worthwhile feature to me :) 1: https://bugzilla.proxmox.com/show_bug.cgi?id=2264 From t.lamprecht at proxmox.com Mon Jul 8 09:10:48 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 8 Jul 2019 09:10:48 +0200 Subject: [PVE-User] Debian buster inside PVE KVM In-Reply-To: <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> References: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> Message-ID: Am 7/8/19 um 8:05 AM schrieb Fabian Gr?nbichler: > On Mon, Jul 08, 2019 at 02:16:34AM +0200, Chris Hofstaedtler | Deduktiva wrote: >> Hello, >> >> while doing some test upgrades I ran into the buster RNG problem [1], >> where the newer kernel and systemd use a lot more randomness during >> boot, causing startup delays. >> >> Very clearly noticable in dmesg: >> [ 1.500056] random: fast init done >> [ 191.700840] random: crng init done >> [ 191.701445] random: 7 urandom warning(s) missed due to ratelimiting >> >> I couldn't find a supported way of enabling virtio_rng [2] in PVE >> 5.4 or the 6.0 beta. As a test, I've set "args: -device >> virtio-rng-pci" and that appears to work - the VM auto-loads the >> virtio_rng kmod and "crng init done" happens at ~4s after poweron. > > yes, that's the way to go for now. > >> Are there any recommendations at this time or plans for adding >> virtio_rng? > > filed [1] to keep track of adding proper support, as it sounds like a > simple enough but worthwhile feature to me :) > > 1: https://bugzilla.proxmox.com/show_bug.cgi?id=2264 > The request for this is a bit older, and then some concerns about possible depleting the hosts entropy pool were raised. Maybe we want to ship havedged, or at least recommend it in docs if no other "high" bandwitdh (relatively speaking) HW rng source is available on the host... ATM, I cannot find the discussion, sorry, IIRC it was on a mailing list of ours.. From arjenvanweelden at gmail.com Mon Jul 8 09:34:33 2019 From: arjenvanweelden at gmail.com (arjenvanweelden at gmail.com) Date: Mon, 08 Jul 2019 09:34:33 +0200 Subject: [PVE-User] Debian buster inside PVE KVM In-Reply-To: References: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> Message-ID: <43816e03bd58bed77509989b7c50105eac8f3745.camel@gmail.com> On Mon, 2019-07-08 at 09:10 +0200, Thomas Lamprecht wrote: > Am 7/8/19 um 8:05 AM schrieb Fabian Gr?nbichler: > > On Mon, Jul 08, 2019 at 02:16:34AM +0200, Chris Hofstaedtler | > > Deduktiva wrote: > > > Hello, > > > > > > while doing some test upgrades I ran into the buster RNG problem > > > [1], > > > where the newer kernel and systemd use a lot more randomness > > > during > > > boot, causing startup delays. > > > > > > Very clearly noticable in dmesg: > > > [ 1.500056] random: fast init done > > > [ 191.700840] random: crng init done > > > [ 191.701445] random: 7 urandom warning(s) missed due to > > > ratelimiting > > > > > > I couldn't find a supported way of enabling virtio_rng [2] in PVE > > > 5.4 or the 6.0 beta. As a test, I've set "args: -device > > > virtio-rng-pci" and that appears to work - the VM auto-loads the > > > virtio_rng kmod and "crng init done" happens at ~4s after > > > poweron. > > > > yes, that's the way to go for now. > > > > > Are there any recommendations at this time or plans for adding > > > virtio_rng? > > > > filed [1] to keep track of adding proper support, as it sounds like > > a > > simple enough but worthwhile feature to me :) > > > > 1: https://bugzilla.proxmox.com/show_bug.cgi?id=2264 > > > > The request for this is a bit older, and then some concerns about > possible depleting the hosts entropy pool were raised. > Maybe we want to ship havedged, or at least recommend it in docs if > no > other "high" bandwitdh (relatively speaking) HW rng source is > available on the host... ATM, I cannot find the discussion, sorry, > IIRC it was on a mailing list of ours.. > I've been happily using this for some time now, after I figured out which address to use. When adding virtio-devices manually, one must make sure that the virtual PCI-slot (,addr=...) does not conflict with other (optional) devices, which Proxmox can do automatically. Having this (as an option) in the GUI would be very nice, and 'apt-get install haveged' is quick and easy. kind regards, Arjen From t.lamprecht at proxmox.com Mon Jul 8 09:48:11 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 8 Jul 2019 09:48:11 +0200 Subject: [PVE-User] Debian buster inside PVE KVM In-Reply-To: <43816e03bd58bed77509989b7c50105eac8f3745.camel@gmail.com> References: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> <43816e03bd58bed77509989b7c50105eac8f3745.camel@gmail.com> Message-ID: Am 7/8/19 um 9:34 AM schrieb arjenvanweelden at gmail.com: > Having this (as an option) in the GUI would be very nice, > and 'apt-get install haveged' is quick and easy. opt-in is surely no problem, my concerns would be rather for the case where we just add this for VMs with Linux as ostype, because why not, VMs can only profit from it, as said, the single thing to look out is that enough entropy is available. And sure it's easy to install havedged, were using a sane Linux Distro as base, after all ;) But one *needs* to do it, else bad or no entropy can harm too. It'd be great if one would you like to assemble a patch for this, shouldn't be to much work. From arjenvanweelden at gmail.com Mon Jul 8 09:56:40 2019 From: arjenvanweelden at gmail.com (arjenvanweelden at gmail.com) Date: Mon, 08 Jul 2019 09:56:40 +0200 Subject: [PVE-User] Debian buster inside PVE KVM In-Reply-To: References: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> <43816e03bd58bed77509989b7c50105eac8f3745.camel@gmail.com> Message-ID: On Mon, 2019-07-08 at 09:48 +0200, Thomas Lamprecht wrote: > Am 7/8/19 um 9:34 AM schrieb arjenvanweelden at gmail.com: > > Having this (as an option) in the GUI would be very nice, > > and 'apt-get install haveged' is quick and easy. > > opt-in is surely no problem, my concerns would be rather for > the case where we just add this for VMs with Linux as ostype, > because why not, VMs can only profit from it, as said, the > single thing to look out is that enough entropy is available. > > And sure it's easy to install havedged, were using a sane Linux > Distro as base, after all ;) But one *needs* to do it, else bad > or no entropy can harm too. Is just installing haveged sufficient? Can the Proxmox-team decide to add haveged to it's dependencies? Or is more discussion required? > It'd be great if one would you like to assemble a patch for this, > shouldn't be to much work. I'll have a look but cannot guarantee anything. From t.lamprecht at proxmox.com Mon Jul 8 10:16:51 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 8 Jul 2019 10:16:51 +0200 Subject: [PVE-User] Debian buster inside PVE KVM In-Reply-To: References: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> <43816e03bd58bed77509989b7c50105eac8f3745.camel@gmail.com> Message-ID: Am 7/8/19 um 9:56 AM schrieb arjenvanweelden at gmail.com: > Is just installing haveged sufficient? Can the Proxmox-team decide to > add haveged to it's dependencies? Or is more discussion required? It'd be, the service is then enabled and running by default. For me it'd be OK to add as a dependency or recommends somewhere, but I have to say that I did not looked to much into possible bad implications or what people with good knowledge of statistics/ randomness think about havedged, but it should not be to bad, AFAICT, or at least hopefully better than nothing ^^ > I'll have a look but cannot guarantee anything. Appreciated! From chris.hofstaedtler at deduktiva.com Mon Jul 8 10:43:54 2019 From: chris.hofstaedtler at deduktiva.com (Chris Hofstaedtler | Deduktiva) Date: Mon, 8 Jul 2019 10:43:54 +0200 Subject: [PVE-User] Debian buster inside PVE KVM In-Reply-To: References: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> Message-ID: <20190708084354.wisv2ew4hs63zleg@tx.intern.ewave.at> * Thomas Lamprecht [190708 09:13]: > Am 7/8/19 um 8:05 AM schrieb Fabian Gr?nbichler: > > On Mon, Jul 08, 2019 at 02:16:34AM +0200, Chris Hofstaedtler | Deduktiva wrote: [..] > >> Are there any recommendations at this time or plans for adding > >> virtio_rng? > > > > filed [1] to keep track of adding proper support, as it sounds like a > > simple enough but worthwhile feature to me :) > > > > 1: https://bugzilla.proxmox.com/show_bug.cgi?id=2264 > > The request for this is a bit older, and then some concerns about > possible depleting the hosts entropy pool were raised. > Maybe we want to ship havedged, or at least recommend it in docs if no > other "high" bandwitdh (relatively speaking) HW rng source is > available on the host... Right, makes sense. OTOH on modern hosts with RDRAND and CONFIG_RANDOM_TRUST_CPU=y (as in 5.0.15-1-pve) this shouldn't be much of a problem (guessing here). Cheers, Chris -- Chris Hofstaedtler / Deduktiva GmbH (FN 418592 b, HG Wien) www.deduktiva.com / +43 1 353 1707 From f.gruenbichler at proxmox.com Mon Jul 8 12:13:37 2019 From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?Q?Gr=FCnbichler?=) Date: Mon, 8 Jul 2019 12:13:37 +0200 Subject: [PVE-User] Debian buster inside PVE KVM In-Reply-To: References: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> Message-ID: <20190708101337.vdfhbhbvfea457cj@nora.maurer-it.com> On Mon, Jul 08, 2019 at 09:10:48AM +0200, Thomas Lamprecht wrote: > Am 7/8/19 um 8:05 AM schrieb Fabian Gr?nbichler: > > On Mon, Jul 08, 2019 at 02:16:34AM +0200, Chris Hofstaedtler | Deduktiva wrote: > >> Hello, > >> > >> while doing some test upgrades I ran into the buster RNG problem [1], > >> where the newer kernel and systemd use a lot more randomness during > >> boot, causing startup delays. > >> > >> Very clearly noticable in dmesg: > >> [ 1.500056] random: fast init done > >> [ 191.700840] random: crng init done > >> [ 191.701445] random: 7 urandom warning(s) missed due to ratelimiting > >> > >> I couldn't find a supported way of enabling virtio_rng [2] in PVE > >> 5.4 or the 6.0 beta. As a test, I've set "args: -device > >> virtio-rng-pci" and that appears to work - the VM auto-loads the > >> virtio_rng kmod and "crng init done" happens at ~4s after poweron. > > > > yes, that's the way to go for now. > > > >> Are there any recommendations at this time or plans for adding > >> virtio_rng? > > > > filed [1] to keep track of adding proper support, as it sounds like a > > simple enough but worthwhile feature to me :) > > > > 1: https://bugzilla.proxmox.com/show_bug.cgi?id=2264 > > > > The request for this is a bit older, and then some concerns about > possible depleting the hosts entropy pool were raised. > Maybe we want to ship havedged, or at least recommend it in docs if no > other "high" bandwitdh (relatively speaking) HW rng source is > available on the host... ATM, I cannot find the discussion, sorry, > IIRC it was on a mailing list of ours.. haveged is surrounded by some controversy especially for usage inside VMs, since it relies on jitter via timer instructions that may or may not be passed through to the actual hardware, and most recommendations actually err on the side of "stay away unless you have no choice"(see 1, 2 and the stuff linked there). virtio-rng does have the issue of potentially depleting the host's entropy pool, with a proper HWRNG, this is not really an issue. it is possible to ratelimit the virtio-rng device (max-bytes/period parameter). offering as opt-in it with the proper caveat ("only enable if your host can provide lots of entropy") is probably better than pointing at potentially problematic solutions? VMs with CPU types that pass in rdrand/rdseed are also "fixed". 1: https://wiki.debian.org/BoottimeEntropyStarvation 2: https://wiki.archlinux.org/index.php/Haveged From f.gruenbichler at proxmox.com Mon Jul 8 12:16:46 2019 From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?Q?Gr=FCnbichler?=) Date: Mon, 8 Jul 2019 12:16:46 +0200 Subject: [PVE-User] Debian buster inside PVE KVM In-Reply-To: <20190708084354.wisv2ew4hs63zleg@tx.intern.ewave.at> References: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> <20190708084354.wisv2ew4hs63zleg@tx.intern.ewave.at> Message-ID: <20190708101646.nktnqbu6rrmjb2pr@nora.maurer-it.com> On Mon, Jul 08, 2019 at 10:43:54AM +0200, Chris Hofstaedtler | Deduktiva wrote: > * Thomas Lamprecht [190708 09:13]: > > Am 7/8/19 um 8:05 AM schrieb Fabian Gr?nbichler: > > > On Mon, Jul 08, 2019 at 02:16:34AM +0200, Chris Hofstaedtler | Deduktiva wrote: > [..] > > >> Are there any recommendations at this time or plans for adding > > >> virtio_rng? > > > > > > filed [1] to keep track of adding proper support, as it sounds like a > > > simple enough but worthwhile feature to me :) > > > > > > 1: https://bugzilla.proxmox.com/show_bug.cgi?id=2264 > > > > The request for this is a bit older, and then some concerns about > > possible depleting the hosts entropy pool were raised. > > Maybe we want to ship havedged, or at least recommend it in docs if no > > other "high" bandwitdh (relatively speaking) HW rng source is > > available on the host... > > Right, makes sense. OTOH on modern hosts with RDRAND and > CONFIG_RANDOM_TRUST_CPU=y (as in 5.0.15-1-pve) this shouldn't be > much of a problem (guessing here). I can still deplete my host's entropy pool from a VM with virtio-rng easily, even with RDRAND/RDSEED and CONFIG_RANDOM_TRUST_CPU. I think even with a "proper" HWRNG you'd still want rate-limiting unless you trust all your virtio-rng users to not request large amounts of randomness at once. The small amounts needed for booting or the occasional key generation are fine. From t.lamprecht at proxmox.com Mon Jul 8 12:19:23 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 8 Jul 2019 12:19:23 +0200 Subject: [PVE-User] Debian buster inside PVE KVM In-Reply-To: <20190708101337.vdfhbhbvfea457cj@nora.maurer-it.com> References: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> <20190708101337.vdfhbhbvfea457cj@nora.maurer-it.com> Message-ID: <7e533094-7a0e-ce29-7927-ac76bbb79eb0@proxmox.com> Am 7/8/19 um 12:13 PM schrieb Fabian Gr?nbichler: > On Mon, Jul 08, 2019 at 09:10:48AM +0200, Thomas Lamprecht wrote: >> Am 7/8/19 um 8:05 AM schrieb Fabian Gr?nbichler: >>> On Mon, Jul 08, 2019 at 02:16:34AM +0200, Chris Hofstaedtler | Deduktiva wrote: >>>> Hello, >>>> >>>> while doing some test upgrades I ran into the buster RNG problem [1], >>>> where the newer kernel and systemd use a lot more randomness during >>>> boot, causing startup delays. >>>> >>>> Very clearly noticable in dmesg: >>>> [ 1.500056] random: fast init done >>>> [ 191.700840] random: crng init done >>>> [ 191.701445] random: 7 urandom warning(s) missed due to ratelimiting >>>> >>>> I couldn't find a supported way of enabling virtio_rng [2] in PVE >>>> 5.4 or the 6.0 beta. As a test, I've set "args: -device >>>> virtio-rng-pci" and that appears to work - the VM auto-loads the >>>> virtio_rng kmod and "crng init done" happens at ~4s after poweron. >>> >>> yes, that's the way to go for now. >>> >>>> Are there any recommendations at this time or plans for adding >>>> virtio_rng? >>> >>> filed [1] to keep track of adding proper support, as it sounds like a >>> simple enough but worthwhile feature to me :) >>> >>> 1: https://bugzilla.proxmox.com/show_bug.cgi?id=2264 >>> >> >> The request for this is a bit older, and then some concerns about >> possible depleting the hosts entropy pool were raised. >> Maybe we want to ship havedged, or at least recommend it in docs if no >> other "high" bandwitdh (relatively speaking) HW rng source is >> available on the host... ATM, I cannot find the discussion, sorry, >> IIRC it was on a mailing list of ours.. > > haveged is surrounded by some controversy especially for usage inside > VMs, since it relies on jitter via timer instructions that may or may > not be passed through to the actual hardware, and most recommendations > actually err on the side of "stay away unless you have no choice"(see > 1, 2 and the stuff linked there). OK, that are the issues I was concerned about possibly existing. Thanks for pointing at them! > > virtio-rng does have the issue of potentially depleting the host's > entropy pool, with a proper HWRNG, this is not really an issue. it is > possible to ratelimit the virtio-rng device (max-bytes/period > parameter). > > offering as opt-in it with the proper caveat ("only enable if your host > can provide lots of entropy") is probably better than pointing at > potentially problematic solutions? Definitively. > > VMs with CPU types that pass in rdrand/rdseed are also "fixed". > > 1: https://wiki.debian.org/BoottimeEntropyStarvation > 2: https://wiki.archlinux.org/index.php/Haveged > h From gilberto.nunes32 at gmail.com Mon Jul 8 14:03:51 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Mon, 8 Jul 2019 09:03:51 -0300 Subject: [PVE-User] Trouble with LXC Backup Message-ID: Hi There! Somebody here has trouble when make LXC Backup with 1 TB of rootdisk??? I have one LXC that takes forever and stuck... This is pretty anoying... Anybody? Thanks a lot --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From venefax at gmail.com Mon Jul 8 14:15:55 2019 From: venefax at gmail.com (Saint Michael) Date: Mon, 8 Jul 2019 08:15:55 -0400 Subject: [PVE-User] Debian buster inside PVE KVM In-Reply-To: <7e533094-7a0e-ce29-7927-ac76bbb79eb0@proxmox.com> References: <20190708001634.hj7ltag3jv4r3zcb@percival.namespace.at> <20190708060540.u5gbece27moclpvu@nora.maurer-it.com> <20190708101337.vdfhbhbvfea457cj@nora.maurer-it.com> <7e533094-7a0e-ce29-7927-ac76bbb79eb0@proxmox.com> Message-ID: > > I have tried all tricks and still my vrirtual machines consume, idle, more > tha one entire CPU, often 1.3. My proxmox is licensed and fully updated. > Something is wrong. All of them use vortio drivers. Pease look at this > ps ax | grep kvm > 24983 ? Sl 127:41 /usr/bin/kvm -id 103 -name Mikrotik -chardev socket,id=qmp,path=/var/run/qemu-server/103.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/103.pid -daemonize -smbios type=1,uuid=4bf6a4d8-f4db-41c4-bdcb-447eb00286c9 -smp 2,sockets=1,cores=2,maxcpus=2 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/103.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 512 -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device VGA,id=vga,bus=pci.0,addr=0x2 -chardev socket,path=/var/run/qemu-server/103.qga,server,nowait,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:f8f3ad9164fd -drive file=/nfs2/images/103/vm-103-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,cache=none,aio=native,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap103i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=DE:48:15:62:49:0B,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -netdev type=tap,id=net1,ifname=tap103i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=5A:26:46:C7:5C:AE,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301 -machine type=pc 25220 ? S 0:00 [kvm-pit/24983] 38573 ? Sl 278:51 /usr/bin/kvm -id 113 -name VIRTUALIZOR-1 -chardev socket,id=qmp,path=/var/run/qemu-server/113.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/113.pid -daemonize -smbios type=1,uuid=2b4f9e6a-d3e7-480d-972b-13ea394d80ef -smp 120,sockets=4,cores=30,maxcpus=120 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/113.vnc,x509,password -cpu host,+pcid,+spec-ctrl,+kvm_pv_unhalt,+kvm_pv_eoi -m 32000 -object memory-backend-ram,id=ram-node0,size=8000M -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 -object memory-backend-ram,id=ram-node1,size=8000M -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 -object memory-backend-ram,id=ram-node2,size=8000M -numa node,nodeid=2,cpus=60-89,memdev=ram-node2 -object memory-backend-ram,id=ram-node3,size=8000M -numa node,nodeid=3,cpus=90-119,memdev=ram-node3 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device vmgenid,guid=6b67f79b-88eb-4714-91ce-24cb1ef9d701 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-vga,id=vga,bus=pci.0,addr=0x2 -chardev socket,path=/var/run/qemu-server/113.qga,server,nowait,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:f8f3ad9164fd -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/nfs/images/113/vm-113-disk-1.qcow2,if=none,id=drive-scsi0,cache=writeback,format=qcow2,aio=threads,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200 -drive file=/nfs/images/113/vm-113-disk-2.qcow2,if=none,id=drive-scsi1,cache=writeback,format=qcow2,aio=threads,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1 -netdev type=tap,id=net0,ifname=tap113i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on,queues=8 -device virtio-net-pci,mac=D2:0D:5E:D0:DA:A1,netdev=net0,bus=pci.0,addr=0x12,id=net0,vectors=18,mq=on,bootindex=300 -machine type=pc 40075 ? S 0:00 [kvm-pit/38573] 41044 ? Sl 262:52 /usr/bin/kvm -id 114 -name VIRTUALIZOR-2 -chardev socket,id=qmp,path=/var/run/qemu-server/114.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/114.pid -daemonize -smbios type=1,uuid=199e23e5-172d-4f6e-9c9f-b5c80d2159f9 -smp 120,sockets=4,cores=30,maxcpus=120 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/114.vnc,x509,password -cpu host,+pcid,+spec-ctrl,+kvm_pv_unhalt,+kvm_pv_eoi -m 32000 -object memory-backend-ram,id=ram-node0,size=8000M -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 -object memory-backend-ram,id=ram-node1,size=8000M -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 -object memory-backend-ram,id=ram-node2,size=8000M -numa node,nodeid=2,cpus=60-89,memdev=ram-node2 -object memory-backend-ram,id=ram-node3,size=8000M -numa node,nodeid=3,cpus=90-119,memdev=ram-node3 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device vmgenid,guid=1bbea63f-08c3-40fa-9b10-215e798675ad -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-vga,id=vga,bus=pci.0,addr=0x2 -chardev socket,path=/var/run/qemu-server/114.qga,server,nowait,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:f8f3ad9164fd -drive if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100 -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/nfs/images/114/vm-114-disk-0.qcow2,if=none,id=drive-scsi0,cache=writeback,format=qcow2,aio=threads,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200 -drive file=/nfs/images/114/vm-114-disk-1.qcow2,if=none,id=drive-scsi1,cache=writeback,format=qcow2,aio=threads,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1 -netdev type=tap,id=net0,ifname=tap114i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on,queues=8 -device virtio-net-pci,mac=E2:DF:7C:7F:97:A6,netdev=net0,bus=pci.0,addr=0x12,id=net0,vectors=18,mq=on,bootindex=300 -machine type=pc 41549 ? S 0:00 [kvm-pit/41044] 41786 ? Sl 355:16 /usr/bin/kvm -id 115 -name VIRTUALIZOR-3 -chardev socket,id=qmp,path=/var/run/qemu-server/115.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/115.pid -daemonize -smbios type=1,uuid=39d8175e-7a08-4040-8de3-6452621612fd -smp 120,sockets=4,cores=30,maxcpus=120 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/115.vnc,x509,password -cpu host,+pcid,+spec-ctrl,+kvm_pv_unhalt,+kvm_pv_eoi -m 32000 -object memory-backend-ram,id=ram-node0,size=8000M -numa node,nodeid=0,cpus=0-29,memdev=ram-node0 -object memory-backend-ram,id=ram-node1,size=8000M -numa node,nodeid=1,cpus=30-59,memdev=ram-node1 -object memory-backend-ram,id=ram-node2,size=8000M -numa node,nodeid=2,cpus=60-89,memdev=ram-node2 -object memory-backend-ram,id=ram-node3,size=8000M -numa node,nodeid=3,cpus=90-119,memdev=ram-node3 -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device vmgenid,guid=3853f064-4730-4bb4-864b-93dbd962f1b1 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-vga,id=vga,bus=pci.0,addr=0x2 -chardev socket,path=/var/run/qemu-server/115.qga,server,nowait,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:f8f3ad9164fd -drive if=none,id=drive-ide2,media=cdrom,aio=threads -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100 -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/nfs/images/115/vm-115-disk-0.qcow2,if=none,id=drive-scsi0,cache=writeback,format=qcow2,aio=threads,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200 -drive file=/nfs/images/115/vm-115-disk-1.qcow2,if=none,id=drive-scsi1,cache=writeback,format=qcow2,aio=threads,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1 -netdev type=tap,id=net0,ifname=tap115i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on,queues=8 -device virtio-net-pci,mac=AA:09:6F:87:13:CA,netdev=net0,bus=pci.0,addr=0x12,id=net0,vectors=18,mq=on,bootindex=300 -machine type=pc 42010 ? S 0:00 [kvm-pit/41786] 65523 ? Sl 1860:06 /usr/bin/kvm -id 104 -name FedericoWindows2016 -chardev socket,id=qmp,path=/var/run/qemu-server/104.qmp,server,nowait -mon chardev=qmp,mode=control -chardev socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5 -mon chardev=qmp-event,mode=control -pidfile /var/run/qemu-server/104.pid -daemonize -smbios type=1,uuid=449f474c-c776-4c53-81a8-2a574b32283f -smp 2,sockets=1,cores=2,maxcpus=2 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg -vnc unix:/var/run/qemu-server/104.vnc,x509,password -no-hpet -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer -m 32000 -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device VGA,id=vga,bus=pci.0,addr=0x2 -chardev socket,path=/var/run/qemu-server/104.qga,server,nowait,id=qga0 -device virtio-serial,id=qga0,bus=pci.0,addr=0x8 -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:f8f3ad9164fd -drive if=none,id=drive-ide0,media=cdrom,aio=threads -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=200 -drive file=/nfs1/images/104/vm-104-disk-1.qcow2,if=none,id=drive-virtio0,cache=writeback,format=qcow2,aio=threads,detect-zeroes=on -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap104i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=D2:EC:32:08:27:57,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 -rtc driftfix=slew,base=localtime -machine type=pc -global kvm-pit.lost_tick_policy=discard 65611 ? S 0:30 [kvm-pit/65523] From bsd at todoo.biz Mon Jul 8 18:04:03 2019 From: bsd at todoo.biz (bsd at todoo.biz) Date: Mon, 8 Jul 2019 18:04:03 +0200 Subject: JS reloading page Message-ID: <0927561D-0512-4923-9F80-5E2E56F05271@todoo.biz> Hello, There is a JS in Proxmox VE v.5.4.6 which reloads the page and forces all menu item at the top every 5". This is really very annoying because we have a quite extensive list of hosts / devices and it always puts back the list at the top ! Is there a way to remove this somehow ? You should really consider removing this JS feature it is painful and totally useless. Thanks. From tonci at suma-informatika.hr Mon Jul 8 18:50:07 2019 From: tonci at suma-informatika.hr (=?UTF-8?B?VG9uxI1pIFN0aXBpxI1ldmnEhw==?=) Date: Mon, 8 Jul 2019 18:50:07 +0200 Subject: [PVE-User] sharing zfs experience Message-ID: <8678ddf7-98ef-f87b-bb16-09ad4953804e@suma-informatika.hr> Hi to all, A customer of mine runs two clusters : 1. 2node with ibm v370 san as shared strage? (hared lvm) 2.? 3node cluster all nodes run zfs ...? no shared storage Couple days ago he had an power outage and during that period of time I was kind a worrying how apcupsd & proxmox will handle this situation. 1. Both nodes were properly shut down but one of 2 them dies , independent from power outage :) but just in the same time. I booted up remaining node , adjusted "votes" and started all vm-s residing on the shared lvm storage ...? No further questions ... prox handled that correctly 2. all 3 nodes started up but the most important lxc conteiner cloud not start. Reason: Job for pve-container at 104.service failed because the control process exited with error code. See "systemctl status pve-container at 104.service" and "journalctl -xe" for details. TASK ERROR: command 'systemctl start pve-container at 104' failed: exit code 1 Upgrading, restarting etc etc did not helped at all. The problem was that rootfs from this contaier? was completely empty ( it contained only /dev/ and /mnt/? dirs . Fortunately second mount point (aka 2nd disk) with 2T of data was pretty healthy and visible. So one option was to restore it from backup but zfs list command showed that this data set still holds data as much as it should (disk 0) root at pve01-hrz-zm:~# ls -al /rpool/data/subvol-104-disk-0/ total 10 drwxr-xr-x 4 root root 4 Srp? 4 14:07 . drwxr-xr-x 9 root root 9 Srp? 4 23:17 .. drwxr-xr-x 2 root root 2 Srp? 4 14:07 dev drwxr-xr-x 3 root root 3 Srp? 4 14:07 mnt root at pve01-hrz-zm:~# zfs list NAME??????????????????????????? USED? AVAIL? REFER? MOUNTPOINT rpool????????????????????????? 2,15T? 1,36T?? 104K? /rpool rpool/data???????????????????? 2,15T? 1,36T?? 128K? /rpool/data rpool/data/subvol-104-disk-0??? 751M? 15,3G?? 751M /rpool/data/subvol-104-disk-0 rpool/data/subvol-104-disk-1?? 2,15T?? 894G? 2,15T /rpool/data/subvol-104-disk-1 Interesting was that both lcx containers from this node had "empty" disk-0? (but the other one was not that big, it had only disk-0) and none of them could start. After many tries I decided to migrate this little container to other just to see what will happen :? migration was successfull and starting up as well .? OK (true relief finally :). then I tried to make backup of this vm just to see what will happen. No, backup was not successfull ... backup archive was only 1.7KB big. Ok, let's get back to migration scenario. So, the final conclusion was that migration itself was not the solution but snapshot was the right one. Snapshot was the step that revived this disk-0. So , at the end I just made snapshot of the 104-disk-0, cloned it back right after to 1044-disk-0 and then just change the reference in lxc configuration. After that lxc started successfully. I'm very wondering why this happened but am also very happy that above simple steps saved my day. Hopefully this information helps somebody that will run into same problem , but in the same time I truly hope that it won't happen :) BR Tonci Stipicevic From t.lamprecht at proxmox.com Mon Jul 8 19:37:02 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 8 Jul 2019 19:37:02 +0200 Subject: [PVE-User] JS reloading page In-Reply-To: References: Message-ID: <7c1478c9-b4f1-e068-6401-6b990f97b21e@proxmox.com> Hi, On 7/8/19 6:04 PM, bsd--- via pve-user wrote: > Hello, > > There is a JS in Proxmox VE v.5.4.6 which reloads the page and forces all menu item at the top every 5". A full page reload? We only do that on cluster creation, as there the websites TLS certificate changed, and thus it's necessary. > This is really very annoying because we have a quite extensive list of hosts / devices and it always puts back the list at the top ! Do you mean in the "Resource Tree" on the left? Could you please share some details like browser/OS used, also exactly on which component the issues is showing up? thanks! > > Is there a way to remove this somehow ? > > You should really consider removing this JS feature it is painful and totally useless. > > > Thanks. From s.ivanov at proxmox.com Mon Jul 8 20:19:18 2019 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Mon, 8 Jul 2019 20:19:18 +0200 Subject: [PVE-User] sharing zfs experience In-Reply-To: <8678ddf7-98ef-f87b-bb16-09ad4953804e@suma-informatika.hr> References: <8678ddf7-98ef-f87b-bb16-09ad4953804e@suma-informatika.hr> Message-ID: <20190708201918.00ca0f22@rosa.proxmox.com> hi, On Mon, 8 Jul 2019 18:50:07 +0200 Ton?i Stipi?evi? wrote: > Hi to all, > > A customer of mine runs two clusters : > > 1. 2node with ibm v370 san as shared strage? (hared lvm) > > 2.? 3node cluster all nodes run zfs ...? no shared storage > > > Couple days ago he had an power outage and during that period of time > I was kind a worrying how apcupsd & proxmox will handle this > situation. > > 1. Both nodes were properly shut down but one of 2 them dies , > independent from power outage :) but just in the same time. I booted > up remaining node , adjusted "votes" and started all vm-s residing on > the shared lvm storage ...? No further questions ... prox handled > that correctly > > 2. all 3 nodes started up but the most important lxc conteiner cloud > not start. > > Reason: Job for pve-container at 104.service failed because the control > process exited with error code. See "systemctl status > pve-container at 104.service" and "journalctl -xe" for details. TASK > ERROR: command 'systemctl start pve-container at 104' failed: exit code 1 > > Upgrading, restarting etc etc did not helped at all. The problem was > that rootfs from this contaier? was completely empty ( it contained > only /dev/ and /mnt/? dirs . Fortunately second mount point (aka 2nd > disk) with 2T of data was pretty healthy and visible. So one option > was to restore it from backup but zfs list command showed that this > data set still holds data as much as it should (disk 0) This somehow reminds me of a recent thread in the forum: https://forum.proxmox.com/threads/reboot-of-pve-host-breaks-lxc-container-startup.55486/#post-255641 did the rpool get imported completely - or are there some errors in the journal while the system booted? In any case - glad you manged to resolve the issue! > > root at pve01-hrz-zm:~# ls -al /rpool/data/subvol-104-disk-0/ > total 10 > drwxr-xr-x 4 root root 4 Srp? 4 14:07 . > drwxr-xr-x 9 root root 9 Srp? 4 23:17 .. > drwxr-xr-x 2 root root 2 Srp? 4 14:07 dev > drwxr-xr-x 3 root root 3 Srp? 4 14:07 mnt > > root at pve01-hrz-zm:~# zfs list > NAME??????????????????????????? USED? AVAIL? REFER? MOUNTPOINT > rpool????????????????????????? 2,15T? 1,36T?? 104K? /rpool > rpool/data???????????????????? 2,15T? 1,36T?? 128K? /rpool/data > rpool/data/subvol-104-disk-0??? 751M? 15,3G?? 751M > /rpool/data/subvol-104-disk-0 > rpool/data/subvol-104-disk-1?? 2,15T?? 894G? 2,15T > /rpool/data/subvol-104-disk-1 > > > Interesting was that both lcx containers from this node had "empty" > disk-0? (but the other one was not that big, it had only disk-0) and > none of them could start. > > After many tries I decided to migrate this little container to other > just to see what will happen :? migration was successfull and > starting up as well .? OK (true relief finally :). then I tried to > make backup of this vm just to see what will happen. No, backup was > not successfull ... backup archive was only 1.7KB big. Ok, let's get > back to migration scenario. So, the final conclusion was that > migration itself was not the solution but snapshot was the right one. > Snapshot was the step that revived this disk-0. > > So , at the end I just made snapshot of the 104-disk-0, cloned it > back right after to 1044-disk-0 and then just change the reference in > lxc configuration. After that lxc started successfully. > > > I'm very wondering why this happened but am also very happy that > above simple steps saved my day. > > Hopefully this information helps somebody that will run into same > problem , but in the same time I truly hope that it won't happen :) > > > BR > > Tonci Stipicevic > > > > > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From gilberto.nunes32 at gmail.com Tue Jul 9 01:47:11 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Mon, 8 Jul 2019 20:47:11 -0300 Subject: [PVE-User] ixgbe Message-ID: Hi there We have some issues with driver ixgbe in Proxmox 5.4.10! Server is fully update but the NIC doesn't show any link at all! Somebody can help, please! Thanks a lot. --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From mityapetuhov at gmail.com Tue Jul 9 06:44:33 2019 From: mityapetuhov at gmail.com (Dmitry Petuhov) Date: Tue, 9 Jul 2019 07:44:33 +0300 Subject: [PVE-User] ixgbe In-Reply-To: References: Message-ID: <1f4034fb-6129-309e-1f57-4707bf5fb057@gmail.com> What command dmesg | grep ixgbe shows after boot? 09.07.2019 2:47, Gilberto Nunes ?????: > Hi there > > We have some issues with driver ixgbe in Proxmox 5.4.10! > Server is fully update but the NIC doesn't show any link at all! > Somebody can help, please! > Thanks a lot. > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From mark at openvs.co.uk Tue Jul 9 10:47:20 2019 From: mark at openvs.co.uk (Mark Adams) Date: Tue, 9 Jul 2019 09:47:20 +0100 Subject: [PVE-User] zfs 0.7 no discard Message-ID: Hi All, Currently having an issue on a few servers where more space is being "used" in the host (zfs), than is actually being used inside the VM. Discard is enabled, but zfs 0.7 does not have support for it. zfs 0.8 has brought in discard support, so I was wondering if anyone else has upgraded to it to resolve this kind of issue? I realise it's in proxmox 6, but I don't want to go there until it's out of beta. Any comments appreciated! Cheers, Mark From aheinlein at gmx.com Tue Jul 9 11:03:03 2019 From: aheinlein at gmx.com (Andreas Heinlein) Date: Tue, 9 Jul 2019 11:03:03 +0200 Subject: [PVE-User] pve-zsync broken? Message-ID: Hello, I have been trying to setup pve-zsync, with Host 2 (backup) pulling from Host 1 (origin). This worked for the first VM, but failed for the second because I misunderstood the maxsnap setting, which lead to out-of-space on Host 1 because of too many snapshots. I tried to cleanup everything and could delete the snapshots from Host 1, but not Host 2. I simply can't find the snapshots. I created a subvol "rpool/backup" on Host 2, which I can see with 'zfs list -t all'. But 'zfs list -t snapshot' only gives "No datasets found". 'zfs list -t all' also shows no snapshots. Shouldn't I at least be able to see the snapshots from the first - still enabled and working - job? Thanks, Andreas From tonci at suma-informatika.hr Tue Jul 9 12:34:03 2019 From: tonci at suma-informatika.hr (=?UTF-8?B?VG9uxI1pIFN0aXBpxI1ldmnEhw==?=) Date: Tue, 9 Jul 2019 12:34:03 +0200 Subject: [PVE-User] sharing zfs experience In-Reply-To: <20190708201918.00ca0f22@rosa.proxmox.com> References: <8678ddf7-98ef-f87b-bb16-09ad4953804e@suma-informatika.hr> <20190708201918.00ca0f22@rosa.proxmox.com> Message-ID: <649afeba-9626-6688-7967-3ee604eea2bb@suma-informatika.hr> Stoiko hi , thank you for your reply Now I'm even more worried after reading this recent thread you sent me :(? ...? I'm not sure any more what to expect after next reboot :) So the question is how to avoid such scenarios in the future ? ... My pools seems to be fully correct ... zpool status shows no errors at all. I think something went wrong on container level ... How come that disk-1 survived and disk-0 did not? I can send some reports like syslog? or something so just please tell me which ? Thank you very much in advance /srda?an pozdrav / best regards / Ton?i Stipi?evi?, dipl. ing. elektr. /direktor / manager/** ** d.o.o. ltd. *podr?ka / upravljanje **IT*/?sustavima za male i srednje tvrtke/ /Small & Medium Business /*IT*//*support / management* Badali?eva 27 / 10000 Zagreb / Hrvatska ? Croatia url: www.suma-informatika.hr mob: +385 91 1234003 fax: +385 1? 5560007 On 08. 07. 2019. 20:19, Stoiko Ivanov wrote: > hi, > Plus Hosting > > On Mon, 8 Jul 2019 18:50:07 +0200 > Ton?i Stipi?evi? wrote: > >> Hi to all, >> >> A customer of mine runs two clusters : >> >> 1. 2node with ibm v370 san as shared strage? (hared lvm) >> >> 2.? 3node cluster all nodes run zfs ...? no shared storage >> >> >> Couple days ago he had an power outage and during that period of time >> I was kind a worrying how apcupsd & proxmox will handle this >> situation. >> >> 1. Both nodes were properly shut down but one of 2 them dies , >> independent from power outage :) but just in the same time. I booted >> up remaining node , adjusted "votes" and started all vm-s residing on >> the shared lvm storage ...? No further questions ... prox handled >> that correctly >> >> 2. all 3 nodes started up but the most important lxc conteiner cloud >> not start. >> >> Reason: Job forpve-container at 104.service failed because the control >> process exited with error code. See "systemctl status >> pve-container at 104.service" and "journalctl -xe" for details. TASK >> ERROR: command 'systemctl start pve-container at 104' failed: exit code 1 >> >> Upgrading, restarting etc etc did not helped at all. The problem was >> that rootfs from this contaier? was completely empty ( it contained >> only /dev/ and /mnt/? dirs . Fortunately second mount point (aka 2nd >> disk) with 2T of data was pretty healthy and visible. So one option >> was to restore it from backup but zfs list command showed that this >> data set still holds data as much as it should (disk 0) > This somehow reminds me of a recent thread in the forum: > https://forum.proxmox.com/threads/reboot-of-pve-host-breaks-lxc-container-startup.55486/#post-255641 > > did the rpool get imported completely - or are there some errors in the > journal while the system booted? > > In any case - glad you manged to resolve the issue! > > >> root at pve01-hrz-zm:~# ls -al /rpool/data/subvol-104-disk-0/ >> total 10 >> drwxr-xr-x 4 root root 4 Srp? 4 14:07 . >> drwxr-xr-x 9 root root 9 Srp? 4 23:17 .. >> drwxr-xr-x 2 root root 2 Srp? 4 14:07 dev >> drwxr-xr-x 3 root root 3 Srp? 4 14:07 mnt >> >> root at pve01-hrz-zm:~# zfs list >> NAME??????????????????????????? USED? AVAIL? REFER? MOUNTPOINT >> rpool????????????????????????? 2,15T? 1,36T?? 104K? /rpool >> rpool/data???????????????????? 2,15T? 1,36T?? 128K? /rpool/data >> rpool/data/subvol-104-disk-0??? 751M? 15,3G?? 751M >> /rpool/data/subvol-104-disk-0 >> rpool/data/subvol-104-disk-1?? 2,15T?? 894G? 2,15T >> /rpool/data/subvol-104-disk-1 >> >> >> Interesting was that both lcx containers from this node had "empty" >> disk-0? (but the other one was not that big, it had only disk-0) and >> none of them could start. >> >> After many tries I decided to migrate this little container to other >> just to see what will happen :? migration was successfull and >> starting up as well .? OK (true relief finally :). then I tried to >> make backup of this vm just to see what will happen. No, backup was >> not successfull ... backup archive was only 1.7KB big. Ok, let's get >> back to migration scenario. So, the final conclusion was that >> migration itself was not the solution but snapshot was the right one. >> Snapshot was the step that revived this disk-0. >> >> So , at the end I just made snapshot of the 104-disk-0, cloned it >> back right after to 1044-disk-0 and then just change the reference in >> lxc configuration. After that lxc started successfully. >> >> >> I'm very wondering why this happened but am also very happy that >> above simple steps saved my day. >> >> Hopefully this information helps somebody that will run into same >> problem , but in the same time I truly hope that it won't happen :) >> >> >> BR >> >> Tonci Stipicevic >> >> >> >> >> >> >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From gilberto.nunes32 at gmail.com Tue Jul 9 12:59:11 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Tue, 9 Jul 2019 07:59:11 -0300 Subject: [PVE-User] ixgbe In-Reply-To: <1f4034fb-6129-309e-1f57-4707bf5fb057@gmail.com> References: <1f4034fb-6129-309e-1f57-4707bf5fb057@gmail.com> Message-ID: Hi there I have created the ixgbe.conf file into /etc/modprobe.d, with this: options ixgbe allow_unsupported_sfp=1.1 I haved compile the module from Intel source code... The worst thing is that this network are thousands of kilometers from my location... So a partner is check that network, looking for some issues in the switch, FC cable whatever... I am still waiting for him.... Any way, thanks for answer... I'ill keep you posted. Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em ter, 9 de jul de 2019 ?s 01:45, Dmitry Petuhov escreveu: > What command > > dmesg | grep ixgbe > > shows after boot? > > > 09.07.2019 2:47, Gilberto Nunes ?????: > > Hi there > > > > We have some issues with driver ixgbe in Proxmox 5.4.10! > > Server is fully update but the NIC doesn't show any link at all! > > Somebody can help, please! > > Thanks a lot. > > > > --- > > Gilberto Nunes Ferreira > > > > (47) 3025-5907 > > (47) 99676-7530 - Whatsapp / Telegram > > > > Skype: gilberto.nunes36 > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From guy at britewhite.net Tue Jul 9 13:12:13 2019 From: guy at britewhite.net (Guy Plunkett) Date: Tue, 9 Jul 2019 12:12:13 +0100 Subject: [PVE-User] ixgbe In-Reply-To: References: <1f4034fb-6129-309e-1f57-4707bf5fb057@gmail.com> Message-ID: <55A844F3-E3C2-47D2-A353-F225BBEEC51E@britewhite.net> I?m running the ixgbe interfaces as well. In 5.0 the ixgbe-options.conf changed slightly. Here?s a copy of mine: options ixgbe allow_unsupported_sfp=1 NOTE.. I've not compiled this driver.. it?s included in proxmox. So you might want to change the file and remove the .1 on the end. Thanks, --Guy > On 9 Jul 2019, at 11:59, Gilberto Nunes wrote: > > Hi there > > I have created the ixgbe.conf file into /etc/modprobe.d, with this: > > options ixgbe allow_unsupported_sfp=1.1 > I haved compile the module from Intel source code... > The worst thing is that this network are thousands of kilometers from my > location... > So a partner is check that network, looking for some issues in the switch, > FC cable whatever... > I am still waiting for him.... > Any way, thanks for answer... > I'ill keep you posted. > > Thanks > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em ter, 9 de jul de 2019 ?s 01:45, Dmitry Petuhov > escreveu: > >> What command >> >> dmesg | grep ixgbe >> >> shows after boot? >> >> >> 09.07.2019 2:47, Gilberto Nunes ?????: >>> Hi there >>> >>> We have some issues with driver ixgbe in Proxmox 5.4.10! >>> Server is fully update but the NIC doesn't show any link at all! >>> Somebody can help, please! >>> Thanks a lot. >>> >>> --- >>> Gilberto Nunes Ferreira >>> >>> (47) 3025-5907 >>> (47) 99676-7530 - Whatsapp / Telegram >>> >>> Skype: gilberto.nunes36 >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From gianni.milo22 at gmail.com Tue Jul 9 21:52:12 2019 From: gianni.milo22 at gmail.com (Gianni Milo) Date: Tue, 9 Jul 2019 20:52:12 +0100 Subject: [PVE-User] zfs 0.7 no discard In-Reply-To: References: Message-ID: Haven't moved to zfs 0.8 but you could have similar results by enabling compression on zfs and by periodically executing fstrim (linux) or sdelete (windows) inside VMs to reclaim unused space ? On Linux VMs, adding "discard" in fstab mount options (ext4) may have similar results as when executing fstrim. On Windows 10/2012+ VMs, defrag command can be used to perform free space consolidation. Make sure you enable "discard" option on the VM disk options tab (which you seem to have done already). Check this article on wiki as well... https://pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files Gianni On Tue, 9 Jul 2019 at 09:49, Mark Adams wrote: > Hi All, > > Currently having an issue on a few servers where more space is being "used" > in the host (zfs), than is actually being used inside the VM. Discard is > enabled, but zfs 0.7 does not have support for it. > > zfs 0.8 has brought in discard support, so I was wondering if anyone else > has upgraded to it to resolve this kind of issue? > > I realise it's in proxmox 6, but I don't want to go there until it's out of > beta. > > Any comments appreciated! > > Cheers, > Mark > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From mark at openvs.co.uk Tue Jul 9 22:00:00 2019 From: mark at openvs.co.uk (Mark Adams) Date: Tue, 9 Jul 2019 21:00:00 +0100 Subject: [PVE-User] zfs 0.7 no discard In-Reply-To: References: Message-ID: Hi GIanni, Thanks for your response, what I forgot to mention is I am also using zfs inside of the VM, which I think is the main problem - I need it to be 0.8 to actually send the discard/trim command to the underlying zfs... what I'm not clear on is if I need both to be 0.8. Also, I assume this can be done retrospectively ? so if I upgrade then I can run fstrim and it will clear the space in the host...? Maybe that question is better posed to the zfsonlinux list though... Cheers, Mark On Tue, 9 Jul 2019 at 20:53, Gianni Milo wrote: > Haven't moved to zfs 0.8 but you could have similar results by enabling > compression on zfs and by periodically executing fstrim (linux) or sdelete > (windows) inside VMs to reclaim unused space ? > > On Linux VMs, adding "discard" in fstab mount options (ext4) may have > similar results as when executing fstrim. > > On Windows 10/2012+ VMs, defrag command can be used to perform free space > consolidation. > > Make sure you enable "discard" option on the VM disk options tab (which you > seem to have done already). > > Check this article on wiki as well... > https://pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files > > Gianni > > > On Tue, 9 Jul 2019 at 09:49, Mark Adams wrote: > > > Hi All, > > > > Currently having an issue on a few servers where more space is being > "used" > > in the host (zfs), than is actually being used inside the VM. Discard is > > enabled, but zfs 0.7 does not have support for it. > > > > zfs 0.8 has brought in discard support, so I was wondering if anyone else > > has upgraded to it to resolve this kind of issue? > > > > I realise it's in proxmox 6, but I don't want to go there until it's out > of > > beta. > > > > Any comments appreciated! > > > > Cheers, > > Mark > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gianni.milo22 at gmail.com Tue Jul 9 22:18:04 2019 From: gianni.milo22 at gmail.com (Gianni Milo) Date: Tue, 9 Jul 2019 21:18:04 +0100 Subject: [PVE-User] zfs 0.7 no discard In-Reply-To: References: Message-ID: On that case, I would upgrade the VM to zfs 0.8, leaving the host on zfs 0.7 with compression enabled. Then I would run zfs trim command inside the VM and see if the space is reclaimed back on the host. Note: fstrim command only works on specific filesystems, not in zfs. Gianni On Tue, 9 Jul 2019 at 21:01, Mark Adams wrote: > Hi GIanni, > > Thanks for your response, what I forgot to mention is I am also using zfs > inside of the VM, which I think is the main problem - I need it to be 0.8 > to actually send the discard/trim command to the underlying zfs... what I'm > not clear on is if I need both to be 0.8. > > Also, I assume this can be done retrospectively ? so if I upgrade then I > can run fstrim and it will clear the space in the host...? Maybe that > question is better posed to the zfsonlinux list though... > > Cheers, > Mark > > On Tue, 9 Jul 2019 at 20:53, Gianni Milo wrote: > > > Haven't moved to zfs 0.8 but you could have similar results by enabling > > compression on zfs and by periodically executing fstrim (linux) or > sdelete > > (windows) inside VMs to reclaim unused space ? > > > > On Linux VMs, adding "discard" in fstab mount options (ext4) may have > > similar results as when executing fstrim. > > > > On Windows 10/2012+ VMs, defrag command can be used to perform free space > > consolidation. > > > > Make sure you enable "discard" option on the VM disk options tab (which > you > > seem to have done already). > > > > Check this article on wiki as well... > > https://pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files > > > > Gianni > > > > > > On Tue, 9 Jul 2019 at 09:49, Mark Adams wrote: > > > > > Hi All, > > > > > > Currently having an issue on a few servers where more space is being > > "used" > > > in the host (zfs), than is actually being used inside the VM. Discard > is > > > enabled, but zfs 0.7 does not have support for it. > > > > > > zfs 0.8 has brought in discard support, so I was wondering if anyone > else > > > has upgraded to it to resolve this kind of issue? > > > > > > I realise it's in proxmox 6, but I don't want to go there until it's > out > > of > > > beta. > > > > > > Any comments appreciated! > > > > > > Cheers, > > > Mark > > > _______________________________________________ > > > pve-user mailing list > > > pve-user at pve.proxmox.com > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From mark at openvs.co.uk Tue Jul 9 22:24:39 2019 From: mark at openvs.co.uk (Mark Adams) Date: Tue, 9 Jul 2019 21:24:39 +0100 Subject: [PVE-User] zfs 0.7 no discard In-Reply-To: References: Message-ID: Ah right! thanks for that info, I had assumed it would use fstrim. Looks like zpool trim is what I need to be running once upgraded. Cheers, Mark On Tue, 9 Jul 2019 at 21:19, Gianni Milo wrote: > On that case, I would upgrade the VM to zfs 0.8, leaving the host on zfs > 0.7 with compression enabled. > Then I would run zfs trim command inside the VM and see if the space is > reclaimed back on the host. > > Note: fstrim command only works on specific filesystems, not in zfs. > > Gianni > > > > On Tue, 9 Jul 2019 at 21:01, Mark Adams wrote: > > > Hi GIanni, > > > > Thanks for your response, what I forgot to mention is I am also using zfs > > inside of the VM, which I think is the main problem - I need it to be 0.8 > > to actually send the discard/trim command to the underlying zfs... what > I'm > > not clear on is if I need both to be 0.8. > > > > Also, I assume this can be done retrospectively ? so if I upgrade then I > > can run fstrim and it will clear the space in the host...? Maybe that > > question is better posed to the zfsonlinux list though... > > > > Cheers, > > Mark > > > > On Tue, 9 Jul 2019 at 20:53, Gianni Milo > wrote: > > > > > Haven't moved to zfs 0.8 but you could have similar results by enabling > > > compression on zfs and by periodically executing fstrim (linux) or > > sdelete > > > (windows) inside VMs to reclaim unused space ? > > > > > > On Linux VMs, adding "discard" in fstab mount options (ext4) may have > > > similar results as when executing fstrim. > > > > > > On Windows 10/2012+ VMs, defrag command can be used to perform free > space > > > consolidation. > > > > > > Make sure you enable "discard" option on the VM disk options tab (which > > you > > > seem to have done already). > > > > > > Check this article on wiki as well... > > > https://pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files > > > > > > Gianni > > > > > > > > > On Tue, 9 Jul 2019 at 09:49, Mark Adams wrote: > > > > > > > Hi All, > > > > > > > > Currently having an issue on a few servers where more space is being > > > "used" > > > > in the host (zfs), than is actually being used inside the VM. Discard > > is > > > > enabled, but zfs 0.7 does not have support for it. > > > > > > > > zfs 0.8 has brought in discard support, so I was wondering if anyone > > else > > > > has upgraded to it to resolve this kind of issue? > > > > > > > > I realise it's in proxmox 6, but I don't want to go there until it's > > out > > > of > > > > beta. > > > > > > > > Any comments appreciated! > > > > > > > > Cheers, > > > > Mark > > > > _____________________________________________ From devzero at web.de Wed Jul 10 11:49:33 2019 From: devzero at web.de (Roland @web.de) Date: Wed, 10 Jul 2019 11:49:33 +0200 Subject: [PVE-User] how to add existing partition as Volume Group ? Message-ID: Hello, i want to make remaining space of the two SSD boot disks available to Proxmox VMs, so i created a forth Partition on those of type "LVM". How can i make these available in Disks->LVM (i.e. proxmox managed LVM storage), as the webgui only lets select me whole disks as LVM storage ? I want to add LVM volumes on SSD to virtual machine to be used as ZFS Slog/L2Arc in virtual FreeNas. regards Roland From jmr.richardson at gmail.com Wed Jul 10 17:04:53 2019 From: jmr.richardson at gmail.com (JR Richardson) Date: Wed, 10 Jul 2019 10:04:53 -0500 Subject: [PVE-User] Intermittent NIC Connectivity after Host Upgrade to PVE 4.15.18-43 Message-ID: Hi All, I recently upgraded a host within a 4 Node Cluster from pve-manager/5.2-5/eb24855a (running kernel: 4.15.18-1-pve) to pve-manager/5.4-10/9603c337 (running kernel: 4.15.18-17-pve). I had several VMs running on the host node, some Win2016 Servers and Linux Servers. I migrated the VM off the host node, performed the upgrade, them migrated the VMs back to the upgraded host. All seemed to go well, next day I notices the Win2016 servers having intermittent network connectivity, Linux servers running fine. Performing continuous pings to Win2016 servers, the NIC were dropping for 45-60 seconds every hour or so, time was not consistent. I was using VitIO for NICs. I did windows update and this crashed the NICs, would not come up. I downloaded the latest virtio-win-0.1.171.iso and updated drivers, This did restore the NICs n the VMs but still have the network drops. I switched the VM NICs to Intel E1000 and still have the network drops. I don't see anything in the host node logs or the Win2016 VM logs indicating why the network drops periodically. Could this be a PVE kernel issue? I think this is more of a Win2016 issue, maybe the Intel NIC driver, its using the embedded Windows driver from 2010. Should I try downloading the latest driver from Intel? Dell R710 host node NICs: Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet (rev 20) Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) The NIC on the host for VMs is Broadcom. But the Linux hosts do not experience any drops so i don't think this is a host node or host kernel issue but I'm not sure. I appreciate any guidance. Thanks. JR -- JR Richardson Engineering for the Masses Chasing the Azeotrope From gianni.milo22 at gmail.com Wed Jul 10 18:46:01 2019 From: gianni.milo22 at gmail.com (Gianni Milo) Date: Wed, 10 Jul 2019 17:46:01 +0100 Subject: [PVE-User] Intermittent NIC Connectivity after Host Upgrade to PVE 4.15.18-43 In-Reply-To: References: Message-ID: Are you able to boot on a previous kernel and see if the problem returns ? Gianni On Wed, 10 Jul 2019 at 16:06, JR Richardson wrote: > Hi All, > > I recently upgraded a host within a 4 Node Cluster from > pve-manager/5.2-5/eb24855a (running kernel: 4.15.18-1-pve) to > pve-manager/5.4-10/9603c337 (running kernel: 4.15.18-17-pve). > > I had several VMs running on the host node, some Win2016 Servers and > Linux Servers. I migrated the VM off the host node, performed the > upgrade, them migrated the VMs back to the upgraded host. All seemed > to go well, next day I notices the Win2016 servers having intermittent > network connectivity, Linux servers running fine. Performing > continuous pings to Win2016 servers, the NIC were dropping for 45-60 > seconds every hour or so, time was not consistent. I was using VitIO > for NICs. > > I did windows update and this crashed the NICs, would not come up. I > downloaded the latest virtio-win-0.1.171.iso and updated drivers, This > did restore the NICs n the VMs but still have the network drops. I > switched the VM NICs to Intel E1000 and still have the network drops. > > I don't see anything in the host node logs or the Win2016 VM logs > indicating why the network drops periodically. Could this be a PVE > kernel issue? I think this is more of a Win2016 issue, maybe the Intel > NIC driver, its using the embedded Windows driver from 2010. Should I > try downloading the latest driver from Intel? > > Dell R710 host node NICs: > Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit > Ethernet (rev 20) > Ethernet controller: Intel Corporation 82576 Gigabit Network Connection > (rev 01) > > The NIC on the host for VMs is Broadcom. But the Linux hosts do not > experience any drops so i don't think this is a host node or host > kernel issue but I'm not sure. > > I appreciate any guidance. > > Thanks. > > JR > -- > JR Richardson > Engineering for the Masses > Chasing the Azeotrope > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gianni.milo22 at gmail.com Wed Jul 10 19:05:16 2019 From: gianni.milo22 at gmail.com (Gianni Milo) Date: Wed, 10 Jul 2019 18:05:16 +0100 Subject: [PVE-User] how to add existing partition as Volume Group ? In-Reply-To: References: Message-ID: Have you created a PV and VG to be able to actually use that space on PVE? Once you do that, you can add that VG to PVE storage providers. You could then add a LVM based disk, from that storage provider, to the Freenas VM. P.S Don't think that a virtualized Freenas is a good idea, especially for production use, but that's up to you ... Gianni From: Roland @web.de Date: Wed, 10 Jul 2019 at 10:49 Subject: [PVE-User] how to add existing partition as Volume Group ? To: PVE User List Hello, i want to make remaining space of the two SSD boot disks available to Proxmox VMs, so i created a forth Partition on those of type "LVM". How can i make these available in Disks->LVM (i.e. proxmox managed LVM storage), as the webgui only lets select me whole disks as LVM storage ? I want to add LVM volumes on SSD to virtual machine to be used as ZFS Slog/L2Arc in virtual FreeNas. regards Roland _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Sent from Gmail Mobile From jmr.richardson at gmail.com Thu Jul 11 14:00:05 2019 From: jmr.richardson at gmail.com (JR Richardson) Date: Thu, 11 Jul 2019 07:00:05 -0500 Subject: [PVE-User] Intermittent NIC Connectivity after Host Upgrade to PVE Message-ID: <001101d537e0$2dcfe920$896fbb60$@gmail.com> Oddly enough, there was another kernel update yesterday, so I updated and so far the intermittent network connectivity problem seems to have went away. Problem appears on kernel: 4.15.18-17-pve but so far not on kernel: 4.15.18-18-pve. The issue with the VirtIO NIC driver is still present on Win2016. Still have to run the Intel NIC on the VM. I will do some more testing today and follow up. Thanks. JR Are you able to boot on a previous kernel and see if the problem returns ? Gianni On Wed, 10 Jul 2019 at 16:06, JR Richardson wrote: > Hi All, > > I recently upgraded a host within a 4 Node Cluster from > pve-manager/5.2-5/eb24855a (running kernel: 4.15.18-1-pve) to > pve-manager/5.4-10/9603c337 (running kernel: 4.15.18-17-pve). > > I had several VMs running on the host node, some Win2016 Servers and > Linux Servers. I migrated the VM off the host node, performed the > upgrade, them migrated the VMs back to the upgraded host. All seemed > to go well, next day I notices the Win2016 servers having intermittent > network connectivity, Linux servers running fine. Performing > continuous pings to Win2016 servers, the NIC were dropping for 45-60 > seconds every hour or so, time was not consistent. I was using VitIO > for NICs. > > I did windows update and this crashed the NICs, would not come up. I > downloaded the latest virtio-win-0.1.171.iso and updated drivers, This > did restore the NICs n the VMs but still have the network drops. I > switched the VM NICs to Intel E1000 and still have the network drops. > > I don't see anything in the host node logs or the Win2016 VM logs > indicating why the network drops periodically. Could this be a PVE > kernel issue? I think this is more of a Win2016 issue, maybe the Intel > NIC driver, its using the embedded Windows driver from 2010. Should I > try downloading the latest driver from Intel? > > Dell R710 host node NICs: > Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit > Ethernet (rev 20) > Ethernet controller: Intel Corporation 82576 Gigabit Network Connection > (rev 01) > > The NIC on the host for VMs is Broadcom. But the Linux hosts do not > experience any drops so i don't think this is a host node or host > kernel issue but I'm not sure. > > I appreciate any guidance. > > Thanks. > > JR From wolfgang.bucher at netland-mn.de Thu Jul 11 14:42:45 2019 From: wolfgang.bucher at netland-mn.de (=?utf-8?Q?Wolfgang_Bucher?=) Date: Thu, 11 Jul 2019 14:42:45 +0200 Subject: [PVE-User] Intermittent NIC Connectivity after Host Upgrade to PVE Message-ID: Have you tried virtio drivers 0.1.141 latest stable Wolfgang -----Urspr?ngliche Nachricht----- Von: JR Richardson? Gesendet: Donnerstag 11 Juli 2019 14:02 An: pve-user at pve.proxmox.com Betreff: Re: [PVE-User] Intermittent NIC Connectivity after Host Upgrade to PVE Oddly enough, there was another kernel update yesterday, so I updated and so far the intermittent network connectivity problem seems to have went away. Problem appears on kernel: 4.15.18-17-pve but so far not on kernel: 4.15.18-18-pve. The issue with the VirtIO NIC driver is still present on Win2016. Still have to run the Intel NIC on the VM. I will do some more testing today and follow up. Thanks. JR Are you able to boot on a previous kernel and see if the problem returns ? Gianni On Wed, 10 Jul 2019 at 16:06, JR Richardson wrote: > Hi All, > > I recently upgraded a host within a 4 Node Cluster from > pve-manager/5.2-5/eb24855a (running kernel: 4.15.18-1-pve) to > pve-manager/5.4-10/9603c337 (running kernel: 4.15.18-17-pve). > > I had several VMs running on the host node, some Win2016 Servers and > Linux Servers. I migrated the VM off the host node, performed the > upgrade, them migrated the VMs back to the upgraded host. All seemed > to go well, next day I notices the Win2016 servers having intermittent > network connectivity, Linux servers running fine. Performing > continuous pings to Win2016 servers, the NIC were dropping for 45-60 > seconds every hour or so, time was not consistent. I was using VitIO > for NICs. > > I did windows update and this crashed the NICs, would not come up. I > downloaded the latest virtio-win-0.1.171.iso and updated drivers, This > did restore the NICs n the VMs but still have the network drops. I > switched the VM NICs to Intel E1000 and still have the network drops. > > I don't see anything in the host node logs or the Win2016 VM logs > indicating why the network drops periodically. Could this be a PVE > kernel issue? I think this is more of a Win2016 issue, maybe the Intel > NIC driver, its using the embedded Windows driver from 2010. Should I > try downloading the latest driver from Intel? > > Dell R710 host node NICs: > Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit > Ethernet (rev 20) > Ethernet controller: Intel Corporation 82576 Gigabit Network Connection > (rev 01) > > The NIC on the host for VMs is Broadcom. But the Linux hosts do not > experience any drops so i don't think this is a host node or host > kernel issue but I'm not sure. > > I appreciate any guidance. > > Thanks. > > JR _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From jmr.richardson at gmail.com Fri Jul 12 13:31:46 2019 From: jmr.richardson at gmail.com (JR Richardson) Date: Fri, 12 Jul 2019 06:31:46 -0500 Subject: [PVE-User] Intermittent NIC Connectivity after Host Upgrade to PVE Message-ID: <000701d538a5$63e578c0$2bb06a40$@gmail.com> Yes, I was using latest stable, them went to unstable, it worked for a bit until reboot of VM then virtio-win-0.1.171.iso also failed. Intel E1000 NIC seems to be stable with latest PVE kernel update. JR Have you tried virtio drivers 0.1.141 latest stable Wolfgang -----Urspr ngliche Nachricht----- Von: JR Richardson Gesendet: Donnerstag 11 Juli 2019 14:02 An: pve-user at pve.proxmox.com Betreff: Re: [PVE-User] Intermittent NIC Connectivity after Host Upgrade to PVE Oddly enough, there was another kernel update yesterday, so I updated and so far the intermittent network connectivity problem seems to have went away. Problem appears on kernel: 4.15.18-17-pve but so far not on kernel: 4.15.18-18-pve. The issue with the VirtIO NIC driver is still present on Win2016. Still have to run the Intel NIC on the VM. I will do some more testing today and follow up. Thanks. JR Are you able to boot on a previous kernel and see if the problem returns ? Gianni On Wed, 10 Jul 2019 at 16:06, JR Richardson wrote: > Hi All, > > I recently upgraded a host within a 4 Node Cluster from > pve-manager/5.2-5/eb24855a (running kernel: 4.15.18-1-pve) to > pve-manager/5.4-10/9603c337 (running kernel: 4.15.18-17-pve). > > I had several VMs running on the host node, some Win2016 Servers and > Linux Servers. I migrated the VM off the host node, performed the > upgrade, them migrated the VMs back to the upgraded host. All seemed > to go well, next day I notices the Win2016 servers having intermittent > network connectivity, Linux servers running fine. Performing > continuous pings to Win2016 servers, the NIC were dropping for 45-60 > seconds every hour or so, time was not consistent. I was using VitIO > for NICs. > > I did windows update and this crashed the NICs, would not come up. I > downloaded the latest virtio-win-0.1.171.iso and updated drivers, This > did restore the NICs n the VMs but still have the network drops. I > switched the VM NICs to Intel E1000 and still have the network drops. > > I don't see anything in the host node logs or the Win2016 VM logs > indicating why the network drops periodically. Could this be a PVE > kernel issue? I think this is more of a Win2016 issue, maybe the Intel > NIC driver, its using the embedded Windows driver from 2010. Should I > try downloading the latest driver from Intel? > > Dell R710 host node NICs: > Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit > Ethernet (rev 20) Ethernet controller: Intel Corporation 82576 Gigabit > Network Connection (rev 01) > > The NIC on the host for VMs is Broadcom. But the Linux hosts do not > experience any drops so i don't think this is a host node or host > kernel issue but I'm not sure. > > I appreciate any guidance. > > Thanks. > > JR From venefax at gmail.com Fri Jul 12 14:19:45 2019 From: venefax at gmail.com (Saint Michael) Date: Fri, 12 Jul 2019 08:19:45 -0400 Subject: [PVE-User] Intermittent NIC Connectivity after Host Upgrade to PVE In-Reply-To: <000701d538a5$63e578c0$2bb06a40$@gmail.com> References: <000701d538a5$63e578c0$2bb06a40$@gmail.com> Message-ID: I read the mention of kernel Linux proxmox 4.15.18-18-pve, but I just run the update and my kernel is still Linux proxmox 4.15.18-17-pve. I installation is licensed and I am supposed to have all the updates. I also see a bad problem with the network on Windows using virtio. I get 10% or less of the bandwidth of a physical virtual machine on the same network. On Fri, Jul 12, 2019 at 7:32 AM JR Richardson wrote: > Yes, I was using latest stable, them went to unstable, it worked for a bit > until reboot of VM then virtio-win-0.1.171.iso also failed. Intel E1000 NIC > seems to be stable with latest PVE kernel update. > > JR > > > > Have you tried virtio drivers 0.1.141 latest stable > > > > Wolfgang > > > > -----Urspr ngliche Nachricht----- > Von: JR Richardson > Gesendet: Donnerstag 11 Juli 2019 14:02 > An: pve-user at pve.proxmox.com > Betreff: Re: [PVE-User] Intermittent NIC Connectivity after Host Upgrade > to PVE > > > Oddly enough, there was another kernel update yesterday, so I updated and > so far the intermittent network connectivity problem seems to have went > away. > Problem appears on kernel: 4.15.18-17-pve but so far not on kernel: > 4.15.18-18-pve. > > The issue with the VirtIO NIC driver is still present on Win2016. Still > have to run the Intel NIC on the VM. > > I will do some more testing today and follow up. > > Thanks. > > JR > > > Are you able to boot on a previous kernel and see if the problem returns ? > > Gianni > > On Wed, 10 Jul 2019 at 16:06, JR Richardson > wrote: > > > Hi All, > > > > I recently upgraded a host within a 4 Node Cluster from > > pve-manager/5.2-5/eb24855a (running kernel: 4.15.18-1-pve) to > > pve-manager/5.4-10/9603c337 (running kernel: 4.15.18-17-pve). > > > > I had several VMs running on the host node, some Win2016 Servers and > > Linux Servers. I migrated the VM off the host node, performed the > > upgrade, them migrated the VMs back to the upgraded host. All seemed > > to go well, next day I notices the Win2016 servers having intermittent > > network connectivity, Linux servers running fine. Performing > > continuous pings to Win2016 servers, the NIC were dropping for 45-60 > > seconds every hour or so, time was not consistent. I was using VitIO > > for NICs. > > > > I did windows update and this crashed the NICs, would not come up. I > > downloaded the latest virtio-win-0.1.171.iso and updated drivers, This > > did restore the NICs n the VMs but still have the network drops. I > > switched the VM NICs to Intel E1000 and still have the network drops. > > > > I don't see anything in the host node logs or the Win2016 VM logs > > indicating why the network drops periodically. Could this be a PVE > > kernel issue? I think this is more of a Win2016 issue, maybe the Intel > > NIC driver, its using the embedded Windows driver from 2010. Should I > > try downloading the latest driver from Intel? > > > > Dell R710 host node NICs: > > Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit > > Ethernet (rev 20) Ethernet controller: Intel Corporation 82576 Gigabit > > Network Connection (rev 01) > > > > The NIC on the host for VMs is Broadcom. But the Linux hosts do not > > experience any drops so i don't think this is a host node or host > > kernel issue but I'm not sure. > > > > I appreciate any guidance. > > > > Thanks. > > > > JR > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From jmr.richardson at gmail.com Sat Jul 13 15:58:51 2019 From: jmr.richardson at gmail.com (JR Richardson) Date: Sat, 13 Jul 2019 08:58:51 -0500 Subject: [PVE-User] Intermittent NIC Connectivity after Host Upgrade to PVE Message-ID: <000a01d53983$1a660410$4f320c30$@gmail.com> You might want to update your packages directly and see if the kernel update is present, I'm using community repository and there is definitely a new kernel 4.15.18.-18-pve, since two days ago. I've been updating all my hosts nodes with it, but does still have the win2016 VirtIO NIC issue. So I'm switching all my Win2016 VMs to use the Intel E1000 NIC. JR I read the mention of kernel Linux proxmox 4.15.18-18-pve, but I just run the update and my kernel is still Linux proxmox 4.15.18-17-pve. I installation is licensed and I am supposed to have all the updates. I also see a bad problem with the network on Windows using virtio. I get 10% or less of the bandwidth of a physical virtual machine on the same network. On Fri, Jul 12, 2019 at 7:32 AM JR Richardson wrote: > Yes, I was using latest stable, them went to unstable, it worked for a > bit until reboot of VM then virtio-win-0.1.171.iso also failed. Intel > E1000 NIC seems to be stable with latest PVE kernel update. > > JR > > > > Have you tried virtio drivers 0.1.141 latest stable > > > > Wolfgang > > > > -----Urspr ngliche Nachricht----- > Von: JR Richardson > Gesendet: Donnerstag 11 Juli 2019 14:02 > An: pve-user at pve.proxmox.com > Betreff: Re: [PVE-User] Intermittent NIC Connectivity after Host > Upgrade to PVE > > > Oddly enough, there was another kernel update yesterday, so I updated > and so far the intermittent network connectivity problem seems to have > went away. > Problem appears on kernel: 4.15.18-17-pve but so far not on kernel: > 4.15.18-18-pve. > > The issue with the VirtIO NIC driver is still present on Win2016. > Still have to run the Intel NIC on the VM. > > I will do some more testing today and follow up. > > Thanks. > > JR > > > Are you able to boot on a previous kernel and see if the problem returns ? > > Gianni > > On Wed, 10 Jul 2019 at 16:06, JR Richardson > wrote: > > > Hi All, > > > > I recently upgraded a host within a 4 Node Cluster from > > pve-manager/5.2-5/eb24855a (running kernel: 4.15.18-1-pve) to > > pve-manager/5.4-10/9603c337 (running kernel: 4.15.18-17-pve). > > > > I had several VMs running on the host node, some Win2016 Servers and > > Linux Servers. I migrated the VM off the host node, performed the > > upgrade, them migrated the VMs back to the upgraded host. All seemed > > to go well, next day I notices the Win2016 servers having > > intermittent network connectivity, Linux servers running fine. > > Performing continuous pings to Win2016 servers, the NIC were > > dropping for 45-60 seconds every hour or so, time was not > > consistent. I was using VitIO for NICs. > > > > I did windows update and this crashed the NICs, would not come up. I > > downloaded the latest virtio-win-0.1.171.iso and updated drivers, > > This did restore the NICs n the VMs but still have the network > > drops. I switched the VM NICs to Intel E1000 and still have the network drops. > > > > I don't see anything in the host node logs or the Win2016 VM logs > > indicating why the network drops periodically. Could this be a PVE > > kernel issue? I think this is more of a Win2016 issue, maybe the > > Intel NIC driver, its using the embedded Windows driver from 2010. > > Should I try downloading the latest driver from Intel? > > > > Dell R710 host node NICs: > > Ethernet controller: Broadcom Limited NetXtreme II BCM5709 Gigabit > > Ethernet (rev 20) Ethernet controller: Intel Corporation 82576 > > Gigabit Network Connection (rev 01) > > > > The NIC on the host for VMs is Broadcom. But the Linux hosts do not > > experience any drops so i don't think this is a host node or host > > kernel issue but I'm not sure. > > > > I appreciate any guidance. > > > > Thanks. > > > > JR From martin at proxmox.com Tue Jul 16 13:19:15 2019 From: martin at proxmox.com (Martin Maurer) Date: Tue, 16 Jul 2019 13:19:15 +0200 Subject: [PVE-User] Proxmox VE 6.0 released! Message-ID: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> Hi all, We're excited to announce the final release of our Proxmox VE 6.0! It's based on the great Debian 10 codename "Buster" and the latest 5.0 Linux kernel, QEMU 4.0, LXC 3.1.0, ZFS 0.8.1, Ceph 14.2, Corosync 3.0, and more. This major release includes the latest Ceph Nautilus feautures and an improved Ceph management dashboard. We have updated the cluster communication stack to Corosync 3 using Kronosnet, and have a new selection widget for the network making it simple to select the correct link address in the cluster creation wizard. With ZFS 0.8.1 we have included TRIM support for SSDs and also support for native encryption with comfortable key-handling. The new installer supports ZFS root via UEFI, for example you can boot a ZFS mirror on NVMe SSDs (using systemd-boot instead of grub). And as always we have included countless bugfixes and improvements on a lot of places; see the release notes for all details. Release notes https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_6.0 Video intro https://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-6-0 Download https://www.proxmox.com/en/downloads Alternate ISO download: http://download.proxmox.com/iso/ Documentation https://pve.proxmox.com/pve-docs/ Community Forum https://forum.proxmox.com Source Code https://git.proxmox.com Bugtracker https://bugzilla.proxmox.com FAQ Q: Can I dist-upgrade Proxmox VE 5.4 to 6.0 with apt? A: Please follow the upgrade instructions exactly, as there is a major version bump of corosync (2.x to 3.x) https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 Q: Can I install Proxmox VE 6.0 on top of Debian Buster? A: Yes, see https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Buster Q: Can I upgrade my Proxmox VE 5.4 cluster with Ceph Luminous to 6.0 with Ceph Nautilus? A: This is a two step process. First, you have to upgrade Proxmox VE from 5.4 to 6.0, and afterwards upgrade Ceph from Luminous to Nautilus. There are a lot of improvements and changes, please follow exactly the upgrade documentation. https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus Q: Where can I get more information about future feature updates? A: Check our roadmap, forum, mailing list and subscribe to our newsletter. A big THANK YOU to our active community for all your feedback, testing, bug reporting and patch submitting! -- Best Regards, Martin Maurer Proxmox VE project leader martin at proxmox.com https://www.proxmox.com From lists at merit.unu.edu Tue Jul 16 13:59:53 2019 From: lists at merit.unu.edu (lists) Date: Tue, 16 Jul 2019 13:59:53 +0200 Subject: [PVE-User] Proxmox VE 6.0 released! In-Reply-To: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> References: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> Message-ID: Many congratulations! This definitely looks like a great release! We will checkout the new features in our test cluster! Thanks. MJ On 16-7-2019 13:19, Martin Maurer wrote: > Hi all, > > We're excited to announce the final release of our Proxmox VE 6.0! It's > based on the great Debian 10 codename "Buster" and the latest 5.0 Linux > kernel, QEMU 4.0, LXC 3.1.0, ZFS 0.8.1, Ceph 14.2, Corosync 3.0, and more. > > This major release includes the latest Ceph Nautilus feautures and an > improved Ceph management dashboard. We have updated the cluster > communication stack to Corosync 3 using Kronosnet, and have a new > selection widget for the network making it simple to select the correct > link address in the cluster creation wizard. > > With ZFS 0.8.1 we have included TRIM support for SSDs and also support > for native encryption with comfortable key-handling. > > The new installer supports ZFS root via UEFI, for example you can boot a > ZFS mirror on NVMe SSDs (using systemd-boot instead of grub). > > And as always we have included countless bugfixes and improvements on a > lot of places; see the release notes for all details. > > Release notes > https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_6.0 > > Video intro > https://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-6-0 > > > Download > https://www.proxmox.com/en/downloads > Alternate ISO download: > http://download.proxmox.com/iso/ > > Documentation > https://pve.proxmox.com/pve-docs/ > > Community Forum > https://forum.proxmox.com > > Source Code > https://git.proxmox.com > > Bugtracker > https://bugzilla.proxmox.com > > FAQ > Q: Can I dist-upgrade Proxmox VE 5.4 to 6.0 with apt? > A: Please follow the upgrade instructions exactly, as there is a major > version bump of corosync (2.x to 3.x) > https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 > > Q: Can I install Proxmox VE 6.0 on top of Debian Buster? > A: Yes, see > https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Buster > > Q: Can I upgrade my Proxmox VE 5.4 cluster with Ceph Luminous to 6.0 > with Ceph Nautilus? > A: This is a two step process. First, you have to upgrade Proxmox VE > from 5.4 to 6.0, and afterwards upgrade Ceph from Luminous to Nautilus. > There are a lot of improvements and changes, please follow exactly the > upgrade documentation. > https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 > https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus > > Q: Where can I get more information about future feature updates? > A: Check our roadmap, forum, mailing list and subscribe to our newsletter. > > A big THANK YOU to our active community for all your feedback, testing, > bug reporting and patch submitting! > From gilberto.nunes32 at gmail.com Tue Jul 16 14:34:36 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Tue, 16 Jul 2019 09:34:36 -0300 Subject: [PVE-User] Proxmox VE 6.0 released! In-Reply-To: References: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> Message-ID: Thanks to all Proxmox staff... You guys make a marvels job... I would like to know if qemu 4.X bring the Vm fault tolerance, like COLO ou microcheckpoint and if Proxmox will incorporeted that features in the next future! Thanks a lot --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em ter, 16 de jul de 2019 ?s 09:00, lists escreveu: > Many congratulations! > > This definitely looks like a great release! > > We will checkout the new features in our test cluster! > > Thanks. > > MJ > > On 16-7-2019 13:19, Martin Maurer wrote: > > Hi all, > > > > We're excited to announce the final release of our Proxmox VE 6.0! It's > > based on the great Debian 10 codename "Buster" and the latest 5.0 Linux > > kernel, QEMU 4.0, LXC 3.1.0, ZFS 0.8.1, Ceph 14.2, Corosync 3.0, and > more. > > > > This major release includes the latest Ceph Nautilus feautures and an > > improved Ceph management dashboard. We have updated the cluster > > communication stack to Corosync 3 using Kronosnet, and have a new > > selection widget for the network making it simple to select the correct > > link address in the cluster creation wizard. > > > > With ZFS 0.8.1 we have included TRIM support for SSDs and also support > > for native encryption with comfortable key-handling. > > > > The new installer supports ZFS root via UEFI, for example you can boot a > > ZFS mirror on NVMe SSDs (using systemd-boot instead of grub). > > > > And as always we have included countless bugfixes and improvements on a > > lot of places; see the release notes for all details. > > > > Release notes > > https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_6.0 > > > > Video intro > > > https://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-6-0 > > > > > > Download > > https://www.proxmox.com/en/downloads > > Alternate ISO download: > > http://download.proxmox.com/iso/ > > > > Documentation > > https://pve.proxmox.com/pve-docs/ > > > > Community Forum > > https://forum.proxmox.com > > > > Source Code > > https://git.proxmox.com > > > > Bugtracker > > https://bugzilla.proxmox.com > > > > FAQ > > Q: Can I dist-upgrade Proxmox VE 5.4 to 6.0 with apt? > > A: Please follow the upgrade instructions exactly, as there is a major > > version bump of corosync (2.x to 3.x) > > https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 > > > > Q: Can I install Proxmox VE 6.0 on top of Debian Buster? > > A: Yes, see > > https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Buster > > > > Q: Can I upgrade my Proxmox VE 5.4 cluster with Ceph Luminous to 6.0 > > with Ceph Nautilus? > > A: This is a two step process. First, you have to upgrade Proxmox VE > > from 5.4 to 6.0, and afterwards upgrade Ceph from Luminous to Nautilus. > > There are a lot of improvements and changes, please follow exactly the > > upgrade documentation. > > https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 > > https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus > > > > Q: Where can I get more information about future feature updates? > > A: Check our roadmap, forum, mailing list and subscribe to our > newsletter. > > > > A big THANK YOU to our active community for all your feedback, testing, > > bug reporting and patch submitting! > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From dietmar at proxmox.com Tue Jul 16 16:13:34 2019 From: dietmar at proxmox.com (Dietmar Maurer) Date: Tue, 16 Jul 2019 16:13:34 +0200 (CEST) Subject: [PVE-User] Proxmox VE 6.0 released! In-Reply-To: References: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> Message-ID: <565475394.59.1563286414870@webmail.proxmox.com> > I would like to know if qemu 4.X bring the Vm fault tolerance, like COLO ou > microcheckpoint and if Proxmox will incorporeted that features in the next > future! Those things are not stable yet ... From alain.pean at c2n.upsaclay.fr Tue Jul 16 16:38:43 2019 From: alain.pean at c2n.upsaclay.fr (=?UTF-8?Q?Alain_p=c3=a9an?=) Date: Tue, 16 Jul 2019 16:38:43 +0200 Subject: [PVE-User] Proxmox VE 6.0 released! In-Reply-To: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> References: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> Message-ID: <536a2271-3338-3e82-0fb6-3786ae65aa11@c2n.upsaclay.fr> Le 16/07/2019 ? 13:19, Martin Maurer a ?crit?: > We're excited to announce the final release of our Proxmox VE 6.0! > It's based on the great Debian 10 codename "Buster" and the latest 5.0 > Linux kernel, QEMU 4.0, LXC 3.1.0, ZFS 0.8.1, Ceph 14.2, Corosync 3.0, > and more. > > This major release includes the latest Ceph Nautilus feautures and an > improved Ceph management dashboard. We have updated the cluster > communication stack to Corosync 3 using Kronosnet, and have a new > selection widget for the network making it simple to select the > correct link address in the cluster creation wizard. > > With ZFS 0.8.1 we have included TRIM support for SSDs and also support > for native encryption with comfortable key-handling. > > The new installer supports ZFS root via UEFI, for example you can boot > a ZFS mirror on NVMe SSDs (using systemd-boot instead of grub). > > And as always we have included countless bugfixes and improvements on > a lot of places; see the release notes for all details. > > Release notes > https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_6.0 Hello Maurer Team, We see that Proxmox is under a continuous developpement, that's good news, but this version has a lot of changes, that make me fear a bit about an upgrade. We have a new version of Debian, a new version of corosync, icompatible with the old one, and a new version of Ceph, and I use Ceph. It's a lot... In the releases notes, I see : *ceph-disk has been removed*: After upgrading it is not possible to create new OSDs without upgrading to Ceph Nautilus. So it willbe mandatory to upgrade to Ceph Nautilus, in addition to the other changes ? Best Regards, Alain -- Administrateur Syst?me/R?seau C2N (ex LPN) Centre de Nanosciences et Nanotechnologies (UMR 9001) Avenue de La vauve, 91920 Palaiseau Tel : 01-70-27-06-88 From t.lamprecht at proxmox.com Tue Jul 16 16:57:29 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Tue, 16 Jul 2019 16:57:29 +0200 Subject: [PVE-User] Proxmox VE 6.0 released! In-Reply-To: <536a2271-3338-3e82-0fb6-3786ae65aa11@c2n.upsaclay.fr> References: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> <536a2271-3338-3e82-0fb6-3786ae65aa11@c2n.upsaclay.fr> Message-ID: On 7/16/19 4:38 PM, Alain p?an wrote: > *ceph-disk has been removed*: After upgrading it is not possible to create?new?OSDs?without?upgrading?to?Ceph?Nautilus. > > So it willbe mandatory to upgrade to Ceph Nautilus, in addition to the other?changes?? yes, if you upgrade to 6.x you will need to upgrade Ceph to Nautilus sooner or later. See: http://intranet.proxmox.com/index.php/Upgrade_from_5.x_to_6.0 http://intranet.proxmox.com/index.php/Ceph_Luminous_to_Nautilus We upgraded internal production clusters and a lot of test setups without issues, so I think it should be dooable, especially if tried out in a test setup first. cheers, Thomas From t.lamprecht at proxmox.com Tue Jul 16 17:00:28 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Tue, 16 Jul 2019 17:00:28 +0200 Subject: [PVE-User] Proxmox VE 6.0 released! In-Reply-To: References: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> <536a2271-3338-3e82-0fb6-3786ae65aa11@c2n.upsaclay.fr> Message-ID: <82831686-04f2-a7c0-5a36-829c62c21631@proxmox.com> On 7/16/19 4:57 PM, Thomas Lamprecht wrote: > On 7/16/19 4:38 PM, Alain p?an wrote: >> *ceph-disk has been removed*: After upgrading it is not possible to create?new?OSDs?without?upgrading?to?Ceph?Nautilus. >> >> So it willbe mandatory to upgrade to Ceph Nautilus, in addition to the other?changes?? > > yes, if you upgrade to 6.x you will need to upgrade Ceph to Nautilus sooner or later. > > See: > > http://intranet.proxmox.com/index.php/Upgrade_from_5.x_to_6.0 > http://intranet.proxmox.com/index.php/Ceph_Luminous_to_Nautilus sorry, meant: https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus From adamw at matrixscience.com Tue Jul 16 17:16:51 2019 From: adamw at matrixscience.com (Adam Weremczuk) Date: Tue, 16 Jul 2019 16:16:51 +0100 Subject: [PVE-User] running Buster CT on 5.4.6 Message-ID: <8bb5435c-b6fb-e198-b679-8150273d092a@matrixscience.com> Hi all, I've just deployed a test Debian 10.0 container on PVE 5.4.6 from the default template. It installed fine, network is working ok across the LAN and I can ssh to it. Regardless whether I disable IPv6 or not (net.ipv6.conf.ens4.disable_ipv6 = 1) I'm getting the following errors: ping 8.8.8.8 connect: Network is unreachable ping google.com connect: Cannot assign requested address host google.com google.com has address 172.217.169.46 (DNS working fine) I've never had such problems for any out of the box Debian 9 containers. Any idea what's wrong and how to fix it? Thanks, Adam From alain.pean at c2n.upsaclay.fr Tue Jul 16 17:37:05 2019 From: alain.pean at c2n.upsaclay.fr (=?UTF-8?Q?Alain_p=c3=a9an?=) Date: Tue, 16 Jul 2019 17:37:05 +0200 Subject: [PVE-User] Proxmox VE 6.0 released! In-Reply-To: References: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> <536a2271-3338-3e82-0fb6-3786ae65aa11@c2n.upsaclay.fr> Message-ID: <270a902a-1540-813b-89f6-3e95dcc17e8a@c2n.upsaclay.fr> Le 16/07/2019 ? 16:57, Thomas Lamprecht a ?crit?: > We upgraded internal production clusters and a lot of test setups > without issues, so I think it should be dooable, especially if tried > out in a test setup first. Thanks for the answer Thomas, I shall indeed test carefully on a test cluster. But the problem is that I have one still in filestore, and the other in bluestore, so perhaps, I shall have to migrate all to bluestore in a first step... Alain -- Administrateur Syst?me/R?seau C2N (ex LPN) Centre de Nanosciences et Nanotechnologies (UMR 9001) Avenue de La vauve, 91920 Palaiseau Tel : 01-70-27-06-88 From r.correa.r at gmail.com Tue Jul 16 21:28:21 2019 From: r.correa.r at gmail.com (Ricardo Correa) Date: Tue, 16 Jul 2019 21:28:21 +0200 Subject: [PVE-User] Corosync Upgrade Issue (v2 -> v3) Message-ID: Hello all, While following the instructions for upgrade I encountered the following issue: ~# systemctl stop pve-ha-lrm ~# systemctl stop pve-ha-crm ~# echo "deb http://download.proxmox.com/debian/corosync-3/ stretch main" > /etc/apt/sources.list.d/corosync3.list ~# apt update ...snip... ~# apt dist-upgrade --download-only ...snip... ~# apt dist-upgrade Reading package lists... Done ...snip... The following packages will be upgraded: corosync libcmap4 libcorosync-common4 libcpg4 libqb0 libquorum5 libvotequorum8 ...snip... Setting up libcorosync-common4:amd64 (3.0.2-pve2~bpo9) ... Processing triggers for pve-ha-manager (2.0-9) ... Processing triggers for man-db (2.7.6.1-2) ... Setting up libqb0:amd64 (1.0.5-1~bpo9+2) ... systemctl start pve-ha-lrmJob for pve-cluster.service failed because a timeout was exceeded. See "systemctl status pve-cluster.service" and "journalctl -xe" for details. dpkg: error processing package libqb0:amd64 (--configure): subprocess installed post-installation script returned error exit status 1 Setting up libknet1:amd64 (1.10-pve1~bpo9) ... dpkg: dependency problems prevent configuration of libvotequorum8:amd64: libvotequorum8:amd64 depends on libqb0 (>= 1.0.3); however: Package libqb0:amd64 is not configured yet. dpkg: error processing package libvotequorum8:amd64 (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of libcpg4:amd64: libcpg4:amd64 depends on libqb0 (>= 1.0.3); however: Package libqb0:amd64 is not configured yet. dpkg: error processing package libcpg4:amd64 (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of libcmap4:amd64: libcmap4:amd64 depends on libqb0 (>= 1.0.3); however: Package libqb0:amd64 is not configured yet. dpkg: error processing package libcmap4:amd64 (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of libquorum5:amd64: libquorum5:amd64 depends on libqb0 (>= 1.0.3); however: Package libqb0:amd64 is not configured yet. dpkg: error processing package libquorum5:amd64 (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of libcfg7:amd64: libcfg7:amd64 depends on libqb0 (>= 1.0.3); however: Package libqb0:amd64 is not configured yet. dpkg: error processing package libcfg7:amd64 (--configure): dependency problems - leaving unconfigured dpkg: dependency problems prevent configuration of corosync: corosync depends on libcfg7 (>= 2.99.5); however: Package libcfg7:amd64 is not configured yet. corosync depends on libcmap4 (>= 2.99.3); however: Package libcmap4:amd64 is not configured yet. corosync depends on libcpg4 (>= 1.1.0); however: Package libcpg4:amd64 is not configured yet. corosync depends on libqb0 (>= 1.0.3); however: Package libqb0:amd64 is not configured yet. corosync depends on libquorum5 (>= 1.99.9); however: Package libquorum5:amd64 is not configured yet. corosync depends on libvotequorum8 (>= 2.4.2); however: Package libvotequorum8:amd64 is not configured yet. dpkg: error processing package corosync (--configure): dependency problems - leaving unconfigured Processing triggers for libc-bin (2.24-11+deb9u4) ... Processing triggers for pve-ha-manager (2.0-9) ... Processing triggers for pve-manager (5.4-11) ... Errors were encountered while processing: libqb0:amd64 libvotequorum8:amd64 libcpg4:amd64 libcmap4:amd64 libquorum5:amd64 libcfg7:amd64 corosync E: Sub-process /usr/bin/dpkg returned an error code (1) W: Operation was interrupted before it could finish This was while updating all 3 of my nodes at the same time (taking second route of the upgrade instructions, i.e., tmuxing into all nodes while having the main tmux syncying commands). Only one of those nodes failed to update, other two are running fine right now and in quorum apparently (pvecm status says quorate: yes). Right now I'm not sure how to proceed, pve-cluster in the failing node is looping with: systemd[1]: Starting The Proxmox VE cluster filesystem... systemd[1]: pve-cluster.service: Start operation timed out. Terminating. pmxcfs[13267]: [main] crit: read error: Interrupted system call pmxcfs[13267]: [main] crit: read error: Interrupted system call pmxcfs[13267]: [main] notice: exit proxmox configuration filesystem (-1) systemd[1]: pve-cluster.service: Control process exited, code=exited status=255 systemd[1]: Failed to start The Proxmox VE cluster filesystem. systemd[1]: pve-cluster.service: Unit entered failed state. systemd[1]: pve-cluster.service: Failed with result 'timeout'. systemd[1]: pve-cluster.service: Service hold-off time over, scheduling restart. systemd[1]: Stopped The Proxmox VE cluster filesystem. And pmxcfs is unavailable (ls /etc/pve timeouts). Any help is appreciated! (BTW, just joined the ML to post this, also joined the forums but can't post). Best regards, Ricardo From f.gruenbichler at proxmox.com Tue Jul 16 21:55:39 2019 From: f.gruenbichler at proxmox.com (=?UTF-8?Q?Fabian_Gr=C3=BCnbichler?=) Date: Tue, 16 Jul 2019 21:55:39 +0200 (CEST) Subject: [PVE-User] Corosync Upgrade Issue (v2 -> v3) In-Reply-To: References: Message-ID: <1048846865.82.1563306939140@webmail.proxmox.com> > Ricardo Correa hat am 16. Juli 2019 21:28 geschrieben: > > > Hello all, > > While following the instructions for upgrade I encountered the following > issue: > > ~# systemctl stop pve-ha-lrm > ~# systemctl stop pve-ha-crm > ~# echo "deb http://download.proxmox.com/debian/corosync-3/ stretch main" > > /etc/apt/sources.list.d/corosync3.list > ~# apt update > ...snip... > ~# apt dist-upgrade --download-only > ...snip... > ~# apt dist-upgrade > Reading package lists... Done > ...snip... > The following packages will be upgraded: > corosync libcmap4 libcorosync-common4 libcpg4 libqb0 libquorum5 > libvotequorum8 > ...snip... please provide "unsnipped" logs, both of these commands and from the journal around the same time. > Setting up libcorosync-common4:amd64 (3.0.2-pve2~bpo9) ... > Processing triggers for pve-ha-manager (2.0-9) ... > Processing triggers for man-db (2.7.6.1-2) ... > Setting up libqb0:amd64 (1.0.5-1~bpo9+2) ... > systemctl start pve-ha-lrmJob for pve-cluster.service failed because a > timeout was exceeded. > See "systemctl status pve-cluster.service" and "journalctl -xe" for details. > dpkg: error processing package libqb0:amd64 (--configure): > subprocess installed post-installation script returned error exit status 1 > Setting up libknet1:amd64 (1.10-pve1~bpo9) ... > dpkg: dependency problems prevent configuration of libvotequorum8:amd64: > libvotequorum8:amd64 depends on libqb0 (>= 1.0.3); however: > Package libqb0:amd64 is not configured yet. > > dpkg: error processing package libvotequorum8:amd64 (--configure): > dependency problems - leaving unconfigured > dpkg: dependency problems prevent configuration of libcpg4:amd64: > libcpg4:amd64 depends on libqb0 (>= 1.0.3); however: > Package libqb0:amd64 is not configured yet. > > dpkg: error processing package libcpg4:amd64 (--configure): > dependency problems - leaving unconfigured > dpkg: dependency problems prevent configuration of libcmap4:amd64: > libcmap4:amd64 depends on libqb0 (>= 1.0.3); however: > Package libqb0:amd64 is not configured yet. > > dpkg: error processing package libcmap4:amd64 (--configure): > dependency problems - leaving unconfigured > dpkg: dependency problems prevent configuration of libquorum5:amd64: > libquorum5:amd64 depends on libqb0 (>= 1.0.3); however: > Package libqb0:amd64 is not configured yet. > > dpkg: error processing package libquorum5:amd64 (--configure): > dependency problems - leaving unconfigured > dpkg: dependency problems prevent configuration of libcfg7:amd64: > libcfg7:amd64 depends on libqb0 (>= 1.0.3); however: > Package libqb0:amd64 is not configured yet. > > dpkg: error processing package libcfg7:amd64 (--configure): > dependency problems - leaving unconfigured > dpkg: dependency problems prevent configuration of corosync: > corosync depends on libcfg7 (>= 2.99.5); however: > Package libcfg7:amd64 is not configured yet. > corosync depends on libcmap4 (>= 2.99.3); however: > Package libcmap4:amd64 is not configured yet. > corosync depends on libcpg4 (>= 1.1.0); however: > Package libcpg4:amd64 is not configured yet. > corosync depends on libqb0 (>= 1.0.3); however: > Package libqb0:amd64 is not configured yet. > corosync depends on libquorum5 (>= 1.99.9); however: > Package libquorum5:amd64 is not configured yet. > corosync depends on libvotequorum8 (>= 2.4.2); however: > Package libvotequorum8:amd64 is not configured yet. > > dpkg: error processing package corosync (--configure): > dependency problems - leaving unconfigured > Processing triggers for libc-bin (2.24-11+deb9u4) ... > Processing triggers for pve-ha-manager (2.0-9) ... > Processing triggers for pve-manager (5.4-11) ... if you followed the upgrade guidelines and ONLY upgraded corosync here, these two triggers should not be triggered... > Errors were encountered while processing: > libqb0:amd64 > libvotequorum8:amd64 > libcpg4:amd64 > libcmap4:amd64 > libquorum5:amd64 > libcfg7:amd64 > corosync > E: Sub-process /usr/bin/dpkg returned an error code (1) > W: Operation was interrupted before it could finish > > This was while updating all 3 of my nodes at the same time (taking second > route > of the upgrade instructions, i.e., tmuxing into all nodes while having the > main > tmux syncying commands). Only one of those nodes failed to update, other > two are > running fine right now and in quorum apparently (pvecm status says quorate: > yes). > > Right now I'm not sure how to proceed, pve-cluster in the failing node > is looping with: > > systemd[1]: Starting The Proxmox VE cluster filesystem... > systemd[1]: pve-cluster.service: Start operation timed out. Terminating. > pmxcfs[13267]: [main] crit: read error: Interrupted system call > pmxcfs[13267]: [main] crit: read error: Interrupted system call > pmxcfs[13267]: [main] notice: exit proxmox configuration filesystem (-1) > systemd[1]: pve-cluster.service: Control process exited, code=exited > status=255 > systemd[1]: Failed to start The Proxmox VE cluster filesystem. > systemd[1]: pve-cluster.service: Unit entered failed state. > systemd[1]: pve-cluster.service: Failed with result 'timeout'. > systemd[1]: pve-cluster.service: Service hold-off time over, scheduling > restart. > systemd[1]: Stopped The Proxmox VE cluster filesystem. > > And pmxcfs is unavailable (ls /etc/pve timeouts). > > Any help is appreciated! (BTW, just joined the ML to post this, also joined > the > forums but can't post). > > Best regards, > Ricardo > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From r.correa.r at gmail.com Tue Jul 16 22:37:45 2019 From: r.correa.r at gmail.com (Ricardo Correa) Date: Tue, 16 Jul 2019 22:37:45 +0200 Subject: [PVE-User] Corosync Upgrade Issue (v2 -> v3) In-Reply-To: <1048846865.82.1563306939140@webmail.proxmox.com> References: <1048846865.82.1563306939140@webmail.proxmox.com> Message-ID: Hi Fabian, Thanks for replying. Here are the complete logs: root at pve01:~# systemctl stop pve-ha-lrm root at pve01:~# systemctl stop pve-ha-crm root at pve01:~# echo "deb http://download.proxmox.com/debian/corosync-3/ stretch main" > /etc/apt/sources.list.d/corosync3.list root at pve01:~# apt update Ign:1 http://ftp.ch.debian.org/debian stretch InRelease Hit:2 http://ftp.ch.debian.org/debian stretch-updates InRelease Hit:3 http://ftp.ch.debian.org/debian stretch Release Get:4 http://download.proxmox.com/debian/corosync-3 stretch InRelease [1,977 B] Hit:5 http://download.proxmox.com/debian/ceph-luminous stretch InRelease Hit:6 http://download.proxmox.com/debian/pve stretch InRelease Hit:7 http://security.debian.org stretch/updates InRelease Get:9 http://download.proxmox.com/debian/corosync-3 stretch/main amd64 Packages [12.4 kB] Fetched 14.3 kB in 0s (33.0 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done 7 packages can be upgraded. Run 'apt list --upgradable' to see them. root at pve01:~# apt list --upgradeable Listing... Done corosync/stable 3.0.2-pve2~bpo9 amd64 [upgradable from: 2.4.4-pve1] libcmap4/stable 3.0.2-pve2~bpo9 amd64 [upgradable from: 2.4.4-pve1] libcorosync-common4/stable 3.0.2-pve2~bpo9 amd64 [upgradable from: 2.4.4-pve1] libcpg4/stable 3.0.2-pve2~bpo9 amd64 [upgradable from: 2.4.4-pve1] libqb0/stable 1.0.5-1~bpo9+2 amd64 [upgradable from: 1.0.3-1~bpo9] libquorum5/stable 3.0.2-pve2~bpo9 amd64 [upgradable from: 2.4.4-pve1] libvotequorum8/stable 3.0.2-pve2~bpo9 amd64 [upgradable from: 2.4.4-pve1] root at pve01:~# apt dist-upgrade --download-only Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following NEW packages will be installed: libcfg7 libknet1 libzstd1 The following packages will be upgraded: corosync libcmap4 libcorosync-common4 libcpg4 libqb0 libquorum5 libvotequorum8 7 upgraded, 3 newly installed, 0 to remove and 0 not upgraded. Need to get 2,405 kB of archives. After this operation, 1,671 kB of additional disk space will be used. Do you want to continue? [Y/n] Get:1 http://ftp.ch.debian.org/debian stretch/main amd64 libzstd1 amd64 1.1.2-1 [193 kB] Get:2 http://download.proxmox.com/debian/corosync-3 stretch/main amd64 libcorosync-common4 amd64 3.0.2-pve2~bpo9 [244 kB] Get:3 http://download.proxmox.com/debian/corosync-3 stretch/main amd64 libqb0 amd64 1.0.5-1~bpo9+2 [140 kB] Get:4 http://download.proxmox.com/debian/corosync-3 stretch/main amd64 libcfg7 amd64 3.0.2-pve2~bpo9 [247 kB] Get:5 http://download.proxmox.com/debian/corosync-3 stretch/main amd64 libcmap4 amd64 3.0.2-pve2~bpo9 [248 kB] Get:6 http://download.proxmox.com/debian/corosync-3 stretch/main amd64 libcpg4 amd64 3.0.2-pve2~bpo9 [250 kB] Get:7 http://download.proxmox.com/debian/corosync-3 stretch/main amd64 libknet1 amd64 1.10-pve1~bpo9 [129 kB] Get:8 http://download.proxmox.com/debian/corosync-3 stretch/main amd64 libquorum5 amd64 3.0.2-pve2~bpo9 [246 kB] Get:9 http://download.proxmox.com/debian/corosync-3 stretch/main amd64 libvotequorum8 amd64 3.0.2-pve2~bpo9 [247 kB] Get:10 http://download.proxmox.com/debian/corosync-3 stretch/main amd64 corosync amd64 3.0.2-pve2~bpo9 [461 kB] Fetched 2,405 kB in 0s (2,434 kB/s) Download complete and in download only mode root at pve01:~# apt dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following NEW packages will be installed: libcfg7 libknet1 libzstd1 The following packages will be upgraded: corosync libcmap4 libcorosync-common4 libcpg4 libqb0 libquorum5 libvotequorum8 7 upgraded, 3 newly installed, 0 to remove and 0 not upgraded. Need to get 0 B/2,405 kB of archives. After this operation, 1,671 kB of additional disk space will be used. Do you want to continue? [Y/n] Reading changelogs... Done (Reading database ... 59056 files and directories currently installed.) Preparing to unpack .../0-libcorosync-common4_3.0.2-pve2~bpo9_amd64.deb ... Unpacking libcorosync-common4:amd64 (3.0.2-pve2~bpo9) over (2.4.4-pve1) ... Preparing to unpack .../1-libqb0_1.0.5-1~bpo9+2_amd64.deb ... Unpacking libqb0:amd64 (1.0.5-1~bpo9+2) over (1.0.3-1~bpo9) ... Selecting previously unselected package libcfg7:amd64. Preparing to unpack .../2-libcfg7_3.0.2-pve2~bpo9_amd64.deb ... Unpacking libcfg7:amd64 (3.0.2-pve2~bpo9) ... Preparing to unpack .../3-libcmap4_3.0.2-pve2~bpo9_amd64.deb ... Unpacking libcmap4:amd64 (3.0.2-pve2~bpo9) over (2.4.4-pve1) ... Preparing to unpack .../4-libcpg4_3.0.2-pve2~bpo9_amd64.deb ... Unpacking libcpg4:amd64 (3.0.2-pve2~bpo9) over (2.4.4-pve1) ... Selecting previously unselected package libzstd1. Preparing to unpack .../5-libzstd1_1.1.2-1_amd64.deb ... Unpacking libzstd1 (1.1.2-1) ... Selecting previously unselected package libknet1:amd64. Preparing to unpack .../6-libknet1_1.10-pve1~bpo9_amd64.deb ... Unpacking libknet1:amd64 (1.10-pve1~bpo9) ... Preparing to unpack .../7-libquorum5_3.0.2-pve2~bpo9_amd64.deb ... Unpacking libquorum5:amd64 (3.0.2-pve2~bpo9) over (2.4.4-pve1) ... Preparing to unpack .../8-libvotequorum8_3.0.2-pve2~bpo9_amd64.deb ... Unpacking libvotequorum8:amd64 (3.0.2-pve2~bpo9) over (2.4.4-pve1) ... Preparing to unpack .../9-corosync_3.0.2-pve2~bpo9_amd64.deb ... Unpacking corosync (3.0.2-pve2~bpo9) over (2.4.4-pve1) ... Processing triggers for pve-manager (5.4-11) ... Setting up libzstd1 (1.1.2-1) ... Errors were encountered while processing: libqb0:amd64 libvotequorum8:amd64 libcpg4:amd64 libcmap4:amd64 libquorum5:amd64 libcfg7:amd64 corosync E: Sub-process /usr/bin/dpkg returned an error code (1) W: Operation was interrupted before it could finish I do see three new packages being installed, is it possible these caused it? libcfg7 libknet1 libzstd1 I'm just looking at these and they are dependencies of corosync (3) and each other so it couldn't have been these alone. Kind regards, Ricardo Correa On Tue, Jul 16, 2019 at 9:55 PM Fabian Gr?nbichler < f.gruenbichler at proxmox.com> wrote: > > Ricardo Correa hat am 16. Juli 2019 21:28 > geschrieben: > > > > > > Hello all, > > > > While following the instructions for upgrade I encountered the following > > issue: > > > > ~# systemctl stop pve-ha-lrm > > ~# systemctl stop pve-ha-crm > > ~# echo "deb http://download.proxmox.com/debian/corosync-3/ stretch > main" > > > /etc/apt/sources.list.d/corosync3.list > > ~# apt update > > ...snip... > > ~# apt dist-upgrade --download-only > > ...snip... > > ~# apt dist-upgrade > > Reading package lists... Done > > ...snip... > > The following packages will be upgraded: > > corosync libcmap4 libcorosync-common4 libcpg4 libqb0 libquorum5 > > libvotequorum8 > > ...snip... > > please provide "unsnipped" logs, both of these commands and from the > journal around the same time. > > > Setting up libcorosync-common4:amd64 (3.0.2-pve2~bpo9) ... > > Processing triggers for pve-ha-manager (2.0-9) ... > > Processing triggers for man-db (2.7.6.1-2) ... > > Setting up libqb0:amd64 (1.0.5-1~bpo9+2) ... > > systemctl start pve-ha-lrmJob for pve-cluster.service failed because a > > timeout was exceeded. > > See "systemctl status pve-cluster.service" and "journalctl -xe" for > details. > > dpkg: error processing package libqb0:amd64 (--configure): > > subprocess installed post-installation script returned error exit > status 1 > > Setting up libknet1:amd64 (1.10-pve1~bpo9) ... > > dpkg: dependency problems prevent configuration of libvotequorum8:amd64: > > libvotequorum8:amd64 depends on libqb0 (>= 1.0.3); however: > > Package libqb0:amd64 is not configured yet. > > > > dpkg: error processing package libvotequorum8:amd64 (--configure): > > dependency problems - leaving unconfigured > > dpkg: dependency problems prevent configuration of libcpg4:amd64: > > libcpg4:amd64 depends on libqb0 (>= 1.0.3); however: > > Package libqb0:amd64 is not configured yet. > > > > dpkg: error processing package libcpg4:amd64 (--configure): > > dependency problems - leaving unconfigured > > dpkg: dependency problems prevent configuration of libcmap4:amd64: > > libcmap4:amd64 depends on libqb0 (>= 1.0.3); however: > > Package libqb0:amd64 is not configured yet. > > > > dpkg: error processing package libcmap4:amd64 (--configure): > > dependency problems - leaving unconfigured > > dpkg: dependency problems prevent configuration of libquorum5:amd64: > > libquorum5:amd64 depends on libqb0 (>= 1.0.3); however: > > Package libqb0:amd64 is not configured yet. > > > > dpkg: error processing package libquorum5:amd64 (--configure): > > dependency problems - leaving unconfigured > > dpkg: dependency problems prevent configuration of libcfg7:amd64: > > libcfg7:amd64 depends on libqb0 (>= 1.0.3); however: > > Package libqb0:amd64 is not configured yet. > > > > dpkg: error processing package libcfg7:amd64 (--configure): > > dependency problems - leaving unconfigured > > dpkg: dependency problems prevent configuration of corosync: > > corosync depends on libcfg7 (>= 2.99.5); however: > > Package libcfg7:amd64 is not configured yet. > > corosync depends on libcmap4 (>= 2.99.3); however: > > Package libcmap4:amd64 is not configured yet. > > corosync depends on libcpg4 (>= 1.1.0); however: > > Package libcpg4:amd64 is not configured yet. > > corosync depends on libqb0 (>= 1.0.3); however: > > Package libqb0:amd64 is not configured yet. > > corosync depends on libquorum5 (>= 1.99.9); however: > > Package libquorum5:amd64 is not configured yet. > > corosync depends on libvotequorum8 (>= 2.4.2); however: > > Package libvotequorum8:amd64 is not configured yet. > > > > dpkg: error processing package corosync (--configure): > > dependency problems - leaving unconfigured > > Processing triggers for libc-bin (2.24-11+deb9u4) ... > > Processing triggers for pve-ha-manager (2.0-9) ... > > Processing triggers for pve-manager (5.4-11) ... > > if you followed the upgrade guidelines and ONLY upgraded corosync here, > these two triggers should not be triggered... > > > Errors were encountered while processing: > > libqb0:amd64 > > libvotequorum8:amd64 > > libcpg4:amd64 > > libcmap4:amd64 > > libquorum5:amd64 > > libcfg7:amd64 > > corosync > > E: Sub-process /usr/bin/dpkg returned an error code (1) > > W: Operation was interrupted before it could finish > > > > This was while updating all 3 of my nodes at the same time (taking second > > route > > of the upgrade instructions, i.e., tmuxing into all nodes while having > the > > main > > tmux syncying commands). Only one of those nodes failed to update, other > > two are > > running fine right now and in quorum apparently (pvecm status says > quorate: > > yes). > > > > Right now I'm not sure how to proceed, pve-cluster in the failing node > > is looping with: > > > > systemd[1]: Starting The Proxmox VE cluster filesystem... > > systemd[1]: pve-cluster.service: Start operation timed out. Terminating. > > pmxcfs[13267]: [main] crit: read error: Interrupted system call > > pmxcfs[13267]: [main] crit: read error: Interrupted system call > > pmxcfs[13267]: [main] notice: exit proxmox configuration filesystem (-1) > > systemd[1]: pve-cluster.service: Control process exited, code=exited > > status=255 > > systemd[1]: Failed to start The Proxmox VE cluster filesystem. > > systemd[1]: pve-cluster.service: Unit entered failed state. > > systemd[1]: pve-cluster.service: Failed with result 'timeout'. > > systemd[1]: pve-cluster.service: Service hold-off time over, scheduling > > restart. > > systemd[1]: Stopped The Proxmox VE cluster filesystem. > > > > And pmxcfs is unavailable (ls /etc/pve timeouts). > > > > Any help is appreciated! (BTW, just joined the ML to post this, also > joined > > the > > forums but can't post). > > > > Best regards, > > Ricardo > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From chris.hofstaedtler at deduktiva.com Tue Jul 16 23:26:58 2019 From: chris.hofstaedtler at deduktiva.com (Chris Hofstaedtler | Deduktiva) Date: Tue, 16 Jul 2019 23:26:58 +0200 Subject: [PVE-User] Corosync Upgrade Issue (v2 -> v3) In-Reply-To: <1048846865.82.1563306939140@webmail.proxmox.com> References: <1048846865.82.1563306939140@webmail.proxmox.com> Message-ID: <20190716212657.btpdgtxu62zwv3n2@tx.local> * Fabian Gr?nbichler [190716 21:55]: [..] > > > > dpkg: error processing package corosync (--configure): > > dependency problems - leaving unconfigured > > Processing triggers for libc-bin (2.24-11+deb9u4) ... > > Processing triggers for pve-ha-manager (2.0-9) ... > > Processing triggers for pve-manager (5.4-11) ... > > if you followed the upgrade guidelines and ONLY upgraded corosync here, these two triggers should not be triggered... I've done a corosync-only upgrade the other day and the triggers are indeed triggered. dpkg.log (truncated): 2019-07-07 21:14:45 startup archives unpack 2019-07-07 21:14:46 upgrade libcorosync-common4:amd64 2.4.4-pve1 3.0.2-pve2~bpo9 2019-07-07 21:14:46 status triggers-pending libc-bin:amd64 2.24-11+deb9u4 2019-07-07 21:14:46 status half-configured libcorosync-common4:amd64 2.4.4-pve1 2019-07-07 21:14:46 status unpacked libcorosync-common4:amd64 2.4.4-pve1 2019-07-07 21:14:46 status half-installed libcorosync-common4:amd64 2.4.4-pve1 2019-07-07 21:14:46 status half-installed libcorosync-common4:amd64 2.4.4-pve1 2019-07-07 21:14:46 status unpacked libcorosync-common4:amd64 3.0.2-pve2~bpo9 2019-07-07 21:14:46 status unpacked libcorosync-common4:amd64 3.0.2-pve2~bpo9 2019-07-07 21:14:46 upgrade libqb0:amd64 1.0.3-1~bpo9 1.0.5-1~bpo9+2 2019-07-07 21:14:46 status triggers-pending pve-ha-manager:amd64 2.0-9 2019-07-07 21:14:46 status triggers-pending pve-manager:amd64 5.4-10 ... full dpkg.log: https://gist.github.com/zeha/9d47a95776d375d6f386b89c5be4a35a Chris From tonci at suma-informatika.hr Wed Jul 17 01:54:51 2019 From: tonci at suma-informatika.hr (=?UTF-8?B?VG9uxI1pIFN0aXBpxI1ldmnEhw==?=) Date: Wed, 17 Jul 2019 01:54:51 +0200 Subject: [PVE-User] lxc pve-zsync In-Reply-To: <8678ddf7-98ef-f87b-bb16-09ad4953804e@suma-informatika.hr> References: <8678ddf7-98ef-f87b-bb16-09ad4953804e@suma-informatika.hr> Message-ID: <4bc5f882-10ad-e2f9-6831-a8db0798057f@suma-informatika.hr> Hello to all, ??? I need to backup one lxc (rootfs+mp0) to another zfs pve-backup host , but it seems that only rootfs gets synced to another site (?!) and not mp0 pve-zsync job fits my needs at the moment unlike storage-repl so this command will transfer only rootfs : pve-zsync sync --source 105 --dest 192.168.71.53:rpool/bck --verbose arch: amd64 cores: 4 hostname: TK-DC01 memory: 4096 mp0: zfs01:subvol-105-disk-1,mp=/mnt/data1,size=8G nameserver: 8.8.8.8 net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.71.1,hwaddr=96:BD:EC:CC:B0:93,ip=192.168.71.252/2$ ostype: debian rootfs: zfs01:subvol-105-disk-0,size=16G searchdomain: lin-dc01.net.local swap: 512 Any help would be very appreciated Thank very much you in advance BR Tonci > > > > > From t.lamprecht at proxmox.com Wed Jul 17 08:10:11 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Wed, 17 Jul 2019 08:10:11 +0200 Subject: [PVE-User] Corosync Upgrade Issue (v2 -> v3) In-Reply-To: <20190716212657.btpdgtxu62zwv3n2@tx.local> References: <1048846865.82.1563306939140@webmail.proxmox.com> <20190716212657.btpdgtxu62zwv3n2@tx.local> Message-ID: On 7/16/19 11:26 PM, Chris Hofstaedtler | Deduktiva wrote: > * Fabian Gr?nbichler [190716 21:55]: > [..] >>> >>> dpkg: error processing package corosync (--configure): >>> dependency problems - leaving unconfigured >>> Processing triggers for libc-bin (2.24-11+deb9u4) ... >>> Processing triggers for pve-ha-manager (2.0-9) ... >>> Processing triggers for pve-manager (5.4-11) ... >> >> if you followed the upgrade guidelines and ONLY upgraded corosync here, these two triggers should not be triggered... > > I've done a corosync-only upgrade the other day and the triggers are > indeed triggered. Yes, and that's correct. We actively trigger the pve-cluster (pmxcfs) service and the "pve-api-updates" trigger - which then trigger manager and ha-manager. Se, and the in-lined commit message for details (basically new libqb is incompat, as new corosync is freshly stared it load the new libqb, pmxcfs has still the old loaded and cannot communicate anymore, thus it needs to be restarted, and subsequently all daemons using IPCC calls to pmxcfs): https://git.proxmox.com/?p=libqb.git;a=commitdiff;h=5abd5865b8d2d0cf245e4b3085a08fb22bf6e7fd cheers, Thomas > > dpkg.log (truncated): > > 2019-07-07 21:14:45 startup archives unpack > 2019-07-07 21:14:46 upgrade libcorosync-common4:amd64 2.4.4-pve1 3.0.2-pve2~bpo9 > 2019-07-07 21:14:46 status triggers-pending libc-bin:amd64 2.24-11+deb9u4 > 2019-07-07 21:14:46 status half-configured libcorosync-common4:amd64 2.4.4-pve1 > 2019-07-07 21:14:46 status unpacked libcorosync-common4:amd64 2.4.4-pve1 > 2019-07-07 21:14:46 status half-installed libcorosync-common4:amd64 2.4.4-pve1 > 2019-07-07 21:14:46 status half-installed libcorosync-common4:amd64 2.4.4-pve1 > 2019-07-07 21:14:46 status unpacked libcorosync-common4:amd64 3.0.2-pve2~bpo9 > 2019-07-07 21:14:46 status unpacked libcorosync-common4:amd64 3.0.2-pve2~bpo9 > 2019-07-07 21:14:46 upgrade libqb0:amd64 1.0.3-1~bpo9 1.0.5-1~bpo9+2 > 2019-07-07 21:14:46 status triggers-pending pve-ha-manager:amd64 2.0-9 > 2019-07-07 21:14:46 status triggers-pending pve-manager:amd64 5.4-10 > ... > > full dpkg.log: > https://gist.github.com/zeha/9d47a95776d375d6f386b89c5be4a35a > > > Chris > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From t.lamprecht at proxmox.com Wed Jul 17 08:22:19 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Wed, 17 Jul 2019 08:22:19 +0200 Subject: [PVE-User] Corosync Upgrade Issue (v2 -> v3) In-Reply-To: References: Message-ID: On 7/16/19 9:28 PM, Ricardo Correa wrote: > systemd[1]: Starting The Proxmox VE cluster filesystem... > systemd[1]: pve-cluster.service: Start operation timed out. Terminating. > pmxcfs[13267]: [main] crit: read error: Interrupted system call That's strange, that an error happening initially at the startup code, after we fork the child process which becomes the daemon doing the actual work we wait in the parent for it to be read, for that a simple pipe is used from where a byte is read, your read is getting interrupted - something which normally should not happen.. Can you try to start in it the foreground: # systemctl stop pve-cluster # pmxcfs -f and see what happens their Also you probably should try to finish the upgrade: # apt -f install From f.gruenbichler at proxmox.com Wed Jul 17 09:27:13 2019 From: f.gruenbichler at proxmox.com (Fabian =?utf-8?Q?Gr=C3=BCnbichler?=) Date: Wed, 17 Jul 2019 09:27:13 +0200 Subject: [PVE-User] Corosync Upgrade Issue (v2 -> v3) In-Reply-To: References: <1048846865.82.1563306939140@webmail.proxmox.com> <20190716212657.btpdgtxu62zwv3n2@tx.local> Message-ID: <20190717072713.mf7pjqfbsoiwoet4@nora.maurer-it.com> On Wed, Jul 17, 2019 at 08:10:11AM +0200, Thomas Lamprecht wrote: > On 7/16/19 11:26 PM, Chris Hofstaedtler | Deduktiva wrote: > > * Fabian Gr?nbichler [190716 21:55]: > > [..] > >>> > >>> dpkg: error processing package corosync (--configure): > >>> dependency problems - leaving unconfigured > >>> Processing triggers for libc-bin (2.24-11+deb9u4) ... > >>> Processing triggers for pve-ha-manager (2.0-9) ... > >>> Processing triggers for pve-manager (5.4-11) ... > >> > >> if you followed the upgrade guidelines and ONLY upgraded corosync here, these two triggers should not be triggered... > > > > I've done a corosync-only upgrade the other day and the triggers are > > indeed triggered. > > Yes, and that's correct. We actively trigger the pve-cluster (pmxcfs) service > and the "pve-api-updates" trigger - which then trigger manager and ha-manager. > > Se, and the in-lined commit message for details (basically new libqb is incompat, > as new corosync is freshly stared it load the new libqb, pmxcfs has still the old > loaded and cannot communicate anymore, thus it needs to be restarted, and > subsequently all daemons using IPCC calls to pmxcfs): > > https://git.proxmox.com/?p=libqb.git;a=commitdiff;h=5abd5865b8d2d0cf245e4b3085a08fb22bf6e7fd sorry for the wrong info - I indeed only checked Buster packages for triggers, not the special Corosync 3.x Stretch packages.. From r.correa.r at gmail.com Wed Jul 17 09:31:11 2019 From: r.correa.r at gmail.com (Ricardo Correa) Date: Wed, 17 Jul 2019 09:31:11 +0200 Subject: [PVE-User] Corosync Upgrade Issue (v2 -> v3) In-Reply-To: References: Message-ID: On Wed, Jul 17, 2019 at 8:22 AM Thomas Lamprecht wrote: > On 7/16/19 9:28 PM, Ricardo Correa wrote: > > systemd[1]: Starting The Proxmox VE cluster filesystem... > > systemd[1]: pve-cluster.service: Start operation timed out. Terminating. > > pmxcfs[13267]: [main] crit: read error: Interrupted system call > > That's strange, that an error happening initially at the startup code, > after we fork the child process which becomes the daemon doing the actual > work we wait in the parent for it to be read, for that a simple pipe is > used from where a byte is read, your read is getting interrupted - > something > which normally should not happen.. > > Can you try to start in it the foreground: > # systemctl stop pve-cluster > # pmxcfs -f > > and see what happens their > > Also you probably should try to finish the upgrade: > # apt -f install > > That did it! There was a message I haven't seen before though: root at pve01:~# apt -f install Reading package lists... Done Building dependency tree Reading state information... Done 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 8 not fully installed or removed. After this operation, 0 B of additional disk space will be used. Setting up pve-manager (5.4-11) ... Setting up libqb0:amd64 (1.0.5-1~bpo9+2) ... Setting up libvotequorum8:amd64 (3.0.2-pve2~bpo9) ... Setting up libcpg4:amd64 (3.0.2-pve2~bpo9) ... Setting up libcmap4:amd64 (3.0.2-pve2~bpo9) ... Setting up libquorum5:amd64 (3.0.2-pve2~bpo9) ... Setting up libcfg7:amd64 (3.0.2-pve2~bpo9) ... Setting up corosync (3.0.2-pve2~bpo9) ... Installing new version of config file /etc/default/corosync ... Processing triggers for pve-ha-manager (2.0-9) ... Processing triggers for pve-manager (5.4-11) ... Processing triggers for libc-bin (2.24-11+deb9u4) ... W: APT had planned for dpkg to do more than it reported back (18 vs 25). Affected packages: libqb0:amd64 pve-manager:amd64 That "APT had planned for dpkg to do more than it reported back (18 vs 25)", could it be due to me actually trying to apt -f install many times before I had stopped the pve-cluster and started pmxcfs by hand? From lists at merit.unu.edu Wed Jul 17 12:47:32 2019 From: lists at merit.unu.edu (mj) Date: Wed, 17 Jul 2019 12:47:32 +0200 Subject: [PVE-User] adding ceph osd nodes Message-ID: Hi, We are running a three-node licensed hyper-converged proxmox cluster with ceph storage. Question: is it possible to add some extra ceph OSD storage nodes, without proxmox virtualisation, and thus without the need to purchase additional proxmox licenses? Anyone doing that? We are wondering for example if the extra mon nodes & OSDs would show up in the pve gui. Of course we could setup a test cluster and simply try it out, but is anyone doing this? Any reasons why we should / should not consider this? MJ From t.lamprecht at proxmox.com Wed Jul 17 13:05:05 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Wed, 17 Jul 2019 13:05:05 +0200 Subject: [PVE-User] adding ceph osd nodes In-Reply-To: References: Message-ID: On 7/17/19 12:47 PM, mj wrote: > Question: is it possible to add some extra ceph OSD storage nodes, without proxmox virtualisation, and thus without the need to purchase additional?proxmox?licenses? > > Anyone?doing?that? > > We are wondering for example if the extra mon nodes & OSDs would show up in?the?pve?gui. Most of our data comes from directly talking to the monitors / ceph cluster over RADOS, so the ceph point of view is our point of view. You may need to do a bit more manual work in setting the others "external nodes" up, though. Copying authkey(s), bootstrap keyring, initial config... From t.lamprecht at proxmox.com Wed Jul 17 14:45:57 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Wed, 17 Jul 2019 14:45:57 +0200 Subject: [PVE-User] Proxmox VE 6.0 released! In-Reply-To: <270a902a-1540-813b-89f6-3e95dcc17e8a@c2n.upsaclay.fr> References: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> <536a2271-3338-3e82-0fb6-3786ae65aa11@c2n.upsaclay.fr> <270a902a-1540-813b-89f6-3e95dcc17e8a@c2n.upsaclay.fr> Message-ID: On 7/16/19 5:37 PM, Alain p?an wrote: > I shall indeed test carefully on a test cluster. But the problem is that I have one still in filestore, and the other in bluestore, so perhaps, I shall?have?to?migrate?all?to?bluestore?in?a?first?step... You can still use Filestore backed Clusters, you cannot use our tooling to add new Filestore backed OSDs (but you could use ceph-volume for that), see: https://pve.proxmox.com/pve-docs/chapter-pveceph.html#_ceph_filestore So there's no direct immediate need to upgrade them, although I'd setup new OSDs with bluestore, as this will be more future proof. cheers, Thomas From a.antreich at proxmox.com Wed Jul 17 14:47:45 2019 From: a.antreich at proxmox.com (Alwin Antreich) Date: Wed, 17 Jul 2019 14:47:45 +0200 Subject: [PVE-User] adding ceph osd nodes In-Reply-To: References: Message-ID: <20190717124745.GD80377@dona.proxmox.com> On Wed, Jul 17, 2019 at 12:47:32PM +0200, mj wrote: > Hi, > > We are running a three-node licensed hyper-converged proxmox cluster with > ceph storage. > > Question: is it possible to add some extra ceph OSD storage nodes, without > proxmox virtualisation, and thus without the need to purchase additional > proxmox licenses? I like to add, though not explicitly asked. While it is technically possible, the cluster will lose its enterprise support. As Ceph is under support on Proxmox VE nodes too. Better split out Ceph to a separate cluster and use the Proxmox VE nodes as clients. To keep the Ceph client and cluster package versions close together, you could use Proxmox VE without subscription for the separate Ceph cluster. -- Cheers, Alwin From t.lamprecht at proxmox.com Wed Jul 17 14:52:21 2019 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Wed, 17 Jul 2019 14:52:21 +0200 Subject: [PVE-User] running Buster CT on 5.4.6 In-Reply-To: <8bb5435c-b6fb-e198-b679-8150273d092a@matrixscience.com> References: <8bb5435c-b6fb-e198-b679-8150273d092a@matrixscience.com> Message-ID: Hi, On 7/16/19 5:16 PM, Adam Weremczuk wrote: > I've just deployed a test Debian 10.0 container on PVE 5.4.6 from the default template. > > It installed fine, network is working ok across the LAN and I can ssh to it. > > Regardless whether I disable IPv6 or not (net.ipv6.conf.ens4.disable_ipv6 = 1) I'm getting the following errors: > > ping 8.8.8.8 > connect: Network is unreachable Hmm, strange, works just fine here (tested on both PVE 6.0 and 5.4). > > ping google.com > connect: Cannot assign requested address > > host google.com > google.com has address 172.217.169.46 > (DNS working fine) > > I've never had such problems for any out of the box Debian 9 containers. > > Any idea what's wrong and how to fix it? Any firewall setup? Also, can you post the # ip addr # ip route outputs from inside the CT? cheers, Thomas From adamw at matrixscience.com Wed Jul 17 16:00:07 2019 From: adamw at matrixscience.com (Adam Weremczuk) Date: Wed, 17 Jul 2019 15:00:07 +0100 Subject: [PVE-User] running Buster CT on 5.4.6 In-Reply-To: References: <8bb5435c-b6fb-e198-b679-8150273d092a@matrixscience.com> Message-ID: On 17/07/19 13:52, Thomas Lamprecht wrote: > Hi, > > On 7/16/19 5:16 PM, Adam Weremczuk wrote: >> I've just deployed a test Debian 10.0 container on PVE 5.4.6 from the default template. >> >> It installed fine, network is working ok across the LAN and I can ssh to it. >> >> Regardless whether I disable IPv6 or not (net.ipv6.conf.ens4.disable_ipv6 = 1) I'm getting the following errors: >> >> ping 8.8.8.8 >> connect: Network is unreachable > Hmm, strange, works just fine here (tested on both PVE 6.0 and 5.4). > >> ping google.com >> connect: Cannot assign requested address >> >> host google.com >> google.com has address 172.217.169.46 >> (DNS working fine) >> >> I've never had such problems for any out of the box Debian 9 containers. >> >> Any idea what's wrong and how to fix it? > Any firewall setup? Also, can you post the > > # ip addr > # ip route > > outputs from inside the CT? > > cheers, > Thomas > Hi Thomas, I probably typed self IP into gateway field when spawning CT as it was set to that. I had to change it in Proxmox as any changes made to local /etc/network/interfaces were overwritten on reboot. Everything is working fine now. Thanks, Adam From gilberto.nunes32 at gmail.com Wed Jul 17 18:32:09 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 17 Jul 2019 13:32:09 -0300 Subject: [PVE-User] PVE 5.4 and Intel ixgbe Message-ID: Hi there everybody! I have installed PVE 5.4 and try to up ixgbe driver for Intel 10GB SFI/SPF+ NIC... I already do ixgbe-options.conf with options ixgbe allow_unsupported_sfp=1 But when try to load the module I still get this error: [ 170.008236] ixgbe 0000:05:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected. [ 170.008262] ixgbe 0000:05:00.0: Reload the driver after installing a supported module. [ 170.022268] ixgbe 0000:05:00.1: failed to load because an unsupported SFP+ or QSFP module type was detected. [ 170.022291] ixgbe 0000:05:00.1: Reload the driver after installing a supported module. I already try to compile Intel module from scratch, but seems to failed too! Thanks for any help! lspci 08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 08:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) pveversion pve-manager/5.4-11/6df3d8d0 (running kernel: 4.15.18-18-pve) --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From alex at calicolabs.com Wed Jul 17 19:44:07 2019 From: alex at calicolabs.com (Alex Chekholko) Date: Wed, 17 Jul 2019 10:44:07 -0700 Subject: [PVE-User] PVE 5.4 and Intel ixgbe In-Reply-To: References: Message-ID: You can try 'modinfo ixgbe' to query your actual installed version to see all the parameters it knows about. I see on one of my hosts # modinfo ixgbe filename: /lib/modules/4.15.0-54-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko version: 5.1.0-k ... parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ modules on 82599-based adapters (uint) And you can check the exact release notes for your version to see the allowed values for that parameter. IME, you may have some kind of incompable cable/optic anyway; see if you can try a different one. Regards, Alex On Wed, Jul 17, 2019 at 9:33 AM Gilberto Nunes wrote: > Hi there everybody! > > I have installed PVE 5.4 and try to up ixgbe driver for Intel 10GB SFI/SPF+ > NIC... > > I already do ixgbe-options.conf with > > options ixgbe allow_unsupported_sfp=1 > > But when try to load the module I still get this error: > > [ 170.008236] ixgbe 0000:05:00.0: failed to load because an unsupported > SFP+ or QSFP module type was detected. > > [ 170.008262] ixgbe 0000:05:00.0: Reload the driver after installing a > supported module. > > [ 170.022268] ixgbe 0000:05:00.1: failed to load because an unsupported > SFP+ or QSFP module type was detected. > > [ 170.022291] ixgbe 0000:05:00.1: Reload the driver after installing a > supported module. > > I already try to compile Intel module from scratch, but seems to failed > too! > > Thanks for any help! > > lspci > > 08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ > Network Connection (rev 01) > 08:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ > Network Connection (rev 01) > > > pveversion > > pve-manager/5.4-11/6df3d8d0 (running kernel: 4.15.18-18-pve) > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Wed Jul 17 19:53:27 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 17 Jul 2019 14:53:27 -0300 Subject: [PVE-User] PVE 5.4 and Intel ixgbe In-Reply-To: References: Message-ID: I am not sure about it, because now, after a fresh installation, even de enp4sf0 appears... This is so frustrated! --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 17 de jul de 2019 ?s 14:45, Alex Chekholko via pve-user < pve-user at pve.proxmox.com> escreveu: > > > > ---------- Forwarded message ---------- > From: Alex Chekholko > To: PVE User List > Cc: > Bcc: > Date: Wed, 17 Jul 2019 10:44:07 -0700 > Subject: Re: [PVE-User] PVE 5.4 and Intel ixgbe > You can try 'modinfo ixgbe' to query your actual installed version to see > all the parameters it knows about. > > I see on one of my hosts > # modinfo ixgbe > filename: > > /lib/modules/4.15.0-54-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko > version: 5.1.0-k > ... > parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ > modules on 82599-based adapters (uint) > > And you can check the exact release notes for your version to see the > allowed values for that parameter. > > IME, you may have some kind of incompable cable/optic anyway; see if you > can try a different one. > > Regards, > Alex > > On Wed, Jul 17, 2019 at 9:33 AM Gilberto Nunes > > wrote: > > > Hi there everybody! > > > > I have installed PVE 5.4 and try to up ixgbe driver for Intel 10GB > SFI/SPF+ > > NIC... > > > > I already do ixgbe-options.conf with > > > > options ixgbe allow_unsupported_sfp=1 > > > > But when try to load the module I still get this error: > > > > [ 170.008236] ixgbe 0000:05:00.0: failed to load because an unsupported > > SFP+ or QSFP module type was detected. > > > > [ 170.008262] ixgbe 0000:05:00.0: Reload the driver after installing a > > supported module. > > > > [ 170.022268] ixgbe 0000:05:00.1: failed to load because an unsupported > > SFP+ or QSFP module type was detected. > > > > [ 170.022291] ixgbe 0000:05:00.1: Reload the driver after installing a > > supported module. > > > > I already try to compile Intel module from scratch, but seems to failed > > too! > > > > Thanks for any help! > > > > lspci > > > > 08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > SFI/SFP+ > > Network Connection (rev 01) > > 08:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > SFI/SFP+ > > Network Connection (rev 01) > > > > > > pveversion > > > > pve-manager/5.4-11/6df3d8d0 (running kernel: 4.15.18-18-pve) > > --- > > Gilberto Nunes Ferreira > > > > (47) 3025-5907 > > (47) 99676-7530 - Whatsapp / Telegram > > > > Skype: gilberto.nunes36 > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > > > ---------- Forwarded message ---------- > From: Alex Chekholko via pve-user > To: PVE User List > Cc: Alex Chekholko > Bcc: > Date: Wed, 17 Jul 2019 10:44:07 -0700 > Subject: Re: [PVE-User] PVE 5.4 and Intel ixgbe > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gilberto.nunes32 at gmail.com Wed Jul 17 20:06:04 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 17 Jul 2019 15:06:04 -0300 Subject: [PVE-User] PVE 5.4 and Intel ixgbe In-Reply-To: References: Message-ID: So here the scenario... The customer (which is in other city, far away from my current localtion!) has the Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection but the GBIC is CISCO and the DAC cable is Mikrotik... Should CISCO GBIC incompatible with Intel card?? Or maybe the DAC cable from Mikrotik?? --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 17 de jul de 2019 ?s 14:53, Gilberto Nunes < gilberto.nunes32 at gmail.com> escreveu: > I am not sure about it, because now, after a fresh installation, even de > enp4sf0 appears... > This is so frustrated! > > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em qua, 17 de jul de 2019 ?s 14:45, Alex Chekholko via pve-user < > pve-user at pve.proxmox.com> escreveu: > >> >> >> >> ---------- Forwarded message ---------- >> From: Alex Chekholko >> To: PVE User List >> Cc: >> Bcc: >> Date: Wed, 17 Jul 2019 10:44:07 -0700 >> Subject: Re: [PVE-User] PVE 5.4 and Intel ixgbe >> You can try 'modinfo ixgbe' to query your actual installed version to see >> all the parameters it knows about. >> >> I see on one of my hosts >> # modinfo ixgbe >> filename: >> >> /lib/modules/4.15.0-54-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko >> version: 5.1.0-k >> ... >> parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ >> modules on 82599-based adapters (uint) >> >> And you can check the exact release notes for your version to see the >> allowed values for that parameter. >> >> IME, you may have some kind of incompable cable/optic anyway; see if you >> can try a different one. >> >> Regards, >> Alex >> >> On Wed, Jul 17, 2019 at 9:33 AM Gilberto Nunes < >> gilberto.nunes32 at gmail.com> >> wrote: >> >> > Hi there everybody! >> > >> > I have installed PVE 5.4 and try to up ixgbe driver for Intel 10GB >> SFI/SPF+ >> > NIC... >> > >> > I already do ixgbe-options.conf with >> > >> > options ixgbe allow_unsupported_sfp=1 >> > >> > But when try to load the module I still get this error: >> > >> > [ 170.008236] ixgbe 0000:05:00.0: failed to load because an unsupported >> > SFP+ or QSFP module type was detected. >> > >> > [ 170.008262] ixgbe 0000:05:00.0: Reload the driver after installing a >> > supported module. >> > >> > [ 170.022268] ixgbe 0000:05:00.1: failed to load because an unsupported >> > SFP+ or QSFP module type was detected. >> > >> > [ 170.022291] ixgbe 0000:05:00.1: Reload the driver after installing a >> > supported module. >> > >> > I already try to compile Intel module from scratch, but seems to failed >> > too! >> > >> > Thanks for any help! >> > >> > lspci >> > >> > 08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit >> SFI/SFP+ >> > Network Connection (rev 01) >> > 08:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit >> SFI/SFP+ >> > Network Connection (rev 01) >> > >> > >> > pveversion >> > >> > pve-manager/5.4-11/6df3d8d0 (running kernel: 4.15.18-18-pve) >> > --- >> > Gilberto Nunes Ferreira >> > >> > (47) 3025-5907 >> > (47) 99676-7530 - Whatsapp / Telegram >> > >> > Skype: gilberto.nunes36 >> > _______________________________________________ >> > pve-user mailing list >> > pve-user at pve.proxmox.com >> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > >> >> >> >> ---------- Forwarded message ---------- >> From: Alex Chekholko via pve-user >> To: PVE User List >> Cc: Alex Chekholko >> Bcc: >> Date: Wed, 17 Jul 2019 10:44:07 -0700 >> Subject: Re: [PVE-User] PVE 5.4 and Intel ixgbe >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > From tonci at suma-informatika.hr Wed Jul 17 22:22:16 2019 From: tonci at suma-informatika.hr (=?UTF-8?B?VG9uxI1pIFN0aXBpxI1ldmnEhw==?=) Date: Wed, 17 Jul 2019 22:22:16 +0200 Subject: [PVE-User] lxc pve-zsync In-Reply-To: <31ecc400-7efd-7d6c-fa95-c9c8c6e6ea73@jabra-server.net> References: <8678ddf7-98ef-f87b-bb16-09ad4953804e@suma-informatika.hr> <4bc5f882-10ad-e2f9-6831-a8db0798057f@suma-informatika.hr> <31ecc400-7efd-7d6c-fa95-c9c8c6e6ea73@jabra-server.net> Message-ID: Hi Jan, thank you very much !! That's it !:) BR Tonci > Hi Tonci, > > enable "Backup" on the MP, otherwise pve-zsync will not sync the mountpoint. > > > BR > > Jan > > On 17.07.2019 01:54, Ton?i Stipi?evi? wrote: >> Hello to all, >> >> ??? I need to backup one lxc (rootfs+mp0) to another zfs pve-backup host >> , but it seems that only rootfs gets synced to another site (?!) and not >> mp0 >> >> pve-zsync job fits my needs at the moment unlike storage-repl >> >> so this command will transfer only rootfs : >> >> pve-zsync sync --source 105 --dest 192.168.71.53:rpool/bck --verbose >> >> arch: amd64 >> cores: 4 >> hostname: TK-DC01 >> memory: 4096 >> mp0: zfs01:subvol-105-disk-1,mp=/mnt/data1,size=8G >> nameserver: 8.8.8.8 >> net0: >> name=eth0,bridge=vmbr0,firewall=1,gw=192.168.71.1,hwaddr=96:BD:EC:CC:B0:93,ip=192.168.71.252/2$ >> >> ostype: debian >> rootfs: zfs01:subvol-105-disk-0,size=16G >> searchdomain: lin-dc01.net.local >> swap: 512 >> >> Any help would be very appreciated >> >> Thank very much you in advance >> >> BR >> >> Tonci >> >>> >>> >>> >>> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From lists at merit.unu.edu Thu Jul 18 13:43:32 2019 From: lists at merit.unu.edu (mj) Date: Thu, 18 Jul 2019 13:43:32 +0200 Subject: [PVE-User] adding ceph osd nodes In-Reply-To: <20190717124745.GD80377@dona.proxmox.com> References: <20190717124745.GD80377@dona.proxmox.com> Message-ID: <4cae7b46-15cb-72d9-7ab0-8f201900b5f9@merit.unu.edu> Hi, On 7/17/19 2:47 PM, Alwin Antreich wrote: > I like to add, though not explicitly asked. While it is technically > possible, the cluster will lose its enterprise support. As Ceph is under > support on Proxmox VE nodes too. Hmm. That is a disappointing consequence of simply adding some storage-only nodes, with the sole purpose to increase ceph performance. > Better split out Ceph to a separate cluster and use the Proxmox VE nodes > as clients. To keep the Ceph client and cluster package versions close > together, you could use Proxmox VE without subscription for the separate > Ceph cluster. I find it surprising that a setup such as described above *would* be supported, and a (in our eyes simpler and more elegant) setup with one or two ceph-osd-only nodes is not supprted. MJ From elacunza at binovo.es Thu Jul 18 14:22:13 2019 From: elacunza at binovo.es (Eneko Lacunza) Date: Thu, 18 Jul 2019 14:22:13 +0200 Subject: [PVE-User] adding ceph osd nodes In-Reply-To: <4cae7b46-15cb-72d9-7ab0-8f201900b5f9@merit.unu.edu> References: <20190717124745.GD80377@dona.proxmox.com> <4cae7b46-15cb-72d9-7ab0-8f201900b5f9@merit.unu.edu> Message-ID: Hi, El 18/7/19 a las 13:43, mj escribi?: > > On 7/17/19 2:47 PM, Alwin Antreich wrote: > >> I like to add, though not explicitly asked. While it is technically >> possible, the cluster will lose its enterprise support. As Ceph is under >> support on Proxmox VE nodes too. > > Hmm. That is a disappointing consequence of simply adding some > storage-only nodes, with the sole purpose to increase ceph performance. > >> Better split out Ceph to a separate cluster and use the Proxmox VE nodes >> as clients. To keep the Ceph client and cluster package versions close >> together, you could use Proxmox VE without subscription for the separate >> Ceph cluster. > > I find it surprising that a setup such as described above *would* be > supported, and a (in our eyes simpler and more elegant) setup with one > or two ceph-osd-only nodes is not supprted. I don't understand your surprise. In the first scenario, without added ceph nodes, Proxmox subscription will support ceph storage server too. If you add non-Proxmox nodes, they can cause trouble in Ceph server duties. In the second scenario, Proxmox subscription will support only ceph as client, but not ceph storage nodes... Cheers Eneko -- Zuzendari Teknikoa / Director T?cnico Binovo IT Human Project, S.L. Telf. 943569206 Astigarraga bidea 2, 2? izq. oficina 11; 20180 Oiartzun (Gipuzkoa) www.binovo.es From a.antreich at proxmox.com Thu Jul 18 14:37:24 2019 From: a.antreich at proxmox.com (Alwin Antreich) Date: Thu, 18 Jul 2019 14:37:24 +0200 Subject: [PVE-User] adding ceph osd nodes In-Reply-To: <4cae7b46-15cb-72d9-7ab0-8f201900b5f9@merit.unu.edu> References: <20190717124745.GD80377@dona.proxmox.com> <4cae7b46-15cb-72d9-7ab0-8f201900b5f9@merit.unu.edu> Message-ID: <20190718123724.GE80377@dona.proxmox.com> On Thu, Jul 18, 2019 at 01:43:32PM +0200, mj wrote: > Hi, > > On 7/17/19 2:47 PM, Alwin Antreich wrote: > > > I like to add, though not explicitly asked. While it is technically > > possible, the cluster will lose its enterprise support. As Ceph is under > > support on Proxmox VE nodes too. > > Hmm. That is a disappointing consequence of simply adding some storage-only > nodes, with the sole purpose to increase ceph performance. > > > Better split out Ceph to a separate cluster and use the Proxmox VE nodes > > as clients. To keep the Ceph client and cluster package versions close > > together, you could use Proxmox VE without subscription for the separate > > Ceph cluster. > > I find it surprising that a setup such as described above *would* be > supported, and a (in our eyes simpler and more elegant) setup with one or > two ceph-osd-only nodes is not supprted. In the above setup, the Proxmox VE cluster is only a Ceph client. That accesses a Ceph cluster. That the Ceph cluster also consists out of Proxmox VE boxes, is merely a suggestion. With the above, the Proxmox VE cluster is under support but the storage cluster is not. This gives it a clear line. With a distributed storage like Ceph it would not be clear where support would start or end, as all nodes are involved in the storage. I hope this clarifies it a bit. -- Cheers, Alwin From ronny+pve-user at aasen.cx Fri Jul 19 10:32:30 2019 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Fri, 19 Jul 2019 10:32:30 +0200 Subject: [PVE-User] Proxmox VE 6.0 released! In-Reply-To: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> References: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> Message-ID: <8a6f5909-109c-f10a-a738-70ab8062d851@aasen.cx> Congratulations on the release. So happy to see Proxmox 6.0 so shortly after the Debian 10 release. It is impressive to say the least. Upgraded my singlenode dev/test homelab prox+ceph and it was no problems with the instructions you provided. pve5to6 was a really useful tool :) giving it a week to see how it feels. and Ill upgrade the work dev cluster as well :) Thanks for your hard work! kind regards Ronny On 16.07.2019 13:19, Martin Maurer wrote: > Hi all, > > We're excited to announce the final release of our Proxmox VE 6.0! It's > based on the great Debian 10 codename "Buster" and the latest 5.0 Linux > kernel, QEMU 4.0, LXC 3.1.0, ZFS 0.8.1, Ceph 14.2, Corosync 3.0, and more. > > This major release includes the latest Ceph Nautilus feautures and an > improved Ceph management dashboard. We have updated the cluster > communication stack to Corosync 3 using Kronosnet, and have a new > selection widget for the network making it simple to select the correct > link address in the cluster creation wizard. > > With ZFS 0.8.1 we have included TRIM support for SSDs and also support > for native encryption with comfortable key-handling. > > The new installer supports ZFS root via UEFI, for example you can boot a > ZFS mirror on NVMe SSDs (using systemd-boot instead of grub). > > And as always we have included countless bugfixes and improvements on a > lot of places; see the release notes for all details. > > Release notes > https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_6.0 > > Video intro > https://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-6-0 > > > Download > https://www.proxmox.com/en/downloads > Alternate ISO download: > http://download.proxmox.com/iso/ > > Documentation > https://pve.proxmox.com/pve-docs/ > > Community Forum > https://forum.proxmox.com > > Source Code > https://git.proxmox.com > > Bugtracker > https://bugzilla.proxmox.com > > FAQ > Q: Can I dist-upgrade Proxmox VE 5.4 to 6.0 with apt? > A: Please follow the upgrade instructions exactly, as there is a major > version bump of corosync (2.x to 3.x) > https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 > > Q: Can I install Proxmox VE 6.0 on top of Debian Buster? > A: Yes, see > https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Buster > > Q: Can I upgrade my Proxmox VE 5.4 cluster with Ceph Luminous to 6.0 > with Ceph Nautilus? > A: This is a two step process. First, you have to upgrade Proxmox VE > from 5.4 to 6.0, and afterwards upgrade Ceph from Luminous to Nautilus. > There are a lot of improvements and changes, please follow exactly the > upgrade documentation. > https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0 > https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus > > Q: Where can I get more information about future feature updates? > A: Check our roadmap, forum, mailing list and subscribe to our newsletter. > > A big THANK YOU to our active community for all your feedback, testing, > bug reporting and patch submitting! > From gilberto.nunes32 at gmail.com Fri Jul 19 14:39:12 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Fri, 19 Jul 2019 09:39:12 -0300 Subject: [PVE-User] Proxmox VE old versions... Message-ID: Hi there I am looking for old Proxmox version, 1.X series... Somebody can point a download site?? Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 From admin at extremeshok.com Fri Jul 19 15:58:52 2019 From: admin at extremeshok.com (admin at extremeshok.com) Date: Fri, 19 Jul 2019 15:58:52 +0200 Subject: [PVE-User] Proxmox VE old versions... In-Reply-To: References: Message-ID: https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_1.6-5261-4.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_1.7-5323-5.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_1.9-6542-6.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_2.0-4b59ea39-23.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_2.1-f9b0f63a-26.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_2.2-7f9cfa4c-28.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_2.3-ad9c5c05-30.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_3.0-0428106c-13.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_3.1-93bf03d4-8.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_3.2-1933730b-2.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_3.2-5a885216-5.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_3.3-a06c9f73-2.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_3.4-102d4547-6.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_3.4-3f2d890e-1.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_4.0-0d8559d0-17.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_4.1-2f9650d4-21.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_4.2-725d76f0-28.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_4.3-e7cdc165-2.iso https://archive.org/download/Proxmox-ve_Released_Iso/proxmox-ve_4.4-eb2d6f1e-1.iso On 7/19/19 2:39 PM, Gilberto Nunes wrote: > Somebody can point a download site?? > > Thanks > > --- > Gilberto Nunes Ferreira From sysadmin at tashicell.com Mon Jul 22 09:06:28 2019 From: sysadmin at tashicell.com (System Admin) Date: Mon, 22 Jul 2019 13:06:28 +0600 Subject: Help on setting up cloud server! Message-ID: Hi all, I'm new to Proxmox & Ceph. I would like to seek your help on setting up cloud server. I've three PVE (version 5.5-3) nodes configured with Ceph storage on hardware RAID 0 (MegaRAID SAS). I couldn't find a way to flash to IT HBA mode. Now, I would like to install CentOS 7 VM on *ceph-vm* pools and then, will configure NextCloud web application on other disk partition using different storage pool. Now this is where I need help. Which storage type RBD or CephFS is best for the cloud and how would I load them on VM. Your help will be appreciated. Thank you. Sonam From ronny+pve-user at aasen.cx Mon Jul 22 10:00:21 2019 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Mon, 22 Jul 2019 10:00:21 +0200 Subject: [PVE-User] Help on setting up cloud server! In-Reply-To: References: Message-ID: On 22.07.2019 09:06, System Admin via pve-user wrote: > Hi all, > > I'm new to Proxmox & Ceph. I would like to seek your help on setting up > cloud server. > > I've three PVE (version 5.5-3) nodes configured with Ceph storage on > hardware RAID 0 (MegaRAID SAS). I couldn't find a way to flash to IT HBA > mode. > > Now, I would like to install CentOS 7 VM on *ceph-vm* pools and then, > will configure NextCloud web application on other disk partition using > different storage pool. Now this is where I need help. > > Which storage type RBD or CephFS is best for the cloud and how would I > load them on VM. Your help will be appreciated. > > > Thank you. > > Sonam generally on a VM you want your OS/boot disk to be RBD image. since RBD is made for this purpose. nextcloud is a bit special tho. since if you want to scale your nextcloud across multiple servers, for High availabillity or performance reasons, you will need a shared storage for the nextcloud servers in addition to the OS disk for each VM. And for this you can use cephfs and mount it as a regular client from the VM as the nextcloud storage area on each nextcloud VM. I am a bit perplexed by what you mean by "other disk partition" since in ceph you do not split storage pools by partitions. give ceph the whole disk as osd. you can place different pools on different classes of disk, eg if you have fast or slow disks. but multiple pools live on the same disks. good luck Ronny From sysadmin at tashicell.com Mon Jul 22 12:11:49 2019 From: sysadmin at tashicell.com (System Admin) Date: Mon, 22 Jul 2019 16:11:49 +0600 Subject: [PVE-User] Help on setting up cloud server! In-Reply-To: References: Message-ID: Thanks for the help Ronny. Yes, VM will be installed on ceph-vm pool which is RDB storage. For storing nextcloud data, I'll mount CephFS but how do mount cephfs on vm? (Sorry, that "other disk partition" was meant to be some "mount point" ). But it is also possible to use RBD storage right? or CephFS is always the winner for cloud storage? Thank you. On 7/22/19 2:00 PM, Ronny Aasen wrote: > On 22.07.2019 09:06, System Admin via pve-user wrote: >> Hi all, >> >> I'm new to Proxmox & Ceph. I would like to seek your help on setting >> up cloud server. >> >> I've three PVE (version 5.5-3) nodes configured with Ceph storage on >> hardware RAID 0 (MegaRAID SAS). I couldn't find a way to flash to IT >> HBA mode. >> >> Now, I would like to install CentOS 7 VM on *ceph-vm* pools and then, >> will configure NextCloud web application on other disk partition >> using different storage pool. Now this is where I need help. >> >> Which storage type RBD or CephFS is best for the cloud and how would >> I load them on VM. Your help will be appreciated. >> >> >> Thank you. >> >> Sonam > > > generally on a VM you want your OS/boot disk to be RBD image. since > RBD is made for this purpose. > > nextcloud is a bit special tho. since if you want to scale your > nextcloud across multiple servers, for High availabillity or > performance reasons, you will need a shared storage for the nextcloud > servers in addition to the OS disk for each VM. > And for this you can use cephfs and mount it as a regular client from > the VM as the nextcloud storage area on each nextcloud VM. > > I am a bit perplexed by what you mean by "other disk partition" since > in ceph you do not split storage pools by partitions. give ceph the > whole disk as osd. > you can place different pools on different classes of disk, eg if you > have fast or slow disks. but multiple pools live on the same disks. > > > good luck > Ronny > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From ronny+pve-user at aasen.cx Mon Jul 22 12:35:41 2019 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Mon, 22 Jul 2019 12:35:41 +0200 Subject: [PVE-User] Help on setting up cloud server! In-Reply-To: References: Message-ID: you can use RBD image for nextcloud data as well. but you must either limit yourself to a single nexcloud server since there is no shared data between the servers. OR use a cluster aware filesystem on top of the shared RBD image, something like ocfs, gfs. in order to mount cephfs on the nextcloud vm, the vm must be able to talk to the whole public ceph network. either via a router, or have an interface in that network. then you can mount cephs on the vm using the kernel http://docs.ceph.com/docs/master/cephfs/kernel/ or using fuse http://docs.ceph.com/docs/master/cephfs/fuse/ generally fuse have a newer version so supports more features, but i think kernel have better performance. good luck Ronny On 22.07.2019 12:11, System Admin via pve-user wrote: > Thanks for the help Ronny. > > Yes, VM will be installed on ceph-vm pool which is RDB storage. For > storing nextcloud data, I'll mount CephFS but how do mount cephfs on vm? > (Sorry, that "other disk partition" was meant to be some "mount point" ). > > But it is also possible to use RBD storage right? or CephFS is always > the winner for cloud storage? > > > Thank you. > > > > On 7/22/19 2:00 PM, Ronny Aasen wrote: >> On 22.07.2019 09:06, System Admin via pve-user wrote: >>> Hi all, >>> >>> I'm new to Proxmox & Ceph. I would like to seek your help on setting >>> up cloud server. >>> >>> I've three PVE (version 5.5-3) nodes configured with Ceph storage on >>> hardware RAID 0 (MegaRAID SAS). I couldn't find a way to flash to IT >>> HBA mode. >>> >>> Now, I would like to install CentOS 7 VM on *ceph-vm* pools and then, >>> will configure NextCloud web application on other disk partition >>> using different storage pool. Now this is where I need help. >>> >>> Which storage type RBD or CephFS is best for the cloud and how would >>> I load them on VM. Your help will be appreciated. >>> >>> >>> Thank you. >>> >>> Sonam >> >> >> generally on a VM you want your OS/boot disk to be RBD image. since >> RBD is made for this purpose. >> >> nextcloud is a bit special tho. since if you want to scale your >> nextcloud across multiple servers, for High availabillity or >> performance reasons, you will need a shared storage for the nextcloud >> servers in addition to the OS disk for each VM. >> And for this you can use cephfs and mount it as a regular client from >> the VM as the nextcloud storage area on each nextcloud VM. >> >> I am a bit perplexed by what you mean by "other disk partition" since >> in ceph you do not split storage pools by partitions. give ceph the >> whole disk as osd. >> you can place different pools on different classes of disk, eg if you >> have fast or slow disks. but multiple pools live on the same disks. >> >> >> good luck >> Ronny From devzero at web.de Tue Jul 23 12:53:08 2019 From: devzero at web.de (Roland @web.de) Date: Tue, 23 Jul 2019 12:53:08 +0200 Subject: [PVE-User] network interfaces renamed after update to proxmox6 Message-ID: hello, the following network interfaces? got lost after update to proxmox6 enp4s0f0 enp4s0f1 enp5s0f0 enp5s0f1 apparently , they got renamed to ens7f0 ens7f1 rename2 rename7 Does anybody know the reason for that or how to fix it or to get to "stable ethernet interface naming scheme" ? roland network config, configured with proxmox 5.4 - still existant with proxmox6 root at pve1-knju:/etc/network# cat interfaces # network interface settings; autogenerated # Please do NOT modify this file directly, unless you know what # you're doing. # # If you want to manage parts of the network configuration manually, # please utilize the 'source' or 'source-directory' directives to do # so. # PVE will preserve these directives, but will NOT read its network # configuration from sourced files, so do not attempt to move any of # the PVE managed interfaces into external files! auto lo iface lo inet loopback iface enp11s0f0 inet manual auto enp4s0f0 iface enp4s0f0 inet manual auto enp4s0f1 iface enp4s0f1 inet manual iface enp11s0f1 inet manual auto enp5s0f0 iface enp5s0f0 inet manual auto enp5s0f1 iface enp5s0f1 inet manual auto bond0 iface bond0 inet manual ??? bond-slaves enp4s0f0 enp4s0f1 enp5s0f0 enp5s0f1 ??? bond-miimon 100 ??? bond-mode balance-rr ??? post-up ip link set enp4s0f0 mtu 9000 && ip link set enp4s0f1 mtu 9000 && ip link set enp5s0f0 mtu 9000 && ip link set enp5s0f1 mtu 9000 ??? post-up ip link set bond0 mtu 9000 auto vmbr0 iface vmbr0 inet static ??? address? 172.16.37.101 ??? netmask? 255.255.255.0 ??? gateway? 172.16.37.1 ??? bridge-ports enp11s0f0 ??? bridge-stp off ??? bridge-fd 0 auto vmbr1 iface vmbr1 inet static ??? address? 10.0.0.1 ??? netmask? 24 ??? bridge-ports bond0 ??? bridge-stp off ??? bridge-fd 0 ??????? post-up ip link set vmbr1 mtu 9000 real network interfaces: root at pve1-knju:~# ifconfig -a bond0: flags=5123? mtu 1500 ??????? ether aa:7a:96:ed:b2:be? txqueuelen 1000? (Ethernet) ??????? RX packets 0? bytes 0 (0.0 B) ??????? RX errors 0? dropped 0? overruns 0? frame 0 ??????? TX packets 0? bytes 0 (0.0 B) ??????? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0 enp11s0f0: flags=4163? mtu 1500 ??????? ether 00:19:99:83:71:e6? txqueuelen 1000? (Ethernet) ??????? RX packets 5245? bytes 1131420 (1.0 MiB) ??????? RX errors 0? dropped 62? overruns 0? frame 0 ??????? TX packets 1995? bytes 1000577 (977.1 KiB) ??????? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0 ??????? device memory 0xce460000-ce47ffff enp11s0f1: flags=4098? mtu 1500 ??????? ether 00:19:99:83:71:e7? txqueuelen 1000? (Ethernet) ??????? RX packets 0? bytes 0 (0.0 B) ??????? RX errors 0? dropped 0? overruns 0? frame 0 ??????? TX packets 0? bytes 0 (0.0 B) ??????? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0 ??????? device memory 0xce4c0000-ce4dffff ens7f0: flags=4098? mtu 1500 ??????? ether 00:1b:21:67:d9:77? txqueuelen 1000? (Ethernet) ??????? RX packets 0? bytes 0 (0.0 B) ??????? RX errors 0? dropped 0? overruns 0? frame 0 ??????? TX packets 0? bytes 0 (0.0 B) ??????? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0 ??????? device interrupt 54? memory 0xce340000-ce360000 ens7f1: flags=4098? mtu 1500 ??????? ether 00:1b:21:67:d9:74? txqueuelen 1000? (Ethernet) ??????? RX packets 0? bytes 0 (0.0 B) ??????? RX errors 0? dropped 0? overruns 0? frame 0 ??????? TX packets 0? bytes 0 (0.0 B) ??????? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0 ??????? device interrupt 43? memory 0xce2a0000-ce2c0000 lo: flags=73? mtu 65536 ??????? inet 127.0.0.1? netmask 255.0.0.0 ??????? inet6 ::1? prefixlen 128? scopeid 0x10 ??????? loop? txqueuelen 1000? (Local Loopback) ??????? RX packets 1184? bytes 322642 (315.0 KiB) ??????? RX errors 0? dropped 0? overruns 0? frame 0 ??????? TX packets 1184? bytes 322642 (315.0 KiB) ??????? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0 rename2: flags=4098? mtu 1500 ??????? ether 00:1b:21:67:d9:75? txqueuelen 1000? (Ethernet) ??????? RX packets 0? bytes 0 (0.0 B) ??????? RX errors 0? dropped 0? overruns 0? frame 0 ??????? TX packets 0? bytes 0 (0.0 B) ??????? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0 ??????? device interrupt 31? memory 0xce240000-ce260000 rename7: flags=4098? mtu 1500 ??????? ether 00:1b:21:67:d9:76? txqueuelen 1000? (Ethernet) ??????? RX packets 0? bytes 0 (0.0 B) ??????? RX errors 0? dropped 0? overruns 0? frame 0 ??????? TX packets 0? bytes 0 (0.0 B) ??????? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0 ??????? device interrupt 25? memory 0xce3a0000-ce3c0000 vmbr0: flags=4163? mtu 1500 ??????? inet 172.16.37.101? netmask 255.255.255.0? broadcast 172.16.37.255 ??????? inet6 fe80::219:99ff:fe83:71e6? prefixlen 64? scopeid 0x20 ??????? ether 00:19:99:83:71:e6? txqueuelen 1000? (Ethernet) ??????? RX packets 5178? bytes 1042478 (1018.0 KiB) ??????? RX errors 0? dropped 0? overruns 0? frame 0 ??????? TX packets 1925? bytes 995957 (972.6 KiB) ??????? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0 vmbr1: flags=4099? mtu 9000 ??????? inet 10.0.0.1? netmask 255.255.255.0? broadcast 10.0.0.255 ??????? ether aa:7a:96:ed:b2:be? txqueuelen 1000? (Ethernet) ??????? RX packets 0? bytes 0 (0.0 B) ??????? RX errors 0? dropped 0? overruns 0? frame 0 ??????? TX packets 0? bytes 0 (0.0 B) ??????? TX errors 0? dropped 0 overruns 0? carrier 0? collisions 0 From d.csapak at proxmox.com Tue Jul 23 13:30:21 2019 From: d.csapak at proxmox.com (Dominik Csapak) Date: Tue, 23 Jul 2019 13:30:21 +0200 Subject: [PVE-User] network interfaces renamed after update to proxmox6 In-Reply-To: References: Message-ID: Hi, we changed from the out-of-tree intel driver to the intree kernel driver. maybe there is some bug there did yo change anything regarding network device naming (e.g. with udev)? can you post an 'lspci --nnk' ? maybe there is a firmware update for your card? From d.csapak at proxmox.com Tue Jul 23 13:34:35 2019 From: d.csapak at proxmox.com (Dominik Csapak) Date: Tue, 23 Jul 2019 13:34:35 +0200 Subject: [PVE-User] network interfaces renamed after update to proxmox6 In-Reply-To: References: Message-ID: On 7/23/19 1:30 PM, Dominik Csapak wrote: > Hi, > > we changed from the out-of-tree intel driver to the intree kernel driver. > > maybe there is some bug there > > did yo change anything regarding network device naming (e.g. with udev)? > > can you post an 'lspci --nnk' ? sorry this is a typo, should be 'lspci -nnk' (only one -) also dmesg output would be interesting > > maybe there is a firmware update for your card? > From devzero at web.de Tue Jul 23 14:39:19 2019 From: devzero at web.de (Roland @web.de) Date: Tue, 23 Jul 2019 14:39:19 +0200 Subject: [PVE-User] network interfaces renamed after update to proxmox6 In-Reply-To: References: Message-ID: <1e9bb43f-69e5-f2e6-941a-ab898c073215@web.de> >did yo change anything regarding network device naming (e.g. with udev)? no, i did not change anything regarding naming. besides creating an additional bond and bridge and adding mtu 9000 to all those, nothing was "special". >can you post an 'lspci --nnk' ? here is the ethernet controller information from that commands output: 04:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [8086:10bc] (rev 06) ??? Subsystem: Intel Corporation PRO/1000 PT Quad Port LP Server Adapter [8086:11bc] ??? Kernel driver in use: e1000e ??? Kernel modules: e1000e 04:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [8086:10bc] (rev 06) ??? Subsystem: Intel Corporation PRO/1000 PT Quad Port LP Server Adapter [8086:11bc] ??? Kernel driver in use: e1000e ??? Kernel modules: e1000e 05:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [8086:10bc] (rev 06) ??? Subsystem: Intel Corporation PRO/1000 PT Quad Port LP Server Adapter [8086:11bc] ??? Kernel driver in use: e1000e ??? Kernel modules: e1000e 05:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) [8086:10bc] (rev 06) ??? Subsystem: Intel Corporation PRO/1000 PT Quad Port LP Server Adapter [8086:11bc] ??? Kernel driver in use: e1000e ??? Kernel modules: e1000e 0b:00.0 Ethernet controller [0200]: Intel Corporation 82575EB Gigabit Network Connection [8086:10a7] (rev 02) ??? Subsystem: Fujitsu Technology Solutions 82575EB Gigabit Network Connection [1734:1128] ??? Kernel driver in use: igb ??? Kernel modules: igb 0b:00.1 Ethernet controller [0200]: Intel Corporation 82575EB Gigabit Network Connection [8086:10a7] (rev 02) ??? Subsystem: Fujitsu Technology Solutions 82575EB Gigabit Network Connection [1734:1128] ??? Kernel driver in use: igb ??? Kernel modules: igb >maybe there is a firmware update for your card? possible. I will have a look regards Roland Am 23.07.19 um 13:34 schrieb Dominik Csapak: > On 7/23/19 1:30 PM, Dominik Csapak wrote: >> Hi, >> >> we changed from the out-of-tree intel driver to the intree kernel >> driver. >> >> maybe there is some bug there >> >> did yo change anything regarding network device naming (e.g. with udev)? >> >> can you post an 'lspci --nnk' ? > > sorry this is a typo, should be 'lspci -nnk' (only one -) > > also dmesg output would be interesting > >> >> maybe there is a firmware update for your card? >> > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From ronny+pve-user at aasen.cx Wed Jul 24 08:54:53 2019 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Wed, 24 Jul 2019 08:54:53 +0200 Subject: [PVE-User] Shared same rbd disk on 2 Vms In-Reply-To: <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> Message-ID: <77aa5ad9-dcff-c5df-cbcf-21e1834ad5c2@aasen.cx> On 03.07.2019 14:20, Herv? Ballans wrote: > Le 02/07/2019 ? 15:50, Tobias Kropf a ?crit?: >> With any Network connection between the vms ... you can use DRBD on >> top of the rbd volumes? > > Thanks Tobias for your reply. > > From what I understood in the past on drbd, it requires 2 disks to be > synchronized between them (sort of network RAID 1 ?). Now I want exactly > the opposite, a single disk that I can use on 2 machines... > > rv with OCFS or GFS you can run the same blockdevice on multiple machines. but it is a bit of extra complexity vs just using cephfs. so if you basically want a filesystem i would look at cephfs. if you absolutely need multi access block devices, you can research OCFS or GFS kind regards Ronny From sir_Misiek1 at o2.pl Wed Jul 24 12:39:37 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Wed, 24 Jul 2019 12:39:37 +0200 Subject: [PVE-User] proxmox installation problem In-Reply-To: <77aa5ad9-dcff-c5df-cbcf-21e1834ad5c2@aasen.cx> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> <77aa5ad9-dcff-c5df-cbcf-21e1834ad5c2@aasen.cx> Message-ID: Hello, I have a problem with the Proxmox installation. IBM 3650 (7979) 6 * SAS hdd server. Proxmox only starts with RAID Hardware. It does not start with RAID-Z and RAID1. Error in the: https://help.komandor.pl/aaa.jpg kind regards Gregor From mityapetuhov at gmail.com Wed Jul 24 12:49:17 2019 From: mityapetuhov at gmail.com (Dmitry Petuhov) Date: Wed, 24 Jul 2019 13:49:17 +0300 Subject: [PVE-User] proxmox installation problem In-Reply-To: References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> <77aa5ad9-dcff-c5df-cbcf-21e1834ad5c2@aasen.cx> Message-ID: Try to look at https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Boot_fails_and_goes_into_busybox These parameters give linux kernel time to detect hard disks present in system and find ZFS on them. 24.07.2019 13:39, lord_Niedzwiedz ?????: > Hello, > > I have a problem with the Proxmox installation. > IBM 3650 (7979) 6 * SAS hdd server. > Proxmox only starts with RAID Hardware. > It does not start with RAID-Z and RAID1. > Error in the: https://help.komandor.pl/aaa.jpg > > kind regards > Gregor > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From venefax at gmail.com Wed Jul 24 12:53:12 2019 From: venefax at gmail.com (Saint Michael) Date: Wed, 24 Jul 2019 12:53:12 +0200 Subject: [PVE-User] Windows VM does not start automatically In-Reply-To: References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> <77aa5ad9-dcff-c5df-cbcf-21e1834ad5c2@aasen.cx> Message-ID: In spice of being set ?auto?, a windows 2019 VM does not start when I reboot the box. Is there any idea of what may be going wrong, or a particular log I could read to find out the culprit? From sir_Misiek1 at o2.pl Wed Jul 24 13:34:15 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Wed, 24 Jul 2019 13:34:15 +0200 Subject: [PVE-User] proxmox installation problem In-Reply-To: References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> <77aa5ad9-dcff-c5df-cbcf-21e1834ad5c2@aasen.cx> Message-ID: <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> Option A) I dont see any grub file ;-/ And command /usr/sbin/update-grub B) I dont see in system command update-initramfs W dniu 24.07.2019 o?12:49, Dmitry Petuhov pisze: > Try to look at > https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Boot_fails_and_goes_into_busybox > > These parameters give linux kernel time to detect hard disks present > in system and find ZFS on them. > > > 24.07.2019 13:39, lord_Niedzwiedz ?????: >> Hello, >> >> I have a problem with the Proxmox installation. >> IBM 3650 (7979) 6 * SAS hdd server. >> Proxmox only starts with RAID Hardware. >> It does not start with RAID-Z and RAID1. >> Error in the: https://help.komandor.pl/aaa.jpg >> >> kind regards >> Gregor >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From mityapetuhov at gmail.com Wed Jul 24 13:57:09 2019 From: mityapetuhov at gmail.com (Dmitry Petuhov) Date: Wed, 24 Jul 2019 14:57:09 +0300 Subject: [PVE-User] proxmox installation problem In-Reply-To: <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> <77aa5ad9-dcff-c5df-cbcf-21e1834ad5c2@aasen.cx> <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> Message-ID: <0069e79b-2391-90ee-b6e7-061315e8e6af@gmail.com> Oh. I see. It seems that you created ZFS pool several times on same set of disks (operations like RAID array deletion do not actually deletes ZFS from disks). Do `zpool import` command to see pools with same name 'rpool' that system sees. Then delete not actual (they will be in faulty state) ones with `zpool destroy `. After that you will be able to import actual pool, exit from that busybox recovery shell and continue. If there'll be no importable pools then you will need to delete all pools and reinstall system from DC again. 24.07.2019 14:34, lord_Niedzwiedz ?????: > Option A) I dont see any grub file ;-/ > And command /usr/sbin/update-grub > > B) I dont see in system command update-initramfs > > W dniu 24.07.2019 o?12:49, Dmitry Petuhov pisze: >> Try to look at >> https://pve.proxmox.com/wiki/ZFS:_Tips_and_Tricks#Boot_fails_and_goes_into_busybox >> >> These parameters give linux kernel time to detect hard disks present >> in system and find ZFS on them. >> >> >> 24.07.2019 13:39, lord_Niedzwiedz ?????: >>> Hello, >>> >>> I have a problem with the Proxmox installation. >>> IBM 3650 (7979) 6 * SAS hdd server. >>> Proxmox only starts with RAID Hardware. >>> It does not start with RAID-Z and RAID1. >>> Error in the: https://help.komandor.pl/aaa.jpg >>> >>> kind regards >>> Gregor >>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From sir_Misiek1 at o2.pl Mon Jul 29 10:55:35 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Mon, 29 Jul 2019 10:55:35 +0200 Subject: [PVE-User] Proxmox - BIG PROBLEM In-Reply-To: <0069e79b-2391-90ee-b6e7-061315e8e6af@gmail.com> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> <77aa5ad9-dcff-c5df-cbcf-21e1834ad5c2@aasen.cx> <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> <0069e79b-2391-90ee-b6e7-061315e8e6af@gmail.com> Message-ID: <6a4eb127-c16b-c12f-5598-7450c0a015b5@o2.pl> I ran a command on the server by mistake: rm /* rm: cannot remove '/Backup': Is a directory rm: cannot remove '/boot': Is a directory rm: cannot remove '/dev': Is a directory rm: cannot remove '/etc': Is a directory rm: cannot remove '/home': Is a directory rm: cannot remove '/media': Is a directory rm: cannot remove '/mnt': Is a directory rm: cannot remove '/opt': Is a directory rm: cannot remove '/proc': Is a directory rm: cannot remove '/Roboczy': Is a directory rm: cannot remove '/root': Is a directory rm: cannot remove '/rpool': Is a directory rm: cannot remove '/run': Is a directory rm: cannot remove '/srv': Is a directory rm: cannot remove '/sys': Is a directory rm: cannot remove '/tmp': Is a directory rm: cannot remove '/usr': Is a directory rm: cannot remove '/var': Is a directory Strange machines work. I'm logged in gui. But I can not get to the machine VM. Do not execute any commands. What to do ??!! From what I see, I deleted my catalogs: / bin / lib / lib64 / sbin WITH /. How is this possible ??!! I'm still logged in on one console after the shell, but I can not do any commandos. Even: qm -bash: /usr/sbin/qm: /usr/bin/perl: bad interpreter: No such file or directory root at tomas:/usr/bin# ls -bash: /usr/bin/ls: No such file or directory root at tomas:/usr/bin# echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Any Idea ?? Please Help Me. Gregor From f.cuseo at panservice.it Mon Jul 29 11:00:27 2019 From: f.cuseo at panservice.it (Fabrizio Cuseo) Date: Mon, 29 Jul 2019 11:00:27 +0200 (CEST) Subject: [PVE-User] Proxmox - BIG PROBLEM In-Reply-To: <6a4eb127-c16b-c12f-5598-7450c0a015b5@o2.pl> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> <77aa5ad9-dcff-c5df-cbcf-21e1834ad5c2@aasen.cx> <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> <0069e79b-2391-90ee-b6e7-061315e8e6af@gmail.com> <6a4eb127-c16b-c12f-5598-7450c0a015b5@o2.pl> Message-ID: <42180642.1911160.1564390827580.JavaMail.zimbra@zimbra.panservice.it> Where are located the VM's disks ? LVM ? ZFS ? Is possibile that you still have your disks (if LVM, for example), but i think that is better that you install a fresh Proxmox server, and move the disks from the old hard drive to the new one. You need some knowledge about linux, lvm, and you can save all your data. ----- Il 29-lug-19, alle 10:55, lord_Niedzwiedz sir_Misiek1 at o2.pl ha scritto: > I ran a command on the server by mistake: > > rm /* > rm: cannot remove '/Backup': Is a directory > rm: cannot remove '/boot': Is a directory > rm: cannot remove '/dev': Is a directory > rm: cannot remove '/etc': Is a directory > rm: cannot remove '/home': Is a directory > rm: cannot remove '/media': Is a directory > rm: cannot remove '/mnt': Is a directory > rm: cannot remove '/opt': Is a directory > rm: cannot remove '/proc': Is a directory > rm: cannot remove '/Roboczy': Is a directory > rm: cannot remove '/root': Is a directory > rm: cannot remove '/rpool': Is a directory > rm: cannot remove '/run': Is a directory > rm: cannot remove '/srv': Is a directory > rm: cannot remove '/sys': Is a directory > rm: cannot remove '/tmp': Is a directory > rm: cannot remove '/usr': Is a directory > rm: cannot remove '/var': Is a directory > > Strange machines work. > I'm logged in gui. > But I can not get to the machine VM. > Do not execute any commands. > What to do ??!! > From what I see, I deleted my catalogs: > / bin > / lib > / lib64 > / sbin > WITH /. > How is this possible ??!! > I'm still logged in on one console after the shell, but I can not do any > commandos. > Even: > qm > -bash: /usr/sbin/qm: /usr/bin/perl: bad interpreter: No such file or > directory > root at tomas:/usr/bin# ls > -bash: /usr/bin/ls: No such file or directory > root at tomas:/usr/bin# echo $PATH > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > > Any Idea ?? > Please Help Me. > > Gregor > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- --- Fabrizio Cuseo - mailto:f.cuseo at panservice.it Direzione Generale - Panservice InterNetWorking Servizi Professionali per Internet ed il Networking Panservice e' associata AIIP - RIPE Local Registry Phone: +39 0773 410020 - Fax: +39 0773 470219 http://www.panservice.it mailto:info at panservice.it Numero verde nazionale: 800 901492 From sir_Misiek1 at o2.pl Mon Jul 29 11:08:37 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Mon, 29 Jul 2019 11:08:37 +0200 Subject: [PVE-User] Proxmox - BIG PROBLEM In-Reply-To: <42180642.1911160.1564390827580.JavaMail.zimbra@zimbra.panservice.it> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> <77aa5ad9-dcff-c5df-cbcf-21e1834ad5c2@aasen.cx> <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> <0069e79b-2391-90ee-b6e7-061315e8e6af@gmail.com> <6a4eb127-c16b-c12f-5598-7450c0a015b5@o2.pl> <42180642.1911160.1564390827580.JavaMail.zimbra@zimbra.panservice.it> Message-ID: <5dc4752b-0827-332f-ad77-9e7a59cfbc0b@o2.pl> ??? ??? VM at local-zfs. But local-zfs not available with gui !! VM still work. And I see in: cd /mnt/pve/ directory: nvme0n1 / nvme1n1 / sda / Here is one virtual. The rest on local-zfs (and they work, and I can not see the space). Proxmox it's still working. I lostmybe only : /bin /lib /lib64 /sbin How is it possible that command: rm / * removed them ?? !! Without the -r option. And the rest of the catalogs did not delete ?? !! Maybe these were symbolic links? Gregor > Where are located the VM's disks ? LVM ? ZFS ? > Is possibile that you still have your disks (if LVM, for example), but i think that is better that you install a fresh Proxmox server, and move the disks from the old hard drive to the new one. > You need some knowledge about linux, lvm, and you can save all your data. > > > > ----- Il 29-lug-19, alle 10:55, lord_Niedzwiedz sir_Misiek1 at o2.pl ha scritto: > >> I ran a command on the server by mistake: >> >> rm /* >> rm: cannot remove '/Backup': Is a directory >> rm: cannot remove '/boot': Is a directory >> rm: cannot remove '/dev': Is a directory >> rm: cannot remove '/etc': Is a directory >> rm: cannot remove '/home': Is a directory >> rm: cannot remove '/media': Is a directory >> rm: cannot remove '/mnt': Is a directory >> rm: cannot remove '/opt': Is a directory >> rm: cannot remove '/proc': Is a directory >> rm: cannot remove '/Roboczy': Is a directory >> rm: cannot remove '/root': Is a directory >> rm: cannot remove '/rpool': Is a directory >> rm: cannot remove '/run': Is a directory >> rm: cannot remove '/srv': Is a directory >> rm: cannot remove '/sys': Is a directory >> rm: cannot remove '/tmp': Is a directory >> rm: cannot remove '/usr': Is a directory >> rm: cannot remove '/var': Is a directory >> >> Strange machines work. >> I'm logged in gui. >> But I can not get to the machine VM. >> Do not execute any commands. >> What to do ??!! >> From what I see, I deleted my catalogs: >> / bin >> / lib >> / lib64 >> / sbin >> WITH /. >> How is this possible ??!! >> I'm still logged in on one console after the shell, but I can not do any >> commandos. >> Even: >> qm >> -bash: /usr/sbin/qm: /usr/bin/perl: bad interpreter: No such file or >> directory >> root at tomas:/usr/bin# ls >> -bash: /usr/bin/ls: No such file or directory >> root at tomas:/usr/bin# echo $PATH >> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >> >> Any Idea ?? >> Please Help Me. >> >> Gregor >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From f.cuseo at panservice.it Mon Jul 29 11:15:29 2019 From: f.cuseo at panservice.it (Fabrizio Cuseo) Date: Mon, 29 Jul 2019 11:15:29 +0200 (CEST) Subject: [PVE-User] Proxmox - BIG PROBLEM In-Reply-To: <5dc4752b-0827-332f-ad77-9e7a59cfbc0b@o2.pl> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> <0069e79b-2391-90ee-b6e7-061315e8e6af@gmail.com> <6a4eb127-c16b-c12f-5598-7450c0a015b5@o2.pl> <42180642.1911160.1564390827580.JavaMail.zimbra@zimbra.panservice.it> <5dc4752b-0827-332f-ad77-9e7a59cfbc0b@o2.pl> Message-ID: <911742305.1912254.1564391729733.JavaMail.zimbra@zimbra.panservice.it> Your system without /bin/ and /lib can't be usable, you need to totally recover it. I personally prefer to install a new system and migrate the vm files (that you have on local-zfs). But forget to use the GUI ----- Il 29-lug-19, alle 11:08, lord_Niedzwiedz ha scritto: > VM at local-zfs. > But local-zfs not available with gui !! > VM still work. > And I see in: > cd /mnt/pve/ > directory: > nvme0n1 / nvme1n1 / sda / > Here is one virtual. > The rest on local-zfs (and they work, and I can not see the space). > Proxmox it's still working. > I lost mybe only : > /bin > / lib > / lib64 > / sbin > How is it possible that command: > rm / * > removed them ?? !! > Without the -r option. > And the rest of the catalogs did not delete ?? !! > Maybe these were symbolic links? > Gregor >> Where are located the VM's disks ? LVM ? ZFS ? >> Is possibile that you still have your disks (if LVM, for example), but i think >> that is better that you install a fresh Proxmox server, and move the disks from >> the old hard drive to the new one. >> You need some knowledge about linux, lvm, and you can save all your data. >> ----- Il 29-lug-19, alle 10:55, lord_Niedzwiedz [ mailto:sir_Misiek1 at o2.pl | >> sir_Misiek1 at o2.pl ] ha scritto: >>> I ran a command on the server by mistake: >>> rm /* >>> rm: cannot remove '/Backup': Is a directory >>> rm: cannot remove '/boot': Is a directory >>> rm: cannot remove '/dev': Is a directory >>> rm: cannot remove '/etc': Is a directory >>> rm: cannot remove '/home': Is a directory >>> rm: cannot remove '/media': Is a directory >>> rm: cannot remove '/mnt': Is a directory >>> rm: cannot remove '/opt': Is a directory >>> rm: cannot remove '/proc': Is a directory >>> rm: cannot remove '/Roboczy': Is a directory >>> rm: cannot remove '/root': Is a directory >>> rm: cannot remove '/rpool': Is a directory >>> rm: cannot remove '/run': Is a directory >>> rm: cannot remove '/srv': Is a directory >>> rm: cannot remove '/sys': Is a directory >>> rm: cannot remove '/tmp': Is a directory >>> rm: cannot remove '/usr': Is a directory >>> rm: cannot remove '/var': Is a directory >>> Strange machines work. >>> I'm logged in gui. >>> But I can not get to the machine VM. >>> Do not execute any commands. >>> What to do ??!! >>> From what I see, I deleted my catalogs: >>> / bin >>> / lib >>> / lib64 >>> / sbin >>> WITH /. >>> How is this possible ??!! >>> I'm still logged in on one console after the shell, but I can not do any >>> commandos. >>> Even: >>> qm >>> -bash: /usr/sbin/qm: /usr/bin/perl: bad interpreter: No such file or >>> directory >>> root at tomas:/usr/bin# ls >>> -bash: /usr/bin/ls: No such file or directory >>> root at tomas:/usr/bin# echo $PATH >>> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>> Any Idea ?? >>> Please Help Me. >>> Gregor >>> _______________________________________________ >>> pve-user mailing list [ mailto:pve-user at pve.proxmox.com | >>> pve-user at pve.proxmox.com ] [ >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user | >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user ] -- --- Fabrizio Cuseo - mailto:f.cuseo at panservice.it Direzione Generale - Panservice InterNetWorking Servizi Professionali per Internet ed il Networking Panservice e' associata AIIP - RIPE Local Registry Phone: +39 0773 410020 - Fax: +39 0773 470219 http://www.panservice.it mailto:info at panservice.it Numero verde nazionale: 800 901492 From sir_Misiek1 at o2.pl Mon Jul 29 11:24:50 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Mon, 29 Jul 2019 11:24:50 +0200 Subject: [PVE-User] Proxmox - BIG PROBLEM In-Reply-To: <911742305.1912254.1564391729733.JavaMail.zimbra@zimbra.panservice.it> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> <0069e79b-2391-90ee-b6e7-061315e8e6af@gmail.com> <6a4eb127-c16b-c12f-5598-7450c0a015b5@o2.pl> <42180642.1911160.1564390827580.JavaMail.zimbra@zimbra.panservice.it> <5dc4752b-0827-332f-ad77-9e7a59cfbc0b@o2.pl> <911742305.1912254.1564391729733.JavaMail.zimbra@zimbra.panservice.it> Message-ID: <91af1f0c-f7c1-770b-6e54-92eb929d1f1f@o2.pl> > Your system without /bin/ and /lib can't be usable, you need to > totally recover it. But why did rm erase them. And the rest of the catalogs did not remove? I have a working VM there that I can not clone or stop. > I personally prefer to install a new system and migrate the vm files > (that you? have on local-zfs). But forget to use the GUI > > > > > ----- Il 29-lug-19, alle 11:08, lord_Niedzwiedz ha > scritto: > > ??? VM at local-zfs. > But local-zfs not available with gui !! > VM still work. > > And I see in: > cd /mnt/pve/ > directory: > nvme0n1 / nvme1n1 / sda / > > Here is one virtual. > The rest on local-zfs (and they work, and I can not see the space). > > Proxmox it's still working. > > I lostmybe only : > /bin > /lib > /lib64 > /sbin > How is it possible that command: > rm / * > removed them ?? !! > > Without the -r option. > > And the rest of the catalogs did not delete ?? !! > Maybe these were symbolic links? > > Gregor > > Where are located the VM's disks ? LVM ? ZFS ? > Is possibile that you still have your disks (if LVM, for example), but i think that is better that you install a fresh Proxmox server, and move the disks from the old hard drive to the new one. > You need some knowledge about linux, lvm, and you can save all your data. > > > > ----- Il 29-lug-19, alle 10:55, lord_Niedzwiedzsir_Misiek1 at o2.pl ha scritto: > > I ran a command on the server by mistake: > > rm /* > rm: cannot remove '/Backup': Is a directory > rm: cannot remove '/boot': Is a directory > rm: cannot remove '/dev': Is a directory > rm: cannot remove '/etc': Is a directory > rm: cannot remove '/home': Is a directory > rm: cannot remove '/media': Is a directory > rm: cannot remove '/mnt': Is a directory > rm: cannot remove '/opt': Is a directory > rm: cannot remove '/proc': Is a directory > rm: cannot remove '/Roboczy': Is a directory > rm: cannot remove '/root': Is a directory > rm: cannot remove '/rpool': Is a directory > rm: cannot remove '/run': Is a directory > rm: cannot remove '/srv': Is a directory > rm: cannot remove '/sys': Is a directory > rm: cannot remove '/tmp': Is a directory > rm: cannot remove '/usr': Is a directory > rm: cannot remove '/var': Is a directory > > Strange machines work. > I'm logged in gui. > But I can not get to the machine VM. > Do not execute any commands. > What to do ??!! > From what I see, I deleted my catalogs: > / bin > / lib > / lib64 > / sbin > WITH /. > How is this possible ??!! > I'm still logged in on one console after the shell, but I can not do any > commandos. > Even: > qm > -bash: /usr/sbin/qm: /usr/bin/perl: bad interpreter: No such file or > directory > root at tomas:/usr/bin# ls > -bash: /usr/bin/ls: No such file or directory > root at tomas:/usr/bin# echo $PATH > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin > > Any Idea ?? > Please Help Me. > > Gregor > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > -- > --- > Fabrizio Cuseo - mailto:f.cuseo at panservice.it > Direzione Generale - Panservice InterNetWorking > Servizi Professionali per Internet ed il Networking > Panservice e' associata AIIP - RIPE Local Registry > Phone: +39 0773 410020 - Fax: +39 0773 470219 > http://www.panservice.it ?mailto:info at panservice.it > Numero verde nazionale: 800 901492 From sir_Misiek1 at o2.pl Mon Jul 29 11:52:07 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Mon, 29 Jul 2019 11:52:07 +0200 Subject: [PVE-User] Proxmox - BIG PROBLEM In-Reply-To: <91af1f0c-f7c1-770b-6e54-92eb929d1f1f@o2.pl> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> <0069e79b-2391-90ee-b6e7-061315e8e6af@gmail.com> <6a4eb127-c16b-c12f-5598-7450c0a015b5@o2.pl> <42180642.1911160.1564390827580.JavaMail.zimbra@zimbra.panservice.it> <5dc4752b-0827-332f-ad77-9e7a59cfbc0b@o2.pl> <911742305.1912254.1564391729733.JavaMail.zimbra@zimbra.panservice.it> <91af1f0c-f7c1-770b-6e54-92eb929d1f1f@o2.pl> Message-ID: <03f3a852-795a-86c7-7c74-a0a145a9e692@o2.pl> But i heave in directory binary files and command. And this not working. root at tom12:/usr/bin# ./ls ls?????????? lsblk??????? lsinitramfs? lslocks????? lsmem lsns???????? lspci??????? lsusb lsattr?????? lscpu??????? lsipc??????? lslogins???? lsmod lsof???????? lspgpot root at tomas12:/usr/bin# ./ls -bash: ./ls: No such file or directory root at tom12:/usr/bin# /usr/bin/ls -bash: /usr/bin/ls: No such file or directory Why ??!! W dniu 29.07.2019 o?11:24, lord_Niedzwiedz pisze: > >> Your system without /bin/ and /lib can't be usable, you need to >> totally recover it. > But why did rm erase them. > And the rest of the catalogs did not remove? > > I have a working VM there that I can not clone or stop. >> I personally prefer to install a new system and migrate the vm files >> (that you? have on local-zfs). But forget to use the GUI >> >> >> >> >> ----- Il 29-lug-19, alle 11:08, lord_Niedzwiedz >> ha scritto: >> >> ??? ??? VM at local-zfs. >> ??? But local-zfs not available with gui !! >> ??? VM still work. >> >> ??? And I see in: >> ??? cd /mnt/pve/ >> ??? directory: >> ??? nvme0n1 / nvme1n1 / sda / >> >> ??? Here is one virtual. >> ??? The rest on local-zfs (and they work, and I can not see the space). >> >> ??? Proxmox it's still working. >> >> ??? I lostmybe only : >> ??? /bin >> ??? /lib >> ??? /lib64 >> ??? /sbin >> ??? How is it possible that command: >> ??? rm / * >> ??? removed them ?? !! >> >> ??? Without the -r option. >> >> ??? And the rest of the catalogs did not delete ?? !! >> ??? Maybe these were symbolic links? >> >> ??? Gregor >> >> ??????? Where are located the VM's disks ? LVM ? ZFS ? >> ??????? Is possibile that you still have your disks (if LVM, for >> example), but i think that is better that you install a fresh Proxmox >> server, and move the disks from the old hard drive to the new one. >> ??????? You need some knowledge about linux, lvm, and you can save >> all your data. >> >> >> >> ??????? ----- Il 29-lug-19, alle 10:55, >> lord_Niedzwiedzsir_Misiek1 at o2.pl? ha scritto: >> >> ??????????? I ran a command on the server by mistake: >> >> ??????????? rm /* >> ??????????? rm: cannot remove '/Backup': Is a directory >> ??????????? rm: cannot remove '/boot': Is a directory >> ??????????? rm: cannot remove '/dev': Is a directory >> ??????????? rm: cannot remove '/etc': Is a directory >> ??????????? rm: cannot remove '/home': Is a directory >> ??????????? rm: cannot remove '/media': Is a directory >> ??????????? rm: cannot remove '/mnt': Is a directory >> ??????????? rm: cannot remove '/opt': Is a directory >> ??????????? rm: cannot remove '/proc': Is a directory >> ??????????? rm: cannot remove '/Roboczy': Is a directory >> ??????????? rm: cannot remove '/root': Is a directory >> ??????????? rm: cannot remove '/rpool': Is a directory >> ??????????? rm: cannot remove '/run': Is a directory >> ??????????? rm: cannot remove '/srv': Is a directory >> ??????????? rm: cannot remove '/sys': Is a directory >> ??????????? rm: cannot remove '/tmp': Is a directory >> ??????????? rm: cannot remove '/usr': Is a directory >> ??????????? rm: cannot remove '/var': Is a directory >> >> ??????????? Strange machines work. >> ??????????? I'm logged in gui. >> ??????????? But I can not get to the machine VM. >> ??????????? Do not execute any commands. >> ??????????? What to do ??!! >> ???????????? From what I see, I deleted my catalogs: >> ??????????? / bin >> ??????????? / lib >> ??????????? / lib64 >> ??????????? / sbin >> ??????????? WITH /. >> ??????????? How is this possible ??!! >> ??????????? I'm still logged in on one console after the shell, but I >> can not do any >> ??????????? commandos. >> ??????????? Even: >> ??????????? qm >> ??????????? -bash: /usr/sbin/qm: /usr/bin/perl: bad interpreter: No >> such file or >> ??????????? directory >> ??????????? root at tomas:/usr/bin# ls >> ??????????? -bash: /usr/bin/ls: No such file or directory >> ??????????? root at tomas:/usr/bin# echo $PATH >> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >> >> ??????????? Any Idea ?? >> ??????????? Please Help Me. >> >> ??????????? Gregor >> >> ??????????? _______________________________________________ >> ??????????? pve-user mailing list >> ??????????? pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> >> >> -- >> --- >> Fabrizio Cuseo - mailto:f.cuseo at panservice.it >> Direzione Generale - Panservice InterNetWorking >> Servizi Professionali per Internet ed il Networking >> Panservice e' associata AIIP - RIPE Local Registry >> Phone: +39 0773 410020 - Fax: +39 0773 470219 >> http://www.panservice.it ?mailto:info at panservice.it >> Numero verde nazionale: 800 901492 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From f.cuseo at panservice.it Mon Jul 29 11:54:37 2019 From: f.cuseo at panservice.it (f.cuseo at panservice.it) Date: Mon, 29 Jul 2019 11:54:37 +0200 (CEST) Subject: [PVE-User] Ris: Proxmox - BIG PROBLEM In-Reply-To: <03f3a852-795a-86c7-7c74-a0a145a9e692@o2.pl> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> <0069e79b-2391-90ee-b6e7-061315e8e6af@gmail.com> <6a4eb127-c16b-c12f-5598-7450c0a015b5@o2.pl> <42180642.1911160.1564390827580.JavaMail.zimbra@zimbra.panservice.it> <5dc4752b-0827-332f-ad77-9e7a59cfbc0b@o2.pl> <911742305.1912254.1564391729733.JavaMail.zimbra@zimbra.panservice.it> <91af1f0c-f7c1-770b-6e54-92eb929d1f1f@o2.pl> <03f3a852-795a-86c7-7c74-a0a145a9e692@o2.pl> Message-ID: <966946832.1913576.1564394077868.JavaMail.zimbra@zimbra.panservice.it> Because your files are dynamically linked and you don't have libraries Inviato dal mio dispositivo Huawei -------- Messaggio originale -------- Oggetto: Re: [PVE-User] Proxmox - BIG PROBLEM Da: lord_Niedzwiedz A: Fabrizio Cuseo CC: pve-user But i heave in directory binary files and command. And this not working. root at tom12:/usr/bin# ./ls ls?????????? lsblk??????? lsinitramfs? lslocks????? lsmem lsns???????? lspci??????? lsusb lsattr?????? lscpu??????? lsipc??????? lslogins???? lsmod lsof???????? lspgpot root at tomas12:/usr/bin# ./ls -bash: ./ls: No such file or directory root at tom12:/usr/bin# /usr/bin/ls -bash: /usr/bin/ls: No such file or directory Why ??!! W dniu 29.07.2019 o?11:24, lord_Niedzwiedz pisze: > >> Your system without /bin/ and /lib can't be usable, you need to >> totally recover it. > But why did rm erase them. > And the rest of the catalogs did not remove? > > I have a working VM there that I can not clone or stop. >> I personally prefer to install a new system and migrate the vm files >> (that you? have on local-zfs). But forget to use the GUI >> >> >> >> >> ----- Il 29-lug-19, alle 11:08, lord_Niedzwiedz >> ha scritto: >> >> ??? ??? VM at local-zfs. >> ??? But local-zfs not available with gui !! >> ??? VM still work. >> >> ??? And I see in: >> ??? cd /mnt/pve/ >> ??? directory: >> ??? nvme0n1 / nvme1n1 / sda / >> >> ??? Here is one virtual. >> ??? The rest on local-zfs (and they work, and I can not see the space). >> >> ??? Proxmox it's still working. >> >> ??? I lostmybe only : >> ??? /bin >> ??? /lib >> ??? /lib64 >> ??? /sbin >> ??? How is it possible that command: >> ??? rm / * >> ??? removed them ?? !! >> >> ??? Without the -r option. >> >> ??? And the rest of the catalogs did not delete ?? !! >> ??? Maybe these were symbolic links? >> >> ??? Gregor >> >> ??????? Where are located the VM's disks ? LVM ? ZFS ? >> ??????? Is possibile that you still have your disks (if LVM, for >> example), but i think that is better that you install a fresh Proxmox >> server, and move the disks from the old hard drive to the new one. >> ??????? You need some knowledge about linux, lvm, and you can save >> all your data. >> >> >> >> ??????? ----- Il 29-lug-19, alle 10:55, >> lord_Niedzwiedzsir_Misiek1 at o2.pl? ha scritto: >> >> ??????????? I ran a command on the server by mistake: >> >> ??????????? rm /* >> ??????????? rm: cannot remove '/Backup': Is a directory >> ??????????? rm: cannot remove '/boot': Is a directory >> ??????????? rm: cannot remove '/dev': Is a directory >> ??????????? rm: cannot remove '/etc': Is a directory >> ??????????? rm: cannot remove '/home': Is a directory >> ??????????? rm: cannot remove '/media': Is a directory >> ??????????? rm: cannot remove '/mnt': Is a directory >> ??????????? rm: cannot remove '/opt': Is a directory >> ??????????? rm: cannot remove '/proc': Is a directory >> ??????????? rm: cannot remove '/Roboczy': Is a directory >> ??????????? rm: cannot remove '/root': Is a directory >> ??????????? rm: cannot remove '/rpool': Is a directory >> ??????????? rm: cannot remove '/run': Is a directory >> ??????????? rm: cannot remove '/srv': Is a directory >> ??????????? rm: cannot remove '/sys': Is a directory >> ??????????? rm: cannot remove '/tmp': Is a directory >> ??????????? rm: cannot remove '/usr': Is a directory >> ??????????? rm: cannot remove '/var': Is a directory >> >> ??????????? Strange machines work. >> ??????????? I'm logged in gui. >> ??????????? But I can not get to the machine VM. >> ??????????? Do not execute any commands. >> ??????????? What to do ??!! >> ???????????? From what I see, I deleted my catalogs: >> ??????????? / bin >> ??????????? / lib >> ??????????? / lib64 >> ??????????? / sbin >> ??????????? WITH /. >> ??????????? How is this possible ??!! >> ??????????? I'm still logged in on one console after the shell, but I >> can not do any >> ??????????? commandos. >> ??????????? Even: >> ??????????? qm >> ??????????? -bash: /usr/sbin/qm: /usr/bin/perl: bad interpreter: No >> such file or >> ??????????? directory >> ??????????? root at tomas:/usr/bin# ls >> ??????????? -bash: /usr/bin/ls: No such file or directory >> ??????????? root at tomas:/usr/bin# echo $PATH >> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >> >> ??????????? Any Idea ?? >> ??????????? Please Help Me. >> >> ??????????? Gregor >> >> ??????????? _______________________________________________ >> ??????????? pve-user mailing list >> ??????????? pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> >> >> -- >> --- >> Fabrizio Cuseo - mailto:f.cuseo at panservice.it >> Direzione Generale - Panservice InterNetWorking >> Servizi Professionali per Internet ed il Networking >> Panservice e' associata AIIP - RIPE Local Registry >> Phone: +39 0773 410020 - Fax: +39 0773 470219 >> http://www.panservice.it ?mailto:info at panservice.it >> Numero verde nazionale: 800 901492 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From immo.wetzel at adtran.com Mon Jul 29 14:29:39 2019 From: immo.wetzel at adtran.com (Immo Wetzel) Date: Mon, 29 Jul 2019 12:29:39 +0000 Subject: [PVE-User] PVE 5.4 and Intel ixgbe In-Reply-To: References: Message-ID: Yes you point to the right corner. The SFP+ must be an intel one for the Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection card. Even some Finisars are right labled but not officialy supported. Check with FS.com or other supplier if you find a suitable SFP+ for your setup. Immo This message has been classified General Business by Immo Wetzel on Montag, 29. Juli 2019 at 14:27:10. From: pve-user [mailto:pve-user-bounces at pve.proxmox.com] On Behalf Of Gilberto Nunes Sent: Wednesday, July 17, 2019 8:06 PM To: PVE User List Subject: Re: [PVE-User] PVE 5.4 and Intel ixgbe So here the scenario... The customer (which is in other city, far away from my current localtion!) has the Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection but the GBIC is CISCO and the DAC cable is Mikrotik... Should CISCO GBIC incompatible with Intel card?? Or maybe the DAC cable from Mikrotik?? --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 17 de jul de 2019 ?s 14:53, Gilberto Nunes < gilberto.nunes32 at gmail.com> escreveu: > I am not sure about it, because now, after a fresh installation, even de > enp4sf0 appears... > This is so frustrated! > > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em qua, 17 de jul de 2019 ?s 14:45, Alex Chekholko via pve-user < > pve-user at pve.proxmox.com> escreveu: > >> >> >> >> ---------- Forwarded message ---------- >> From: Alex Chekholko >> To: PVE User List >> Cc: >> Bcc: >> Date: Wed, 17 Jul 2019 10:44:07 -0700 >> Subject: Re: [PVE-User] PVE 5.4 and Intel ixgbe >> You can try 'modinfo ixgbe' to query your actual installed version to see >> all the parameters it knows about. >> >> I see on one of my hosts >> # modinfo ixgbe >> filename: >> >> /lib/modules/4.15.0-54-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko >> version: 5.1.0-k >> ... >> parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ >> modules on 82599-based adapters (uint) >> >> And you can check the exact release notes for your version to see the >> allowed values for that parameter. >> >> IME, you may have some kind of incompable cable/optic anyway; see if you >> can try a different one. >> >> Regards, >> Alex >> >> On Wed, Jul 17, 2019 at 9:33 AM Gilberto Nunes < >> gilberto.nunes32 at gmail.com> >> wrote: >> >> > Hi there everybody! >> > >> > I have installed PVE 5.4 and try to up ixgbe driver for Intel 10GB >> SFI/SPF+ >> > NIC... >> > >> > I already do ixgbe-options.conf with >> > >> > options ixgbe allow_unsupported_sfp=1 >> > >> > But when try to load the module I still get this error: >> > >> > [ 170.008236] ixgbe 0000:05:00.0: failed to load because an unsupported >> > SFP+ or QSFP module type was detected. >> > >> > [ 170.008262] ixgbe 0000:05:00.0: Reload the driver after installing a >> > supported module. >> > >> > [ 170.022268] ixgbe 0000:05:00.1: failed to load because an unsupported >> > SFP+ or QSFP module type was detected. >> > >> > [ 170.022291] ixgbe 0000:05:00.1: Reload the driver after installing a >> > supported module. >> > >> > I already try to compile Intel module from scratch, but seems to failed >> > too! >> > >> > Thanks for any help! >> > >> > lspci >> > >> > 08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit >> SFI/SFP+ >> > Network Connection (rev 01) >> > 08:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit >> SFI/SFP+ >> > Network Connection (rev 01) >> > >> > >> > pveversion >> > >> > pve-manager/5.4-11/6df3d8d0 (running kernel: 4.15.18-18-pve) >> > --- >> > Gilberto Nunes Ferreira >> > >> > (47) 3025-5907 >> > (47) 99676-7530 - Whatsapp / Telegram >> > >> > Skype: gilberto.nunes36 >> > _______________________________________________ >> > pve-user mailing list >> > pve-user at pve.proxmox.com >> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > >> >> >> >> ---------- Forwarded message ---------- >> From: Alex Chekholko via pve-user >> To: PVE User List >> Cc: Alex Chekholko >> Bcc: >> Date: Wed, 17 Jul 2019 10:44:07 -0700 >> Subject: Re: [PVE-User] PVE 5.4 and Intel ixgbe >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From gaio at sv.lnf.it Mon Jul 29 15:36:38 2019 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Mon, 29 Jul 2019 15:36:38 +0200 Subject: [PVE-User] Again trouble, but this time with ext4/trim... In-Reply-To: <20190701090048.GA3471@sv.lnf.it> References: <20190701090048.GA3471@sv.lnf.it> Message-ID: <20190729133638.GK2670@sv.lnf.it> > In that servers i've also some other FS, but the ext4 ones are low > varying, mounted RO or noatime, but also some FS in XFS fs, that seems > does not suffer. I've disabled 'discard' for /dev/sda in both server, and keeped it for /dev/sdb (squid spool, ext4) and /dev/sdc (XFS filesystem). I've also added a 'randomization' for the fstrim task, but still: Jul 29 01:14:10 vdmsv1 systemd[1]: Starting Discard unused blocks... Jul 29 01:14:14 vdmsv1 kernel: [455671.833357] EXT4-fs error (device sdb1): ext4_validate_block_bitmap:386: comm fstrim: bg 97: bad block bitmap checksum Jul 29 01:14:14 vdmsv1 kernel: [455671.859823] EXT4-fs warning (device sdb1): ext4_trim_all_free:5130: Error -74 loading buddy information for 97 Jul 29 01:14:14 vdmsv1 fstrim[23986]: fstrim: /var/spool/squid: FITRIM ioctl failed: Messaggio errato Jul 29 01:15:02 vdmsv1 CRON[24072]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.service: Main process exited, code=exited, status=64/n/a Jul 29 01:15:29 vdmsv1 systemd[1]: Failed to start Discard unused blocks. Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.timer: Adding 2h 8min 41.708255s random time. Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.service: Unit entered failed state. Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.timer: Adding 3h 56min 32.453953s random time. Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.service: Failed with result 'exit-code'. So seems that i've some trouble with 'discard', at least with ext4 FS... -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From gilberto.nunes32 at gmail.com Mon Jul 29 15:54:47 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Mon, 29 Jul 2019 10:54:47 -0300 Subject: [PVE-User] PVE 5.4 and Intel ixgbe In-Reply-To: References: Message-ID: Yep! The customer (after a long discussion! Customer stubborn!) he changed from CISCO to INTEL SPF+ and everything works fine! Thanks --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em seg, 29 de jul de 2019 ?s 09:29, Immo Wetzel escreveu: > > Yes you point to the right corner. > The SFP+ must be an intel one for the Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection card. Even some Finisars are right labled but not officialy supported. > Check with FS.com or other supplier if you find a suitable SFP+ for your setup. > > Immo > > This message has been classified General Business by Immo Wetzel on Montag, 29. Juli 2019 at 14:27:10. > > From: pve-user [mailto:pve-user-bounces at pve.proxmox.com] On Behalf Of Gilberto Nunes > Sent: Wednesday, July 17, 2019 8:06 PM > To: PVE User List > Subject: Re: [PVE-User] PVE 5.4 and Intel ixgbe > > So here the scenario... > The customer (which is in other city, far away from my current localtion!) > has the Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection > but the GBIC is CISCO and the DAC cable is Mikrotik... > Should CISCO GBIC incompatible with Intel card?? Or maybe the DAC cable > from Mikrotik?? > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > > Em qua, 17 de jul de 2019 ?s 14:53, Gilberto Nunes < > gilberto.nunes32 at gmail.com> escreveu: > > > I am not sure about it, because now, after a fresh installation, even de > > enp4sf0 appears... > > This is so frustrated! > > > > > > --- > > Gilberto Nunes Ferreira > > > > (47) 3025-5907 > > (47) 99676-7530 - Whatsapp / Telegram > > > > Skype: gilberto.nunes36 > > > > > > > > > > > > Em qua, 17 de jul de 2019 ?s 14:45, Alex Chekholko via pve-user < > > pve-user at pve.proxmox.com> escreveu: > > > >> > >> > >> > >> ---------- Forwarded message ---------- > >> From: Alex Chekholko > >> To: PVE User List > >> Cc: > >> Bcc: > >> Date: Wed, 17 Jul 2019 10:44:07 -0700 > >> Subject: Re: [PVE-User] PVE 5.4 and Intel ixgbe > >> You can try 'modinfo ixgbe' to query your actual installed version to see > >> all the parameters it knows about. > >> > >> I see on one of my hosts > >> # modinfo ixgbe > >> filename: > >> > >> /lib/modules/4.15.0-54-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko > >> version: 5.1.0-k > >> ... > >> parm: allow_unsupported_sfp:Allow unsupported and untested SFP+ > >> modules on 82599-based adapters (uint) > >> > >> And you can check the exact release notes for your version to see the > >> allowed values for that parameter. > >> > >> IME, you may have some kind of incompable cable/optic anyway; see if you > >> can try a different one. > >> > >> Regards, > >> Alex > >> > >> On Wed, Jul 17, 2019 at 9:33 AM Gilberto Nunes < > >> gilberto.nunes32 at gmail.com> > >> wrote: > >> > >> > Hi there everybody! > >> > > >> > I have installed PVE 5.4 and try to up ixgbe driver for Intel 10GB > >> SFI/SPF+ > >> > NIC... > >> > > >> > I already do ixgbe-options.conf with > >> > > >> > options ixgbe allow_unsupported_sfp=1 > >> > > >> > But when try to load the module I still get this error: > >> > > >> > [ 170.008236] ixgbe 0000:05:00.0: failed to load because an unsupported > >> > SFP+ or QSFP module type was detected. > >> > > >> > [ 170.008262] ixgbe 0000:05:00.0: Reload the driver after installing a > >> > supported module. > >> > > >> > [ 170.022268] ixgbe 0000:05:00.1: failed to load because an unsupported > >> > SFP+ or QSFP module type was detected. > >> > > >> > [ 170.022291] ixgbe 0000:05:00.1: Reload the driver after installing a > >> > supported module. > >> > > >> > I already try to compile Intel module from scratch, but seems to failed > >> > too! > >> > > >> > Thanks for any help! > >> > > >> > lspci > >> > > >> > 08:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >> SFI/SFP+ > >> > Network Connection (rev 01) > >> > 08:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > >> SFI/SFP+ > >> > Network Connection (rev 01) > >> > > >> > > >> > pveversion > >> > > >> > pve-manager/5.4-11/6df3d8d0 (running kernel: 4.15.18-18-pve) > >> > --- > >> > Gilberto Nunes Ferreira > >> > > >> > (47) 3025-5907 > >> > (47) 99676-7530 - Whatsapp / Telegram > >> > > >> > Skype: gilberto.nunes36 > >> > _______________________________________________ > >> > pve-user mailing list > >> > pve-user at pve.proxmox.com > >> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >> > > >> > >> > >> > >> ---------- Forwarded message ---------- > >> From: Alex Chekholko via pve-user > >> To: PVE User List > >> Cc: Alex Chekholko > >> Bcc: > >> Date: Wed, 17 Jul 2019 10:44:07 -0700 > >> Subject: Re: [PVE-User] PVE 5.4 and Intel ixgbe > >> _______________________________________________ > >> pve-user mailing list > >> pve-user at pve.proxmox.com > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >> > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From f.gruenbichler at proxmox.com Mon Jul 29 16:05:03 2019 From: f.gruenbichler at proxmox.com (Fabian =?utf-8?Q?Gr=C3=BCnbichler?=) Date: Mon, 29 Jul 2019 16:05:03 +0200 Subject: [PVE-User] Again trouble, but this time with ext4/trim... In-Reply-To: <20190729133638.GK2670@sv.lnf.it> References: <20190701090048.GA3471@sv.lnf.it> <20190729133638.GK2670@sv.lnf.it> Message-ID: <20190729140503.h4udxcwi4dpzara7@nora.maurer-it.com> On Mon, Jul 29, 2019 at 03:36:38PM +0200, Marco Gaiarin wrote: > > > In that servers i've also some other FS, but the ext4 ones are low > > varying, mounted RO or noatime, but also some FS in XFS fs, that seems > > does not suffer. > > I've disabled 'discard' for /dev/sda in both server, and keeped it for > /dev/sdb (squid spool, ext4) and /dev/sdc (XFS filesystem). > > I've also added a 'randomization' for the fstrim task, but still: > > Jul 29 01:14:10 vdmsv1 systemd[1]: Starting Discard unused blocks... > Jul 29 01:14:14 vdmsv1 kernel: [455671.833357] EXT4-fs error (device sdb1): ext4_validate_block_bitmap:386: comm fstrim: bg 97: bad block bitmap checksum > Jul 29 01:14:14 vdmsv1 kernel: [455671.859823] EXT4-fs warning (device sdb1): ext4_trim_all_free:5130: Error -74 loading buddy information for 97 > Jul 29 01:14:14 vdmsv1 fstrim[23986]: fstrim: /var/spool/squid: FITRIM ioctl failed: Messaggio errato > Jul 29 01:15:02 vdmsv1 CRON[24072]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) > Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.service: Main process exited, code=exited, status=64/n/a > Jul 29 01:15:29 vdmsv1 systemd[1]: Failed to start Discard unused blocks. > Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.timer: Adding 2h 8min 41.708255s random time. > Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.service: Unit entered failed state. > Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.timer: Adding 3h 56min 32.453953s random time. > Jul 29 01:15:29 vdmsv1 systemd[1]: fstrim.service: Failed with result 'exit-code'. > > So seems that i've some trouble with 'discard', at least with ext4 > FS... are you sure this storage box supports fstrim at all? thin provisioning and trim/discard often go together, but they don't have to.. quick googling only showed some vmware specific integration to 'reclaim no longer used space of thin provisioned volumes', but no explicit mention of discard support. From gaio at sv.lnf.it Mon Jul 29 16:46:19 2019 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Mon, 29 Jul 2019 16:46:19 +0200 Subject: [PVE-User] Again trouble, but this time with ext4/trim... In-Reply-To: <20190729140503.h4udxcwi4dpzara7@nora.maurer-it.com> References: <20190701090048.GA3471@sv.lnf.it> <20190729133638.GK2670@sv.lnf.it> <20190729140503.h4udxcwi4dpzara7@nora.maurer-it.com> Message-ID: <20190729144619.GL2670@sv.lnf.it> Mandi! Fabian Gr?nbichler In chel di` si favelave... > are you sure this storage box supports fstrim at all? thin provisioning > and trim/discard often go together, but they don't have to.. Whai i can say is: 1) In the past, i've done 'direct iSCSI assignment' to some VM, eg the VM mount the iSCSI volume, and i can trim them flawlessy. 2) in autumn, i've created a new set ov VMs, where i've abandoned direct iSCSI mount and i use PVE volumes instead. Clearly, i've done some test for trim/discard, verifying that space get correctly reclaimed. After some weeks, i've suffered ext4 corruption as now. 3) in december i've upgraded the cluster to PVE 5; filesystem corruption desappear. 4) some weeks ago i was forced ot reboot the cluster (to change an UPS), and FS corruption reappeared. For now, i've removed the 'discard=on' options from volumes that have ext4 FS. The crazy things is that volumes formatted in XFS, residing on the same SAN, seems does not suffer at all of FS corruption. If i can do some test, please, say me. > quick googling only showed some vmware specific integration to 'reclaim > no longer used space of thin provisioned volumes', but no explicit > mention of discard support. Ahem, but trim/discard are not the same things? -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From gaio at sv.lnf.it Mon Jul 29 17:19:26 2019 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Mon, 29 Jul 2019 17:19:26 +0200 Subject: [PVE-User] Again trouble, but this time with ext4/trim... In-Reply-To: <20190729144619.GL2670@sv.lnf.it> References: <20190701090048.GA3471@sv.lnf.it> <20190729133638.GK2670@sv.lnf.it> <20190729140503.h4udxcwi4dpzara7@nora.maurer-it.com> <20190729144619.GL2670@sv.lnf.it> Message-ID: <20190729151926.GA18646@sv.lnf.it> Ah, forgot to say. > 3) in december i've upgraded the cluster to PVE 5; filesystem > corruption desappear. > 4) some weeks ago i was forced ot reboot the cluster (to change an > UPS), and FS corruption reappeared. 5) a (unused, indeed) W2016 server that was installed month before, start to complain about corrupted 'EDB' (internal database) files. After removing the 'discard' options on disk (again, on main/OS disk; there's a second 'DATA' disk with discard enabled), problem desappear. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From frederic at juliana-multimedia.com Mon Jul 29 18:55:22 2019 From: frederic at juliana-multimedia.com (=?UTF-8?B?RnLDqWTDqXJpYyBNQVNTT1Q=?=) Date: Mon, 29 Jul 2019 18:55:22 +0200 Subject: [PVE-User] Proxmox - BIG PROBLEM In-Reply-To: <6a4eb127-c16b-c12f-5598-7450c0a015b5@o2.pl> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <0f229df7-d77b-fc64-f3a6-f4bb42d2c6b3@binovo.es> <5dc024bc-fbfe-ebf9-859f-e0b1269172ab@ias.u-psud.fr> <77aa5ad9-dcff-c5df-cbcf-21e1834ad5c2@aasen.cx> <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> <0069e79b-2391-90ee-b6e7-061315e8e6af@gmail.com> <6a4eb127-c16b-c12f-5598-7450c0a015b5@o2.pl> Message-ID: Le 29/07/2019 ? 10:55, lord_Niedzwiedz a ?crit?: > I ran a command on the server by mistake: > > rm /* > rm: cannot remove '/Backup': Is a directory > rm: cannot remove '/boot': Is a directory > rm: cannot remove '/dev': Is a directory > rm: cannot remove '/etc': Is a directory > rm: cannot remove '/home': Is a directory > rm: cannot remove '/media': Is a directory > rm: cannot remove '/mnt': Is a directory > rm: cannot remove '/opt': Is a directory > rm: cannot remove '/proc': Is a directory > rm: cannot remove '/Roboczy': Is a directory > rm: cannot remove '/root': Is a directory > rm: cannot remove '/rpool': Is a directory > rm: cannot remove '/run': Is a directory > rm: cannot remove '/srv': Is a directory > rm: cannot remove '/sys': Is a directory > rm: cannot remove '/tmp': Is a directory > rm: cannot remove '/usr': Is a directory > rm: cannot remove '/var': Is a directory > > Strange machines work. > I'm logged in gui. > But I can not get to the machine VM. > Do not execute any commands. > What to do ??!! > From what I see, I deleted my catalogs: > / bin > / lib > / lib64 > / sbin > WITH /. > How is this possible ??!! You definitely have a distribution with a "/usr" merged. https://wiki.debian.org/UsrMerge The "rm" command has removed all files on root except folders. Symbolic links from "/usr" merged have also been removed. You have to recreate them by hand. Maybe by starting the machine on a live or rescue system. bin -> usr/bin lib -> usr/lib lib64 -> usr/lib64 sbin -> usr/sbin -- ============================================== | FR?D?RIC MASSOT | | http://www.juliana-multimedia.com | | mailto:frederic at juliana-multimedia.com | | +33.(0)2.97.54.77.94 +33.(0)6.67.19.95.69 | ===========================Debian=GNU/Linux=== From ntrsboy at gmail.com Tue Jul 30 20:05:03 2019 From: ntrsboy at gmail.com (Ntrs Ntrs) Date: Tue, 30 Jul 2019 13:05:03 -0500 Subject: [PVE-User] Unable to provision a virtual machine on local storage by cloning a template from a shared storage Message-ID: <79584914-074A-46D6-95DF-64F26D2164BC@gmail.com> Hi We have 2 nodes (Prox1 and Prox2) in a cluster. Each node has local-lvm LVM for KVM guests and shared NFS mount for templates available on both nodes. Prox1 storage: - Local LVM - shared NFS mount Prox2 storage: - Local LVM - shared NFS mount The templates are located on Prox1 on the NFS storage. When cloning a template from Prox1 to Prox2 the only option for 'Target storage' is the shared NFS storage, which is a deal-breaker for us. Is it possible to clone a template from one node to another if the destination storage is not shared (i.e. not available on the source node)? Are we doing something wrong? In our case we want to clone a template from Prox 1 stored on the shared NFS mount to Prox 2 on the local LVM. This is required in order to use commercially available billing software that lists the templates from a single common location (the shared NFS mount) on the order form and provisions the KVM virtual server on the first available node on it?s local LVM storage. Thanks. NT From ronny+pve-user at aasen.cx Tue Jul 30 23:23:43 2019 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Tue, 30 Jul 2019 23:23:43 +0200 Subject: [PVE-User] Unable to provision a virtual machine on local storage by cloning a template from a shared storage In-Reply-To: <79584914-074A-46D6-95DF-64F26D2164BC@gmail.com> References: <79584914-074A-46D6-95DF-64F26D2164BC@gmail.com> Message-ID: On 30.07.2019 20:05, Ntrs Ntrs wrote: > Hi > > We have 2 nodes (Prox1 and Prox2) in a cluster. Each node has local-lvm LVM for KVM guests and shared NFS mount for templates available on both nodes. > > Prox1 storage: > - Local LVM > - shared NFS mount > > Prox2 storage: > - Local LVM > - shared NFS mount > > The templates are located on Prox1 on the NFS storage. When cloning a template from Prox1 to Prox2 the only option for 'Target storage' is the shared NFS storage, which is a deal-breaker for us. > > Is it possible to clone a template from one node to another if the destination storage is not shared (i.e. not available on the source node)? Are we doing something wrong? > > In our case we want to clone a template from Prox 1 stored on the shared NFS mount to Prox 2 on the local LVM. This is required in order to use commercially available billing software that lists the templates from a single common location (the shared NFS mount) on the order form and provisions the KVM virtual server on the first available node on it?s local LVM storage. just a check... since you have only 2 nodes I assume you have found a different way to deal with the split brain problem of no quorum, when a node dies/restarts ? your cluster will of course crash and die on a node reboot if you have not. If the template is on non-shared storage on a node, it can only be cloned to storage on that node. If the template is on shared storage, it can be cloned to shared storage on any node. so you can have the template as qcow2image on shared nfs you can clone it to a? vm on the shared nfs on any node. once the vm is up you can live migrate the local if you need that. i do not know the software you talk about. but perhaps you can do some tricks like.. not have a prox template, only the disk image on nfs and instantiate a new vm with a copy of that disk image. or perhaps have the same template as a copy on each node using the same source disk. good luck Ronny From ntrsboy at gmail.com Wed Jul 31 11:49:23 2019 From: ntrsboy at gmail.com (Ntrs Ntrs) Date: Wed, 31 Jul 2019 04:49:23 -0500 Subject: [PVE-User] Unable to provision a virtual machine on local storage by cloning a template from a shared storage In-Reply-To: References: <79584914-074A-46D6-95DF-64F26D2164BC@gmail.com> Message-ID: This is just a test setup, so the risk with running two nodes is ok. Thanks for your answer. If I understand correctly our assumption that our use case is not possible with proxmox is correct. We want to be able to clone a template from a shared storage to a local storage on the node. I don?t see why that would not be technically possible, it is just copying one file from one disk to another. Both disks are visible on the node, one as an NFS mount and the other as a local disk. Is this something that proxmox can possibly add as a feature in the future or is there some limitation we are not able to see as end users? Thanks. NT > On Jul 30, 2019, at 4:23 PM, Ronny Aasen wrote: > > On 30.07.2019 20:05, Ntrs Ntrs wrote: >> Hi >> >> We have 2 nodes (Prox1 and Prox2) in a cluster. Each node has local-lvm LVM for KVM guests and shared NFS mount for templates available on both nodes. >> >> Prox1 storage: >> - Local LVM >> - shared NFS mount >> >> Prox2 storage: >> - Local LVM >> - shared NFS mount >> >> The templates are located on Prox1 on the NFS storage. When cloning a template from Prox1 to Prox2 the only option for 'Target storage' is the shared NFS storage, which is a deal-breaker for us. >> >> Is it possible to clone a template from one node to another if the destination storage is not shared (i.e. not available on the source node)? Are we doing something wrong? >> >> In our case we want to clone a template from Prox 1 stored on the shared NFS mount to Prox 2 on the local LVM. This is required in order to use commercially available billing software that lists the templates from a single common location (the shared NFS mount) on the order form and provisions the KVM virtual server on the first available node on it?s local LVM storage. > > > just a check... since you have only 2 nodes I assume you have found a different way to deal with the split brain problem of no quorum, when a node dies/restarts ? your cluster will of course crash and die on a node reboot if you have not. > > > If the template is on non-shared storage on a node, it can only be cloned to storage on that node. If the template is on shared storage, it can be cloned to shared storage on any node. > so you can have the template as qcow2image on shared nfs you can clone it to a vm on the shared nfs on any node. once the vm is up you can live migrate the local if you need that. > > i do not know the software you talk about. but perhaps you can do some tricks like.. not have a prox template, only the disk image on nfs and instantiate a new vm with a copy of that disk image. or perhaps have the same template as a copy on each node using the same source disk. > > > good luck > > Ronny > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From cj0nes at live.com Wed Jul 31 14:27:35 2019 From: cj0nes at live.com (Craig Jones) Date: Wed, 31 Jul 2019 12:27:35 +0000 Subject: [PVE-User] Host Rebooting when using PCIe Passthrough In-Reply-To: <70c519ac-d73d-1dbd-9a3c-1c3ae5fc89e4@proxmox.com> References: <70c519ac-d73d-1dbd-9a3c-1c3ae5fc89e4@proxmox.com> Message-ID: Just wanted to follow through on this. I took the NIC out and was still having reboots. I swapped the GPU with a different one and have not had anymore reboots. Hopefully this remains when I add a NIC back, but the resolution was swapping the card. There's something funky with the original GPU I was using. Thanks for the suggestions! On 7/5/2019 1:23 AM, Dominik Csapak wrote: > On 7/4/19 8:45 PM, Craig Jones wrote: >> Hello, >> >> I have a VM that I'm passing a GPU through to. The passthrough itself >> works great. The issue is that whenever this VM is powered on, the host >> will reboot without any interaction from me. The reboot happens anywhere >> from 3 - 15 minutes after the VM has been powered on. I have many other >> VMs that don't cause this. The only difference between them and this one >> is the passthrough GPU. Attached are some potentially helpful outputs. >> The syslogs have been truncated from when the VM had been powered on to >> the last entry right before the host rebooted. >> >> Thanks, >> Craig >> >> > > one thing you could do is setup kernel crash logging (kdump) to see > if the kernel crashes and why > > aside from that the only thing i see is that your gpu is not > in an isolated iommu group: > > ----8<---- > /sys/kernel/iommu_groups/1/devices/0000:00:01.0 > /sys/kernel/iommu_groups/1/devices/0000:00:01.1 > /sys/kernel/iommu_groups/1/devices/0000:01:00.0 > /sys/kernel/iommu_groups/1/devices/0000:01:00.1 > /sys/kernel/iommu_groups/1/devices/0000:02:00.0 > ---->8---- > > 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. > [AMD/ATI] RV770 [Radeon HD 4870] > 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV770 > HDMI Audio [Radeon HD 4850/4870] > 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. > RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07) > > it seems it is together in a group with your nic > > this can be the cause for the crashes... > > hope this helps > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus From gianni.milo22 at gmail.com Wed Jul 31 14:35:37 2019 From: gianni.milo22 at gmail.com (Gianni Milo) Date: Wed, 31 Jul 2019 13:35:37 +0100 Subject: [PVE-User] Unable to provision a virtual machine on local storage by cloning a template from a shared storage In-Reply-To: References: <79584914-074A-46D6-95DF-64F26D2164BC@gmail.com> Message-ID: > We want to be able to clone a template from a shared storage to a local > storage on the node. I don?t see why that would not be technically > possible, it is just copying one file from one disk to another. When you right click on the "template -> clone" and you choose "mode:full clone", it gives you the option to select the "target storage" and the "target node". You can choose for example as "target storage:Local LVM" and as target node: the node you want to clone this VM to. Is this what you want to achieve ? You cannot choose a different target storage other than the source, when the selected mode is "linked clone". Gianni From ntrsboy at gmail.com Wed Jul 31 14:45:49 2019 From: ntrsboy at gmail.com (Ntrs Ntrs) Date: Wed, 31 Jul 2019 07:45:49 -0500 Subject: [PVE-User] Unable to provision a virtual machine on local storage by cloning a template from a shared storage In-Reply-To: References: <79584914-074A-46D6-95DF-64F26D2164BC@gmail.com> Message-ID: <4F16560A-ACA3-4924-BF47-CE5E2C5F26BF@gmail.com> I want to "full clone? a template from a shared storage on a node (NFS) to the local storage on the same node (SSD) and the local storage is not listed in the target storage drop down box. Take a look at the attached image. Maybe it?s important to say this was with PVE 5.1. If anything about this changed with 6.0 perhaps we should retry with a later version. > On Jul 31, 2019, at 7:35 AM, Gianni Milo wrote: > >> We want to be able to clone a template from a shared storage to a local >> storage on the node. I don?t see why that would not be technically >> possible, it is just copying one file from one disk to another. > > > When you right click on the "template -> clone" and you choose "mode:full > clone", it gives you the option to select the "target storage" and the > "target node". > You can choose for example as "target storage:Local LVM" and as target > node: the node you want to clone this VM to. Is this what you want to > achieve ? > You cannot choose a different target storage other than the source, when > the selected mode is "linked clone". > > Gianni > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From gianni.milo22 at gmail.com Wed Jul 31 14:54:35 2019 From: gianni.milo22 at gmail.com (Gianni Milo) Date: Wed, 31 Jul 2019 13:54:35 +0100 Subject: [PVE-User] Unable to provision a virtual machine on local storage by cloning a template from a shared storage In-Reply-To: <4F16560A-ACA3-4924-BF47-CE5E2C5F26BF@gmail.com> References: <79584914-074A-46D6-95DF-64F26D2164BC@gmail.com> <4F16560A-ACA3-4924-BF47-CE5E2C5F26BF@gmail.com> Message-ID: > I want to "full clone? a template from a shared storage on a node (NFS) to > the local storage on the same node (SSD) and the local storage is not > listed in the target storage drop down box. Take a look at the attached > image. Maybe it?s important to say this was with PVE 5.1. If anything about > this changed with 6.0 perhaps we should retry with a later version. > This definitely should be possible as long as you have selected in the "Datacenter" -> "Storage" -> "Local SSD storage" -> "Enable:yes" -> "Nodes:all" -> "Content:Disk Image,Container" . From adamw at matrixscience.com Wed Jul 31 17:37:32 2019 From: adamw at matrixscience.com (Adam Weremczuk) Date: Wed, 31 Jul 2019 16:37:32 +0100 Subject: [PVE-User] converting privileged CT to unprivileged Message-ID: <62bf7660-2b9b-a1c7-f92c-72efda0ec661@matrixscience.com> Hi all, PVE 5.4.6. My container was created as privileged and runs on zfs pool shared by 2 hosts. I've unsuccessfully tried to convert it from GUI: - stopped the container - took a backup - clicked "restore" ("unprivileged" ticked - default) extracting archive '/var/lib/vz/dump/vzdump-lxc-100-2019_07_31-16_15_48.tar.lzo' tar: ./var/spool/postfix/dev/urandom: Cannot mknod: Operation not permitted tar: ./var/spool/postfix/dev/random: Cannot mknod: Operation not permitted Total bytes read: 619950080 (592MiB, 42MiB/s) tar: Exiting with failure status due to previous errors TASK ERROR: unable to restore CT 100 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar xpf - --lzop --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/100/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 2 CT 100 completely disappeared from the list! Earlier attempt from shell (105 was the first available ID): pct restore 105 /var/lib/vz/dump/vzdump-lxc-100-2019_07_31-16_15_48.tar.lzo -ignore-unpack-errors 1 -unprivileged 400 Parameter verification failed. storage: storage 'local' does not support container directories pct restore [OPTIONS] Any hints? Thanks, Adam From gilberto.nunes32 at gmail.com Wed Jul 31 17:39:30 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 31 Jul 2019 12:39:30 -0300 Subject: [PVE-User] converting privileged CT to unprivileged In-Reply-To: <62bf7660-2b9b-a1c7-f92c-72efda0ec661@matrixscience.com> References: <62bf7660-2b9b-a1c7-f92c-72efda0ec661@matrixscience.com> Message-ID: You can uncompress the backup for any other directory, delete urandom and random and then, compress again the whole directory... Then try restore into PVE again. --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 31 de jul de 2019 ?s 12:37, Adam Weremczuk escreveu: > > Hi all, > > PVE 5.4.6. > > My container was created as privileged and runs on zfs pool shared by 2 > hosts. > > I've unsuccessfully tried to convert it from GUI: > - stopped the container > - took a backup > - clicked "restore" ("unprivileged" ticked - default) > > extracting archive > '/var/lib/vz/dump/vzdump-lxc-100-2019_07_31-16_15_48.tar.lzo' > tar: ./var/spool/postfix/dev/urandom: Cannot mknod: Operation not permitted > tar: ./var/spool/postfix/dev/random: Cannot mknod: Operation not permitted > Total bytes read: 619950080 (592MiB, 42MiB/s) > tar: Exiting with failure status due to previous errors > TASK ERROR: unable to restore CT 100 - command 'lxc-usernsexec -m > u:0:100000:65536 -m g:0:100000:65536 -- tar xpf - --lzop --totals > --one-file-system -p --sparse --numeric-owner --acls --xattrs > '--xattrs-include=user.*' '--xattrs-include=security.capability' > '--warning=no-file-ignored' '--warning=no-xattr-write' -C > /var/lib/lxc/100/rootfs --skip-old-files --anchored --exclude './dev/*'' > failed: exit code 2 > > CT 100 completely disappeared from the list! > > Earlier attempt from shell (105 was the first available ID): > > pct restore 105 > /var/lib/vz/dump/vzdump-lxc-100-2019_07_31-16_15_48.tar.lzo > -ignore-unpack-errors 1 -unprivileged > 400 Parameter verification failed. > storage: storage 'local' does not support container directories > pct restore [OPTIONS] > > Any hints? > > Thanks, > Adam > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From pve at junkyard.4t2.com Wed Jul 31 18:05:28 2019 From: pve at junkyard.4t2.com (Tom Weber) Date: Wed, 31 Jul 2019 18:05:28 +0200 Subject: [PVE-User] converting privileged CT to unprivileged In-Reply-To: References: <62bf7660-2b9b-a1c7-f92c-72efda0ec661@matrixscience.com> Message-ID: Or just delete these files before shutting down the container and making the backup. Tom Am Mittwoch, den 31.07.2019, 12:39 -0300 schrieb Gilberto Nunes: > You can uncompress the backup for any other directory, delete urandom > and random and then, compress again the whole directory... > Then try restore into PVE again. > > --- > Gilberto Nunes Ferreira > > (47) 3025-5907 > (47) 99676-7530 - Whatsapp / Telegram > > Skype: gilberto.nunes36 > > > > > Em qua, 31 de jul de 2019 ?s 12:37, Adam Weremczuk > escreveu: > > > > Hi all, > > > > PVE 5.4.6. > > > > My container was created as privileged and runs on zfs pool shared > > by 2 > > hosts. > > > > I've unsuccessfully tried to convert it from GUI: > > - stopped the container > > - took a backup > > - clicked "restore" ("unprivileged" ticked - default) > > > > extracting archive > > '/var/lib/vz/dump/vzdump-lxc-100-2019_07_31-16_15_48.tar.lzo' > > tar: ./var/spool/postfix/dev/urandom: Cannot mknod: Operation not > > permitted > > tar: ./var/spool/postfix/dev/random: Cannot mknod: Operation not > > permitted > > Total bytes read: 619950080 (592MiB, 42MiB/s) > > tar: Exiting with failure status due to previous errors > > TASK ERROR: unable to restore CT 100 - command 'lxc-usernsexec -m > > u:0:100000:65536 -m g:0:100000:65536 -- tar xpf - --lzop --totals > > --one-file-system -p --sparse --numeric-owner --acls --xattrs > > '--xattrs-include=user.*' '--xattrs-include=security.capability' > > '--warning=no-file-ignored' '--warning=no-xattr-write' -C > > /var/lib/lxc/100/rootfs --skip-old-files --anchored --exclude > > './dev/*'' > > failed: exit code 2 > > > > CT 100 completely disappeared from the list! > > > > Earlier attempt from shell (105 was the first available ID): > > > > pct restore 105 > > /var/lib/vz/dump/vzdump-lxc-100-2019_07_31-16_15_48.tar.lzo > > -ignore-unpack-errors 1 -unprivileged > > 400 Parameter verification failed. > > storage: storage 'local' does not support container directories > > pct restore [OPTIONS] > > > > Any hints? > > > > Thanks, > > Adam > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From gilberto.nunes32 at gmail.com Wed Jul 31 19:44:33 2019 From: gilberto.nunes32 at gmail.com (Gilberto Nunes) Date: Wed, 31 Jul 2019 14:44:33 -0300 Subject: [PVE-User] converting privileged CT to unprivileged In-Reply-To: References: <62bf7660-2b9b-a1c7-f92c-72efda0ec661@matrixscience.com> Message-ID: Or do that! Yep --- Gilberto Nunes Ferreira (47) 3025-5907 (47) 99676-7530 - Whatsapp / Telegram Skype: gilberto.nunes36 Em qua, 31 de jul de 2019 ?s 13:05, Tom Weber escreveu: > > Or just delete these files before shutting down the container and > making the backup. > > Tom > > Am Mittwoch, den 31.07.2019, 12:39 -0300 schrieb Gilberto Nunes: > > You can uncompress the backup for any other directory, delete urandom > > and random and then, compress again the whole directory... > > Then try restore into PVE again. > > > > --- > > Gilberto Nunes Ferreira > > > > (47) 3025-5907 > > (47) 99676-7530 - Whatsapp / Telegram > > > > Skype: gilberto.nunes36 > > > > > > > > > > Em qua, 31 de jul de 2019 ?s 12:37, Adam Weremczuk > > escreveu: > > > > > > Hi all, > > > > > > PVE 5.4.6. > > > > > > My container was created as privileged and runs on zfs pool shared > > > by 2 > > > hosts. > > > > > > I've unsuccessfully tried to convert it from GUI: > > > - stopped the container > > > - took a backup > > > - clicked "restore" ("unprivileged" ticked - default) > > > > > > extracting archive > > > '/var/lib/vz/dump/vzdump-lxc-100-2019_07_31-16_15_48.tar.lzo' > > > tar: ./var/spool/postfix/dev/urandom: Cannot mknod: Operation not > > > permitted > > > tar: ./var/spool/postfix/dev/random: Cannot mknod: Operation not > > > permitted > > > Total bytes read: 619950080 (592MiB, 42MiB/s) > > > tar: Exiting with failure status due to previous errors > > > TASK ERROR: unable to restore CT 100 - command 'lxc-usernsexec -m > > > u:0:100000:65536 -m g:0:100000:65536 -- tar xpf - --lzop --totals > > > --one-file-system -p --sparse --numeric-owner --acls --xattrs > > > '--xattrs-include=user.*' '--xattrs-include=security.capability' > > > '--warning=no-file-ignored' '--warning=no-xattr-write' -C > > > /var/lib/lxc/100/rootfs --skip-old-files --anchored --exclude > > > './dev/*'' > > > failed: exit code 2 > > > > > > CT 100 completely disappeared from the list! > > > > > > Earlier attempt from shell (105 was the first available ID): > > > > > > pct restore 105 > > > /var/lib/vz/dump/vzdump-lxc-100-2019_07_31-16_15_48.tar.lzo > > > -ignore-unpack-errors 1 -unprivileged > > > 400 Parameter verification failed. > > > storage: storage 'local' does not support container directories > > > pct restore [OPTIONS] > > > > > > Any hints? > > > > > > Thanks, > > > Adam > > > > > > _______________________________________________ > > > pve-user mailing list > > > pve-user at pve.proxmox.com > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From dmateos at redsa.com Mon Jul 29 11:31:13 2019 From: dmateos at redsa.com (David Mateos Murillo) Date: Mon, 29 Jul 2019 11:31:13 +0200 Subject: [PVE-User] Proxmox - BIG PROBLEM In-Reply-To: <91af1f0c-f7c1-770b-6e54-92eb929d1f1f@o2.pl> References: <58b2cd7b-032d-2b63-1302-db798acf7843@ias.u-psud.fr> <5854eebd-03ed-1835-a503-fed82b578dcc@o2.pl> <0069e79b-2391-90ee-b6e7-061315e8e6af@gmail.com> <6a4eb127-c16b-c12f-5598-7450c0a015b5@o2.pl> <42180642.1911160.1564390827580.JavaMail.zimbra@zimbra.panservice.it> <5dc4752b-0827-332f-ad77-9e7a59cfbc0b@o2.pl> <911742305.1912254.1564391729733.JavaMail.zimbra@zimbra.panservice.it> <91af1f0c-f7c1-770b-6e54-92eb929d1f1f@o2.pl> Message-ID: <573abbea-ce95-651e-b6e2-6b2084b5dfd7@redsa.com> Probably the rest of the folders are under other partitions El 29/7/19 a las 11:24, lord_Niedzwiedz escribi?: > >> Your system without /bin/ and /lib can't be usable, you need to >> totally recover it. > But why did rm erase them. > And the rest of the catalogs did not remove? > > I have a working VM there that I can not clone or stop. >> I personally prefer to install a new system and migrate the vm files >> (that you? have on local-zfs). But forget to use the GUI >> >> >> >> >> ----- Il 29-lug-19, alle 11:08, lord_Niedzwiedz >> ha scritto: >> >> ??? ??? VM at local-zfs. >> ??? But local-zfs not available with gui !! >> ??? VM still work. >> >> ??? And I see in: >> ??? cd /mnt/pve/ >> ??? directory: >> ??? nvme0n1 / nvme1n1 / sda / >> >> ??? Here is one virtual. >> ??? The rest on local-zfs (and they work, and I can not see the space). >> >> ??? Proxmox it's still working. >> >> ??? I lostmybe only : >> ??? /bin >> ??? /lib >> ??? /lib64 >> ??? /sbin >> ??? How is it possible that command: >> ??? rm / * >> ??? removed them ?? !! >> >> ??? Without the -r option. >> >> ??? And the rest of the catalogs did not delete ?? !! >> ??? Maybe these were symbolic links? >> >> ??? Gregor >> >> ??????? Where are located the VM's disks ? LVM ? ZFS ? >> ??????? Is possibile that you still have your disks (if LVM, for >> example), but i think that is better that you install a fresh Proxmox >> server, and move the disks from the old hard drive to the new one. >> ??????? You need some knowledge about linux, lvm, and you can save >> all your data. >> >> >> >> ??????? ----- Il 29-lug-19, alle 10:55, >> lord_Niedzwiedzsir_Misiek1 at o2.pl? ha scritto: >> >> ??????????? I ran a command on the server by mistake: >> >> ??????????? rm /* >> ??????????? rm: cannot remove '/Backup': Is a directory >> ??????????? rm: cannot remove '/boot': Is a directory >> ??????????? rm: cannot remove '/dev': Is a directory >> ??????????? rm: cannot remove '/etc': Is a directory >> ??????????? rm: cannot remove '/home': Is a directory >> ??????????? rm: cannot remove '/media': Is a directory >> ??????????? rm: cannot remove '/mnt': Is a directory >> ??????????? rm: cannot remove '/opt': Is a directory >> ??????????? rm: cannot remove '/proc': Is a directory >> ??????????? rm: cannot remove '/Roboczy': Is a directory >> ??????????? rm: cannot remove '/root': Is a directory >> ??????????? rm: cannot remove '/rpool': Is a directory >> ??????????? rm: cannot remove '/run': Is a directory >> ??????????? rm: cannot remove '/srv': Is a directory >> ??????????? rm: cannot remove '/sys': Is a directory >> ??????????? rm: cannot remove '/tmp': Is a directory >> ??????????? rm: cannot remove '/usr': Is a directory >> ??????????? rm: cannot remove '/var': Is a directory >> >> ??????????? Strange machines work. >> ??????????? I'm logged in gui. >> ??????????? But I can not get to the machine VM. >> ??????????? Do not execute any commands. >> ??????????? What to do ??!! >> ???????????? From what I see, I deleted my catalogs: >> ??????????? / bin >> ??????????? / lib >> ??????????? / lib64 >> ??????????? / sbin >> ??????????? WITH /. >> ??????????? How is this possible ??!! >> ??????????? I'm still logged in on one console after the shell, but I >> can not do any >> ??????????? commandos. >> ??????????? Even: >> ??????????? qm >> ??????????? -bash: /usr/sbin/qm: /usr/bin/perl: bad interpreter: No >> such file or >> ??????????? directory >> ??????????? root at tomas:/usr/bin# ls >> ??????????? -bash: /usr/bin/ls: No such file or directory >> ??????????? root at tomas:/usr/bin# echo $PATH >> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >> >> ??????????? Any Idea ?? >> ??????????? Please Help Me. >> >> ??????????? Gregor >> >> ??????????? _______________________________________________ >> ??????????? pve-user mailing list >> ??????????? pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >> >> >> -- >> --- >> Fabrizio Cuseo - mailto:f.cuseo at panservice.it >> Direzione Generale - Panservice InterNetWorking >> Servizi Professionali per Internet ed il Networking >> Panservice e' associata AIIP - RIPE Local Registry >> Phone: +39 0773 410020 - Fax: +39 0773 470219 >> http://www.panservice.it ?mailto:info at panservice.it >> Numero verde nazionale: 800 901492 > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From sir_Misiek1 at o2.pl Wed Jul 24 12:03:48 2019 From: sir_Misiek1 at o2.pl (lord_Niedzwiedz) Date: Wed, 24 Jul 2019 12:03:48 +0200 Subject: [PVE-User] Proxmox instalation problem In-Reply-To: <8a6f5909-109c-f10a-a738-70ab8062d851@aasen.cx> References: <7402932b-7743-1cfa-d1ee-2b29ceac91c2@proxmox.com> <8a6f5909-109c-f10a-a738-70ab8062d851@aasen.cx> Message-ID: Hello, I have a problem with the Proxmox installation. IBM 3650 (7979) 6 * SAS hdd server. Proxmox only starts with RAID Hardware. It does not start with RAID-Z and RAID1. Error in the attachment kind regards Gregor