From viniciuschagas2 at gmail.com Mon Jul 2 04:16:36 2018 From: viniciuschagas2 at gmail.com (Vinicius Barreto) Date: Sun, 1 Jul 2018 23:16:36 -0300 Subject: [PVE-User] which service is responsible for mounting the NFS storages during the startup of the Proxmox? Message-ID: Hello, please, would anyone know to tell me which service is responsible for mounting the NFS storages during the startup of the Proxmox? Note: Added by GUI or Pvesm. I ask because there is another service that I added to the server and it depends on the NFS drives being mounted at startup. So I plan to configure my service to start after this service that mounts the NFS drives during the proxmox boot. Thanks in advance. King regards -- *Marcos Vin?cius Barreto das Chagas* From t.lamprecht at proxmox.com Mon Jul 2 09:00:44 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 2 Jul 2018 09:00:44 +0200 Subject: [PVE-User] which service is responsible for mounting the NFS storages during the startup of the Proxmox? In-Reply-To: References: Message-ID: On 7/2/18 4:16 AM, Vinicius Barreto wrote: > Hello, > please, would anyone know to tell me which service is responsible for > mounting the NFS storages during the startup of the Proxmox? > Note: Added by GUI or Pvesm. > We have a logic that we activate volumes of what we need. E.g., when a VM is started then we activate it's volume which in turn mounts the underlying storage, if not already mounted. pvestatd checks on all configured enabled storages and also mount's them to be able to get it's status/usage information. > I ask because there is another service that I added to the server and it > depends on the NFS drives being mounted at startup. > So I plan to configure my service to start after this service that mounts > the NFS drives during the proxmox boot. > IMO you should do an after/require on pve-storage.targetd and pvestatd.service - you then could eventually wait heuristically a bit to ensure that all storages are mounted. Or you use PVE libraries to activate all storages (i.e., mount them) it could look like: ---- #!/usr/bin/perl use strict; use warnings; use PVE::Storage; use PVE::Cluster; PVE::Cluster::cfs_update(); my $storage_cfg = PVE::Storage::config(); my $all_sids = [ keys %{$storage_cfg->{ids}} ]; PVE::Storage::activate_storage_list($storage_cfg, $all_sids); ---- This could be done a pre start hook. cheers, Thomas From mark at openvs.co.uk Tue Jul 3 02:05:51 2018 From: mark at openvs.co.uk (Mark Adams) Date: Tue, 3 Jul 2018 01:05:51 +0100 Subject: [PVE-User] pveceph createosd after destroyed osd Message-ID: Currently running the newest 5.2-1 version, I had a test cluster which was working fine. I since added more disks, first stopping, then setting out, then destroying each osd so I could recreate it all from scratch. However, when adding a new osd (either via GUI or pveceph CLI) it seems to show a successful create, however does not show in the gui as an osd under the host. It's like the osd information is being stored by proxmox/ceph somewhere else and not being correctly removed and recreated? I can see that the newly created disk (after it being destroyed) is down/out. Is this by design? is there a way to force the disk back? shouldn't it show in the gui once you create it again? Thanks! From ken.woods at alaska.gov Tue Jul 3 02:34:15 2018 From: ken.woods at alaska.gov (Woods, Ken A (DNR)) Date: Tue, 3 Jul 2018 00:34:15 +0000 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: References: Message-ID: http://docs.ceph.com/docs/mimic/rados/operations/add-or-rm-osds/#removing-osds-manual Are you sure you followed the directions? ________________________________ From: pve-user on behalf of Mark Adams Sent: Monday, July 2, 2018 4:05:51 PM To: pve-user at pve.proxmox.com Subject: [PVE-User] pveceph createosd after destroyed osd Currently running the newest 5.2-1 version, I had a test cluster which was working fine. I since added more disks, first stopping, then setting out, then destroying each osd so I could recreate it all from scratch. However, when adding a new osd (either via GUI or pveceph CLI) it seems to show a successful create, however does not show in the gui as an osd under the host. It's like the osd information is being stored by proxmox/ceph somewhere else and not being correctly removed and recreated? I can see that the newly created disk (after it being destroyed) is down/out. Is this by design? is there a way to force the disk back? shouldn't it show in the gui once you create it again? Thanks! _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From mark at openvs.co.uk Tue Jul 3 02:41:39 2018 From: mark at openvs.co.uk (Mark Adams) Date: Tue, 3 Jul 2018 01:41:39 +0100 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: References: Message-ID: Hi, Thanks for your response! No, I didn't do any of that on the cli - I just did stop in the webgui, then out, then destroy. Note that there was no VM's or data at all on this test ceph cluster - I had deleted it all before doing this. I was basically just removing it all so the OSD numbers looked "nicer" for the final setup. It's not a huge deal, I can just reinstall proxmox. But it concerns me that it seems so fragile using the webgui to do this. I want to know where I went wrong? Is there somewhere that a signature is being stored so when you try to add that same drive again (even though I ticked "remove partitions") it doesn't add back in to the ceph cluster in the next sequential order from the last current "live" or "valid" drive? Is it just a rule that you never actually remove drives? you just set them stopped/out? Regards, Mark On 3 July 2018 at 01:34, Woods, Ken A (DNR) wrote: > http://docs.ceph.com/docs/mimic/rados/operations/add-or- > rm-osds/#removing-osds-manual > > Are you sure you followed the directions? > > ________________________________ > From: pve-user on behalf of Mark Adams > > Sent: Monday, July 2, 2018 4:05:51 PM > To: pve-user at pve.proxmox.com > Subject: [PVE-User] pveceph createosd after destroyed osd > > Currently running the newest 5.2-1 version, I had a test cluster which was > working fine. I since added more disks, first stopping, then setting out, > then destroying each osd so I could recreate it all from scratch. > > However, when adding a new osd (either via GUI or pveceph CLI) it seems to > show a successful create, however does not show in the gui as an osd under > the host. > > It's like the osd information is being stored by proxmox/ceph somewhere > else and not being correctly removed and recreated? > > I can see that the newly created disk (after it being destroyed) is > down/out. > > Is this by design? is there a way to force the disk back? shouldn't it show > in the gui once you create it again? > > Thanks! > > From ken.woods at alaska.gov Tue Jul 3 02:48:30 2018 From: ken.woods at alaska.gov (Woods, Ken A (DNR)) Date: Tue, 3 Jul 2018 00:48:30 +0000 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: References: , Message-ID: You're thinking "proxmox". Try thinking "ceph" instead. Sure, ceph runs with proxmox, but what you're really doing is using a pretty GUI that sits on top of debian, running ceph and kvm. Anyway, perhaps the GUI does all the steps needed? Perhaps not. If it were me, I'd NOT reinstall, as that's likely not going to fix the issue. Follow the directions in the page I linked and see if that helps. ________________________________ From: pve-user on behalf of Mark Adams Sent: Monday, July 2, 2018 4:41:39 PM To: PVE User List Subject: Re: [PVE-User] pveceph createosd after destroyed osd Hi, Thanks for your response! No, I didn't do any of that on the cli - I just did stop in the webgui, then out, then destroy. Note that there was no VM's or data at all on this test ceph cluster - I had deleted it all before doing this. I was basically just removing it all so the OSD numbers looked "nicer" for the final setup. It's not a huge deal, I can just reinstall proxmox. But it concerns me that it seems so fragile using the webgui to do this. I want to know where I went wrong? Is there somewhere that a signature is being stored so when you try to add that same drive again (even though I ticked "remove partitions") it doesn't add back in to the ceph cluster in the next sequential order from the last current "live" or "valid" drive? Is it just a rule that you never actually remove drives? you just set them stopped/out? Regards, Mark On 3 July 2018 at 01:34, Woods, Ken A (DNR) wrote: > http://docs.ceph.com/docs/mimic/rados/operations/add-or- > rm-osds/#removing-osds-manual > > Are you sure you followed the directions? > > ________________________________ > From: pve-user on behalf of Mark Adams > > Sent: Monday, July 2, 2018 4:05:51 PM > To: pve-user at pve.proxmox.com > Subject: [PVE-User] pveceph createosd after destroyed osd > > Currently running the newest 5.2-1 version, I had a test cluster which was > working fine. I since added more disks, first stopping, then setting out, > then destroying each osd so I could recreate it all from scratch. > > However, when adding a new osd (either via GUI or pveceph CLI) it seems to > show a successful create, however does not show in the gui as an osd under > the host. > > It's like the osd information is being stored by proxmox/ceph somewhere > else and not being correctly removed and recreated? > > I can see that the newly created disk (after it being destroyed) is > down/out. > > Is this by design? is there a way to force the disk back? shouldn't it show > in the gui once you create it again? > > Thanks! > > _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From ken.woods at alaska.gov Tue Jul 3 02:50:34 2018 From: ken.woods at alaska.gov (Woods, Ken A (DNR)) Date: Tue, 3 Jul 2018 00:50:34 +0000 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: References: , , Message-ID: 1. ceph osd purge {id} --yes-i-really-mean-it 2. Navigate to the host where you keep the master copy of the cluster?s ceph.conf file. ssh {admin-host} cd /etc/ceph vim ceph.conf 3. Remove the OSD entry from your ceph.conf file (if it exists). [osd.1] host = {hostname} ________________________________ From: Woods, Ken A (DNR) Sent: Monday, July 2, 2018 4:48:30 PM To: PVE User List Subject: Re: [PVE-User] pveceph createosd after destroyed osd You're thinking "proxmox". Try thinking "ceph" instead. Sure, ceph runs with proxmox, but what you're really doing is using a pretty GUI that sits on top of debian, running ceph and kvm. Anyway, perhaps the GUI does all the steps needed? Perhaps not. If it were me, I'd NOT reinstall, as that's likely not going to fix the issue. Follow the directions in the page I linked and see if that helps. ________________________________ From: pve-user on behalf of Mark Adams Sent: Monday, July 2, 2018 4:41:39 PM To: PVE User List Subject: Re: [PVE-User] pveceph createosd after destroyed osd Hi, Thanks for your response! No, I didn't do any of that on the cli - I just did stop in the webgui, then out, then destroy. Note that there was no VM's or data at all on this test ceph cluster - I had deleted it all before doing this. I was basically just removing it all so the OSD numbers looked "nicer" for the final setup. It's not a huge deal, I can just reinstall proxmox. But it concerns me that it seems so fragile using the webgui to do this. I want to know where I went wrong? Is there somewhere that a signature is being stored so when you try to add that same drive again (even though I ticked "remove partitions") it doesn't add back in to the ceph cluster in the next sequential order from the last current "live" or "valid" drive? Is it just a rule that you never actually remove drives? you just set them stopped/out? Regards, Mark On 3 July 2018 at 01:34, Woods, Ken A (DNR) wrote: > http://docs.ceph.com/docs/mimic/rados/operations/add-or- > rm-osds/#removing-osds-manual > > Are you sure you followed the directions? > > ________________________________ > From: pve-user on behalf of Mark Adams > > Sent: Monday, July 2, 2018 4:05:51 PM > To: pve-user at pve.proxmox.com > Subject: [PVE-User] pveceph createosd after destroyed osd > > Currently running the newest 5.2-1 version, I had a test cluster which was > working fine. I since added more disks, first stopping, then setting out, > then destroying each osd so I could recreate it all from scratch. > > However, when adding a new osd (either via GUI or pveceph CLI) it seems to > show a successful create, however does not show in the gui as an osd under > the host. > > It's like the osd information is being stored by proxmox/ceph somewhere > else and not being correctly removed and recreated? > > I can see that the newly created disk (after it being destroyed) is > down/out. > > Is this by design? is there a way to force the disk back? shouldn't it show > in the gui once you create it again? > > Thanks! > > _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From a.antreich at proxmox.com Tue Jul 3 11:07:43 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Tue, 3 Jul 2018 11:07:43 +0200 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: References: Message-ID: <20180703090743.4gszjjg77g4matln@dona.proxmox.com> On Tue, Jul 03, 2018 at 01:05:51AM +0100, Mark Adams wrote: > Currently running the newest 5.2-1 version, I had a test cluster which was > working fine. I since added more disks, first stopping, then setting out, > then destroying each osd so I could recreate it all from scratch. > > However, when adding a new osd (either via GUI or pveceph CLI) it seems to > show a successful create, however does not show in the gui as an osd under > the host. > > It's like the osd information is being stored by proxmox/ceph somewhere > else and not being correctly removed and recreated? > > I can see that the newly created disk (after it being destroyed) is > down/out. > > Is this by design? is there a way to force the disk back? shouldn't it show > in the gui once you create it again? > Did you zero your disks after removal? On the first ~200 MB of the disk, there are leftovers that need to be zeroed before use. After that the OSD should be added fine. -- Cheers, Alwin From mark at tuxis.nl Tue Jul 3 12:23:19 2018 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 03 Jul 2018 12:23:19 +0200 Subject: [PVE-User] VM remains in snap-delete state In-Reply-To: <1530104951.2625.3.camel@tuxis.nl> References: <1530104951.2625.3.camel@tuxis.nl> Message-ID: <1530613399.12123.12.camel@tuxis.nl> Hi, On Wed, 2018-06-27 at 15:09 +0200, Mark Schouten wrote: > I have a VM that remains in snap-delete state. I'm wondering what the > safest way is to proceed. I think I can do a qm unlock, and click > remove again, but I'm not sure. Please advise. See current qm config > below: Just found the following error in the task list: trying to aquire cfs lock 'storage-REDACTED' ...TASK ERROR: got lock request timeout I see the snapshot still exists, as does the state-image. Would the best solution be to just qm unlock and click delete snapshot again? -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From mark at openvs.co.uk Tue Jul 3 13:18:53 2018 From: mark at openvs.co.uk (Mark Adams) Date: Tue, 3 Jul 2018 12:18:53 +0100 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: <20180703090743.4gszjjg77g4matln@dona.proxmox.com> References: <20180703090743.4gszjjg77g4matln@dona.proxmox.com> Message-ID: Hi Alwin, please see my response below. On 3 July 2018 at 10:07, Alwin Antreich wrote: > On Tue, Jul 03, 2018 at 01:05:51AM +0100, Mark Adams wrote: > > Currently running the newest 5.2-1 version, I had a test cluster which > was > > working fine. I since added more disks, first stopping, then setting out, > > then destroying each osd so I could recreate it all from scratch. > > > > However, when adding a new osd (either via GUI or pveceph CLI) it seems > to > > show a successful create, however does not show in the gui as an osd > under > > the host. > > > > It's like the osd information is being stored by proxmox/ceph somewhere > > else and not being correctly removed and recreated? > > > > I can see that the newly created disk (after it being destroyed) is > > down/out. > > > > Is this by design? is there a way to force the disk back? shouldn't it > show > > in the gui once you create it again? > > > Did you zero your disks after removal? On the first ~200 MB of the disk, > there are leftovers that need to be zeroed before use. After that the > OSD should be added fine. > > I hadn't done this, no - it has helped with the majority of disks thanks and I can now re-add them (I also had to remove the folders from /var/lib/ceph/osd which had other osd names - not sure if the destroy process is supposed to remove them also?) However I have a strange problem on the 2nd host, where it will not make osd.12 ... I get no error output from the gui or pveceph createosd /dev/sda - it just doesn't appear as an osd. It successfully partitions the disk, but doesn't create a folder in /var/lib/ceph/osd/ for the osd mount. I can see there is lock files in /var/lib/ceph/tmp/ ... which I would think should only be there whilst the creation is taking place? journalctl -xe is showing me the problem I think, "command_with_stdin: Error EEXIST: entity osd.12 exists but key does not match" Where is this key? how should I be clearing it out so it will create? Thanks, Mark -- > Cheers, > Alwin > > From a.antreich at proxmox.com Tue Jul 3 17:16:05 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Tue, 3 Jul 2018 17:16:05 +0200 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: References: <20180703090743.4gszjjg77g4matln@dona.proxmox.com> Message-ID: <20180703151605.f2fzneycdmpa5zq3@dona.proxmox.com> On Tue, Jul 03, 2018 at 12:18:53PM +0100, Mark Adams wrote: > Hi Alwin, please see my response below. > > On 3 July 2018 at 10:07, Alwin Antreich wrote: > > > On Tue, Jul 03, 2018 at 01:05:51AM +0100, Mark Adams wrote: > > > Currently running the newest 5.2-1 version, I had a test cluster which > > was > > > working fine. I since added more disks, first stopping, then setting out, > > > then destroying each osd so I could recreate it all from scratch. > > > > > > However, when adding a new osd (either via GUI or pveceph CLI) it seems > > to > > > show a successful create, however does not show in the gui as an osd > > under > > > the host. > > > > > > It's like the osd information is being stored by proxmox/ceph somewhere > > > else and not being correctly removed and recreated? > > > > > > I can see that the newly created disk (after it being destroyed) is > > > down/out. > > > > > > Is this by design? is there a way to force the disk back? shouldn't it > > show > > > in the gui once you create it again? > > > > > Did you zero your disks after removal? On the first ~200 MB of the disk, > > there are leftovers that need to be zeroed before use. After that the > > OSD should be added fine. > > > > > I hadn't done this, no - it has helped with the majority of disks thanks > and I can now re-add them (I also had to remove the folders from > /var/lib/ceph/osd which had other osd names - not sure if the destroy > process is supposed to remove them also?) They will not interfere. ;) > > However I have a strange problem on the 2nd host, where it will not make > osd.12 ... I get no error output from the gui or pveceph createosd /dev/sda > - it just doesn't appear as an osd. > > It successfully partitions the disk, but doesn't create a folder in > /var/lib/ceph/osd/ for the osd mount. I can see there is lock files in > /var/lib/ceph/tmp/ ... which I would think should only be there whilst the > creation is taking place? >From the OSD creation or different lock? Shouldn't make troubles either. > > journalctl -xe is showing me the problem I think, "command_with_stdin: > Error EEXIST: entity osd.12 exists but key does not match" > > Where is this key? how should I be clearing it out so it will create? > 'ceph auth list' will show you all keys in ceph, there will be an orphan osd.12. Removal 'ceph auth del '. -- Cheers, Alwin From aa at ipnerd.net Wed Jul 4 08:21:45 2018 From: aa at ipnerd.net (Alwin Antreich) Date: Wed, 4 Jul 2018 08:21:45 +0200 Subject: [PVE-User] VM remains in snap-delete state In-Reply-To: <1530613399.12123.12.camel@tuxis.nl> References: <1530104951.2625.3.camel@tuxis.nl> <1530613399.12123.12.camel@tuxis.nl> Message-ID: On Tue, Jul 3, 2018, 12:23 Mark Schouten wrote: > Hi, > > On Wed, 2018-06-27 at 15:09 +0200, Mark Schouten wrote: > > I have a VM that remains in snap-delete state. I'm wondering what the > > safest way is to proceed. I think I can do a qm unlock, and click > > remove again, but I'm not sure. Please advise. See current qm config > > below: > > > Just found the following error in the task list: > trying to aquire cfs lock 'storage-REDACTED' ...TASK ERROR: got lock > request timeout > > I see the snapshot still exists, as does the state-image. Would the > best solution be to just qm unlock and click delete snapshot again? > The lock was not acquired, that's why it ran into the timeout. You need to find out what other task is holding the lock and get that one to complete first. -- Cheers, Alwin > From viniciuschagas2 at gmail.com Wed Jul 4 17:11:32 2018 From: viniciuschagas2 at gmail.com (Vinicius Barreto) Date: Wed, 4 Jul 2018 12:11:32 -0300 Subject: [PVE-User] which service is responsible for mounting the NFS storages during the startup of the Proxmox? In-Reply-To: References: Message-ID: Hello Thomas, I was able to understand perfectly. Your information was a great help. Thank you very much. Kind regards Em seg, 2 de jul de 2018 ?s 04:00, Thomas Lamprecht escreveu: > On 7/2/18 4:16 AM, Vinicius Barreto wrote: > > Hello, > > please, would anyone know to tell me which service is responsible for > > mounting the NFS storages during the startup of the Proxmox? > > Note: Added by GUI or Pvesm. > > > > We have a logic that we activate volumes of what we need. > E.g., when a VM is started then we activate it's volume which > in turn mounts the underlying storage, if not already mounted. > > pvestatd checks on all configured enabled storages and also mount's > them to be able to get it's status/usage information. > > > I ask because there is another service that I added to the server and it > > depends on the NFS drives being mounted at startup. > > So I plan to configure my service to start after this service that mounts > > the NFS drives during the proxmox boot. > > > > IMO you should do an after/require on pve-storage.targetd and > pvestatd.service - you then could eventually wait heuristically a bit > to ensure that all storages are mounted. > > Or you use PVE libraries to activate all storages (i.e., mount them) > > it could look like: > > ---- > #!/usr/bin/perl > > use strict; > use warnings; > > use PVE::Storage; > use PVE::Cluster; > > PVE::Cluster::cfs_update(); > > my $storage_cfg = PVE::Storage::config(); > my $all_sids = [ keys %{$storage_cfg->{ids}} ]; > > PVE::Storage::activate_storage_list($storage_cfg, $all_sids); > ---- > > This could be done a pre start hook. > > cheers, > Thomas > > > -- *Marcos Vin?cius Barreto das Chagas* From mark at openvs.co.uk Thu Jul 5 11:26:34 2018 From: mark at openvs.co.uk (Mark Adams) Date: Thu, 5 Jul 2018 10:26:34 +0100 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: <20180703151605.f2fzneycdmpa5zq3@dona.proxmox.com> References: <20180703090743.4gszjjg77g4matln@dona.proxmox.com> <20180703151605.f2fzneycdmpa5zq3@dona.proxmox.com> Message-ID: Hi Anwin; Thanks for that - It's all working now! Just to confirm though, shouldn't the destroy button handle some of these actions? or is it left out on purpose? Regards, Mark On 3 July 2018 at 16:16, Alwin Antreich wrote: > On Tue, Jul 03, 2018 at 12:18:53PM +0100, Mark Adams wrote: > > Hi Alwin, please see my response below. > > > > On 3 July 2018 at 10:07, Alwin Antreich wrote: > > > > > On Tue, Jul 03, 2018 at 01:05:51AM +0100, Mark Adams wrote: > > > > Currently running the newest 5.2-1 version, I had a test cluster > which > > > was > > > > working fine. I since added more disks, first stopping, then setting > out, > > > > then destroying each osd so I could recreate it all from scratch. > > > > > > > > However, when adding a new osd (either via GUI or pveceph CLI) it > seems > > > to > > > > show a successful create, however does not show in the gui as an osd > > > under > > > > the host. > > > > > > > > It's like the osd information is being stored by proxmox/ceph > somewhere > > > > else and not being correctly removed and recreated? > > > > > > > > I can see that the newly created disk (after it being destroyed) is > > > > down/out. > > > > > > > > Is this by design? is there a way to force the disk back? shouldn't > it > > > show > > > > in the gui once you create it again? > > > > > > > Did you zero your disks after removal? On the first ~200 MB of the > disk, > > > there are leftovers that need to be zeroed before use. After that the > > > OSD should be added fine. > > > > > > > > I hadn't done this, no - it has helped with the majority of disks thanks > > and I can now re-add them (I also had to remove the folders from > > /var/lib/ceph/osd which had other osd names - not sure if the destroy > > process is supposed to remove them also?) > They will not interfere. ;) > > > > > However I have a strange problem on the 2nd host, where it will not make > > osd.12 ... I get no error output from the gui or pveceph createosd > /dev/sda > > - it just doesn't appear as an osd. > > > > It successfully partitions the disk, but doesn't create a folder in > > /var/lib/ceph/osd/ for the osd mount. I can see there is lock files in > > /var/lib/ceph/tmp/ ... which I would think should only be there whilst > the > > creation is taking place? > From the OSD creation or different lock? Shouldn't make troubles either. > > > > > journalctl -xe is showing me the problem I think, "command_with_stdin: > > Error EEXIST: entity osd.12 exists but key does not match" > > > > Where is this key? how should I be clearing it out so it will create? > > > 'ceph auth list' will show you all keys in ceph, there will be an orphan > osd.12. Removal 'ceph auth del '. > > From a.antreich at proxmox.com Thu Jul 5 12:04:19 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Thu, 5 Jul 2018 12:04:19 +0200 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: References: <20180703090743.4gszjjg77g4matln@dona.proxmox.com> <20180703151605.f2fzneycdmpa5zq3@dona.proxmox.com> Message-ID: <20180705100419.xzckakgopaqludx5@dona.proxmox.com> On Thu, Jul 05, 2018 at 10:26:34AM +0100, Mark Adams wrote: > Hi Anwin; > > Thanks for that - It's all working now! Just to confirm though, shouldn't > the destroy button handle some of these actions? or is it left out on > purpose? > > Regards, > Mark > I am not sure, what you mean exactly but the destroyosd (CLI/GUI) is doing more then those two steps. -- Cheers, Alwin From mark at openvs.co.uk Thu Jul 5 12:05:52 2018 From: mark at openvs.co.uk (Mark Adams) Date: Thu, 5 Jul 2018 11:05:52 +0100 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: <20180705100419.xzckakgopaqludx5@dona.proxmox.com> References: <20180703090743.4gszjjg77g4matln@dona.proxmox.com> <20180703151605.f2fzneycdmpa5zq3@dona.proxmox.com> <20180705100419.xzckakgopaqludx5@dona.proxmox.com> Message-ID: On 5 July 2018 at 11:04, Alwin Antreich wrote: > On Thu, Jul 05, 2018 at 10:26:34AM +0100, Mark Adams wrote: > > Hi Anwin; > > > > Thanks for that - It's all working now! Just to confirm though, shouldn't > > the destroy button handle some of these actions? or is it left out on > > purpose? > > > > Regards, > > Mark > > > I am not sure, what you mean exactly but the destroyosd (CLI/GUI) is > doing more then those two steps. > > Yes I realise it is, what I'm saying is should it also be doing those steps? From a.antreich at proxmox.com Thu Jul 5 12:53:50 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Thu, 5 Jul 2018 12:53:50 +0200 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: References: <20180703090743.4gszjjg77g4matln@dona.proxmox.com> <20180703151605.f2fzneycdmpa5zq3@dona.proxmox.com> <20180705100419.xzckakgopaqludx5@dona.proxmox.com> Message-ID: <20180705105350.5lymz3ih4wtraw34@dona.proxmox.com> On Thu, Jul 05, 2018 at 11:05:52AM +0100, Mark Adams wrote: > On 5 July 2018 at 11:04, Alwin Antreich wrote: > > > On Thu, Jul 05, 2018 at 10:26:34AM +0100, Mark Adams wrote: > > > Hi Anwin; > > > > > > Thanks for that - It's all working now! Just to confirm though, shouldn't > > > the destroy button handle some of these actions? or is it left out on > > > purpose? > > > > > > Regards, > > > Mark > > > > > I am not sure, what you mean exactly but the destroyosd (CLI/GUI) is > > doing more then those two steps. > > > > > Yes I realise it is, what I'm saying is should it also be doing those > steps? Well, it is doing those too. Just with the failed creation of the OSD not all entries are set and the destroy might fail on some (eg. no service, no mount). The osd create/destroy is up for a change anyway with the move from ceph-disk (deprecated in Mimic) to ceph-volume. Sure room for improvement. ;) -- Cheers, Alwin From yannis.milios at gmail.com Thu Jul 5 13:00:58 2018 From: yannis.milios at gmail.com (Yannis Milios) Date: Thu, 5 Jul 2018 12:00:58 +0100 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: <20180705105350.5lymz3ih4wtraw34@dona.proxmox.com> References: <20180703090743.4gszjjg77g4matln@dona.proxmox.com> <20180703151605.f2fzneycdmpa5zq3@dona.proxmox.com> <20180705100419.xzckakgopaqludx5@dona.proxmox.com> <20180705105350.5lymz3ih4wtraw34@dona.proxmox.com> Message-ID: > Yes I realise it is, what I'm saying is should it also be doing those > steps? Usually you don't have to, but as things often can go wrong you *may* have to do things manually sometimes. GUI is great and saves lots of work, however knowing how to manually solve problems when they arise via the CLI in my opinion is also a must.Especially when you deal with a complicated storage like Ceph .... Y On Thu, Jul 5, 2018 at 11:53 AM Alwin Antreich wrote: > On Thu, Jul 05, 2018 at 11:05:52AM +0100, Mark Adams wrote: > > On 5 July 2018 at 11:04, Alwin Antreich wrote: > > > > > On Thu, Jul 05, 2018 at 10:26:34AM +0100, Mark Adams wrote: > > > > Hi Anwin; > > > > > > > > Thanks for that - It's all working now! Just to confirm though, > shouldn't > > > > the destroy button handle some of these actions? or is it left out on > > > > purpose? > > > > > > > > Regards, > > > > Mark > > > > > > > I am not sure, what you mean exactly but the destroyosd (CLI/GUI) is > > > doing more then those two steps. > > > > > > > > Yes I realise it is, what I'm saying is should it also be doing those > > steps? > Well, it is doing those too. Just with the failed creation of the OSD > not all entries are set and the destroy might fail on some (eg. no > service, no mount). > > The osd create/destroy is up for a change anyway with the move from > ceph-disk (deprecated in Mimic) to ceph-volume. Sure room for > improvement. ;) > > > -- > Cheers, > Alwin > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From mark at openvs.co.uk Thu Jul 5 13:16:00 2018 From: mark at openvs.co.uk (Mark Adams) Date: Thu, 5 Jul 2018 12:16:00 +0100 Subject: [PVE-User] pveceph createosd after destroyed osd In-Reply-To: References: <20180703090743.4gszjjg77g4matln@dona.proxmox.com> <20180703151605.f2fzneycdmpa5zq3@dona.proxmox.com> <20180705100419.xzckakgopaqludx5@dona.proxmox.com> <20180705105350.5lymz3ih4wtraw34@dona.proxmox.com> Message-ID: Hi Alwin, Yannis, Yes I definitely agree with you there Yannis - Good to know how to resolve via cli when things don't seem right. It's also good to know that things can go wrong sometimes via the gui. Alwin - Any improvements are good of course! Regards, Mark On 5 July 2018 at 12:00, Yannis Milios wrote: > > Yes I realise it is, what I'm saying is should it also be doing those > > steps? > > Usually you don't have to, but as things often can go wrong you *may* have > to do things manually sometimes. > GUI is great and saves lots of work, however knowing how to manually solve > problems when they arise via the CLI in my opinion is also a > must.Especially when you deal with a complicated storage like Ceph .... > > Y > On Thu, Jul 5, 2018 at 11:53 AM Alwin Antreich > wrote: > > > On Thu, Jul 05, 2018 at 11:05:52AM +0100, Mark Adams wrote: > > > On 5 July 2018 at 11:04, Alwin Antreich > wrote: > > > > > > > On Thu, Jul 05, 2018 at 10:26:34AM +0100, Mark Adams wrote: > > > > > Hi Anwin; > > > > > > > > > > Thanks for that - It's all working now! Just to confirm though, > > shouldn't > > > > > the destroy button handle some of these actions? or is it left out > on > > > > > purpose? > > > > > > > > > > Regards, > > > > > Mark > > > > > > > > > I am not sure, what you mean exactly but the destroyosd (CLI/GUI) is > > > > doing more then those two steps. > > > > > > > > > > > Yes I realise it is, what I'm saying is should it also be doing those > > > steps? > > Well, it is doing those too. Just with the failed creation of the OSD > > not all entries are set and the destroy might fail on some (eg. no > > service, no mount). > > > > The osd create/destroy is up for a change anyway with the move from > > ceph-disk (deprecated in Mimic) to ceph-volume. Sure room for > > improvement. ;) > > > From arne.hude at arnehude.de Fri Jul 6 21:45:52 2018 From: arne.hude at arnehude.de (Arne Hude) Date: Fri, 6 Jul 2018 21:45:52 +0200 Subject: [PVE-User] Proxmox 5.2.1 LXC can send packets but doesnt receive any packets Message-ID: <7ef447a4-db5d-19ed-8794-fde85dbbee6e@arnehude.de> Hello, i've just set up a Proxmox VE (newest stable release) Setup on a KVM from Netcup. Also i've got a secondary IP for use at this Server. What my plans are: - Hypervisor with one external IP - A LXC-Container with another external IP Future: - Hypervisor on external IP - up to 16 bridges each with another local network for interconnecting the LXC-Containers (Failure-Safeing) - at least one Container from each bridge reachable from external via an ip bought from my hoster This is my?*HOST?*network-config: auto lo iface lo inet loopback iface ens3 inet manual auto vmbr0 iface vmbr0 inet static ?? ? ?? address 94.16.xxx.232 ?? ? ?? netmask 255.255.252.0 ?? ? ?? gateway 94.16.xxx.1 ?? ? ?? bridge_ports ens3 ?? ? ?? bridge_stp off ?? ? ?? bridge_fd 1 ?? ? ?? pointopoint 94.16.xxx.1 ?? ? ?? bridge_hello 2 ?? ? ?? bridge_maxage 12 ?? ? ?? bridge_maxwait 0 ?? ? ?? post-up route add 46.38.xxx.250/32 dev vmbr0 Also i've set sysctl -w net.ipv4.ip_forward=1 sysctl -w net.ipv6.conf.all.forwarding=1 sysctl -w net.ipv6.conf.all.proxy_ndp=1 Firewall and iptables are off or empty. The container network is set up like this: auto lo iface lo inet loopback auto eth0 iface eth0 inet static ?? ? ?? address 46.38.xxx.250 ?? ? ?? netmask 255.255.255.255 ?? ? ?? post-up ip route add 94.16.xxx.232 dev eth0 ?? ? ?? post-up ip route add default via 94.16.xxx.232 dev eth0 ?? ? ?? pre-down ip route del default via 94.16.xx.232 dev eth0 ?? ? ?? pre-down ip route del 94.16.xxx.232 dev eth0 So far so good. Now i get into trouble. if i want to ping my container, the ICMP-Request get lost between my hypervisor and the container. In Conclusion, i can reach the outside world from inside the container but as far as i want a respond its getting lost in the internal network. For setup i used a "tutorial" as far as it is suitable for me (German)::? anysrc:Netcup KVM vServer und LXC Container I'll hope you can help me with this problem im worring with... Hopefully someone of you has a clue., whats wrong about my setup.. From lists at merit.unu.edu Mon Jul 9 15:38:01 2018 From: lists at merit.unu.edu (lists) Date: Mon, 9 Jul 2018 15:38:01 +0200 Subject: [PVE-User] ceph jewel -> luminous upgrade docs Message-ID: Hi, We're reading up on the jewel -> luminous upgrade which we are planning to perform after the holidays. There seems to be a discrepancy between the proxmox and ceph docs: http://docs.ceph.com/docs/master/releases/luminous/#upgrade-from-jewel-or-kraken Step six in the ceph upgrade docs says: "Verify that all RBD client users have sufficient caps to blacklist other client users. RBD client users with only "allow r" monitor caps should to be updated as follows" Does this not apply to a 'regular' proxmox install? We are managing the cluster (and installed it) using the regular proxmox 4 installer. (this gave us hammer at the time, which we have upgraded to jewel) Since there were issues on the ceph mailinglist where QEMU corrupted VM images, I thought I'd ask here. Reports: [1] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-November/022278.html [2] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-November/022694.html [3] http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-May/026496.html [4] https://www.spinics.net/lists/ceph-users/msg45665.html Is this step 6 relevant to us, proxmox users? MJ From mark at tuxis.nl Mon Jul 9 15:57:01 2018 From: mark at tuxis.nl (Mark Schouten) Date: Mon, 09 Jul 2018 15:57:01 +0200 Subject: [PVE-User] ceph jewel -> luminous upgrade docs In-Reply-To: References: Message-ID: On Mon, 2018-07-09 at 15:38 +0200, lists wrote: > "Verify that all RBD client users have sufficient caps to blacklist > other client users. RBD client users with only "allow r" monitor > caps > should to be updated as follows" > > Does this not apply to a 'regular' proxmox install? We are managing > the > cluster (and installed it) using the regular proxmox 4 installer. > (this > gave us hammer at the time, which we have upgraded to jewel) A default Proxmox setup runs as admin, doesn't it? With all caps? -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From lists at merit.unu.edu Mon Jul 9 16:32:07 2018 From: lists at merit.unu.edu (lists) Date: Mon, 9 Jul 2018 16:32:07 +0200 Subject: [PVE-User] ceph jewel -> luminous upgrade docs In-Reply-To: References: Message-ID: Hi Mark, On 9-7-2018 15:57, Mark Schouten wrote: > A default Proxmox setup runs as admin, doesn't it? With all caps? I think the answer to #1 is yes, but I'm not sure how to tell about the second caps question. Perhaps this tell you something: > root at pve:~# ceph auth list | grep -v key > installed auth entries: > > osd.0 > caps: [mon] allow profile osd > caps: [osd] allow * > osd.1 > caps: [mon] allow profile osd > caps: [osd] allow * >skipping >many >more >OSD >lines > osd.17 > caps: [mon] allow profile osd > caps: [osd] allow * > osd.18 > caps: [mon] allow profile osd > caps: [osd] allow * > osd.19 > caps: [mon] allow profile osd > caps: [osd] allow * > client.admin > auid: 0 > caps: [mds] allow > caps: [mon] allow * > caps: [osd] allow * > client.bootstrap-mds > caps: [mon] allow profile bootstrap-mds > client.bootstrap-osd > caps: [mon] allow profile bootstrap-osd > client.bootstrap-rgw > caps: [mon] allow profile bootstrap-rgw Reason I ask here: there is no "blacklist" in any of the output above... MJ From mark at tuxis.nl Mon Jul 9 16:50:37 2018 From: mark at tuxis.nl (Mark Schouten) Date: Mon, 09 Jul 2018 16:50:37 +0200 Subject: [PVE-User] ceph jewel -> luminous upgrade docs In-Reply-To: References: Message-ID: <6d405133937c0de11aa4d2ded6dec87f7f051665.camel@tuxis.nl> On Mon, 2018-07-09 at 16:32 +0200, lists wrote: > Hi Mark, > > On 9-7-2018 15:57, Mark Schouten wrote: > > A default Proxmox setup runs as admin, doesn't it? With all caps? > > I think the answer to #1 is yes, but I'm not sure how to tell about > the > second caps question. Perhaps this tell you something: In /etc/pve/priv/ceph/ is the file you (and your VM's) are using as a key-file for Ceph. > > > > caps: [osd] allow * This (*) allows all capabilities. -- Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ Mark Schouten | Tuxis Internet Engineering KvK: 61527076 | http://www.tuxis.nl/ T: 0318 200208 | info at tuxis.nl From lists at merit.unu.edu Mon Jul 9 17:34:38 2018 From: lists at merit.unu.edu (lists) Date: Mon, 9 Jul 2018 17:34:38 +0200 Subject: [PVE-User] ceph jewel -> luminous upgrade docs In-Reply-To: <6d405133937c0de11aa4d2ded6dec87f7f051665.camel@tuxis.nl> References: <6d405133937c0de11aa4d2ded6dec87f7f051665.camel@tuxis.nl> Message-ID: Hi Mark, On 9-7-2018 16:50, Mark Schouten wrote: >>> >>> caps: [osd] allow * > > This (*) allows all capabilities. > Super, so no need to worry about that :-) Thanks! MJ From lyt_yudi at icloud.com Tue Jul 10 03:41:38 2018 From: lyt_yudi at icloud.com (lyt_yudi) Date: Tue, 10 Jul 2018 09:41:38 +0800 Subject: [PVE-User] About the FRR integrated into the PVE? References: Message-ID: <30E114F1-7B32-4FD0-98CF-6382C334AD9F@icloud.com> Hi, will it be better integrated into the PVE network architecture? About https://frrouting.org/ https://github.com/FRRouting/frr _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From Jan.Dreyer at arvato.com Tue Jul 10 09:15:00 2018 From: Jan.Dreyer at arvato.com (Dreyer, Jan, SCM-IT) Date: Tue, 10 Jul 2018 07:15:00 +0000 Subject: [PVE-User] zfs: cannot import rpool after reboot Message-ID: Hi, after upgrading my Debian (and proxmox) to stretch (PVE 5.2), I got into trouble when rebooting my servers. It?s similar to https://forum.proxmox.com/threads/zfs-cannot-import-rpool-after-reboot.37611/page-2 beside that a manual import does _not_ work. When booting with the old kernel (4.4.128-1-pve) the machine comes up, but has no access to it?s (zfs-based) swap device. I already tried to boot from the most recent 4.10, which seems to hang in a CPU-stalled loop for > 5 minutes. ?zdb" shows me the configured pools, so basically they are available. As a workaround I set the 4.4-kernel as default for booting, but of course I want to upgrade. Does anyone has a hint on how to do it? Regards ? Jan Dreyer IT Administrator ? Operations ? A-SCM-IT IOX Expert This email may contain confidential and/or privileged information. If you are not the intended recipient (or have received this email in error) please notify the sender immediately and destroy this email. Any unauthorized copying, disclosure or distribution of the information in this email is strictly forbidden. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From aderumier at odiso.com Tue Jul 10 09:58:39 2018 From: aderumier at odiso.com (Alexandre DERUMIER) Date: Tue, 10 Jul 2018 09:58:39 +0200 (CEST) Subject: [PVE-User] About the FRR integrated into the PVE? In-Reply-To: <30E114F1-7B32-4FD0-98CF-6382C334AD9F@icloud.com> References: <30E114F1-7B32-4FD0-98CF-6382C334AD9F@icloud.com> Message-ID: <1428369101.109208.1531209519961.JavaMail.zimbra@oxygem.tv> Hi, I have sent a package for proxmox last month, I need to rebase it on frr 5.0.1. (I need it for vxlan bgp evpn) ----- Mail original ----- De: "lyt_yudi" ?: "proxmoxve" Envoy?: Mardi 10 Juillet 2018 03:41:38 Objet: [PVE-User] About the FRR integrated into the PVE? Hi, will it be better integrated into the PVE network architecture? About https://frrouting.org/ https://github.com/FRRouting/frr _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From lyt_yudi at icloud.com Wed Jul 11 09:58:26 2018 From: lyt_yudi at icloud.com (lyt_yudi) Date: Wed, 11 Jul 2018 15:58:26 +0800 Subject: [PVE-User] About the FRR integrated into the PVE? In-Reply-To: <1428369101.109208.1531209519961.JavaMail.zimbra@oxygem.tv> References: <30E114F1-7B32-4FD0-98CF-6382C334AD9F@icloud.com> <1428369101.109208.1531209519961.JavaMail.zimbra@oxygem.tv> Message-ID: <7C5B4CE0-63E0-4F64-8137-6E97223FA677@icloud.com> > ? 2018?7?10????3:58?Alexandre DERUMIER ??? > > > I have sent a package for proxmox last month, > > I need to rebase it on frr 5.0.1. > > (I need it for vxlan bgp evpn) It's too fast you are. Great! From aderumier at odiso.com Wed Jul 11 15:17:38 2018 From: aderumier at odiso.com (Alexandre DERUMIER) Date: Wed, 11 Jul 2018 15:17:38 +0200 (CEST) Subject: [PVE-User] About the FRR integrated into the PVE? In-Reply-To: <7C5B4CE0-63E0-4F64-8137-6E97223FA677@icloud.com> References: <30E114F1-7B32-4FD0-98CF-6382C334AD9F@icloud.com> <1428369101.109208.1531209519961.JavaMail.zimbra@oxygem.tv> <7C5B4CE0-63E0-4F64-8137-6E97223FA677@icloud.com> Message-ID: <1885894804.289481.1531315058163.JavaMail.zimbra@oxygem.tv> Just curious, What do you want to do with frr ? (vxlan - bgp evpn ?) ----- Mail original ----- De: "lyt_yudi" ?: "proxmoxve" Envoy?: Mercredi 11 Juillet 2018 09:58:26 Objet: Re: [PVE-User] About the FRR integrated into the PVE? > ? 2018?7?10????3:58?Alexandre DERUMIER ??? > > > I have sent a package for proxmox last month, > > I need to rebase it on frr 5.0.1. > > (I need it for vxlan bgp evpn) It's too fast you are. Great! _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From basarakrystian at gmail.com Wed Jul 11 22:45:11 2018 From: basarakrystian at gmail.com (Krystian Basara) Date: Wed, 11 Jul 2018 22:45:11 +0200 Subject: [PVE-User] PROXMOX API RESTORE PROBLEM Message-ID: Hello. I've got problem with restoring VM. ( all other api functions work correct for me ) Example of what i'm using : POST methods: /nodes/NODE/qemu?vmid=102&storage=local&volume=local:backup/vzdump-kvm-102-2018_07_10-09_45_29.tar /nodes/NODE/qemu?vmid=101&storage=local&ostemplate=local:backup/vzdump-lxc-101-2018_07_10-10_24_51.tar End point is /nodes/NODE/qemu or lxc, right? And vars are volume or ostemplate if restoring? Am I doing it in wrong way? Thanks for help. From lyt_yudi at icloud.com Thu Jul 12 03:20:47 2018 From: lyt_yudi at icloud.com (lyt_yudi) Date: Thu, 12 Jul 2018 09:20:47 +0800 Subject: [PVE-User] About the FRR integrated into the PVE? In-Reply-To: <1885894804.289481.1531315058163.JavaMail.zimbra@oxygem.tv> References: <30E114F1-7B32-4FD0-98CF-6382C334AD9F@icloud.com> <1428369101.109208.1531209519961.JavaMail.zimbra@oxygem.tv> <7C5B4CE0-63E0-4F64-8137-6E97223FA677@icloud.com> <1885894804.289481.1531315058163.JavaMail.zimbra@oxygem.tv> Message-ID: <2E2474DD-696F-4B64-BD96-70DBD9D0449C@icloud.com> > ? 2018?7?11????9:17?Alexandre DERUMIER ??? > > What do you want to do with frr ? (vxlan - bgp evpn ?) yes, this is more urgent! From lyt_yudi at icloud.com Thu Jul 12 04:03:46 2018 From: lyt_yudi at icloud.com (lyt_yudi) Date: Thu, 12 Jul 2018 10:03:46 +0800 Subject: [PVE-User] About the FRR integrated into the PVE? In-Reply-To: <2E2474DD-696F-4B64-BD96-70DBD9D0449C@icloud.com> References: <30E114F1-7B32-4FD0-98CF-6382C334AD9F@icloud.com> <1428369101.109208.1531209519961.JavaMail.zimbra@oxygem.tv> <7C5B4CE0-63E0-4F64-8137-6E97223FA677@icloud.com> <1885894804.289481.1531315058163.JavaMail.zimbra@oxygem.tv> <2E2474DD-696F-4B64-BD96-70DBD9D0449C@icloud.com> Message-ID: <4646BF4D-A1FF-4236-B82B-70FF05F89CFD@icloud.com> > ? 2018?7?12????9:20?lyt_yudi ??? > >> ? 2018?7?11????9:17?Alexandre DERUMIER > ??? >> >> What do you want to do with frr ? (vxlan - bgp evpn ?) > > yes, this is more urgent! Add ospf/ospf6 ? In small and medium networks, the OSPF will be more advantageous! By the way, BGP also needs to establish neighbors first, while establishing neighbors is usually OSPF. It's just a suggestion. I really don't have the experience of large-scale network environment. Thanks! From f.gruenbichler at proxmox.com Thu Jul 12 10:28:50 2018 From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?Q?Gr=FCnbichler?=) Date: Thu, 12 Jul 2018 10:28:50 +0200 Subject: [PVE-User] PROXMOX API RESTORE PROBLEM In-Reply-To: References: Message-ID: <20180712082850.6hkuihnl5q5d7jfs@nora.maurer-it.com> On Wed, Jul 11, 2018 at 10:45:11PM +0200, Krystian Basara wrote: > Hello. > I've got problem with restoring VM. > ( all other api functions work correct for me ) > Example of what i'm using : > POST methods: > > /nodes/NODE/qemu?vmid=102&storage=local&volume=local:backup/vzdump-kvm-102-2018_07_10-09_45_29.tar > > /nodes/NODE/qemu?vmid=101&storage=local&ostemplate=local:backup/vzdump-lxc-101-2018_07_10-10_24_51.tar > > > End point is /nodes/NODE/qemu or lxc, right? > And vars are volume or ostemplate if restoring? "archive" is the parameter for the backup archive, volume is for VM disks, ostemplate is for creating a container from a template. check the API docs (shipped as part of pve-docs and linked at the bottom of the documentation). From aderumier at odiso.com Thu Jul 12 12:59:19 2018 From: aderumier at odiso.com (Alexandre DERUMIER) Date: Thu, 12 Jul 2018 12:59:19 +0200 (CEST) Subject: [PVE-User] About the FRR integrated into the PVE? In-Reply-To: <2E2474DD-696F-4B64-BD96-70DBD9D0449C@icloud.com> References: <30E114F1-7B32-4FD0-98CF-6382C334AD9F@icloud.com> <1428369101.109208.1531209519961.JavaMail.zimbra@oxygem.tv> <7C5B4CE0-63E0-4F64-8137-6E97223FA677@icloud.com> <1885894804.289481.1531315058163.JavaMail.zimbra@oxygem.tv> <2E2474DD-696F-4B64-BD96-70DBD9D0449C@icloud.com> Message-ID: <437417501.396535.1531393159883.JavaMail.zimbra@oxygem.tv> > What do you want to do with frr ? (vxlan - bgp evpn ?) >>yes, this is more urgent! Great . with me, we are already 3 differents users who's need it :) I'll send documentation soon, but it's already working in my test lab. I'm currently trying to polish promox integration, make network config reloading possible,... If you have time to test next month, it could be great :) >>Add ospf/ospf6 ? >> >>In small and medium networks, the OSPF will be more advantageous! >>By the way, BGP also needs to establish neighbors first, while establishing neighbors is usually OSPF. >> >>It's just a suggestion. I really don't have the experience of large-scale network environment. well, you can do bgp,ospf with frr. I don't known yet if we need to manage frr.conf from proxmox gui or not. As they are so many setup possible, depending of infrastructure. The advantage of bgp-evpn, is that vm live migration is working, because control plane is done through bgp, learning mac address from vm, and dynamicaly create routes. bgp/ospf, is more for the routing of the proxmox hosts underlay. (if you have a layer3 network) ----- Mail original ----- De: "lyt_yudi" ?: "proxmoxve" Envoy?: Jeudi 12 Juillet 2018 03:20:47 Objet: Re: [PVE-User] About the FRR integrated into the PVE? > ? 2018?7?11????9:17?Alexandre DERUMIER ??? > > What do you want to do with frr ? (vxlan - bgp evpn ?) yes, this is more urgent! _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From lyt_yudi at icloud.com Thu Jul 12 14:35:53 2018 From: lyt_yudi at icloud.com (lyt_yudi) Date: Thu, 12 Jul 2018 20:35:53 +0800 Subject: [PVE-User] About the FRR integrated into the PVE? In-Reply-To: <437417501.396535.1531393159883.JavaMail.zimbra@oxygem.tv> References: <30E114F1-7B32-4FD0-98CF-6382C334AD9F@icloud.com> <1428369101.109208.1531209519961.JavaMail.zimbra@oxygem.tv> <7C5B4CE0-63E0-4F64-8137-6E97223FA677@icloud.com> <1885894804.289481.1531315058163.JavaMail.zimbra@oxygem.tv> <2E2474DD-696F-4B64-BD96-70DBD9D0449C@icloud.com> <437417501.396535.1531393159883.JavaMail.zimbra@oxygem.tv> Message-ID: > ? 2018?7?12????6:59?Alexandre DERUMIER ??? > > Great . with me, we are already 3 differents users who's need it :) > I'll send documentation soon, but it's already working in my test lab. > > I'm currently trying to polish promox integration, make network config reloading possible,... > > If you have time to test next month, it could be great :) Yes, very happy, if I can to test it! > well, you can do bgp,ospf with frr. > I don't known yet if we need to manage frr.conf from proxmox gui or not. As they are so many setup possible, depending of infrastructure. > > > The advantage of bgp-evpn, is that vm live migration is working, because control plane is done through bgp, learning mac address from vm, > and dynamicaly create routes. > > > bgp/ospf, is more for the routing of the proxmox hosts underlay. (if you have a layer3 network) Yes, the layer3 network it?s possible! If it can be managed by the PVE UI, it will be more convenient and more network functions. Thanks! From michiel at nexigon.net Wed Jul 18 14:27:21 2018 From: michiel at nexigon.net (Michiel Janssens) Date: Wed, 18 Jul 2018 14:27:21 +0200 Subject: [PVE-User] Virt-viewer issue, advice needed for filing bug Message-ID: Hi, I'm using openSUSE Tumbleweed and recently got an update for virt- viewer. https://build.opensuse.org/package/view_file/openSUSE:Factory/virt-view er/virt-viewer.changes?expand=1 After this change I can no longer use the spice session from proxmox webgui. The temporary connection file from the webgui is downloaded and started with remote viewer. After about 10 seconds I get: "cannot connect to grafical server" "time-out at socket I/O Locally on the laptop at which I initiate the connection to my proxmox host everything works just fine. So connecting with remote-viewer via spice://localhost:5900 to a local kvm spice session works ok. I also tried to get some debugging info by starting remote-viewer with this temporary file (have to be fast as the ticket expires), but i get no useful info. I'm willing to file a bug, but I i'm in doubt if this has to do with the spice implementation inside Proxmox or that its just a bug inside Tumbleweed. >From the link above, the linked bug mentions a change: "With spice-gtk 0.35 the spice-controller library has been removed." please advise to which bug tracker this should be filed. Thanks Michiel Janssens From gaio at sv.lnf.it Thu Jul 19 18:34:59 2018 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Thu, 19 Jul 2018 18:34:59 +0200 Subject: [PVE-User] ZFS, grub cannot load second stage... Message-ID: <20180719163459.GA15982@sv.lnf.it> I've installed a little server vith PVE 5.2, using ZFS, as just done before. The only differences is that server have a raid-1 zpool with 4 disks (2 1TB and 2 4TB disks) all in a single 5TB rpool. The server have a plain AHCI controller, no RAID hardware. After configuring the server, configuring some VM/LXC inside, probably a user inadvertitely powered off it. The server never gone back, and print in console: error: no such device: aeXXXXX. error: unknown filesystem. Entering rescue mode... grub rescue> in rescue mode, i can 'ls' all the disks and seems OK (number of disks and partition match), but i cannot 'ls' inside (filesystem not found or similar error). I've googled and found: https://forum.proxmox.com/threads/crashes-with-zfs-root-and-stuck-on-grub-rescue-prompt.34172/ and i was able to run the install CD (rescue mode does nothing, say there's no rpool) and with: zpool import -f -d /dev/ -R /mnt rpool i've mounted the pool, chmod-ed on it and tried to reinstall grub and rebuild grub confing and initrd. Data seems OK. But after a reboot, throw the same error. How can fix that? The hex code in 'error: no such device: aeXXXXX.' is relative to the 'ROOT' zfs volume? How can i determine if match correctly? Please, help me. Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From gaio at sv.lnf.it Thu Jul 19 18:47:57 2018 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Thu, 19 Jul 2018 18:47:57 +0200 Subject: [PVE-User] ZFS, grub cannot load second stage... In-Reply-To: <20180719163459.GA15982@sv.lnf.it> References: <20180719163459.GA15982@sv.lnf.it> Message-ID: <20180719164757.GC15982@sv.lnf.it> Mandi! Marco Gaiarin In chel di` si favelave... > zpool import -f -d /dev/ -R /mnt rpool Sorry, a little notes. I was forced to use -d /dev because if i omit it, zpool found only two disks out of 4 (does not found the two disks added after the first install). This seems strange. As just stated, the server was rebooted many time, so, anyway, it worked. Again, thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From gaio at sv.lnf.it Fri Jul 20 17:56:39 2018 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Fri, 20 Jul 2018 17:56:39 +0200 Subject: [PVE-User] ZFS, grub cannot load second stage... In-Reply-To: <20180719164757.GC15982@sv.lnf.it> References: <20180719163459.GA15982@sv.lnf.it> <20180719164757.GC15982@sv.lnf.it> Message-ID: <20180720155639.GS3540@sv.lnf.it> > Sorry, a little notes. Again, two little notes. Finally i've, in some way, extracted the VMs and containers form the 'dead' system and reinstalled, but this time using TWO zfs pools. Anyway... a) i've noted that, even if i've used many tools, the 4TB set of disk remains with label of provious setup (linux software raid); i was forced to use dd and write zeroes to some megabytes of the disks, not the first 512 bytes only. I suppose that this is the 'mother of all my troubles', but still i don't understand why the system worked perfectly, until i ''filled'' disks with data. b) i've discovered that systemd does not work in chroot environment; so i was not able to run proxmox and only after fiddling a bit with services i was able to mount /etc/pve filesystem and backup containers with vzdump. Still VMs backup does not work, so i've done a disk dump with dd and recreated a identical machine on the reinstalled system. Probably a page on the wiki on how to work on such 'emergency mode' would be useful. Sorry, i was on time shortage and so i've not appointed the procedure i used, but roughly: a) boot the proxmox 5.2 cd; do Ctrl-C on terminal 1 to interrupt the installer and have a shell (please, add some shells on the installer cd!). b) import the 'broken' pool and mount it somewhere; mount also /proc, /sys and /dev and chroot within. c) start manually pmxcfs to mount /etc/pve d) backup containers with 'vzdump' e) backup VMs disks with dd ('zfs list' for the list of zfs devices) f) backup /etc for reference and for VMs parameters (eg, MAC addresses) Clearly, to test all that stuff there's no need to have a 'broken' system, simply boot the installer cd on a working one. Thanks. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From jmr.richardson at gmail.com Tue Jul 24 19:36:45 2018 From: jmr.richardson at gmail.com (JR Richardson) Date: Tue, 24 Jul 2018 12:36:45 -0500 Subject: [PVE-User] Datacenter Summary Status Webpage for View-Olny User Message-ID: Hi All, Which permission role grants view access to the datacenter summary page? I configured a user for view-only with PVEAuditor but only see Nodes and VMs. The idea is to export the datacenter summary page out to our NOC so they can see red/green light status on a monitor. Thanks. JR -- JR Richardson Engineering for the Masses Chasing the Azeotrope From mark at openvs.co.uk Wed Jul 25 02:19:59 2018 From: mark at openvs.co.uk (Mark Adams) Date: Wed, 25 Jul 2018 01:19:59 +0100 Subject: [PVE-User] Poor CEPH performance? or normal? Message-ID: Hi All, I have a proxmox 5.1 + ceph cluster of 3 nodes, each with 12 x WD 10TB GOLD drives. Network is 10Gbps on X550-T2, separate network for the ceph cluster. I have 1 VM currently running on this cluster, which is debian stretch with a zpool on it. I'm zfs sending in to it, but only getting around ~15MiB/s write speed. does this sound right? it seems very slow to me. Not only that, but when this zfs send is running - I can not do any parallel sends to any other zfs datasets inside of the same VM. They just seem to hang, then eventually say "dataset is busy". Any pointers or insights greatly appreciated! Thanks From aa at ipnerd.net Wed Jul 25 08:10:59 2018 From: aa at ipnerd.net (Alwin Antreich) Date: Wed, 25 Jul 2018 08:10:59 +0200 Subject: [PVE-User] Poor CEPH performance? or normal? In-Reply-To: References: Message-ID: Hi, On Wed, Jul 25, 2018, 02:20 Mark Adams wrote: > Hi All, > > I have a proxmox 5.1 + ceph cluster of 3 nodes, each with 12 x WD 10TB GOLD > drives. Network is 10Gbps on X550-T2, separate network for the ceph > cluster. > Do a rados bench for testing the cluster performance, spinners are not fast. > I have 1 VM currently running on this cluster, which is debian stretch with > a zpool on it. I'm zfs sending in to it, but only getting around ~15MiB/s > write speed. does this sound right? it seems very slow to me. > Never ever use a CoW filesystem on top of another CoW system. This doubles the writes that need to be made. > Not only that, but when this zfs send is running - I can not do any > parallel sends to any other zfs datasets inside of the same VM. They just > seem to hang, then eventually say "dataset is busy". > Ceph already gives you the possibility of snapshots. You can let PVE do this through CLI or GUI. -- Cheers, Alwin From t.lamprecht at proxmox.com Wed Jul 25 09:18:53 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Wed, 25 Jul 2018 09:18:53 +0200 Subject: [PVE-User] Datacenter Summary Status Webpage for View-Olny User In-Reply-To: References: Message-ID: Hi, Am 07/24/2018 um 07:36 PM schrieb JR Richardson: > Hi All, > > Which permission role grants view access to the datacenter summary > page? I configured a user for view-only with PVEAuditor but only see > Nodes and VMs. The idea is to export the datacenter summary page out > to our NOC so they can see red/green light status on a monitor. > Hmm, cannot reproduce this here. When I add a new user an give him the PVEAuditor permission on the root path / (with propagate on), and then login in a new browser session I can see the DC/Summary page with all it's elements just fine. Are you sure you added the correct user (or it's group) permissions? Or are you using an older version of PVE? I just tested on 5.2. cheers, Thomas From mark at openvs.co.uk Wed Jul 25 11:20:57 2018 From: mark at openvs.co.uk (Mark Adams) Date: Wed, 25 Jul 2018 10:20:57 +0100 Subject: [PVE-User] Poor CEPH performance? or normal? In-Reply-To: References: Message-ID: Hi Alwin, On 25 July 2018 at 07:10, Alwin Antreich wrote: > Hi, > > On Wed, Jul 25, 2018, 02:20 Mark Adams wrote: > > > Hi All, > > > > I have a proxmox 5.1 + ceph cluster of 3 nodes, each with 12 x WD 10TB > GOLD > > drives. Network is 10Gbps on X550-T2, separate network for the ceph > > cluster. > > > Do a rados bench for testing the cluster performance, spinners are not > fast. > > This was a typo - I'm actually on 5.2-1. I'll give rados bench a try to see what it comes back with. > > I have 1 VM currently running on this cluster, which is debian stretch > with > > a zpool on it. I'm zfs sending in to it, but only getting around ~15MiB/s > > write speed. does this sound right? it seems very slow to me. > > > Never ever use a CoW filesystem on top of another CoW system. This doubles > the writes that need to be made. > > > Not only that, but when this zfs send is running - I can not do any > > parallel sends to any other zfs datasets inside of the same VM. They just > > seem to hang, then eventually say "dataset is busy". > > > Ceph already gives you the possibility of snapshots. You can let PVE do > this through CLI or GUI. > The problem with this is the required features.. I need an HA cluster, and zfs doesn't support this - so ceph is ideal, however I also need "restore previous versions" usable inside a file server VM in samba, which ceph snapshots at the VM layer is no use for.... Unless there is some other smart way of doing this I don't know about! I guess my main question is, is there any other config hints to speed this up whether ceph or in ZFS inside the VM, and is the blocking of other IO normal with ceph when "max" write speed is being reached? That bit doesn't seem right to me. > -- > Cheers, > Alwin > Regards, Mark From jmr.richardson at gmail.com Wed Jul 25 13:11:14 2018 From: jmr.richardson at gmail.com (JR Richardson) Date: Wed, 25 Jul 2018 06:11:14 -0500 Subject: [PVE-User] Datacenter Summary Status Webpage for View-Olny User In-Reply-To: References: Message-ID: <000f01d42408$33cf29e0$9b6d7da0$@gmail.com> I'm using 5.2, gave user path / and see the summary page OK now. Thanks. JR JR Richardson Engineering for the Masses Chasing the Azeotrope -----Original Message----- From: Thomas Lamprecht [mailto:t.lamprecht at proxmox.com] Sent: Wednesday, July 25, 2018 2:19 AM To: PVE User List; JR Richardson Subject: Re: [PVE-User] Datacenter Summary Status Webpage for View-Olny User Hi, Am 07/24/2018 um 07:36 PM schrieb JR Richardson: > Hi All, > > Which permission role grants view access to the datacenter summary > page? I configured a user for view-only with PVEAuditor but only see > Nodes and VMs. The idea is to export the datacenter summary page out > to our NOC so they can see red/green light status on a monitor. > Hmm, cannot reproduce this here. When I add a new user an give him the PVEAuditor permission on the root path / (with propagate on), and then login in a new browser session I can see the DC/Summary page with all it's elements just fine. Are you sure you added the correct user (or it's group) permissions? Or are you using an older version of PVE? I just tested on 5.2. cheers, Thomas From ronny+pve-user at aasen.cx Wed Jul 25 20:20:34 2018 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Wed, 25 Jul 2018 20:20:34 +0200 Subject: [PVE-User] Poor CEPH performance? or normal? In-Reply-To: References: Message-ID: On 25. juli 2018 02:19, Mark Adams wrote: > Hi All, > > I have a proxmox 5.1 + ceph cluster of 3 nodes, each with 12 x WD 10TB GOLD > drives. Network is 10Gbps on X550-T2, separate network for the ceph cluster. > > I have 1 VM currently running on this cluster, which is debian stretch with > a zpool on it. I'm zfs sending in to it, but only getting around ~15MiB/s > write speed. does this sound right? it seems very slow to me. > > Not only that, but when this zfs send is running - I can not do any > parallel sends to any other zfs datasets inside of the same VM. They just > seem to hang, then eventually say "dataset is busy". > > Any pointers or insights greatly appreciated! Greetings alwin gave you some good advice about filesystems and vm's, i wanted to say a little about ceph. with 3 nodes, and the default and reccomended size=3 pools, you can not tolerate any node failures. IOW, if you loose a node, or need to do lengthy maintainance on it, you are running degraded. I allways have a 4th "failure domain" node. so my cluster can selfheal (one of cephs killer features) from a node failure. your cluster should be 3+[how-many-node-failures-i-want-to-be-able-to-survive-and-still-operate-sanely] spinning osd's with bluestore benefit greatly from ssd DB/WAL's if your osd's have ondisk DB/WAL you can gain a lot of performance by having the DB/WAL on a SSD or better. ceph gains performance with scale(number of osd nodes) . so while ceph's aggeregate performance is awesome, an individual single thread will not be amazing. A given set of data will exist on all 3 nodes, and you will hit 100% of nodes with any write. so by using ceph with 3 nodes you give ceph the worst case for performance. eg with 4 nodes a write would hit 75%, with 6 nodes it would hit 50% of the cluster. you see where this is going... But a single write will only hit one disk in 3 nodes, and will not have a better performance then the disk it hits. you can cheat more performance with rbd caching. and it is important for performance to get a higher queue depth. afaik zfs uses a queue depth of 1, for ceph the worst possible. you may have some success by buffering on one or both ends of the transfer [1] if the vm have a RBD disk, you may (or may not) benefit from rbd fancy striping[2], since operations can hit more osd's in parallel. good luck Ronny Aasen [1] https://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/ [2] http://docs.ceph.com/docs/master/architecture/#data-striping From ronny+pve-user at aasen.cx Wed Jul 25 21:11:32 2018 From: ronny+pve-user at aasen.cx (Ronny Aasen) Date: Wed, 25 Jul 2018 21:11:32 +0200 Subject: [PVE-User] Poor CEPH performance? or normal? In-Reply-To: References: Message-ID: On 25.07.2018 11:20, Mark Adams wrote: > Hi Alwin, > > On 25 July 2018 at 07:10, Alwin Antreich wrote: > >> Hi, >> >> On Wed, Jul 25, 2018, 02:20 Mark Adams wrote: >> >>> Hi All, >>> >>> I have a proxmox 5.1 + ceph cluster of 3 nodes, each with 12 x WD 10TB >> GOLD >>> drives. Network is 10Gbps on X550-T2, separate network for the ceph >>> cluster. >>> >> Do a rados bench for testing the cluster performance, spinners are not >> fast. >> >> > This was a typo - I'm actually on 5.2-1. I'll give rados bench a try to see > what it comes back with. > > >>> I have 1 VM currently running on this cluster, which is debian stretch >> with >>> a zpool on it. I'm zfs sending in to it, but only getting around ~15MiB/s >>> write speed. does this sound right? it seems very slow to me. >>> >> Never ever use a CoW filesystem on top of another CoW system. This doubles >> the writes that need to be made. > >>> Not only that, but when this zfs send is running - I can not do any >>> parallel sends to any other zfs datasets inside of the same VM. They just >>> seem to hang, then eventually say "dataset is busy". >>> >> Ceph already gives you the possibility of snapshots. You can let PVE do >> this through CLI or GUI. >> > The problem with this is the required features.. I need an HA cluster, and > zfs doesn't support this - so ceph is ideal, however I also need "restore > previous versions" usable inside a file server VM in samba, which ceph > snapshots at the VM layer is no use for.... Unless there is some other > smart way of doing this I don't know about! > > I guess my main question is, is there any other config hints to speed this > up whether ceph or in ZFS inside the VM, and is the blocking of other IO > normal with ceph when "max" write speed is being reached? That bit doesn't > seem right to me. > you can expose rbd snapshots in samba using this script. https://github.com/ksperis/autosnap-rbd-shadow-copy -- but having ceph-> rbd -> filesystem -> vm mounting filesystem -> samba export? is a long and overly complex chain. if you mount and export a cephfs filesystem you can remove the need for a filesystem on a rbd ceph-> cephfs -> vm with cephfs mounted -> samba export You can even take it a step further, by using the vfs_ceph samba module you can translate samba calls to cephfs calls directly https://fosdem.org/2018/schedule/event/cephfs_gateways/attachments/slides/2636/export/events/attachments/cephfs_gateways/slides/2636/cephfs_samba_and_nfs.pdf good luck Ronny Aasen From brentgclarklist at gmail.com Thu Jul 26 11:05:32 2018 From: brentgclarklist at gmail.com (Brent Clark) Date: Thu, 26 Jul 2018 11:05:32 +0200 Subject: [PVE-User] Proxmox disable TLS 1 Message-ID: <1606b411-3907-0de1-c73a-ba171462cc0f@gmail.com> Good day Guys I did a sslscan on my proxmox host, and I got the following: snippet: Preferred TLSv1.0? 256 bits? ECDHE-RSA-AES256-SHA????????? Curve P-256 DHE 256 Accepted? TLSv1.0? 256 bits? DHE-RSA-AES256-SHA??????????? DHE 2048 bits Accepted? TLSv1.0? 256 bits? DHE-RSA-CAMELLIA256-SHA?????? DHE 2048 bits Accepted? TLSv1.0? 256 bits? AES256-SHA Accepted? TLSv1.0? 256 bits? CAMELLIA256-SHA Accepted? TLSv1.0? 128 bits? ECDHE-RSA-AES128-SHA????????? Curve P-256 DHE 256 Accepted? TLSv1.0? 128 bits? DHE-RSA-AES128-SHA??????????? DHE 2048 bits Accepted? TLSv1.0? 128 bits? DHE-RSA-SEED-SHA????????????? DHE 2048 bits Accepted? TLSv1.0? 128 bits? DHE-RSA-CAMELLIA128-SHA?????? DHE 2048 bits Accepted? TLSv1.0? 128 bits? AES128-SHA Accepted? TLSv1.0? 128 bits? SEED-SHA Accepted? TLSv1.0? 128 bits? CAMELLIA128-SHA I need to remove / disable TLSv1.0. Google has not been able to be of much help, for I get suggestions to edit /usr/bin/pveproxy and /etc/default/pveproxy and the list goes on. Can someone suggest how to fix this issue. Kind Regards Brent Clark From t.lamprecht at proxmox.com Thu Jul 26 11:22:12 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Thu, 26 Jul 2018 11:22:12 +0200 Subject: [PVE-User] Proxmox disable TLS 1 In-Reply-To: <1606b411-3907-0de1-c73a-ba171462cc0f@gmail.com> References: <1606b411-3907-0de1-c73a-ba171462cc0f@gmail.com> Message-ID: <9bcb6168-2c9a-2d52-5535-bdcabde87884@proxmox.com> Hi, Am 07/26/2018 um 11:05 AM schrieb Brent Clark: > Good day Guys > > I did a sslscan on my proxmox host, and I got the following: > > snippet: > Preferred TLSv1.0? 256 bits? ECDHE-RSA-AES256-SHA????????? Curve P-256 > DHE 256 > Accepted? TLSv1.0? 256 bits? DHE-RSA-AES256-SHA??????????? DHE 2048 bits > Accepted? TLSv1.0? 256 bits? DHE-RSA-CAMELLIA256-SHA?????? DHE 2048 bits > Accepted? TLSv1.0? 256 bits? AES256-SHA > Accepted? TLSv1.0? 256 bits? CAMELLIA256-SHA > Accepted? TLSv1.0? 128 bits? ECDHE-RSA-AES128-SHA????????? Curve P-256 > DHE 256 > Accepted? TLSv1.0? 128 bits? DHE-RSA-AES128-SHA??????????? DHE 2048 bits > Accepted? TLSv1.0? 128 bits? DHE-RSA-SEED-SHA????????????? DHE 2048 bits > Accepted? TLSv1.0? 128 bits? DHE-RSA-CAMELLIA128-SHA?????? DHE 2048 bits > Accepted? TLSv1.0? 128 bits? AES128-SHA > Accepted? TLSv1.0? 128 bits? SEED-SHA > Accepted? TLSv1.0? 128 bits? CAMELLIA128-SHA > > I need to remove / disable TLSv1.0. Google has not been able to be of > much help, for I get suggestions to edit > /usr/bin/pveproxy and /etc/default/pveproxy and the list goes on. > > Can someone suggest how to fix this issue. Ah yes, I posted a possible quick solution for this in the forum a bit ago [0]. Edit /etc/default/pveproxy to have a line with: CIPHERS="HIGH:!TLSv1:!SSLv3:!aNULL:!MD5" then systemctl restart pveproxy and you should be good to go :-) cheers, Thomas [0]: https://forum.proxmox.com/threads/disabling-tls-1-0-and-1-1-in-proxmox.35814/#post-175643 From uwe.sauter.de at gmail.com Thu Jul 26 11:23:18 2018 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Thu, 26 Jul 2018 11:23:18 +0200 Subject: [PVE-User] Proxmox disable TLS 1 In-Reply-To: <9bcb6168-2c9a-2d52-5535-bdcabde87884@proxmox.com> References: <1606b411-3907-0de1-c73a-ba171462cc0f@gmail.com> <9bcb6168-2c9a-2d52-5535-bdcabde87884@proxmox.com> Message-ID: <1296b25b-a728-4a5a-d460-2babf4914b36@gmail.com> Am 26.07.2018 um 11:22 schrieb Thomas Lamprecht: > Hi, > > Am 07/26/2018 um 11:05 AM schrieb Brent Clark: >> Good day Guys >> >> I did a sslscan on my proxmox host, and I got the following: >> >> snippet: >> Preferred TLSv1.0? 256 bits? ECDHE-RSA-AES256-SHA????????? Curve P-256 DHE 256 >> Accepted? TLSv1.0? 256 bits? DHE-RSA-AES256-SHA??????????? DHE 2048 bits >> Accepted? TLSv1.0? 256 bits? DHE-RSA-CAMELLIA256-SHA?????? DHE 2048 bits >> Accepted? TLSv1.0? 256 bits? AES256-SHA >> Accepted? TLSv1.0? 256 bits? CAMELLIA256-SHA >> Accepted? TLSv1.0? 128 bits? ECDHE-RSA-AES128-SHA????????? Curve P-256 DHE 256 >> Accepted? TLSv1.0? 128 bits? DHE-RSA-AES128-SHA??????????? DHE 2048 bits >> Accepted? TLSv1.0? 128 bits? DHE-RSA-SEED-SHA????????????? DHE 2048 bits >> Accepted? TLSv1.0? 128 bits? DHE-RSA-CAMELLIA128-SHA?????? DHE 2048 bits >> Accepted? TLSv1.0? 128 bits? AES128-SHA >> Accepted? TLSv1.0? 128 bits? SEED-SHA >> Accepted? TLSv1.0? 128 bits? CAMELLIA128-SHA >> >> I need to remove / disable TLSv1.0. Google has not been able to be of much help, for I get suggestions to edit >> /usr/bin/pveproxy and /etc/default/pveproxy and the list goes on. >> >> Can someone suggest how to fix this issue. > > Ah yes, I posted a possible quick solution for this in the forum a bit > ago [0]. > > Edit /etc/default/pveproxy to have a line with: > > CIPHERS="HIGH:!TLSv1:!SSLv3:!aNULL:!MD5" Or you are a bit more selective: CIPHERS="ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256" > > then > systemctl restart pveproxy > > and you should be good to go :-) > > cheers, > Thomas > > [0]: https://forum.proxmox.com/threads/disabling-tls-1-0-and-1-1-in-proxmox.35814/#post-175643 > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From brentgclarklist at gmail.com Thu Jul 26 11:30:53 2018 From: brentgclarklist at gmail.com (Brent Clark) Date: Thu, 26 Jul 2018 11:30:53 +0200 Subject: [PVE-User] Proxmox disable TLS 1 In-Reply-To: <9bcb6168-2c9a-2d52-5535-bdcabde87884@proxmox.com> References: <1606b411-3907-0de1-c73a-ba171462cc0f@gmail.com> <9bcb6168-2c9a-2d52-5535-bdcabde87884@proxmox.com> Message-ID: <4d08cee4-908e-39f8-371d-a7fe5c7a9440@gmail.com> Thanks for replying Interesting, I do not have that file / package. root at chs-vmh01 ~ # dpkg -l | awk '/pve-/{print $2" "$3}' snippet pve-cluster 5.0-28 pve-container 2.0-24 pve-docs 5.2-4 pve-edk2-firmware 1.20180612-1 pve-firewall 3.0-13 pve-firmware 2.0-5 pve-ha-manager 2.0-5 pve-i18n 1.0-6 pve-kernel-4.15 5.2-4 pve-kernel-4.15.15-1-pve 4.15.15-6 pve-kernel-4.15.17-1-pve 4.15.17-9 pve-kernel-4.15.17-3-pve 4.15.17-14 pve-kernel-4.15.18-1-pve 4.15.18-15 pve-libspice-server1 0.12.8-3 pve-manager 5.2-5 pve-qemu-kvm 2.11.2-1 pve-xtermjs 1.0-5 Regards Brent On 26/07/2018 11:22, Thomas Lamprecht wrote: > Hi, > > Am 07/26/2018 um 11:05 AM schrieb Brent Clark: >> Good day Guys >> >> I did a sslscan on my proxmox host, and I got the following: >> >> snippet: >> Preferred TLSv1.0? 256 bits? ECDHE-RSA-AES256-SHA????????? Curve P-256 >> DHE 256 >> Accepted? TLSv1.0? 256 bits? DHE-RSA-AES256-SHA??????????? DHE 2048 bits >> Accepted? TLSv1.0? 256 bits? DHE-RSA-CAMELLIA256-SHA?????? DHE 2048 bits >> Accepted? TLSv1.0? 256 bits? AES256-SHA >> Accepted? TLSv1.0? 256 bits? CAMELLIA256-SHA >> Accepted? TLSv1.0? 128 bits? ECDHE-RSA-AES128-SHA????????? Curve P-256 >> DHE 256 >> Accepted? TLSv1.0? 128 bits? DHE-RSA-AES128-SHA??????????? DHE 2048 bits >> Accepted? TLSv1.0? 128 bits? DHE-RSA-SEED-SHA????????????? DHE 2048 bits >> Accepted? TLSv1.0? 128 bits? DHE-RSA-CAMELLIA128-SHA?????? DHE 2048 bits >> Accepted? TLSv1.0? 128 bits? AES128-SHA >> Accepted? TLSv1.0? 128 bits? SEED-SHA >> Accepted? TLSv1.0? 128 bits? CAMELLIA128-SHA >> >> I need to remove / disable TLSv1.0. Google has not been able to be of >> much help, for I get suggestions to edit >> /usr/bin/pveproxy and /etc/default/pveproxy and the list goes on. >> > > Can someone suggest how to fix this issue. > > Ah yes, I posted a possible quick solution for this in the forum a bit > ago [0]. > > Edit /etc/default/pveproxy to have a line with: > > CIPHERS="HIGH:!TLSv1:!SSLv3:!aNULL:!MD5" > > then > systemctl restart pveproxy > > and you should be good to go :-) > > cheers, > Thomas > > [0]: > https://forum.proxmox.com/threads/disabling-tls-1-0-and-1-1-in-proxmox.35814/#post-175643 > > > > From t.lamprecht at proxmox.com Thu Jul 26 11:38:01 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Thu, 26 Jul 2018 11:38:01 +0200 Subject: [PVE-User] Proxmox disable TLS 1 In-Reply-To: <4d08cee4-908e-39f8-371d-a7fe5c7a9440@gmail.com> References: <1606b411-3907-0de1-c73a-ba171462cc0f@gmail.com> <9bcb6168-2c9a-2d52-5535-bdcabde87884@proxmox.com> <4d08cee4-908e-39f8-371d-a7fe5c7a9440@gmail.com> Message-ID: <0500b375-8dad-e5bd-275d-5b3d143f409a@proxmox.com> Am 07/26/2018 um 11:30 AM schrieb Brent Clark: > Thanks for replying > > Interesting, I do not have that file / package. > Just create it, it's not there by default but gets considered by pveproxy - if there. And pveproxy resides in the pve-manager package > root at chs-vmh01 ~ # dpkg -l | awk '/pve-/{print $2" "$3}' Not all our packages begin with pve, e.g. qemu-server, novnc-pve, etc. Use `pveversion -v` to get a list of all packages directly relevant to PVE and it's version quickly, can also be found in the Web UI (Node -> Summary -> Package versions) cheers, Thomas > > snippet > > pve-cluster 5.0-28 > pve-container 2.0-24 > pve-docs 5.2-4 > pve-edk2-firmware 1.20180612-1 > pve-firewall 3.0-13 > pve-firmware 2.0-5 > pve-ha-manager 2.0-5 > pve-i18n 1.0-6 > pve-kernel-4.15 5.2-4 > pve-kernel-4.15.15-1-pve 4.15.15-6 > pve-kernel-4.15.17-1-pve 4.15.17-9 > pve-kernel-4.15.17-3-pve 4.15.17-14 > pve-kernel-4.15.18-1-pve 4.15.18-15 > pve-libspice-server1 0.12.8-3 > pve-manager 5.2-5 > pve-qemu-kvm 2.11.2-1 > pve-xtermjs 1.0-5 > > Regards > Brent > > > On 26/07/2018 11:22, Thomas Lamprecht wrote: >> Hi, >> >> Am 07/26/2018 um 11:05 AM schrieb Brent Clark: >>> Good day Guys >>> >>> I did a sslscan on my proxmox host, and I got the following: >>> >>> snippet: >>> Preferred TLSv1.0? 256 bits? ECDHE-RSA-AES256-SHA????????? Curve >>> P-256 DHE 256 >>> Accepted? TLSv1.0? 256 bits? DHE-RSA-AES256-SHA??????????? DHE 2048 bits >>> Accepted? TLSv1.0? 256 bits? DHE-RSA-CAMELLIA256-SHA?????? DHE 2048 bits >>> Accepted? TLSv1.0? 256 bits? AES256-SHA >>> Accepted? TLSv1.0? 256 bits? CAMELLIA256-SHA >>> Accepted? TLSv1.0? 128 bits? ECDHE-RSA-AES128-SHA????????? Curve >>> P-256 DHE 256 >>> Accepted? TLSv1.0? 128 bits? DHE-RSA-AES128-SHA??????????? DHE 2048 bits >>> Accepted? TLSv1.0? 128 bits? DHE-RSA-SEED-SHA????????????? DHE 2048 bits >>> Accepted? TLSv1.0? 128 bits? DHE-RSA-CAMELLIA128-SHA?????? DHE 2048 bits >>> Accepted? TLSv1.0? 128 bits? AES128-SHA >>> Accepted? TLSv1.0? 128 bits? SEED-SHA >>> Accepted? TLSv1.0? 128 bits? CAMELLIA128-SHA >>> >>> I need to remove / disable TLSv1.0. Google has not been able to be of >>> much help, for I get suggestions to edit >>> /usr/bin/pveproxy and /etc/default/pveproxy and the list goes on. >>> >> ?> Can someone suggest how to fix this issue. >> >> Ah yes, I posted a possible quick solution for this in the forum a bit >> ago [0]. >> >> Edit /etc/default/pveproxy to have a line with: >> >> CIPHERS="HIGH:!TLSv1:!SSLv3:!aNULL:!MD5" >> >> then >> systemctl restart pveproxy >> >> and you should be good to go :-) >> >> cheers, >> Thomas >> >> [0]: >> https://forum.proxmox.com/threads/disabling-tls-1-0-and-1-1-in-proxmox.35814/#post-175643 >> >> >> >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From brentgclarklist at gmail.com Thu Jul 26 11:47:21 2018 From: brentgclarklist at gmail.com (Brent Clark) Date: Thu, 26 Jul 2018 11:47:21 +0200 Subject: [PVE-User] Proxmox disable TLS 1 In-Reply-To: <0500b375-8dad-e5bd-275d-5b3d143f409a@proxmox.com> References: <1606b411-3907-0de1-c73a-ba171462cc0f@gmail.com> <9bcb6168-2c9a-2d52-5535-bdcabde87884@proxmox.com> <4d08cee4-908e-39f8-371d-a7fe5c7a9440@gmail.com> <0500b375-8dad-e5bd-275d-5b3d143f409a@proxmox.com> Message-ID: <8afaa353-6ec1-e133-5f8a-679ad9fff8c0@gmail.com> Thomas, you the man !!! Thank you so much. Snippet of sslscan: ---------------------------------------------------------------------- Heartbleed: TLS 1.2 not vulnerable to heartbleed TLS 1.1 not vulnerable to heartbleed TLS 1.0 not vulnerable to heartbleed Supported Server Cipher(s): Preferred TLSv1.2 256 bits ECDHE-RSA-AES256-GCM-SHA384 Curve P-256 DHE 256 Accepted TLSv1.2 256 bits ECDHE-RSA-AES256-SHA384 Curve P-256 DHE 256 Accepted TLSv1.2 256 bits DHE-RSA-AES256-GCM-SHA384 DHE 2048 bits Accepted TLSv1.2 256 bits DHE-RSA-AES256-SHA256 DHE 2048 bits Accepted TLSv1.2 256 bits AES256-GCM-SHA384 Accepted TLSv1.2 256 bits AES256-SHA256 Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-GCM-SHA256 Curve P-256 DHE 256 Accepted TLSv1.2 128 bits ECDHE-RSA-AES128-SHA256 Curve P-256 DHE 256 Accepted TLSv1.2 128 bits DHE-RSA-AES128-GCM-SHA256 DHE 2048 bits Accepted TLSv1.2 128 bits DHE-RSA-AES128-SHA256 DHE 2048 bits Accepted TLSv1.2 128 bits AES128-GCM-SHA256 Accepted TLSv1.2 128 bits AES128-SHA256 SSL Certificate: Signature Algorithm: sha1WithRSAEncryption RSA Key Strength: 2048 ---------------------------------------------------------------------- Regards Brent On 26/07/2018 11:38, Thomas Lamprecht wrote: > Am 07/26/2018 um 11:30 AM schrieb Brent Clark: >> Thanks for replying >> >> Interesting, I do not have that file / package. >> > > Just create it, it's not there by default but gets considered by > pveproxy - if there. And pveproxy resides in the pve-manager package > >> root at chs-vmh01 ~ # dpkg -l | awk '/pve-/{print $2" "$3}' > > Not all our packages begin with pve, e.g. qemu-server, novnc-pve, etc. > > Use `pveversion -v` to get a list of all packages directly relevant to > PVE and it's version quickly, can also be found in the Web UI (Node -> > Summary -> Package versions) > > cheers, > Thomas > >> >> snippet >> >> pve-cluster 5.0-28 >> pve-container 2.0-24 >> pve-docs 5.2-4 >> pve-edk2-firmware 1.20180612-1 >> pve-firewall 3.0-13 >> pve-firmware 2.0-5 >> pve-ha-manager 2.0-5 >> pve-i18n 1.0-6 >> pve-kernel-4.15 5.2-4 >> pve-kernel-4.15.15-1-pve 4.15.15-6 >> pve-kernel-4.15.17-1-pve 4.15.17-9 >> pve-kernel-4.15.17-3-pve 4.15.17-14 >> pve-kernel-4.15.18-1-pve 4.15.18-15 >> pve-libspice-server1 0.12.8-3 >> pve-manager 5.2-5 >> pve-qemu-kvm 2.11.2-1 >> pve-xtermjs 1.0-5 >> >> Regards >> Brent >> >> >> On 26/07/2018 11:22, Thomas Lamprecht wrote: >>> Hi, >>> >>> Am 07/26/2018 um 11:05 AM schrieb Brent Clark: >>>> Good day Guys >>>> >>>> I did a sslscan on my proxmox host, and I got the following: >>>> >>>> snippet: >>>> Preferred TLSv1.0? 256 bits? ECDHE-RSA-AES256-SHA????????? Curve >>>> P-256 DHE 256 >>>> Accepted? TLSv1.0? 256 bits? DHE-RSA-AES256-SHA??????????? DHE 2048 >>>> bits >>>> Accepted? TLSv1.0? 256 bits? DHE-RSA-CAMELLIA256-SHA?????? DHE 2048 >>>> bits >>>> Accepted? TLSv1.0? 256 bits? AES256-SHA >>>> Accepted? TLSv1.0? 256 bits? CAMELLIA256-SHA >>>> Accepted? TLSv1.0? 128 bits? ECDHE-RSA-AES128-SHA????????? Curve >>>> P-256 DHE 256 >>>> Accepted? TLSv1.0? 128 bits? DHE-RSA-AES128-SHA??????????? DHE 2048 >>>> bits >>>> Accepted? TLSv1.0? 128 bits? DHE-RSA-SEED-SHA????????????? DHE 2048 >>>> bits >>>> Accepted? TLSv1.0? 128 bits? DHE-RSA-CAMELLIA128-SHA?????? DHE 2048 >>>> bits >>>> Accepted? TLSv1.0? 128 bits? AES128-SHA >>>> Accepted? TLSv1.0? 128 bits? SEED-SHA >>>> Accepted? TLSv1.0? 128 bits? CAMELLIA128-SHA >>>> >>>> I need to remove / disable TLSv1.0. Google has not been able to be >>>> of much help, for I get suggestions to edit >>>> /usr/bin/pveproxy and /etc/default/pveproxy and the list goes on. >>>> >>> ?> Can someone suggest how to fix this issue. >>> >>> Ah yes, I posted a possible quick solution for this in the forum a bit >>> ago [0]. >>> >>> Edit /etc/default/pveproxy to have a line with: >>> >>> CIPHERS="HIGH:!TLSv1:!SSLv3:!aNULL:!MD5" >>> >>> then >>> systemctl restart pveproxy >>> >>> and you should be good to go :-) >>> >>> cheers, >>> Thomas >>> >>> [0]: >>> https://forum.proxmox.com/threads/disabling-tls-1-0-and-1-1-in-proxmox.35814/#post-175643 >>> >>> >>> >>> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From mark at openvs.co.uk Thu Jul 26 12:25:45 2018 From: mark at openvs.co.uk (Mark Adams) Date: Thu, 26 Jul 2018 11:25:45 +0100 Subject: [PVE-User] Poor CEPH performance? or normal? In-Reply-To: References: Message-ID: Hi Ronny, Thanks for your suggestions. Do you know if it is possible to change an existing rbd pool to striping? or does this have to be done on first setup? Regards, Mark On Wed, 25 Jul 2018, 19:20 Ronny Aasen, wrote: > On 25. juli 2018 02:19, Mark Adams wrote: > > Hi All, > > > > I have a proxmox 5.1 + ceph cluster of 3 nodes, each with 12 x WD 10TB > GOLD > > drives. Network is 10Gbps on X550-T2, separate network for the ceph > cluster. > > > > I have 1 VM currently running on this cluster, which is debian stretch > with > > a zpool on it. I'm zfs sending in to it, but only getting around ~15MiB/s > > write speed. does this sound right? it seems very slow to me. > > > > Not only that, but when this zfs send is running - I can not do any > > parallel sends to any other zfs datasets inside of the same VM. They just > > seem to hang, then eventually say "dataset is busy". > > > > Any pointers or insights greatly appreciated! > > Greetings > > alwin gave you some good advice about filesystems and vm's, i wanted to > say a little about ceph. > > with 3 nodes, and the default and reccomended size=3 pools, you can not > tolerate any node failures. IOW, if you loose a node, or need to do > lengthy maintainance on it, you are running degraded. I allways have a > 4th "failure domain" node. so my cluster can selfheal (one of cephs > killer features) from a node failure. your cluster should be > > 3+[how-many-node-failures-i-want-to-be-able-to-survive-and-still-operate-sanely] > > spinning osd's with bluestore benefit greatly from ssd DB/WAL's if your > osd's have ondisk DB/WAL you can gain a lot of performance by having the > DB/WAL on a SSD or better. > > ceph gains performance with scale(number of osd nodes) . so while ceph's > aggeregate performance is awesome, an individual single thread will not > be amazing. A given set of data will exist on all 3 nodes, and you will > hit 100% of nodes with any write. so by using ceph with 3 nodes you > give ceph the worst case for performance. eg > with 4 nodes a write would hit 75%, with 6 nodes it would hit 50% of the > cluster. you see where this is going... > > But a single write will only hit one disk in 3 nodes, and will not have > a better performance then the disk it hits. you can cheat more > performance with rbd caching. and it is important for performance to get > a higher queue depth. afaik zfs uses a queue depth of 1, for ceph the > worst possible. you may have some success by buffering on one or both > ends of the transfer [1] > > if the vm have a RBD disk, you may (or may not) benefit from rbd fancy > striping[2], since operations can hit more osd's in parallel. > > > good luck > Ronny Aasen > > > [1] > > https://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/ > [2] http://docs.ceph.com/docs/master/architecture/#data-striping > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From proxmox at iancoetzee.za.net Thu Jul 26 13:22:47 2018 From: proxmox at iancoetzee.za.net (Ian Coetzee) Date: Thu, 26 Jul 2018 13:22:47 +0200 Subject: [PVE-User] Proxmox disable TLS 1 In-Reply-To: <0500b375-8dad-e5bd-275d-5b3d143f409a@proxmox.com> References: <1606b411-3907-0de1-c73a-ba171462cc0f@gmail.com> <9bcb6168-2c9a-2d52-5535-bdcabde87884@proxmox.com> <4d08cee4-908e-39f8-371d-a7fe5c7a9440@gmail.com> <0500b375-8dad-e5bd-275d-5b3d143f409a@proxmox.com> Message-ID: Hi All, I know this has been answered. What I did was to drop a reverse proxy (nginx) in front of pveproxy listening on port 443 then make the needed changes in the nginx config Makes my life way easier :) On 26 July 2018 at 11:38, Thomas Lamprecht wrote: > Am 07/26/2018 um 11:30 AM schrieb Brent Clark: > >> Thanks for replying >> >> Interesting, I do not have that file / package. >> >> > Just create it, it's not there by default but gets considered by > pveproxy - if there. And pveproxy resides in the pve-manager package > > root at chs-vmh01 ~ # dpkg -l | awk '/pve-/{print $2" "$3}' >> > > Not all our packages begin with pve, e.g. qemu-server, novnc-pve, etc. > > Use `pveversion -v` to get a list of all packages directly relevant to > PVE and it's version quickly, can also be found in the Web UI (Node -> > Summary -> Package versions) > > cheers, > Thomas > > > >> snippet >> >> pve-cluster 5.0-28 >> pve-container 2.0-24 >> pve-docs 5.2-4 >> pve-edk2-firmware 1.20180612-1 >> pve-firewall 3.0-13 >> pve-firmware 2.0-5 >> pve-ha-manager 2.0-5 >> pve-i18n 1.0-6 >> pve-kernel-4.15 5.2-4 >> pve-kernel-4.15.15-1-pve 4.15.15-6 >> pve-kernel-4.15.17-1-pve 4.15.17-9 >> pve-kernel-4.15.17-3-pve 4.15.17-14 >> pve-kernel-4.15.18-1-pve 4.15.18-15 >> pve-libspice-server1 0.12.8-3 >> pve-manager 5.2-5 >> pve-qemu-kvm 2.11.2-1 >> pve-xtermjs 1.0-5 >> >> Regards >> Brent >> >> >> On 26/07/2018 11:22, Thomas Lamprecht wrote: >> >>> Hi, >>> >>> Am 07/26/2018 um 11:05 AM schrieb Brent Clark: >>> >>>> Good day Guys >>>> >>>> I did a sslscan on my proxmox host, and I got the following: >>>> >>>> snippet: >>>> Preferred TLSv1.0 256 bits ECDHE-RSA-AES256-SHA Curve P-256 >>>> DHE 256 >>>> Accepted TLSv1.0 256 bits DHE-RSA-AES256-SHA DHE 2048 bits >>>> Accepted TLSv1.0 256 bits DHE-RSA-CAMELLIA256-SHA DHE 2048 bits >>>> Accepted TLSv1.0 256 bits AES256-SHA >>>> Accepted TLSv1.0 256 bits CAMELLIA256-SHA >>>> Accepted TLSv1.0 128 bits ECDHE-RSA-AES128-SHA Curve P-256 >>>> DHE 256 >>>> Accepted TLSv1.0 128 bits DHE-RSA-AES128-SHA DHE 2048 bits >>>> Accepted TLSv1.0 128 bits DHE-RSA-SEED-SHA DHE 2048 bits >>>> Accepted TLSv1.0 128 bits DHE-RSA-CAMELLIA128-SHA DHE 2048 bits >>>> Accepted TLSv1.0 128 bits AES128-SHA >>>> Accepted TLSv1.0 128 bits SEED-SHA >>>> Accepted TLSv1.0 128 bits CAMELLIA128-SHA >>>> >>>> I need to remove / disable TLSv1.0. Google has not been able to be of >>>> much help, for I get suggestions to edit >>>> /usr/bin/pveproxy and /etc/default/pveproxy and the list goes on. >>>> >>>> > Can someone suggest how to fix this issue. >>> >>> Ah yes, I posted a possible quick solution for this in the forum a bit >>> ago [0]. >>> >>> Edit /etc/default/pveproxy to have a line with: >>> >>> CIPHERS="HIGH:!TLSv1:!SSLv3:!aNULL:!MD5" >>> >>> then >>> systemctl restart pveproxy >>> >>> and you should be good to go :-) >>> >>> cheers, >>> Thomas >>> >>> [0]: https://forum.proxmox.com/threads/disabling-tls-1-0-and-1-1- >>> in-proxmox.35814/#post-175643 >>> >>> >>> >>> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From uwe.sauter.de at gmail.com Thu Jul 26 13:27:35 2018 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Thu, 26 Jul 2018 13:27:35 +0200 Subject: [PVE-User] Proxmox disable TLS 1 In-Reply-To: References: <1606b411-3907-0de1-c73a-ba171462cc0f@gmail.com> <9bcb6168-2c9a-2d52-5535-bdcabde87884@proxmox.com> <4d08cee4-908e-39f8-371d-a7fe5c7a9440@gmail.com> <0500b375-8dad-e5bd-275d-5b3d143f409a@proxmox.com> Message-ID: <1dd12d51-922b-3e0d-97be-ea7d4d2052c2@gmail.com> Would you mind to share the relevant parts of your nginx config? Does forwarding NoVNC traffic work? Am 26.07.2018 um 13:22 schrieb Ian Coetzee: > Hi All, > > I know this has been answered. > > What I did was to drop a reverse proxy (nginx) in front of pveproxy > listening on port 443 then make the needed changes in the nginx config > > Makes my life way easier :) > > On 26 July 2018 at 11:38, Thomas Lamprecht wrote: > >> Am 07/26/2018 um 11:30 AM schrieb Brent Clark: >> >>> Thanks for replying >>> >>> Interesting, I do not have that file / package. >>> >>> >> Just create it, it's not there by default but gets considered by >> pveproxy - if there. And pveproxy resides in the pve-manager package >> >> root at chs-vmh01 ~ # dpkg -l | awk '/pve-/{print $2" "$3}' >>> >> >> Not all our packages begin with pve, e.g. qemu-server, novnc-pve, etc. >> >> Use `pveversion -v` to get a list of all packages directly relevant to >> PVE and it's version quickly, can also be found in the Web UI (Node -> >> Summary -> Package versions) >> >> cheers, >> Thomas >> >> >> >>> snippet >>> >>> pve-cluster 5.0-28 >>> pve-container 2.0-24 >>> pve-docs 5.2-4 >>> pve-edk2-firmware 1.20180612-1 >>> pve-firewall 3.0-13 >>> pve-firmware 2.0-5 >>> pve-ha-manager 2.0-5 >>> pve-i18n 1.0-6 >>> pve-kernel-4.15 5.2-4 >>> pve-kernel-4.15.15-1-pve 4.15.15-6 >>> pve-kernel-4.15.17-1-pve 4.15.17-9 >>> pve-kernel-4.15.17-3-pve 4.15.17-14 >>> pve-kernel-4.15.18-1-pve 4.15.18-15 >>> pve-libspice-server1 0.12.8-3 >>> pve-manager 5.2-5 >>> pve-qemu-kvm 2.11.2-1 >>> pve-xtermjs 1.0-5 >>> >>> Regards >>> Brent >>> >>> >>> On 26/07/2018 11:22, Thomas Lamprecht wrote: >>> >>>> Hi, >>>> >>>> Am 07/26/2018 um 11:05 AM schrieb Brent Clark: >>>> >>>>> Good day Guys >>>>> >>>>> I did a sslscan on my proxmox host, and I got the following: >>>>> >>>>> snippet: >>>>> Preferred TLSv1.0 256 bits ECDHE-RSA-AES256-SHA Curve P-256 >>>>> DHE 256 >>>>> Accepted TLSv1.0 256 bits DHE-RSA-AES256-SHA DHE 2048 bits >>>>> Accepted TLSv1.0 256 bits DHE-RSA-CAMELLIA256-SHA DHE 2048 bits >>>>> Accepted TLSv1.0 256 bits AES256-SHA >>>>> Accepted TLSv1.0 256 bits CAMELLIA256-SHA >>>>> Accepted TLSv1.0 128 bits ECDHE-RSA-AES128-SHA Curve P-256 >>>>> DHE 256 >>>>> Accepted TLSv1.0 128 bits DHE-RSA-AES128-SHA DHE 2048 bits >>>>> Accepted TLSv1.0 128 bits DHE-RSA-SEED-SHA DHE 2048 bits >>>>> Accepted TLSv1.0 128 bits DHE-RSA-CAMELLIA128-SHA DHE 2048 bits >>>>> Accepted TLSv1.0 128 bits AES128-SHA >>>>> Accepted TLSv1.0 128 bits SEED-SHA >>>>> Accepted TLSv1.0 128 bits CAMELLIA128-SHA >>>>> >>>>> I need to remove / disable TLSv1.0. Google has not been able to be of >>>>> much help, for I get suggestions to edit >>>>> /usr/bin/pveproxy and /etc/default/pveproxy and the list goes on. >>>>> >>>>> > Can someone suggest how to fix this issue. >>>> >>>> Ah yes, I posted a possible quick solution for this in the forum a bit >>>> ago [0]. >>>> >>>> Edit /etc/default/pveproxy to have a line with: >>>> >>>> CIPHERS="HIGH:!TLSv1:!SSLv3:!aNULL:!MD5" >>>> >>>> then >>>> systemctl restart pveproxy >>>> >>>> and you should be good to go :-) >>>> >>>> cheers, >>>> Thomas >>>> >>>> [0]: https://forum.proxmox.com/threads/disabling-tls-1-0-and-1-1- >>>> in-proxmox.35814/#post-175643 >>>> >>>> >>>> >>>> _______________________________________________ >>> pve-user mailing list >>> pve-user at pve.proxmox.com >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >>> >> >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From proxmox at iancoetzee.za.net Thu Jul 26 13:50:31 2018 From: proxmox at iancoetzee.za.net (Ian Coetzee) Date: Thu, 26 Jul 2018 13:50:31 +0200 Subject: [PVE-User] Proxmox disable TLS 1 In-Reply-To: <1dd12d51-922b-3e0d-97be-ea7d4d2052c2@gmail.com> References: <1606b411-3907-0de1-c73a-ba171462cc0f@gmail.com> <9bcb6168-2c9a-2d52-5535-bdcabde87884@proxmox.com> <4d08cee4-908e-39f8-371d-a7fe5c7a9440@gmail.com> <0500b375-8dad-e5bd-275d-5b3d143f409a@proxmox.com> <1dd12d51-922b-3e0d-97be-ea7d4d2052c2@gmail.com> Message-ID: Hi Uwe, I won't mind, sanitised version below. This config gives me an A+ score on www.ssllabs.com It was originally sourced from https://pve.proxmox.com/wiki/Web_Interface_Via_Nginx_Proxy with some tweaks added by me over a period $ sudo cat /etc/nginx/sites-enabled/proxmox-gui > upstream proxmox { > server localhost:8006; > } > > server { > listen 80 default_server; > rewrite ^(.*) https:// permanent; > } > > server { > listen 443; > server_name _; > ssl on; > ssl_certificate ***; > ssl_certificate_key ***; > ssl_protocols TLSv1.2; > ssl_prefer_server_ciphers on; > ssl_ciphers > ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5; > proxy_redirect off; > server_tokens off; > add_header X-XSS-Protection "1; mode=block"; > add_header X-Content-Type-Options nosniff; > add_header X-Frame-Options SAMEORIGIN; > proxy_hide_header X-Powered-By; > location / { > proxy_http_version 1.1; > proxy_set_header Upgrade $http_upgrade; > proxy_set_header Connection "upgrade"; > proxy_pass https://localhost:8006; > proxy_buffering off; > client_max_body_size 0; > proxy_connect_timeout 3600s; > proxy_read_timeout 3600s; > proxy_send_timeout 3600s; > send_timeout 3600s; > } > } > NoVNC, xterm.js and Spice works perfectly. Kind regards On 26 July 2018 at 13:27, Uwe Sauter wrote: > Would you mind to share the relevant parts of your nginx config? Does > forwarding NoVNC traffic work? > > > > Am 26.07.2018 um 13:22 schrieb Ian Coetzee: > > Hi All, > > > > I know this has been answered. > > > > What I did was to drop a reverse proxy (nginx) in front of pveproxy > > listening on port 443 then make the needed changes in the nginx config > > > > Makes my life way easier :) > > > > On 26 July 2018 at 11:38, Thomas Lamprecht > wrote: > > > >> Am 07/26/2018 um 11:30 AM schrieb Brent Clark: > >> > >>> Thanks for replying > >>> > >>> Interesting, I do not have that file / package. > >>> > >>> > >> Just create it, it's not there by default but gets considered by > >> pveproxy - if there. And pveproxy resides in the pve-manager package > >> > >> root at chs-vmh01 ~ # dpkg -l | awk '/pve-/{print $2" "$3}' > >>> > >> > >> Not all our packages begin with pve, e.g. qemu-server, novnc-pve, etc. > >> > >> Use `pveversion -v` to get a list of all packages directly relevant to > >> PVE and it's version quickly, can also be found in the Web UI (Node -> > >> Summary -> Package versions) > >> > >> cheers, > >> Thomas > >> > >> > >> > >>> snippet > >>> > >>> pve-cluster 5.0-28 > >>> pve-container 2.0-24 > >>> pve-docs 5.2-4 > >>> pve-edk2-firmware 1.20180612-1 > >>> pve-firewall 3.0-13 > >>> pve-firmware 2.0-5 > >>> pve-ha-manager 2.0-5 > >>> pve-i18n 1.0-6 > >>> pve-kernel-4.15 5.2-4 > >>> pve-kernel-4.15.15-1-pve 4.15.15-6 > >>> pve-kernel-4.15.17-1-pve 4.15.17-9 > >>> pve-kernel-4.15.17-3-pve 4.15.17-14 > >>> pve-kernel-4.15.18-1-pve 4.15.18-15 > >>> pve-libspice-server1 0.12.8-3 > >>> pve-manager 5.2-5 > >>> pve-qemu-kvm 2.11.2-1 > >>> pve-xtermjs 1.0-5 > >>> > >>> Regards > >>> Brent > >>> > >>> > >>> On 26/07/2018 11:22, Thomas Lamprecht wrote: > >>> > >>>> Hi, > >>>> > >>>> Am 07/26/2018 um 11:05 AM schrieb Brent Clark: > >>>> > >>>>> Good day Guys > >>>>> > >>>>> I did a sslscan on my proxmox host, and I got the following: > >>>>> > >>>>> snippet: > >>>>> Preferred TLSv1.0 256 bits ECDHE-RSA-AES256-SHA Curve > P-256 > >>>>> DHE 256 > >>>>> Accepted TLSv1.0 256 bits DHE-RSA-AES256-SHA DHE 2048 > bits > >>>>> Accepted TLSv1.0 256 bits DHE-RSA-CAMELLIA256-SHA DHE 2048 > bits > >>>>> Accepted TLSv1.0 256 bits AES256-SHA > >>>>> Accepted TLSv1.0 256 bits CAMELLIA256-SHA > >>>>> Accepted TLSv1.0 128 bits ECDHE-RSA-AES128-SHA Curve > P-256 > >>>>> DHE 256 > >>>>> Accepted TLSv1.0 128 bits DHE-RSA-AES128-SHA DHE 2048 > bits > >>>>> Accepted TLSv1.0 128 bits DHE-RSA-SEED-SHA DHE 2048 > bits > >>>>> Accepted TLSv1.0 128 bits DHE-RSA-CAMELLIA128-SHA DHE 2048 > bits > >>>>> Accepted TLSv1.0 128 bits AES128-SHA > >>>>> Accepted TLSv1.0 128 bits SEED-SHA > >>>>> Accepted TLSv1.0 128 bits CAMELLIA128-SHA > >>>>> > >>>>> I need to remove / disable TLSv1.0. Google has not been able to be of > >>>>> much help, for I get suggestions to edit > >>>>> /usr/bin/pveproxy and /etc/default/pveproxy and the list goes on. > >>>>> > >>>>> > Can someone suggest how to fix this issue. > >>>> > >>>> Ah yes, I posted a possible quick solution for this in the forum a bit > >>>> ago [0]. > >>>> > >>>> Edit /etc/default/pveproxy to have a line with: > >>>> > >>>> CIPHERS="HIGH:!TLSv1:!SSLv3:!aNULL:!MD5" > >>>> > >>>> then > >>>> systemctl restart pveproxy > >>>> > >>>> and you should be good to go :-) > >>>> > >>>> cheers, > >>>> Thomas > >>>> > >>>> [0]: https://forum.proxmox.com/threads/disabling-tls-1-0-and-1-1- > >>>> in-proxmox.35814/#post-175643 > >>>> > >>>> > >>>> > >>>> _______________________________________________ > >>> pve-user mailing list > >>> pve-user at pve.proxmox.com > >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>> > >> > >> > >> _______________________________________________ > >> pve-user mailing list > >> pve-user at pve.proxmox.com > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >> > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From uwe.sauter.de at gmail.com Thu Jul 26 13:54:33 2018 From: uwe.sauter.de at gmail.com (Uwe Sauter) Date: Thu, 26 Jul 2018 13:54:33 +0200 Subject: [PVE-User] Proxmox disable TLS 1 In-Reply-To: References: <1606b411-3907-0de1-c73a-ba171462cc0f@gmail.com> <9bcb6168-2c9a-2d52-5535-bdcabde87884@proxmox.com> <4d08cee4-908e-39f8-371d-a7fe5c7a9440@gmail.com> <0500b375-8dad-e5bd-275d-5b3d143f409a@proxmox.com> <1dd12d51-922b-3e0d-97be-ea7d4d2052c2@gmail.com> Message-ID: <10e7e94d-3617-4c5f-7de6-9ddcd419257e@gmail.com> Thank you very much, Uwe Am 26.07.2018 um 13:50 schrieb Ian Coetzee: > Hi Uwe, > > I won't mind, sanitised version below. This config gives me an A+ score on www.ssllabs.com > > It was originally sourced from https://pve.proxmox.com/wiki/Web_Interface_Via_Nginx_Proxy with some tweaks added by me over a period > > $ sudo cat /etc/nginx/sites-enabled/proxmox-gui > upstream proxmox { > ??? server localhost:8006; > } > > server { > ??? listen 80 default_server; > ??? rewrite ^(.*) https:// permanent; > } > > server { > ??? listen 443; > ??? server_name _; > ??? ssl on; > ??? ssl_certificate ***; > ??? ssl_certificate_key ***; > ??? ssl_protocols TLSv1.2; > ??? ssl_prefer_server_ciphers on; > ??? ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5; > ??? proxy_redirect off; > ??? server_tokens off; > ??? add_header X-XSS-Protection "1; mode=block"; > ??? add_header X-Content-Type-Options nosniff; > ??? add_header X-Frame-Options SAMEORIGIN; > ??? proxy_hide_header X-Powered-By; > ??? location / { > ??????? proxy_http_version 1.1; > ??????? proxy_set_header Upgrade $http_upgrade; > ??????? proxy_set_header Connection "upgrade"; > ??????? proxy_pass https://localhost:8006; > ??????? proxy_buffering off; > ??????? client_max_body_size 0; > ??????? proxy_connect_timeout? 3600s; > ??????? proxy_read_timeout? 3600s; > ??????? proxy_send_timeout? 3600s; > ??????? send_timeout? 3600s; > ??? } > } > > > NoVNC, xterm.js and Spice works perfectly. > > Kind regards > > > On 26 July 2018 at 13:27, Uwe Sauter > wrote: > > Would you mind to share the relevant parts of your nginx config? Does forwarding NoVNC traffic work? > > > > Am 26.07.2018 um 13:22 schrieb Ian Coetzee: > > Hi All, > > > > I know this has been answered. > > > > What I did was to drop a reverse proxy (nginx) in front of pveproxy > > listening on port 443 then make the needed changes in the nginx config > > > > Makes my life way easier :) > > > > On 26 July 2018 at 11:38, Thomas Lamprecht > wrote: > > > >> Am 07/26/2018 um 11:30 AM schrieb Brent Clark: > >> > >>> Thanks for replying > >>> > >>> Interesting, I do not have that file / package. > >>> > >>> > >> Just create it, it's not there by default but gets considered by > >> pveproxy - if there. And pveproxy resides in the pve-manager package > >> > >> root at chs-vmh01 ~ # dpkg -l | awk '/pve-/{print $2" "$3}' > >>> > >> > >> Not all our packages begin with pve, e.g. qemu-server, novnc-pve, etc. > >> > >> Use `pveversion -v` to get a list of all packages directly relevant to > >> PVE and it's version quickly, can also be found in the Web UI (Node -> > >> Summary -> Package versions) > >> > >> cheers, > >> Thomas > >> > >> > >> > >>> snippet > >>> > >>> pve-cluster 5.0-28 > >>> pve-container 2.0-24 > >>> pve-docs 5.2-4 > >>> pve-edk2-firmware 1.20180612-1 > >>> pve-firewall 3.0-13 > >>> pve-firmware 2.0-5 > >>> pve-ha-manager 2.0-5 > >>> pve-i18n 1.0-6 > >>> pve-kernel-4.15 5.2-4 > >>> pve-kernel-4.15.15-1-pve 4.15.15-6 > >>> pve-kernel-4.15.17-1-pve 4.15.17-9 > >>> pve-kernel-4.15.17-3-pve 4.15.17-14 > >>> pve-kernel-4.15.18-1-pve 4.15.18-15 > >>> pve-libspice-server1 0.12.8-3 > >>> pve-manager 5.2-5 > >>> pve-qemu-kvm 2.11.2-1 > >>> pve-xtermjs 1.0-5 > >>> > >>> Regards > >>> Brent > >>> > >>> > >>> On 26/07/2018 11:22, Thomas Lamprecht wrote: > >>> > >>>> Hi, > >>>> > >>>> Am 07/26/2018 um 11:05 AM schrieb Brent Clark: > >>>> > >>>>> Good day Guys > >>>>> > >>>>> I did a sslscan on my proxmox host, and I got the following: > >>>>> > >>>>> snippet: > >>>>> Preferred TLSv1.0? 256 bits? ECDHE-RSA-AES256-SHA? ? ? ? ? Curve P-256 > >>>>> DHE 256 > >>>>> Accepted? TLSv1.0? 256 bits? DHE-RSA-AES256-SHA? ? ? ? ? ? DHE 2048 bits > >>>>> Accepted? TLSv1.0? 256 bits? DHE-RSA-CAMELLIA256-SHA? ? ? ?DHE 2048 bits > >>>>> Accepted? TLSv1.0? 256 bits? AES256-SHA > >>>>> Accepted? TLSv1.0? 256 bits? CAMELLIA256-SHA > >>>>> Accepted? TLSv1.0? 128 bits? ECDHE-RSA-AES128-SHA? ? ? ? ? Curve P-256 > >>>>> DHE 256 > >>>>> Accepted? TLSv1.0? 128 bits? DHE-RSA-AES128-SHA? ? ? ? ? ? DHE 2048 bits > >>>>> Accepted? TLSv1.0? 128 bits? DHE-RSA-SEED-SHA? ? ? ? ? ? ? DHE 2048 bits > >>>>> Accepted? TLSv1.0? 128 bits? DHE-RSA-CAMELLIA128-SHA? ? ? ?DHE 2048 bits > >>>>> Accepted? TLSv1.0? 128 bits? AES128-SHA > >>>>> Accepted? TLSv1.0? 128 bits? SEED-SHA > >>>>> Accepted? TLSv1.0? 128 bits? CAMELLIA128-SHA > >>>>> > >>>>> I need to remove / disable TLSv1.0. Google has not been able to be of > >>>>> much help, for I get suggestions to edit > >>>>> /usr/bin/pveproxy and /etc/default/pveproxy and the list goes on. > >>>>> > >>>>>? > Can someone suggest how to fix this issue. > >>>> > >>>> Ah yes, I posted a possible quick solution for this in the forum a bit > >>>> ago [0]. > >>>> > >>>> Edit /etc/default/pveproxy to have a line with: > >>>> > >>>> CIPHERS="HIGH:!TLSv1:!SSLv3:!aNULL:!MD5" > >>>> > >>>> then > >>>> systemctl restart pveproxy > >>>> > >>>> and you should be good to go :-) > >>>> > >>>> cheers, > >>>> Thomas > >>>> > >>>> [0]: https://forum.proxmox.com/threads/disabling-tls-1-0-and-1-1- > > >>>> in-proxmox.35814/#post-175643 > >>>> > >>>> > >>>> > >>>> _______________________________________________ > >>> pve-user mailing list > >>> pve-user at pve.proxmox.com > >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >>> > >> > >> > >> _______________________________________________ > >> pve-user mailing list > >> pve-user at pve.proxmox.com > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >> > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From marcus.haarmann at midoco.de Fri Jul 27 11:02:05 2018 From: marcus.haarmann at midoco.de (Marcus Haarmann) Date: Fri, 27 Jul 2018 11:02:05 +0200 (CEST) Subject: [PVE-User] Proxmox Kernel / Ceph Integration Message-ID: <1578731983.1178474.1532682125473.JavaMail.zimbra@midoco.de> Hi experts, we are using a Proxmox cluster with an underlying ceph storage. Versions are pve 5.2-2 with kernel 4.15.18-1-pve and ceph luminous 12.2.5 We are running a couple of VM and also Containers there. 3 virtual NIC (as bond balance-alb), ceph uses 2 bonded 10GBit interfaces (public/cluster separated) It occurs during nightly backup that backup stalls. In parallel, we get lots of messages in the dmesg: [137612.371311] libceph: mon0 192.168.16.31:6789 session established [137643.090541] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon [137643.091383] libceph: mon1 192.168.16.32:6789 session established [137673.810526] libceph: mon1 192.168.16.32:6789 session lost, hunting for new mon [137673.811388] libceph: mon2 192.168.16.34:6789 session established [137704.530567] libceph: mon2 192.168.16.34:6789 session lost, hunting for new mon [137704.531363] libceph: mon0 192.168.16.31:6789 session established [137735.250593] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon [137735.251352] libceph: mon1 192.168.16.32:6789 session established [137765.970608] libceph: mon1 192.168.16.32:6789 session lost, hunting for new mon [137765.971544] libceph: mon0 192.168.16.31:6789 session established [137796.690605] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon [137796.691412] libceph: mon1 192.168.16.32:6789 session established We are searching for the issue for a while, since the blocking backup is not easy to overcome (unblocking does not help, only stop and migrate to a different server, since the rbd device seems to block). It seems to be related to the ceph messages. We found the following patch related to these messages (which may lead to a blocking state in kernel): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7b4c443d139f1d2b5570da475f7a9cbcef86740c We have tried to patch the kernel ourselfes, but this was not successful. Although I presume the real error situation is related to a network problem, it would be nice to have an official backport of this patch in the pve kernel. Anybody can do that ? (only one line of code) We are trying to modify the bonding mode because the network connection seems to be unstable, maybe this solves the issue. Thank you very much and best regards, Marcus Haarmann From t.lamprecht at proxmox.com Fri Jul 27 11:12:24 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Fri, 27 Jul 2018 11:12:24 +0200 Subject: [PVE-User] Proxmox Kernel / Ceph Integration In-Reply-To: <1578731983.1178474.1532682125473.JavaMail.zimbra@midoco.de> References: <1578731983.1178474.1532682125473.JavaMail.zimbra@midoco.de> Message-ID: Hi, Am 07/27/2018 um 11:02 AM schrieb Marcus Haarmann: > Hi experts, > > we are using a Proxmox cluster with an underlying ceph storage. > Versions are pve 5.2-2 with kernel 4.15.18-1-pve and ceph luminous 12.2.5 > We are running a couple of VM and also Containers there. > 3 virtual NIC (as bond balance-alb), ceph uses 2 bonded 10GBit interfaces (public/cluster separated) > > It occurs during nightly backup that backup stalls. In parallel, we get lots of messages in the dmesg: > [137612.371311] libceph: mon0 192.168.16.31:6789 session established > [137643.090541] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon > [137643.091383] libceph: mon1 192.168.16.32:6789 session established > [137673.810526] libceph: mon1 192.168.16.32:6789 session lost, hunting for new mon > [137673.811388] libceph: mon2 192.168.16.34:6789 session established > [137704.530567] libceph: mon2 192.168.16.34:6789 session lost, hunting for new mon > [137704.531363] libceph: mon0 192.168.16.31:6789 session established > [137735.250593] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon > [137735.251352] libceph: mon1 192.168.16.32:6789 session established > [137765.970608] libceph: mon1 192.168.16.32:6789 session lost, hunting for new mon > [137765.971544] libceph: mon0 192.168.16.31:6789 session established > [137796.690605] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon > [137796.691412] libceph: mon1 192.168.16.32:6789 session established > > We are searching for the issue for a while, since the blocking backup is not easy to overcome (unblocking does not help, > only stop and migrate to a different server, since the rbd device seems to block). > It seems to be related to the ceph messages. > We found the following patch related to these messages (which may lead to a blocking state in kernel): > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7b4c443d139f1d2b5570da475f7a9cbcef86740c > > We have tried to patch the kernel ourselfes, but this was not successful. > Porting the patch was not successful or the patch did not worked as expected? > Although I presume the real error situation is related to a network problem, it would be nice to have an > official backport of this patch in the pve kernel. > Anybody can do that ? (only one line of code) > A single line can also wreak havoc just fine ;-) But this one seems/sounds harmless, regression-wise. But it would be really good to first know if the patch addresses the issue at all. > We are trying to modify the bonding mode because the network connection seems to be unstable, > maybe this solves the issue. > Sounds like it's worth a shot, if you already know that the network may not be fully stable, as you may want to do something about that sooner or later anyway. cheers, Thomas From marcus.haarmann at midoco.de Fri Jul 27 12:05:16 2018 From: marcus.haarmann at midoco.de (Marcus Haarmann) Date: Fri, 27 Jul 2018 12:05:16 +0200 (CEST) Subject: [PVE-User] Proxmox Kernel / Ceph Integration In-Reply-To: References: <1578731983.1178474.1532682125473.JavaMail.zimbra@midoco.de> Message-ID: <1656958966.1182029.1532685916671.JavaMail.zimbra@midoco.de> Hi, we tried to build the kernel and the message vanished, yes. However, the error situation was more unstable than before, so we went back to an official version of the kernel, because the build process was "rough" for us ... (we do not do this kind of things very often). Since this patch is officially backported in several kernels (obviously from a ceph team member who knows what he was doing), but not in 4.15, I would presume this should not make things worse. The situation which we get here seems to be the same as descibed: a mon connection is lost for some reason and the kernel ceph client seems to be stuck in a kind of endless loop, because of two identical timers. That is what the patch addresses. But you are right, we should solve the issue why the mon connection is lost (this is during a backup -> high I/O on network, which might cause the mon connection loss). Next step is to modify the bond to be just a failover instead of balance-alb (more conservative ...) Marcus Haarmann Von: "Thomas Lamprecht" An: "pve-user" , "Marcus Haarmann" Gesendet: Freitag, 27. Juli 2018 11:12:24 Betreff: Re: [PVE-User] Proxmox Kernel / Ceph Integration Hi, Am 07/27/2018 um 11:02 AM schrieb Marcus Haarmann: > Hi experts, > > we are using a Proxmox cluster with an underlying ceph storage. > Versions are pve 5.2-2 with kernel 4.15.18-1-pve and ceph luminous 12.2.5 > We are running a couple of VM and also Containers there. > 3 virtual NIC (as bond balance-alb), ceph uses 2 bonded 10GBit interfaces (public/cluster separated) > > It occurs during nightly backup that backup stalls. In parallel, we get lots of messages in the dmesg: > [137612.371311] libceph: mon0 192.168.16.31:6789 session established > [137643.090541] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon > [137643.091383] libceph: mon1 192.168.16.32:6789 session established > [137673.810526] libceph: mon1 192.168.16.32:6789 session lost, hunting for new mon > [137673.811388] libceph: mon2 192.168.16.34:6789 session established > [137704.530567] libceph: mon2 192.168.16.34:6789 session lost, hunting for new mon > [137704.531363] libceph: mon0 192.168.16.31:6789 session established > [137735.250593] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon > [137735.251352] libceph: mon1 192.168.16.32:6789 session established > [137765.970608] libceph: mon1 192.168.16.32:6789 session lost, hunting for new mon > [137765.971544] libceph: mon0 192.168.16.31:6789 session established > [137796.690605] libceph: mon0 192.168.16.31:6789 session lost, hunting for new mon > [137796.691412] libceph: mon1 192.168.16.32:6789 session established > > We are searching for the issue for a while, since the blocking backup is not easy to overcome (unblocking does not help, > only stop and migrate to a different server, since the rbd device seems to block). > It seems to be related to the ceph messages. > We found the following patch related to these messages (which may lead to a blocking state in kernel): > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7b4c443d139f1d2b5570da475f7a9cbcef86740c > > We have tried to patch the kernel ourselfes, but this was not successful. > Porting the patch was not successful or the patch did not worked as expected? > Although I presume the real error situation is related to a network problem, it would be nice to have an > official backport of this patch in the pve kernel. > Anybody can do that ? (only one line of code) > A single line can also wreak havoc just fine ;-) But this one seems/sounds harmless, regression-wise. But it would be really good to first know if the patch addresses the issue at all. > We are trying to modify the bonding mode because the network connection seems to be unstable, > maybe this solves the issue. > Sounds like it's worth a shot, if you already know that the network may not be fully stable, as you may want to do something about that sooner or later anyway. cheers, Thomas From ronny+pve-user at aasen.cx Fri Jul 27 14:05:53 2018 From: ronny+pve-user at aasen.cx (ronny+pve-user at aasen.cx) Date: Fri, 27 Jul 2018 14:05:53 +0200 Subject: [PVE-User] Poor CEPH performance? or normal? In-Reply-To: References: Message-ID: <3d59359e-e4a5-aa32-a43a-757732d27cfe@aasen.cx> rbd striping is a per image setting. you may need to make the rbd image and migrate data. kind regards Ronny Aasen On 07/26/18 12:25, Mark Adams wrote: > Hi Ronny, > > Thanks for your suggestions. Do you know if it is possible to change an > existing rbd pool to striping? or does this have to be done on first setup? > > Regards, > Mark > > On Wed, 25 Jul 2018, 19:20 Ronny Aasen, wrote: > >> On 25. juli 2018 02:19, Mark Adams wrote: >>> Hi All, >>> >>> I have a proxmox 5.1 + ceph cluster of 3 nodes, each with 12 x WD 10TB >> GOLD >>> drives. Network is 10Gbps on X550-T2, separate network for the ceph >> cluster. >>> I have 1 VM currently running on this cluster, which is debian stretch >> with >>> a zpool on it. I'm zfs sending in to it, but only getting around ~15MiB/s >>> write speed. does this sound right? it seems very slow to me. >>> >>> Not only that, but when this zfs send is running - I can not do any >>> parallel sends to any other zfs datasets inside of the same VM. They just >>> seem to hang, then eventually say "dataset is busy". >>> >>> Any pointers or insights greatly appreciated! >> Greetings >> >> alwin gave you some good advice about filesystems and vm's, i wanted to >> say a little about ceph. >> >> with 3 nodes, and the default and reccomended size=3 pools, you can not >> tolerate any node failures. IOW, if you loose a node, or need to do >> lengthy maintainance on it, you are running degraded. I allways have a >> 4th "failure domain" node. so my cluster can selfheal (one of cephs >> killer features) from a node failure. your cluster should be >> >> 3+[how-many-node-failures-i-want-to-be-able-to-survive-and-still-operate-sanely] >> >> spinning osd's with bluestore benefit greatly from ssd DB/WAL's if your >> osd's have ondisk DB/WAL you can gain a lot of performance by having the >> DB/WAL on a SSD or better. >> >> ceph gains performance with scale(number of osd nodes) . so while ceph's >> aggeregate performance is awesome, an individual single thread will not >> be amazing. A given set of data will exist on all 3 nodes, and you will >> hit 100% of nodes with any write. so by using ceph with 3 nodes you >> give ceph the worst case for performance. eg >> with 4 nodes a write would hit 75%, with 6 nodes it would hit 50% of the >> cluster. you see where this is going... >> >> But a single write will only hit one disk in 3 nodes, and will not have >> a better performance then the disk it hits. you can cheat more >> performance with rbd caching. and it is important for performance to get >> a higher queue depth. afaik zfs uses a queue depth of 1, for ceph the >> worst possible. you may have some success by buffering on one or both >> ends of the transfer [1] >> >> if the vm have a RBD disk, you may (or may not) benefit from rbd fancy >> striping[2], since operations can hit more osd's in parallel. >> >> >> good luck >> Ronny Aasen >> >> >> [1] >> >> https://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/ >> [2] http://docs.ceph.com/docs/master/architecture/#data-striping >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From athompso at athompso.net Fri Jul 27 15:24:37 2018 From: athompso at athompso.net (Adam Thompson) Date: Fri, 27 Jul 2018 08:24:37 -0500 Subject: [PVE-User] Proxmox Kernel / Ceph Integration In-Reply-To: <1578731983.1178474.1532682125473.JavaMail.zimbra@midoco.de> References: <1578731983.1178474.1532682125473.JavaMail.zimbra@midoco.de> Message-ID: On 2018-07-27 04:02, Marcus Haarmann wrote: > Hi experts, > > we are using a Proxmox cluster with an underlying ceph storage. > Versions are pve 5.2-2 with kernel 4.15.18-1-pve and ceph luminous > 12.2.5 > We are running a couple of VM and also Containers there. > 3 virtual NIC (as bond balance-alb), ceph uses 2 bonded 10GBit > interfaces (public/cluster separated) I have a thought, but need to know which network subnets are attached to which bondX interfaces. Also, you mention you have 3 "virtual NIC" in ALB mode. Is this a V-in-V situation? What bonding mode are you using for the two 10GE interfaces you dedicate to CEPH? (Feel free to just paste /etc/network/interfaces if that's easier than typing it all out - just make notes about which i/f does what.) -Adam From athompso at athompso.net Fri Jul 27 15:46:14 2018 From: athompso at athompso.net (Adam Thompson) Date: Fri, 27 Jul 2018 08:46:14 -0500 Subject: [PVE-User] Poor CEPH performance? or normal? In-Reply-To: <3d59359e-e4a5-aa32-a43a-757732d27cfe@aasen.cx> References: <3d59359e-e4a5-aa32-a43a-757732d27cfe@aasen.cx> Message-ID: <2a9e3b016473338e6e43ac66ae80e962@athompso.net> On 2018-07-27 07:05, ronny+pve-user at aasen.cx wrote: > rbd striping is a per image setting. you may need to make the rbd > image and migrate data. > > On 07/26/18 12:25, Mark Adams wrote: >> Thanks for your suggestions. Do you know if it is possible to change >> an >> existing rbd pool to striping? or does this have to be done on first >> setup? Please be aware that striping will not result in any increased performance, if you are using "safe" I/O modes, i.e. your VM waits for a successful flush-to-disk after every sector. In that scenario, CEPH will never give you write performance equal to a local disk because you're limited to the bandwidth of a single remote disk [subsystem] *plus* the network round-trip latency, which even if measured in microseconds, still adds up. Based on my experience with this and other distributed storage systems, I believe you will likely find that you get large write-performance gains by: 1. use the largest possible block size during writes. 512B sectors are the worst-case scenario for any remote storage. Try to write in chunks of *at least* 1 MByte, and it's not unreasonable nowadays to write in chunks of 64MB or larger. The rationale here is that you're spending more time sending data, and less time waiting for ACKs. The more you can tilt that in favor of data, the better off you are. (There are downsides to huge sector/block/chunk sizes, though - this isn't a "free lunch" scenario. See #5.) 2. relax your write-consistency requirements. If you can tolerate the small risk with "Write Back" you should see better performance, especially during burst writes. During large sequential writes, there are not many ways to violate the laws of physics, and CEPH automatically amplifies your writes by (in your case) a factor of 2x due to replication. 3. switch to storage devices with the best possible local write speed, for OSDs. OSDs are limited by the performance of the underlying device or virtual device. (e.g. it's totally possible to run OSDs on a hardware RAID6 controller) 4. Avoid CoW-on-CoW. Write amplification means you'll lose around 50% of your IOPS and/or I/O bandwidth for each level of CoW nesting, depending on workload. So don't put CEPH OSDs on, ssy, BTRFS or ZFS filesystems. A worst-case scenario would be something like running a VM using ZFS on top of CEPH, where the OSDs are located on BTRFS filsystems, which are in turn virtual devices hosted on ZFS filesystems. Welcome to 1980's storage performance, in that case! (I did it without realizing once... seriously, 5 MBps sequential writes was a good day!) FWIW, CoW filesystems are generally awesome - just not when stacked. A sufficiently fast external NAS running ZFS with VMs stored over NFS can provide decent performance, *if* tuned correctly. iX Systems, for example, spends a lot of time & effort making this work well, including some lovely HA NAS appliances. 5. Remember the triangle. You can optimize a distributed storage system for any TWO of: a) cost, b) resiliency/reliability/HA, or c) speed. (This is a specific case of the traditional good/fast/cheap:pick-any-2 adage.) I'm not sure I'm saying anything new here, I may have just summarized the discussion, but the points remain valid. Good luck with your performance problems. -Adam From marcus.haarmann at midoco.de Fri Jul 27 17:42:13 2018 From: marcus.haarmann at midoco.de (Marcus Haarmann) Date: Fri, 27 Jul 2018 17:42:13 +0200 (CEST) Subject: [PVE-User] Proxmox Kernel / Ceph Integration In-Reply-To: References: <1578731983.1178474.1532682125473.JavaMail.zimbra@midoco.de> Message-ID: <940961156.1205895.1532706133980.JavaMail.zimbra@midoco.de> Hi Adam, here is the setup: auto lo iface lo inet loopback iface eth0 inet manual iface eth1 inet manual iface eth2 inet manual iface eth3 inet manual iface eth4 inet manual iface eth5 inet manual auto bond0 iface bond0 inet manual slaves eth0 eth1 bond_miimon 100 bond_mode balance-alb #frontside auto bond1 iface bond1 inet static address 192.168.16.31 netmask 255.255.255.0 slaves eth2 eth3 bond_miimon 100 bond_mode balance-alb pre-up (ifconfig eth2 mtu 8996 && ifconfig eth3 mtu 8996) mtu 8996 #corosync auto bond2 iface bond2 inet static address 192.168.17.31 netmask 255.255.255.0 slaves eth4 eth5 bond_miimon 100 bond_mode balance-alb pre-up (ifconfig eth4 mtu 8996 && ifconfig eth5 mtu 8996) mtu 8996 #ceph auto vmbr0 iface vmbr0 inet static address 192.168.19.31 netmask 255.255.255.0 gateway 192.168.19.1 bridge_ports bond0 bridge_stp off bridge_fd 0 bond0/vmbr0 is for using the VMs (frontend side) bond1 is ceph public net bond2 is ceph cluster net corosync is running in 192.168.16.x Marcus Haarmann Von: "Adam Thompson" An: "pve-user" CC: "Marcus Haarmann" Gesendet: Freitag, 27. Juli 2018 15:24:37 Betreff: Re: [PVE-User] Proxmox Kernel / Ceph Integration On 2018-07-27 04:02, Marcus Haarmann wrote: > Hi experts, > > we are using a Proxmox cluster with an underlying ceph storage. > Versions are pve 5.2-2 with kernel 4.15.18-1-pve and ceph luminous > 12.2.5 > We are running a couple of VM and also Containers there. > 3 virtual NIC (as bond balance-alb), ceph uses 2 bonded 10GBit > interfaces (public/cluster separated) I have a thought, but need to know which network subnets are attached to which bondX interfaces. Also, you mention you have 3 "virtual NIC" in ALB mode. Is this a V-in-V situation? What bonding mode are you using for the two 10GE interfaces you dedicate to CEPH? (Feel free to just paste /etc/network/interfaces if that's easier than typing it all out - just make notes about which i/f does what.) -Adam From rakesh.jain at FireEye.com Fri Jul 27 18:24:29 2018 From: rakesh.jain at FireEye.com (Rakesh Jain) Date: Fri, 27 Jul 2018 16:24:29 +0000 Subject: [PVE-User] Increase RAW Disk SIZE from UI Message-ID: <0BE18A1B-AB0A-4476-83DC-C2FC8819E44E@fireeye.com> Hello team, We are using Virtual Environment 5.2-1 in our environment. I have Resized raw disk image from 30 GB to 112 GB. We don?t have LVM configured. What I did is -> Shutdown the VM Virtual Machine 100 (labs-provision) on node pve01 from GUI Increase the DISK size, Memory Size and Increased CPU cores from GUI Restarted the VM from GUI Now when I see the disk it shows the increased size but how can I make this change reflecting to /dev/sda3 ?? rakesh.jain at labs-provision:~$ sudo fdisk -l [sudo] password for rakesh.jain: GPT PMBR size mismatch (67108863 != 234881023) will be corrected by w(rite). Disk /dev/sda: 112 GiB, 120259084288 bytes, 234881024 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 5101DB8D-88D6-487F-9660-CBA9D05EE18F Device Start End Sectors Size Type /dev/sda1 2048 4095 2048 1M BIOS boot /dev/sda2 4096 268287 264192 129M Linux filesystem /dev/sda3 268288 67106815 66838528 31.9G Linux filesystem rakesh.jain at labs-provision:~$ df -hT Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 2.0G 0 2.0G 0% /dev tmpfs tmpfs 396M 41M 356M 11% /run /dev/sda3 ext4 32G 18G 12G 61% / tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup /dev/sda2 ext2 125M 35M 84M 30% /boot tmpfs tmpfs 396M 0 396M 0% /run/user/26528 tmpfs tmpfs 396M 0 396M 0% /run/user/37379 Please let me know. Any help is appreciated. -Rakesh Jain This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto. From yannis.milios at gmail.com Fri Jul 27 18:38:48 2018 From: yannis.milios at gmail.com (Yannis Milios) Date: Fri, 27 Jul 2018 17:38:48 +0100 Subject: [PVE-User] Increase RAW Disk SIZE from UI In-Reply-To: <0BE18A1B-AB0A-4476-83DC-C2FC8819E44E@fireeye.com> References: <0BE18A1B-AB0A-4476-83DC-C2FC8819E44E@fireeye.com> Message-ID: Well, that?s basic linux sysadmin, nothing related to pve ... You can find dozens of articles by googling. If you feel lazy to search, here?s some examples ... https://access.redhat.com/articles/1190213 https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/storage_administration_guide/ext4grow On Fri, 27 Jul 2018 at 17:24, Rakesh Jain wrote: > Hello team, > > We are using Virtual Environment 5.2-1 in our environment. > > I have Resized raw disk image from 30 GB to 112 GB. We don?t have LVM > configured. > > What I did is -> > Shutdown the VM Virtual Machine 100 (labs-provision) on node pve01 from GUI > Increase the DISK size, Memory Size and Increased CPU cores from GUI > Restarted the VM from GUI > > Now when I see the disk it shows the increased size but how can I make > this change reflecting to /dev/sda3 ?? > > rakesh.jain at labs-provision:~$ sudo fdisk -l > [sudo] password for rakesh.jain: > GPT PMBR size mismatch (67108863 != 234881023) will be corrected by > w(rite). > Disk /dev/sda: 112 GiB, 120259084288 bytes, 234881024 sectors > Units: sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disklabel type: gpt > Disk identifier: 5101DB8D-88D6-487F-9660-CBA9D05EE18F > > Device Start End Sectors Size Type > /dev/sda1 2048 4095 2048 1M BIOS boot > /dev/sda2 4096 268287 264192 129M Linux filesystem > /dev/sda3 268288 67106815 66838528 31.9G Linux filesystem > > rakesh.jain at labs-provision:~$ df -hT > Filesystem Type Size Used Avail Use% Mounted on > udev devtmpfs 2.0G 0 2.0G 0% /dev > tmpfs tmpfs 396M 41M 356M 11% /run > /dev/sda3 ext4 32G 18G 12G 61% / > tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm > tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock > tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup > /dev/sda2 ext2 125M 35M 84M 30% /boot > tmpfs tmpfs 396M 0 396M 0% /run/user/26528 > tmpfs tmpfs 396M 0 396M 0% /run/user/37379 > > Please let me know. Any help is appreciated. > > -Rakesh Jain > > This email and any attachments thereto may contain private, confidential, > and/or privileged material for the sole use of the intended recipient. Any > review, copying, or distribution of this email (or any attachments thereto) > by others is strictly prohibited. If you are not the intended recipient, > please contact the sender immediately and permanently delete the original > and any copies of this email and any attachments thereto. > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Sent from Gmail Mobile From rakesh.jain at FireEye.com Fri Jul 27 18:46:40 2018 From: rakesh.jain at FireEye.com (Rakesh Jain) Date: Fri, 27 Jul 2018 16:46:40 +0000 Subject: [PVE-User] Increase RAW Disk SIZE from UI In-Reply-To: References: <0BE18A1B-AB0A-4476-83DC-C2FC8819E44E@fireeye.com> Message-ID: <27D989DC-BEC2-4976-AE96-92DC6A8F61E6@fireeye.com> Thanks for reply. I know but the PVE documentation mentions a different command -> qm command You can resize your disks online or offline with command line: qm resize exemple: to add 5G to your virtio0 disk on vmid100: qm resize 100 virtio0 +5G So, got confused. -Rakesh Jain ?On 27/07/18, 10:09 PM, "pve-user on behalf of Yannis Milios" wrote: Well, that?s basic linux sysadmin, nothing related to pve ... You can find dozens of articles by googling. If you feel lazy to search, here?s some examples ... https://access.redhat.com/articles/1190213 https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/storage_administration_guide/ext4grow On Fri, 27 Jul 2018 at 17:24, Rakesh Jain wrote: > Hello team, > > We are using Virtual Environment 5.2-1 in our environment. > > I have Resized raw disk image from 30 GB to 112 GB. We don?t have LVM > configured. > > What I did is -> > Shutdown the VM Virtual Machine 100 (labs-provision) on node pve01 from GUI > Increase the DISK size, Memory Size and Increased CPU cores from GUI > Restarted the VM from GUI > > Now when I see the disk it shows the increased size but how can I make > this change reflecting to /dev/sda3 ?? > > rakesh.jain at labs-provision:~$ sudo fdisk -l > [sudo] password for rakesh.jain: > GPT PMBR size mismatch (67108863 != 234881023) will be corrected by > w(rite). > Disk /dev/sda: 112 GiB, 120259084288 bytes, 234881024 sectors > Units: sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disklabel type: gpt > Disk identifier: 5101DB8D-88D6-487F-9660-CBA9D05EE18F > > Device Start End Sectors Size Type > /dev/sda1 2048 4095 2048 1M BIOS boot > /dev/sda2 4096 268287 264192 129M Linux filesystem > /dev/sda3 268288 67106815 66838528 31.9G Linux filesystem > > rakesh.jain at labs-provision:~$ df -hT > Filesystem Type Size Used Avail Use% Mounted on > udev devtmpfs 2.0G 0 2.0G 0% /dev > tmpfs tmpfs 396M 41M 356M 11% /run > /dev/sda3 ext4 32G 18G 12G 61% / > tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm > tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock > tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup > /dev/sda2 ext2 125M 35M 84M 30% /boot > tmpfs tmpfs 396M 0 396M 0% /run/user/26528 > tmpfs tmpfs 396M 0 396M 0% /run/user/37379 > > Please let me know. Any help is appreciated. > > -Rakesh Jain > > This email and any attachments thereto may contain private, confidential, > and/or privileged material for the sole use of the intended recipient. Any > review, copying, or distribution of this email (or any attachments thereto) > by others is strictly prohibited. If you are not the intended recipient, > please contact the sender immediately and permanently delete the original > and any copies of this email and any attachments thereto. > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Sent from Gmail Mobile _______________________________________________ pve-user mailing list pve-user at pve.proxmox.com https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto. From ml+pve-user at valo.at Fri Jul 27 18:48:59 2018 From: ml+pve-user at valo.at (Christian Kivalo) Date: Fri, 27 Jul 2018 18:48:59 +0200 Subject: [PVE-User] Increase RAW Disk SIZE from UI In-Reply-To: <27D989DC-BEC2-4976-AE96-92DC6A8F61E6@fireeye.com> References: <0BE18A1B-AB0A-4476-83DC-C2FC8819E44E@fireeye.com> <27D989DC-BEC2-4976-AE96-92DC6A8F61E6@fireeye.com> Message-ID: <8B30CAF2-02CA-4EF5-9FD9-624CD8408E7D@valo.at> On July 27, 2018 6:46:40 PM GMT+02:00, Rakesh Jain wrote: >Thanks for reply. I know but the PVE documentation mentions a different >command -> > >qm command > >You can resize your disks online or offline with command line: >qm resize > >exemple: to add 5G to your virtio0 disk on vmid100: >qm resize 100 virtio0 +5G This just does on the command line what you did from the GUI. You still need to grow the filesystem inside the VM yourself. >So, got confused. > >-Rakesh Jain > >?On 27/07/18, 10:09 PM, "pve-user on behalf of Yannis Milios" > >wrote: > > Well, that?s basic linux sysadmin, nothing related to pve ... > >You can find dozens of articles by googling. If you feel lazy to >search, > here?s some examples ... > > https://access.redhat.com/articles/1190213 > > >https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/storage_administration_guide/ext4grow > > > >On Fri, 27 Jul 2018 at 17:24, Rakesh Jain >wrote: > > > Hello team, > > > > We are using Virtual Environment 5.2-1 in our environment. > > >> I have Resized raw disk image from 30 GB to 112 GB. We don?t have LVM > > configured. > > > > What I did is -> >> Shutdown the VM Virtual Machine 100 (labs-provision) on node pve01 >from GUI > > Increase the DISK size, Memory Size and Increased CPU cores from GUI > > Restarted the VM from GUI > > >> Now when I see the disk it shows the increased size but how can I >make > > this change reflecting to /dev/sda3 ?? > > > > rakesh.jain at labs-provision:~$ sudo fdisk -l > > [sudo] password for rakesh.jain: > > GPT PMBR size mismatch (67108863 != 234881023) will be corrected by > > w(rite). > > Disk /dev/sda: 112 GiB, 120259084288 bytes, 234881024 sectors > > Units: sectors of 1 * 512 = 512 bytes > > Sector size (logical/physical): 512 bytes / 512 bytes > > I/O size (minimum/optimal): 512 bytes / 512 bytes > > Disklabel type: gpt > > Disk identifier: 5101DB8D-88D6-487F-9660-CBA9D05EE18F > > > > Device Start End Sectors Size Type > > /dev/sda1 2048 4095 2048 1M BIOS boot > > /dev/sda2 4096 268287 264192 129M Linux filesystem > > /dev/sda3 268288 67106815 66838528 31.9G Linux filesystem > > > > rakesh.jain at labs-provision:~$ df -hT > > Filesystem Type Size Used Avail Use% Mounted on > > udev devtmpfs 2.0G 0 2.0G 0% /dev > > tmpfs tmpfs 396M 41M 356M 11% /run > > /dev/sda3 ext4 32G 18G 12G 61% / > > tmpfs tmpfs 2.0G 0 2.0G 0% /dev/shm > > tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock > > tmpfs tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup > > /dev/sda2 ext2 125M 35M 84M 30% /boot > > tmpfs tmpfs 396M 0 396M 0% /run/user/26528 > > tmpfs tmpfs 396M 0 396M 0% /run/user/37379 > > > > Please let me know. Any help is appreciated. > > > > -Rakesh Jain > > >> This email and any attachments thereto may contain private, >confidential, >> and/or privileged material for the sole use of the intended >recipient. Any >> review, copying, or distribution of this email (or any attachments >thereto) >> by others is strictly prohibited. If you are not the intended >recipient, >> please contact the sender immediately and permanently delete the >original > > and any copies of this email and any attachments thereto. > > _______________________________________________ > > pve-user mailing list > > pve-user at pve.proxmox.com > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > > -- > Sent from Gmail Mobile > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > >This email and any attachments thereto may contain private, >confidential, and/or privileged material for the sole use of the >intended recipient. Any review, copying, or distribution of this email >(or any attachments thereto) by others is strictly prohibited. If you >are not the intended recipient, please contact the sender immediately >and permanently delete the original and any copies of this email and >any attachments thereto. >_______________________________________________ >pve-user mailing list >pve-user at pve.proxmox.com >https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Christian Kivalo From ekgermann at semperen.com Fri Jul 27 20:45:17 2018 From: ekgermann at semperen.com (Eric Germann) Date: Fri, 27 Jul 2018 14:45:17 -0400 Subject: [PVE-User] Unable to join cluster Message-ID: I have two new Proxmox boxes in a virgin cluster. No VM?s, etc. The only thing setup on them is networking. I created a cluster on the first one successfully. However, I try to join the second to the cluster, I get the following error: Starting worker failed: unable to parse worker upid 'UPID:pve02:00001876:00026B6B:5B5B6751:clusterjoin:2001:470:e2fc:10::160:root at pam:' (500) The nodes are all IPv4 and IPv6 enabled. The cluster IP as show in the config is IPv6. Is this the issue? If I put the v6 address in brackets, same error. If I substitute the IPv4 address, I get the following error Etablishing API connection with host '172.28.10.160' TASK ERROR: 401 401 authentication failure Thoughts? They haven?t been up more than 1 hr when this occurs. Thanks in advance EKg From ken.woods at alaska.gov Fri Jul 27 20:58:12 2018 From: ken.woods at alaska.gov (Woods, Ken A (DNR)) Date: Fri, 27 Jul 2018 18:58:12 +0000 Subject: [PVE-User] Unable to join cluster In-Reply-To: References: Message-ID: What's the output of "pvecm status"? Same version on both nodes? Time synchronized? How are you adding the second node? Try "pvecm add XXX.XXX.XXX.XXX" Is multicast enabled? Does omping work and not drop packets? (Also, can somebody fix the typo of the missing "s" in "Etablishing API connection with host XXXXX") kw -----Original Message----- From: pve-user [mailto:pve-user-bounces at pve.proxmox.com] On Behalf Of Eric Germann Sent: Friday, July 27, 2018 10:45 AM To: pve-user at pve.proxmox.com Subject: [PVE-User] Unable to join cluster I have two new Proxmox boxes in a virgin cluster. No VM?s, etc. The only thing setup on them is networking. I created a cluster on the first one successfully. However, I try to join the second to the cluster, I get the following error: Starting worker failed: unable to parse worker upid 'UPID:pve02:00001876:00026B6B:5B5B6751:clusterjoin:2001:470:e2fc:10::160:root at pam:' (500) The nodes are all IPv4 and IPv6 enabled. The cluster IP as show in the config is IPv6. Is this the issue? If I put the v6 address in brackets, same error. If I substitute the IPv4 address, I get the following error Etablishing API connection with host '172.28.10.160' TASK ERROR: 401 401 authentication failure Thoughts? They haven?t been up more than 1 hr when this occurs. Thanks in advance EKg From ekgermann at semperen.com Fri Jul 27 21:18:43 2018 From: ekgermann at semperen.com (Eric Germann) Date: Fri, 27 Jul 2018 15:18:43 -0400 Subject: [PVE-User] Unable to join cluster In-Reply-To: References: Message-ID: <7A03D429-52E0-49EA-8419-F96F5E0E02EA@semperen.com> > On Jul 27, 2018, at 2:58 PM, Woods, Ken A (DNR) wrote: > > What's the output of "pvecm status?? root at pve01:~# pvwcm status -bash: pvwcm: command not found root at pve01:~# pvecm status Quorum information ------------------ Date: Fri Jul 27 19:09:38 2018 Quorum provider: corosync_votequorum Nodes: 1 Node ID: 0x00000001 Ring ID: 1/56 Quorate: Yes Votequorum information ---------------------- Expected votes: 1 Highest expected: 1 Total votes: 1 Quorum: 1 Flags: Quorate Membership information ---------------------- Nodeid Votes Name 0x00000001 1 2001:470:e2fc:10::160 (local) > Same version on both nodes? Yes > Time synchronized? Yes. NTP > How are you adding the second node? Try "pvecm add XXX.XXX.XXX.XXX? Both through the GUI and from the command line. Oddity is V6 and V4 errors are different. > Is multicast enabled? Does omping work and not drop packets? > Need to look in to that. moping returns no messages. It?s on a dumb switch. I?ll reconfig on to an intelligent switch I can enable mcast on EKG > > (Also, can somebody fix the typo of the missing "s" in "Etablishing API connection with host XXXXX") > > kw > > > -----Original Message----- > From: pve-user [mailto:pve-user-bounces at pve.proxmox.com] On Behalf Of Eric Germann > Sent: Friday, July 27, 2018 10:45 AM > To: pve-user at pve.proxmox.com > Subject: [PVE-User] Unable to join cluster > > I have two new Proxmox boxes in a virgin cluster. No VM?s, etc. The only thing setup on them is networking. > > I created a cluster on the first one successfully. > > However, I try to join the second to the cluster, I get the following error: > > Starting worker failed: unable to parse worker upid 'UPID:pve02:00001876:00026B6B:5B5B6751:clusterjoin:2001:470:e2fc:10::160:root at pam:' (500) > > The nodes are all IPv4 and IPv6 enabled. The cluster IP as show in the config is IPv6. Is this the issue? > > If I put the v6 address in brackets, same error. > > If I substitute the IPv4 address, I get the following error > > Etablishing API connection with host '172.28.10.160' > TASK ERROR: 401 401 authentication failure > > Thoughts? They haven?t been up more than 1 hr when this occurs. > > Thanks in advance > > EKg > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From mark at openvs.co.uk Sat Jul 28 13:00:15 2018 From: mark at openvs.co.uk (Mark Adams) Date: Sat, 28 Jul 2018 12:00:15 +0100 Subject: [PVE-User] Poor CEPH performance? or normal? In-Reply-To: <2a9e3b016473338e6e43ac66ae80e962@athompso.net> References: <3d59359e-e4a5-aa32-a43a-757732d27cfe@aasen.cx> <2a9e3b016473338e6e43ac66ae80e962@athompso.net> Message-ID: Hi Adam, Thanks for your great round up there - Your points are excellent. What I have ended up doing a few days ago (apologies have been too busy to respond..) was set rbd cache = true under each client in the ceph.conf - This got me from 15MB/s up to about 70MB/s. I then set the disk holding the zfs dataset to writeback cache in proxmox (as you note below) and that has bumped it up to about 130MB/s -- Which I am happy with for this setup. Regards, Mark On 27 July 2018 at 14:46, Adam Thompson wrote: > On 2018-07-27 07:05, ronny+pve-user at aasen.cx wrote: > >> rbd striping is a per image setting. you may need to make the rbd >> image and migrate data. >> >> On 07/26/18 12:25, Mark Adams wrote: >> >>> Thanks for your suggestions. Do you know if it is possible to change an >>> existing rbd pool to striping? or does this have to be done on first >>> setup? >>> >> > Please be aware that striping will not result in any increased > performance, if you are using "safe" I/O modes, i.e. your VM waits for a > successful flush-to-disk after every sector. In that scenario, CEPH will > never give you write performance equal to a local disk because you're > limited to the bandwidth of a single remote disk [subsystem] *plus* the > network round-trip latency, which even if measured in microseconds, still > adds up. > > Based on my experience with this and other distributed storage systems, I > believe you will likely find that you get large write-performance gains by: > > 1. use the largest possible block size during writes. 512B sectors are > the worst-case scenario for any remote storage. Try to write in chunks of > *at least* 1 MByte, and it's not unreasonable nowadays to write in chunks > of 64MB or larger. The rationale here is that you're spending more time > sending data, and less time waiting for ACKs. The more you can tilt that > in favor of data, the better off you are. (There are downsides to huge > sector/block/chunk sizes, though - this isn't a "free lunch" scenario. See > #5.) > > 2. relax your write-consistency requirements. If you can tolerate the > small risk with "Write Back" you should see better performance, especially > during burst writes. During large sequential writes, there are not many > ways to violate the laws of physics, and CEPH automatically amplifies your > writes by (in your case) a factor of 2x due to replication. > > 3. switch to storage devices with the best possible local write speed, for > OSDs. OSDs are limited by the performance of the underlying device or > virtual device. (e.g. it's totally possible to run OSDs on a hardware > RAID6 controller) > > 4. Avoid CoW-on-CoW. Write amplification means you'll lose around 50% of > your IOPS and/or I/O bandwidth for each level of CoW nesting, depending on > workload. So don't put CEPH OSDs on, ssy, BTRFS or ZFS filesystems. A > worst-case scenario would be something like running a VM using ZFS on top > of CEPH, where the OSDs are located on BTRFS filsystems, which are in turn > virtual devices hosted on ZFS filesystems. Welcome to 1980's storage > performance, in that case! (I did it without realizing once... seriously, > 5 MBps sequential writes was a good day!) FWIW, CoW filesystems are > generally awesome - just not when stacked. A sufficiently fast external > NAS running ZFS with VMs stored over NFS can provide decent performance, > *if* tuned correctly. iX Systems, for example, spends a lot of time & > effort making this work well, including some lovely HA NAS appliances. > > 5. Remember the triangle. You can optimize a distributed storage system > for any TWO of: a) cost, b) resiliency/reliability/HA, or c) speed. (This > is a specific case of the traditional good/fast/cheap:pick-any-2 adage.) > > > I'm not sure I'm saying anything new here, I may have just summarized the > discussion, but the points remain valid. > > Good luck with your performance problems. > -Adam > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From gavin.henry at gmail.com Sun Jul 29 11:39:03 2018 From: gavin.henry at gmail.com (Gavin Henry) Date: Sun, 29 Jul 2018 10:39:03 +0100 Subject: [PVE-User] vzdump causing complete box hang Message-ID: Hi all, Once a week when we run vzdump on some database VMs we get the following and the whole box hang and needs a cold remote reboot. Any suggestions? We're seeing no other issues, no disk issues on the RAID, no filesystem issues: Jul 29 05:15:03 leela vzdump[280112]: starting task UPID:leela:0004465A:038A102F:5B5D4D57:vzdump::root at pam: Jul 29 05:15:03 leela vzdump[280154]: INFO: starting new backup job: vzdump --mailnotification always --quiet 1 --mode snapshot --compress lzo --storage Backups --all 1 Jul 29 05:15:03 leela pmxcfs[6094]: [status] notice: received log Jul 29 05:15:03 leela vzdump[280154]: INFO: Starting Backup of VM 151 (openvz) Jul 29 05:15:04 leela kernel: EXT3-fs: barriers disabled Jul 29 05:15:04 leela kernel: kjournald starting. Commit interval 5 seconds Jul 29 05:15:04 leela kernel: EXT3-fs (dm-2): using internal journal Jul 29 05:15:04 leela kernel: ext3_orphan_cleanup: deleting unreferenced inode 132645031 Jul 29 05:15:04 leela kernel: EXT3-fs (dm-2): 45 orphan inodes deleted Jul 29 05:15:04 leela kernel: EXT3-fs (dm-2): recovery complete Jul 29 05:15:04 leela kernel: EXT3-fs (dm-2): mounted filesystem with ordered data mode Jul 29 05:17:01 leela /USR/SBIN/CRON[282355]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Jul 29 05:17:23 leela vzdump[280154]: INFO: Finished Backup of VM 151 (00:02:20) Jul 29 05:17:24 leela vzdump[280154]: INFO: Starting Backup of VM 153 (openvz) Jul 29 05:17:24 leela kernel: EXT3-fs: barriers disabled Jul 29 05:17:24 leela kernel: kjournald starting. Commit interval 5 seconds Jul 29 05:17:24 leela kernel: EXT3-fs (dm-2): using internal journal Jul 29 05:24:33 leela kernel: Buffer I/O error on device dm-2, logical block 433061993 Jul 29 05:24:33 leela kernel: lost page write due to I/O error on dm-2 Jul 29 05:25:02 leela /USR/SBIN/CRON[291615]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Jul 29 05:28:59 leela kernel: EXT3-fs (dm-2): error: ext3_put_super: Couldn't clean up the journal Jul 29 05:29:00 leela vzdump[280154]: ERROR: Backup of VM 153 failed - command '(cd /mnt/vzsnap0/private/153;find . '(' -regex '^\.$' ')' -o '(' -type 's' -prune ')' -o -print0|sed 's/\\/\\\\/g'|tar cpf - --totals --sparse --numeric-owner --no-recursion --one-file-system ./etc/vzdump/vps.conf --null -T -|lzop) >/mnt/replicated/backups/dump/vzdump-openvz-153-2018_07_29-05_17_24.tar.dat' failed: exit code 2 Jul 29 05:29:00 leela vzdump[280154]: INFO: Starting Backup of VM 157 (openvz) Jul 29 05:29:01 leela kernel: EXT3-fs: barriers disabled Jul 29 05:29:01 leela kernel: kjournald starting. Commit interval 5 seconds Jul 29 05:29:01 leela kernel: EXT3-fs (dm-2): using internal journal Jul 29 05:29:01 leela kernel: ext3_orphan_cleanup: deleting unreferenced inode 132645031 Jul 29 05:29:01 leela kernel: ext3_orphan_cleanup: deleting unreferenced inode 122814467 Jul 29 05:29:01 leela kernel: EXT3-fs (dm-2): 45 orphan inodes deleted Jul 29 05:29:01 leela kernel: EXT3-fs (dm-2): recovery complete Jul 29 05:29:01 leela kernel: EXT3-fs (dm-2): mounted filesystem with ordered data mode Jul 29 05:30:04 leela ansible-setup: Invoked with filter=* gather_subset=['all'] fact_path=/etc/ansible/facts.d gather_timeout=10 Jul 29 05:32:05 leela ansible-setup: Invoked with filter=* gather_subset=['all'] fact_path=/etc/ansible/facts.d gather_timeout=10 Jul 29 05:35:02 leela /USR/SBIN/CRON[304114]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Jul 29 05:35:03 leela kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception. Jul 29 05:35:48 leela kernel: JBD: I/O error detected when updating journal superblock for dm-2. Jul 29 05:35:54 leela kernel: EXT3-fs (dm-2): error: ext3_journal_start_sb: Detected aborted journal Jul 29 05:35:54 leela kernel: EXT3-fs (dm-2): error: remounting filesystem read-only Jul 29 05:36:40 leela kernel: EXT3-fs (dm-2): error: ext3_put_super: Couldn't clean up the journal Jul 29 05:36:41 leela vzdump[280154]: ERROR: Backup of VM 157 failed - command '(cd /mnt/vzsnap0/private/157;find . '(' -regex '^\.$' ')' -o '(' -type 's' -prune ')' -o -print0|sed 's/\\/\\\\/g'|tar cpf - --totals --sparse --numeric-owner --no-recursion --one-file-system ./etc/vzdump/vps.conf --null -T -|lzop) >/mnt/replicated/backups/dump/vzdump-openvz-157-2018_07_29-05_29_00.tar.dat' failed: exit code 2 Jul 29 05:36:41 leela vzdump[280154]: INFO: Starting Backup of VM 161 (openvz) Jul 29 05:36:42 leela kernel: EXT3-fs: barriers disabled Jul 29 05:36:42 leela kernel: kjournald starting. Commit interval 5 seconds Jul 29 05:36:42 leela kernel: EXT3-fs (dm-2): using internal journal Jul 29 05:35:42 leela kernel: EXT3-fs error (device dm-2): ext3_get_inode_loc: unable to read inode block - inode=15024130, block=60096514 Jul 29 05:35:42 leela kernel: Buffer I/O error on device dm-2, logical block 0 Jul 29 05:35:42 leela kernel: lost page write due to I/O error on dm-2 Jul 29 05:35:42 leela kernel: EXT3-fs (dm-2): I/O error while writing superblock Jul 29 05:35:42 leela kernel: EXT3-fs (dm-2): error in ext3_reserve_inode_write: IO failure Jul 29 05:35:42 leela kernel: Buffer I/O error on device dm-2, logical block 0 Jul 29 05:35:42 leela kernel: lost page write due to I/O error on dm-2 Jul 29 05:35:42 leela kernel: EXT3-fs (dm-2): I/O error while writing superblock Jul 29 05:35:43 leela kernel: EXT3-fs error (device dm-2): ext3_get_inode_loc: unable to read inode block - inode=23347202, block=93388802 Jul 29 05:35:43 leela kernel: Buffer I/O error on device dm-2, logical block 0 Jul 29 05:35:43 leela kernel: lost page write due to I/O error on dm-2 Jul 29 05:35:43 leela kernel: EXT3-fs (dm-2): I/O error while writing superblock Jul 29 05:35:43 leela kernel: EXT3-fs (dm-2): error in ext3_reserve_inode_write: IO failure Jul 29 05:35:43 leela kernel: Buffer I/O error on device dm-2, logical block 0 They are getting dumped on to a glusterfs replicated set up which is shared with another proxmox in the same rack in a two node cluster. Thanks, Gavin. From gavin.henry at gmail.com Sun Jul 29 16:21:02 2018 From: gavin.henry at gmail.com (Gavin Henry) Date: Sun, 29 Jul 2018 15:21:02 +0100 Subject: [PVE-User] vzdump causing complete box hang In-Reply-To: References: Message-ID: On 29 July 2018 at 14:09, Woods, Ken A (DNR) wrote: > I?d start by figure out what?s going on with with all the errors on DM-2 Yep, that's: dmsetup info Name: pve-data-real State: ACTIVE Read Ahead: 256 Tables present: LIVE Open count: 2 Event number: 0 Major, minor: 253, 2 Number of targets: 1 UUID: LVM-RDfBSyRHSFN0z6KxboxgFoJfPn4nD1tFumUZGz4X9NTfxAhccnDZ6Y3oQ0EM72HI-real dm-2 mount: /dev/mapper/pve-data on /var/lib/vz type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered) dmsetup ls --tree pve-swap (253:1) ?? (8:3) pve-root (253:0) ?? (8:3) pve-data (253:3) ??pve-data-real (253:2) ?? (8:3) pve-vzsnap--leela--0 (253:5) ??pve-vzsnap--leela--0-cow (253:4) ? ?? (8:3) ??pve-data-real (253:2) ?? (8:3) ls -lha /dev/mapper/ total 0 drwxr-xr-x 2 root root 180 Jul 29 08:06 . drwxr-xr-x 16 root root 6.3K Jul 29 08:14 .. crw------- 1 root root 10, 61 Jul 29 08:06 control lrwxrwxrwx 1 root root 7 Jul 29 08:06 pve-data -> ../dm-3 lrwxrwxrwx 1 root root 7 Jul 29 08:06 pve-data-real -> ../dm-2 lrwxrwxrwx 1 root root 7 Jul 29 08:06 pve-root -> ../dm-0 lrwxrwxrwx 1 root root 7 Jul 29 08:06 pve-swap -> ../dm-1 lrwxrwxrwx 1 root root 7 Jul 29 08:06 pve-vzsnap--leela--0 -> ../dm-5 lrwxrwxrwx 1 root root 7 Jul 29 08:06 pve-vzsnap--leela--0-cow -> ../dm-4 Some downtime with a full fsck I think... From athompso at athompso.net Sun Jul 29 16:41:24 2018 From: athompso at athompso.net (Adam Thompson) Date: Sun, 29 Jul 2018 09:41:24 -0500 Subject: [PVE-User] Poor CEPH performance? or normal? In-Reply-To: References: <3d59359e-e4a5-aa32-a43a-757732d27cfe@aasen.cx> <2a9e3b016473338e6e43ac66ae80e962@athompso.net> Message-ID: <2BC86780-1C83-402C-81B1-1BE7560100EF@athompso.net> On July 28, 2018 6:00:15 AM CDT, Mark Adams wrote: >Hi Adam, > >Thanks for your great round up there - Your points are excellent. > >What I have ended up doing a few days ago (apologies have been too busy >to >respond..) was set rbd cache = true under each client in the ceph.conf >- >This got me from 15MB/s up to about 70MB/s. I then set the disk holding >the >zfs dataset to writeback cache in proxmox (as you note below) and that >has >bumped it up to about 130MB/s -- Which I am happy with for this setup. > >Regards, >Mark > >On 27 July 2018 at 14:46, Adam Thompson wrote: > >> On 2018-07-27 07:05, ronny+pve-user at aasen.cx wrote: >> >>> rbd striping is a per image setting. you may need to make the rbd >>> image and migrate data. >>> >>> On 07/26/18 12:25, Mark Adams wrote: >>> >>>> Thanks for your suggestions. Do you know if it is possible to >change an >>>> existing rbd pool to striping? or does this have to be done on >first >>>> setup? >>>> >>> >> Please be aware that striping will not result in any increased >> performance, if you are using "safe" I/O modes, i.e. your VM waits >for a >> successful flush-to-disk after every sector. In that scenario, CEPH >will >> never give you write performance equal to a local disk because you're >> limited to the bandwidth of a single remote disk [subsystem] *plus* >the >> network round-trip latency, which even if measured in microseconds, >still >> adds up. >> >> Based on my experience with this and other distributed storage >systems, I >> believe you will likely find that you get large write-performance >gains by: >> >> 1. use the largest possible block size during writes. 512B sectors >are >> the worst-case scenario for any remote storage. Try to write in >chunks of >> *at least* 1 MByte, and it's not unreasonable nowadays to write in >chunks >> of 64MB or larger. The rationale here is that you're spending more >time >> sending data, and less time waiting for ACKs. The more you can tilt >that >> in favor of data, the better off you are. (There are downsides to >huge >> sector/block/chunk sizes, though - this isn't a "free lunch" >scenario. See >> #5.) >> >> 2. relax your write-consistency requirements. If you can tolerate >the >> small risk with "Write Back" you should see better performance, >especially >> during burst writes. During large sequential writes, there are not >many >> ways to violate the laws of physics, and CEPH automatically amplifies >your >> writes by (in your case) a factor of 2x due to replication. >> >> 3. switch to storage devices with the best possible local write >speed, for >> OSDs. OSDs are limited by the performance of the underlying device >or >> virtual device. (e.g. it's totally possible to run OSDs on a >hardware >> RAID6 controller) >> >> 4. Avoid CoW-on-CoW. Write amplification means you'll lose around >50% of >> your IOPS and/or I/O bandwidth for each level of CoW nesting, >depending on >> workload. So don't put CEPH OSDs on, ssy, BTRFS or ZFS filesystems. >A >> worst-case scenario would be something like running a VM using ZFS on >top >> of CEPH, where the OSDs are located on BTRFS filsystems, which are in >turn >> virtual devices hosted on ZFS filesystems. Welcome to 1980's storage >> performance, in that case! (I did it without realizing once... >seriously, >> 5 MBps sequential writes was a good day!) FWIW, CoW filesystems are >> generally awesome - just not when stacked. A sufficiently fast >external >> NAS running ZFS with VMs stored over NFS can provide decent >performance, >> *if* tuned correctly. iX Systems, for example, spends a lot of time >& >> effort making this work well, including some lovely HA NAS >appliances. >> >> 5. Remember the triangle. You can optimize a distributed storage >system >> for any TWO of: a) cost, b) resiliency/reliability/HA, or c) speed. >(This >> is a specific case of the traditional good/fast/cheap:pick-any-2 >adage.) >> >> >> I'm not sure I'm saying anything new here, I may have just summarized >the >> discussion, but the points remain valid. >> >> Good luck with your performance problems. >> -Adam >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user >> >_______________________________________________ >pve-user mailing list >pve-user at pve.proxmox.com >https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user That's a pretty good result. You now have some very small windows where recently-written data could be lost, but in most applications not unreasonably so. In exchange, you get very good throughout for spinning rust. (FWIW, I gave up on CEPH because my nodes only have 2Gbps network each, but I am seeing similar speeds with local ZFS+ZIL+L2ARC on 15k SAS drives. These are older systems, obviously.) Thanks for sharing your solution! -Adam -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From gavin.henry at gmail.com Sun Jul 29 16:50:08 2018 From: gavin.henry at gmail.com (Gavin Henry) Date: Sun, 29 Jul 2018 15:50:08 +0100 Subject: [PVE-User] vzdump causing complete box hang In-Reply-To: References: Message-ID: I notice that I removed this lvm last week and it's back. What's creating it again? The proxmox lvm snapshot tool? vgdisplay /dev/pve/vzsnap-leela-0: read failed after 0 of 4096 at 2212310745088: Input/output error /dev/pve/vzsnap-leela-0: read failed after 0 of 4096 at 2212310802432: Input/output error /dev/pve/vzsnap-leela-0: read failed after 0 of 4096 at 0: Input/output error /dev/pve/vzsnap-leela-0: read failed after 0 of 4096 at 4096: Input/output error --- Volume group --- VG Name pve System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 1038 VG Access read/write VG Status resizable MAX LV 0 Cur LV 4 Open LV 3 Max PV 0 Cur PV 1 Act PV 1 VG Size 2.18 TiB PE Size 4.00 MiB Total PE 571999 Alloc PE / Size 568160 / 2.17 TiB Free PE / Size 3839 / 15.00 GiB VG UUID RDfBSy-RHSF-N0z6-Kxbo-xgFo-JfPn-4nD1tF lvremove /dev/pve/vzsnap-leela-0 /dev/pve/vzsnap-leela-0: read failed after 0 of 4096 at 2212310745088: Input/output error /dev/pve/vzsnap-leela-0: read failed after 0 of 4096 at 2212310802432: Input/output error /dev/pve/vzsnap-leela-0: read failed after 0 of 4096 at 0: Input/output error /dev/pve/vzsnap-leela-0: read failed after 0 of 4096 at 4096: Input/output error Do you really want to remove and DISCARD active logical volume vzsnap-leela-0? [y/n]: y Logical volume "vzsnap-leela-0" successfully removed From gavin.henry at gmail.com Sun Jul 29 23:09:59 2018 From: gavin.henry at gmail.com (Gavin Henry) Date: Sun, 29 Jul 2018 22:09:59 +0100 Subject: [PVE-User] vzdump causing complete box hang In-Reply-To: References: Message-ID: dm-2 is fine. Did a full forced fsck on it. No disk issues either. Very strange. From lyt_yudi at icloud.com Mon Jul 30 04:12:36 2018 From: lyt_yudi at icloud.com (lyt_yudi) Date: Mon, 30 Jul 2018 10:12:36 +0800 Subject: [PVE-User] Can't set password with qm guest passwd Message-ID: Hi, Here use two version?s qemu-guest-agent, it can?t set up password. 1. qemu-guest-agent: "version" : "2.8.0" # qm guest passwd 131 root Enter new password: ********** Retype new password: ********** Use of inherited AUTOLOAD for non-method PVE::API2::Qemu::Agent::agent_cmd() is deprecated at /usr/share/perl5/PVE/API2/Qemu/Agent.pm line 255. Can't locate object method "map_method_by_name" via package "131" (perhaps you forgot to load "131"?) at /usr/share/perl5/PVE/RESTHandler.pm line 326. 2. qemu-guest-agent: "version" : "2.11.1" # qm guest passwd 123 root Enter new password: ********** Retype new password: ********** Use of inherited AUTOLOAD for non-method PVE::API2::Qemu::Agent::agent_cmd() is deprecated at /usr/share/perl5/PVE/API2/Qemu/Agent.pm line 255. Can't locate object method "map_method_by_name" via package "123" (perhaps you forgot to load "123"?) at /usr/share/perl5/PVE/RESTHandler.pm line 326. Are there any related examples? Did anyone succeed? Thanks! From lyt_yudi at icloud.com Mon Jul 30 05:46:02 2018 From: lyt_yudi at icloud.com (lyt_yudi) Date: Mon, 30 Jul 2018 11:46:02 +0800 Subject: [PVE-User] Can't set password with qm guest passwd In-Reply-To: References: Message-ID: <6AA683E8-2A91-4FA8-B3EC-36B131CF3E1F@icloud.com> By pvesh result: # pvesh create nodes/localhost/qemu/131/agent/set-user-password --password test123456 --username root --crypted 1 Use of inherited AUTOLOAD for non-method PVE::API2::Qemu::Agent::agent_cmd() is deprecated at /usr/share/perl5/PVE/API2/Qemu/Agent.pm line 255. Can't locate object method "map_method_by_name" via package "131" (perhaps you forgot to load "131"?) at /usr/share/perl5/PVE/RESTHandler.pm line 326. # pvesh create nodes/localhost/qemu/131/agent/set-user-password --password test123456 --username root --crypted 0 Use of inherited AUTOLOAD for non-method PVE::API2::Qemu::Agent::agent_cmd() is deprecated at /usr/share/perl5/PVE/API2/Qemu/Agent.pm line 255. Can't locate object method "map_method_by_name" via package "131" (perhaps you forgot to load "131"?) at /usr/share/perl5/PVE/RESTHandler.pm line 326. > ? 2018?7?30????10:12?lyt_yudi ??? > > Hi, > > Here use two version?s qemu-guest-agent, it can?t set up password. > > 1. qemu-guest-agent: "version" : "2.8.0" > > # qm guest passwd 131 root > Enter new password: ********** > Retype new password: ********** > Use of inherited AUTOLOAD for non-method PVE::API2::Qemu::Agent::agent_cmd() is deprecated at /usr/share/perl5/PVE/API2/Qemu/Agent.pm line 255. > Can't locate object method "map_method_by_name" via package "131" (perhaps you forgot to load "131"?) at /usr/share/perl5/PVE/RESTHandler.pm line 326. > > 2. qemu-guest-agent: "version" : "2.11.1" > > # qm guest passwd 123 root > Enter new password: ********** > Retype new password: ********** > Use of inherited AUTOLOAD for non-method PVE::API2::Qemu::Agent::agent_cmd() is deprecated at /usr/share/perl5/PVE/API2/Qemu/Agent.pm line 255. > Can't locate object method "map_method_by_name" via package "123" (perhaps you forgot to load "123"?) at /usr/share/perl5/PVE/RESTHandler.pm line 326. > > Are there any related examples? > > Did anyone succeed? > > Thanks! > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From d.csapak at proxmox.com Mon Jul 30 08:25:44 2018 From: d.csapak at proxmox.com (Dominik Csapak) Date: Mon, 30 Jul 2018 08:25:44 +0200 Subject: [PVE-User] Can't set password with qm guest passwd In-Reply-To: <6AA683E8-2A91-4FA8-B3EC-36B131CF3E1F@icloud.com> References: <6AA683E8-2A91-4FA8-B3EC-36B131CF3E1F@icloud.com> Message-ID: <42f09909-50b9-1f68-d078-2cb7e3b0a3a9@proxmox.com> yes there was a perl import missing, i already sent a fix on the devel list, see: https://pve.proxmox.com/pipermail/pve-devel/2018-July/033180.html From t.lamprecht at proxmox.com Mon Jul 30 08:52:45 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 30 Jul 2018 08:52:45 +0200 Subject: [PVE-User] Unable to join cluster In-Reply-To: <7A03D429-52E0-49EA-8419-F96F5E0E02EA@semperen.com> References: <7A03D429-52E0-49EA-8419-F96F5E0E02EA@semperen.com> Message-ID: <6da7b71d-7165-a27d-6e1b-c86713ed2aa6@proxmox.com> (note: re-send as I forgot to hit answer all, thus the list wasn't included) Am 07/27/2018 um 08:45 PM schrieb Eric Germann: > I have two new Proxmox boxes in a virgin cluster. No VM?s, etc. The only thing setup on them is networking. > > I created a cluster on the first one successfully. > > However, I try to join the second to the cluster, I get the following error: > > Starting worker failed: unable to parse worker upid 'UPID:pve02:00001876:00026B6B:5B5B6751:clusterjoin:2001:470:e2fc:10::160:root at pam:' (500) > > The nodes are all IPv4 and IPv6 enabled. The cluster IP as show in the config is IPv6. Is this the issue? > Yes, this could be indeed a bug where our UPID parser cannot handle the encoded IPv6 address... > If I put the v6 address in brackets, same error. > > If I substitute the IPv4 address, I get the following error > > Etablishing API connection with host '172.28.10.160' > TASK ERROR: 401 401 authentication failure > Huh, that's a bit weird, sure you have the correct credentials? > Thoughts? They haven?t been up more than 1 hr when this occurs. > For now you may want to use the 'pvecm add' CLI command with --use_ssh as parameter, with this you should workaround the issue while we take a look at this. cheers, Thomas From lyt_yudi at icloud.com Mon Jul 30 09:15:35 2018 From: lyt_yudi at icloud.com (lyt_yudi) Date: Mon, 30 Jul 2018 15:15:35 +0800 Subject: [PVE-User] Can't set password with qm guest passwd In-Reply-To: <42f09909-50b9-1f68-d078-2cb7e3b0a3a9@proxmox.com> References: <6AA683E8-2A91-4FA8-B3EC-36B131CF3E1F@icloud.com> <42f09909-50b9-1f68-d078-2cb7e3b0a3a9@proxmox.com> Message-ID: > ? 2018?7?30????2:25?Dominik Csapak ??? > > yes there was a perl import missing, i already sent a fix on the devel list, see: > > https://pve.proxmox.com/pipermail/pve-devel/2018-July/033180.html thanks This is a new error # pvesh create nodes/localhost/qemu/131/agent/set-user-password --password test123456 --username root --crypted 0 Can't use string ("password") as a HASH ref while "strict refs" in use at /usr/share/perl5/PVE/QemuServer/Agent.pm line 62. > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From t.lamprecht at proxmox.com Mon Jul 30 10:43:53 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 30 Jul 2018 10:43:53 +0200 Subject: [PVE-User] Can't set password with qm guest passwd In-Reply-To: References: <6AA683E8-2A91-4FA8-B3EC-36B131CF3E1F@icloud.com> <42f09909-50b9-1f68-d078-2cb7e3b0a3a9@proxmox.com> Message-ID: <12deb1fe-c83d-2d36-657d-8164584507d1@proxmox.com> Am 07/30/2018 um 09:15 AM schrieb lyt_yudi: > > >> ? 2018?7?30????2:25?Dominik Csapak ??? >> >> yes there was a perl import missing, i already sent a fix on the devel list, see: >> >> https://pve.proxmox.com/pipermail/pve-devel/2018-July/033180.html > > thanks > > This is a new error > > # pvesh create nodes/localhost/qemu/131/agent/set-user-password --password test123456 --username root --crypted 0 > Can't use string ("password") as a HASH ref while "strict refs" in use at /usr/share/perl5/PVE/QemuServer/Agent.pm line 62. > Ah, yes, I found the issue and pushed a fix. Much thanks for the report. cheers, Thomas From lyt_yudi at icloud.com Mon Jul 30 11:54:54 2018 From: lyt_yudi at icloud.com (lyt_yudi) Date: Mon, 30 Jul 2018 17:54:54 +0800 Subject: [PVE-User] [pve-devel] applied: [PATCH qemu-server] agent: import used check_agent_error method In-Reply-To: <20180730090519.81816-1-t.lamprecht@proxmox.com> References: <20180730090519.81816-1-t.lamprecht@proxmox.com> Message-ID: Hi > ? 2018?7?30????4:43?Thomas Lamprecht ??? > >> # pvesh create nodes/localhost/qemu/131/agent/set-user-password --password test123456 --username root --crypted 0 >> Can't use string ("password") as a HASH ref while "strict refs" in use at /usr/share/perl5/PVE/QemuServer/Agent.pm line 62. > > Ah, yes, I found the issue and pushed a fix. Much thanks for the report. Thanks But with this new patch, report errors: > ? 2018?7?30????5:05?Thomas Lamprecht ??? > > -use PVE::QemuServer::Agent qw(agent_available agent_cmd); > +use PVE::QemuServer::Agent qw(agent_available agent_cmd check_agent_error); > use MIME::Base64 qw(encode_base64 decode_base64); > use JSON; # pvesh create nodes/localhost/qemu/131/agent/set-user-password --password test123456 --username root --crypted 0 Can't use string ("password") as a HASH ref while "strict refs" in use at /usr/share/perl5/PVE/QemuServer/Agent.pm line 62. From lyt_yudi at icloud.com Mon Jul 30 11:57:31 2018 From: lyt_yudi at icloud.com (lyt_yudi) Date: Mon, 30 Jul 2018 17:57:31 +0800 Subject: [PVE-User] [pve-devel] applied: [PATCH qemu-server] agent: import used check_agent_error method In-Reply-To: References: <20180730090519.81816-1-t.lamprecht@proxmox.com> Message-ID: <5CC461CF-0C1F-4DF0-B7BD-ED908A9F19EE@icloud.com> sorry, got it It?s fixed! > ? 2018?7?30????5:54?lyt_yudi ??? > > Hi > > >> ? 2018?7?30????4:43?Thomas Lamprecht ??? >> >>> # pvesh create nodes/localhost/qemu/131/agent/set-user-password --password test123456 --username root --crypted 0 >>> Can't use string ("password") as a HASH ref while "strict refs" in use at /usr/share/perl5/PVE/QemuServer/Agent.pm line 62. >> >> Ah, yes, I found the issue and pushed a fix. Much thanks for the report. > Thanks > > > But with this new patch, report errors: > >> ? 2018?7?30????5:05?Thomas Lamprecht ??? >> >> -use PVE::QemuServer::Agent qw(agent_available agent_cmd); >> +use PVE::QemuServer::Agent qw(agent_available agent_cmd check_agent_error); >> use MIME::Base64 qw(encode_base64 decode_base64); >> use JSON; > > # pvesh create nodes/localhost/qemu/131/agent/set-user-password --password test123456 --username root --crypted 0 > Can't use string ("password") as a HASH ref while "strict refs" in use at /usr/share/perl5/PVE/QemuServer/Agent.pm line 62. > > > _______________________________________________ > pve-user mailing list > pve-user at pve.proxmox.com > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user From t.lamprecht at proxmox.com Mon Jul 30 12:03:35 2018 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 30 Jul 2018 12:03:35 +0200 Subject: [PVE-User] [pve-devel] applied: [PATCH qemu-server] agent: import used check_agent_error method In-Reply-To: <5CC461CF-0C1F-4DF0-B7BD-ED908A9F19EE@icloud.com> References: <20180730090519.81816-1-t.lamprecht@proxmox.com> <5CC461CF-0C1F-4DF0-B7BD-ED908A9F19EE@icloud.com> Message-ID: <7ec1ff62-a3fd-9e43-305c-831680db77c3@proxmox.com> Am 07/30/2018 um 11:57 AM schrieb lyt_yudi: > sorry, got it > > It?s fixed! > great, thanks for reporting and testing again! >> ? 2018?7?30????5:54?lyt_yudi ??? >> >> Hi >> >> >>> ? 2018?7?30????4:43?Thomas Lamprecht ??? >>> >>>> # pvesh create nodes/localhost/qemu/131/agent/set-user-password --password test123456 --username root --crypted 0 >>>> Can't use string ("password") as a HASH ref while "strict refs" in use at /usr/share/perl5/PVE/QemuServer/Agent.pm line 62. >>> >>> Ah, yes, I found the issue and pushed a fix. Much thanks for the report. >> Thanks >> >> >> But with this new patch, report errors: >> >>> ? 2018?7?30????5:05?Thomas Lamprecht ??? >>> >>> -use PVE::QemuServer::Agent qw(agent_available agent_cmd); >>> +use PVE::QemuServer::Agent qw(agent_available agent_cmd check_agent_error); >>> use MIME::Base64 qw(encode_base64 decode_base64); >>> use JSON; >> >> # pvesh create nodes/localhost/qemu/131/agent/set-user-password --password test123456 --username root --crypted 0 >> Can't use string ("password") as a HASH ref while "strict refs" in use at /usr/share/perl5/PVE/QemuServer/Agent.pm line 62. >> >> >> _______________________________________________ >> pve-user mailing list >> pve-user at pve.proxmox.com >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From a.antreich at proxmox.com Tue Jul 31 11:09:54 2018 From: a.antreich at proxmox.com (Alwin Antreich) Date: Tue, 31 Jul 2018 11:09:54 +0200 Subject: [PVE-User] Ceph v12.2.7-pve1 on pvetest Message-ID: <20180731090954.czx2bw42tjo2ps6k@dona.proxmox.com> Hello @all, the ceph luminous v12.2.7-pve1 update has been released on the pvetest repository. https://pve.proxmox.com/wiki/Package_Repositories#_proxmox_ve_test_repository Note to the update: If you used ceph __v12.2.5__ in combination with __erasure coded__ (EC) pools and __v12.2.6__ with bluestore there is a small risk of corruption under certain workloads. For example, OSD service restarts may be a trigger. Aforementioned operation cases are unsupported by Proxmox. See the link below for further upgrade instructions. http://docs.ceph.com/docs/master/releases/luminous/#v12-2-7-luminous -- Cheers, Alwin