From mhill at inett.de Tue Jun 1 13:55:05 2021 From: mhill at inett.de (Maximilian Hill) Date: Tue, 1 Jun 2021 13:55:05 +0200 Subject: [PVE-User] Monitoring Proxmox Backup Server Message-ID: Hello, I'm currently developing a monitoring check_mk monitoring extension for PBS and found myself in this situation: Collecting with proxmox-backup-manager works quite fine, but there are some things, like verify jobs, I can't examine without proxmox-backup-client. proxmox-backup-client on the other hand forces me to authenticate even running on the proxmox backup server when I'm logged in as root via ssh. Is there some way to overcome this since I'm already authenticated as a user of the realm 'pam'? Kind regards, Maximilian Hill -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From proxmox at elchaka.de Sun Jun 13 20:36:01 2021 From: proxmox at elchaka.de (proxmox at elchaka.de) Date: Sun, 13 Jun 2021 20:36:01 +0200 Subject: [PVE-User] PBS backup/restore issue In-Reply-To: References: Message-ID: <8078D056-30A0-4CE9-A117-3269EA5CD331@elchaka.de> Any update on this? I had read an issue with pbs and restore of vms stored on ceph. Which is hopefully solved in actual verion of pbs and qemu... Kind regards Mehmet Am 24. Februar 2021 11:06:14 MEZ schrieb Eneko Lacunza via pve-user : >_______________________________________________ >pve-user mailing list >pve-user at lists.proxmox.com >https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user From proxmox at elchaka.de Sun Jun 13 20:36:01 2021 From: proxmox at elchaka.de (proxmox at elchaka.de) Date: Sun, 13 Jun 2021 20:36:01 +0200 Subject: [PVE-User] PBS backup/restore issue In-Reply-To: References: Message-ID: <8078D056-30A0-4CE9-A117-3269EA5CD331@elchaka.de> Any update on this? I had read an issue with pbs and restore of vms stored on ceph. Which is hopefully solved in actual verion of pbs and qemu... Kind regards Mehmet Am 24. Februar 2021 11:06:14 MEZ schrieb Eneko Lacunza via pve-user : >_______________________________________________ >pve-user mailing list >pve-user at lists.proxmox.com >https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user From proxmox at elchaka.de Sun Jun 13 20:36:01 2021 From: proxmox at elchaka.de (proxmox at elchaka.de) Date: Sun, 13 Jun 2021 20:36:01 +0200 Subject: [PVE-User] PBS backup/restore issue In-Reply-To: References: Message-ID: <8078D056-30A0-4CE9-A117-3269EA5CD331@elchaka.de> Any update on this? I had read an issue with pbs and restore of vms stored on ceph. Which is hopefully solved in actual verion of pbs and qemu... Kind regards Mehmet Am 24. Februar 2021 11:06:14 MEZ schrieb Eneko Lacunza via pve-user : >_______________________________________________ >pve-user mailing list >pve-user at lists.proxmox.com >https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user From elacunza at binovo.es Mon Jun 14 10:08:41 2021 From: elacunza at binovo.es (Eneko Lacunza) Date: Mon, 14 Jun 2021 10:08:41 +0200 Subject: [PVE-User] PBS backup/restore issue In-Reply-To: <8078D056-30A0-4CE9-A117-3269EA5CD331@elchaka.de> References: <8078D056-30A0-4CE9-A117-3269EA5CD331@elchaka.de> Message-ID: <6496c087-e801-f3e6-5361-024fa508af68@binovo.es> Hi, The issue I reported was fixed fastly. We haven't had any issue since. El 13/6/21 a las 20:36, proxmox at elchaka.de escribi?: > Any update on this? > > I had read an issue with pbs and restore of vms stored on ceph. Which > is hopefully solved in actual verion of pbs and qemu... > > Kind regards > Mehmet > > Am 24. Februar 2021 11:06:14 MEZ schrieb Eneko Lacunza via pve-user > : > > ------------------------------------------------------------------------ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > Eneko Lacunza Zuzendari teknikoa | Director t?cnico Binovo IT Human Project Tel. +34 943 569 206 | https://www.binovo.es Astigarragako Bidea, 2 - 2? izda. Oficina 10-11, 20180 Oiartzun https://www.youtube.com/user/CANALBINOVO https://www.linkedin.com/company/37269706/ From humbertos at ifsc.edu.br Mon Jun 14 22:50:27 2021 From: humbertos at ifsc.edu.br (Humberto Jose de Sousa) Date: Mon, 14 Jun 2021 17:50:27 -0300 Subject: Problem with OSD and rrdcached (I think) Message-ID: Hi everyone. I have a cluster with proxmox in the last version (pve-manager/6.4-8/185e14db (running kernel: 5.4.119-1-pve)). One of the nodes did not start your 3 OSDs. I tried debugging without success. At last I tried to destroy and recreate the OSDs. When I tried to recreate it, I got a lot of errors from rrdcached: https://pastebin.com/mg1wfyD3 Can anyone help me? Thanks. Humberto. From contact+dev at gilouweb.com Tue Jun 15 21:34:42 2021 From: contact+dev at gilouweb.com (Gilles Pietri) Date: Tue, 15 Jun 2021 21:34:42 +0200 Subject: [PVE-User] Nested KVM on AMD EPYC processor, oops Message-ID: <38cc69ca-b0e9-6ee1-afb6-86a19c0db098@gilouweb.com> Hi, I'm running qemu (through openstack) on a proxmox instance running: # pveversion -v proxmox-ve: 6.4-1 (running kernel: 5.4.119-1-pve) pve-manager: 6.4-8 (running version: 6.4-8/185e14db) pve-kernel-5.4: 6.4-3 pve-kernel-helper: 6.4-3 [...] pve-qemu-kvm: 5.2.0-6 qemu-server: 6.4-2 The qemu version in Openstack (Wallaby) is $ qemu-system-x86_64 -version QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.16) Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers The VM is running using the host CPU, which is AMD EPYC 7451 24-Core with relevant parameters here: cores: 8 cpu: host memory: 32768 numa: 0 rng0: source=/dev/urandom sockets: 1 And I get spammed a lot with that kind of traces: [Tue Jun 15 19:13:29 2021] ------------[ cut here ]------------ [Tue Jun 15 19:13:29 2021] WARNING: CPU: 6 PID: 47530 at arch/x86/kvm/mmu.c:2250 nonpaging_update_pte+0x9/0x10 [kvm] [Tue Jun 15 19:13:30 2021] Modules linked in: xt_nat nf_conntrack_netlink tcp_diag inet_diag xt_MASQUERADE xfrm_user iptable_nat nf_nat overlay binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache sctp veth ebt_arp ebtable_filter ebtables ip6table_raw ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tables iptable_raw xt_mac ipt_REJECT nf_reject_ipv4 xt_mark xt_set xt_physdev xt_addrtype xt_comment xt_multiport xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp ip_set_hash_net ip_set iptable_filter bpfilter softdog nfnetlink_log nfnetlink ipmi_ssif amd64_edac_mod edac_mce_amd kvm_amd kvm drm_vram_helper irqbypass ttm crct10dif_pclmul crc32_pclmul drm_kms_helper ghash_clmulni_intel drm aesni_intel i2c_algo_bit crypto_simd fb_sys_fops cryptd syscopyarea sysfillrect glue_helper k10temp sysimgblt ccp ipmi_si ipmi_devintf ipmi_msghandler mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser rdma_cm iw_cm ib_cm [Tue Jun 15 19:13:30 2021] ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi sunrpc ip_tables x_tables autofs4 dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear ixgbe ahci xhci_pci xfrm_algo raid1 i2c_piix4 dca libahci xhci_hcd mdio [Tue Jun 15 19:13:30 2021] CPU: 6 PID: 47530 Comm: kvm Tainted: P W O 5.4.119-1-pve #1 [Tue Jun 15 19:13:30 2021] Hardware name: empty empty/S8026GM2NRE-HOV-B, BIOS V8.711 07/09/2020 [Tue Jun 15 19:13:30 2021] RIP: 0010:nonpaging_update_pte+0x9/0x10 [kvm] [Tue Jun 15 19:13:30 2021] Code: 00 0f 1f 44 00 00 55 31 c0 48 89 e5 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 5d c3 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 5d c3 0f 1f 00 0f 1f 44 00 00 31 f6 48 8b 04 77 48 63 54 37 [Tue Jun 15 19:13:30 2021] RSP: 0018:ffffb904e4c7ba78 EFLAGS: 00010202 [Tue Jun 15 19:13:30 2021] RAX: ffffffffc0dc0500 RBX: 0000000000000701 RCX: ffffb904e4c7bac0 [Tue Jun 15 19:13:30 2021] RDX: ffff909fa2bd0000 RSI: ffff908f2fe61e30 RDI: ffff90a63593bca0 [Tue Jun 15 19:13:30 2021] RBP: ffffb904e4c7ba78 R08: 000000000054a7ae R09: ffff909fa2bd0000 [Tue Jun 15 19:13:30 2021] R10: 0000000000000000 R11: 0000000000001970 R12: ffff90a63593bca0 [Tue Jun 15 19:13:30 2021] R13: 0000000000000000 R14: ffff909fa2bd0000 R15: ffffb904e4c7bac8 [Tue Jun 15 19:13:30 2021] FS: 00007f85da5fc700(0000) GS:ffff9098ef800000(0000) knlGS:0000000000000000 [Tue Jun 15 19:13:30 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Tue Jun 15 19:13:30 2021] CR2: 000000c420563010 CR3: 0000001fe1698000 CR4: 00000000003406e0 [Tue Jun 15 19:13:30 2021] Call Trace: [Tue Jun 15 19:13:30 2021] kvm_mmu_pte_write+0x421/0x430 [kvm] [Tue Jun 15 19:13:30 2021] kvm_page_track_write+0x82/0xc0 [kvm] [Tue Jun 15 19:13:30 2021] emulator_write_phys+0x3b/0x50 [kvm] [Tue Jun 15 19:13:30 2021] write_emulate+0xe/0x10 [kvm] [Tue Jun 15 19:13:30 2021] emulator_read_write_onepage+0xfc/0x320 [kvm] [Tue Jun 15 19:13:30 2021] emulator_read_write+0xd6/0x190 [kvm] [Tue Jun 15 19:13:30 2021] emulator_write_emulated+0x15/0x20 [kvm] [Tue Jun 15 19:13:30 2021] segmented_write+0x5d/0x80 [kvm] [Tue Jun 15 19:13:30 2021] writeback+0x203/0x2e0 [kvm] [Tue Jun 15 19:13:30 2021] x86_emulate_insn+0x990/0x1050 [kvm] [Tue Jun 15 19:13:30 2021] x86_emulate_instruction+0x350/0x710 [kvm] [Tue Jun 15 19:13:30 2021] complete_emulated_pio+0x3f/0x70 [kvm] [Tue Jun 15 19:13:30 2021] kvm_arch_vcpu_ioctl_run+0x4cb/0x570 [kvm] [Tue Jun 15 19:13:30 2021] kvm_vcpu_ioctl+0x24b/0x610 [kvm] [Tue Jun 15 19:13:30 2021] do_vfs_ioctl+0xa9/0x640 [Tue Jun 15 19:13:30 2021] ? task_numa_work+0x228/0x300 [Tue Jun 15 19:13:30 2021] ksys_ioctl+0x67/0x90 [Tue Jun 15 19:13:30 2021] __x64_sys_ioctl+0x1a/0x20 [Tue Jun 15 19:13:30 2021] do_syscall_64+0x57/0x190 [Tue Jun 15 19:13:30 2021] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [Tue Jun 15 19:13:30 2021] RIP: 0033:0x7f8df6dec427 [Tue Jun 15 19:13:30 2021] Code: 00 00 90 48 8b 05 69 aa 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 aa 0c 00 f7 d8 64 89 01 48 [Tue Jun 15 19:13:30 2021] RSP: 002b:00007f85da5f8c08 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [Tue Jun 15 19:13:30 2021] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: 00007f8df6dec427 [Tue Jun 15 19:13:30 2021] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000022 [Tue Jun 15 19:13:30 2021] RBP: 0000000000000000 R08: 000055be0e527f58 R09: 0000000000000000 [Tue Jun 15 19:13:30 2021] R10: 0000000000000001 R11: 0000000000000246 R12: 000055be0f426eb0 [Tue Jun 15 19:13:30 2021] R13: 00007f8dea1f5000 R14: 0000000000000000 R15: 000055be0f426eb0 [Tue Jun 15 19:13:30 2021] ---[ end trace 4e3f65d27e26463c ]--- I'm guessing this is more of a qemu bug / issue, but it does have a lot of impact on the performance of nested VMs, though it does not crash. I was wondering if any of proxmox users noticed that with these CPUs / versions before going upstream? Regards, Gilles From marcosnegrini at yahoo.com.ar Thu Jun 17 16:47:11 2021 From: marcosnegrini at yahoo.com.ar (marcos negrini) Date: Thu, 17 Jun 2021 14:47:11 +0000 (UTC) Subject: [PVE-User] memory management of a vm in HA References: <1375274439.1944470.1623941231996.ref@mail.yahoo.com> Message-ID: <1375274439.1944470.1623941231996@mail.yahoo.com> Hello:I am administrator of a proxmox cluster and I have been testing High Availability, and I am very satisfied with the performance; but I wanted to understand a little more in depth how the memory management works. I did the tests in a cluster of 3 servers with a SAN storage, I tried to cut the power to a physical server and almost instantly the vm's that were configured with HA went to the next node, my question is, how do you manage the memory of each vm's? do you pre-share it in the other physical servers so that the memory status of each one is not lost? how do you manage the loss of the information that was not copied? is there any technical document of this implementation in proxmox?Regardspd: sorry for my english level, I hope my doubt is interpreted. From Christoph.Weber at xpecto.com Fri Jun 18 08:25:54 2021 From: Christoph.Weber at xpecto.com (Christoph Weber) Date: Fri, 18 Jun 2021 06:25:54 +0000 Subject: [PVE-User] WG: Problem with OSD and rrdcached (I think) In-Reply-To: References: <1c71de3e503c4fc38b2f1965d6670264@xpecto.com> Message-ID: > At last I tried to destroy and recreate the OSDs. When I tried to > recreate it, I got a lot of errors from rrdcached: I think the relevant error line is this the one before the rrdcached messages: stderr: /usr/bin/ceph-osd: error while loading shared libraries: 00881:353989632:757542912 Seems there is a shared library missing for the ceph installation which is required bei the ceph-osd binary. Try to call it directly to see more ( -h for help): /usr/bin/ceph-osd -h The rrdcache errors afterwards are either irrelevant, have the same origin or at least a similar problem. Maybe you need to fsck your root filesystem and see if apt shows any errors. apt-get check regards christoph From f.gruenbichler at proxmox.com Fri Jun 18 08:44:08 2021 From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=) Date: Fri, 18 Jun 2021 08:44:08 +0200 Subject: [PVE-User] memory management of a vm in HA In-Reply-To: <1375274439.1944470.1623941231996@mail.yahoo.com> References: <1375274439.1944470.1623941231996.ref@mail.yahoo.com> <1375274439.1944470.1623941231996@mail.yahoo.com> Message-ID: <1623998195.h71kiadsrn.astroid@nora.none> On June 17, 2021 4:47 pm, marcos negrini wrote: > Hello:I am administrator of a proxmox cluster and I have been testing High Availability, and I am very satisfied with the performance; but I wanted to understand a little more in depth how the memory management works. I did the tests in a cluster of 3 servers with a SAN storage, I tried to cut the power to a physical server and almost instantly the vm's that were configured with HA went to the next node, my question is, how do you manage the memory of each vm's? do you pre-share it in the other physical servers so that the memory status of each one is not lost? how do you manage the loss of the information that was not copied? is there any technical document of this implementation in proxmox?Regardspd: sorry for my english level, I hope my doubt is interpreted. I'd suggest reading [1] as a starting point. To answer your questions: - guest memory is not replicated or shared between nodes, HA just tries to ensure the guest is running "somewhere" according to the HA configuration - ideally your guests' volumes are on shared storage, but if you can live with losing data since the last replication, ZFS with replication can also be an option - if a node disappears/crashes/loses quorum/.. it gets fenced, the still quorate part of the cluster will notice and "steal" the affected HA resources -- if the fenced node is still responsive, it's watchdog timer will expire and it will shutdown (stopping all running guests in the process) -- the stealing node will wait a certain amount of time to give the fenced node time to be completely fenced, then it will take over the guest configs and start the guest - additionally, you can configure what should happen to HA resources on (orderly) node shutdown/reboot (see "Node Maintenance" in the admin guide) - here one of the options is to migrate them to other nodes, which is possibly what you triggered in your test? 1: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_ha_manager From f.cuseo at panservice.it Fri Jun 18 15:46:29 2021 From: f.cuseo at panservice.it (Fabrizio Cuseo) Date: Fri, 18 Jun 2021 15:46:29 +0200 (CEST) Subject: [PVE-User] Live restore & Vlan Config In-Reply-To: <0f85eeee-3ebe-3e91-e32d-7831ea8999f4@gmail.com> References: <4817f902-0e8b-5ecc-fc0d-8bfccc255bd8@proxmox.com> <0f85eeee-3ebe-3e91-e32d-7831ea8999f4@gmail.com> Message-ID: <2043695330.93705.1624023988967.JavaMail.zimbra@zimbra.panservice.it> Hello. Do you know if there is the possibility to change vlan id (or some other VM configuration data) before starting a live restore ? It will be useful to make some test from backup without waiting all the time needed for the traditional restore. Regards, Fabrizio -- --- Fabrizio Cuseo - mailto:f.cuseo at panservice.it Direzione Generale - Panservice InterNetWorking Servizi Professionali per Internet ed il Networking Panservice e' associata AIIP - RIPE Local Registry Phone: +39 0773 410020 - Fax: +39 0773 470219 http://www.panservice.it mailto:info at panservice.it Numero verde nazionale: 800 901492 From leandro at tecnetmza.com.ar Fri Jun 18 16:21:12 2021 From: leandro at tecnetmza.com.ar (Leandro Roggerone) Date: Fri, 18 Jun 2021 11:21:12 -0300 Subject: [PVE-User] my proxmox stoped creating backups. Message-ID: Hi guys. Im not sure what have changed on my platform but noticed that some days ago scheduled backups fails with error status. I tried create vm backup manually but it also fails. It is a critical VM in production environment. This is my output: INFO: starting new backup job: vzdump 106 --compress lzo --storage local --mode snapshot --remove 0 --node pve INFO: Starting Backup of VM 106 (qemu) INFO: Backup started at 2021-06-18 11:00:17 INFO: status = running INFO: update VM 106: -lock backup INFO: VM Name: MonitorVM INFO: include disk 'scsi0' 'local-lvm:vm-106-disk-0' 200G INFO: backup mode: snapshot INFO: ionice priority: 7 INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-106-2021_06_18-11_00_17.vma.lzo' ERROR: got timeout INFO: aborting backup job ERROR: interrupted by signal ERROR: Backup of VM 106 failed - got timeout INFO: Failed at 2021-06-18 11:01:54 INFO: Backup job finished with errors TASK ERROR: job errors I found some links mentioning this issue but no solution yet. Any ideas ? Libre de virus. www.avast.com <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> From humbertos at ifsc.edu.br Fri Jun 18 22:58:36 2021 From: humbertos at ifsc.edu.br (Humberto Jose de Sousa) Date: Fri, 18 Jun 2021 17:58:36 -0300 Subject: [PVE-User] WG: Problem with OSD and rrdcached (I think) In-Reply-To: References: <1c71de3e503c4fc38b2f1965d6670264@xpecto.com> Message-ID: Thanks a lot Christoph! I was looking for the wrong error. I ran the command "apt install ceph-osd --reinstall" and solved the problem. Regards, Humberto. Em sex., 18 de jun. de 2021 ?s 03:26, Christoph Weber < Christoph.Weber at xpecto.com> escreveu: > > At last I tried to destroy and recreate the OSDs. When I tried to > > recreate it, I got a lot of errors from rrdcached: > > I think the relevant error line is this the one before the rrdcached > messages: > > stderr: /usr/bin/ceph-osd: error while loading shared libraries: > 00881:353989632:757542912 > > Seems there is a shared library missing for the ceph installation which is > required bei the ceph-osd binary. > > Try to call it directly to see more ( -h for help): > > /usr/bin/ceph-osd -h > > > The rrdcache errors afterwards are either irrelevant, have the same origin > or at least a similar problem. > > Maybe you need to fsck your root filesystem and see if apt shows any > errors. > > apt-get check > > regards > christoph > > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > > From mhill at inett.de Mon Jun 21 09:56:55 2021 From: mhill at inett.de (Maximilian Hill) Date: Mon, 21 Jun 2021 09:56:55 +0200 Subject: [PVE-User] PBS Remote Sync http request timeout Message-ID: Hello, from time to time we experience some failed Sync Jobs exiting with: TASK ERROR: Failed to retrieve backup groups from remote - http request timed out Triggering it manually afterwards succeeds. Version of both Proxmox Backup Servers: 1.1-9 Has someone experienced that before? For now I just have a wild guess: Because the storage isn't the fastest and backups were made shortly before, it might take so mutch time to gather metadata, that the request times out. Is there a way to adjust the timeout? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: From s.reiter at proxmox.com Mon Jun 21 10:20:13 2021 From: s.reiter at proxmox.com (Stefan Reiter) Date: Mon, 21 Jun 2021 10:20:13 +0200 Subject: [PVE-User] Live restore & Vlan Config In-Reply-To: <2043695330.93705.1624023988967.JavaMail.zimbra@zimbra.panservice.it> References: <4817f902-0e8b-5ecc-fc0d-8bfccc255bd8@proxmox.com> <0f85eeee-3ebe-3e91-e32d-7831ea8999f4@gmail.com> <2043695330.93705.1624023988967.JavaMail.zimbra@zimbra.panservice.it> Message-ID: <34dde7c2-04b5-cacc-6d7e-f9f3792ed4fb@proxmox.com> On 6/18/21 3:46 PM, Fabrizio Cuseo wrote: > Hello. > Do you know if there is the possibility to change vlan id (or some other VM configuration data) before starting a live restore ? It will be useful to make some test from backup without waiting all the time needed for the traditional restore. > Hi, at the moment this is not possible, you have to use a regular restore and change the config before starting. This is something we're considering adding in the future though. See also this forum thread: https://forum.proxmox.com/threads/error-occured-during-live-restore-max-8-vcpus-allowed-per-vm-on-this-node.88472/ ~ Stefan > Regards, Fabrizio > > > > From proxmox at elchaka.de Wed Jun 23 20:50:18 2021 From: proxmox at elchaka.de (proxmox at elchaka.de) Date: Wed, 23 Jun 2021 20:50:18 +0200 Subject: [PVE-User] E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. In-Reply-To: References: Message-ID: <740EEB9E-D4FC-4C8C-A63C-DEEBB5D20483@elchaka.de> Did you tried something like apt -f install Hth Mehmet Am 20. Mai 2021 15:00:44 MESZ schrieb Hongyi Zhao : >The pve 6.3-1 on my machine has been in use for some time. Recently, >due to unexpected power failure, the apt package management system was >corrupted with the following error when I install any package, say, >the following: > >root at pve:~# apt install inxi >E: dpkg was interrupted, you must manually run 'dpkg --configure -a' >to correct the problem. > >I tried to run 'dpkg --configure -a' as suggested above, but it didn't >work at all. > >Are there any suggestions to solve this problem? > >Regards, >-- >Assoc. Prof. Hongyi Zhao >Theory and Simulation of Materials >Hebei Vocational University of Technology and Engineering >NO. 552 North Gangtie Road, Xingtai, China > >_______________________________________________ >pve-user mailing list >pve-user at lists.proxmox.com >https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user From hongyi.zhao at gmail.com Thu Jun 24 01:27:47 2021 From: hongyi.zhao at gmail.com (Hongyi Zhao) Date: Thu, 24 Jun 2021 07:27:47 +0800 Subject: [PVE-User] E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. In-Reply-To: <740EEB9E-D4FC-4C8C-A63C-DEEBB5D20483@elchaka.de> References: <740EEB9E-D4FC-4C8C-A63C-DEEBB5D20483@elchaka.de> Message-ID: On Thu, Jun 24, 2021 at 2:50 AM wrote: > > Did you tried something like > > apt -f install Tried by failed. See the screenshot in attachment. HY > > Hth > Mehmet > > Am 20. Mai 2021 15:00:44 MESZ schrieb Hongyi Zhao : >> >> The pve 6.3-1 on my machine has been in use for some time. Recently, >> due to unexpected power failure, the apt package management system was >> corrupted with the following error when I install any package, say, >> the following: >> >> root at pve:~# apt install inxi >> E: dpkg was interrupted, you must manually run 'dpkg --configure -a' >> to correct the problem. >> >> I tried to run 'dpkg --configure -a' as suggested above, but it didn't >> work at all. >> >> Are there any suggestions to solve this problem? >> >> Regards, -- Assoc. Prof. Hongyi Zhao Theory and Simulation of Materials Hebei Vocational University of Technology and Engineering NO. 552 North Gangtie Road, Xingtai, China From asterisk at raouf.net Thu Jun 24 14:02:05 2021 From: asterisk at raouf.net (Faris Raouf) Date: Thu, 24 Jun 2021 13:02:05 +0100 Subject: [PVE-User] E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. In-Reply-To: References: <740EEB9E-D4FC-4C8C-A63C-DEEBB5D20483@elchaka.de> Message-ID: <008001d768f0$c02503d0$406f0b70$@raouf.net> =========== >>On Thu, Jun 24, 2021 at 2:50 AM wrote: >> >> Did you tried something like >> >> apt -f install > >Tried by failed. See the screenshot in attachment. > >HY =========== (sorry I'm using Outlook and not configured correctly for mailing list replies) If you are using the console within Proxmox, maybe try connecting via ssh instead. I know that sounds crazy, but I had similar issues on PBS (as opposed to proxmox) where dpkg --configure -a caused the console to sort of crash or freeze and the command didn't actually complete. Doing it via ssh instead resolved the problem. I can't see your screen shots, and you mentioned "doesn't work" as opposed to "console does something strange" so maybe this is of no help whatsoever. From martin at proxmox.com Thu Jun 24 15:16:26 2021 From: martin at proxmox.com (Martin Maurer) Date: Thu, 24 Jun 2021 15:16:26 +0200 Subject: [PVE-User] Proxmox VE 7.0 (beta) released! Message-ID: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> Hi all, We are pleased to announce the first beta release of Proxmox Virtual Environment 7.0! The 7.x family is based on the great Debian 11 "Bullseye" and comes with a 5.11 kernel, QEMU 6.0, LXC 4.0, OpenZFS 2.0.4. Note: The current release of Proxmox Virtual Environment 7.0 is a beta version. If you test or upgrade, make sure to first create backups of your data. We recommend https://www.proxmox.com/en/proxmox-backup-server to do so. Here are some of the highlights of the Proxmox VE 7.0 beta version: - Ceph Server: Ceph Pacific 16.2 is the new default. Ceph Octopus 15.2 comes with continued support. - BTRFS: modern copy on write file system natively supported by the Linux kernel, implementing features such as snapshots, built-in RAID, and self healing via checksums for data and metadata. - ifupdown2 is the default for new installations using the Proxmox VE official ISO. - QEMU 6.0 has support for io_uring as asynchronous I/O engine for virtual drives - this is now the default for newly started or migrated guests. - Countless GUI improvements - and much more... Release notes https://pve.proxmox.com/wiki/Roadmap Download[ http://download.proxmox.com/iso Community Forum https://forum.proxmox.com Bugtracker https://bugzilla.proxmox.com Source code https://git.proxmox.com FAQ Q: Can I upgrade Proxmox VE 6.4 to 7.0 beta with apt? A: Yes, please follow the upgrade instructions on https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0 Q: Can I upgrade a 7.0 beta installation to the stable 7.0 release via apt? A: Yes, upgrading from beta to stable installation will be possible via apt. Q: Which apt repository can I use for Proxmox VE 7.0 beta? A: deb http://download.proxmox.com/debian/pve bullseye pvetest Q: Can I install Proxmox VE 7.0 beta on top of Debian 11 "Bullseye"? A: Yes. Q: Can I upgrade my Proxmox VE 6.4 cluster with Ceph Octopus to 7.0 beta? A: This is a two step process. First, you have to upgrade Proxmox VE from 6.4 to 7.0, and afterwards upgrade Ceph from Octopus to Pacific. There are a lot of improvements and changes, so please follow exactly the upgrade documentation: https://pve.proxmox.com/wiki/Upgrade_from_6.x_to_7.0 https://pve.proxmox.com/wiki/Ceph_Octopus_to_Pacific Q: When do you expect the stable Proxmox VE 7.0 release? A: The final Proxmox VE 7.0 will be available as soon as all Proxmox VE 7.0 release critical bugs are fixed. Q: Where can I get more information about feature updates? A: Check the https://pve.proxmox.com/wiki/Roadmap, https://forum.proxmox.com, the https://lists.proxmox.com/cgi-bin/mailman/listinfo, and/or subscribe to our https://www.proxmox.com/en/news. You are welcome to test your hardware and your upgrade path and we are looking forward to your feedback, bug reports, or ideas. Thank you for getting involved! -- Best Regards, Martin Maurer martin at proxmox.com https://www.proxmox.com ____________________________________________________________________ Proxmox Server Solutions GmbH Br?uhausgasse 37, 1050 Vienna, Austria Commercial register no.: FN 258879 f Registration office: Handelsgericht Wien From hongyi.zhao at gmail.com Thu Jun 24 15:41:35 2021 From: hongyi.zhao at gmail.com (Hongyi Zhao) Date: Thu, 24 Jun 2021 21:41:35 +0800 Subject: [PVE-User] E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. In-Reply-To: References: <740EEB9E-D4FC-4C8C-A63C-DEEBB5D20483@elchaka.de> Message-ID: On Thu, Jun 24, 2021 at 8:11 PM Faris Raouf via pve-user wrote: > > > > > ---------- Forwarded message ---------- > From: Faris Raouf > To: "'Proxmox VE user list'" > Cc: > Bcc: > Date: Thu, 24 Jun 2021 13:02:05 +0100 > Subject: RE: [PVE-User] E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. > =========== > >>On Thu, Jun 24, 2021 at 2:50 AM wrote: > >> > >> Did you tried something like > >> > >> apt -f install > > > >Tried by failed. See the screenshot in attachment. > > > >HY > =========== > > (sorry I'm using Outlook and not configured correctly for mailing list > replies) > > If you are using the console within Proxmox, maybe try connecting via ssh > instead. > > I know that sounds crazy, but I had similar issues on PBS (as opposed to > proxmox) where dpkg --configure -a caused the console to sort of crash or > freeze and the command didn't actually complete. > > Doing it via ssh instead resolved the problem. > > I can't see your screen shots, and you mentioned "doesn't work" as opposed > to "console does something strange" so maybe this is of no help whatsoever. See the following info seen from the console/shell within Proxmox web UI: root at pve:~# apt -f install E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. And the following info seen under ssh terminal: werner at X10DAi:~$ ssh root at 192.168.10.254 The authenticity of host '192.168.10.254 (192.168.10.254)' can't be established. ECDSA key fingerprint is SHA256:s6hATbCfE+vH/h2HBl2RcKJM4xHwjfXiF5mt/c8igog. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.10.254' (ECDSA) to the list of known hosts. root at 192.168.10.254's password: Linux pve 5.4.73-1-pve #1 SMP PVE 5.4.73-1 (Mon, 16 Nov 2020 10:52:16 +0100) x86_64 The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Last login: Thu Jun 24 21:36:35 2021 root at pve:~# apt -f install E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. root at pve:~# HY > > > > ---------- Forwarded message ---------- > From: Faris Raouf via pve-user > To: "'Proxmox VE user list'" > Cc: Faris Raouf > Bcc: > Date: Thu, 24 Jun 2021 13:02:05 +0100 > Subject: Re: [PVE-User] E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user -- Assoc. Prof. Hongyi Zhao Theory and Simulation of Materials Hebei Vocational University of Technology and Engineering NO. 552 North Gangtie Road, Xingtai, China From ralf.storm at konzept-is.de Thu Jun 24 16:08:06 2021 From: ralf.storm at konzept-is.de (Ralf Storm) Date: Thu, 24 Jun 2021 16:08:06 +0200 Subject: [PVE-User] Error updating Ceph from Nautilus to Octopus In-Reply-To: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> References: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> Message-ID: <087e4881-4b9c-791f-77d6-30f826152bd5@konzept-is.de> Hello, I upgraded my nodes according to the instructions in the Proxmox wiki. After restarting the monitors and managers i restarted the first 2 osds and saw the following error in syslog: fsck warning: #6:fc6c85f8:::rbd_header.08a3e84bf25fd0:head# has omap that is not per-pool or pgmeta I am unsure if this is bad or not - anyone had this before or knows if it is a problem? With best regards Ralf Storm From elacunza at binovo.es Thu Jun 24 16:30:31 2021 From: elacunza at binovo.es (Eneko Lacunza) Date: Thu, 24 Jun 2021 16:30:31 +0200 Subject: BIG cluster questions Message-ID: Hi all, We're currently helping a customer to configure a virtualization cluster with 88 servers for VDI. Right know we're testing the feasibility of building just one Proxmox cluster of 88 nodes. A 4-node cluster has been configured too for comparing both (same server and networking/racks). Nodes have 2 NICs 2x25Gbps each. Currently there are two LACP bonds configured (one for each NIC); one for storage (NFS v4.2) and the other for the rest (VMs, cluster). Cluster has two rings, one on each bond. - With clusters at rest (no significant number of VMs running), we see quite a different corosync/knet latency average on our 88 node cluster (~300-400) and our 4-node cluster (<100). For 88-node cluster: - Creating some VMs (let's say 16), one each 30s, works well. - Destroying some VMs (let's say 16), one each 30s, outputs error messages (storage cfs lock related) and fails removing some of the VMs. - Rebooting 32 nodes, one each 30 seconds (boot for a node is about 120s) so that no quorum is lost, creates a cluster traffic "flood". Some of the rebooted nodes don't rejoin the cluster, and WUI shows all nodes in cluster quorum with a grey ?, instead of green OK. In this situation corosying latency in some nodes can skyrocket to 10s or 100s times the values before the reboots. Access to pmxcfs is very slow and we have been able to fix the issue only rebooting all nodes. - We have tried changing the transport of knet in a ring from UDP to SCTP as reported here: https://forum.proxmox.com/threads/proxmox-6-2-corosync-3-rare-and-spontaneous-disruptive-udp-5405-storm-flood.75871/page-2 that gives better latencies for corosync, but the reboot issue continues. We don't know whether both issues are related or not. Could LACP bonds be the issue? https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration " If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using the corresponding bonding mode (802.3ad). Otherwise you should generally use the active-backup mode. If you intend to run your cluster network on the bonding interfaces, then you have to use active-passive mode on the bonding interfaces, other modes are unsupported. " As per second line, we understand that running cluster networking over a LACP bond is not supported (just to confirm our interpretation)? We're in the process of reconfiguring nodes/switches to test without a bond, to see if that gives us a stable cluster (will report on this). Do you think this could be the issue? Now for more general questions; do you think a 88-node Proxmox VE cluster is feasible? Those 88 nodes will host about 14.000 VMs. Will HA manager be able to manage them, or are they too many? (HA for those VMs doesn't seem to be a requirement right know). Thanks a lot Eneko EnekoLacunza CTO | Zuzendari teknikoa Binovo IT Human Project 943 569 206 elacunza at binovo.es binovo.es Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun youtube linkedin From ralf.storm at konzept-is.de Thu Jun 24 16:52:47 2021 From: ralf.storm at konzept-is.de (Ralf Storm) Date: Thu, 24 Jun 2021 16:52:47 +0200 Subject: [PVE-User] Problem with wrong node status in manager_status Message-ID: <579943a8-d47a-d930-d78c-0572b745f68b@konzept-is.de> Hello, I have 7 node cluster and all seems to work fine, but it is impossible to migrate ha-vms to one node, non ha is possible the problem is that its status is "fence" also it is online on all views in ha view and in pvecm status it is online everywhere but not when i look in the filesystem under /etc/pve/ha/manager status What I already tried is to restart all crm services on all nodes, the nodes were also updatet and rebooted several times, but the "fence" status dos not go away, haing this problem for weeks now. any suggestion? Best regards Ralf Storm From contact+dev at gilouweb.com Thu Jun 24 16:58:46 2021 From: contact+dev at gilouweb.com (Gilles Pietri) Date: Thu, 24 Jun 2021 16:58:46 +0200 Subject: [PVE-User] Nested KVM on AMD EPYC processor, oops In-Reply-To: <38cc69ca-b0e9-6ee1-afb6-86a19c0db098@gilouweb.com> References: <38cc69ca-b0e9-6ee1-afb6-86a19c0db098@gilouweb.com> Message-ID: <08400e55-6f85-ae8b-6363-8279ae2be279@gilouweb.com> Le 15/06/2021 ? 21:34, Gilles Pietri a ?crit?: > Hi, > > I'm running qemu (through openstack) on a proxmox instance running: > # pveversion -v > proxmox-ve: 6.4-1 (running kernel: 5.4.119-1-pve) > pve-manager: 6.4-8 (running version: 6.4-8/185e14db) > pve-kernel-5.4: 6.4-3 > pve-kernel-helper: 6.4-3 > [...] > pve-qemu-kvm: 5.2.0-6 > qemu-server: 6.4-2 Hi, I switched to pve-kernel 5.11 (7.0-2~bpo10), as suggested on #qemu, and nested virt works nicely! Regards, Gilles > > > The qemu version in Openstack (Wallaby) is > $ qemu-system-x86_64 -version > QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.16) > Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers > > The VM is running using the host CPU, which is AMD EPYC 7451 24-Core > with relevant parameters here: > cores: 8 > cpu: host > memory: 32768 > numa: 0 > rng0: source=/dev/urandom > sockets: 1 > > > And I get spammed a lot with that kind of traces: > > [Tue Jun 15 19:13:29 2021] ------------[ cut here ]------------ > [Tue Jun 15 19:13:29 2021] WARNING: CPU: 6 PID: 47530 at > arch/x86/kvm/mmu.c:2250 nonpaging_update_pte+0x9/0x10 [kvm] > [Tue Jun 15 19:13:30 2021] Modules linked in: xt_nat > nf_conntrack_netlink tcp_diag inet_diag xt_MASQUERADE xfrm_user > iptable_nat nf_nat overlay binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4 > nfs lockd grace fscache sctp veth ebt_arp ebtable_filter ebtables > ip6table_raw ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6_tables > iptable_raw xt_mac ipt_REJECT nf_reject_ipv4 xt_mark xt_set xt_physdev > xt_addrtype xt_comment xt_multiport xt_conntrack nf_conntrack > nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp ip_set_hash_net ip_set > iptable_filter bpfilter softdog nfnetlink_log nfnetlink ipmi_ssif > amd64_edac_mod edac_mce_amd kvm_amd kvm drm_vram_helper irqbypass ttm > crct10dif_pclmul crc32_pclmul drm_kms_helper ghash_clmulni_intel drm > aesni_intel i2c_algo_bit crypto_simd fb_sys_fops cryptd syscopyarea > sysfillrect glue_helper k10temp sysimgblt ccp ipmi_si ipmi_devintf > ipmi_msghandler mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) > icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost tap ib_iser > rdma_cm iw_cm ib_cm > [Tue Jun 15 19:13:30 2021] ib_core iscsi_tcp libiscsi_tcp libiscsi > scsi_transport_iscsi sunrpc ip_tables x_tables autofs4 dm_thin_pool > dm_persistent_data dm_bio_prison dm_bufio raid10 raid456 > async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq > libcrc32c raid0 multipath linear ixgbe ahci xhci_pci xfrm_algo raid1 > i2c_piix4 dca libahci xhci_hcd mdio > [Tue Jun 15 19:13:30 2021] CPU: 6 PID: 47530 Comm: kvm Tainted: P > W O 5.4.119-1-pve #1 > [Tue Jun 15 19:13:30 2021] Hardware name: empty empty/S8026GM2NRE-HOV-B, > BIOS V8.711 07/09/2020 > [Tue Jun 15 19:13:30 2021] RIP: 0010:nonpaging_update_pte+0x9/0x10 [kvm] > [Tue Jun 15 19:13:30 2021] Code: 00 0f 1f 44 00 00 55 31 c0 48 89 e5 5d > c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 5d c3 0f 1f 44 00 00 0f 1f 44 00 > 00 55 48 89 e5 <0f> 0b 5d c3 0f 1f 00 0f 1f 44 00 00 31 f6 48 8b 04 77 > 48 63 54 37 > [Tue Jun 15 19:13:30 2021] RSP: 0018:ffffb904e4c7ba78 EFLAGS: 00010202 > [Tue Jun 15 19:13:30 2021] RAX: ffffffffc0dc0500 RBX: 0000000000000701 > RCX: ffffb904e4c7bac0 > [Tue Jun 15 19:13:30 2021] RDX: ffff909fa2bd0000 RSI: ffff908f2fe61e30 > RDI: ffff90a63593bca0 > [Tue Jun 15 19:13:30 2021] RBP: ffffb904e4c7ba78 R08: 000000000054a7ae > R09: ffff909fa2bd0000 > [Tue Jun 15 19:13:30 2021] R10: 0000000000000000 R11: 0000000000001970 > R12: ffff90a63593bca0 > [Tue Jun 15 19:13:30 2021] R13: 0000000000000000 R14: ffff909fa2bd0000 > R15: ffffb904e4c7bac8 > [Tue Jun 15 19:13:30 2021] FS: 00007f85da5fc700(0000) > GS:ffff9098ef800000(0000) knlGS:0000000000000000 > [Tue Jun 15 19:13:30 2021] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [Tue Jun 15 19:13:30 2021] CR2: 000000c420563010 CR3: 0000001fe1698000 > CR4: 00000000003406e0 > [Tue Jun 15 19:13:30 2021] Call Trace: > [Tue Jun 15 19:13:30 2021] kvm_mmu_pte_write+0x421/0x430 [kvm] > [Tue Jun 15 19:13:30 2021] kvm_page_track_write+0x82/0xc0 [kvm] > [Tue Jun 15 19:13:30 2021] emulator_write_phys+0x3b/0x50 [kvm] > [Tue Jun 15 19:13:30 2021] write_emulate+0xe/0x10 [kvm] > [Tue Jun 15 19:13:30 2021] emulator_read_write_onepage+0xfc/0x320 [kvm] > [Tue Jun 15 19:13:30 2021] emulator_read_write+0xd6/0x190 [kvm] > [Tue Jun 15 19:13:30 2021] emulator_write_emulated+0x15/0x20 [kvm] > [Tue Jun 15 19:13:30 2021] segmented_write+0x5d/0x80 [kvm] > [Tue Jun 15 19:13:30 2021] writeback+0x203/0x2e0 [kvm] > [Tue Jun 15 19:13:30 2021] x86_emulate_insn+0x990/0x1050 [kvm] > [Tue Jun 15 19:13:30 2021] x86_emulate_instruction+0x350/0x710 [kvm] > [Tue Jun 15 19:13:30 2021] complete_emulated_pio+0x3f/0x70 [kvm] > [Tue Jun 15 19:13:30 2021] kvm_arch_vcpu_ioctl_run+0x4cb/0x570 [kvm] > [Tue Jun 15 19:13:30 2021] kvm_vcpu_ioctl+0x24b/0x610 [kvm] > [Tue Jun 15 19:13:30 2021] do_vfs_ioctl+0xa9/0x640 > [Tue Jun 15 19:13:30 2021] ? task_numa_work+0x228/0x300 > [Tue Jun 15 19:13:30 2021] ksys_ioctl+0x67/0x90 > [Tue Jun 15 19:13:30 2021] __x64_sys_ioctl+0x1a/0x20 > [Tue Jun 15 19:13:30 2021] do_syscall_64+0x57/0x190 > [Tue Jun 15 19:13:30 2021] entry_SYSCALL_64_after_hwframe+0x44/0xa9 > [Tue Jun 15 19:13:30 2021] RIP: 0033:0x7f8df6dec427 > [Tue Jun 15 19:13:30 2021] Code: 00 00 90 48 8b 05 69 aa 0c 00 64 c7 00 > 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 > 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 aa 0c 00 f7 d8 > 64 89 01 48 > [Tue Jun 15 19:13:30 2021] RSP: 002b:00007f85da5f8c08 EFLAGS: 00000246 > ORIG_RAX: 0000000000000010 > [Tue Jun 15 19:13:30 2021] RAX: ffffffffffffffda RBX: 000000000000ae80 > RCX: 00007f8df6dec427 > [Tue Jun 15 19:13:30 2021] RDX: 0000000000000000 RSI: 000000000000ae80 > RDI: 0000000000000022 > [Tue Jun 15 19:13:30 2021] RBP: 0000000000000000 R08: 000055be0e527f58 > R09: 0000000000000000 > [Tue Jun 15 19:13:30 2021] R10: 0000000000000001 R11: 0000000000000246 > R12: 000055be0f426eb0 > [Tue Jun 15 19:13:30 2021] R13: 00007f8dea1f5000 R14: 0000000000000000 > R15: 000055be0f426eb0 > [Tue Jun 15 19:13:30 2021] ---[ end trace 4e3f65d27e26463c ]--- > > I'm guessing this is more of a qemu bug / issue, but it does have a lot > of impact on the performance of nested VMs, though it does not crash. I > was wondering if any of proxmox users noticed that with these CPUs / > versions before going upstream? > > Regards, > > Gilles > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > -- Gilles?Pietri 29 avenue de la Lib?ration, F-44400 Rez? 0634745199 From elacunza at binovo.es Fri Jun 25 10:06:20 2021 From: elacunza at binovo.es (Eneko Lacunza) Date: Fri, 25 Jun 2021 10:06:20 +0200 Subject: BIG cluster questions In-Reply-To: References: Message-ID: <10800fc7-ef19-a2ea-6490-3b70161601ef@binovo.es> Hi, We have tested without bonding, same issues. El 24/6/21 a las 16:30, Eneko Lacunza escribi?: > Hi all, > > We're currently helping a customer to configure a virtualization > cluster with 88 servers for VDI. > > Right know we're testing the feasibility of building just one Proxmox > cluster of 88 nodes. A 4-node cluster has been configured too for > comparing both (same server and networking/racks). > > Nodes have 2 NICs 2x25Gbps each. Currently there are two LACP bonds > configured (one for each NIC); one for storage (NFS v4.2) and the > other for the rest (VMs, cluster). > > Cluster has two rings, one on each bond. > > - With clusters at rest (no significant number of VMs running), we see > quite a different corosync/knet latency average on our 88 node cluster > (~300-400) and our 4-node cluster (<100). > > > For 88-node cluster: > > - Creating some VMs (let's say 16), one each 30s, works well. > - Destroying some VMs (let's say 16), one each 30s, outputs error > messages (storage cfs lock related) and fails removing some of the VMs. > > - Rebooting 32 nodes, one each 30 seconds (boot for a node is about > 120s) so that no quorum is lost, creates a cluster traffic "flood". > Some of the rebooted nodes don't rejoin the cluster, and WUI shows all > nodes in cluster quorum with a grey ?, instead of green OK. In this > situation corosying latency in some nodes can skyrocket to 10s or 100s > times the values before the reboots. Access to pmxcfs is very slow and > we have been able to fix the issue only rebooting all nodes. > > - We have tried changing the transport of knet in a ring from UDP to > SCTP as reported here: > https://forum.proxmox.com/threads/proxmox-6-2-corosync-3-rare-and-spontaneous-disruptive-udp-5405-storm-flood.75871/page-2 > that gives better latencies for corosync, but the reboot issue continues. > > We don't know whether both issues are related or not. > > Could LACP bonds be the issue? > https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration > " > If your switch support the LACP (IEEE 802.3ad) protocol then we > recommend using the corresponding bonding mode (802.3ad). Otherwise > you should generally use the active-backup mode. > If you intend to run your cluster network on the bonding interfaces, > then you have to use active-passive mode on the bonding interfaces, > other modes are unsupported. > " > As per second line, we understand that running cluster networking over > a LACP bond is not supported (just to confirm our interpretation)? > We're in the process of reconfiguring nodes/switches to test without a > bond, to see if that gives us a stable cluster (will report on this). > Do you think this could be the issue? > > > Now for more general questions; do you think a 88-node Proxmox VE > cluster is feasible? > > Those 88 nodes will host about 14.000 VMs. Will HA manager be able to > manage them, or are they too many? (HA for those VMs doesn't seem to > be a requirement right know). > > > Thanks a lot > Eneko > EnekoLacunza CTO | Zuzendari teknikoa Binovo IT Human Project 943 569 206 elacunza at binovo.es binovo.es Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun youtube linkedin From laurentfdumont at gmail.com Fri Jun 25 19:33:37 2021 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Fri, 25 Jun 2021 13:33:37 -0400 Subject: [PVE-User] BIG cluster questions In-Reply-To: References: Message-ID: This is anecdotal but I have never seen one cluster that big. You might want to inquire about professional support which would give you a better perspective for that kind of scale. On Thu, Jun 24, 2021 at 10:30 AM Eneko Lacunza via pve-user < pve-user at lists.proxmox.com> wrote: > > > > ---------- Forwarded message ---------- > From: Eneko Lacunza > To: "pve-user at pve.proxmox.com" > Cc: > Bcc: > Date: Thu, 24 Jun 2021 16:30:31 +0200 > Subject: BIG cluster questions > Hi all, > > We're currently helping a customer to configure a virtualization cluster > with 88 servers for VDI. > > Right know we're testing the feasibility of building just one Proxmox > cluster of 88 nodes. A 4-node cluster has been configured too for > comparing both (same server and networking/racks). > > Nodes have 2 NICs 2x25Gbps each. Currently there are two LACP bonds > configured (one for each NIC); one for storage (NFS v4.2) and the other > for the rest (VMs, cluster). > > Cluster has two rings, one on each bond. > > - With clusters at rest (no significant number of VMs running), we see > quite a different corosync/knet latency average on our 88 node cluster > (~300-400) and our 4-node cluster (<100). > > > For 88-node cluster: > > - Creating some VMs (let's say 16), one each 30s, works well. > - Destroying some VMs (let's say 16), one each 30s, outputs error > messages (storage cfs lock related) and fails removing some of the VMs. > > - Rebooting 32 nodes, one each 30 seconds (boot for a node is about > 120s) so that no quorum is lost, creates a cluster traffic "flood". Some > of the rebooted nodes don't rejoin the cluster, and WUI shows all nodes > in cluster quorum with a grey ?, instead of green OK. In this situation > corosying latency in some nodes can skyrocket to 10s or 100s times the > values before the reboots. Access to pmxcfs is very slow and we have > been able to fix the issue only rebooting all nodes. > > - We have tried changing the transport of knet in a ring from UDP to > SCTP as reported here: > > https://forum.proxmox.com/threads/proxmox-6-2-corosync-3-rare-and-spontaneous-disruptive-udp-5405-storm-flood.75871/page-2 > that gives better latencies for corosync, but the reboot issue continues. > > We don't know whether both issues are related or not. > > Could LACP bonds be the issue? > > https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration > " > If your switch support the LACP (IEEE 802.3ad) protocol then we > recommend using the corresponding bonding mode (802.3ad). Otherwise you > should generally use the active-backup mode. > If you intend to run your cluster network on the bonding interfaces, > then you have to use active-passive mode on the bonding interfaces, > other modes are unsupported. > " > As per second line, we understand that running cluster networking over a > LACP bond is not supported (just to confirm our interpretation)? We're > in the process of reconfiguring nodes/switches to test without a bond, > to see if that gives us a stable cluster (will report on this). Do you > think this could be the issue? > > > Now for more general questions; do you think a 88-node Proxmox VE > cluster is feasible? > > Those 88 nodes will host about 14.000 VMs. Will HA manager be able to > manage them, or are they too many? (HA for those VMs doesn't seem to be > a requirement right know). > > > Thanks a lot > Eneko > > > EnekoLacunza > > CTO | Zuzendari teknikoa > > Binovo IT Human Project > > 943 569 206 > > elacunza at binovo.es > > binovo.es > > Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun > > > youtube > linkedin > > > > > ---------- Forwarded message ---------- > From: Eneko Lacunza via pve-user > To: "pve-user at pve.proxmox.com" > Cc: Eneko Lacunza > Bcc: > Date: Thu, 24 Jun 2021 16:30:31 +0200 > Subject: [PVE-User] BIG cluster questions > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From laurentfdumont at gmail.com Fri Jun 25 19:33:37 2021 From: laurentfdumont at gmail.com (Laurent Dumont) Date: Fri, 25 Jun 2021 13:33:37 -0400 Subject: [PVE-User] BIG cluster questions In-Reply-To: References: Message-ID: This is anecdotal but I have never seen one cluster that big. You might want to inquire about professional support which would give you a better perspective for that kind of scale. On Thu, Jun 24, 2021 at 10:30 AM Eneko Lacunza via pve-user < pve-user at lists.proxmox.com> wrote: > > > > ---------- Forwarded message ---------- > From: Eneko Lacunza > To: "pve-user at pve.proxmox.com" > Cc: > Bcc: > Date: Thu, 24 Jun 2021 16:30:31 +0200 > Subject: BIG cluster questions > Hi all, > > We're currently helping a customer to configure a virtualization cluster > with 88 servers for VDI. > > Right know we're testing the feasibility of building just one Proxmox > cluster of 88 nodes. A 4-node cluster has been configured too for > comparing both (same server and networking/racks). > > Nodes have 2 NICs 2x25Gbps each. Currently there are two LACP bonds > configured (one for each NIC); one for storage (NFS v4.2) and the other > for the rest (VMs, cluster). > > Cluster has two rings, one on each bond. > > - With clusters at rest (no significant number of VMs running), we see > quite a different corosync/knet latency average on our 88 node cluster > (~300-400) and our 4-node cluster (<100). > > > For 88-node cluster: > > - Creating some VMs (let's say 16), one each 30s, works well. > - Destroying some VMs (let's say 16), one each 30s, outputs error > messages (storage cfs lock related) and fails removing some of the VMs. > > - Rebooting 32 nodes, one each 30 seconds (boot for a node is about > 120s) so that no quorum is lost, creates a cluster traffic "flood". Some > of the rebooted nodes don't rejoin the cluster, and WUI shows all nodes > in cluster quorum with a grey ?, instead of green OK. In this situation > corosying latency in some nodes can skyrocket to 10s or 100s times the > values before the reboots. Access to pmxcfs is very slow and we have > been able to fix the issue only rebooting all nodes. > > - We have tried changing the transport of knet in a ring from UDP to > SCTP as reported here: > > https://forum.proxmox.com/threads/proxmox-6-2-corosync-3-rare-and-spontaneous-disruptive-udp-5405-storm-flood.75871/page-2 > that gives better latencies for corosync, but the reboot issue continues. > > We don't know whether both issues are related or not. > > Could LACP bonds be the issue? > > https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration > " > If your switch support the LACP (IEEE 802.3ad) protocol then we > recommend using the corresponding bonding mode (802.3ad). Otherwise you > should generally use the active-backup mode. > If you intend to run your cluster network on the bonding interfaces, > then you have to use active-passive mode on the bonding interfaces, > other modes are unsupported. > " > As per second line, we understand that running cluster networking over a > LACP bond is not supported (just to confirm our interpretation)? We're > in the process of reconfiguring nodes/switches to test without a bond, > to see if that gives us a stable cluster (will report on this). Do you > think this could be the issue? > > > Now for more general questions; do you think a 88-node Proxmox VE > cluster is feasible? > > Those 88 nodes will host about 14.000 VMs. Will HA manager be able to > manage them, or are they too many? (HA for those VMs doesn't seem to be > a requirement right know). > > > Thanks a lot > Eneko > > > EnekoLacunza > > CTO | Zuzendari teknikoa > > Binovo IT Human Project > > 943 569 206 > > elacunza at binovo.es > > binovo.es > > Astigarragako Bidea, 2 - 2 izda. Oficina 10-11, 20180 Oiartzun > > > youtube > linkedin > > > > > ---------- Forwarded message ---------- > From: Eneko Lacunza via pve-user > To: "pve-user at pve.proxmox.com" > Cc: Eneko Lacunza > Bcc: > Date: Thu, 24 Jun 2021 16:30:31 +0200 > Subject: [PVE-User] BIG cluster questions > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > From kyleaschmitt at gmail.com Sat Jun 26 01:34:38 2021 From: kyleaschmitt at gmail.com (Kyle Schmitt) Date: Fri, 25 Jun 2021 18:34:38 -0500 Subject: [PVE-User] Is it safe to remove orphaned disk images? Message-ID: I have a root disk image for a container on an old network share, and I forgot to tell the system to remove the old image after migrating it to the newer network share. Because the VMID still exists, the web-ui is refusing to remove the disk image. If I manually remove it from the share, via linux, will it break anything? Thanks --Kyle From leesteken at protonmail.ch Sat Jun 26 09:11:13 2021 From: leesteken at protonmail.ch (Arjen) Date: Sat, 26 Jun 2021 07:11:13 +0000 Subject: [PVE-User] Is it safe to remove orphaned disk images? In-Reply-To: References: Message-ID: On Saturday, June 26th, 2021 at 01:34, Kyle Schmitt wrote: > I have a root disk image for a container on an old network share, and > I forgot to tell the system to remove the old image after migrating it > to the newer network share. Because the VMID still exists, the web-ui > is refusing to remove the disk image. If I manually remove it from > the share, via linux, will it break anything? Usually that works fine. If you want to test it first, just rename it instead. And delete it only after you made sure there are no problems. From aderumier at odiso.com Sat Jun 26 13:16:27 2021 From: aderumier at odiso.com (aderumier at odiso.com) Date: Sat, 26 Jun 2021 13:16:27 +0200 Subject: [PVE-User] BIG cluster questions In-Reply-To: References: Message-ID: Le jeudi 24 juin 2021 ? 16:30 +0200, Eneko Lacunza via pve-user a ?crit?: > Now for more general questions; do you think a 88-node Proxmox VE > cluster is feasible? Well, corosync is not really done to this amount of node. (I think the hardcoded limit is around 100), but in practice, I have seen a lot of users having problem with corosync communication starting around 30 nodes (Maybe with low-latency switches + fast frequencies cpu, it's possible to keep latency enough low to get it working) > Those 88 nodes will host about 14.000 VMs. Will HA manager be able to > manage them, or are they too many? (HA for those VMs doesn't seem to > be > a requirement right know). I have cluster with 2500vms, it's working fine. (on a 15 nodes cluster with around 200vms on each node). I don't known with 15000vms, maybe the main pve-crm loop will take more time, I'm not sure about the timeouts. At work, I'm doing nodes 20 nodes cluster (1 by rack) to avoid to have big cluster. Multi-cluster central management is not yet on the roadmap, but they are some tool like https://maas.io/ which allow to manage multi-cluster in a central way.? From aderumier at odiso.com Sat Jun 26 13:16:27 2021 From: aderumier at odiso.com (aderumier at odiso.com) Date: Sat, 26 Jun 2021 13:16:27 +0200 Subject: [PVE-User] BIG cluster questions In-Reply-To: References: Message-ID: Le jeudi 24 juin 2021 ? 16:30 +0200, Eneko Lacunza via pve-user a ?crit?: > Now for more general questions; do you think a 88-node Proxmox VE > cluster is feasible? Well, corosync is not really done to this amount of node. (I think the hardcoded limit is around 100), but in practice, I have seen a lot of users having problem with corosync communication starting around 30 nodes (Maybe with low-latency switches + fast frequencies cpu, it's possible to keep latency enough low to get it working) > Those 88 nodes will host about 14.000 VMs. Will HA manager be able to > manage them, or are they too many? (HA for those VMs doesn't seem to > be > a requirement right know). I have cluster with 2500vms, it's working fine. (on a 15 nodes cluster with around 200vms on each node). I don't known with 15000vms, maybe the main pve-crm loop will take more time, I'm not sure about the timeouts. At work, I'm doing nodes 20 nodes cluster (1 by rack) to avoid to have big cluster. Multi-cluster central management is not yet on the roadmap, but they are some tool like https://maas.io/ which allow to manage multi-cluster in a central way.? From jmr.richardson at gmail.com Sat Jun 26 14:59:43 2021 From: jmr.richardson at gmail.com (JR Richardson) Date: Sat, 26 Jun 2021 07:59:43 -0500 Subject: [PVE-User] BIG cluster questions Message-ID: <000001d76a8b$2271a2f0$6754e8d0$@gmail.com> That is a big cluster, I like it, hope it works out. You should separate the corosync/heartbeat network on its own physical Ethernet link. This is probably where you are getting latency from. Even though you are using 25Gig NICs, pushing all your data/migration traffic/heartbeat traffic, across one physical link bonded or not, you can experience situations with a busy link where your corosync traffic is queued, even for a few milli seconds, this will add up across many nodes. Think about jumbo frames as well, slamming a NIC with 9000 byte packets for storage, and poor little heartbeat packets start queueing up in the waiting pool. In the design notes for proxmox, it's highly recommended to separate all needed networks on physical NICs and switches as well. Good luck. JR Richardson Engineering for the Masses Chasing the Azeotrope JRx DistillCo 1'st Place Brisket 1'st Place Chili This is anecdotal but I have never seen one cluster that big. You might want to inquire about professional support which would give you a better perspective for that kind of scale. On Thu, Jun 24, 2021 at 10:30 AM Eneko Lacunza via pve-user < pve-user at lists.proxmox.com> wrote: > > > > ---------- Forwarded message ---------- > From: Eneko Lacunza > To: "pve-user at pve.proxmox.com" > Cc: > Bcc: > Date: Thu, 24 Jun 2021 16:30:31 +0200 > Subject: BIG cluster questions > Hi all, > > We're currently helping a customer to configure a virtualization > cluster with 88 servers for VDI. > From leandro at tecnetmza.com.ar Mon Jun 28 13:49:25 2021 From: leandro at tecnetmza.com.ar (Leandro Roggerone) Date: Mon, 28 Jun 2021 08:49:25 -0300 Subject: [PVE-User] timeout when creating backup Message-ID: Hello. I got this message when trying to create backup for specific machine: INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-106-2021_06_28-08_45_42.vma.lzo' ERROR: got timeout INFO: aborting backup job Dont know what happened. It was working ok some days ago. Is there anything I can do ? Thanks!! Leandro. Libre de virus. www.avast.com <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> From adamw at matrixscience.com Mon Jun 28 17:09:44 2021 From: adamw at matrixscience.com (Adam Weremczuk) Date: Mon, 28 Jun 2021 16:09:44 +0100 Subject: [PVE-User] invoking ssh crashes container Message-ID: <267f44a3-24e1-1a44-d222-837f144bdfdb@matrixscience.com> Hi all, I run a minimalistic Debian 10.10 container on Proxmox 6.2-6 dual host stack. It has 512 MB of memory assigned and most of the time uses about 10%. Every time I try to ssh from it (using bash in web console) to a different host the container crashes (powers off). Syslog records the following: (...) Jun 28 13:51:46 debian10 kernel: [26284286.210180] [? 25629]? 1006 25629???? 3205???? 1290??? 61440??????? 0???????????? 0 ssh Jun 28 13:51:46 debian10 kernel: [26284286.210188] [? 25634]? 1006 25634???? 3205???? 1289??? 61440??????? 0???????????? 0 ssh Jun 28 13:51:46 debian10 kernel: [26284286.210196] [? 25640]? 1006 25640???? 3205???? 1292??? 65536??????? 0???????????? 0 ssh Jun 28 13:51:46 debian10 kernel: [26284286.210207] [? 23745]???? 0 23745???? 4880???? 1753??? 73728??????? 0???????????? 0 systemd-logind Jun 28 13:51:46 debian10 kernel: [26284286.210218] [??? 810] 0?? 810???? 3962???? 1576??? 65536??????? 0???????? -1000 sshd Jun 28 13:51:46 debian10 kernel: [26284286.213678] ssh invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Jun 28 13:51:46 debian10 kernel: [26284286.213729] dump_header+0x4f/0x1e1 Jun 28 13:51:46 debian10 kernel: [26284286.213770] mem_cgroup_try_charge+0x71/0x190 Jun 28 13:51:46 debian10 kernel: [26284286.213820] filemap_fault+0x86f/0xa60 Jun 28 13:51:46 debian10 kernel: [26284286.213971] __handle_mm_fault+0xeb0/0x12e0 Jun 28 13:51:46 debian10 kernel: [26284286.214035] RIP: 0033:0x5619b5c48930 Jun 28 13:51:46 debian10 kernel: [26284286.214286] [? 24101]? 1006 24101???? 3205???? 1088??? 65536????? 172???????????? 0 ssh Jun 28 13:51:46 debian10 kernel: [26284286.214293] [? 24105]? 1006 24105???? 3205???? 1108??? 65536????? 173???????????? 0 ssh Jun 28 13:51:46 debian10 kernel: [26284286.214299] [? 24109]? 1006 24109???? 3205???? 1103??? 65536????? 173???????????? 0 ssh (...) MANUAL POWER ON Jun 28 13:58:49 debian10 systemd-sysctl[44]: Couldn't write '1' to 'fs/protected_hardlinks', ignoring: Read-only file system Jun 28 13:58:49 debian10 systemd-sysctl[44]: Couldn't write '1' to 'fs/protected_symlinks', ignoring: Read-only file system Jun 28 13:58:49 debian10 systemd[1]: Starting Flush Journal to Persistent Storage... Jun 28 13:58:49 debian10 systemd[1]: Started Create System Users. (...) Any ideas? Regards, Adam From t.lamprecht at proxmox.com Mon Jun 28 11:32:58 2021 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Mon, 28 Jun 2021 11:32:58 +0200 Subject: [PVE-User] BIG cluster questions In-Reply-To: References: Message-ID: Hi, On 24.06.21 16:30, Eneko Lacunza wrote: > We're currently helping a customer to configure a virtualization cluster with 88 servers for VDI. > > Right know we're testing the feasibility of building just one Proxmox cluster of 88 nodes. A 4-node cluster has been configured too for comparing both (same server and networking/racks). > > Nodes have 2 NICs 2x25Gbps each. Currently there are two LACP bonds configured (one for each NIC); one for storage (NFS v4.2) and the other for the rest (VMs, cluster). > > Cluster has two rings, one on each bond. > > - With clusters at rest (no significant number of VMs running), we see quite a different corosync/knet latency average on our 88 node cluster (~300-400) and our 4-node cluster (<100). > > > For 88-node cluster: > > - Creating some VMs (let's say 16), one each 30s, works well. > - Destroying some VMs (let's say 16), one each 30s, outputs error messages (storage cfs lock related) and fails removing some of the VMs. some storages operationss are still cluster-locked on a not so fine-grained basis, that could be probably improved - mostly affects parallel creation/removals though - once the guests are setup it should not be such a big issue as then removal and creations frequency normally will go down. > > - Rebooting 32 nodes, one each 30 seconds (boot for a node is about 120s) so that no quorum is lost, creates a cluster traffic "flood". Some of the rebooted nodes don't rejoin the cluster, and WUI shows all nodes in cluster quorum with a grey ?, instead of green OK. In this situation corosying latency in some nodes can skyrocket to 10s or 100s times the values before the reboots. Access to pmxcfs is very slow and we have been able to fix the issue only rebooting all nodes. > > - We have tried changing the transport of knet in a ring from UDP to SCTP as reported here: > https://forum.proxmox.com/threads/proxmox-6-2-corosync-3-rare-and-spontaneous-disruptive-udp-5405-storm-flood.75871/page-2 > that gives better latencies for corosync, but the reboot issue continues. > > We don't know whether both issues are related or not. > > Could LACP bonds be the issue? > https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_network_configuration > " > If your switch support the LACP (IEEE 802.3ad) protocol then we recommend using the corresponding bonding mode (802.3ad). Otherwise you should generally use the active-backup mode. > If you intend to run your cluster network on the bonding interfaces, then you have to use active-passive mode on the bonding interfaces, other modes are unsupported. > " > As per second line, we understand that running cluster networking over a LACP bond is not supported (just to confirm our interpretation)? We're in the process of reconfiguring nodes/switches to test without a bond, to see if that gives us a stable cluster (will report on this). Do you think this could be the issue? > We general think that there's something that can still improve on leave/join of the cluster, those floods are not ideal and we tackled a few issues with it already (so ensure latest Proxmox VE 6.4 is in use), but there are still some issues left, which sadly are really hard to debug and reproduce nicely. > Now for more general questions; do you think a 88-node Proxmox VE cluster is feasible? > To be honest, currently I rather do not think so. The biggest cluster we know of is 51 nodes big, and they had quite some fine-tuning going on and really high-end "golden" HW to get there. > Those 88 nodes will host about 14.000 VMs. Will HA manager be able to manage them, or are they too many? (HA for those VMs doesn't seem to be a requirement right know). The HA stack has some limitations due to physical maximum file size, so currently it can track about 3500 - 4000 services, so 14k would be to much regarding that. I'd advise to split that cluster into 4 separate clusters, for example 2 x 21-node + 2 x 23-node cluster (odd sized node count are slightly better regarding quorum), then you'd have roughly 3500 HA services per cluster, which can be feasible. As there's semi-active work going on in cross-cluster migrations, the biggest drawback one gets when splitting up into separate clusters would be gone, sooner or later. From mark at tuxis.nl Tue Jun 29 10:05:44 2021 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 29 Jun 2021 10:05:44 +0200 Subject: [PVE-User] Proxmox VE 7.0 (beta) released! In-Reply-To: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> References: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> Message-ID: <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> Hi, Op 24-06-2021 om 15:16 schreef Martin Maurer: > We are pleased to announce the first beta release of Proxmox Virtual > Environment 7.0! The 7.x family is based on the great Debian 11 > "Bullseye" and comes with a 5.11 kernel, QEMU 6.0, LXC 4.0, OpenZFS 2.0.4. I just upgraded a node in our demo cluster and all seemed fine. Except for non-working cluster network. I was unable to ping the node through the cluster interface, pvecm saw no other nodes and ceph was broken. However, if I ran tcpdump, ping started working, but not the rest. Interesting situation, which I 'fixed' by disabling vlan-aware-bridge for that interface. After the reboot, everything works (AFAICS). If Proxmox wants to debug this, feel free to reach out to me, I can grant you access to this node so you can check it out. -- Mark Schouten CTO, Tuxis B.V. | https://www.tuxis.nl/ | +31 318 200208 From s.ivanov at proxmox.com Tue Jun 29 10:23:17 2021 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Tue, 29 Jun 2021 10:23:17 +0200 Subject: [PVE-User] Proxmox VE 7.0 (beta) released! In-Reply-To: <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> References: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> Message-ID: <20210629102317.0a29b58f@rosa.proxmox.com> Hi, On Tue, 29 Jun 2021 10:05:44 +0200 Mark Schouten wrote: > Hi, > > Op 24-06-2021 om 15:16 schreef Martin Maurer: > > We are pleased to announce the first beta release of Proxmox Virtual > > Environment 7.0! The 7.x family is based on the great Debian 11 > > "Bullseye" and comes with a 5.11 kernel, QEMU 6.0, LXC 4.0, OpenZFS 2.0.4. > > I just upgraded a node in our demo cluster and all seemed fine. Except > for non-working cluster network. I was unable to ping the node through > the cluster interface, pvecm saw no other nodes and ceph was broken. Thanks for the report - could you provide some details on the upgraded node? Mostly which NICs are used - but also the complete hardware -setup (If you prefer you can send me a pvereport to my e-mail) > > However, if I ran tcpdump, ping started working, but not the rest. quite odd - last time I ran into something like this was with an OpenBSD router, where the promisc flag did not get passed down to the physical port of a bridge) the output of `ip -details a` and `ip -details l` might provide some insight > > Interesting situation, which I 'fixed' by disabling vlan-aware-bridge > for that interface. After the reboot, everything works (AFAICS). Thanks for sharing the mitigation (sadly this won't work for everybody) > > If Proxmox wants to debug this, feel free to reach out to me, I can > grant you access to this node so you can check it out. > Kind Regards, stoiko From f.gruenbichler at proxmox.com Tue Jun 29 10:32:24 2021 From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=) Date: Tue, 29 Jun 2021 10:32:24 +0200 Subject: [PVE-User] invoking ssh crashes container In-Reply-To: <267f44a3-24e1-1a44-d222-837f144bdfdb@matrixscience.com> References: <267f44a3-24e1-1a44-d222-837f144bdfdb@matrixscience.com> Message-ID: <1624955440.81gresm1m7.astroid@nora.none> On June 28, 2021 5:09 pm, Adam Weremczuk wrote: > Hi all, > > I run a minimalistic Debian 10.10 container on Proxmox 6.2-6 dual host > stack. > > It has 512 MB of memory assigned and most of the time uses about 10%. > > Every time I try to ssh from it (using bash in web console) to a > different host the container crashes (powers off). > > Syslog records the following: > > (...) > Jun 28 13:51:46 debian10 kernel: [26284286.210180] [? 25629]? 1006 > 25629???? 3205???? 1290??? 61440??????? 0???????????? 0 ssh > Jun 28 13:51:46 debian10 kernel: [26284286.210188] [? 25634]? 1006 > 25634???? 3205???? 1289??? 61440??????? 0???????????? 0 ssh > Jun 28 13:51:46 debian10 kernel: [26284286.210196] [? 25640]? 1006 > 25640???? 3205???? 1292??? 65536??????? 0???????????? 0 ssh > Jun 28 13:51:46 debian10 kernel: [26284286.210207] [? 23745]???? 0 > 23745???? 4880???? 1753??? 73728??????? 0???????????? 0 systemd-logind > Jun 28 13:51:46 debian10 kernel: [26284286.210218] [??? 810] 0?? 810???? > 3962???? 1576??? 65536??????? 0???????? -1000 sshd > Jun 28 13:51:46 debian10 kernel: [26284286.213678] ssh invoked > oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, > oom_score_adj=0 > Jun 28 13:51:46 debian10 kernel: [26284286.213729] dump_header+0x4f/0x1e1 > Jun 28 13:51:46 debian10 kernel: [26284286.213770] > mem_cgroup_try_charge+0x71/0x190 > Jun 28 13:51:46 debian10 kernel: [26284286.213820] filemap_fault+0x86f/0xa60 > Jun 28 13:51:46 debian10 kernel: [26284286.213971] > __handle_mm_fault+0xeb0/0x12e0 > Jun 28 13:51:46 debian10 kernel: [26284286.214035] RIP: 0033:0x5619b5c48930 > Jun 28 13:51:46 debian10 kernel: [26284286.214286] [? 24101]? 1006 > 24101???? 3205???? 1088??? 65536????? 172???????????? 0 ssh > Jun 28 13:51:46 debian10 kernel: [26284286.214293] [? 24105]? 1006 > 24105???? 3205???? 1108??? 65536????? 173???????????? 0 ssh > Jun 28 13:51:46 debian10 kernel: [26284286.214299] [? 24109]? 1006 > 24109???? 3205???? 1103??? 65536????? 173???????????? 0 ssh > (...) > MANUAL POWER ON > Jun 28 13:58:49 debian10 systemd-sysctl[44]: Couldn't write '1' to > 'fs/protected_hardlinks', ignoring: Read-only file system > Jun 28 13:58:49 debian10 systemd-sysctl[44]: Couldn't write '1' to > 'fs/protected_symlinks', ignoring: Read-only file system > Jun 28 13:58:49 debian10 systemd[1]: Starting Flush Journal to > Persistent Storage... > Jun 28 13:58:49 debian10 systemd[1]: Started Create System Users. > (...) > > Any ideas? the log says it got OOM killed. did you enable persistent journal inside the container? if not, that might be the culprit (tmpfs usage in the container gets accounted towards the container's resource limits). `mkdir /var/log/journal && systemctl restart systemd-journald` From f.gruenbichler at proxmox.com Tue Jun 29 10:32:24 2021 From: f.gruenbichler at proxmox.com (Fabian =?iso-8859-1?q?Gr=FCnbichler?=) Date: Tue, 29 Jun 2021 10:32:24 +0200 Subject: [PVE-User] invoking ssh crashes container In-Reply-To: <267f44a3-24e1-1a44-d222-837f144bdfdb@matrixscience.com> References: <267f44a3-24e1-1a44-d222-837f144bdfdb@matrixscience.com> Message-ID: <1624955440.81gresm1m7.astroid@nora.none> On June 28, 2021 5:09 pm, Adam Weremczuk wrote: > Hi all, > > I run a minimalistic Debian 10.10 container on Proxmox 6.2-6 dual host > stack. > > It has 512 MB of memory assigned and most of the time uses about 10%. > > Every time I try to ssh from it (using bash in web console) to a > different host the container crashes (powers off). > > Syslog records the following: > > (...) > Jun 28 13:51:46 debian10 kernel: [26284286.210180] [? 25629]? 1006 > 25629???? 3205???? 1290??? 61440??????? 0???????????? 0 ssh > Jun 28 13:51:46 debian10 kernel: [26284286.210188] [? 25634]? 1006 > 25634???? 3205???? 1289??? 61440??????? 0???????????? 0 ssh > Jun 28 13:51:46 debian10 kernel: [26284286.210196] [? 25640]? 1006 > 25640???? 3205???? 1292??? 65536??????? 0???????????? 0 ssh > Jun 28 13:51:46 debian10 kernel: [26284286.210207] [? 23745]???? 0 > 23745???? 4880???? 1753??? 73728??????? 0???????????? 0 systemd-logind > Jun 28 13:51:46 debian10 kernel: [26284286.210218] [??? 810] 0?? 810???? > 3962???? 1576??? 65536??????? 0???????? -1000 sshd > Jun 28 13:51:46 debian10 kernel: [26284286.213678] ssh invoked > oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, > oom_score_adj=0 > Jun 28 13:51:46 debian10 kernel: [26284286.213729] dump_header+0x4f/0x1e1 > Jun 28 13:51:46 debian10 kernel: [26284286.213770] > mem_cgroup_try_charge+0x71/0x190 > Jun 28 13:51:46 debian10 kernel: [26284286.213820] filemap_fault+0x86f/0xa60 > Jun 28 13:51:46 debian10 kernel: [26284286.213971] > __handle_mm_fault+0xeb0/0x12e0 > Jun 28 13:51:46 debian10 kernel: [26284286.214035] RIP: 0033:0x5619b5c48930 > Jun 28 13:51:46 debian10 kernel: [26284286.214286] [? 24101]? 1006 > 24101???? 3205???? 1088??? 65536????? 172???????????? 0 ssh > Jun 28 13:51:46 debian10 kernel: [26284286.214293] [? 24105]? 1006 > 24105???? 3205???? 1108??? 65536????? 173???????????? 0 ssh > Jun 28 13:51:46 debian10 kernel: [26284286.214299] [? 24109]? 1006 > 24109???? 3205???? 1103??? 65536????? 173???????????? 0 ssh > (...) > MANUAL POWER ON > Jun 28 13:58:49 debian10 systemd-sysctl[44]: Couldn't write '1' to > 'fs/protected_hardlinks', ignoring: Read-only file system > Jun 28 13:58:49 debian10 systemd-sysctl[44]: Couldn't write '1' to > 'fs/protected_symlinks', ignoring: Read-only file system > Jun 28 13:58:49 debian10 systemd[1]: Starting Flush Journal to > Persistent Storage... > Jun 28 13:58:49 debian10 systemd[1]: Started Create System Users. > (...) > > Any ideas? the log says it got OOM killed. did you enable persistent journal inside the container? if not, that might be the culprit (tmpfs usage in the container gets accounted towards the container's resource limits). `mkdir /var/log/journal && systemctl restart systemd-journald` From t.lamprecht at proxmox.com Tue Jun 29 11:46:58 2021 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Tue, 29 Jun 2021 11:46:58 +0200 Subject: [PVE-User] Proxmox VE 7.0 (beta) released! In-Reply-To: <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> References: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> Message-ID: Hi, On 29.06.21 10:05, Mark Schouten wrote: > Op 24-06-2021 om 15:16 schreef Martin Maurer: >> We are pleased to announce the first beta release of Proxmox Virtual Environment 7.0! The 7.x family is based on the great Debian 11 "Bullseye" and comes with a 5.11 kernel, QEMU 6.0, LXC 4.0, OpenZFS 2.0.4. > > I just upgraded a node in our demo cluster and all seemed fine. Except for non-working cluster network. I was unable to ping the node through the cluster interface, pvecm saw no other nodes and ceph was broken. > > However, if I ran tcpdump, ping started working, but not the rest. > > Interesting situation, which I 'fixed' by disabling vlan-aware-bridge for that interface. After the reboot, everything works (AFAICS). > > If Proxmox wants to debug this, feel free to reach out to me, I can grant you access to this node so you can check it out. > Do you have some FW rules regarding MAC-Addresses or the like? As the MAC-Address selection changed in Proxmox VE 7 due to new default n systemd's network link policy, as listed in our known issues[0]. It's now not the one of the first port anymore, but derived from interface name and `/etc/machine-id`, which in combination should be unique but also persistent. But, for some ISO releases (4.0 to 5.3) the machine-id for the installed host was not always re-generated, which could result in duplication of a MAC for identical named interfaces on two hosts. We try to actively catch and fix that on upgrade by checking if the ID is one of the known static ones (it's just a handful of possible IDs) on upgrade. But if one cloned an machine (e.g., a colleague run into this in a demo virtualized PVE test clusters they cloned from a template) that ID will be duplicated and thus make problems. That could be easily checked by comparing the `/etc/machine-id` content and be fixed by re-generation[1]. Just noting that for completness sake, to avoid more investigation if it's just that. - Thomas [0]: https://pve.proxmox.com/wiki/Roadmap#7.0-beta-known-issues [1]: https://wiki.debian.org/MachineId#machine_id_and_cloned_systems.2C_generating_a_new_machine_id From mark at tuxis.nl Tue Jun 29 10:34:28 2021 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 29 Jun 2021 10:34:28 +0200 Subject: [PVE-User] Proxmox VE 7.0 (beta) released! In-Reply-To: <20210629102317.0a29b58f@rosa.proxmox.com> References: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> <20210629102317.0a29b58f@rosa.proxmox.com> Message-ID: Hi Stoiko, Op 29-06-2021 om 10:23 schreef Stoiko Ivanov: >> I just upgraded a node in our demo cluster and all seemed fine. Except >> for non-working cluster network. I was unable to ping the node through >> the cluster interface, pvecm saw no other nodes and ceph was broken. > Thanks for the report - could you provide some details on the upgraded > node? Mostly which NICs are used - but also the complete hardware -setup > > (If you prefer you can send me a pvereport to my e-mail) See attached! -- Mark Schouten CTO, Tuxis B.V. | https://www.tuxis.nl/ | +31 318 200208 -------------- next part -------------- ==== general system info ==== # hostname node06 # pveversion --verbose proxmox-ve: 7.0-2 (running kernel: 5.11.22-1-pve) pve-manager: 7.0-5 (running version: 7.0-5/cce9b25f) pve-kernel-5.11: 7.0-3 pve-kernel-helper: 7.0-3 pve-kernel-5.4: 6.4-3 pve-kernel-5.11.22-1-pve: 5.11.22-1 pve-kernel-5.4.119-1-pve: 5.4.119-1 pve-kernel-5.4.106-1-pve: 5.4.106-1 pve-kernel-5.4.34-1-pve: 5.4.34-2 ceph: 15.2.13-pve1 ceph-fuse: 15.2.13-pve1 corosync: 3.1.2-pve2 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown: residual config ifupdown2: 3.0.0-1+pve5 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.21-pve1 libproxmox-acme-perl: 1.1.0 libproxmox-backup-qemu0: 1.0.3-1 libpve-access-control: 7.0-3 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.0-4 libpve-guest-common-perl: 4.0-2 libpve-http-server-perl: 4.0-2 libpve-storage-perl: 7.0-6 libqb0: 1.0.5-1 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 4.0.9-1 lxcfs: 4.0.8-pve1 novnc-pve: 1.2.0-3 proxmox-backup-client: 1.1.10-1 proxmox-backup-file-restore: 1.1.10-1 proxmox-mini-journalreader: 1.2-1 proxmox-widget-toolkit: 3.1-4 pve-cluster: 7.0-2 pve-container: 4.0-3 pve-docs: 7.0-3 pve-edk2-firmware: 3.20200531-1 pve-firewall: 4.2-2 pve-firmware: 3.2-4 pve-ha-manager: 3.2-2 pve-i18n: 2.3-1 pve-qemu-kvm: 6.0.0-2 pve-xtermjs: 4.12.0-1 qemu-server: 7.0-5 smartmontools: 7.2-pve2 spiceterm: 3.2-2 vncterm: 1.7-1 zfsutils-linux: 2.0.4-pve1 # cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 2a03:7900:111::dc:6 node06.demo.customers.tuxis.net node06 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts #2a03:7900:111:0:90d4:a7ff:fe7f:ccf6 pbs.tuxis.net # pvesubscription get message: There is no subscription key serverid: F3A59435D1B87C5A2460F965646A3177 status: NotFound url: https://www.proxmox.com/proxmox-ve/pricing # cat /etc/apt/sources.list ## Managed via Ansible deb http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free deb-src http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free deb http://security.debian.org/ bullseye-security main contrib non-free deb-src http://security.debian.org/ bullseye-security main contrib non-free deb http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free deb-src http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free # cat /etc/apt/sources.list.d/pvetest-for-beta.list deb http://download.proxmox.com/debian/pve bullseye pvetest # cat /etc/apt/sources.list.d/ceph.list deb http://download.proxmox.com/debian/ceph-octopus bullseye main # cat /etc/apt/sources.list.d/apt_tuxis_nl_tuxis.list deb https://apt.tuxis.nl/tuxis/ tuxis-cron main deb https://apt.tuxis.nl/tuxis/ monitoring main deb https://apt.tuxis.nl/tuxis/ pmrb main # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz Stepping: 2 CPU MHz: 3600.000 CPU max MHz: 3600.0000 CPU min MHz: 1200.0000 BogoMIPS: 7000.21 Virtualization: VT-x L1d cache: 128 KiB L1i cache: 128 KiB L2 cache: 1 MiB L3 cache: 10 MiB NUMA node0 CPU(s): 0-7 Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d # pvesh get /cluster/resources --type node --output-format=yaml --- - cpu: 0.011763712868024 disk: 2595618816 id: node/node06 level: '' maxcpu: 8 maxdisk: 115451363328 maxmem: 67322179584 mem: 3120074752 node: node06 status: online type: node uptime: 2238 - cpu: 0.188962970750996 disk: 21875785728 id: node/node04 level: '' maxcpu: 8 maxdisk: 101597184000 maxmem: 67331584000 mem: 23079858176 node: node04 status: online type: node uptime: 13969251 - cpu: 0.0102829414157591 disk: 3339059200 id: node/node05 level: '' maxcpu: 8 maxdisk: 115422265344 maxmem: 67331592192 mem: 8491147264 node: node05 status: online type: node uptime: 2830727 ==== overall system load info ==== # top -b -c -w512 -n 1 -o TIME | head -n 30 top - 10:26:43 up 37 min, 1 user, load average: 0.13, 0.08, 0.06 Tasks: 295 total, 1 running, 294 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.8 us, 1.5 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 64203.4 total, 60470.7 free, 2954.7 used, 778.0 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 60187.0 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1647 root rt 0 571448 178096 53256 S 6.2 0.3 0:53.45 /usr/sbin/corosync -f 1663 ceph 20 0 1400308 570700 28140 S 0.0 0.9 0:39.35 /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph 1656 ceph 20 0 1298416 546100 28064 S 0.0 0.8 0:27.35 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph 1644 ceph 20 0 507260 123516 20868 S 0.0 0.2 0:24.52 /usr/bin/ceph-mon -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 1762 root 20 0 270424 95672 9780 S 0.0 0.1 0:23.76 pvestatd 1888 www-data 20 0 346704 127472 8076 S 0.0 0.2 0:09.48 pveproxy worker 1887 www-data 20 0 346736 126700 7424 S 0.0 0.2 0:08.86 pveproxy worker 1889 www-data 20 0 346604 126552 7296 S 0.0 0.2 0:07.14 pveproxy worker 1763 root 20 0 264748 84656 4284 S 0.0 0.1 0:06.55 pve-firewall 1550 root 20 0 618448 55796 50792 S 0.0 0.1 0:05.45 /usr/bin/pmxcfs 1643 ceph 20 0 503992 176144 19884 S 0.0 0.3 0:04.03 /usr/bin/ceph-mgr -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 1789 root 20 0 345064 124220 6524 S 0.0 0.2 0:02.80 pvedaemon worker 13 root 20 0 0 0 0 I 0.0 0.0 0:02.48 [rcu_sched] 1788 root 20 0 345068 123856 6204 S 0.0 0.2 0:02.40 pvedaemon worker 1790 root 20 0 345064 123804 6168 S 0.0 0.2 0:02.40 pvedaemon worker 1 root 20 0 165772 8880 5280 S 0.0 0.0 0:02.17 /sbin/init 1335 Debian-+ 20 0 24352 10532 5780 S 0.0 0.0 0:01.40 /usr/sbin/snmpd -LOw -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf -f -p /run/snmpd.pid 1642 ceph 20 0 302080 27288 11188 S 0.0 0.0 0:00.85 /usr/bin/ceph-mds -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 421 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss] 422 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss] 423 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss] 424 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss] 425 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss] # head /proc/pressure/* ==> /proc/pressure/cpu <== some avg10=0.00 avg60=0.00 avg300=0.00 total=5430541 ==> /proc/pressure/io <== some avg10=0.00 avg60=0.00 avg300=0.00 total=1300222 full avg10=0.00 avg60=0.00 avg300=0.00 total=1149011 ==> /proc/pressure/memory <== some avg10=0.00 avg60=0.00 avg300=0.00 total=0 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 ==== info about storage ==== # cat /etc/pve/storage.cfg dir: local path /var/lib/vz content images,backup,iso,vztmpl prune-backups keep-last=1 shared 0 zfspool: local-zfs pool rpool/data content rootdir,images sparse 1 rbd: Ceph content rootdir,images krbd 0 pool Ceph cephfs: CephFS path /mnt/pve/CephFS content iso,snippets,backup,vztmpl prune-backups keep-last=1 dir: Tuxis_Marketplace path /mnt/pve/Tuxis_Marketplace content iso,backup is_mountpoint yes mkdir 0 shared 1 dir: Tuxis_Marketplace_Beta path /mnt/pve/Tuxis_Marketplace_Beta content backup,iso is_mountpoint yes mkdir 0 shared 1 rbd: CephKRBD content images krbd 1 pool Ceph pbs: pbs002.tuxis.nl datastore DB0220_demo server pbs002.tuxis.nl content backup encryption-key 68:d5:89:f6:f1:f4:67:59:1b:74:6a:78:99:11:ad:09:a0:b0:12:db:43:8d:41:19:af:38:90:77:12:c1:6d:f8 fingerprint 45:f8:79:eb:27:96:88:6b:29:ad:21:00:13:c6:bd:b8:30:f6:f3:9b:f0:bf:dd:f3:ad:f0:09:d5:d2:9a:34:79 prune-backups keep-last=1 username DB0220 at pbs # pvesm status Name Type Status Total Used Available % Ceph rbd active 802501642 337252874 465248768 42.03% CephFS cephfs active 500432896 35184640 465248256 7.03% CephKRBD rbd active 802501642 337252874 465248768 42.03% Tuxis_Marketplace dir active 274877906944 0 274877906944 0.00% Tuxis_Marketplace_Beta dir active 274877906944 0 274877906944 0.00% local dir active 112745472 2534784 110210688 2.25% local-zfs zfspool active 110210824 96 110210728 0.00% pbs002.tuxis.nl pbs active 1701798656 10516096 1691282560 0.62% # cat /etc/fstab # proc /proc proc defaults 0 0 # findmnt --ascii TARGET SOURCE FSTYPE OPTIONS / rpool/ROOT/pve-1 zfs rw,relatime,xattr,noacl |-/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime | |-/sys/kernel/security securityfs securityfs rw,nosuid,nodev,noexec,relatime | |-/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime | |-/sys/fs/pstore pstore pstore rw,nosuid,nodev,noexec,relatime | |-/sys/fs/bpf none bpf rw,nosuid,nodev,noexec,relatime,mode=700 | |-/sys/kernel/debug debugfs debugfs rw,nosuid,nodev,noexec,relatime | |-/sys/kernel/tracing tracefs tracefs rw,nosuid,nodev,noexec,relatime | |-/sys/fs/fuse/connections fusectl fusectl rw,nosuid,nodev,noexec,relatime | `-/sys/kernel/config configfs configfs rw,nosuid,nodev,noexec,relatime |-/proc proc proc rw,relatime | `-/proc/sys/fs/binfmt_misc systemd-1 autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=561 |-/dev udev devtmpfs rw,nosuid,relatime,size=32838164k,nr_inodes=8209541,mode=755,inode64 | |-/dev/pts devpts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 | |-/dev/shm tmpfs tmpfs rw,nosuid,nodev,inode64 | |-/dev/mqueue mqueue mqueue rw,nosuid,nodev,noexec,relatime | `-/dev/hugepages hugetlbfs hugetlbfs rw,relatime,pagesize=2M |-/run tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=6574432k,mode=755,inode64 | |-/run/lock tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 | |-/run/rpc_pipefs sunrpc rpc_pipefs rw,relatime | `-/run/user/0 tmpfs tmpfs rw,nosuid,nodev,relatime,size=6574428k,nr_inodes=1643607,mode=700,inode64 |-/rpool rpool zfs rw,noatime,xattr,noacl | |-/rpool/ROOT rpool/ROOT zfs rw,noatime,xattr,noacl | `-/rpool/data rpool/data zfs rw,noatime,xattr,noacl |-/var/lib/ceph/osd/ceph-4 tmpfs tmpfs rw,relatime,inode64 |-/var/lib/ceph/osd/ceph-5 tmpfs tmpfs rw,relatime,inode64 |-/mnt/pve/Tuxis_Marketplace_Beta s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0 |-/mnt/pve/Tuxis_Marketplace s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0 |-/etc/pve /dev/fuse fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other |-/var/lib/lxcfs lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other `-/mnt/pve/CephFS [fdb0:5bd1:dc::4],[fdb0:5bd1:dc::5],[fdb0:5bd1:dc::6]:/ ceph rw,relatime,name=admin,secret=,acl # df --human Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G 1.3M 6.3G 1% /run rpool/ROOT/pve-1 108G 2.5G 106G 3% / tmpfs 32G 63M 32G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock rpool 106G 128K 106G 1% /rpool rpool/ROOT 106G 128K 106G 1% /rpool/ROOT rpool/data 106G 128K 106G 1% /rpool/data tmpfs 32G 28K 32G 1% /var/lib/ceph/osd/ceph-4 tmpfs 32G 28K 32G 1% /var/lib/ceph/osd/ceph-5 s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace_Beta s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace /dev/fuse 30M 40K 30M 1% /etc/pve [fdb0:5bd1:dc::4],[fdb0:5bd1:dc::5],[fdb0:5bd1:dc::6]:/ 478G 34G 444G 8% /mnt/pve/CephFS tmpfs 6.3G 4.0K 6.3G 1% /run/user/0 ==== info about network ==== # ip -details -statistics address 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 55254397 58681 0 0 0 0 TX: bytes packets errors dropped carrier collsns 55254397 58681 0 0 0 0 2: eno1: mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000 link/ether 0c:c4:7a:d9:1d:f6 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9216 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp6s0 RX: bytes packets errors dropped missed mcast 15982113 105257 0 74 0 5642 TX: bytes packets errors dropped carrier collsns 12624090 36080 0 0 0 0 3: eno2: mtu 1500 qdisc mq master vmbr999 state UP group default qlen 1000 link/ether 0c:c4:7a:d9:1d:f7 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9216 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp7s0 RX: bytes packets errors dropped missed mcast 489871250 699399 0 0 0 33 TX: bytes packets errors dropped carrier collsns 268886467 574470 0 0 0 0 4: vmbr0: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:ed:27:6f:7b:f3 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 23.76 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet6 2a03:7900:111::dc:6/64 scope global valid_lft forever preferred_lft forever inet6 fe80::50ed:27ff:fe6f:7bf3/64 scope link valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 14379299 103674 0 0 0 5565 TX: bytes packets errors dropped carrier collsns 12356802 32972 0 0 0 0 5: vmbr999: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 16:2d:db:6c:6d:8a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 160.33 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet6 fdb0:5bd1:dc::6/64 scope global valid_lft forever preferred_lft forever inet6 fe80::142d:dbff:fe6c:6d8a/64 scope link valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 465681104 499419 0 0 0 33 TX: bytes packets errors dropped carrier collsns 260240217 473931 0 0 0 0 # ip -details -4 route show # ip -details -6 route show unicast ::1 dev lo proto kernel scope global metric 256 pref medium unicast 2a03:7900:111::/64 dev vmbr0 proto kernel scope global metric 256 pref medium unicast fdb0:5bd1:dc::/64 dev vmbr999 proto kernel scope global metric 256 pref medium unicast fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff dev vmbr999 proto boot scope global metric 1024 pref medium unicast fe80::/64 dev vmbr999 proto kernel scope global metric 256 pref medium unicast fe80::/64 dev vmbr0 proto kernel scope global metric 256 pref medium unicast default via 2a03:7900:111::1 dev vmbr0 proto kernel scope global metric 1024 onlink pref medium # cat /etc/network/interfaces # network interface settings; autogenerated # Please do NOT modify this file directly, unless you know what # you're doing. # # If you want to manage parts of the network configuration manually, # please utilize the 'source' or 'source-directory' directives to do # so. # PVE will preserve these directives, but will NOT read its network # configuration from sourced files, so do not attempt to move any of # the PVE managed interfaces into external files! auto lo iface lo inet loopback iface eno1 inet6 manual iface eno2 inet6 manual auto vmbr0 iface vmbr0 inet6 static address 2a03:7900:111::dc:6/64 gateway 2a03:7900:111::1 bridge-ports eno1 bridge-stp off bridge-fd 0 auto vmbr999 iface vmbr999 inet6 static address fdb0:5bd1:dc::6/64 bridge-ports eno2 bridge-stp off bridge-fd 0 #bridge-vlan-aware yes #bridge-vids 2-4094 post-up /usr/sbin/ip ro add fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff ==== info about virtual guests ==== # qm list # pct list ==== info about firewall ==== # cat /etc/pve/local/host.fw cat: /etc/pve/local/host.fw: No such file or directory # iptables-save # Generated by iptables-save v1.8.7 on Tue Jun 29 10:26:45 2021 *raw :PREROUTING ACCEPT [10644:3275736] :OUTPUT ACCEPT [8292:3186298] COMMIT # Completed on Tue Jun 29 10:26:45 2021 # Generated by iptables-save v1.8.7 on Tue Jun 29 10:26:45 2021 *filter :INPUT ACCEPT [8292:3186298] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [8293:3186338] COMMIT # Completed on Tue Jun 29 10:26:45 2021 ==== info about cluster ==== # pvecm nodes Membership information ---------------------- Nodeid Votes Name 1 1 node04 2 1 node05 3 1 node06 (local) # pvecm status Cluster information ------------------- Name: Demo Config Version: 3 Transport: knet Secure auth: on Quorum information ------------------ Date: Tue Jun 29 10:26:46 2021 Quorum provider: corosync_votequorum Nodes: 3 Node ID: 0x00000003 Ring ID: 1.f47 Quorate: Yes Votequorum information ---------------------- Expected votes: 3 Highest expected: 3 Total votes: 3 Quorum: 2 Flags: Quorate Membership information ---------------------- Nodeid Votes Name 0x00000001 1 fdb0:5bd1:dc::4%32617 0x00000002 1 fdb0:5bd1:dc::5%32617 0x00000003 1 fdb0:5bd1:dc::6%32617 (local) # cat /etc/pve/corosync.conf 2>/dev/null logging { debug: off to_syslog: yes } nodelist { node { name: node04 nodeid: 1 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::4 } node { name: node05 nodeid: 2 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::5 } node { name: node06 nodeid: 3 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::6 } } quorum { provider: corosync_votequorum } totem { cluster_name: Demo config_version: 3 interface { linknumber: 0 } ip_version: ipv4-6 link_mode: passive secauth: on version: 2 } # ha-manager status quorum OK master node06 (idle, Tue Jun 9 16:20:27 2020) lrm node04 (idle, Tue Jun 29 10:26:43 2021) lrm node05 (idle, Tue Jun 29 10:26:43 2021) lrm node06 (idle, Tue Jun 29 10:26:41 2021) ==== info about hardware ==== # dmidecode -t bios # dmidecode 3.3 Getting SMBIOS data from sysfs. SMBIOS 3.0 present. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: American Megatrends Inc. Version: 2.0a Release Date: 08/01/2016 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16 MB Characteristics: PCI is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported BIOS ROM is socketed EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) ACPI is supported USB legacy is supported BIOS boot specification is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 5.6 # lspci -nnk 00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [8086:2f00] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [15d9:0832] 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f02] (rev 02) Kernel driver in use: pcieport 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f03] (rev 02) Kernel driver in use: pcieport 00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02) Kernel driver in use: pcieport 00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f0a] (rev 02) Kernel driver in use: pcieport 00:04.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [8086:2f20] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [8086:2f21] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [8086:2f22] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [8086:2f23] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [8086:2f24] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [8086:2f25] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [8086:2f26] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [8086:2f27] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [8086:2f28] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [15d9:0832] 00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [8086:2f29] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [15d9:0832] 00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [8086:2f2a] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [15d9:0832] 00:05.4 PIC [0800]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [8086:2f2c] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [15d9:0832] 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR [8086:8d7c] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] 00:11.4 SATA controller [0106]: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] [8086:8d62] (rev 05) Subsystem: Super Micro Computer Inc C610/X99 series chipset sSATA Controller [AHCI mode] [15d9:0832] Kernel driver in use: ahci Kernel modules: ahci 00:14.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: xhci_hcd Kernel modules: xhci_pci 00:16.0 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel modules: mei_me 00:16.1 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #2 [8086:8d3b] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] 00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 [8086:8d10] (rev d5) Kernel driver in use: pcieport 00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 [8086:8d18] (rev d5) Kernel driver in use: pcieport 00:1c.5 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #6 [8086:8d1a] (rev d5) Kernel driver in use: pcieport 00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 [8086:8d1c] (rev d5) Kernel driver in use: pcieport 00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series chipset LPC Controller [8086:8d44] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: lpc_ich Kernel modules: lpc_ich 00:1f.2 SATA controller [0106]: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [8086:8d02] (rev 05) Subsystem: Super Micro Computer Inc C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [15d9:0832] Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset SMBus Controller [8086:8d22] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: i801_smbus Kernel modules: i2c_i801 06:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) DeviceName: Intel Ethernet i210AT #1 Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533] Kernel driver in use: igb Kernel modules: igb 07:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) DeviceName: Intel Ethernet i210AT #2 Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533] Kernel driver in use: igb Kernel modules: igb 08:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03) 09:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30) DeviceName: ASPEED Video AST2400 Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ast Kernel modules: ast ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f81] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f36] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] Kernel driver in use: hswep_uncore ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f37] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] Kernel driver in use: hswep_uncore ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe1] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe2] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe3] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff8] (rev 02) ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff9] (rev 02) ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffc] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffd] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffe] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f1d] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832] ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f34] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832] Kernel driver in use: hswep_uncore ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1e] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f7d] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1f] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2fa0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832] ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2f30] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832] Kernel driver in use: hswep_uncore ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832] ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832] ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2faa] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fab] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fac] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fad] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast [8086:2fae] (rev 02) ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2faf] (rev 02) ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [8086:2fb0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [8086:2fb1] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [8086:2fb2] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [15d9:0832] ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [8086:2fb3] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [15d9:0832] ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbc] (rev 02) ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbd] (rev 02) ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbe] (rev 02) ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbf] (rev 02) ff:15.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [8086:2fb4] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:15.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [8086:2fb5] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:15.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [8086:2fb6] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [15d9:0832] ff:15.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [8086:2fb7] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [15d9:0832] ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers [8086... (rev 02) ff:16.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast [8086:2f6e] (rev 02) ff:16.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2f6f] (rev 02) ff:17.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control [8086:2fd0] (rev 02) Kernel driver in use: hswep_uncore ff:17.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb8] (rev 02) ff:17.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb9] (rev 02) ff:17.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fba] (rev 02) ff:17.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fbb] (rev 02) ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f98] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f99] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9a] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2fc0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9c] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f88] (rev 02) ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f8a] (rev 02) ==== info about block devices ==== # lsblk --ascii NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 111.8G 0 disk |-sda1 8:1 0 1007K 0 part |-sda2 8:2 0 512M 0 part `-sda3 8:3 0 111.3G 0 part sdb 8:16 0 111.8G 0 disk |-sdb1 8:17 0 1007K 0 part |-sdb2 8:18 0 512M 0 part `-sdb3 8:19 0 111.3G 0 part sdc 8:32 0 447.1G 0 disk `-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 253:0 0 447.1G 0 lvm sdd 8:48 0 447.1G 0 disk `-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 253:1 0 447.1G 0 lvm # ls -l /dev/disk/by-*/ /dev/disk/by-id/: total 0 lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102524 -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102531 -> ../../sdc lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-name-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 -> ../../dm-0 lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-name-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 -> ../../dm-1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-uuid-LVM-GHM6Bwl9TQ7jv5GJd8ORRD6XDearTRZhgvpxQ22a3TWdlBd9iGk1oHhop5lXn8lL -> ../../dm-1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-uuid-LVM-hoOm4ydEDwKOrnNdVuCBCsY31it5n1ZRDsf4uP4Irce8u2hubaahZCqfMz9IpwhI -> ../../dm-0 lrwxrwxrwx 1 root root 9 Jun 29 09:49 lvm-pv-uuid-AGbSTn-aDmD-AbAR-ngCX-8glc-2KVW-xal2xh -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 09:49 lvm-pv-uuid-QPw8aR-Rbbe-LzZ7-0j3t-n8gn-OeOs-YWPaoV -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x5002538c00018347 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x5002538c00018347-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x5002538c00018347-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x5002538c00018347-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x5002538c40146ccb -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x5002538c40146cd2 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x55cd2e414db345fd -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x55cd2e414db345fd-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x55cd2e414db345fd-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x55cd2e414db345fd-part3 -> ../../sda3 /dev/disk/by-label/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 09:49 rpool -> ../../sda3 /dev/disk/by-partuuid/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 09:49 4f42744a-eef7-49f5-bfa4-5cb3ca1ee4b2 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 70166c71-7a1f-400e-bd39-f8f4be867d3e -> ../../sda3 lrwxrwxrwx 1 root root 10 Jun 29 09:49 87402126-9aa6-4be9-9c13-4704492a974b -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 a52ed3d9-d18c-4d5b-9d8a-c92b235fd9e1 -> ../../sdb3 lrwxrwxrwx 1 root root 10 Jun 29 09:49 de77a2cb-a1df-460e-97a2-3c8c8ae9fad5 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 fb306c92-2607-46a5-a32d-7556b04dd494 -> ../../sda2 /dev/disk/by-path/: total 0 lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-3 -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-3.0 -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3.0-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3.0-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3.0-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-4 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-4.0 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4.0-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4.0-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4.0-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-1 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-1.0 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-2 -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-2.0 -> ../../sdd /dev/disk/by-uuid/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 09:49 17716103480993325194 -> ../../sda3 lrwxrwxrwx 1 root root 10 Jun 29 09:49 B851-E178 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 09:49 B852-ACFC -> ../../sdb2 # iscsiadm -m node iscsiadm: No records found # iscsiadm -m session iscsiadm: No active sessions. ==== info about volumes ==== # pvs PV VG Fmt Attr PSize PFree /dev/sdc ceph-33bdcbd7-07be-4373-97ca-0678dda8888d lvm2 a-- <447.13g 0 /dev/sdd ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 lvm2 a-- <447.13g 0 # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd-block-e2deed6d-596f-4837-b14e-88c9afdbe531 ceph-33bdcbd7-07be-4373-97ca-0678dda8888d -wi-ao---- <447.13g osd-block-55199458-8b33-44f2-b4d2-3a876072a622 ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 -wi-ao---- <447.13g # vgs VG #PV #LV #SN Attr VSize VFree ceph-33bdcbd7-07be-4373-97ca-0678dda8888d 1 1 0 wz--n- <447.13g 0 ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 1 1 0 wz--n- <447.13g 0 # zpool status pool: rpool state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 00:00:19 with 0 errors on Sun Jun 13 00:24:20 2021 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 ONLINE 0 0 0 errors: No known data errors # zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT rpool 111G 2.43G 109G - - 5% 2% 1.00x ONLINE - ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 111G 2.43G 109G - - 5% 2.18% - ONLINE # zfs list NAME USED AVAIL REFER MOUNTPOINT rpool 2.43G 105G 104K /rpool rpool/ROOT 2.42G 105G 96K /rpool/ROOT rpool/ROOT/pve-1 2.42G 105G 2.42G / rpool/data 96K 105G 96K /rpool/data # pveceph status cluster: id: 73045ca5-eead-4e44-a0c1-b6796ed3d7d5 health: HEALTH_WARN client is using insecure global_id reclaim mons are allowing insecure global_id reclaim 4 slow ops, oldest one blocked for 4666 sec, mon.node04 has slow ops services: mon: 3 daemons, quorum node04,node05,node06 (age 36m) mgr: node04(active, since 97m), standbys: node05, node06 mds: CephFS:1 {0=node05=up:active} 2 up:standby osd: 6 osds: 6 up (since 37m), 6 in (since 37m) data: pools: 4 pools, 193 pgs objects: 96.62k objects, 365 GiB usage: 1.0 TiB used, 1.6 TiB / 2.6 TiB avail pgs: 193 active+clean io: client: 0 B/s rd, 20 KiB/s wr, 0 op/s rd, 3 op/s wr # ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 node04 154G 292G 1 7372 0 0 exists,up 1 node04 202G 244G 1 5733 0 0 exists,up 2 node05 163G 283G 0 7371 0 0 exists,up 3 node05 193G 253G 0 4095 0 0 exists,up 4 node06 180G 266G 0 4095 0 0 exists,up 5 node06 177G 270G 0 0 0 0 exists,up # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 2.6 TiB 1.6 TiB 1.0 TiB 1.0 TiB 39.96 TOTAL 2.6 TiB 1.6 TiB 1.0 TiB 1.0 TiB 39.96 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL Ceph 2 128 322 GiB 87.91k 965 GiB 42.04 444 GiB CephFS_data 3 32 34 GiB 8.68k 101 GiB 7.03 444 GiB CephFS_metadata 4 32 7.9 MiB 24 24 MiB 0 444 GiB device_health_metrics 5 1 28 MiB 6 84 MiB 0 444 GiB # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 2.61960 - 2.6 TiB 1.0 TiB 1.0 TiB 98 MiB 5.9 GiB 1.6 TiB 39.96 1.00 - root default -3 0.87320 - 894 GiB 357 GiB 355 GiB 43 MiB 2.0 GiB 537 GiB 39.96 1.00 - host node04 0 ssd 0.43660 1.00000 447 GiB 154 GiB 153 GiB 5.5 MiB 1018 MiB 293 GiB 34.53 0.86 82 up osd.0 1 ssd 0.43660 1.00000 447 GiB 203 GiB 202 GiB 38 MiB 986 MiB 244 GiB 45.38 1.14 111 up osd.1 -5 0.87320 - 894 GiB 357 GiB 355 GiB 30 MiB 2.0 GiB 537 GiB 39.96 1.00 - host node05 2 ssd 0.43660 1.00000 447 GiB 164 GiB 163 GiB 25 MiB 1.0 GiB 284 GiB 36.58 0.92 94 up osd.2 3 ssd 0.43660 1.00000 447 GiB 194 GiB 193 GiB 4.8 MiB 1019 MiB 253 GiB 43.34 1.08 99 up osd.3 -7 0.87320 - 894 GiB 357 GiB 355 GiB 25 MiB 2.0 GiB 537 GiB 39.96 1.00 - host node06 4 ssd 0.43660 1.00000 447 GiB 180 GiB 179 GiB 22 MiB 1002 MiB 267 GiB 40.32 1.01 97 up osd.4 5 ssd 0.43660 1.00000 447 GiB 177 GiB 176 GiB 2.9 MiB 1021 MiB 270 GiB 39.60 0.99 96 up osd.5 TOTAL 2.6 TiB 1.0 TiB 1.0 TiB 98 MiB 5.9 GiB 1.6 TiB 39.96 MIN/MAX VAR: 0.86/1.14 STDDEV: 3.70 # cat /etc/ceph/ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = fdb0:5bd1:dc::4/64 fsid = 73045ca5-eead-4e44-a0c1-b6796ed3d7d5 mon_allow_pool_delete = true mon_host = fdb0:5bd1:dc::4 fdb0:5bd1:dc::5 fdb0:5bd1:dc::6 ms_bind_ipv4 = false ms_bind_ipv6 = true osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = fdb0:5bd1:dc::4/64 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.node04] host = node04 mds_standby_for_name = pve [mds.node05] host = node05 mds_standby_for_name = pve [mds.node06] host = node06 mds standby for name = pve [mon.node04] public_addr = fdb0:5bd1:dc::4 [mon.node05] public_addr = fdb0:5bd1:dc::5 [mon.node06] public_addr = fdb0:5bd1:dc::6 # ceph config dump WHO MASK LEVEL OPTION VALUE RO mgr unknown mgr/dashboard/server_addr ::1 * mgr unknown mgr/dashboard/ssl false * # pveceph pool ls +-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+ | Name | Size | Min Size | PG Num | min. PG Num | Optimal PG Num | PG Autoscale Mode | PG Autoscale Target Size | PG Autoscale Target Ratio | Crush Rule Name | %-Used | Used | +=======================+======+==========+========+=============+================+===================+==========================+===========================+=================+======================+===============+ | Ceph | 3 | 2 | 128 | | 64 | on | | | replicated_rule | 0.420354932546616 | 1036478722631 | +-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+ | CephFS_data | 3 | 2 | 32 | | 32 | on | | | replicated_rule | 0.0703079700469971 | 108086587392 | +-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+ | CephFS_metadata | 3 | 2 | 32 | 16 | 16 | on | | | replicated_rule | 1.73890748556005e-05 | 24853666 | +-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+ | device_health_metrics | 3 | 2 | 1 | 1 | 1 | on | | | replicated_rule | 6.18295161984861e-05 | 88374925 | +-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+ # ceph versions { "mon": { "ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 2, "ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 1 }, "mgr": { "ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 2, "ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 1 }, "osd": { "ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 4, "ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 2 }, "mds": { "ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 2, "ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 1 }, "overall": { "ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 10, "ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 5 } } -------------- next part -------------- ==== general system info ==== # hostname node06 # pveversion --verbose proxmox-ve: 7.0-2 (running kernel: 5.11.22-1-pve) pve-manager: 7.0-5 (running version: 7.0-5/cce9b25f) pve-kernel-5.11: 7.0-3 pve-kernel-helper: 7.0-3 pve-kernel-5.4: 6.4-3 pve-kernel-5.11.22-1-pve: 5.11.22-1 pve-kernel-5.4.119-1-pve: 5.4.119-1 pve-kernel-5.4.106-1-pve: 5.4.106-1 pve-kernel-5.4.34-1-pve: 5.4.34-2 ceph: 15.2.13-pve1 ceph-fuse: 15.2.13-pve1 corosync: 3.1.2-pve2 criu: 3.15-1+pve-1 glusterfs-client: 9.2-1 ifupdown: residual config ifupdown2: 3.0.0-1+pve5 ksm-control-daemon: 1.4-1 libjs-extjs: 7.0.0-1 libknet1: 1.21-pve1 libproxmox-acme-perl: 1.1.0 libproxmox-backup-qemu0: 1.0.3-1 libpve-access-control: 7.0-3 libpve-apiclient-perl: 3.2-1 libpve-common-perl: 7.0-4 libpve-guest-common-perl: 4.0-2 libpve-http-server-perl: 4.0-2 libpve-storage-perl: 7.0-6 libqb0: 1.0.5-1 libspice-server1: 0.14.3-2.1 lvm2: 2.03.11-2.1 lxc-pve: 4.0.9-1 lxcfs: 4.0.8-pve1 novnc-pve: 1.2.0-3 proxmox-backup-client: 1.1.10-1 proxmox-backup-file-restore: 1.1.10-1 proxmox-mini-journalreader: 1.2-1 proxmox-widget-toolkit: 3.1-4 pve-cluster: 7.0-2 pve-container: 4.0-3 pve-docs: 7.0-3 pve-edk2-firmware: 3.20200531-1 pve-firewall: 4.2-2 pve-firmware: 3.2-4 pve-ha-manager: 3.2-2 pve-i18n: 2.3-1 pve-qemu-kvm: 6.0.0-2 pve-xtermjs: 4.12.0-1 qemu-server: 7.0-5 smartmontools: 7.2-pve2 spiceterm: 3.2-2 vncterm: 1.7-1 zfsutils-linux: 2.0.4-pve1 # cat /etc/hosts 127.0.0.1 localhost.localdomain localhost 2a03:7900:111::dc:6 node06.demo.customers.tuxis.net node06 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts #2a03:7900:111:0:90d4:a7ff:fe7f:ccf6 pbs.tuxis.net # pvesubscription get message: There is no subscription key serverid: F3A59435D1B87C5A2460F965646A3177 status: NotFound url: https://www.proxmox.com/proxmox-ve/pricing # cat /etc/apt/sources.list ## Managed via Ansible deb http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free deb-src http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free deb http://security.debian.org/ bullseye-security main contrib non-free deb-src http://security.debian.org/ bullseye-security main contrib non-free deb http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free deb-src http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free # cat /etc/apt/sources.list.d/pvetest-for-beta.list deb http://download.proxmox.com/debian/pve bullseye pvetest # cat /etc/apt/sources.list.d/ceph.list deb http://download.proxmox.com/debian/ceph-octopus bullseye main # cat /etc/apt/sources.list.d/apt_tuxis_nl_tuxis.list deb https://apt.tuxis.nl/tuxis/ tuxis-cron main deb https://apt.tuxis.nl/tuxis/ monitoring main deb https://apt.tuxis.nl/tuxis/ pmrb main # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 46 bits physical, 48 bits virtual CPU(s): 8 On-line CPU(s) list: 0-7 Thread(s) per core: 2 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 63 Model name: Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz Stepping: 2 CPU MHz: 1200.000 CPU max MHz: 3600.0000 CPU min MHz: 1200.0000 BogoMIPS: 6999.74 Virtualization: VT-x L1d cache: 128 KiB L1i cache: 128 KiB L2 cache: 1 MiB L3 cache: 10 MiB NUMA node0 CPU(s): 0-7 Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d # pvesh get /cluster/resources --type node --output-format=yaml --- - id: node/node04 node: node04 status: offline type: node - cpu: 0 disk: 2595225600 id: node/node06 level: '' maxcpu: 8 maxdisk: 115451232256 maxmem: 67322179584 mem: 1778470912 node: node06 status: online type: node uptime: 17 - id: node/node05 node: node05 status: offline type: node ==== overall system load info ==== # top -b -c -w512 -n 1 -o TIME | head -n 30 top - 10:30:25 up 1 min, 1 user, load average: 0.56, 0.23, 0.08 Tasks: 316 total, 3 running, 313 sleeping, 0 stopped, 0 zombie %Cpu(s): 1.5 us, 3.1 sy, 0.0 ni, 95.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 64203.4 total, 62239.5 free, 1723.6 used, 240.4 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 61675.9 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 165796 8924 5312 S 0.0 0.0 0:01.96 /sbin/init 1908 root 20 0 333916 125364 18596 S 0.0 0.2 0:00.63 /usr/bin/perl /usr/bin/pvesh --nooutput create /nodes/localhost/startall 1900 www-data 20 0 346652 126348 7264 S 0.0 0.2 0:00.53 pveproxy worker 1684 root rt 0 309108 177892 53256 R 0.0 0.3 0:00.52 /usr/sbin/corosync -f 1680 ceph 20 0 255064 19864 10604 S 0.0 0.0 0:00.35 /usr/bin/ceph-mgr -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 1681 ceph 20 0 409952 32068 18780 S 6.2 0.0 0:00.34 /usr/bin/ceph-mon -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 1679 ceph 20 0 250860 18892 10156 S 0.0 0.0 0:00.33 /usr/bin/ceph-mds -f --cluster ceph --id node06 --setuser ceph --setgroup ceph 1699 ceph 20 0 267296 20544 11224 S 0.0 0.0 0:00.31 /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph 31 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/3] 37 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/4] 43 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/5] 49 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/6] 55 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/7] 1701 ceph 20 0 267292 20304 11036 S 0.0 0.0 0:00.30 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph 19 root rt 0 0 0 0 S 0.0 0.0 0:00.29 [migration/1] 25 root rt 0 0 0 0 S 0.0 0.0 0:00.29 [migration/2] 1787 root 20 0 264744 84344 3988 S 0.0 0.1 0:00.28 pve-firewall 1901 www-data 20 0 346460 126360 7280 S 0.0 0.2 0:00.28 pveproxy worker 1813 root 20 0 345068 123768 6148 S 0.0 0.2 0:00.25 pvedaemon worker 1121 root 20 0 98428 3324 2640 S 0.0 0.0 0:00.23 /usr/sbin/zed -F 734 root 20 0 40140 14964 13888 S 0.0 0.0 0:00.22 /lib/systemd/systemd-journald 780 root 20 0 22124 3568 2380 S 0.0 0.0 0:00.19 /lib/systemd/systemd-udevd 1339 root 20 0 396620 18592 7216 S 0.0 0.0 0:00.17 /usr/bin/python3 /usr/bin/fail2ban-server -xf start # head /proc/pressure/* ==> /proc/pressure/cpu <== some avg10=0.00 avg60=0.24 avg300=0.15 total=727501 ==> /proc/pressure/io <== some avg10=0.00 avg60=0.10 avg300=0.06 total=376684 full avg10=0.00 avg60=0.08 avg300=0.05 total=320128 ==> /proc/pressure/memory <== some avg10=0.00 avg60=0.00 avg300=0.00 total=0 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 ==== info about storage ==== # cat /etc/pve/storage.cfg dir: local path /var/lib/vz content images,backup,iso,vztmpl prune-backups keep-last=1 shared 0 zfspool: local-zfs pool rpool/data content rootdir,images sparse 1 rbd: Ceph content rootdir,images krbd 0 pool Ceph cephfs: CephFS path /mnt/pve/CephFS content iso,snippets,backup,vztmpl prune-backups keep-last=1 dir: Tuxis_Marketplace path /mnt/pve/Tuxis_Marketplace content iso,backup is_mountpoint yes mkdir 0 shared 1 dir: Tuxis_Marketplace_Beta path /mnt/pve/Tuxis_Marketplace_Beta content backup,iso is_mountpoint yes mkdir 0 shared 1 rbd: CephKRBD content images krbd 1 pool Ceph pbs: pbs002.tuxis.nl datastore DB0220_demo server pbs002.tuxis.nl content backup encryption-key 68:d5:89:f6:f1:f4:67:59:1b:74:6a:78:99:11:ad:09:a0:b0:12:db:43:8d:41:19:af:38:90:77:12:c1:6d:f8 fingerprint 45:f8:79:eb:27:96:88:6b:29:ad:21:00:13:c6:bd:b8:30:f6:f3:9b:f0:bf:dd:f3:ad:f0:09:d5:d2:9a:34:79 prune-backups keep-last=1 username DB0220 at pbs # pvesm status got timeout ERROR: command 'pvesm status' failed: got timeout # cat /etc/fstab # proc /proc proc defaults 0 0 # findmnt --ascii TARGET SOURCE FSTYPE OPTIONS / rpool/ROOT/pve-1 zfs rw,relatime,xattr,noacl |-/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime | |-/sys/kernel/security securityfs securityfs rw,nosuid,nodev,noexec,relatime | |-/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime | |-/sys/fs/pstore pstore pstore rw,nosuid,nodev,noexec,relatime | |-/sys/fs/bpf none bpf rw,nosuid,nodev,noexec,relatime,mode=700 | |-/sys/kernel/debug debugfs debugfs rw,nosuid,nodev,noexec,relatime | |-/sys/kernel/tracing tracefs tracefs rw,nosuid,nodev,noexec,relatime | |-/sys/fs/fuse/connections fusectl fusectl rw,nosuid,nodev,noexec,relatime | `-/sys/kernel/config configfs configfs rw,nosuid,nodev,noexec,relatime |-/proc proc proc rw,relatime | `-/proc/sys/fs/binfmt_misc systemd-1 autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=23036 |-/dev udev devtmpfs rw,nosuid,relatime,size=32838164k,nr_inodes=8209541,mode=755,inode64 | |-/dev/pts devpts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 | |-/dev/shm tmpfs tmpfs rw,nosuid,nodev,inode64 | |-/dev/hugepages hugetlbfs hugetlbfs rw,relatime,pagesize=2M | `-/dev/mqueue mqueue mqueue rw,nosuid,nodev,noexec,relatime |-/run tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=6574432k,mode=755,inode64 | |-/run/lock tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 | |-/run/rpc_pipefs sunrpc rpc_pipefs rw,relatime | `-/run/user/0 tmpfs tmpfs rw,nosuid,nodev,relatime,size=6574428k,nr_inodes=1643607,mode=700,inode64 |-/rpool rpool zfs rw,noatime,xattr,noacl | |-/rpool/ROOT rpool/ROOT zfs rw,noatime,xattr,noacl | `-/rpool/data rpool/data zfs rw,noatime,xattr,noacl |-/var/lib/ceph/osd/ceph-4 tmpfs tmpfs rw,relatime,inode64 |-/var/lib/ceph/osd/ceph-5 tmpfs tmpfs rw,relatime,inode64 |-/mnt/pve/Tuxis_Marketplace s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0 |-/mnt/pve/Tuxis_Marketplace_Beta s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0 |-/etc/pve /dev/fuse fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other `-/var/lib/lxcfs lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other # df --human Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G 1.3M 6.3G 1% /run rpool/ROOT/pve-1 108G 2.5G 106G 3% / tmpfs 32G 66M 32G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock rpool 106G 128K 106G 1% /rpool rpool/ROOT 106G 128K 106G 1% /rpool/ROOT rpool/data 106G 128K 106G 1% /rpool/data tmpfs 32G 24K 32G 1% /var/lib/ceph/osd/ceph-4 tmpfs 32G 24K 32G 1% /var/lib/ceph/osd/ceph-5 s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace_Beta /dev/fuse 30M 40K 30M 1% /etc/pve tmpfs 6.3G 4.0K 6.3G 1% /run/user/0 ==== info about virtual guests ==== # qm list # pct list ==== info about network ==== # ip -details -statistics address 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 729230 6382 0 0 0 0 TX: bytes packets errors dropped carrier collsns 729230 6382 0 0 0 0 2: eno1: mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000 link/ether 0c:c4:7a:d9:1d:f6 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9216 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp6s0 RX: bytes packets errors dropped missed mcast 507910 4689 0 3 0 270 TX: bytes packets errors dropped carrier collsns 425694 1188 0 0 0 0 3: eno2: mtu 1500 qdisc mq master vmbr999 state UP group default qlen 1000 link/ether 0c:c4:7a:d9:1d:f7 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9216 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp7s0 RX: bytes packets errors dropped missed mcast 720 8 0 0 0 8 TX: bytes packets errors dropped carrier collsns 36292 370 0 0 0 0 4: vmbr0: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:ed:27:6f:7b:f3 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 204.43 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet6 2a03:7900:111::dc:6/64 scope global valid_lft forever preferred_lft forever inet6 fe80::50ed:27ff:fe6f:7bf3/64 scope link valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 439912 4664 0 0 0 267 TX: bytes packets errors dropped carrier collsns 414944 1063 0 0 0 0 5: vmbr999: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 16:2d:db:6c:6d:8a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 1 vlan_protocol 802.1Q bridge_id 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 204.52 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet6 fdb0:5bd1:dc::6/64 scope global valid_lft forever preferred_lft forever inet6 fe80::142d:dbff:fe6c:6d8a/64 scope link valid_lft forever preferred_lft forever RX: bytes packets errors dropped missed mcast 608 8 0 0 0 8 TX: bytes packets errors dropped carrier collsns 36292 370 0 0 0 0 # ip -details -4 route show # ip -details -6 route show unicast ::1 dev lo proto kernel scope global metric 256 pref medium unicast 2a03:7900:111::/64 dev vmbr0 proto kernel scope global metric 256 pref medium unicast fdb0:5bd1:dc::/64 dev vmbr999 proto kernel scope global metric 256 pref medium unicast fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff dev vmbr999 proto boot scope global metric 1024 pref medium unicast fe80::/64 dev vmbr0 proto kernel scope global metric 256 pref medium unicast fe80::/64 dev vmbr999 proto kernel scope global metric 256 pref medium unicast default via 2a03:7900:111::1 dev vmbr0 proto kernel scope global metric 1024 onlink pref medium # cat /etc/network/interfaces # network interface settings; autogenerated # Please do NOT modify this file directly, unless you know what # you're doing. # # If you want to manage parts of the network configuration manually, # please utilize the 'source' or 'source-directory' directives to do # so. # PVE will preserve these directives, but will NOT read its network # configuration from sourced files, so do not attempt to move any of # the PVE managed interfaces into external files! auto lo iface lo inet loopback iface eno1 inet6 manual iface eno2 inet6 manual auto vmbr0 iface vmbr0 inet6 static address 2a03:7900:111::dc:6/64 gateway 2a03:7900:111::1 bridge-ports eno1 bridge-stp off bridge-fd 0 auto vmbr999 iface vmbr999 inet6 static address fdb0:5bd1:dc::6/64 bridge-ports eno2 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 post-up /usr/sbin/ip ro add fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff ==== info about firewall ==== # cat /etc/pve/local/host.fw cat: /etc/pve/local/host.fw: No such file or directory # iptables-save # Generated by iptables-save v1.8.7 on Tue Jun 29 10:30:36 2021 *raw :PREROUTING ACCEPT [376:131554] :OUTPUT ACCEPT [281:128086] COMMIT # Completed on Tue Jun 29 10:30:36 2021 # Generated by iptables-save v1.8.7 on Tue Jun 29 10:30:36 2021 *filter :INPUT ACCEPT [281:128086] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [282:128126] COMMIT # Completed on Tue Jun 29 10:30:36 2021 ==== info about cluster ==== # pvecm nodes Membership information ---------------------- Nodeid Votes Name 3 1 node06 (local) # pvecm status Cluster information ------------------- Name: Demo Config Version: 3 Transport: knet Secure auth: on Quorum information ------------------ Date: Tue Jun 29 10:30:37 2021 Quorum provider: corosync_votequorum Nodes: 1 Node ID: 0x00000003 Ring ID: 3.f4c Quorate: No Votequorum information ---------------------- Expected votes: 3 Highest expected: 3 Total votes: 1 Quorum: 2 Activity blocked Flags: Membership information ---------------------- Nodeid Votes Name 0x00000003 1 fdb0:5bd1:dc::6%32732 (local) # cat /etc/pve/corosync.conf 2>/dev/null logging { debug: off to_syslog: yes } nodelist { node { name: node04 nodeid: 1 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::4 } node { name: node05 nodeid: 2 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::5 } node { name: node06 nodeid: 3 quorum_votes: 1 ring0_addr: fdb0:5bd1:dc::6 } } quorum { provider: corosync_votequorum } totem { cluster_name: Demo config_version: 3 interface { linknumber: 0 } ip_version: ipv4-6 link_mode: passive secauth: on version: 2 } # ha-manager status quorum No quorum on node 'node06'! master node06 (old timestamp - dead?, Tue Jun 9 16:20:27 2020) lrm node04 (old timestamp - dead?, Tue Jun 29 10:27:38 2021) lrm node05 (old timestamp - dead?, Tue Jun 29 10:27:38 2021) lrm node06 (old timestamp - dead?, Tue Jun 29 10:27:37 2021) ==== info about hardware ==== # dmidecode -t bios # dmidecode 3.3 Getting SMBIOS data from sysfs. SMBIOS 3.0 present. Handle 0x0000, DMI type 0, 24 bytes BIOS Information Vendor: American Megatrends Inc. Version: 2.0a Release Date: 08/01/2016 Address: 0xF0000 Runtime Size: 64 kB ROM Size: 16 MB Characteristics: PCI is supported BIOS is upgradeable BIOS shadowing is allowed Boot from CD is supported Selectable boot is supported BIOS ROM is socketed EDD is supported 5.25"/1.2 MB floppy services are supported (int 13h) 3.5"/720 kB floppy services are supported (int 13h) 3.5"/2.88 MB floppy services are supported (int 13h) Print screen service is supported (int 5h) 8042 keyboard services are supported (int 9h) Serial services are supported (int 14h) Printer services are supported (int 17h) ACPI is supported USB legacy is supported BIOS boot specification is supported Targeted content distribution is supported UEFI is supported BIOS Revision: 5.6 # lspci -nnk 00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [8086:2f00] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [15d9:0832] 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f02] (rev 02) Kernel driver in use: pcieport 00:01.1 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f03] (rev 02) Kernel driver in use: pcieport 00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02) Kernel driver in use: pcieport 00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f0a] (rev 02) Kernel driver in use: pcieport 00:04.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [8086:2f20] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [8086:2f21] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [8086:2f22] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [8086:2f23] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [8086:2f24] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [8086:2f25] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [8086:2f26] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:04.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [8086:2f27] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [15d9:0832] Kernel driver in use: ioatdma Kernel modules: ioatdma 00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [8086:2f28] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [15d9:0832] 00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [8086:2f29] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [15d9:0832] 00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [8086:2f2a] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [15d9:0832] 00:05.4 PIC [0800]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [8086:2f2c] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [15d9:0832] 00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR [8086:8d7c] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] 00:11.4 SATA controller [0106]: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] [8086:8d62] (rev 05) Subsystem: Super Micro Computer Inc C610/X99 series chipset sSATA Controller [AHCI mode] [15d9:0832] Kernel driver in use: ahci Kernel modules: ahci 00:14.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: xhci_hcd Kernel modules: xhci_pci 00:16.0 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel modules: mei_me 00:16.1 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #2 [8086:8d3b] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] 00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 [8086:8d10] (rev d5) Kernel driver in use: pcieport 00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 [8086:8d18] (rev d5) Kernel driver in use: pcieport 00:1c.5 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #6 [8086:8d1a] (rev d5) Kernel driver in use: pcieport 00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 [8086:8d1c] (rev d5) Kernel driver in use: pcieport 00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ehci-pci Kernel modules: ehci_pci 00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series chipset LPC Controller [8086:8d44] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: lpc_ich Kernel modules: lpc_ich 00:1f.2 SATA controller [0106]: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [8086:8d02] (rev 05) Subsystem: Super Micro Computer Inc C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [15d9:0832] Kernel driver in use: ahci Kernel modules: ahci 00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset SMBus Controller [8086:8d22] (rev 05) Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: i801_smbus Kernel modules: i2c_i801 06:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) DeviceName: Intel Ethernet i210AT #1 Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533] Kernel driver in use: igb Kernel modules: igb 07:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) DeviceName: Intel Ethernet i210AT #2 Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533] Kernel driver in use: igb Kernel modules: igb 08:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03) 09:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30) DeviceName: ASPEED Video AST2400 Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832] Kernel driver in use: ast Kernel modules: ast ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f81] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f36] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] Kernel driver in use: hswep_uncore ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f37] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832] Kernel driver in use: hswep_uncore ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe1] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe2] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe3] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832] ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff8] (rev 02) ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff9] (rev 02) ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffc] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffd] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffe] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832] ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f1d] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832] ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f34] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832] Kernel driver in use: hswep_uncore ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1e] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f7d] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1f] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832] ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2fa0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832] ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2f30] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832] Kernel driver in use: hswep_uncore ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832] ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832] ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2faa] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fab] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fac] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fad] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832] ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast [8086:2fae] (rev 02) ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2faf] (rev 02) ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [8086:2fb0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [8086:2fb1] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [8086:2fb2] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [15d9:0832] ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [8086:2fb3] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [15d9:0832] ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbc] (rev 02) ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbd] (rev 02) ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbe] (rev 02) ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbf] (rev 02) ff:15.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [8086:2fb4] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:15.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [8086:2fb5] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [15d9:0832] Kernel driver in use: hswep_uncore ff:15.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [8086:2fb6] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [15d9:0832] ff:15.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [8086:2fb7] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [15d9:0832] ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers [8086... (rev 02) ff:16.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast [8086:2f6e] (rev 02) ff:16.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2f6f] (rev 02) ff:17.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control [8086:2fd0] (rev 02) Kernel driver in use: hswep_uncore ff:17.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb8] (rev 02) ff:17.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb9] (rev 02) ff:17.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fba] (rev 02) ff:17.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fbb] (rev 02) ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f98] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f99] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9a] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2fc0] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9c] (rev 02) Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832] ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f88] (rev 02) ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f8a] (rev 02) ==== info about block devices ==== # lsblk --ascii NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 111.8G 0 disk |-sda1 8:1 0 1007K 0 part |-sda2 8:2 0 512M 0 part `-sda3 8:3 0 111.3G 0 part sdb 8:16 0 111.8G 0 disk |-sdb1 8:17 0 1007K 0 part |-sdb2 8:18 0 512M 0 part `-sdb3 8:19 0 111.3G 0 part sdc 8:32 0 447.1G 0 disk `-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 253:0 0 447.1G 0 lvm sdd 8:48 0 447.1G 0 disk `-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 253:1 0 447.1G 0 lvm # ls -l /dev/disk/by-*/ /dev/disk/by-id/: total 0 lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102524 -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102531 -> ../../sdc lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-name-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 -> ../../dm-0 lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-name-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 -> ../../dm-1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-uuid-LVM-GHM6Bwl9TQ7jv5GJd8ORRD6XDearTRZhgvpxQ22a3TWdlBd9iGk1oHhop5lXn8lL -> ../../dm-1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-uuid-LVM-hoOm4ydEDwKOrnNdVuCBCsY31it5n1ZRDsf4uP4Irce8u2hubaahZCqfMz9IpwhI -> ../../dm-0 lrwxrwxrwx 1 root root 9 Jun 29 10:29 lvm-pv-uuid-AGbSTn-aDmD-AbAR-ngCX-8glc-2KVW-xal2xh -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 10:29 lvm-pv-uuid-QPw8aR-Rbbe-LzZ7-0j3t-n8gn-OeOs-YWPaoV -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x5002538c00018347 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x5002538c00018347-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x5002538c00018347-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x5002538c00018347-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x5002538c40146ccb -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x5002538c40146cd2 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x55cd2e414db345fd -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x55cd2e414db345fd-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x55cd2e414db345fd-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x55cd2e414db345fd-part3 -> ../../sda3 /dev/disk/by-label/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 10:29 rpool -> ../../sda3 /dev/disk/by-partuuid/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 10:29 4f42744a-eef7-49f5-bfa4-5cb3ca1ee4b2 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 70166c71-7a1f-400e-bd39-f8f4be867d3e -> ../../sda3 lrwxrwxrwx 1 root root 10 Jun 29 10:29 87402126-9aa6-4be9-9c13-4704492a974b -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 a52ed3d9-d18c-4d5b-9d8a-c92b235fd9e1 -> ../../sdb3 lrwxrwxrwx 1 root root 10 Jun 29 10:29 de77a2cb-a1df-460e-97a2-3c8c8ae9fad5 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 fb306c92-2607-46a5-a32d-7556b04dd494 -> ../../sda2 /dev/disk/by-path/: total 0 lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-3 -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-3.0 -> ../../sda lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3.0-part1 -> ../../sda1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3.0-part2 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3.0-part3 -> ../../sda3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-4 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-4.0 -> ../../sdb lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4.0-part1 -> ../../sdb1 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4.0-part2 -> ../../sdb2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4.0-part3 -> ../../sdb3 lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-1 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-1.0 -> ../../sdc lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-2 -> ../../sdd lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-2.0 -> ../../sdd /dev/disk/by-uuid/: total 0 lrwxrwxrwx 1 root root 10 Jun 29 10:29 17716103480993325194 -> ../../sda3 lrwxrwxrwx 1 root root 10 Jun 29 10:29 B851-E178 -> ../../sda2 lrwxrwxrwx 1 root root 10 Jun 29 10:29 B852-ACFC -> ../../sdb2 # iscsiadm -m node iscsiadm: No records found # iscsiadm -m session iscsiadm: No active sessions. ==== info about volumes ==== # pvs PV VG Fmt Attr PSize PFree /dev/sdc ceph-33bdcbd7-07be-4373-97ca-0678dda8888d lvm2 a-- <447.13g 0 /dev/sdd ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 lvm2 a-- <447.13g 0 # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd-block-e2deed6d-596f-4837-b14e-88c9afdbe531 ceph-33bdcbd7-07be-4373-97ca-0678dda8888d -wi-a----- <447.13g osd-block-55199458-8b33-44f2-b4d2-3a876072a622 ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 -wi-a----- <447.13g # vgs VG #PV #LV #SN Attr VSize VFree ceph-33bdcbd7-07be-4373-97ca-0678dda8888d 1 1 0 wz--n- <447.13g 0 ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 1 1 0 wz--n- <447.13g 0 # zpool status pool: rpool state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 00:00:19 with 0 errors on Sun Jun 13 00:24:20 2021 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 ONLINE 0 0 0 errors: No known data errors # zpool list -v NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT rpool 111G 2.43G 109G - - 5% 2% 1.00x ONLINE - ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 111G 2.43G 109G - - 5% 2.18% - ONLINE # zfs list NAME USED AVAIL REFER MOUNTPOINT rpool 2.43G 105G 104K /rpool rpool/ROOT 2.42G 105G 96K /rpool/ROOT rpool/ROOT/pve-1 2.42G 105G 2.42G / rpool/data 96K 105G 96K /rpool/data # pveceph status ERROR: command 'pveceph status' failed: got timeout # ceph osd status ERROR: command 'ceph osd status' failed: got timeout # ceph df ERROR: command 'ceph df' failed: got timeout # ceph osd df tree ERROR: command 'ceph osd df tree' failed: got timeout # cat /etc/ceph/ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = fdb0:5bd1:dc::4/64 fsid = 73045ca5-eead-4e44-a0c1-b6796ed3d7d5 mon_allow_pool_delete = true mon_host = fdb0:5bd1:dc::4 fdb0:5bd1:dc::5 fdb0:5bd1:dc::6 ms_bind_ipv4 = false ms_bind_ipv6 = true osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = fdb0:5bd1:dc::4/64 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.node04] host = node04 mds_standby_for_name = pve [mds.node05] host = node05 mds_standby_for_name = pve [mds.node06] host = node06 mds standby for name = pve [mon.node04] public_addr = fdb0:5bd1:dc::4 [mon.node05] public_addr = fdb0:5bd1:dc::5 [mon.node06] public_addr = fdb0:5bd1:dc::6 # ceph config dump ERROR: command 'ceph config dump' failed: got timeout # pveceph pool ls got timeout # ceph versions ERROR: command 'ceph versions' failed: got timeout From s.ivanov at proxmox.com Tue Jun 29 11:58:03 2021 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Tue, 29 Jun 2021 11:58:03 +0200 Subject: [PVE-User] Fw: Proxmox VE 7.0 (beta) released! Message-ID: <20210629115803.3b4633af@rosa.proxmox.com> Forwarding to list - mistakenly did not reply-all: Begin forwarded message: Date: Tue, 29 Jun 2021 10:50:40 +0200 From: Stoiko Ivanov To: Mark Schouten Subject: Re: [PVE-User] Proxmox VE 7.0 (beta) released! Hi Mark, Thanks! hm - intel nics - a few questions: * did you run this machine with pve 6.X with the pve-kernel-5.4 or with pve-kernel-5.11 - and did it work? * if possible could you boot this/a similar machine with pve 6.x and pve-kernel-5.11 (this might help us narrow this down to kernel vs. remaining system) * my colleagues and I have gut-feeling, that it might be related to: https://sources.debian.org/src/bridge-utils/1.7-1/debian/NEWS/#L3-L23 (see also the release-notes: https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_7.0_beta1) * could you try to remove the persistent mac-address? (copy /lib/systemd/network/99-default.link to /etc/systemd/network/ and comment out the last line and reboot) Thanks! stoiko From mark at tuxis.nl Tue Jun 29 12:06:02 2021 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 29 Jun 2021 12:06:02 +0200 Subject: [PVE-User] Proxmox VE 7.0 (beta) released! In-Reply-To: References: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> Message-ID: Hi, Op 29-06-2021 om 11:46 schreef Thomas Lamprecht: > Do you have some FW rules regarding MAC-Addresses or the like? > As the MAC-Address selection changed in Proxmox VE 7 due to new default > n systemd's network link policy, as listed in our known issues[0]. There is no firewall configured on this cluster. On Stoiko's advice, I changed the systemd-link-settings and now everything works again. I do not completely understand why that fixes it though. Commenting out MACAddressPolicy=persistent helps, but why? -- Mark Schouten CTO, Tuxis B.V. | https://www.tuxis.nl/ | +31 318 200208 From t.lamprecht at proxmox.com Tue Jun 29 12:31:14 2021 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Tue, 29 Jun 2021 12:31:14 +0200 Subject: [PVE-User] Proxmox VE 7.0 (beta) released! In-Reply-To: References: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> Message-ID: On 29.06.21 12:06, Mark Schouten wrote: > Hi, > > Op 29-06-2021 om 11:46 schreef Thomas Lamprecht: >> Do you have some FW rules regarding MAC-Addresses or the like? >> As the MAC-Address selection changed in Proxmox VE 7 due to new default >> n systemd's network link policy, as listed in our known issues[0]. > > There is no firewall configured on this cluster. On Stoiko's advice, I changed the systemd-link-settings and now everything works again. Ah yeah, that advice was not posted to the list initially so I did not saw that... > > I do not completely understand why that fixes it though.? Commenting out MACAddressPolicy=persistent helps, but why? > Because duplicate MAC addresses are not ideal, to say the least? I.e., quoting the second part of my original reply again: > It's now not the one of the first port anymore, but derived from interface > name and `/etc/machine-id`, which in combination should be unique but also > persistent. > > But, for some ISO releases (4.0 to 5.3) the machine-id for the installed host > was not always re-generated, which could result in duplication of a MAC for > identical named interfaces on two hosts. > We try to actively catch and fix that on upgrade by checking if the ID is one > of the known static ones (it's just a handful of possible IDs) on upgrade. > > But if one cloned an machine (e.g., a colleague run into this in a demo > virtualized PVE test clusters they cloned from a template) that ID will be > duplicated and thus make problems. > That could be easily checked by comparing the `/etc/machine-id` content and > be fixed by re-generation[1]. > From mark at tuxis.nl Tue Jun 29 14:04:05 2021 From: mark at tuxis.nl (Mark Schouten) Date: Tue, 29 Jun 2021 14:04:05 +0200 Subject: [PVE-User] Proxmox VE 7.0 (beta) released! In-Reply-To: References: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> Message-ID: <152e5dc5-8b0c-f182-4d85-1e1b3639209a@tuxis.nl> Hi, Op 29-06-2021 om 12:31 schreef Thomas Lamprecht: >> I do not completely understand why that fixes it though.? Commenting out MACAddressPolicy=persistent helps, but why? >> > > Because duplicate MAC addresses are not ideal, to say the least? That I understand. :) But, the cluster interface works when bridge_vlan_aware is off, regardless of the MacAddressPolicy setting. -- Mark Schouten CTO, Tuxis B.V. | https://www.tuxis.nl/ | +31 318 200208 From w.bumiller at proxmox.com Tue Jun 29 14:27:52 2021 From: w.bumiller at proxmox.com (Wolfgang Bumiller) Date: Tue, 29 Jun 2021 14:27:52 +0200 (CEST) Subject: [PVE-User] Proxmox VE 7.0 (beta) released! Message-ID: <14296680.254.1624969672688@webmail.proxmox.com> > On 06/29/2021 2:04 PM Mark Schouten wrote: > > > Hi, > > Op 29-06-2021 om 12:31 schreef Thomas Lamprecht: > >> I do not completely understand why that fixes it though.? Commenting out MACAddressPolicy=persistent helps, but why? > >> > > > > Because duplicate MAC addresses are not ideal, to say the least? > > That I understand. :) > > But, the cluster interface works when bridge_vlan_aware is off, > regardless of the MacAddressPolicy setting. Yep, this may actually need more investigation, as I also had this issue on a single PVE VM on my ArchLinux host. - definitely no duplicate mac addresses there - no MAC related firewall settings - network traffic *routed* off of a bridge on the host (so the final physical nic being an intel one should also not influence this) - works when disabling `bridge-vlan-aware` - still works when enabling vlan filtering via /sys after the fact - also works with MACAddressPolicy commented out *regardless* of `bridge-vlan-aware`... Also tried using systemd-networkd for the bridge in place of ifupdown2. Same behavior when toggling `VLANFiltering` in the [Bridge] section... Also note that similar to manually enabling vlan filtering via /sys, simply enabling `VLANFiltering` and restarting `systemd-networkd` does not actually break it, only if I delete the bridge first and then let systemd-network recreate it from scratch it'll be broken... Curious stuff... From s.ivanov at proxmox.com Tue Jun 29 15:31:11 2021 From: s.ivanov at proxmox.com (Stoiko Ivanov) Date: Tue, 29 Jun 2021 15:31:11 +0200 Subject: [PVE-User] Proxmox VE 7.0 (beta) released! In-Reply-To: <152e5dc5-8b0c-f182-4d85-1e1b3639209a@tuxis.nl> References: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> <152e5dc5-8b0c-f182-4d85-1e1b3639209a@tuxis.nl> Message-ID: <20210629153111.2a0fbc28@rosa.proxmox.com> On Tue, 29 Jun 2021 14:04:05 +0200 Mark Schouten wrote: > Hi, > > Op 29-06-2021 om 12:31 schreef Thomas Lamprecht: > >> I do not completely understand why that fixes it though.? Commenting out MACAddressPolicy=persistent helps, but why? > >> > > > > Because duplicate MAC addresses are not ideal, to say the least? > > That I understand. :) > > But, the cluster interface works when bridge_vlan_aware is off, > regardless of the MacAddressPolicy setting. > We managed to find a reproducer - my current guess is that it might have something to do with intel NIC drivers or some changes in ifupdown2 (or udev, or in their interaction ;) - Sadly if tcpdump fixes the issues, it makes debugging quite hard :) In any case - as can also be seen in the 2 reports you sent: with vlan-aware bridges the promisc flag of the ethernet interface (the bridge-port) is set to 0, when vlan-aware is not present it is set to 1. This explains the symptoms you're seeing, and why running tcpdump fixes it FWIW: I think simply starting a guest would have fixed the issue as well (when a second bridge_port gets added the kernel sets the promisc flag on the port correctly) As Wolfgang wrote - we'll look into it and will hopefully come up with a sensible solution. Thanks for the beta-test and the report! From aderumier at odiso.com Tue Jun 29 15:51:06 2021 From: aderumier at odiso.com (alexandre derumier) Date: Tue, 29 Jun 2021 15:51:06 +0200 Subject: [PVE-User] Proxmox VE 7.0 (beta) released! In-Reply-To: <20210629153111.2a0fbc28@rosa.proxmox.com> References: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> <152e5dc5-8b0c-f182-4d85-1e1b3639209a@tuxis.nl> <20210629153111.2a0fbc28@rosa.proxmox.com> Message-ID: <0790b6524457e986e6f14176af5e2ee9d88a5d97.camel@odiso.com> Hi, I have found a bug report about promisc && vlan-aware on mellanox site, of a proxmox user. (proxmox 6.4 with kernel (5.12?) https://community.mellanox.com/s/question/0D51T00008ansfP/vlan-aware-linux-bridging-is-not-functional-on-connectx4lx-card-unless-manually-put-in-promiscuous-mode So maybe is it a kernel change ? I don't known is that ifupdown2 has been bumped to last master ? (I don't have seen special commit of this kind of change) Le mardi 29 juin 2021 ? 15:31 +0200, Stoiko Ivanov a ?crit?: > On Tue, 29 Jun 2021 14:04:05 +0200 > Mark Schouten wrote: > > > Hi, > > > > Op 29-06-2021 om 12:31 schreef Thomas Lamprecht: > > > > I do not completely understand why that fixes it though.? > > > > Commenting out MACAddressPolicy=persistent helps, but why? > > > > ? > > > > > > > That I understand. :) > > > > But, the cluster interface works when bridge_vlan_aware is off, > > regardless of the MacAddressPolicy setting. > > > > We managed to find a reproducer - my current guess is that it might > have > something to do with intel NIC drivers or some changes in ifupdown2 (or > udev, or in their interaction ;) - Sadly if tcpdump fixes the issues, > it > makes debugging quite hard :) > > In any case - as can also be seen in the 2 reports you sent: > with vlan-aware bridges the promisc flag of the ethernet interface (the > bridge-port) is set to 0, when vlan-aware is not present it is set to > 1. > This explains the symptoms you're seeing, and why running tcpdump fixes > it > > FWIW: I think simply starting a guest would have fixed the issue as > well > (when a second bridge_port gets added the kernel sets the promisc flag > on > the port correctly) > > As Wolfgang wrote - we'll look into it and will hopefully come up with > a > sensible solution. > > Thanks for the beta-test and the report! > > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user From t.lamprecht at proxmox.com Tue Jun 29 16:14:50 2021 From: t.lamprecht at proxmox.com (Thomas Lamprecht) Date: Tue, 29 Jun 2021 16:14:50 +0200 Subject: [PVE-User] Proxmox VE 7.0 (beta) released! In-Reply-To: <20210629153111.2a0fbc28@rosa.proxmox.com> References: <5377d815-bde4-9ca8-8584-ff63a6eb27ba@proxmox.com> <0d129a03-9a70-e123-5e5a-e7862ef303ac@tuxis.nl> <152e5dc5-8b0c-f182-4d85-1e1b3639209a@tuxis.nl> <20210629153111.2a0fbc28@rosa.proxmox.com> Message-ID: On 29.06.21 15:31, Stoiko Ivanov wrote: > On Tue, 29 Jun 2021 14:04:05 +0200 > Mark Schouten wrote: > >> Hi, >> >> Op 29-06-2021 om 12:31 schreef Thomas Lamprecht: >>>> I do not completely understand why that fixes it though.? Commenting out MACAddressPolicy=persistent helps, but why? >>>> >>> >>> Because duplicate MAC addresses are not ideal, to say the least? >> >> That I understand. :) >> >> But, the cluster interface works when bridge_vlan_aware is off, >> regardless of the MacAddressPolicy setting. >> > > We managed to find a reproducer - my current guess is that it might have > something to do with intel NIC drivers or some changes in ifupdown2 (or > udev, or in their interaction ;) - Sadly if tcpdump fixes the issues, it > makes debugging quite hard :) The issue is that the kernel always (since close to forever) cleared the bridge's promisc mode when there was either no port or exactly one port with flood or learning enabled in the `br_manage_promisc` function. Further, on toggeling VLAN-aware the aforementioned `br_manage_promisc` is called from `br_vlan_filter_toggle` So, why does this breaks now? I really do not think it's due to some driver-specific stuff, not impossible but the following sounds like a better explanation about the "why now": Previously the MAC address of the bridge was the same as the one from the single port, so there it didn't matter to much if promisc was on on the single port itself, the bridge could accept the packages. But now, with the systemd default MACAddresPolicy "persistent" now also applying to bridges, the bridge gets a different MAC than the port, which means the disabled promisc matters on that port quite a bit more. So vlan-aware on "breaks" it by mistake, as then a br_manage_promisc call is made at a time where the "clear promisc for port" logic triggers, so rather a side-effect than a real cause. I quite tempted to drop the br_auto_port special case for the single port case in the kernel as fix, but need to think about this - and probably will send that to LKML first to poke for some comments... From gaio at sv.lnf.it Wed Jun 30 11:28:47 2021 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Wed, 30 Jun 2021 11:28:47 +0200 Subject: [PVE-User] Guest LVM2 overhead/usefulness for thin storages (Ceph, ZFS) Message-ID: <20210630092847.GG3262@sv.lnf.it> A collegue here came from the old school of 'LVM everything', that surely make sense for 'phisical servers'. Also, point me that make sense also for some 'virtual servers' (or better, 'virtual storage') setup. EG, considering a SAN, splitting virtual volumes in predefined chunks (1/2TB) and 'recombine' them via LVM in the guest, permit to move around smaller volumes, transparently for the guest. But if we came to 'modern', thin storage, like ZFS or Ceph, this still make sense? Seems 'no' to me, seems to me only useful to add another layer of abstraction... and the same functionality can be achived splitting data in virtual disks as if was volumes, format with a single partition per disk, and eventually extend the disk... And, this layer, how does it 'cost'? Not only in the term of performance, but also functionality... eg, 'trim' can traverse correctly all the layers? I hope i was clear. Thanks. PS: i've tried to look at the wiki but found nothing; if i've missed something, point me to the doc! -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From piccardi at truelite.it Wed Jun 30 14:01:41 2021 From: piccardi at truelite.it (Simone Piccardi) Date: Wed, 30 Jun 2021 14:01:41 +0200 Subject: [PVE-User] Guest LVM2 overhead/usefulness for thin storages (Ceph, ZFS) In-Reply-To: <20210630092847.GG3262@sv.lnf.it> References: <20210630092847.GG3262@sv.lnf.it> Message-ID: On 30/06/21 11:28, Marco Gaiarin wrote: > Also, point me that make sense also for some 'virtual servers' (or > better, 'virtual storage') setup. > EG, considering a SAN, splitting virtual volumes in predefined chunks > (1/2TB) and 'recombine' them via LVM in the guest, permit to move > around smaller volumes, transparently for the guest. > My personal preference, for VM, is having two partition in a single virtual disk (the first one for the swap, the second one for /). And then resize the disk. If you want to have a separate /home (I see the need only if you want to apply quotas) I just use a second disk. I cannot see any advantage on using LVM inside a VM. Simone -- Simone Piccardi Truelite Srl piccardi at truelite.it (email/jabber) Via Monferrato, 6 Tel. +39-347-1032433 50142 Firenze http://www.truelite.it Tel. +39-055-7879597 From smr at kmi.com Wed Jun 30 14:20:07 2021 From: smr at kmi.com (Stefan M. Radman) Date: Wed, 30 Jun 2021 12:20:07 +0000 Subject: lvmthin woes Message-ID: <7DB04451-5D1E-4ECE-B974-41184828DE15@kmi.com> Hi Today tried to migrate a disk from a thin pool to another thin pool on PVE 6.4-6 but online storage migration failed at 54%. When I attempted the migration a second time, the reason for the first failure became apparent from the error message. WARNING: Remaining free space in metadata of thin pool hdd/data is too low (100.00% >= 96.00%). Resize is recommended. I had run out of metadata on the destination thin pool during storage migration. Jun 30 13:10:41 hrd-srv lvm[857]: WARNING: Thin pool hdd-data-tpool metadata is now 80.97% full. Jun 30 13:11:31 hrd-srv lvm[857]: WARNING: Thin pool hdd-data-tpool metadata is now 85.80% full. Jun 30 13:12:21 hrd-srv lvm[857]: WARNING: Thin pool hdd-data-tpool metadata is now 90.48% full. Jun 30 13:13:11 hrd-srv lvm[857]: WARNING: Thin pool hdd-data-tpool metadata is now 95.46% full. Jun 30 13:13:58 hrd-srv kernel: device-mapper: thin: No free metadata blocks Jun 30 13:13:58 hrd-srv kernel: device-mapper: thin: 253:3: switching pool to read-only mode Jun 30 13:13:59 hrd-srv pvedaemon[20516]: VM 103 qmp command failed - VM 103 qmp command 'block-job-cancel' failed - Block job 'drive-scsi2' not found Jun 30 13:13:59 hrd-srv kernel: device-mapper: thin: 253:3: unable to service pool target messages in READ_ONLY or FAIL mode Jun 30 13:13:59 hrd-srv pvedaemon[20516]: lvremove 'hdd/vm-103-disk-0' error: Failed to update pool hdd/data. Jun 30 13:13:59 hrd-srv pvedaemon[20516]: storage migration failed: block job (mirror) error: drive-scsi2: 'mirror' has been cancelled Jun 30 13:13:59 hrd-srv pvedaemon[13249]: end task UPID:hrd-srv:00005024:15A37C83:60DC42CC:qmmove:103:root at pam: storage migration failed: block job (mirror) error: drive-scsi2: 'mirror' has been cancelled Jun 30 13:14:01 hrd-srv lvm[857]: WARNING: Thin pool hdd-data-tpool metadata is now 100.00% full. Resizing the metadata pool (lvresize --poolmetadatasize ..) worked without a problem (thanks to the hint in the wiki). [3630930.387272] device-mapper: thin: No free metadata blocks [3630930.393240] device-mapper: thin: 253:3: switching pool to read-only mode [3630931.632043] device-mapper: thin: 253:3: unable to service pool target messages in READ_ONLY or FAIL mode [3633079.042694] device-mapper: thin: 253:3: growing the metadata device from 25600 to 51200 blocks [3633079.042699] device-mapper: thin: 253:3: switching pool to write mode [3633462.550930] device-mapper: thin: 253:3: growing the metadata device from 51200 to 76800 blocks [3633489.659827] device-mapper: thin: 253:3: growing the metadata device from 76800 to 90112 blocks [3633497.712154] device-mapper: thin: 253:3: growing the metadata device from 90112 to 103424 blocks [3633514.576483] device-mapper: thin: 253:3: growing the metadata device from 103424 to 116736 blocks [3633531.417242] device-mapper: thin: 253:3: growing the metadata device from 116736 to 130048 blocks [3633552.498115] device-mapper: thin: 253:3: growing the metadata device from 130048 to 143360 blocks [3633563.294272] device-mapper: thin: 253:3: growing the metadata device from 143360 to 156672 blocks [3633573.562695] device-mapper: thin: 253:3: growing the metadata device from 156672 to 169984 blocks [3633595.843620] device-mapper: thin: 253:3: growing the metadata device from 169984 to 262144 blocks The pool is writable again and not overcommitted. root at hrd-srv:~# lvs -a hdd LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert data hdd twi-aotz-- 1.00t 16.86 10.36 [data_tdata] hdd Twi-ao---- 1.00t [data_tmeta] hdd ewi-ao---- 1.00g [lvol0_pmspare] hdd ewi------- 100.00m vm-100-disk-0 hdd Vwi-aotz-- 500.00g data 23.02 vm-102-disk-1 hdd Vwi-a-tz-- 14.90g data 22.66 vz hdd -wi-ao---- 1.00t root at pve:~# vgs hdd VG #PV #LV #SN Attr VSize VFree hdd 4 4 0 wz--n- <3.64t <1.64t Nevertheless, the third storage migration attempt failed with error Thin pool hdd-data-tpool (253:3) transaction_id is 17, while expected 18. Any advice on how to go about this situation? Thank you Stefan FIRST ATTEMPT create full clone of drive scsi2 (ssd:vm-103-disk-2) Logical volume "vm-103-disk-0" created. drive mirror is starting for drive-scsi2 drive-scsi2: transferred 33.0 MiB of 100.0 GiB (0.03%) in 0s ... drive-scsi2: transferred 54.1 GiB of 100.0 GiB (54.11%) in 4m 41s drive-scsi2: Cancelling block job drive-scsi2: Done. device-mapper: message ioctl on (253:3) failed: Operation not supported Failed to process message "delete 3". Failed to suspend hdd/data with queued messages. lvremove 'hdd/vm-103-disk-0' error: Failed to update pool hdd/data. TASK ERROR: storage migration failed: block job (mirror) error: drive-scsi2: 'mirror' has been cancelled SECOND ATTEMPT create full clone of drive scsi2 (ssd:vm-103-disk-2) WARNING: Remaining free space in metadata of thin pool hdd/data is too low (100.00% >= 96.00%). Resize is recommended. TASK ERROR: storage migration failed: lvcreate 'hdd/vm-103-disk-0' error: Cannot create new thin volume, free space in thin pool hdd/data reached threshold. I ran out of metadata on a thin pool today create full clone of drive scsi2 (ssd:vm-103-disk-2) Thin pool hdd-data-tpool (253:3) transaction_id is 17, while expected 18. TASK ERROR: storage migration failed: lvcreate 'hdd/vm-103-disk-0' error: Failed to suspend hdd/data with queued messages. THIRD ATTEMPT create full clone of drive scsi2 (ssd:vm-103-disk-2) Thin pool hdd-data-tpool (253:3) transaction_id is 17, while expected 18. TASK ERROR: storage migration failed: lvcreate 'hdd/vm-103-disk-0' error: Failed to suspend hdd/data with queued messages. CONFIDENTIALITY NOTICE: This communication may contain privileged and confidential information, or may otherwise be protected from disclosure, and is intended solely for use of the intended recipient(s). If you are not the intended recipient of this communication, please notify the sender that you have received this communication in error and delete and destroy all copies in your possession. From gaio at sv.lnf.it Wed Jun 30 15:07:34 2021 From: gaio at sv.lnf.it (Marco Gaiarin) Date: Wed, 30 Jun 2021 15:07:34 +0200 Subject: [PVE-User] Guest LVM2 overhead/usefulness for thin storages (Ceph, ZFS) In-Reply-To: References: <20210630092847.GG3262@sv.lnf.it> Message-ID: <20210630130734.GQ3262@sv.lnf.it> Mandi! Simone Piccardi via pve-user In chel di` si favelave... > My personal preference, for VM, is having two partition in a single virtual > disk (the first one for the swap, the second one for /). And then resize the > disk. > If you want to have a separate /home (I see the need only if you want to > apply quotas) I just use a second disk. > I cannot see any advantage on using LVM inside a VM. Exactly what i meant, exactly what i suppose, thanks Simone. -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) From gianni.milo22 at gmail.com Wed Jun 30 15:40:06 2021 From: gianni.milo22 at gmail.com (Yanni M.) Date: Wed, 30 Jun 2021 14:40:06 +0100 Subject: [PVE-User] Guest LVM2 overhead/usefulness for thin storages (Ceph, ZFS) In-Reply-To: <20210630092847.GG3262@sv.lnf.it> References: <20210630092847.GG3262@sv.lnf.it> Message-ID: Personally I favor the flexibility that LVM offers when it comes to creating/resizing volumes (growing/shrinking), most of times without even needing to reboot the vm. Even if it's possible to do the same with disk partitioning, it's not as trivial as with lvm (hence why it was invented in the first place). Having to create a new vdisk on the host for each individual volume that you might need for the guest in the future - even though possible - it does not look as clean solution as doing all these, with lvm, within the guest. The overhead of the lvm is minimal and the discard/trim should be already enabled by default. Special care should be taken when using ThinLVM to avoid situations where the pools get full etc (same apply when used at the host level). Apart from that, it does not require any special administration skills, apart from familiarity with lvm tools obviously. To be clear, I'm not defending lvm in here. Both your and this method are ok. It's all about with what each person is familiar with and how each organise their work. On Wed, 30 Jun 2021 at 10:29, Marco Gaiarin wrote: > > A collegue here came from the old school of 'LVM everything', that > surely make sense for 'phisical servers'. > > Also, point me that make sense also for some 'virtual servers' (or > better, 'virtual storage') setup. > EG, considering a SAN, splitting virtual volumes in predefined chunks > (1/2TB) and 'recombine' them via LVM in the guest, permit to move > around smaller volumes, transparently for the guest. > > > But if we came to 'modern', thin storage, like ZFS or Ceph, this still > make sense? > Seems 'no' to me, seems to me only useful to add another layer of > abstraction... and the same functionality can be achived splitting data > in virtual disks as if was volumes, format with a single partition per > disk, and eventually extend the disk... > > And, this layer, how does it 'cost'? Not only in the term of > performance, but also functionality... eg, 'trim' can traverse correctly > all the layers? > > > I hope i was clear. Thanks. > > > PS: i've tried to look at the wiki but found nothing; if i've missed > something, point me to the doc! > > -- > dott. Marco Gaiarin GNUPG Key ID: > 240A3D66 > Associazione ``La Nostra Famiglia'' > http://www.lanostrafamiglia.it/ > Polo FVG - Via della Bont?, 7 - 33078 - San Vito al Tagliamento > (PN) > marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f > +39-0434-842797 > > Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! > http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 > (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA) > > _______________________________________________ > pve-user mailing list > pve-user at lists.proxmox.com > https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-user > >