[PVE-User] Proxmox VE 7.0 (beta) released!
Mark Schouten
mark at tuxis.nl
Tue Jun 29 10:34:28 CEST 2021
Hi Stoiko,
Op 29-06-2021 om 10:23 schreef Stoiko Ivanov:
>> I just upgraded a node in our demo cluster and all seemed fine. Except
>> for non-working cluster network. I was unable to ping the node through
>> the cluster interface, pvecm saw no other nodes and ceph was broken.
> Thanks for the report - could you provide some details on the upgraded
> node? Mostly which NICs are used - but also the complete hardware -setup
> > (If you prefer you can send me a pvereport to my e-mail)
See attached!
--
Mark Schouten
CTO, Tuxis B.V. | https://www.tuxis.nl/
<mark at tuxis.nl> | +31 318 200208
-------------- next part --------------
==== general system info ====
# hostname
node06
# pveversion --verbose
proxmox-ve: 7.0-2 (running kernel: 5.11.22-1-pve)
pve-manager: 7.0-5 (running version: 7.0-5/cce9b25f)
pve-kernel-5.11: 7.0-3
pve-kernel-helper: 7.0-3
pve-kernel-5.4: 6.4-3
pve-kernel-5.11.22-1-pve: 5.11.22-1
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 15.2.13-pve1
ceph-fuse: 15.2.13-pve1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.0.0-1+pve5
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 7.0-3
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-4
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-6
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-1
lxcfs: 4.0.8-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 1.1.10-1
proxmox-backup-file-restore: 1.1.10-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.1-4
pve-cluster: 7.0-2
pve-container: 4.0-3
pve-docs: 7.0-3
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.2-2
pve-i18n: 2.3-1
pve-qemu-kvm: 6.0.0-2
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-5
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.4-pve1
# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
2a03:7900:111::dc:6 node06.demo.customers.tuxis.net node06
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
#2a03:7900:111:0:90d4:a7ff:fe7f:ccf6 pbs.tuxis.net
# pvesubscription get
message: There is no subscription key
serverid: F3A59435D1B87C5A2460F965646A3177
status: NotFound
url: https://www.proxmox.com/proxmox-ve/pricing
# cat /etc/apt/sources.list
## Managed via Ansible
deb http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free
deb-src http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free
deb http://security.debian.org/ bullseye-security main contrib non-free
deb-src http://security.debian.org/ bullseye-security main contrib non-free
deb http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free
deb-src http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free
# cat /etc/apt/sources.list.d/pvetest-for-beta.list
deb http://download.proxmox.com/debian/pve bullseye pvetest
# cat /etc/apt/sources.list.d/ceph.list
deb http://download.proxmox.com/debian/ceph-octopus bullseye main
# cat /etc/apt/sources.list.d/apt_tuxis_nl_tuxis.list
deb https://apt.tuxis.nl/tuxis/ tuxis-cron main
deb https://apt.tuxis.nl/tuxis/ monitoring main
deb https://apt.tuxis.nl/tuxis/ pmrb main
# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz
Stepping: 2
CPU MHz: 3600.000
CPU max MHz: 3600.0000
CPU min MHz: 1200.0000
BogoMIPS: 7000.21
Virtualization: VT-x
L1d cache: 128 KiB
L1i cache: 128 KiB
L2 cache: 1 MiB
L3 cache: 10 MiB
NUMA node0 CPU(s): 0-7
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d
# pvesh get /cluster/resources --type node --output-format=yaml
---
- cpu: 0.011763712868024
disk: 2595618816
id: node/node06
level: ''
maxcpu: 8
maxdisk: 115451363328
maxmem: 67322179584
mem: 3120074752
node: node06
status: online
type: node
uptime: 2238
- cpu: 0.188962970750996
disk: 21875785728
id: node/node04
level: ''
maxcpu: 8
maxdisk: 101597184000
maxmem: 67331584000
mem: 23079858176
node: node04
status: online
type: node
uptime: 13969251
- cpu: 0.0102829414157591
disk: 3339059200
id: node/node05
level: ''
maxcpu: 8
maxdisk: 115422265344
maxmem: 67331592192
mem: 8491147264
node: node05
status: online
type: node
uptime: 2830727
==== overall system load info ====
# top -b -c -w512 -n 1 -o TIME | head -n 30
top - 10:26:43 up 37 min, 1 user, load average: 0.13, 0.08, 0.06
Tasks: 295 total, 1 running, 294 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.8 us, 1.5 sy, 0.0 ni, 97.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 64203.4 total, 60470.7 free, 2954.7 used, 778.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 60187.0 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1647 root rt 0 571448 178096 53256 S 6.2 0.3 0:53.45 /usr/sbin/corosync -f
1663 ceph 20 0 1400308 570700 28140 S 0.0 0.9 0:39.35 /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph
1656 ceph 20 0 1298416 546100 28064 S 0.0 0.8 0:27.35 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph
1644 ceph 20 0 507260 123516 20868 S 0.0 0.2 0:24.52 /usr/bin/ceph-mon -f --cluster ceph --id node06 --setuser ceph --setgroup ceph
1762 root 20 0 270424 95672 9780 S 0.0 0.1 0:23.76 pvestatd
1888 www-data 20 0 346704 127472 8076 S 0.0 0.2 0:09.48 pveproxy worker
1887 www-data 20 0 346736 126700 7424 S 0.0 0.2 0:08.86 pveproxy worker
1889 www-data 20 0 346604 126552 7296 S 0.0 0.2 0:07.14 pveproxy worker
1763 root 20 0 264748 84656 4284 S 0.0 0.1 0:06.55 pve-firewall
1550 root 20 0 618448 55796 50792 S 0.0 0.1 0:05.45 /usr/bin/pmxcfs
1643 ceph 20 0 503992 176144 19884 S 0.0 0.3 0:04.03 /usr/bin/ceph-mgr -f --cluster ceph --id node06 --setuser ceph --setgroup ceph
1789 root 20 0 345064 124220 6524 S 0.0 0.2 0:02.80 pvedaemon worker
13 root 20 0 0 0 0 I 0.0 0.0 0:02.48 [rcu_sched]
1788 root 20 0 345068 123856 6204 S 0.0 0.2 0:02.40 pvedaemon worker
1790 root 20 0 345064 123804 6168 S 0.0 0.2 0:02.40 pvedaemon worker
1 root 20 0 165772 8880 5280 S 0.0 0.0 0:02.17 /sbin/init
1335 Debian-+ 20 0 24352 10532 5780 S 0.0 0.0 0:01.40 /usr/sbin/snmpd -LOw -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf -f -p /run/snmpd.pid
1642 ceph 20 0 302080 27288 11188 S 0.0 0.0 0:00.85 /usr/bin/ceph-mds -f --cluster ceph --id node06 --setuser ceph --setgroup ceph
421 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss]
422 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss]
423 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss]
424 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss]
425 root 1 -19 0 0 0 S 0.0 0.0 0:00.82 [z_wr_iss]
# head /proc/pressure/*
==> /proc/pressure/cpu <==
some avg10=0.00 avg60=0.00 avg300=0.00 total=5430541
==> /proc/pressure/io <==
some avg10=0.00 avg60=0.00 avg300=0.00 total=1300222
full avg10=0.00 avg60=0.00 avg300=0.00 total=1149011
==> /proc/pressure/memory <==
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
==== info about storage ====
# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content images,backup,iso,vztmpl
prune-backups keep-last=1
shared 0
zfspool: local-zfs
pool rpool/data
content rootdir,images
sparse 1
rbd: Ceph
content rootdir,images
krbd 0
pool Ceph
cephfs: CephFS
path /mnt/pve/CephFS
content iso,snippets,backup,vztmpl
prune-backups keep-last=1
dir: Tuxis_Marketplace
path /mnt/pve/Tuxis_Marketplace
content iso,backup
is_mountpoint yes
mkdir 0
shared 1
dir: Tuxis_Marketplace_Beta
path /mnt/pve/Tuxis_Marketplace_Beta
content backup,iso
is_mountpoint yes
mkdir 0
shared 1
rbd: CephKRBD
content images
krbd 1
pool Ceph
pbs: pbs002.tuxis.nl
datastore DB0220_demo
server pbs002.tuxis.nl
content backup
encryption-key 68:d5:89:f6:f1:f4:67:59:1b:74:6a:78:99:11:ad:09:a0:b0:12:db:43:8d:41:19:af:38:90:77:12:c1:6d:f8
fingerprint 45:f8:79:eb:27:96:88:6b:29:ad:21:00:13:c6:bd:b8:30:f6:f3:9b:f0:bf:dd:f3:ad:f0:09:d5:d2:9a:34:79
prune-backups keep-last=1
username DB0220 at pbs
# pvesm status
Name Type Status Total Used Available %
Ceph rbd active 802501642 337252874 465248768 42.03%
CephFS cephfs active 500432896 35184640 465248256 7.03%
CephKRBD rbd active 802501642 337252874 465248768 42.03%
Tuxis_Marketplace dir active 274877906944 0 274877906944 0.00%
Tuxis_Marketplace_Beta dir active 274877906944 0 274877906944 0.00%
local dir active 112745472 2534784 110210688 2.25%
local-zfs zfspool active 110210824 96 110210728 0.00%
pbs002.tuxis.nl pbs active 1701798656 10516096 1691282560 0.62%
# cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# findmnt --ascii
TARGET SOURCE FSTYPE OPTIONS
/ rpool/ROOT/pve-1 zfs rw,relatime,xattr,noacl
|-/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime
| |-/sys/kernel/security securityfs securityfs rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/pstore pstore pstore rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/bpf none bpf rw,nosuid,nodev,noexec,relatime,mode=700
| |-/sys/kernel/debug debugfs debugfs rw,nosuid,nodev,noexec,relatime
| |-/sys/kernel/tracing tracefs tracefs rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/fuse/connections fusectl fusectl rw,nosuid,nodev,noexec,relatime
| `-/sys/kernel/config configfs configfs rw,nosuid,nodev,noexec,relatime
|-/proc proc proc rw,relatime
| `-/proc/sys/fs/binfmt_misc systemd-1 autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=561
|-/dev udev devtmpfs rw,nosuid,relatime,size=32838164k,nr_inodes=8209541,mode=755,inode64
| |-/dev/pts devpts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
| |-/dev/shm tmpfs tmpfs rw,nosuid,nodev,inode64
| |-/dev/mqueue mqueue mqueue rw,nosuid,nodev,noexec,relatime
| `-/dev/hugepages hugetlbfs hugetlbfs rw,relatime,pagesize=2M
|-/run tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=6574432k,mode=755,inode64
| |-/run/lock tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64
| |-/run/rpc_pipefs sunrpc rpc_pipefs rw,relatime
| `-/run/user/0 tmpfs tmpfs rw,nosuid,nodev,relatime,size=6574428k,nr_inodes=1643607,mode=700,inode64
|-/rpool rpool zfs rw,noatime,xattr,noacl
| |-/rpool/ROOT rpool/ROOT zfs rw,noatime,xattr,noacl
| `-/rpool/data rpool/data zfs rw,noatime,xattr,noacl
|-/var/lib/ceph/osd/ceph-4 tmpfs tmpfs rw,relatime,inode64
|-/var/lib/ceph/osd/ceph-5 tmpfs tmpfs rw,relatime,inode64
|-/mnt/pve/Tuxis_Marketplace_Beta s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0
|-/mnt/pve/Tuxis_Marketplace s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0
|-/etc/pve /dev/fuse fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
|-/var/lib/lxcfs lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
`-/mnt/pve/CephFS [fdb0:5bd1:dc::4],[fdb0:5bd1:dc::5],[fdb0:5bd1:dc::6]:/ ceph rw,relatime,name=admin,secret=<hidden>,acl
# df --human
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 1.3M 6.3G 1% /run
rpool/ROOT/pve-1 108G 2.5G 106G 3% /
tmpfs 32G 63M 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
rpool 106G 128K 106G 1% /rpool
rpool/ROOT 106G 128K 106G 1% /rpool/ROOT
rpool/data 106G 128K 106G 1% /rpool/data
tmpfs 32G 28K 32G 1% /var/lib/ceph/osd/ceph-4
tmpfs 32G 28K 32G 1% /var/lib/ceph/osd/ceph-5
s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace_Beta
s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace
/dev/fuse 30M 40K 30M 1% /etc/pve
[fdb0:5bd1:dc::4],[fdb0:5bd1:dc::5],[fdb0:5bd1:dc::6]:/ 478G 34G 444G 8% /mnt/pve/CephFS
tmpfs 6.3G 4.0K 6.3G 1% /run/user/0
==== info about network ====
# ip -details -statistics address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
RX: bytes packets errors dropped missed mcast
55254397 58681 0 0 0 0
TX: bytes packets errors dropped carrier collsns
55254397 58681 0 0 0 0
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether 0c:c4:7a:d9:1d:f6 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9216
bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535
altname enp6s0
RX: bytes packets errors dropped missed mcast
15982113 105257 0 74 0 5642
TX: bytes packets errors dropped carrier collsns
12624090 36080 0 0 0 0
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr999 state UP group default qlen 1000
link/ether 0c:c4:7a:d9:1d:f7 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9216
bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535
altname enp7s0
RX: bytes packets errors dropped missed mcast
489871250 699399 0 0 0 33
TX: bytes packets errors dropped carrier collsns
268886467 574470 0 0 0 0
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:ed:27:6f:7b:f3 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 23.76 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet6 2a03:7900:111::dc:6/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::50ed:27ff:fe6f:7bf3/64 scope link
valid_lft forever preferred_lft forever
RX: bytes packets errors dropped missed mcast
14379299 103674 0 0 0 5565
TX: bytes packets errors dropped carrier collsns
12356802 32972 0 0 0 0
5: vmbr999: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 16:2d:db:6c:6d:8a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 160.33 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet6 fdb0:5bd1:dc::6/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::142d:dbff:fe6c:6d8a/64 scope link
valid_lft forever preferred_lft forever
RX: bytes packets errors dropped missed mcast
465681104 499419 0 0 0 33
TX: bytes packets errors dropped carrier collsns
260240217 473931 0 0 0 0
# ip -details -4 route show
# ip -details -6 route show
unicast ::1 dev lo proto kernel scope global metric 256 pref medium
unicast 2a03:7900:111::/64 dev vmbr0 proto kernel scope global metric 256 pref medium
unicast fdb0:5bd1:dc::/64 dev vmbr999 proto kernel scope global metric 256 pref medium
unicast fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff dev vmbr999 proto boot scope global metric 1024 pref medium
unicast fe80::/64 dev vmbr999 proto kernel scope global metric 256 pref medium
unicast fe80::/64 dev vmbr0 proto kernel scope global metric 256 pref medium
unicast default via 2a03:7900:111::1 dev vmbr0 proto kernel scope global metric 1024 onlink pref medium
# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
auto lo
iface lo inet loopback
iface eno1 inet6 manual
iface eno2 inet6 manual
auto vmbr0
iface vmbr0 inet6 static
address 2a03:7900:111::dc:6/64
gateway 2a03:7900:111::1
bridge-ports eno1
bridge-stp off
bridge-fd 0
auto vmbr999
iface vmbr999 inet6 static
address fdb0:5bd1:dc::6/64
bridge-ports eno2
bridge-stp off
bridge-fd 0
#bridge-vlan-aware yes
#bridge-vids 2-4094
post-up /usr/sbin/ip ro add fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff
==== info about virtual guests ====
# qm list
# pct list
==== info about firewall ====
# cat /etc/pve/local/host.fw
cat: /etc/pve/local/host.fw: No such file or directory
# iptables-save
# Generated by iptables-save v1.8.7 on Tue Jun 29 10:26:45 2021
*raw
:PREROUTING ACCEPT [10644:3275736]
:OUTPUT ACCEPT [8292:3186298]
COMMIT
# Completed on Tue Jun 29 10:26:45 2021
# Generated by iptables-save v1.8.7 on Tue Jun 29 10:26:45 2021
*filter
:INPUT ACCEPT [8292:3186298]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [8293:3186338]
COMMIT
# Completed on Tue Jun 29 10:26:45 2021
==== info about cluster ====
# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
1 1 node04
2 1 node05
3 1 node06 (local)
# pvecm status
Cluster information
-------------------
Name: Demo
Config Version: 3
Transport: knet
Secure auth: on
Quorum information
------------------
Date: Tue Jun 29 10:26:46 2021
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000003
Ring ID: 1.f47
Quorate: Yes
Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 fdb0:5bd1:dc::4%32617
0x00000002 1 fdb0:5bd1:dc::5%32617
0x00000003 1 fdb0:5bd1:dc::6%32617 (local)
# cat /etc/pve/corosync.conf 2>/dev/null
logging {
debug: off
to_syslog: yes
}
nodelist {
node {
name: node04
nodeid: 1
quorum_votes: 1
ring0_addr: fdb0:5bd1:dc::4
}
node {
name: node05
nodeid: 2
quorum_votes: 1
ring0_addr: fdb0:5bd1:dc::5
}
node {
name: node06
nodeid: 3
quorum_votes: 1
ring0_addr: fdb0:5bd1:dc::6
}
}
quorum {
provider: corosync_votequorum
}
totem {
cluster_name: Demo
config_version: 3
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}
# ha-manager status
quorum OK
master node06 (idle, Tue Jun 9 16:20:27 2020)
lrm node04 (idle, Tue Jun 29 10:26:43 2021)
lrm node05 (idle, Tue Jun 29 10:26:43 2021)
lrm node06 (idle, Tue Jun 29 10:26:41 2021)
==== info about hardware ====
# dmidecode -t bios
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.0 present.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: American Megatrends Inc.
Version: 2.0a
Release Date: 08/01/2016
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 16 MB
Characteristics:
PCI is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
ACPI is supported
USB legacy is supported
BIOS boot specification is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 5.6
# lspci -nnk
00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [8086:2f00] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [15d9:0832]
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f02] (rev 02)
Kernel driver in use: pcieport
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f03] (rev 02)
Kernel driver in use: pcieport
00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02)
Kernel driver in use: pcieport
00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f0a] (rev 02)
Kernel driver in use: pcieport
00:04.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [8086:2f20] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [8086:2f21] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [8086:2f22] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [8086:2f23] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [8086:2f24] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [8086:2f25] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [8086:2f26] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [8086:2f27] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [8086:2f28] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [15d9:0832]
00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [8086:2f29] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [15d9:0832]
00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [8086:2f2a] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [15d9:0832]
00:05.4 PIC [0800]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [8086:2f2c] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [15d9:0832]
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR [8086:8d7c] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
00:11.4 SATA controller [0106]: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] [8086:8d62] (rev 05)
Subsystem: Super Micro Computer Inc C610/X99 series chipset sSATA Controller [AHCI mode] [15d9:0832]
Kernel driver in use: ahci
Kernel modules: ahci
00:14.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
00:16.0 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel modules: mei_me
00:16.1 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #2 [8086:8d3b] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
Kernel driver in use: pcieport
00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
Kernel driver in use: pcieport
00:1c.5 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #6 [8086:8d1a] (rev d5)
Kernel driver in use: pcieport
00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 [8086:8d1c] (rev d5)
Kernel driver in use: pcieport
00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series chipset LPC Controller [8086:8d44] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: lpc_ich
Kernel modules: lpc_ich
00:1f.2 SATA controller [0106]: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [8086:8d02] (rev 05)
Subsystem: Super Micro Computer Inc C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [15d9:0832]
Kernel driver in use: ahci
Kernel modules: ahci
00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset SMBus Controller [8086:8d22] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
06:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
DeviceName: Intel Ethernet i210AT #1
Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533]
Kernel driver in use: igb
Kernel modules: igb
07:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
DeviceName: Intel Ethernet i210AT #2
Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533]
Kernel driver in use: igb
Kernel modules: igb
08:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03)
09:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30)
DeviceName: ASPEED Video AST2400
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: ast
Kernel modules: ast
ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f81] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832]
ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f36] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832]
Kernel driver in use: hswep_uncore
ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f37] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832]
Kernel driver in use: hswep_uncore
ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe0] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832]
ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe1] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832]
ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe2] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832]
ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe3] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832]
ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff8] (rev 02)
ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff9] (rev 02)
ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffc] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832]
ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffd] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832]
ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffe] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832]
ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f1d] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832]
ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f34] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832]
Kernel driver in use: hswep_uncore
ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1e] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832]
ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f7d] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832]
ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1f] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832]
ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2fa0] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832]
ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2f30] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832]
Kernel driver in use: hswep_uncore
ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832]
ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832]
ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2faa] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832]
ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fab] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832]
ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fac] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832]
ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fad] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832]
ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast [8086:2fae] (rev 02)
ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2faf] (rev 02)
ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [8086:2fb0] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [15d9:0832]
Kernel driver in use: hswep_uncore
ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [8086:2fb1] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [15d9:0832]
Kernel driver in use: hswep_uncore
ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [8086:2fb2] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [15d9:0832]
ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [8086:2fb3] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [15d9:0832]
ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbc] (rev 02)
ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbd] (rev 02)
ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbe] (rev 02)
ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbf] (rev 02)
ff:15.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [8086:2fb4] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [15d9:0832]
Kernel driver in use: hswep_uncore
ff:15.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [8086:2fb5] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [15d9:0832]
Kernel driver in use: hswep_uncore
ff:15.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [8086:2fb6] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [15d9:0832]
ff:15.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [8086:2fb7] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [15d9:0832]
ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers [8086... (rev 02)
ff:16.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast [8086:2f6e] (rev 02)
ff:16.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2f6f] (rev 02)
ff:17.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control [8086:2fd0] (rev 02)
Kernel driver in use: hswep_uncore
ff:17.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb8] (rev 02)
ff:17.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb9] (rev 02)
ff:17.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fba] (rev 02)
ff:17.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fbb] (rev 02)
ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f98] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832]
ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f99] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832]
ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9a] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832]
ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2fc0] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832]
ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9c] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832]
ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f88] (rev 02)
ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f8a] (rev 02)
==== info about block devices ====
# lsblk --ascii
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 111.8G 0 disk
|-sda1 8:1 0 1007K 0 part
|-sda2 8:2 0 512M 0 part
`-sda3 8:3 0 111.3G 0 part
sdb 8:16 0 111.8G 0 disk
|-sdb1 8:17 0 1007K 0 part
|-sdb2 8:18 0 512M 0 part
`-sdb3 8:19 0 111.3G 0 part
sdc 8:32 0 447.1G 0 disk
`-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 253:0 0 447.1G 0 lvm
sdd 8:48 0 447.1G 0 disk
`-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 253:1 0 447.1G 0 lvm
# ls -l /dev/disk/by-*/
/dev/disk/by-id/:
total 0
lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jun 29 09:49 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102524 -> ../../sdd
lrwxrwxrwx 1 root root 9 Jun 29 09:49 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102531 -> ../../sdc
lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-name-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-name-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 -> ../../dm-1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-uuid-LVM-GHM6Bwl9TQ7jv5GJd8ORRD6XDearTRZhgvpxQ22a3TWdlBd9iGk1oHhop5lXn8lL -> ../../dm-1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 dm-uuid-LVM-hoOm4ydEDwKOrnNdVuCBCsY31it5n1ZRDsf4uP4Irce8u2hubaahZCqfMz9IpwhI -> ../../dm-0
lrwxrwxrwx 1 root root 9 Jun 29 09:49 lvm-pv-uuid-AGbSTn-aDmD-AbAR-ngCX-8glc-2KVW-xal2xh -> ../../sdc
lrwxrwxrwx 1 root root 9 Jun 29 09:49 lvm-pv-uuid-QPw8aR-Rbbe-LzZ7-0j3t-n8gn-OeOs-YWPaoV -> ../../sdd
lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x5002538c00018347 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x5002538c00018347-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x5002538c00018347-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x5002538c00018347-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x5002538c40146ccb -> ../../sdd
lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x5002538c40146cd2 -> ../../sdc
lrwxrwxrwx 1 root root 9 Jun 29 09:49 wwn-0x55cd2e414db345fd -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x55cd2e414db345fd-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x55cd2e414db345fd-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 29 09:49 wwn-0x55cd2e414db345fd-part3 -> ../../sda3
/dev/disk/by-label/:
total 0
lrwxrwxrwx 1 root root 10 Jun 29 09:49 rpool -> ../../sda3
/dev/disk/by-partuuid/:
total 0
lrwxrwxrwx 1 root root 10 Jun 29 09:49 4f42744a-eef7-49f5-bfa4-5cb3ca1ee4b2 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 70166c71-7a1f-400e-bd39-f8f4be867d3e -> ../../sda3
lrwxrwxrwx 1 root root 10 Jun 29 09:49 87402126-9aa6-4be9-9c13-4704492a974b -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 a52ed3d9-d18c-4d5b-9d8a-c92b235fd9e1 -> ../../sdb3
lrwxrwxrwx 1 root root 10 Jun 29 09:49 de77a2cb-a1df-460e-97a2-3c8c8ae9fad5 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jun 29 09:49 fb306c92-2607-46a5-a32d-7556b04dd494 -> ../../sda2
/dev/disk/by-path/:
total 0
lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-3 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-3.0 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3.0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3.0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-3.0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-4 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:11.4-ata-4.0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4.0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4.0-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jun 29 09:49 pci-0000:00:11.4-ata-4.0-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-1 -> ../../sdc
lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-1.0 -> ../../sdc
lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-2 -> ../../sdd
lrwxrwxrwx 1 root root 9 Jun 29 09:49 pci-0000:00:1f.2-ata-2.0 -> ../../sdd
/dev/disk/by-uuid/:
total 0
lrwxrwxrwx 1 root root 10 Jun 29 09:49 17716103480993325194 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jun 29 09:49 B851-E178 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 29 09:49 B852-ACFC -> ../../sdb2
# iscsiadm -m node
iscsiadm: No records found
# iscsiadm -m session
iscsiadm: No active sessions.
==== info about volumes ====
# pvs
PV VG Fmt Attr PSize PFree
/dev/sdc ceph-33bdcbd7-07be-4373-97ca-0678dda8888d lvm2 a-- <447.13g 0
/dev/sdd ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 lvm2 a-- <447.13g 0
# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
osd-block-e2deed6d-596f-4837-b14e-88c9afdbe531 ceph-33bdcbd7-07be-4373-97ca-0678dda8888d -wi-ao---- <447.13g
osd-block-55199458-8b33-44f2-b4d2-3a876072a622 ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 -wi-ao---- <447.13g
# vgs
VG #PV #LV #SN Attr VSize VFree
ceph-33bdcbd7-07be-4373-97ca-0678dda8888d 1 1 0 wz--n- <447.13g 0
ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 1 1 0 wz--n- <447.13g 0
# zpool status
pool: rpool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 00:00:19 with 0 errors on Sun Jun 13 00:24:20 2021
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 ONLINE 0 0 0
errors: No known data errors
# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 111G 2.43G 109G - - 5% 2% 1.00x ONLINE -
ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 111G 2.43G 109G - - 5% 2.18% - ONLINE
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 2.43G 105G 104K /rpool
rpool/ROOT 2.42G 105G 96K /rpool/ROOT
rpool/ROOT/pve-1 2.42G 105G 2.42G /
rpool/data 96K 105G 96K /rpool/data
# pveceph status
cluster:
id: 73045ca5-eead-4e44-a0c1-b6796ed3d7d5
health: HEALTH_WARN
client is using insecure global_id reclaim
mons are allowing insecure global_id reclaim
4 slow ops, oldest one blocked for 4666 sec, mon.node04 has slow ops
services:
mon: 3 daemons, quorum node04,node05,node06 (age 36m)
mgr: node04(active, since 97m), standbys: node05, node06
mds: CephFS:1 {0=node05=up:active} 2 up:standby
osd: 6 osds: 6 up (since 37m), 6 in (since 37m)
data:
pools: 4 pools, 193 pgs
objects: 96.62k objects, 365 GiB
usage: 1.0 TiB used, 1.6 TiB / 2.6 TiB avail
pgs: 193 active+clean
io:
client: 0 B/s rd, 20 KiB/s wr, 0 op/s rd, 3 op/s wr
# ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE
0 node04 154G 292G 1 7372 0 0 exists,up
1 node04 202G 244G 1 5733 0 0 exists,up
2 node05 163G 283G 0 7371 0 0 exists,up
3 node05 193G 253G 0 4095 0 0 exists,up
4 node06 180G 266G 0 4095 0 0 exists,up
5 node06 177G 270G 0 0 0 0 exists,up
# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 2.6 TiB 1.6 TiB 1.0 TiB 1.0 TiB 39.96
TOTAL 2.6 TiB 1.6 TiB 1.0 TiB 1.0 TiB 39.96
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
Ceph 2 128 322 GiB 87.91k 965 GiB 42.04 444 GiB
CephFS_data 3 32 34 GiB 8.68k 101 GiB 7.03 444 GiB
CephFS_metadata 4 32 7.9 MiB 24 24 MiB 0 444 GiB
device_health_metrics 5 1 28 MiB 6 84 MiB 0 444 GiB
# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 2.61960 - 2.6 TiB 1.0 TiB 1.0 TiB 98 MiB 5.9 GiB 1.6 TiB 39.96 1.00 - root default
-3 0.87320 - 894 GiB 357 GiB 355 GiB 43 MiB 2.0 GiB 537 GiB 39.96 1.00 - host node04
0 ssd 0.43660 1.00000 447 GiB 154 GiB 153 GiB 5.5 MiB 1018 MiB 293 GiB 34.53 0.86 82 up osd.0
1 ssd 0.43660 1.00000 447 GiB 203 GiB 202 GiB 38 MiB 986 MiB 244 GiB 45.38 1.14 111 up osd.1
-5 0.87320 - 894 GiB 357 GiB 355 GiB 30 MiB 2.0 GiB 537 GiB 39.96 1.00 - host node05
2 ssd 0.43660 1.00000 447 GiB 164 GiB 163 GiB 25 MiB 1.0 GiB 284 GiB 36.58 0.92 94 up osd.2
3 ssd 0.43660 1.00000 447 GiB 194 GiB 193 GiB 4.8 MiB 1019 MiB 253 GiB 43.34 1.08 99 up osd.3
-7 0.87320 - 894 GiB 357 GiB 355 GiB 25 MiB 2.0 GiB 537 GiB 39.96 1.00 - host node06
4 ssd 0.43660 1.00000 447 GiB 180 GiB 179 GiB 22 MiB 1002 MiB 267 GiB 40.32 1.01 97 up osd.4
5 ssd 0.43660 1.00000 447 GiB 177 GiB 176 GiB 2.9 MiB 1021 MiB 270 GiB 39.60 0.99 96 up osd.5
TOTAL 2.6 TiB 1.0 TiB 1.0 TiB 98 MiB 5.9 GiB 1.6 TiB 39.96
MIN/MAX VAR: 0.86/1.14 STDDEV: 3.70
# cat /etc/ceph/ceph.conf
[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = fdb0:5bd1:dc::4/64
fsid = 73045ca5-eead-4e44-a0c1-b6796ed3d7d5
mon_allow_pool_delete = true
mon_host = fdb0:5bd1:dc::4 fdb0:5bd1:dc::5 fdb0:5bd1:dc::6
ms_bind_ipv4 = false
ms_bind_ipv6 = true
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = fdb0:5bd1:dc::4/64
[client]
keyring = /etc/pve/priv/$cluster.$name.keyring
[mds]
keyring = /var/lib/ceph/mds/ceph-$id/keyring
[mds.node04]
host = node04
mds_standby_for_name = pve
[mds.node05]
host = node05
mds_standby_for_name = pve
[mds.node06]
host = node06
mds standby for name = pve
[mon.node04]
public_addr = fdb0:5bd1:dc::4
[mon.node05]
public_addr = fdb0:5bd1:dc::5
[mon.node06]
public_addr = fdb0:5bd1:dc::6
# ceph config dump
WHO MASK LEVEL OPTION VALUE RO
mgr unknown mgr/dashboard/server_addr ::1 *
mgr unknown mgr/dashboard/ssl false *
# pveceph pool ls
+-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+
| Name | Size | Min Size | PG Num | min. PG Num | Optimal PG Num | PG Autoscale Mode | PG Autoscale Target Size | PG Autoscale Target Ratio | Crush Rule Name | %-Used | Used |
+=======================+======+==========+========+=============+================+===================+==========================+===========================+=================+======================+===============+
| Ceph | 3 | 2 | 128 | | 64 | on | | | replicated_rule | 0.420354932546616 | 1036478722631 |
+-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+
| CephFS_data | 3 | 2 | 32 | | 32 | on | | | replicated_rule | 0.0703079700469971 | 108086587392 |
+-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+
| CephFS_metadata | 3 | 2 | 32 | 16 | 16 | on | | | replicated_rule | 1.73890748556005e-05 | 24853666 |
+-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+
| device_health_metrics | 3 | 2 | 1 | 1 | 1 | on | | | replicated_rule | 6.18295161984861e-05 | 88374925 |
+-----------------------+------+----------+--------+-------------+----------------+-------------------+--------------------------+---------------------------+-----------------+----------------------+---------------+
# ceph versions
{
"mon": {
"ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 2,
"ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 1
},
"mgr": {
"ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 2,
"ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 1
},
"osd": {
"ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 4,
"ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 2
},
"mds": {
"ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 2,
"ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 1
},
"overall": {
"ceph version 15.2.13 (1f5c7871ec0e36ade641773b9b05b6211c308b9d) octopus (stable)": 10,
"ceph version 15.2.13 (de5fc19f874b2757d3c0977de8b143f6146af132) octopus (stable)": 5
}
}
-------------- next part --------------
==== general system info ====
# hostname
node06
# pveversion --verbose
proxmox-ve: 7.0-2 (running kernel: 5.11.22-1-pve)
pve-manager: 7.0-5 (running version: 7.0-5/cce9b25f)
pve-kernel-5.11: 7.0-3
pve-kernel-helper: 7.0-3
pve-kernel-5.4: 6.4-3
pve-kernel-5.11.22-1-pve: 5.11.22-1
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph: 15.2.13-pve1
ceph-fuse: 15.2.13-pve1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.0.0-1+pve5
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 7.0-3
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-4
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-6
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-1
lxcfs: 4.0.8-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 1.1.10-1
proxmox-backup-file-restore: 1.1.10-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.1-4
pve-cluster: 7.0-2
pve-container: 4.0-3
pve-docs: 7.0-3
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.2-2
pve-i18n: 2.3-1
pve-qemu-kvm: 6.0.0-2
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-5
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.4-pve1
# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
2a03:7900:111::dc:6 node06.demo.customers.tuxis.net node06
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
#2a03:7900:111:0:90d4:a7ff:fe7f:ccf6 pbs.tuxis.net
# pvesubscription get
message: There is no subscription key
serverid: F3A59435D1B87C5A2460F965646A3177
status: NotFound
url: https://www.proxmox.com/proxmox-ve/pricing
# cat /etc/apt/sources.list
## Managed via Ansible
deb http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free
deb-src http://debmirror.tuxis.nl/debian/ bullseye main contrib non-free
deb http://security.debian.org/ bullseye-security main contrib non-free
deb-src http://security.debian.org/ bullseye-security main contrib non-free
deb http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free
deb-src http://debmirror.tuxis.nl/debian/ bullseye-updates main contrib non-free
# cat /etc/apt/sources.list.d/pvetest-for-beta.list
deb http://download.proxmox.com/debian/pve bullseye pvetest
# cat /etc/apt/sources.list.d/ceph.list
deb http://download.proxmox.com/debian/ceph-octopus bullseye main
# cat /etc/apt/sources.list.d/apt_tuxis_nl_tuxis.list
deb https://apt.tuxis.nl/tuxis/ tuxis-cron main
deb https://apt.tuxis.nl/tuxis/ monitoring main
deb https://apt.tuxis.nl/tuxis/ pmrb main
# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz
Stepping: 2
CPU MHz: 1200.000
CPU max MHz: 3600.0000
CPU min MHz: 1200.0000
BogoMIPS: 6999.74
Virtualization: VT-x
L1d cache: 128 KiB
L1i cache: 128 KiB
L2 cache: 1 MiB
L3 cache: 10 MiB
NUMA node0 CPU(s): 0-7
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d
# pvesh get /cluster/resources --type node --output-format=yaml
---
- id: node/node04
node: node04
status: offline
type: node
- cpu: 0
disk: 2595225600
id: node/node06
level: ''
maxcpu: 8
maxdisk: 115451232256
maxmem: 67322179584
mem: 1778470912
node: node06
status: online
type: node
uptime: 17
- id: node/node05
node: node05
status: offline
type: node
==== overall system load info ====
# top -b -c -w512 -n 1 -o TIME | head -n 30
top - 10:30:25 up 1 min, 1 user, load average: 0.56, 0.23, 0.08
Tasks: 316 total, 3 running, 313 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.5 us, 3.1 sy, 0.0 ni, 95.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 64203.4 total, 62239.5 free, 1723.6 used, 240.4 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 61675.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 165796 8924 5312 S 0.0 0.0 0:01.96 /sbin/init
1908 root 20 0 333916 125364 18596 S 0.0 0.2 0:00.63 /usr/bin/perl /usr/bin/pvesh --nooutput create /nodes/localhost/startall
1900 www-data 20 0 346652 126348 7264 S 0.0 0.2 0:00.53 pveproxy worker
1684 root rt 0 309108 177892 53256 R 0.0 0.3 0:00.52 /usr/sbin/corosync -f
1680 ceph 20 0 255064 19864 10604 S 0.0 0.0 0:00.35 /usr/bin/ceph-mgr -f --cluster ceph --id node06 --setuser ceph --setgroup ceph
1681 ceph 20 0 409952 32068 18780 S 6.2 0.0 0:00.34 /usr/bin/ceph-mon -f --cluster ceph --id node06 --setuser ceph --setgroup ceph
1679 ceph 20 0 250860 18892 10156 S 0.0 0.0 0:00.33 /usr/bin/ceph-mds -f --cluster ceph --id node06 --setuser ceph --setgroup ceph
1699 ceph 20 0 267296 20544 11224 S 0.0 0.0 0:00.31 /usr/bin/ceph-osd -f --cluster ceph --id 5 --setuser ceph --setgroup ceph
31 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/3]
37 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/4]
43 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/5]
49 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/6]
55 root rt 0 0 0 0 S 0.0 0.0 0:00.30 [migration/7]
1701 ceph 20 0 267292 20304 11036 S 0.0 0.0 0:00.30 /usr/bin/ceph-osd -f --cluster ceph --id 4 --setuser ceph --setgroup ceph
19 root rt 0 0 0 0 S 0.0 0.0 0:00.29 [migration/1]
25 root rt 0 0 0 0 S 0.0 0.0 0:00.29 [migration/2]
1787 root 20 0 264744 84344 3988 S 0.0 0.1 0:00.28 pve-firewall
1901 www-data 20 0 346460 126360 7280 S 0.0 0.2 0:00.28 pveproxy worker
1813 root 20 0 345068 123768 6148 S 0.0 0.2 0:00.25 pvedaemon worker
1121 root 20 0 98428 3324 2640 S 0.0 0.0 0:00.23 /usr/sbin/zed -F
734 root 20 0 40140 14964 13888 S 0.0 0.0 0:00.22 /lib/systemd/systemd-journald
780 root 20 0 22124 3568 2380 S 0.0 0.0 0:00.19 /lib/systemd/systemd-udevd
1339 root 20 0 396620 18592 7216 S 0.0 0.0 0:00.17 /usr/bin/python3 /usr/bin/fail2ban-server -xf start
# head /proc/pressure/*
==> /proc/pressure/cpu <==
some avg10=0.00 avg60=0.24 avg300=0.15 total=727501
==> /proc/pressure/io <==
some avg10=0.00 avg60=0.10 avg300=0.06 total=376684
full avg10=0.00 avg60=0.08 avg300=0.05 total=320128
==> /proc/pressure/memory <==
some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
==== info about storage ====
# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content images,backup,iso,vztmpl
prune-backups keep-last=1
shared 0
zfspool: local-zfs
pool rpool/data
content rootdir,images
sparse 1
rbd: Ceph
content rootdir,images
krbd 0
pool Ceph
cephfs: CephFS
path /mnt/pve/CephFS
content iso,snippets,backup,vztmpl
prune-backups keep-last=1
dir: Tuxis_Marketplace
path /mnt/pve/Tuxis_Marketplace
content iso,backup
is_mountpoint yes
mkdir 0
shared 1
dir: Tuxis_Marketplace_Beta
path /mnt/pve/Tuxis_Marketplace_Beta
content backup,iso
is_mountpoint yes
mkdir 0
shared 1
rbd: CephKRBD
content images
krbd 1
pool Ceph
pbs: pbs002.tuxis.nl
datastore DB0220_demo
server pbs002.tuxis.nl
content backup
encryption-key 68:d5:89:f6:f1:f4:67:59:1b:74:6a:78:99:11:ad:09:a0:b0:12:db:43:8d:41:19:af:38:90:77:12:c1:6d:f8
fingerprint 45:f8:79:eb:27:96:88:6b:29:ad:21:00:13:c6:bd:b8:30:f6:f3:9b:f0:bf:dd:f3:ad:f0:09:d5:d2:9a:34:79
prune-backups keep-last=1
username DB0220 at pbs
# pvesm status
got timeout
ERROR: command 'pvesm status' failed: got timeout
# cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
# findmnt --ascii
TARGET SOURCE FSTYPE OPTIONS
/ rpool/ROOT/pve-1 zfs rw,relatime,xattr,noacl
|-/sys sysfs sysfs rw,nosuid,nodev,noexec,relatime
| |-/sys/kernel/security securityfs securityfs rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/cgroup cgroup2 cgroup2 rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/pstore pstore pstore rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/bpf none bpf rw,nosuid,nodev,noexec,relatime,mode=700
| |-/sys/kernel/debug debugfs debugfs rw,nosuid,nodev,noexec,relatime
| |-/sys/kernel/tracing tracefs tracefs rw,nosuid,nodev,noexec,relatime
| |-/sys/fs/fuse/connections fusectl fusectl rw,nosuid,nodev,noexec,relatime
| `-/sys/kernel/config configfs configfs rw,nosuid,nodev,noexec,relatime
|-/proc proc proc rw,relatime
| `-/proc/sys/fs/binfmt_misc systemd-1 autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=23036
|-/dev udev devtmpfs rw,nosuid,relatime,size=32838164k,nr_inodes=8209541,mode=755,inode64
| |-/dev/pts devpts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
| |-/dev/shm tmpfs tmpfs rw,nosuid,nodev,inode64
| |-/dev/hugepages hugetlbfs hugetlbfs rw,relatime,pagesize=2M
| `-/dev/mqueue mqueue mqueue rw,nosuid,nodev,noexec,relatime
|-/run tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=6574432k,mode=755,inode64
| |-/run/lock tmpfs tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64
| |-/run/rpc_pipefs sunrpc rpc_pipefs rw,relatime
| `-/run/user/0 tmpfs tmpfs rw,nosuid,nodev,relatime,size=6574428k,nr_inodes=1643607,mode=700,inode64
|-/rpool rpool zfs rw,noatime,xattr,noacl
| |-/rpool/ROOT rpool/ROOT zfs rw,noatime,xattr,noacl
| `-/rpool/data rpool/data zfs rw,noatime,xattr,noacl
|-/var/lib/ceph/osd/ceph-4 tmpfs tmpfs rw,relatime,inode64
|-/var/lib/ceph/osd/ceph-5 tmpfs tmpfs rw,relatime,inode64
|-/mnt/pve/Tuxis_Marketplace s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0
|-/mnt/pve/Tuxis_Marketplace_Beta s3fs fuse.s3fs rw,nosuid,nodev,relatime,user_id=0,group_id=0
|-/etc/pve /dev/fuse fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
`-/var/lib/lxcfs lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
# df --human
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 1.3M 6.3G 1% /run
rpool/ROOT/pve-1 108G 2.5G 106G 3% /
tmpfs 32G 66M 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
rpool 106G 128K 106G 1% /rpool
rpool/ROOT 106G 128K 106G 1% /rpool/ROOT
rpool/data 106G 128K 106G 1% /rpool/data
tmpfs 32G 24K 32G 1% /var/lib/ceph/osd/ceph-4
tmpfs 32G 24K 32G 1% /var/lib/ceph/osd/ceph-5
s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace
s3fs 256T 0 256T 0% /mnt/pve/Tuxis_Marketplace_Beta
/dev/fuse 30M 40K 30M 1% /etc/pve
tmpfs 6.3G 4.0K 6.3G 1% /run/user/0
==== info about virtual guests ====
# qm list
# pct list
==== info about network ====
# ip -details -statistics address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
RX: bytes packets errors dropped missed mcast
729230 6382 0 0 0 0
TX: bytes packets errors dropped carrier collsns
729230 6382 0 0 0 0
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether 0c:c4:7a:d9:1d:f6 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9216
bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535
altname enp6s0
RX: bytes packets errors dropped missed mcast
507910 4689 0 3 0 270
TX: bytes packets errors dropped carrier collsns
425694 1188 0 0 0 0
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr999 state UP group default qlen 1000
link/ether 0c:c4:7a:d9:1d:f7 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9216
bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535
altname enp7s0
RX: bytes packets errors dropped missed mcast
720 8 0 0 0 8
TX: bytes packets errors dropped carrier collsns
36292 370 0 0 0 0
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:ed:27:6f:7b:f3 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.52:ed:27:6f:7b:f3 designated_root 8000.52:ed:27:6f:7b:f3 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 204.43 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet6 2a03:7900:111::dc:6/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::50ed:27ff:fe6f:7bf3/64 scope link
valid_lft forever preferred_lft forever
RX: bytes packets errors dropped missed mcast
439912 4664 0 0 0 267
TX: bytes packets errors dropped carrier collsns
414944 1063 0 0 0 0
5: vmbr999: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 16:2d:db:6c:6d:8a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 1 vlan_protocol 802.1Q bridge_id 8000.16:2d:db:6c:6d:8a designated_root 8000.16:2d:db:6c:6d:8a root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 204.52 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet6 fdb0:5bd1:dc::6/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::142d:dbff:fe6c:6d8a/64 scope link
valid_lft forever preferred_lft forever
RX: bytes packets errors dropped missed mcast
608 8 0 0 0 8
TX: bytes packets errors dropped carrier collsns
36292 370 0 0 0 0
# ip -details -4 route show
# ip -details -6 route show
unicast ::1 dev lo proto kernel scope global metric 256 pref medium
unicast 2a03:7900:111::/64 dev vmbr0 proto kernel scope global metric 256 pref medium
unicast fdb0:5bd1:dc::/64 dev vmbr999 proto kernel scope global metric 256 pref medium
unicast fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff dev vmbr999 proto boot scope global metric 1024 pref medium
unicast fe80::/64 dev vmbr0 proto kernel scope global metric 256 pref medium
unicast fe80::/64 dev vmbr999 proto kernel scope global metric 256 pref medium
unicast default via 2a03:7900:111::1 dev vmbr0 proto kernel scope global metric 1024 onlink pref medium
# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
auto lo
iface lo inet loopback
iface eno1 inet6 manual
iface eno2 inet6 manual
auto vmbr0
iface vmbr0 inet6 static
address 2a03:7900:111::dc:6/64
gateway 2a03:7900:111::1
bridge-ports eno1
bridge-stp off
bridge-fd 0
auto vmbr999
iface vmbr999 inet6 static
address fdb0:5bd1:dc::6/64
bridge-ports eno2
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
post-up /usr/sbin/ip ro add fdb0:5bd1:cde::/64 via fdb0:5bd1:dc::ffff
==== info about firewall ====
# cat /etc/pve/local/host.fw
cat: /etc/pve/local/host.fw: No such file or directory
# iptables-save
# Generated by iptables-save v1.8.7 on Tue Jun 29 10:30:36 2021
*raw
:PREROUTING ACCEPT [376:131554]
:OUTPUT ACCEPT [281:128086]
COMMIT
# Completed on Tue Jun 29 10:30:36 2021
# Generated by iptables-save v1.8.7 on Tue Jun 29 10:30:36 2021
*filter
:INPUT ACCEPT [281:128086]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [282:128126]
COMMIT
# Completed on Tue Jun 29 10:30:36 2021
==== info about cluster ====
# pvecm nodes
Membership information
----------------------
Nodeid Votes Name
3 1 node06 (local)
# pvecm status
Cluster information
-------------------
Name: Demo
Config Version: 3
Transport: knet
Secure auth: on
Quorum information
------------------
Date: Tue Jun 29 10:30:37 2021
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000003
Ring ID: 3.f4c
Quorate: No
Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 1
Quorum: 2 Activity blocked
Flags:
Membership information
----------------------
Nodeid Votes Name
0x00000003 1 fdb0:5bd1:dc::6%32732 (local)
# cat /etc/pve/corosync.conf 2>/dev/null
logging {
debug: off
to_syslog: yes
}
nodelist {
node {
name: node04
nodeid: 1
quorum_votes: 1
ring0_addr: fdb0:5bd1:dc::4
}
node {
name: node05
nodeid: 2
quorum_votes: 1
ring0_addr: fdb0:5bd1:dc::5
}
node {
name: node06
nodeid: 3
quorum_votes: 1
ring0_addr: fdb0:5bd1:dc::6
}
}
quorum {
provider: corosync_votequorum
}
totem {
cluster_name: Demo
config_version: 3
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}
# ha-manager status
quorum No quorum on node 'node06'!
master node06 (old timestamp - dead?, Tue Jun 9 16:20:27 2020)
lrm node04 (old timestamp - dead?, Tue Jun 29 10:27:38 2021)
lrm node05 (old timestamp - dead?, Tue Jun 29 10:27:38 2021)
lrm node06 (old timestamp - dead?, Tue Jun 29 10:27:37 2021)
==== info about hardware ====
# dmidecode -t bios
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.0 present.
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: American Megatrends Inc.
Version: 2.0a
Release Date: 08/01/2016
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 16 MB
Characteristics:
PCI is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
ACPI is supported
USB legacy is supported
BIOS boot specification is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 5.6
# lspci -nnk
00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [8086:2f00] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [15d9:0832]
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f02] (rev 02)
Kernel driver in use: pcieport
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f03] (rev 02)
Kernel driver in use: pcieport
00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02)
Kernel driver in use: pcieport
00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f0a] (rev 02)
Kernel driver in use: pcieport
00:04.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [8086:2f20] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [8086:2f21] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [8086:2f22] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [8086:2f23] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [8086:2f24] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [8086:2f25] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [8086:2f26] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:04.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [8086:2f27] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [15d9:0832]
Kernel driver in use: ioatdma
Kernel modules: ioatdma
00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [8086:2f28] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [15d9:0832]
00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [8086:2f29] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [15d9:0832]
00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [8086:2f2a] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [15d9:0832]
00:05.4 PIC [0800]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [8086:2f2c] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [15d9:0832]
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR [8086:8d7c] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
00:11.4 SATA controller [0106]: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] [8086:8d62] (rev 05)
Subsystem: Super Micro Computer Inc C610/X99 series chipset sSATA Controller [AHCI mode] [15d9:0832]
Kernel driver in use: ahci
Kernel modules: ahci
00:14.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
00:16.0 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel modules: mei_me
00:16.1 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #2 [8086:8d3b] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
Kernel driver in use: pcieport
00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
Kernel driver in use: pcieport
00:1c.5 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #6 [8086:8d1a] (rev d5)
Kernel driver in use: pcieport
00:1c.6 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 [8086:8d1c] (rev d5)
Kernel driver in use: pcieport
00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series chipset LPC Controller [8086:8d44] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: lpc_ich
Kernel modules: lpc_ich
00:1f.2 SATA controller [0106]: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [8086:8d02] (rev 05)
Subsystem: Super Micro Computer Inc C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [15d9:0832]
Kernel driver in use: ahci
Kernel modules: ahci
00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset SMBus Controller [8086:8d22] (rev 05)
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
06:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
DeviceName: Intel Ethernet i210AT #1
Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533]
Kernel driver in use: igb
Kernel modules: igb
07:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
DeviceName: Intel Ethernet i210AT #2
Subsystem: Super Micro Computer Inc I210 Gigabit Network Connection [15d9:1533]
Kernel driver in use: igb
Kernel modules: igb
08:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03)
09:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30)
DeviceName: ASPEED Video AST2400
Subsystem: Super Micro Computer Inc X10SRL-F [15d9:0832]
Kernel driver in use: ast
Kernel modules: ast
ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f81] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832]
ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f36] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832]
Kernel driver in use: hswep_uncore
ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f37] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [15d9:0832]
Kernel driver in use: hswep_uncore
ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe0] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832]
ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe1] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832]
ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe2] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832]
ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe3] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [15d9:0832]
ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff8] (rev 02)
ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff9] (rev 02)
ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffc] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832]
ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffd] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832]
ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffe] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [15d9:0832]
ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f1d] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832]
ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f34] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [15d9:0832]
Kernel driver in use: hswep_uncore
ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1e] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832]
ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f7d] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832]
ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1f] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [15d9:0832]
ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2fa0] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832]
ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2f30] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [15d9:0832]
Kernel driver in use: hswep_uncore
ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832]
ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [15d9:0832]
ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2faa] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832]
ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fab] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832]
ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fac] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832]
ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fad] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [15d9:0832]
ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast [8086:2fae] (rev 02)
ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2faf] (rev 02)
ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [8086:2fb0] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [15d9:0832]
Kernel driver in use: hswep_uncore
ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [8086:2fb1] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [15d9:0832]
Kernel driver in use: hswep_uncore
ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [8086:2fb2] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [15d9:0832]
ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [8086:2fb3] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [15d9:0832]
ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbc] (rev 02)
ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbd] (rev 02)
ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbe] (rev 02)
ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbf] (rev 02)
ff:15.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [8086:2fb4] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control [15d9:0832]
Kernel driver in use: hswep_uncore
ff:15.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [8086:2fb5] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control [15d9:0832]
Kernel driver in use: hswep_uncore
ff:15.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [8086:2fb6] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers [15d9:0832]
ff:15.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [8086:2fb7] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers [15d9:0832]
ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers [8086... (rev 02)
ff:16.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast [8086:2f6e] (rev 02)
ff:16.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2f6f] (rev 02)
ff:17.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control [8086:2fd0] (rev 02)
Kernel driver in use: hswep_uncore
ff:17.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb8] (rev 02)
ff:17.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb9] (rev 02)
ff:17.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fba] (rev 02)
ff:17.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fbb] (rev 02)
ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f98] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832]
ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f99] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832]
ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9a] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832]
ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2fc0] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832]
ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9c] (rev 02)
Subsystem: Super Micro Computer Inc Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [15d9:0832]
ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f88] (rev 02)
ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f8a] (rev 02)
==== info about block devices ====
# lsblk --ascii
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 111.8G 0 disk
|-sda1 8:1 0 1007K 0 part
|-sda2 8:2 0 512M 0 part
`-sda3 8:3 0 111.3G 0 part
sdb 8:16 0 111.8G 0 disk
|-sdb1 8:17 0 1007K 0 part
|-sdb2 8:18 0 512M 0 part
`-sdb3 8:19 0 111.3G 0 part
sdc 8:32 0 447.1G 0 disk
`-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 253:0 0 447.1G 0 lvm
sdd 8:48 0 447.1G 0 disk
`-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 253:1 0 447.1G 0 lvm
# ls -l /dev/disk/by-*/
/dev/disk/by-id/:
total 0
lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-INTEL_SSDSC2KW120H6_BTLT7124064S120GGN-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jun 29 10:29 ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102524 -> ../../sdd
lrwxrwxrwx 1 root root 9 Jun 29 10:29 ata-SAMSUNG_MZ7LM480HCHP-00003_S1YJNXAH102531 -> ../../sdc
lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-name-ceph--33bdcbd7--07be--4373--97ca--0678dda8888d-osd--block--e2deed6d--596f--4837--b14e--88c9afdbe531 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-name-ceph--97bdf879--bbf1--41ba--8563--81abe42cf617-osd--block--55199458--8b33--44f2--b4d2--3a876072a622 -> ../../dm-1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-uuid-LVM-GHM6Bwl9TQ7jv5GJd8ORRD6XDearTRZhgvpxQ22a3TWdlBd9iGk1oHhop5lXn8lL -> ../../dm-1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 dm-uuid-LVM-hoOm4ydEDwKOrnNdVuCBCsY31it5n1ZRDsf4uP4Irce8u2hubaahZCqfMz9IpwhI -> ../../dm-0
lrwxrwxrwx 1 root root 9 Jun 29 10:29 lvm-pv-uuid-AGbSTn-aDmD-AbAR-ngCX-8glc-2KVW-xal2xh -> ../../sdc
lrwxrwxrwx 1 root root 9 Jun 29 10:29 lvm-pv-uuid-QPw8aR-Rbbe-LzZ7-0j3t-n8gn-OeOs-YWPaoV -> ../../sdd
lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x5002538c00018347 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x5002538c00018347-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x5002538c00018347-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x5002538c00018347-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x5002538c40146ccb -> ../../sdd
lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x5002538c40146cd2 -> ../../sdc
lrwxrwxrwx 1 root root 9 Jun 29 10:29 wwn-0x55cd2e414db345fd -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x55cd2e414db345fd-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x55cd2e414db345fd-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 29 10:29 wwn-0x55cd2e414db345fd-part3 -> ../../sda3
/dev/disk/by-label/:
total 0
lrwxrwxrwx 1 root root 10 Jun 29 10:29 rpool -> ../../sda3
/dev/disk/by-partuuid/:
total 0
lrwxrwxrwx 1 root root 10 Jun 29 10:29 4f42744a-eef7-49f5-bfa4-5cb3ca1ee4b2 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 70166c71-7a1f-400e-bd39-f8f4be867d3e -> ../../sda3
lrwxrwxrwx 1 root root 10 Jun 29 10:29 87402126-9aa6-4be9-9c13-4704492a974b -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 a52ed3d9-d18c-4d5b-9d8a-c92b235fd9e1 -> ../../sdb3
lrwxrwxrwx 1 root root 10 Jun 29 10:29 de77a2cb-a1df-460e-97a2-3c8c8ae9fad5 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jun 29 10:29 fb306c92-2607-46a5-a32d-7556b04dd494 -> ../../sda2
/dev/disk/by-path/:
total 0
lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-3 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-3.0 -> ../../sda
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3.0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3.0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-3.0-part3 -> ../../sda3
lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-4 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:11.4-ata-4.0 -> ../../sdb
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4.0-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4.0-part2 -> ../../sdb2
lrwxrwxrwx 1 root root 10 Jun 29 10:29 pci-0000:00:11.4-ata-4.0-part3 -> ../../sdb3
lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-1 -> ../../sdc
lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-1.0 -> ../../sdc
lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-2 -> ../../sdd
lrwxrwxrwx 1 root root 9 Jun 29 10:29 pci-0000:00:1f.2-ata-2.0 -> ../../sdd
/dev/disk/by-uuid/:
total 0
lrwxrwxrwx 1 root root 10 Jun 29 10:29 17716103480993325194 -> ../../sda3
lrwxrwxrwx 1 root root 10 Jun 29 10:29 B851-E178 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jun 29 10:29 B852-ACFC -> ../../sdb2
# iscsiadm -m node
iscsiadm: No records found
# iscsiadm -m session
iscsiadm: No active sessions.
==== info about volumes ====
# pvs
PV VG Fmt Attr PSize PFree
/dev/sdc ceph-33bdcbd7-07be-4373-97ca-0678dda8888d lvm2 a-- <447.13g 0
/dev/sdd ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 lvm2 a-- <447.13g 0
# lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
osd-block-e2deed6d-596f-4837-b14e-88c9afdbe531 ceph-33bdcbd7-07be-4373-97ca-0678dda8888d -wi-a----- <447.13g
osd-block-55199458-8b33-44f2-b4d2-3a876072a622 ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 -wi-a----- <447.13g
# vgs
VG #PV #LV #SN Attr VSize VFree
ceph-33bdcbd7-07be-4373-97ca-0678dda8888d 1 1 0 wz--n- <447.13g 0
ceph-97bdf879-bbf1-41ba-8563-81abe42cf617 1 1 0 wz--n- <447.13g 0
# zpool status
pool: rpool
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 00:00:19 with 0 errors on Sun Jun 13 00:24:20 2021
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 ONLINE 0 0 0
errors: No known data errors
# zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 111G 2.43G 109G - - 5% 2% 1.00x ONLINE -
ata-SAMSUNG_MZ7LM120HCFD-00003_S22PNYAG500437-part3 111G 2.43G 109G - - 5% 2.18% - ONLINE
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 2.43G 105G 104K /rpool
rpool/ROOT 2.42G 105G 96K /rpool/ROOT
rpool/ROOT/pve-1 2.42G 105G 2.42G /
rpool/data 96K 105G 96K /rpool/data
# pveceph status
ERROR: command 'pveceph status' failed: got timeout
# ceph osd status
ERROR: command 'ceph osd status' failed: got timeout
# ceph df
ERROR: command 'ceph df' failed: got timeout
# ceph osd df tree
ERROR: command 'ceph osd df tree' failed: got timeout
# cat /etc/ceph/ceph.conf
[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = fdb0:5bd1:dc::4/64
fsid = 73045ca5-eead-4e44-a0c1-b6796ed3d7d5
mon_allow_pool_delete = true
mon_host = fdb0:5bd1:dc::4 fdb0:5bd1:dc::5 fdb0:5bd1:dc::6
ms_bind_ipv4 = false
ms_bind_ipv6 = true
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = fdb0:5bd1:dc::4/64
[client]
keyring = /etc/pve/priv/$cluster.$name.keyring
[mds]
keyring = /var/lib/ceph/mds/ceph-$id/keyring
[mds.node04]
host = node04
mds_standby_for_name = pve
[mds.node05]
host = node05
mds_standby_for_name = pve
[mds.node06]
host = node06
mds standby for name = pve
[mon.node04]
public_addr = fdb0:5bd1:dc::4
[mon.node05]
public_addr = fdb0:5bd1:dc::5
[mon.node06]
public_addr = fdb0:5bd1:dc::6
# ceph config dump
ERROR: command 'ceph config dump' failed: got timeout
# pveceph pool ls
got timeout
# ceph versions
ERROR: command 'ceph versions' failed: got timeout
More information about the pve-user
mailing list