[PVE-User] WARNING: Upgrade and Watchdog kills Server in HA-Mode
Andreas Herrmann
andreas at mx20.org
Thu Dec 7 13:08:38 CET 2017
Hi,
On 07.12.2017 08:57, Thomas Lamprecht wrote:
> some more information would be great to check this.
> First, do you have a daemon(like) service loading sysctl
> configs on the fly? If not we may rule out the sysctl config problem
> as a trigger for this.
No. It's a quite new installation from ISO without upgrade from Proxmox 4 and
really less modifications.
> Can you describe your firewall setup a bit?
> Do you use Firewall groups?
We don't use Proxmox firewall at all. We have uif based rules and no
limitations between the proxmox hosts:
# Zugriff der Nodes untereinander
in+ s=nethcn-b-vl58(4),nethcn-b-vl802(4)
# Die beiden Corosync HA Ringe
in+ i=coro1 s=nethcn-b-ha1(4)
in+ i=coro2 s=nethcn-b-ha2(4)
# Ceph Traffic
in+ i=ceph s=nethcn-b-store(4)
> Do you got some log entries around that time?
> Or a persistent journal?
Some logs are attached. nethcn-b5 rebootet after I restarted services with
needrestart. nethcn-b4 rebootet in between the update. Maybe are problem with
communication between watchdog-mux.service und Proxmox.
Maybe I should change to hardware watchdog provided by Supermicro X10SRW-F
mainboard.
Andreas
-------------- next part --------------
Dec 6 17:51:08 nethcn-b2 systemd[1]: Created slice User Slice of root.
Dec 6 17:51:08 nethcn-b2 systemd[1]: Starting User Manager for UID 0...
Dec 6 17:51:08 nethcn-b2 systemd[1]: Started Session 2294 of user root.
Dec 6 17:51:08 nethcn-b2 systemd[25841]: Listening on GnuPG cryptographic agent and passphrase cache.
Dec 6 17:51:08 nethcn-b2 systemd[25841]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Dec 6 17:51:08 nethcn-b2 systemd[25841]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Dec 6 17:51:08 nethcn-b2 systemd[25841]: Reached target Paths.
Dec 6 17:51:08 nethcn-b2 systemd[25841]: Reached target Timers.
Dec 6 17:51:08 nethcn-b2 systemd[25841]: Listening on GnuPG network certificate management daemon.
Dec 6 17:51:08 nethcn-b2 systemd[25841]: Listening on GnuPG cryptographic agent (access for web browsers).
Dec 6 17:51:08 nethcn-b2 systemd[25841]: Reached target Sockets.
Dec 6 17:51:08 nethcn-b2 systemd[25841]: Reached target Basic System.
Dec 6 17:51:08 nethcn-b2 systemd[25841]: Reached target Default.
Dec 6 17:51:08 nethcn-b2 systemd[25841]: Startup finished in 21ms.
Dec 6 17:51:08 nethcn-b2 systemd[1]: Started User Manager for UID 0.
Dec 6 17:51:14 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:14 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 15min 25.622828s random time.
Dec 6 17:51:14 nethcn-b2 systemd[1]: apt-daily.timer: Adding 6h 6min 27.629758s random time.
Dec 6 17:51:14 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 24min 42.371776s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily.timer: Adding 10h 23min 49.731837s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 25min 49.899301s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily.timer: Adding 1h 1min 44.339369s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 53min 41.700970s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily.timer: Adding 4h 19min 32.155871s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 33min 33.939842s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily.timer: Adding 10h 3min 29.743451s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 26min 34.968617s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily.timer: Adding 10h 29min 18.753427s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 28min 47.463310s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily.timer: Adding 1h 32min 44.821502s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 15min 11.470765s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily.timer: Adding 43min 12.485912s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 11min 29.546795s random time.
Dec 6 17:51:15 nethcn-b2 systemd[1]: apt-daily.timer: Adding 2h 35min 42.196692s random time.
Dec 6 17:51:16 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:17 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 49min 31.062780s random time.
Dec 6 17:51:17 nethcn-b2 systemd[1]: apt-daily.timer: Adding 4h 32min 30.982647s random time.
Dec 6 17:51:17 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:17 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 45min 39.993857s random time.
Dec 6 17:51:17 nethcn-b2 systemd[1]: apt-daily.timer: Adding 3h 8min 26.608575s random time.
Dec 6 17:51:17 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:17 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 37min 6.641514s random time.
Dec 6 17:51:17 nethcn-b2 systemd[1]: apt-daily.timer: Adding 11h 34min 54.498924s random time.
Dec 6 17:51:18 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:18 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 8min 17.506967s random time.
Dec 6 17:51:18 nethcn-b2 systemd[1]: apt-daily.timer: Adding 3h 55min 54.889100s random time.
Dec 6 17:51:18 nethcn-b2 systemd[1]: Stopping The Proxmox VE cluster filesystem...
Dec 6 17:51:18 nethcn-b2 pmxcfs[9987]: [main] notice: teardown filesystem
Dec 6 17:51:20 nethcn-b2 pmxcfs[9987]: [main] notice: exit proxmox configuration filesystem (0)
Dec 6 17:51:20 nethcn-b2 systemd[1]: Stopped The Proxmox VE cluster filesystem.
Dec 6 17:51:20 nethcn-b2 systemd[1]: Starting The Proxmox VE cluster filesystem...
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [status] notice: update cluster info (cluster name NETHCN-B, version = 7)
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [status] notice: node has quorum
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [dcdb] notice: members: 1/10104, 2/28566, 3/29106, 4/30188, 5/10652
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [dcdb] notice: starting data syncronisation
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [dcdb] notice: received sync request (epoch 1/10104/00000011)
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [status] notice: members: 1/10104, 2/28566, 3/29106, 4/30188, 5/10652
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [status] notice: starting data syncronisation
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [status] notice: received sync request (epoch 1/10104/00000011)
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [dcdb] notice: received all states
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [dcdb] notice: leader is 1/10104
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [dcdb] notice: synced members: 1/10104, 2/28566, 3/29106, 4/30188, 5/10652
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [dcdb] notice: all data is up to date
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [status] notice: received all states
Dec 6 17:51:20 nethcn-b2 pmxcfs[28566]: [status] notice: all data is up to date
Dec 6 17:51:20 nethcn-b2 pve-ha-crm[10842]: ipcc_send_rec[1] failed: Transport endpoint is not connected
Dec 6 17:51:20 nethcn-b2 pve-ha-crm[10842]: ipcc_send_rec[2] failed: Connection refused
Dec 6 17:51:20 nethcn-b2 pve-ha-crm[10842]: ipcc_send_rec[3] failed: Connection refused
Dec 6 17:51:20 nethcn-b2 pve-ha-crm[10842]: ERROR: Connection refused
Dec 6 17:51:20 nethcn-b2 pve-ha-crm[10842]: server received shutdown request
Dec 6 17:51:20 nethcn-b2 pve-ha-crm[10842]: server stopped
Dec 6 17:51:20 nethcn-b2 watchdog-mux[3397]: client did not stop watchdog - disable watchdog updates
Dec 6 17:51:20 nethcn-b2 systemd[1]: pve-ha-crm.service: Main process exited, code=exited, status=255/n/a
Dec 6 17:51:21 nethcn-b2 systemd[1]: Started The Proxmox VE cluster filesystem.
Dec 6 17:51:21 nethcn-b2 systemd[1]: Reloading Proxmox VE firewall.
Dec 6 17:51:21 nethcn-b2 systemd[1]: pve-ha-crm.service: Unit entered failed state.
Dec 6 17:51:21 nethcn-b2 systemd[1]: pve-ha-crm.service: Failed with result 'exit-code'.
Dec 6 17:51:21 nethcn-b2 pve-ha-lrm[13145]: ipcc_send_rec[1] failed: Transport endpoint is not connected
Dec 6 17:51:21 nethcn-b2 watchdog-mux[3397]: exit watchdog-mux with active connections
Dec 6 17:51:21 nethcn-b2 kernel: [88876.361477] watchdog: watchdog0: watchdog did not stop!
Dec 6 17:51:21 nethcn-b2 pve-firewall[28714]: send HUP to 10566
Dec 6 17:51:21 nethcn-b2 pve-firewall[10566]: received signal HUP
Dec 6 17:51:21 nethcn-b2 pve-firewall[10566]: server shutdown (restart)
Dec 6 17:51:21 nethcn-b2 systemd[1]: Reloaded Proxmox VE firewall.
Dec 6 17:51:22 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:22 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 43min 57.561346s random time.
Dec 6 17:51:22 nethcn-b2 systemd[1]: apt-daily.timer: Adding 1h 53min 46.711159s random time.
Dec 6 17:51:22 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:22 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 23min 666.457ms random time.
Dec 6 17:51:22 nethcn-b2 systemd[1]: apt-daily.timer: Adding 5h 40min 48.607339s random time.
Dec 6 17:51:22 nethcn-b2 systemd[1]: Stopping Proxmox VE firewall logger...
Dec 6 17:51:22 nethcn-b2 pvepw-logger[22772]: received terminate request (signal)
Dec 6 17:51:22 nethcn-b2 pvepw-logger[22772]: stopping pvefw logger
Dec 6 17:51:22 nethcn-b2 pve-firewall[10566]: restarting server
Dec 6 17:51:22 nethcn-b2 systemd[1]: Stopped Proxmox VE firewall logger.
Dec 6 17:51:22 nethcn-b2 systemd[1]: Starting Proxmox VE firewall logger...
Dec 6 17:51:22 nethcn-b2 pvefw-logger[28896]: starting pvefw logger
Dec 6 17:51:22 nethcn-b2 systemd[1]: Started Proxmox VE firewall logger.
Dec 6 17:51:22 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:22 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 16min 39.381721s random time.
Dec 6 17:51:22 nethcn-b2 systemd[1]: apt-daily.timer: Adding 1h 23min 56.458060s random time.
Dec 6 17:51:22 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:22 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 53min 22.748230s random time.
Dec 6 17:51:22 nethcn-b2 systemd[1]: apt-daily.timer: Adding 4h 43min 27.611334s random time.
Dec 6 17:51:22 nethcn-b2 kernel: [88877.490542] audit: type=1400 audit(1512579082.769:14): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/bin/lxc-start" pid=28947 comm="apparmor_parser"
Dec 6 17:51:22 nethcn-b2 kernel: [88877.677013] audit: type=1400 audit(1512579082.955:15): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="lxc-container-default" pid=28951 comm="apparmor_parser"
Dec 6 17:51:22 nethcn-b2 kernel: [88877.693940] audit: type=1400 audit(1512579082.955:16): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="lxc-container-default-cgns" pid=28951 comm="apparmor_parser"
Dec 6 17:51:22 nethcn-b2 kernel: [88877.711368] audit: type=1400 audit(1512579082.956:17): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="lxc-container-default-with-mounting" pid=28951 comm="apparmor_parser"
Dec 6 17:51:23 nethcn-b2 kernel: [88877.729675] audit: type=1400 audit(1512579082.956:18): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="lxc-container-default-with-nesting" pid=28951 comm="apparmor_parser"
Dec 6 17:51:23 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:23 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 8min 32.222778s random time.
Dec 6 17:51:23 nethcn-b2 systemd[1]: apt-daily.timer: Adding 9h 48min 39.647146s random time.
Dec 6 17:51:23 nethcn-b2 pvestatd[10618]: ipcc_send_rec[1] failed: Transport endpoint is not connected
Dec 6 17:51:23 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:24 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 36min 23.950978s random time.
Dec 6 17:51:24 nethcn-b2 systemd[1]: apt-daily.timer: Adding 7h 27min 37.724293s random time.
Dec 6 17:51:24 nethcn-b2 systemd[1]: Reloading.
Dec 6 17:51:24 nethcn-b2 systemd[1]: apt-daily-upgrade.timer: Adding 28min 57.947572s random time.
Dec 6 17:51:24 nethcn-b2 systemd[1]: apt-daily.timer: Adding 3h 8min 51.079398s random time.
Dec 6 17:51:24 nethcn-b2 systemd[1]: Started Session 2296 of user root.
Dec 6 17:51:24 nethcn-b2 systemd[1]: Stopping PVE Local HA Ressource Manager Daemon...
Dec 6 17:51:24 nethcn-b2 pve-ha-lrm[13145]: received signal TERM
Dec 6 17:51:24 nethcn-b2 pve-ha-lrm[13145]: restart LRM, freeze all services
-------------- next part --------------
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [dcdb] notice: members: 1/10104, 2/9987, 3/9969, 4/9839
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [dcdb] notice: starting data syncronisation
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [status] notice: members: 1/10104, 2/9987, 3/9969, 4/9839
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [status] notice: starting data syncronisation
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [dcdb] notice: received sync request (epoch 1/10104/00000008)
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [status] notice: received sync request (epoch 1/10104/00000008)
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [dcdb] notice: received all states
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [dcdb] notice: leader is 1/10104
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [dcdb] notice: synced members: 1/10104, 2/9987, 3/9969, 4/9839
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [dcdb] notice: all data is up to date
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [status] notice: received all states
Dec 6 17:27:33 nethcn-b2 pmxcfs[9987]: [status] notice: all data is up to date
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [dcdb] notice: members: 1/10104, 2/9987, 3/9969, 4/9839, 5/14789
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [dcdb] notice: starting data syncronisation
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [status] notice: members: 1/10104, 2/9987, 3/9969, 4/9839, 5/14789
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [status] notice: starting data syncronisation
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [dcdb] notice: received sync request (epoch 1/10104/00000009)
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [status] notice: received sync request (epoch 1/10104/00000009)
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [dcdb] notice: received all states
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [dcdb] notice: leader is 1/10104
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [dcdb] notice: synced members: 1/10104, 2/9987, 3/9969, 4/9839, 5/14789
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [dcdb] notice: all data is up to date
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [status] notice: received all states
Dec 6 17:27:34 nethcn-b2 pmxcfs[9987]: [status] notice: all data is up to date
Dec 6 17:28:00 nethcn-b2 systemd[1]: Starting Proxmox VE replication runner...
Dec 6 17:28:01 nethcn-b2 systemd[1]: Started Proxmox VE replication runner.
Dec 6 17:28:01 nethcn-b2 CRON[21670]: (root) CMD ( sleep $((RANDOM % 20)); /usr/local/sbin/check_ipmi.sh)
Dec 6 17:28:20 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:20Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:28:23 nethcn-b2 corosync[10210]: notice [TOTEM ] A new membership (192.168.112.1:2524) was formed. Members left: 5
Dec 6 17:28:23 nethcn-b2 corosync[10210]: notice [TOTEM ] Failed to receive the leave message. failed: 5
Dec 6 17:28:23 nethcn-b2 corosync[10210]: [TOTEM ] A new membership (192.168.112.1:2524) was formed. Members left: 5
Dec 6 17:28:23 nethcn-b2 corosync[10210]: [TOTEM ] Failed to receive the leave message. failed: 5
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [dcdb] notice: members: 1/10104, 2/9987, 3/9969, 4/9839
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [dcdb] notice: starting data syncronisation
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [status] notice: members: 1/10104, 2/9987, 3/9969, 4/9839
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [status] notice: starting data syncronisation
Dec 6 17:28:23 nethcn-b2 corosync[10210]: notice [QUORUM] Members[4]: 1 2 3 4
Dec 6 17:28:23 nethcn-b2 corosync[10210]: notice [MAIN ] Completed service synchronization, ready to provide service.
Dec 6 17:28:23 nethcn-b2 corosync[10210]: [QUORUM] Members[4]: 1 2 3 4
Dec 6 17:28:23 nethcn-b2 corosync[10210]: [MAIN ] Completed service synchronization, ready to provide service.
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [dcdb] notice: received sync request (epoch 1/10104/0000000A)
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [status] notice: received sync request (epoch 1/10104/0000000A)
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [dcdb] notice: received all states
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [dcdb] notice: leader is 1/10104
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [dcdb] notice: synced members: 1/10104, 2/9987, 3/9969, 4/9839
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [dcdb] notice: all data is up to date
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [dcdb] notice: dfsm_deliver_queue: queue length 11
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [status] notice: received all states
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [status] notice: all data is up to date
Dec 6 17:28:23 nethcn-b2 pmxcfs[9987]: [status] notice: dfsm_deliver_queue: queue length 26
Dec 6 17:28:24 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:24Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:28:28 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:28Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:28:32 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:32Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:28:32 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:32.057623 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6815 osd.32 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:12.057620)
Dec 6 17:28:32 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:32.057651 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6803 osd.33 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:12.057620)
Dec 6 17:28:32 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:32.057658 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6807 osd.34 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:12.057620)
Dec 6 17:28:32 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:32.057665 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6827 osd.35 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:12.057620)
Dec 6 17:28:32 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:32.057672 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6811 osd.36 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:12.057620)
Dec 6 17:28:32 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:32.057681 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6819 osd.37 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:12.057620)
Dec 6 17:28:32 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:32.057688 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6823 osd.38 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:12.057620)
Dec 6 17:28:32 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:32.057694 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6831 osd.39 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:12.057620)
Dec 6 17:28:33 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:33.058175 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6815 osd.32 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:13.058171)
Dec 6 17:28:33 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:33.058198 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6803 osd.33 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:13.058171)
Dec 6 17:28:33 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:33.058212 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6807 osd.34 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:13.058171)
Dec 6 17:28:33 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:33.058224 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6827 osd.35 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:13.058171)
Dec 6 17:28:33 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:33.058238 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6811 osd.36 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:13.058171)
Dec 6 17:28:33 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:33.058250 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6819 osd.37 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:13.058171)
Dec 6 17:28:33 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:33.058263 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6823 osd.38 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:13.058171)
Dec 6 17:28:33 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:33.058274 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6831 osd.39 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:13.058171)
Dec 6 17:28:33 nethcn-b2 pvestatd[10618]: status update time (9.911 seconds)
Dec 6 17:28:34 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:34.058434 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6815 osd.32 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:14.058430)
Dec 6 17:28:34 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:34.058444 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6803 osd.33 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:14.058430)
Dec 6 17:28:34 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:34.058447 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6807 osd.34 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:14.058430)
Dec 6 17:28:34 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:34.058449 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6827 osd.35 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:14.058430)
Dec 6 17:28:34 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:34.058454 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6811 osd.36 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:14.058430)
Dec 6 17:28:34 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:34.058456 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6819 osd.37 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:14.058430)
Dec 6 17:28:34 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:34.058458 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6823 osd.38 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:14.058430)
Dec 6 17:28:34 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:34.058460 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6831 osd.39 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:14.058430)
Dec 6 17:28:34 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:34.382846 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6815 osd.32 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:14.382840)
Dec 6 17:28:34 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:34.382872 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6803 osd.33 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:14.382840)
Dec 6 17:28:34 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:34.382880 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6807 osd.34 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:14.382840)
Dec 6 17:28:34 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:34.382890 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6827 osd.35 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:14.382840)
Dec 6 17:28:34 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:34.382899 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6811 osd.36 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:14.382840)
Dec 6 17:28:34 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:34.382906 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6819 osd.37 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:14.382840)
Dec 6 17:28:34 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:34.382912 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6823 osd.38 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:14.382840)
Dec 6 17:28:34 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:34.382918 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6831 osd.39 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:14.382840)
Dec 6 17:28:35 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:35.058560 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6815 osd.32 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:15.058555)
Dec 6 17:28:35 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:35.058575 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6803 osd.33 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:15.058555)
Dec 6 17:28:35 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:35.058578 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6807 osd.34 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:15.058555)
Dec 6 17:28:35 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:35.058599 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6827 osd.35 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:15.058555)
Dec 6 17:28:35 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:35.058602 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6811 osd.36 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:15.058555)
Dec 6 17:28:35 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:35.058604 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6819 osd.37 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:15.058555)
Dec 6 17:28:35 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:35.058606 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6823 osd.38 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:15.058555)
Dec 6 17:28:35 nethcn-b2 ceph-osd[10845]: 2017-12-06 17:28:35.058609 7fbf4f84c700 -1 osd.11 37761 heartbeat_check: no reply from 192.168.112.135:6831 osd.39 since back 2017-12-06 17:28:11.439837 front 2017-12-06 17:28:11.439837 (cutoff 2017-12-06 17:28:15.058555)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11486]: 2017-12-06 17:28:35.202181 7f38ff3a1700 -1 osd.10 37761 heartbeat_check: no reply from 192.168.112.135:6815 osd.32 since back 2017-12-06 17:28:14.631386 front 2017-12-06 17:28:14.631386 (cutoff 2017-12-06 17:28:15.202177)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11486]: 2017-12-06 17:28:35.202194 7f38ff3a1700 -1 osd.10 37761 heartbeat_check: no reply from 192.168.112.135:6803 osd.33 since back 2017-12-06 17:28:14.631386 front 2017-12-06 17:28:14.631386 (cutoff 2017-12-06 17:28:15.202177)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11486]: 2017-12-06 17:28:35.202199 7f38ff3a1700 -1 osd.10 37761 heartbeat_check: no reply from 192.168.112.135:6807 osd.34 since back 2017-12-06 17:28:14.631386 front 2017-12-06 17:28:14.631386 (cutoff 2017-12-06 17:28:15.202177)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11486]: 2017-12-06 17:28:35.202202 7f38ff3a1700 -1 osd.10 37761 heartbeat_check: no reply from 192.168.112.135:6827 osd.35 since back 2017-12-06 17:28:14.631386 front 2017-12-06 17:28:14.631386 (cutoff 2017-12-06 17:28:15.202177)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11486]: 2017-12-06 17:28:35.202205 7f38ff3a1700 -1 osd.10 37761 heartbeat_check: no reply from 192.168.112.135:6811 osd.36 since back 2017-12-06 17:28:14.631386 front 2017-12-06 17:28:14.631386 (cutoff 2017-12-06 17:28:15.202177)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11486]: 2017-12-06 17:28:35.202207 7f38ff3a1700 -1 osd.10 37761 heartbeat_check: no reply from 192.168.112.135:6819 osd.37 since back 2017-12-06 17:28:14.631386 front 2017-12-06 17:28:14.631386 (cutoff 2017-12-06 17:28:15.202177)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11486]: 2017-12-06 17:28:35.202210 7f38ff3a1700 -1 osd.10 37761 heartbeat_check: no reply from 192.168.112.135:6823 osd.38 since back 2017-12-06 17:28:14.631386 front 2017-12-06 17:28:14.631386 (cutoff 2017-12-06 17:28:15.202177)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11486]: 2017-12-06 17:28:35.202212 7f38ff3a1700 -1 osd.10 37761 heartbeat_check: no reply from 192.168.112.135:6831 osd.39 since back 2017-12-06 17:28:14.631386 front 2017-12-06 17:28:14.631386 (cutoff 2017-12-06 17:28:15.202177)
Dec 6 17:28:35 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:35Z E! InfluxDB Output Error: Post http://influxdb-b1.as6724.net:8086/write?consistency=any&db=noc_nethcn_telegraf: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Dec 6 17:28:35 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:35Z E! Error writing to output [influxdb]: Could not write to any InfluxDB server in cluster
Dec 6 17:28:35 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:35.383153 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6815 osd.32 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:15.383148)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:35.383170 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6803 osd.33 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:15.383148)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:35.383173 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6807 osd.34 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:15.383148)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:35.383178 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6827 osd.35 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:15.383148)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:35.383180 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6811 osd.36 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:15.383148)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:35.383183 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6819 osd.37 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:15.383148)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:35.383185 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6823 osd.38 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:15.383148)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11856]: 2017-12-06 17:28:35.383187 7f308a81d700 -1 osd.14 37761 heartbeat_check: no reply from 192.168.112.135:6831 osd.39 since back 2017-12-06 17:28:13.944993 front 2017-12-06 17:28:13.944993 (cutoff 2017-12-06 17:28:15.383148)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11590]: 2017-12-06 17:28:35.471821 7f6a57e74700 -1 osd.12 37761 heartbeat_check: no reply from 192.168.112.135:6815 osd.32 since back 2017-12-06 17:28:14.871165 front 2017-12-06 17:28:14.871165 (cutoff 2017-12-06 17:28:15.471814)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11590]: 2017-12-06 17:28:35.471853 7f6a57e74700 -1 osd.12 37761 heartbeat_check: no reply from 192.168.112.135:6803 osd.33 since back 2017-12-06 17:28:14.871165 front 2017-12-06 17:28:14.871165 (cutoff 2017-12-06 17:28:15.471814)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11590]: 2017-12-06 17:28:35.471861 7f6a57e74700 -1 osd.12 37761 heartbeat_check: no reply from 192.168.112.135:6807 osd.34 since back 2017-12-06 17:28:14.871165 front 2017-12-06 17:28:14.871165 (cutoff 2017-12-06 17:28:15.471814)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11590]: 2017-12-06 17:28:35.471871 7f6a57e74700 -1 osd.12 37761 heartbeat_check: no reply from 192.168.112.135:6827 osd.35 since back 2017-12-06 17:28:14.871165 front 2017-12-06 17:28:14.871165 (cutoff 2017-12-06 17:28:15.471814)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11590]: 2017-12-06 17:28:35.471877 7f6a57e74700 -1 osd.12 37761 heartbeat_check: no reply from 192.168.112.135:6811 osd.36 since back 2017-12-06 17:28:14.871165 front 2017-12-06 17:28:14.871165 (cutoff 2017-12-06 17:28:15.471814)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11590]: 2017-12-06 17:28:35.471888 7f6a57e74700 -1 osd.12 37761 heartbeat_check: no reply from 192.168.112.135:6819 osd.37 since back 2017-12-06 17:28:14.871165 front 2017-12-06 17:28:14.871165 (cutoff 2017-12-06 17:28:15.471814)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11590]: 2017-12-06 17:28:35.471909 7f6a57e74700 -1 osd.12 37761 heartbeat_check: no reply from 192.168.112.135:6823 osd.38 since back 2017-12-06 17:28:14.871165 front 2017-12-06 17:28:14.871165 (cutoff 2017-12-06 17:28:15.471814)
Dec 6 17:28:35 nethcn-b2 ceph-osd[11590]: 2017-12-06 17:28:35.471916 7f6a57e74700 -1 osd.12 37761 heartbeat_check: no reply from 192.168.112.135:6831 osd.39 since back 2017-12-06 17:28:14.871165 front 2017-12-06 17:28:14.871165 (cutoff 2017-12-06 17:28:15.471814)
Dec 6 17:28:35 nethcn-b2 kernel: [87510.180709] libceph: osd32 down
Dec 6 17:28:35 nethcn-b2 kernel: [87510.184066] libceph: osd33 down
Dec 6 17:28:35 nethcn-b2 kernel: [87510.187436] libceph: osd34 down
Dec 6 17:28:35 nethcn-b2 kernel: [87510.190854] libceph: osd35 down
Dec 6 17:28:35 nethcn-b2 kernel: [87510.194260] libceph: osd36 down
Dec 6 17:28:35 nethcn-b2 kernel: [87510.197709] libceph: osd37 down
Dec 6 17:28:35 nethcn-b2 kernel: [87510.201060] libceph: osd38 down
Dec 6 17:28:35 nethcn-b2 kernel: [87510.204407] libceph: osd39 down
Dec 6 17:28:36 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:36Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:28:40 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:40Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:28:44 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:44Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:28:48 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:48Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:28:52 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:52Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:28:56 nethcn-b2 telegraf[29224]: 2017-12-06T16:28:56Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:29:00 nethcn-b2 telegraf[29224]: 2017-12-06T16:29:00Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:29:00 nethcn-b2 systemd[1]: Starting Proxmox VE replication runner...
Dec 6 17:29:01 nethcn-b2 systemd[1]: Started Proxmox VE replication runner.
Dec 6 17:29:04 nethcn-b2 telegraf[29224]: 2017-12-06T16:29:04Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:29:08 nethcn-b2 telegraf[29224]: 2017-12-06T16:29:08Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:29:12 nethcn-b2 telegraf[29224]: 2017-12-06T16:29:12Z E! Error in plugin [inputs.ceph]: took longer to collect than collection interval (4s)
Dec 6 17:29:29 nethcn-b2 nullmailer[914]: Rescanning queue.
Dec 6 17:29:55 nethcn-b2 corosync[10210]: notice [TOTEM ] A new membership (192.168.112.1:2528) was formed. Members joined: 5
Dec 6 17:29:55 nethcn-b2 corosync[10210]: [TOTEM ] A new membership (192.168.112.1:2528) was formed. Members joined: 5
Dec 6 17:29:59 nethcn-b2 corosync[10210]: notice [TOTEM ] Retransmit List: 4
Dec 6 17:29:59 nethcn-b2 corosync[10210]: [TOTEM ] Retransmit List: 4
-------------- next part --------------
Dec 6 17:27:20 nethcn-b5 systemd[1]: Created slice User Slice of root.
Dec 6 17:27:20 nethcn-b5 systemd[1]: Starting User Manager for UID 0...
Dec 6 17:27:20 nethcn-b5 systemd[1]: Started Session 2272 of user root.
Dec 6 17:27:20 nethcn-b5 systemd[9782]: Listening on GnuPG cryptographic agent (access for web browsers).
Dec 6 17:27:20 nethcn-b5 systemd[9782]: Listening on GnuPG network certificate management daemon.
Dec 6 17:27:20 nethcn-b5 systemd[9782]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Dec 6 17:27:20 nethcn-b5 systemd[9782]: Reached target Paths.
Dec 6 17:27:20 nethcn-b5 systemd[9782]: Reached target Timers.
Dec 6 17:27:20 nethcn-b5 systemd[9782]: Listening on GnuPG cryptographic agent and passphrase cache.
Dec 6 17:27:20 nethcn-b5 systemd[9782]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Dec 6 17:27:20 nethcn-b5 systemd[9782]: Reached target Sockets.
Dec 6 17:27:20 nethcn-b5 systemd[9782]: Reached target Basic System.
Dec 6 17:27:20 nethcn-b5 systemd[9782]: Reached target Default.
Dec 6 17:27:20 nethcn-b5 systemd[9782]: Startup finished in 19ms.
Dec 6 17:27:20 nethcn-b5 systemd[1]: Started User Manager for UID 0.
Dec 6 17:27:24 nethcn-b5 kernel: [88684.296467] FW INVALID STATE: IN=vlan31 OUT= MAC=24:8a:07:20:c5:56:24:8a:07:20:c5:5e:08:00 SRC=192.168.112.131 DST=192.168.112.135 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=27581 DF PROTO=TCP SPT=34568 DPT=6789 WINDOW=0 RES=0x00 RST URGP=0
Dec 6 17:27:30 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:30 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 54min 17.746014s random time.
Dec 6 17:27:30 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:30 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 7min 30.184150s random time.
Dec 6 17:27:30 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:30 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 9min 42.427373s random time.
Dec 6 17:27:30 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:30 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 35min 49.985856s random time.
Dec 6 17:27:30 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:30 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 57min 39.588322s random time.
Dec 6 17:27:30 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:30 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 41min 14.870258s random time.
Dec 6 17:27:30 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:30 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 14min 20.468467s random time.
Dec 6 17:27:30 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:30 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 9min 11.475661s random time.
Dec 6 17:27:30 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:30 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 38min 19.555617s random time.
Dec 6 17:27:31 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:31 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 30min 37.001210s random time.
Dec 6 17:27:31 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:32 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 2min 37.078602s random time.
Dec 6 17:27:32 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:32 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 40min 55.466580s random time.
Dec 6 17:27:32 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:32 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 1min 34.251377s random time.
Dec 6 17:27:32 nethcn-b5 systemd[1]: Stopping The Proxmox VE cluster filesystem...
Dec 6 17:27:32 nethcn-b5 pmxcfs[10077]: [main] notice: teardown filesystem
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[11175]: ipcc_send_rec[1] failed: Transport endpoint is not connected
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[11175]: ipcc_send_rec[2] failed: Connection refused
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[11175]: ipcc_send_rec[3] failed: Connection refused
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[11175]: ERROR: Connection refused
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[11175]: server received shutdown request
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[11175]: server stopped
Dec 6 17:27:33 nethcn-b5 systemd[1]: pve-ha-crm.service: Main process exited, code=exited, status=255/n/a
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[14737]: ipcc_send_rec[1] failed: Connection refused
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[14737]: ipcc_send_rec[1] failed: Connection refused
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[14737]: ipcc_send_rec[2] failed: Connection refused
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[14737]: ipcc_send_rec[2] failed: Connection refused
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[14737]: ipcc_send_rec[3] failed: Connection refused
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[14737]: ipcc_send_rec[3] failed: Connection refused
Dec 6 17:27:33 nethcn-b5 pve-ha-crm[14737]: Unable to load access control list: Connection refused
Dec 6 17:27:33 nethcn-b5 systemd[1]: pve-ha-crm.service: Control process exited, code=exited status=111
Dec 6 17:27:33 nethcn-b5 systemd[1]: pve-ha-crm.service: Unit entered failed state.
Dec 6 17:27:33 nethcn-b5 systemd[1]: pve-ha-crm.service: Failed with result 'exit-code'.
Dec 6 17:27:34 nethcn-b5 pmxcfs[10077]: [main] notice: exit proxmox configuration filesystem (0)
Dec 6 17:27:34 nethcn-b5 systemd[1]: Stopped The Proxmox VE cluster filesystem.
Dec 6 17:27:34 nethcn-b5 systemd[1]: Starting The Proxmox VE cluster filesystem...
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [status] notice: update cluster info (cluster name NETHCN-B, version = 7)
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [status] notice: node has quorum
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [dcdb] notice: members: 1/10104, 2/9987, 3/9969, 4/9839, 5/14789
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [dcdb] notice: starting data syncronisation
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [status] notice: members: 1/10104, 2/9987, 3/9969, 4/9839, 5/14789
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [status] notice: starting data syncronisation
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [dcdb] notice: received sync request (epoch 1/10104/00000009)
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [status] notice: received sync request (epoch 1/10104/00000009)
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [dcdb] notice: received all states
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [dcdb] notice: leader is 1/10104
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [dcdb] notice: synced members: 1/10104, 2/9987, 3/9969, 4/9839, 5/14789
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [dcdb] notice: all data is up to date
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [status] notice: received all states
Dec 6 17:27:34 nethcn-b5 pmxcfs[14789]: [status] notice: all data is up to date
Dec 6 17:27:35 nethcn-b5 systemd[1]: Started The Proxmox VE cluster filesystem.
Dec 6 17:27:35 nethcn-b5 systemd[1]: Reloading Proxmox VE firewall.
Dec 6 17:27:35 nethcn-b5 pve-firewall[15692]: send HUP to 10709
Dec 6 17:27:35 nethcn-b5 pve-firewall[10709]: received signal HUP
Dec 6 17:27:35 nethcn-b5 pve-firewall[10709]: server shutdown (restart)
Dec 6 17:27:35 nethcn-b5 systemd[1]: Reloaded Proxmox VE firewall.
Dec 6 17:27:35 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:36 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 17min 21.376308s random time.
Dec 6 17:27:36 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:36 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 17min 58.242872s random time.
Dec 6 17:27:36 nethcn-b5 systemd[1]: Stopping Proxmox VE firewall logger...
Dec 6 17:27:36 nethcn-b5 pvepw-logger[24509]: received terminate request (signal)
Dec 6 17:27:36 nethcn-b5 pvepw-logger[24509]: stopping pvefw logger
Dec 6 17:27:36 nethcn-b5 pve-firewall[10709]: restarting server
Dec 6 17:27:36 nethcn-b5 systemd[1]: Stopped Proxmox VE firewall logger.
Dec 6 17:27:36 nethcn-b5 systemd[1]: Starting Proxmox VE firewall logger...
Dec 6 17:27:36 nethcn-b5 pvefw-logger[15777]: starting pvefw logger
Dec 6 17:27:36 nethcn-b5 systemd[1]: Started Proxmox VE firewall logger.
Dec 6 17:27:36 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:36 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 26min 56.751953s random time.
Dec 6 17:27:36 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:36 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 10min 35.995195s random time.
Dec 6 17:27:36 nethcn-b5 kernel: [88695.943725] audit: type=1400 audit(1512577656.632:14): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="/usr/bin/lxc-start" pid=15841 comm="apparmor_parser"
Dec 6 17:27:36 nethcn-b5 kernel: [88696.136335] audit: type=1400 audit(1512577656.825:15): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="lxc-container-default" pid=15845 comm="apparmor_parser"
Dec 6 17:27:36 nethcn-b5 kernel: [88696.153706] audit: type=1400 audit(1512577656.825:16): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="lxc-container-default-cgns" pid=15845 comm="apparmor_parser"
Dec 6 17:27:36 nethcn-b5 kernel: [88696.172585] audit: type=1400 audit(1512577656.825:17): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="lxc-container-default-with-mounting" pid=15845 comm="apparmor_parser"
Dec 6 17:27:36 nethcn-b5 kernel: [88696.191137] audit: type=1400 audit(1512577656.825:18): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="lxc-container-default-with-nesting" pid=15845 comm="apparmor_parser"
Dec 6 17:27:37 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:37 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 25min 26.022837s random time.
Dec 6 17:27:37 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:37 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 43min 7.212239s random time.
Dec 6 17:27:37 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:37 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 45min 48.451806s random time.
Dec 6 17:27:37 nethcn-b5 systemd[1]: Stopping PVE Local HA Ressource Manager Daemon...
Dec 6 17:27:38 nethcn-b5 pve-ha-lrm[14351]: received signal TERM
Dec 6 17:27:38 nethcn-b5 pve-ha-lrm[14351]: restart LRM, freeze all services
Dec 6 17:27:38 nethcn-b5 pve-ha-lrm[14351]: ipcc_send_rec[1] failed: Transport endpoint is not connected
Dec 6 17:27:39 nethcn-b5 pvestatd[10636]: ipcc_send_rec[1] failed: Transport endpoint is not connected
Dec 6 17:27:48 nethcn-b5 pve-ha-lrm[14351]: watchdog closed (disabled)
Dec 6 17:27:48 nethcn-b5 pve-ha-lrm[14351]: server stopped
Dec 6 17:27:49 nethcn-b5 systemd[1]: Stopped PVE Local HA Ressource Manager Daemon.
Dec 6 17:27:49 nethcn-b5 systemd[1]: Starting PVE Cluster Ressource Manager Daemon...
Dec 6 17:27:49 nethcn-b5 pve-ha-crm[16740]: starting server
Dec 6 17:27:49 nethcn-b5 pve-ha-crm[16740]: status change startup => wait_for_quorum
Dec 6 17:27:49 nethcn-b5 systemd[1]: Started PVE Cluster Ressource Manager Daemon.
Dec 6 17:27:49 nethcn-b5 systemd[1]: Starting PVE Local HA Ressource Manager Daemon...
Dec 6 17:27:50 nethcn-b5 pve-ha-lrm[16775]: starting server
Dec 6 17:27:50 nethcn-b5 pve-ha-lrm[16775]: status change startup => wait_for_agent_lock
Dec 6 17:27:50 nethcn-b5 systemd[1]: Started PVE Local HA Ressource Manager Daemon.
Dec 6 17:27:50 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:50 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 42min 4.585055s random time.
Dec 6 17:27:50 nethcn-b5 systemd[1]: Stopping PVE Cluster Ressource Manager Daemon...
Dec 6 17:27:50 nethcn-b5 pve-ha-crm[16740]: received signal TERM
Dec 6 17:27:50 nethcn-b5 pve-ha-crm[16740]: server received shutdown request
Dec 6 17:27:54 nethcn-b5 pve-ha-crm[16740]: status change wait_for_quorum => slave
Dec 6 17:27:54 nethcn-b5 pve-ha-crm[16740]: server stopped
Dec 6 17:27:55 nethcn-b5 systemd[1]: Stopped PVE Cluster Ressource Manager Daemon.
Dec 6 17:27:55 nethcn-b5 systemd[1]: Starting PVE Cluster Ressource Manager Daemon...
Dec 6 17:27:55 nethcn-b5 pve-ha-crm[17429]: starting server
Dec 6 17:27:55 nethcn-b5 pve-ha-crm[17429]: status change startup => wait_for_quorum
Dec 6 17:27:55 nethcn-b5 systemd[1]: Started PVE Cluster Ressource Manager Daemon.
Dec 6 17:27:56 nethcn-b5 systemd[1]: Reloading.
Dec 6 17:27:56 nethcn-b5 systemd[1]: apt-daily-upgrade.timer: Adding 32min 16.418414s random time.
Dec 6 17:27:56 nethcn-b5 systemd[1]: Reloading PVE API Daemon.
Dec 6 17:27:57 nethcn-b5 pvedaemon[17563]: send HUP to 10872
Dec 6 17:27:57 nethcn-b5 pvedaemon[10872]: received signal HUP
Dec 6 17:27:57 nethcn-b5 pvedaemon[10872]: server closing
Dec 6 17:27:57 nethcn-b5 pvedaemon[10872]: server shutdown (restart)
Dec 6 17:27:57 nethcn-b5 pvedaemon[10874]: worker exit
Dec 6 17:27:57 nethcn-b5 pvedaemon[10873]: worker exit
Dec 6 17:27:57 nethcn-b5 pvedaemon[10875]: worker exit
Dec 6 17:27:57 nethcn-b5 systemd[1]: Reloaded PVE API Daemon.
Dec 6 17:27:57 nethcn-b5 systemd[1]: Reloading PVE API Proxy Server.
Dec 6 17:27:58 nethcn-b5 pvedaemon[10872]: restarting server
Dec 6 17:27:58 nethcn-b5 pvedaemon[10872]: starting 3 worker(s)
Dec 6 17:27:58 nethcn-b5 pvedaemon[10872]: worker 17616 started
Dec 6 17:27:58 nethcn-b5 pvedaemon[10872]: worker 17617 started
Dec 6 17:27:58 nethcn-b5 pvedaemon[10872]: worker 17618 started
Dec 6 17:27:58 nethcn-b5 pveproxy[17600]: send HUP to 13530
Dec 6 17:27:58 nethcn-b5 pveproxy[13530]: received signal HUP
Dec 6 17:27:58 nethcn-b5 pveproxy[13530]: server closing
Dec 6 17:27:58 nethcn-b5 pveproxy[13530]: server shutdown (restart)
Dec 6 17:27:58 nethcn-b5 pveproxy[13533]: worker exit
Dec 6 17:27:58 nethcn-b5 pveproxy[13532]: worker exit
Dec 6 17:27:58 nethcn-b5 pveproxy[13531]: worker exit
Dec 6 17:27:58 nethcn-b5 systemd[1]: Reloaded PVE API Proxy Server.
Dec 6 17:27:58 nethcn-b5 systemd[1]: Reloading PVE SPICE Proxy Server.
Dec 6 17:27:58 nethcn-b5 spiceproxy[17623]: send HUP to 13563
Dec 6 17:27:58 nethcn-b5 spiceproxy[13563]: received signal HUP
Dec 6 17:27:58 nethcn-b5 spiceproxy[13563]: server closing
Dec 6 17:27:58 nethcn-b5 spiceproxy[13563]: server shutdown (restart)
Dec 6 17:27:58 nethcn-b5 spiceproxy[13564]: worker exit
Dec 6 17:27:58 nethcn-b5 systemd[1]: Reloaded PVE SPICE Proxy Server.
Dec 6 17:27:58 nethcn-b5 systemd[1]: Reloading PVE Status Daemon.
Dec 6 17:27:58 nethcn-b5 pveproxy[13530]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Dec 6 17:27:58 nethcn-b5 pveproxy[13530]: restarting server
Dec 6 17:27:58 nethcn-b5 pveproxy[13530]: starting 3 worker(s)
Dec 6 17:27:58 nethcn-b5 pveproxy[13530]: worker 17637 started
Dec 6 17:27:58 nethcn-b5 pveproxy[13530]: worker 17639 started
Dec 6 17:27:58 nethcn-b5 pveproxy[13530]: worker 17640 started
Dec 6 17:27:58 nethcn-b5 spiceproxy[13563]: restarting server
Dec 6 17:27:58 nethcn-b5 spiceproxy[13563]: starting 1 worker(s)
Dec 6 17:27:58 nethcn-b5 spiceproxy[13563]: worker 17644 started
Dec 6 17:27:58 nethcn-b5 pvestatd[17634]: send HUP to 10636
Dec 6 17:27:58 nethcn-b5 pvestatd[10636]: received signal HUP
Dec 6 17:27:58 nethcn-b5 pvestatd[10636]: server shutdown (restart)
Dec 6 17:27:58 nethcn-b5 systemd[1]: Reloaded PVE Status Daemon.
Dec 6 17:27:59 nethcn-b5 pvestatd[10636]: restarting server
Dec 6 17:28:00 nethcn-b5 systemd[1]: Starting Proxmox VE replication runner...
Dec 6 17:28:00 nethcn-b5 pve-ha-lrm[16775]: successfully acquired lock 'ha_agent_nethcn-b5_lock'
Dec 6 17:28:00 nethcn-b5 pve-ha-lrm[16775]: watchdog active
Dec 6 17:28:00 nethcn-b5 pve-ha-lrm[16775]: status change wait_for_agent_lock => active
Dec 6 17:28:00 nethcn-b5 systemd[1]: Started Proxmox VE replication runner.
Dec 6 17:28:00 nethcn-b5 pve-ha-crm[17429]: status change wait_for_quorum => slave
Dec 6 17:28:01 nethcn-b5 cron[10376]: (*system*pveupdate) RELOAD (/etc/cron.d/pveupdate)
Dec 6 17:28:01 nethcn-b5 CRON[18584]: (root) CMD ( sleep $((RANDOM % 20)); /usr/local/sbin/check_ipmi.sh)
Dec 6 17:28:03 nethcn-b5 pvedaemon[10872]: worker 10873 finished
Dec 6 17:28:03 nethcn-b5 pvedaemon[10872]: worker 10874 finished
Dec 6 17:28:03 nethcn-b5 pvedaemon[10872]: worker 10875 finished
Dec 6 17:28:03 nethcn-b5 pveproxy[13530]: worker 13531 finished
Dec 6 17:28:03 nethcn-b5 pveproxy[13530]: worker 13532 finished
Dec 6 17:28:03 nethcn-b5 pveproxy[13530]: worker 13533 finished
Dec 6 17:28:03 nethcn-b5 spiceproxy[13563]: worker 13564 finished
Dec 6 17:28:06 nethcn-b5 systemd[1]: Stopping LXC Container Monitoring Daemon...
Dec 6 17:28:06 nethcn-b5 systemd[1]: Stopped LXC Container Monitoring Daemon.
Dec 6 17:28:06 nethcn-b5 systemd[1]: Started LXC Container Monitoring Daemon.
Dec 6 17:28:06 nethcn-b5 systemd[1]: Stopping Proxmox VE watchdog multiplexer...
Dec 6 17:28:06 nethcn-b5 watchdog-mux[3747]: got terminate request
Dec 6 17:28:06 nethcn-b5 watchdog-mux[3747]: exit watchdog-mux with active connections
Dec 6 17:28:06 nethcn-b5 systemd[1]: Stopped Proxmox VE watchdog multiplexer.
Dec 6 17:28:06 nethcn-b5 systemd[1]: Started Proxmox VE watchdog multiplexer.
Dec 6 17:28:06 nethcn-b5 kernel: [88725.955509] watchdog: watchdog0: watchdog did not stop!
Dec 6 17:28:06 nethcn-b5 watchdog-mux[18946]: watchdog active - unable to restart watchdog-mux
Dec 6 17:28:06 nethcn-b5 systemd[1]: watchdog-mux.service: Main process exited, code=exited, status=1/FAILURE
Dec 6 17:28:06 nethcn-b5 systemd[1]: watchdog-mux.service: Unit entered failed state.
Dec 6 17:28:06 nethcn-b5 systemd[1]: watchdog-mux.service: Failed with result 'exit-code'.
Dec 6 17:28:10 nethcn-b5 pve-ha-lrm[16775]: watchdog update failed - Broken pipe
Dec 6 17:29:41 nethcn-b5 systemd-modules-load[1761]: Inserted module 'iscsi_tcp'
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] random: get_random_bytes called from start_kernel+0x42/0x4f3 with crng_init=0
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] Linux version 4.13.8-3-pve (root at nora) (gcc version 6.3.0 20170516 (Debian 6.3.0-18)) #1 SMP PVE 4.13.8-30 (Tue, 5 Dec 2017 13:06:48 +0100) ()
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] Command line: BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-4.13.8-3-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs elevator=noop console=tty0 console=ttyS1,115200n8
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] KERNEL supported cpus:
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] Intel GenuineIntel
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] AMD AuthenticAMD
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] Centaur CentaurHauls
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] e820: BIOS-provided physical RAM map:
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000099bff] usable
Dec 6 17:29:41 nethcn-b5 kernel: [ 0.000000] BIOS-e820: [mem 0x0000000000099c00-0x000000000009ffff] reserved
More information about the pve-user
mailing list