[PVE-User] Corosync Upgrade Issue (v2 -> v3)

[Ricardo Correa]

Hi Fabian,

Thanks for replying. Here are the complete logs:

root at pve01:~# systemctl stop pve-ha-lrm
root at pve01:~# systemctl stop pve-ha-crm
root at pve01:~# echo "deb http://download.proxmox.com/debian/corosync-3/
stretch main" > /etc/apt/sources.list.d/corosync3.list
root at pve01:~# apt update
Ign:1 http://ftp.ch.debian.org/debian stretch InRelease
Hit:2 http://ftp.ch.debian.org/debian stretch-updates InRelease
Hit:3 http://ftp.ch.debian.org/debian stretch Release
Get:4 http://download.proxmox.com/debian/corosync-3 stretch InRelease
[1,977 B]
Hit:5 http://download.proxmox.com/debian/ceph-luminous stretch InRelease
Hit:6 http://download.proxmox.com/debian/pve stretch InRelease
Hit:7 http://security.debian.org stretch/updates InRelease
Get:9 http://download.proxmox.com/debian/corosync-3 stretch/main amd64
Packages [12.4 kB]
Fetched 14.3 kB in 0s (33.0 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
7 packages can be upgraded. Run 'apt list --upgradable' to see them.
root at pve01:~# apt list --upgradeable
Listing... Done
corosync/stable 3.0.2-pve2~bpo9 amd64 [upgradable from: 2.4.4-pve1]
libcmap4/stable 3.0.2-pve2~bpo9 amd64 [upgradable from: 2.4.4-pve1]
libcorosync-common4/stable 3.0.2-pve2~bpo9 amd64 [upgradable from:
2.4.4-pve1]
libcpg4/stable 3.0.2-pve2~bpo9 amd64 [upgradable from: 2.4.4-pve1]
libqb0/stable 1.0.5-1~bpo9+2 amd64 [upgradable from: 1.0.3-1~bpo9]
libquorum5/stable 3.0.2-pve2~bpo9 amd64 [upgradable from: 2.4.4-pve1]
libvotequorum8/stable 3.0.2-pve2~bpo9 amd64 [upgradable from: 2.4.4-pve1]
root at pve01:~# apt dist-upgrade --download-only
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  libcfg7 libknet1 libzstd1
The following packages will be upgraded:
  corosync libcmap4 libcorosync-common4 libcpg4 libqb0 libquorum5
libvotequorum8
7 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,405 kB of archives.
After this operation, 1,671 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://ftp.ch.debian.org/debian stretch/main amd64 libzstd1 amd64
1.1.2-1 [193 kB]
Get:2 http://download.proxmox.com/debian/corosync-3 stretch/main amd64
libcorosync-common4 amd64 3.0.2-pve2~bpo9 [244 kB]
Get:3 http://download.proxmox.com/debian/corosync-3 stretch/main amd64
libqb0 amd64 1.0.5-1~bpo9+2 [140 kB]
Get:4 http://download.proxmox.com/debian/corosync-3 stretch/main amd64
libcfg7 amd64 3.0.2-pve2~bpo9 [247 kB]
Get:5 http://download.proxmox.com/debian/corosync-3 stretch/main amd64
libcmap4 amd64 3.0.2-pve2~bpo9 [248 kB]
Get:6 http://download.proxmox.com/debian/corosync-3 stretch/main amd64
libcpg4 amd64 3.0.2-pve2~bpo9 [250 kB]
Get:7 http://download.proxmox.com/debian/corosync-3 stretch/main amd64
libknet1 amd64 1.10-pve1~bpo9 [129 kB]
Get:8 http://download.proxmox.com/debian/corosync-3 stretch/main amd64
libquorum5 amd64 3.0.2-pve2~bpo9 [246 kB]
Get:9 http://download.proxmox.com/debian/corosync-3 stretch/main amd64
libvotequorum8 amd64 3.0.2-pve2~bpo9 [247 kB]
Get:10 http://download.proxmox.com/debian/corosync-3 stretch/main amd64
corosync amd64 3.0.2-pve2~bpo9 [461 kB]
Fetched 2,405 kB in 0s (2,434 kB/s)
Download complete and in download only mode
root at pve01:~# apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  libcfg7 libknet1 libzstd1
The following packages will be upgraded:
  corosync libcmap4 libcorosync-common4 libcpg4 libqb0 libquorum5
libvotequorum8
7 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/2,405 kB of archives.
After this operation, 1,671 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Reading changelogs... Done
(Reading database ... 59056 files and directories currently installed.)
Preparing to unpack .../0-libcorosync-common4_3.0.2-pve2~bpo9_amd64.deb ...
Unpacking libcorosync-common4:amd64 (3.0.2-pve2~bpo9) over (2.4.4-pve1) ...
Preparing to unpack .../1-libqb0_1.0.5-1~bpo9+2_amd64.deb ...
Unpacking libqb0:amd64 (1.0.5-1~bpo9+2) over (1.0.3-1~bpo9) ...
Selecting previously unselected package libcfg7:amd64.
Preparing to unpack .../2-libcfg7_3.0.2-pve2~bpo9_amd64.deb ...
Unpacking libcfg7:amd64 (3.0.2-pve2~bpo9) ...
Preparing to unpack .../3-libcmap4_3.0.2-pve2~bpo9_amd64.deb ...
Unpacking libcmap4:amd64 (3.0.2-pve2~bpo9) over (2.4.4-pve1) ...
Preparing to unpack .../4-libcpg4_3.0.2-pve2~bpo9_amd64.deb ...
Unpacking libcpg4:amd64 (3.0.2-pve2~bpo9) over (2.4.4-pve1) ...
Selecting previously unselected package libzstd1.
Preparing to unpack .../5-libzstd1_1.1.2-1_amd64.deb ...
Unpacking libzstd1 (1.1.2-1) ...
Selecting previously unselected package libknet1:amd64.
Preparing to unpack .../6-libknet1_1.10-pve1~bpo9_amd64.deb ...
Unpacking libknet1:amd64 (1.10-pve1~bpo9) ...
Preparing to unpack .../7-libquorum5_3.0.2-pve2~bpo9_amd64.deb ...
Unpacking libquorum5:amd64 (3.0.2-pve2~bpo9) over (2.4.4-pve1) ...
Preparing to unpack .../8-libvotequorum8_3.0.2-pve2~bpo9_amd64.deb ...
Unpacking libvotequorum8:amd64 (3.0.2-pve2~bpo9) over (2.4.4-pve1) ...
Preparing to unpack .../9-corosync_3.0.2-pve2~bpo9_amd64.deb ...
Unpacking corosync (3.0.2-pve2~bpo9) over (2.4.4-pve1) ...
Processing triggers for pve-manager (5.4-11) ...
Setting up libzstd1 (1.1.2-1) ...
Errors were encountered while processing:
 libqb0:amd64
 libvotequorum8:amd64
 libcpg4:amd64
 libcmap4:amd64
 libquorum5:amd64
 libcfg7:amd64
 corosync
E: Sub-process /usr/bin/dpkg returned an error code (1)
W: Operation was interrupted before it could finish

I do see three new packages being installed, is it possible these caused it?

libcfg7 libknet1 libzstd1

I'm just looking at these and they are dependencies of corosync (3) and
each other so it couldn't have been
these alone.

Kind regards,
Ricardo Correa

On Tue, Jul 16, 2019 at 9:55 PM Fabian Grünbichler <
f.gruenbichler at proxmox.com> wrote:

> > Ricardo Correa <r.correa.r at gmail.com> hat am 16. Juli 2019 21:28
> geschrieben:
> >
> >
> > Hello all,
> >
> > While following the instructions for upgrade I encountered the following
> > issue:
> >
> > ~# systemctl stop pve-ha-lrm
> > ~# systemctl stop pve-ha-crm
> > ~# echo "deb http://download.proxmox.com/debian/corosync-3/ stretch
> main" >
> > /etc/apt/sources.list.d/corosync3.list
> > ~# apt update
> > ...snip...
> > ~# apt dist-upgrade --download-only
> > ...snip...
> > ~# apt dist-upgrade
> > Reading package lists... Done
> > ...snip...
> > The following packages will be upgraded:
> >   corosync libcmap4 libcorosync-common4 libcpg4 libqb0 libquorum5
> > libvotequorum8
> > ...snip...
>
> please provide "unsnipped" logs, both of these commands and from the
> journal around the same time.
>
> > Setting up libcorosync-common4:amd64 (3.0.2-pve2~bpo9) ...
> > Processing triggers for pve-ha-manager (2.0-9) ...
> > Processing triggers for man-db (2.7.6.1-2) ...
> > Setting up libqb0:amd64 (1.0.5-1~bpo9+2) ...
> > systemctl start pve-ha-lrmJob for pve-cluster.service failed because a
> > timeout was exceeded.
> > See "systemctl status pve-cluster.service" and "journalctl -xe" for
> details.
> > dpkg: error processing package libqb0:amd64 (--configure):
> >  subprocess installed post-installation script returned error exit
> status 1
> > Setting up libknet1:amd64 (1.10-pve1~bpo9) ...
> > dpkg: dependency problems prevent configuration of libvotequorum8:amd64:
> >  libvotequorum8:amd64 depends on libqb0 (>= 1.0.3); however:
> >   Package libqb0:amd64 is not configured yet.
> >
> > dpkg: error processing package libvotequorum8:amd64 (--configure):
> >  dependency problems - leaving unconfigured
> > dpkg: dependency problems prevent configuration of libcpg4:amd64:
> >  libcpg4:amd64 depends on libqb0 (>= 1.0.3); however:
> >   Package libqb0:amd64 is not configured yet.
> >
> > dpkg: error processing package libcpg4:amd64 (--configure):
> >  dependency problems - leaving unconfigured
> > dpkg: dependency problems prevent configuration of libcmap4:amd64:
> >  libcmap4:amd64 depends on libqb0 (>= 1.0.3); however:
> >   Package libqb0:amd64 is not configured yet.
> >
> > dpkg: error processing package libcmap4:amd64 (--configure):
> >  dependency problems - leaving unconfigured
> > dpkg: dependency problems prevent configuration of libquorum5:amd64:
> >  libquorum5:amd64 depends on libqb0 (>= 1.0.3); however:
> >   Package libqb0:amd64 is not configured yet.
> >
> > dpkg: error processing package libquorum5:amd64 (--configure):
> >  dependency problems - leaving unconfigured
> > dpkg: dependency problems prevent configuration of libcfg7:amd64:
> >  libcfg7:amd64 depends on libqb0 (>= 1.0.3); however:
> >   Package libqb0:amd64 is not configured yet.
> >
> > dpkg: error processing package libcfg7:amd64 (--configure):
> >  dependency problems - leaving unconfigured
> > dpkg: dependency problems prevent configuration of corosync:
> >  corosync depends on libcfg7 (>= 2.99.5); however:
> >   Package libcfg7:amd64 is not configured yet.
> >  corosync depends on libcmap4 (>= 2.99.3); however:
> >   Package libcmap4:amd64 is not configured yet.
> >  corosync depends on libcpg4 (>= 1.1.0); however:
> >   Package libcpg4:amd64 is not configured yet.
> >  corosync depends on libqb0 (>= 1.0.3); however:
> >   Package libqb0:amd64 is not configured yet.
> >  corosync depends on libquorum5 (>= 1.99.9); however:
> >   Package libquorum5:amd64 is not configured yet.
> >  corosync depends on libvotequorum8 (>= 2.4.2); however:
> >   Package libvotequorum8:amd64 is not configured yet.
> >
> > dpkg: error processing package corosync (--configure):
> >  dependency problems - leaving unconfigured
> > Processing triggers for libc-bin (2.24-11+deb9u4) ...
> > Processing triggers for pve-ha-manager (2.0-9) ...
> > Processing triggers for pve-manager (5.4-11) ...
>
> if you followed the upgrade guidelines and ONLY upgraded corosync here,
> these two triggers should not be triggered...
>
> > Errors were encountered while processing:
> >  libqb0:amd64
> >  libvotequorum8:amd64
> >  libcpg4:amd64
> >  libcmap4:amd64
> >  libquorum5:amd64
> >  libcfg7:amd64
> >  corosync
> > E: Sub-process /usr/bin/dpkg returned an error code (1)
> > W: Operation was interrupted before it could finish
> >
> > This was while updating all 3 of my nodes at the same time (taking second
> > route
> > of the upgrade instructions, i.e., tmuxing into all nodes while having
> the
> > main
> > tmux syncying commands). Only one of those nodes failed to update, other
> > two are
> > running fine right now and in quorum apparently (pvecm status says
> quorate:
> > yes).
> >
> > Right now I'm not sure how to proceed, pve-cluster in the failing node
> > is looping with:
> >
> > systemd[1]: Starting The Proxmox VE cluster filesystem...
> > systemd[1]: pve-cluster.service: Start operation timed out. Terminating.
> > pmxcfs[13267]: [main] crit: read error: Interrupted system call
> > pmxcfs[13267]: [main] crit: read error: Interrupted system call
> > pmxcfs[13267]: [main] notice: exit proxmox configuration filesystem (-1)
> > systemd[1]: pve-cluster.service: Control process exited, code=exited
> > status=255
> > systemd[1]: Failed to start The Proxmox VE cluster filesystem.
> > systemd[1]: pve-cluster.service: Unit entered failed state.
> > systemd[1]: pve-cluster.service: Failed with result 'timeout'.
> > systemd[1]: pve-cluster.service: Service hold-off time over, scheduling
> > restart.
> > systemd[1]: Stopped The Proxmox VE cluster filesystem.
> >
> > And pmxcfs is unavailable (ls /etc/pve timeouts).
> >
> > Any help is appreciated! (BTW, just joined the ML to post this, also
> joined
> > the
> > forums but can't post).
> >
> > Best regards,
> > Ricardo
> > _______________________________________________
> > pve-user mailing list
> > pve-user at pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
>