[PVE-User] replication problems after upgrade to 5.0-15

nmachkova at verdnatura.es nmachkova at verdnatura.es
Wed Nov 29 12:44:44 CET 2017

I am using ZFS for CTs at PROXMOX cluster with 2 nodes without HA since 
August 2017
I installed proxmox-ve_5.0-af4267bf-4.iso at 2 old servers
and proxmox is really great 8-)))
but I did upgrade at both nodes and something is wrong, because I am 
unable to do CT replications or migrations

example of replication of CT 106(goat) from node mox11 => mox

========== errors from webGUI

2017-11-29 10:42:01 106-0: start replication job
2017-11-29 10:42:01 106-0: guest => CT 106, running => 1
2017-11-29 10:42:01 106-0: volumes => zfs:subvol-106-disk-1
2017-11-29 10:42:02 106-0: freeze guest filesystem
2017-11-29 10:42:03 106-0: create snapshot 
'__replicate_106-0_1511948521__' on zfs:subvol-106-disk-1
2017-11-29 10:42:03 106-0: thaw guest filesystem
2017-11-29 10:42:03 106-0: full sync 'zfs:subvol-106-disk-1' 
2017-11-29 10:42:04 106-0: internal error: Invalid argument
2017-11-29 10:42:04 106-0: command 'zfs send -Rpv -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511948521__' failed: got 
signal 6
2017-11-29 10:42:04 106-0: cannot receive: failed to read from stream
2017-11-29 10:42:04 106-0: cannot open 'ctpool/subvol-106-disk-1': 
dataset does not exist
2017-11-29 10:42:04 106-0: command 'zfs recv -F -- 
ctpool/subvol-106-disk-1' failed: exit code 1
2017-11-29 10:42:04 106-0: delete previous replication snapshot 
'__replicate_106-0_1511948521__' on zfs:subvol-106-disk-1
2017-11-29 10:42:04 106-0: end replication job with error: command 'set 
-o pipefail && pvesm export zfs:subvol-106-disk-1 zfs - -with-snapshots 
1 -snapshot __replicate_106-0_1511948521__ | /usr/bin/ssh -o 
'BatchMode=yes' -o 'HostKeyAlias=mox' root at -- pvesm import 
zfs:subvol-106-disk-1 zfs - -with-snapshots 1' failed: exit code 1

=========== version (same at both nodes)

proxmox-ve: 5.0-15 (running kernel: 4.10.15-1-pve)
pve-manager: 5.1-36 (running version: 5.1-36/131401db)
pve-kernel-4.10.15-1-pve: 4.10.15-15
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-15
qemu-server: 5.0-12
pve-firmware: 2.0-3
libpve-common-perl: 5.0-20
libpve-guest-common-perl: 2.0-13
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-16
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.0-2
pve-container: 2.0-17
pve-firewall: 3.0-3
pve-ha-manager: 2.0-3
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9

========== zfs status

=== node mox
# zfs list -t all -r ctpool
NAME                                        USED  AVAIL  REFER  
ctpool                                     3.41G  25.4G   112K  /ctpool
ctpool/subvol-110-disk-1                    503M   521M   503M  
ctpool/subvol-251-disk-1                    478M  3.54G   468M  
ctpool/subvol-251-disk-1 at dyn01sharednet    10.1M      -   467M  -
ctpool/subvol-252-disk-1                    478M  3.54G   468M  
ctpool/subvol-252-disk-1 at dyn01sharednet    10.2M      -   468M  -
ctpool/subvol-301-disk-1                    690M  3.49G   518M  
ctpool/subvol-301-disk-1 at silla70y100        151M      -   538M  -
ctpool/subvol-301-disk-1 at silla70y100fixed  18.3M      -   526M  -
ctpool/subvol-302-disk-1                    531M  3.54G   470M  
ctpool/subvol-302-disk-1 at silla70y100       40.3M      -   472M  -
ctpool/subvol-302-disk-1 at silla70y100fixed  17.6M      -   478M  -
ctpool/subvol-501-disk-1                    402M   110M   402M  
ctpool/subvol-502-disk-1                    388M   124M   388M  

=== node mox11
# zfs list -t all -r ctpool
NAME                                     USED  AVAIL  REFER  MOUNTPOINT
ctpool                                  3.97G  24.8G    96K  /ctpool
ctpool/subvol-102-disk-1                 589M  7.42G   589M  
ctpool/subvol-103-disk-1                 870M  7.16G   855M  
ctpool/subvol-103-disk-1 at campana        9.42M      -   756M  -
ctpool/subvol-103-disk-1 at beforenodeupg  4.87M      -   855M  -
ctpool/subvol-106-disk-1                 427M  3.59G   424M  
ctpool/subvol-106-disk-1 at goat_apache2   3.08M      -   423M  -
ctpool/subvol-111-disk-1                1.65G  6.58G  1.42G  
ctpool/subvol-111-disk-1 at postgresql     53.4M      -   513M  -
ctpool/subvol-111-disk-1 at phppgadmin     2.00M      -   524M  -
ctpool/subvol-111-disk-1 at pgwebssl       1.93M      -   524M  -
ctpool/subvol-111-disk-1 at redmine01      28.8M      -   714M  -
ctpool/subvol-111-disk-1 at redmine02      31.5M      -   882M  -
ctpool/subvol-111-disk-1 at redmine03      43.4M      -  1.42G  -
ctpool/subvol-111-disk-1 at redmine04      19.0M      -  1.42G  -
ctpool/subvol-111-disk-1 at redmine05      7.14M      -  1.42G  -
ctpool/subvol-220-disk-1                 470M   554M   470M  

============== node mox11
# zpool history ctpool
2017-11-28.14:26:08 zfs destroy 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511875560__
2017-11-28.14:32:05 zpool import -c /etc/zfs/zpool.cache -aN
2017-11-28.15:10:04 zfs snapshot 
ctpool/subvol-103-disk-1 at __replicate_103-0_1511878201__
2017-11-28.15:10:09 zfs destroy 
ctpool/subvol-103-disk-1 at __replicate_103-0_1511878201__
2017-11-28.15:15:04 zfs snapshot 
ctpool/subvol-103-disk-1 at __replicate_103-0_1511878501__
2017-11-28.15:15:09 zfs destroy 
ctpool/subvol-103-disk-1 at __replicate_103-0_1511878501__
2017-11-28.15:43:59 zfs snapshot ctpool/subvol-106-disk-1 at __migration__
2017-11-28.15:44:04 zfs destroy ctpool/subvol-106-disk-1 at __migration__
2017-11-29.10:42:04 zfs snapshot 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511948521__
2017-11-29.10:42:09 zfs destroy 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511948521__
2017-11-29.10:47:04 zfs snapshot 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511948821__
2017-11-29.10:47:09 zfs destroy 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511948821__
2017-11-29.10:57:04 zfs snapshot 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511949421__
2017-11-29.10:57:09 zfs destroy 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511949421__

============== node mox
# zpool history ctpool
2017-11-28.05:10:40 zfs get -o value -Hp available,used ctpool
2017-11-28.05:33:09 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.06:03:09 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.06:09:59 zpool list -o name -H ctpool
2017-11-28.06:21:30 zfs get -o value -Hp available,used ctpool
2017-11-28.06:33:10 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.07:03:09 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.07:04:52 zpool list -o name -H ctpool
2017-11-28.07:04:57 zfs get -o value -Hp available,used ctpool
2017-11-28.07:33:09 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.08:03:09 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.08:04:21 zfs get -o value -Hp available,used ctpool
2017-11-28.08:07:41 zpool list -o name -H ctpool
2017-11-28.08:33:09 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.08:56:12 zpool list -o name -H ctpool
2017-11-28.09:03:10 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.09:04:21 zfs get -o value -Hp available,used ctpool
2017-11-28.09:16:59 zfs get -o value -Hp available,used ctpool
2017-11-28.09:33:08 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.10:03:09 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.10:33:09 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.11:03:09 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.11:13:29 zpool list -o name -H ctpool
2017-11-28.11:33:09 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.12:03:09 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.12:09:49 zpool list -o name -H ctpool
2017-11-28.12:22:47 zpool list -o name -H ctpool
2017-11-28.12:33:08 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.12:38:41 zpool list -o name -H ctpool
2017-11-28.12:42:11 zpool list -o name -H ctpool
2017-11-28.13:07:37 zfs get -o value -Hp available,used ctpool
2017-11-28.13:10:59 zpool import -c /etc/zfs/zpool.cache -aN
2017-11-28.13:33:10 zfs rollback -r -- 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.13:43:02 zfs destroy 
ctpool/subvol-106-disk-1 at __replicate_106-0_1511251201__
2017-11-28.13:43:09 zfs destroy -r ctpool/subvol-106-disk-1
2017-11-28.14:12:19 zfs destroy ctpool/subvol-103-disk-1
2017-11-29.04:50:24 zfs get -o value -Hp available,used ctpool
2017-11-29.10:39:14 zfs set compression=lz4 ctpool

# zpool status
   pool: ctpool
  state: ONLINE
status: Some supported features are not enabled on the pool. The pool 
	still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(5) for details.
   scan: scrub repaired 0B in 0h6m with 0 errors on Sun Nov 26 00:30:42 

	ctpool      ONLINE       0     0     0
	  zfs       ONLINE       0     0     0

errors: No known data errors

root at mox:~# zfs get all ctpool
NAME    PROPERTY              VALUE                  SOURCE
ctpool  type                  filesystem             -
ctpool  creation              Mon Jul 31 16:48 2017  -
ctpool  used                  3.41G                  -
ctpool  available             25.4G                  -
ctpool  referenced            112K                   -
ctpool  compressratio         1.87x                  -
ctpool  mounted               yes                    -
ctpool  quota                 none                   default
ctpool  reservation           none                   default
ctpool  recordsize            128K                   default
ctpool  mountpoint            /ctpool                default
ctpool  sharenfs              off                    default
ctpool  checksum              on                     default
ctpool  compression           lz4                    local
ctpool  atime                 on                     default
ctpool  devices               on                     default
ctpool  exec                  on                     default
ctpool  setuid                on                     default
ctpool  readonly              off                    default
ctpool  zoned                 off                    default
ctpool  snapdir               hidden                 default
ctpool  aclinherit            restricted             default
ctpool  createtxg             1                      -
ctpool  canmount              on                     default
ctpool  xattr                 on                     default
ctpool  copies                1                      default
ctpool  version               5                      -
ctpool  utf8only              off                    -
ctpool  normalization         none                   -
ctpool  casesensitivity       sensitive              -
ctpool  vscan                 off                    default
ctpool  nbmand                off                    default
ctpool  sharesmb              off                    default
ctpool  refquota              none                   default
ctpool  refreservation        none                   default
ctpool  guid                  2622709745618035732    -
ctpool  primarycache          all                    default
ctpool  secondarycache        all                    default
ctpool  usedbysnapshots       0B                     -
ctpool  usedbydataset         112K                   -
ctpool  usedbychildren        3.41G                  -
ctpool  usedbyrefreservation  0B                     -
ctpool  logbias               latency                default
ctpool  dedup                 off                    default
ctpool  mlslabel              none                   default
ctpool  sync                  standard               default
ctpool  dnodesize             legacy                 default
ctpool  refcompressratio      1.00x                  -
ctpool  written               112K                   -
ctpool  logicalused           5.79G                  -
ctpool  logicalreferenced     45.5K                  -
ctpool  volmode               default                default
ctpool  filesystem_limit      none                   default
ctpool  snapshot_limit        none                   default
ctpool  filesystem_count      none                   default
ctpool  snapshot_count        none                   default
ctpool  snapdev               hidden                 default
ctpool  acltype               off                    default
ctpool  context               none                   default
ctpool  fscontext             none                   default
ctpool  defcontext            none                   default
ctpool  rootcontext           none                   default
ctpool  relatime              off                    default
ctpool  redundant_metadata    all                    default
ctpool  overlay               off                    default

I disabled and removed ALL the replication tasks and tried to find some 
answer at
# systemctl status zed
# pvesr status

BUT this is interesting, I tried to upgrade the pool as it is 
recommended via zpool status

# zpool upgrade -a
This system supports ZFS pool feature flags.

cannot set property for 'ctpool': invalid argument for this pool 
=============== end

hm, replication error is occuring just after some "sync" operation so 
these are my packages related to sync

# dpkg -l |grep sync
ii  corosync                             2.4.2-pve3                     
amd64        cluster engine daemon and utilities
ii  libasyncns0:amd64                    0.8-6                          
amd64        Asynchronous name service query library
ii  libcorosync-common4:amd64            2.4.2-pve3                     
amd64        cluster engine common library
ii  libevent-2.0-5:amd64                 2.0.21-stable-3                
amd64        Asynchronous event notification library
ii  libfile-sync-perl                    0.11-2+b3                      
amd64        Perl interface to sync() and fsync()
ii  libpve-http-server-perl              2.0-6                          
all          Proxmox Asynchrounous HTTP Server Implementation
ii  rsync                                3.1.2-1                        
amd64        fast, versatile, remote (and local) file-copying tool

do I have to install pve-zsync ???
MANY THANKS for your time&energy amigos

More information about the pve-user mailing list