[PVE-User] proxmox 5 - replication fails

[dorsy]

Same error again, nothing strange until 3:20 AM:

Jul 10 03:00:02 xxnodenamexx pvesr[32023]: send from 
@__replicate_103-0_1499647500__ to 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499648400__ estimated size 
is 7.27M
Jul 10 03:00:02 xxnodenamexx pvesr[32023]: total estimated size is 7.27M
Jul 10 03:00:02 xxnodenamexx pvesr[32023]: TIME        SENT SNAPSHOT
Jul 10 03:00:02 xxnodenamexx pvesr[32023]: 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499647500__ name 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499647500__        -
Jul 10 03:00:03 xxnodenamexx pvesr[32023]: 03:00:03   3.10M 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499648400__
Jul 10 03:00:06 xxnodenamexx pvesr[32023]: send from 
@__replicate_109-0_1499647505__ to 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499648405__ estimated size 
is 24.0M
Jul 10 03:00:06 xxnodenamexx pvesr[32023]: total estimated size is 24.0M
Jul 10 03:00:06 xxnodenamexx pvesr[32023]: TIME        SENT SNAPSHOT
Jul 10 03:00:06 xxnodenamexx pvesr[32023]: 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499647505__ name 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499647505__        -
Jul 10 03:00:07 xxnodenamexx pvesr[32023]: 03:00:07   17.9M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499648405__
Jul 10 03:15:02 xxnodenamexx pvesr[2801]: send from 
@__replicate_103-0_1499648400__ to 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499649300__ estimated size 
is 63.9M
Jul 10 03:15:02 xxnodenamexx pvesr[2801]: total estimated size is 63.9M
Jul 10 03:15:02 xxnodenamexx pvesr[2801]: TIME        SENT SNAPSHOT
Jul 10 03:15:03 xxnodenamexx pvesr[2801]: 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499648400__ name 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499648400__        -
Jul 10 03:15:03 xxnodenamexx pvesr[2801]: 03:15:03   6.00M 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499649300__
Jul 10 03:15:04 xxnodenamexx pvesr[2801]: 03:15:04   13.5M 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499649300__
Jul 10 03:15:05 xxnodenamexx pvesr[2801]: 03:15:05   13.9M 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499649300__
Jul 10 03:15:06 xxnodenamexx pvesr[2801]: 03:15:06   17.2M 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499649300__
Jul 10 03:15:07 xxnodenamexx pvesr[2801]: 03:15:07   49.7M 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499649300__
Jul 10 03:15:24 xxnodenamexx pvesr[2801]: send from 
@__replicate_109-0_1499648405__ to 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__ estimated size 
is 169M
Jul 10 03:15:24 xxnodenamexx pvesr[2801]: total estimated size is 169M
Jul 10 03:15:24 xxnodenamexx pvesr[2801]: 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499648405__ name 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499648405__        -
Jul 10 03:15:24 xxnodenamexx pvesr[2801]: TIME        SENT SNAPSHOT
Jul 10 03:15:25 xxnodenamexx pvesr[2801]: 03:15:25   18.6M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:26 xxnodenamexx pvesr[2801]: 03:15:26   49.3M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:27 xxnodenamexx pvesr[2801]: 03:15:27   69.9M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:28 xxnodenamexx pvesr[2801]: 03:15:28   74.0M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:29 xxnodenamexx pvesr[2801]: 03:15:29   80.3M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:30 xxnodenamexx pvesr[2801]: 03:15:30   85.5M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:31 xxnodenamexx pvesr[2801]: 03:15:31   92.9M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:32 xxnodenamexx pvesr[2801]: 03:15:32    101M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:33 xxnodenamexx pvesr[2801]: 03:15:33    102M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:34 xxnodenamexx pvesr[2801]: 03:15:34    126M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:35 xxnodenamexx pvesr[2801]: 03:15:35    141M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:36 xxnodenamexx pvesr[2801]: 03:15:36    141M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:37 xxnodenamexx pvesr[2801]: 03:15:37    142M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:38 xxnodenamexx pvesr[2801]: 03:15:38    142M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:39 xxnodenamexx pvesr[2801]: 03:15:39    151M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:40 xxnodenamexx pvesr[2801]: 03:15:40    167M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__
Jul 10 03:15:41 xxnodenamexx pvesr[2801]: 03:15:41    169M 
rpool/data/vm-109-disk-1 at __replicate_109-0_1499649323__


And then it wants to send from "@" isntead of "@__replicate_103-0_[time]__
Also strange that the schedule is */15, but it is starting at 3:20 after 
3:15


Jul 10 03:20:05 xxnodenamexx pvesr[4021]: send from @ to 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499649600__ estimated size 
is 19.5G
Jul 10 03:20:05 xxnodenamexx pvesr[4021]: total estimated size is 19.5G
Jul 10 03:20:05 xxnodenamexx pvesr[4021]: TIME        SENT SNAPSHOT
Jul 10 03:20:05 xxnodenamexx pvesr[4021]: 
rpool/data/vm-103-disk-1        name rpool/data/vm-103-disk-1        -
Jul 10 03:20:05 xxnodenamexx pvesr[4021]: volume 
'rpool/data/vm-103-disk-1' already exists
Jul 10 03:20:05 xxnodenamexx pvesr[4021]: warning: cannot send 
'rpool/data/vm-103-disk-1 at __replicate_103-0_1499649600__': Broken pipe
Jul 10 03:20:06 xxnodenamexx pvesr[4021]: cannot send 
'rpool/data/vm-103-disk-1': I/O error
Jul 10 03:20:06 xxnodenamexx pvesr[4021]: command 'zfs send -Rpv -- 
rpool/data/vm-103-disk-1 at __replicate_103-0_1499649600__' failed: exit code 1
Jul 10 03:20:06 xxnodenamexx pvesr[4021]: send/receive failed, cleaning 
up snapshot(s)..

Hope it helps.