[PVE-User] PVE 4.2 multipathing over iscsi problem

[Sebastian Gruner]

Hi,

I've a 3 node-cluster pve-version 4.2 with iscsi-storage (HP MSA2040) an 10gbit backend.
My Problem is, that I've got multiple multipath errors if I restore a backup to the SAN.

######################################################

Errors from syslog:
Jun 21 09:32:46 proxmox-ew-edv-03 qmrestore[18729]: command 'lzop -d -c /mnt/pve/backup/dump/vzdump-qemu-117-samba-test-2016_02_27-15_30_12.vma.lzo|vma extract -v -r /var/tmp/vzdumptmp18729.fifo - /var/tmp/vzdumptmp18729' failed: interrupted by signal
Jun 21 09:32:46 proxmox-ew-edv-03 qmrestore[18728]: <root at pam> end task UPID:proxmox-ew-edv-03:00004929:00560AAB:5768ED96:qmrestore:102:root at pam: command 'lzop -d -c /mnt/pve/backup/dump/vzdump-qemu-117-samba-test-2016_02_27-15_30_12.vma.lzo|vma extract -v -r /var/tmp/vzdumptmp18729.fifo - /var/tmp/vzdumptmp18729' failed: interrupted by signal
Jun 21 09:38:36 proxmox-ew-edv-03 pmxcfs[2008]: [status] notice: received log
Jun 21 09:49:27 proxmox-ew-edv-03 qmrestore[22023]: <root at pam> starting task UPID:proxmox-ew-edv-03:00005609:005794BE:5768F187:qmrestore:102:root at pam:
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.289216] scsi_io_completion: 255 callbacks suppressed
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.289237] sd 2:0:0:0: [sdd] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.289241] sd 2:0:0:0: [sdd] tag#0 Sense Key : Illegal Request [current] 
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.289245] sd 2:0:0:0: [sdd] tag#0 Add. Sense: Invalid field in cdb
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.289249] sd 2:0:0:0: [sdd] tag#0 CDB: Write(16) 8a 00 00 00 00 00 00 00 08 00 00 00 40 00 00 00
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.289251] blk_update_request: 520 callbacks suppressed
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.289253] blk_update_request: critical target error, dev sdd, sector 2048
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.290935] blk_update_request: critical target error, dev dm-7, sector 2048
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.292264] buffer_io_error: 541862 callbacks suppressed
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.292267] Buffer I/O error on dev dm-8, logical block 0, lost async page write
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.293675] Buffer I/O error on dev dm-8, logical block 1, lost async page write
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.295093] Buffer I/O error on dev dm-8, logical block 2, lost async page write
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.296496] Buffer I/O error on dev dm-8, logical block 3, lost async page write
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.297900] Buffer I/O error on dev dm-8, logical block 4, lost async page write
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.299309] Buffer I/O error on dev dm-8, logical block 5, lost async page write
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.300620] Buffer I/O error on dev dm-8, logical block 6, lost async page write
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.301926] Buffer I/O error on dev dm-8, logical block 7, lost async page write
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.303227] Buffer I/O error on dev dm-8, logical block 8, lost async page write
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.304520] Buffer I/O error on dev dm-8, logical block 9, lost async page write
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.307884] sd 1:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.307888] sd 1:0:0:0: [sdc] tag#0 Sense Key : Illegal Request [current] 
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.307890] sd 1:0:0:0: [sdc] tag#0 Add. Sense: Invalid field in cdb
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.307894] sd 1:0:0:0: [sdc] tag#0 CDB: Write(16) 8a 00 00 00 00 00 00 00 48 00 00 00 40 00 00 00
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.307896] blk_update_request: critical target error, dev sdc, sector 18432
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.309571] sd 1:0:0:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.309573] sd 1:0:0:0: [sdc] tag#1 Sense Key : Illegal Request [current] 
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.309576] sd 1:0:0:0: [sdc] tag#1 Add. Sense: Invalid field in cdb
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.309578] sd 1:0:0:0: [sdc] tag#1 CDB: Write(16) 8a 00 00 00 00 00 00 00 c8 00 00 00 40 00 00 00
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.309580] blk_update_request: critical target error, dev sdc, sector 51200
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.311267] sd 2:0:0:0: [sdd] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.311270] sd 2:0:0:0: [sdd] tag#1 Sense Key : Illegal Request [current] 
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.311273] sd 2:0:0:0: [sdd] tag#1 Add. Sense: Invalid field in cdb
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.311275] sd 2:0:0:0: [sdd] tag#1 CDB: Write(16) 8a 00 00 00 00 00 00 00 88 00 00 00 40 00 00 00
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.311277] blk_update_request: critical target error, dev sdd, sector 34816
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.312937] blk_update_request: critical target error, dev dm-7, sector 18432
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.315953] blk_update_request: critical target error, dev dm-7, sector 51200
Jun 21 09:49:32 proxmox-ew-edv-03 kernel: [57404.320290] blk_update_request: critical target error, dev dm-7, sector 34816

#######################################################

The SAN itself works perfect, I can build VM's, migrate VM's and so on, but the restore failed all the time.
My multipathing is setup by following this guide: https://pve.proxmox.com/wiki/ISCSI_Multipath
The storage is used with a LVM Volume-group, which is mounted as central storage for disk-images.
My other two Clusters (pve-version 3.4) in same configuration work perfectly, only version 4.2 produces these errors.

#######################################################

root at proxmox-ew-edv-03:~# pveversion -v
proxmox-ve: 4.2-54 (running kernel: 4.4.10-1-pve)
pve-manager: 4.2-15 (running version: 4.2-15/6669ad2c)
pve-kernel-4.4.6-1-pve: 4.4.6-48
pve-kernel-4.4.8-1-pve: 4.4.8-52
pve-kernel-4.4.10-1-pve: 4.4.10-54
lvm2: 2.02.153-1
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-42
qemu-server: 4.0-81
pve-firmware: 1.1-8
libpve-common-perl: 4.0-68
libpve-access-control: 4.0-16
libpve-storage-perl: 4.0-55
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-19
pve-container: 1.0-68
pve-firewall: 2.0-29
pve-ha-manager: 1.0-32
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u2
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve9~jessie

###################################################

root at proxmox-ew-edv-03:~# cat /etc/multipath.conf
defaults {
        polling_interval        2
        path_selector           "round-robin 0"
        path_grouping_policy    multibus
        uid_attribute           ID_SERIAL
        rr_min_io               100
        failback                immediate
        no_path_retry           queue
        user_friendly_names     yes }

blacklist {
#    wwid ".*"
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^(td|hd)[a-z]"
        devnode "^dcssblk[0-9]*"
        devnode "^cciss!c[0-9]d[0-9]*"
        device {
                vendor "DGC"
                product "LUNZ"
        }
        device {
                vendor "EMC"
                product "LUNZ"
        }
        device {
                vendor "IBM"
                product "Universal Xport"
        }
        device {
                vendor "IBM"
                product "S/390.*"
        }
        device {
                vendor "DELL"
                product "Universal Xport"
        }
        device {
                vendor "SGI"
                product "Universal Xport"
        }
        device {
                vendor "STK"
                product "Universal Xport"
        }
        device {
                vendor "SUN"
                product "Universal Xport"
        }
        device {
                vendor "(NETAPP|LSI|ENGENIO)"
                product "Universal Xport"
        } }

blacklist_exceptions {
    wwid "3600c0ff0001dfcb041cd4e5701000000"
    wwid "3600c0ff0001e038153cd4e5701000000" }

multipaths {
    multipath {
        wwid "3600c0ff0001dfcb041cd4e5701000000"
        alias vdisk0001
    }
    multipath {
        wwid "3600c0ff0001e038153cd4e5701000000"
        alias vdisk0002
    } }

##################################################

I've tried everything I know and searched for hours, but I can't find any misconfiguration or solution to my problem.
Any ideas? Thanks for help

kind regards,
s.gruner