[PVE-User] VM clone with 2 disks fails on ZFS-iSCSI IET storage
Sebastian Oswald
s.oswald at autogassner.de
Fri Mar 18 17:12:23 CET 2016
Hello Mikhail,
You should _always_ access the zvol block devices via /dev/zvol/... !
I have ZoL on several debian/devuan jessie machines for development, on
my private PCs and servers and on backup systems in production with
really good performance and reliability so far.
On my "playground" I also use zvols as FC SCSI targets for 2 hosts,
both using ZFS on top of that zvol. No Problems so far with these, even
when I purposedly tried to break it (because thats why its called a
test setup, isn't it?).
I get >800MB/s (just under 7GBit/s) at the raw MP-device on the
initiator with a 2x4GBit link and "out of the box" SCST-setup, so
nothing fine-tuned or optimized yet. The pool consists of cheap
SATA-Disks ranging from 1-4TB and 2 SATA-SSDs for caching, so nothing
fancy or fast and even different vdev-sizes mixed together (again:
testing playground!)
You might want to throw in some SSDs for L2ARC and ZIL on your storage
host and maybe some more RAM as ZFS really lives up with proper cache
sizes and will get completely unusable if its caches are flushed to
swap (as with every filesystem or LVM). Also check the (stupidly low)
default settings for max bandwith in the VM/container config - these
have bitten me several times in the past.
I haven't used Proxmox with ZFS yet, thats on the agenda for the next
weeks or so before/when installing the new (third) proxmox production
system. But as long as proxmox isn't doing anything uber-fancy or
completely different from the normal ZoL-approach, I don't expect any
gremlins to bite me from the ZFS side of the setup...
Regards,
Sebastian
PS: For general ZFS knowledge I can HIGHLY recommend "FreeBSD Mastery:
ZFS" from Michael W. Lucas. Most of it applies 1:1 to Zol and it gives
a very good insight to the inner workings and thinking of ZFS.
> My ZFS pool runs 4 x 4TB SAS drives in RAID10:
>
> # zpool status
> pool: rpool
> state: ONLINE
> scan: resilvered 1.55G in 0h0m with 0 errors on Mon Mar 7 23:29:50
> 2016 config:
>
> NAME STATE READ WRITE CKSUM
> rpool ONLINE 0 0 0
> mirror-0 ONLINE 0 0 0
> sda ONLINE 0 0 0
> sdb ONLINE 0 0 0
> mirror-1 ONLINE 0 0 0
> sdc ONLINE 0 0 0
> sdd ONLINE 0 0 0
>
> and just now I tried ZFS send/receive on the storage system to copy
> volumes. I was very much surprised that the speed is getting at
> 100MB/s max..
>
> I just guess this shows how unstable and untested ZFS on Linux is.
> This also proves Proxmox's WIKI page that does not suggest to use ZFS
> over iSCSI on Linux in production.
>
> So I think it is now about time to switch to old school LVM over iSCSI
> in my case, until I put some real data on this cluster..
>
> Mikhail.
>
> On 03/18/2016 05:59 PM, Mikhail wrote:
> > Hello,
> >
> > I'm running 3-node cluster with latest PVE 4.1-1 community edition.
> > My shared storage is ZFS over iSCSI (ZFS storage server is Linux
> > Debian Jessie with IET).
> >
> > There's a problem cloning VM that has 2 (or possibly "2 and more")
> > disks attached to VM in this setup. The problem is that one disk
> > gets copied, and then "Clone" task fails with message: TASK ERROR:
> > clone failed: File exists.
> > at /usr/share/perl5/PVE/Storage/LunCmd/Iet.pm line 376.
> >
> > There's no problems cloning same VM if it has only one disk.
> >
> > Here's steps to reproduce
> >
> > 1) create VM with 2 disks (both on shared storage)
> > 2) shutdown VM
> > 3) attempt to clone this VM to another VM
> >
> > More details in my case:
> >
> > 1) VM 8002 - source VM to clone, here's output from storage server -
> >
> > root at storage:/etc/iet# zfs list|grep 8002
> > rpool/vm-8002-disk-1 30.5G 6.32T 30.5G -
> > rpool/vm-8002-disk-2 77.8G 6.32T 77.8G -
> >
> > root at storage:/etc/iet# cat /etc/iet/ietd.conf
> > Target iqn.2016-03.eu.myorg:rpool
> > Lun 1 Path=/dev/rpool/vm-1001-disk-2,Type=blockio
> > Lun 2 Path=/dev/rpool/vm-1002-disk-1,Type=blockio
> > Lun 7 Path=/dev/rpool/vm-8003-disk-1,Type=blockio
> > Lun 5 Path=/dev/rpool/vm-101-disk-1,Type=blockio
> > Lun 3 Path=/dev/rpool/vm-8002-disk-1,Type=blockio
> > Lun 6 Path=/dev/rpool/vm-8002-disk-2,Type=blockio
> > Lun 0 Path=/dev/rpool/vm-8201-disk-1,Type=blockio
> > Lun 8 Path=/dev/rpool/vm-8301-disk-1,Type=blockio
> > Lun 9 Path=/dev/rpool/vm-8301-disk-2,Type=blockio
> > Lun 10 Path=/dev/rpool/vm-8302-disk-1,Type=blockio
> > Lun 4 Path=/dev/rpool/vm-8001-disk-1,Type=blockio
> >
> > root at storage:/etc/iet# cat /proc/net/iet/volume
> > tid:1 name:iqn.2016-03.eu.myorg:rpool
> > lun:1 state:0 iotype:blockio iomode:wt blocks:1048576000
> > blocksize:512 path:/dev/rpool/vm-1001-disk-2
> > lun:2 state:0 iotype:blockio iomode:wt blocks:67108864
> > blocksize:512 path:/dev/rpool/vm-1002-disk-1
> > lun:5 state:0 iotype:blockio iomode:wt blocks:62914560
> > blocksize:512 path:/dev/rpool/vm-101-disk-1
> > lun:3 state:0 iotype:blockio iomode:wt blocks:62914560
> > blocksize:512 path:/dev/rpool/vm-8002-disk-1
> > lun:6 state:0 iotype:blockio iomode:wt blocks:314572800
> > blocksize:512 path:/dev/rpool/vm-8002-disk-2
> > lun:0 state:0 iotype:blockio iomode:wt blocks:83886080
> > blocksize:512 path:/dev/rpool/vm-8201-disk-1
> > lun:8 state:0 iotype:blockio iomode:wt blocks:31457280
> > blocksize:512 path:/dev/rpool/vm-8301-disk-1
> > lun:9 state:0 iotype:blockio iomode:wt blocks:104857600
> > blocksize:512 path:/dev/rpool/vm-8301-disk-2
> > lun:10 state:0 iotype:blockio iomode:wt blocks:104857600
> > blocksize:512 path:/dev/rpool/vm-8302-disk-1
> > lun:12 state:0 iotype:blockio iomode:wt blocks:62914560
> > blocksize:512 path:/dev/rpool/vm-8091-disk-1
> > lun:4 state:0 iotype:blockio iomode:wt blocks:62914560
> > blocksize:512 path:/dev/rpool/vm-8001-disk-1
> >
> >
> >
> > VM config:
> >
> > root at pm1:/etc/pve/nodes/pm2/qemu-server# pwd
> > /etc/pve/nodes/pm2/qemu-server
> > root at pm1:/etc/pve/nodes/pm2/qemu-server# cat 8002.conf
> > boot: cdn
> > bootdisk: virtio0
> > cores: 1
> > ide2:
> > isoimages:iso/systemrescuecd-x86-4.7.1.iso,media=cdrom,size=469942K
> > memory: 1024 name: rep
> > net0: virtio=32:38:38:39:39:33,bridge=vmbr0,tag=80
> > numa: 0
> > ostype: l26
> > smbios1: uuid=8b0b1ab8-d3e3-48ae-8834-edd0e68a3c0c
> > sockets: 1
> > virtio0: storage-1:vm-8002-disk-1,size=30G
> > virtio1: storage-1:vm-8002-disk-2,size=150G
> >
> > Storage config:
> >
> > virtio1: storage-1:vm-8002-disk-2,size=150G
> > root at pm1:/etc/pve/nodes/pm2/qemu-server# cat /etc/pve/storage.cfg
> > dir: local
> > path /var/lib/vz
> > maxfiles 0
> > content vztmpl,rootdir,images,iso
> >
> > nfs: isoimages
> > path /mnt/pve/isoimages
> > server 192.168.4.1
> > export /rpool/shared/isoimages
> > content iso
> > options vers=3
> > maxfiles 1
> >
> > zfs: storage-1
> > pool rpool
> > blocksize 4k
> > iscsiprovider iet
> > portal 192.168.4.1
> > target iqn.2016-03.eu.myorg:rpool
> > sparse
> > nowritecache
> > content images
> >
> > 2) Attempting to clone 8002 to lets say 80123. Here's task output
> > (cut some % lines):
> >
> > create full clone of drive virtio0 (storage-1:vm-8002-disk-1)
> > transferred: 0 bytes remaining: 32212254720 bytes total: 32212254720
> > bytes progression: 0.00 %
> > qemu-img: iSCSI Failure: SENSE KEY:ILLEGAL_REQUEST(5)
> > ASCQ:INVALID_OPERATION_CODE(0x2000)
> > transferred: 322122547 bytes remaining: 31890132173 bytes total:
> > 32212254720 bytes progression: 1.00 %
> > transferred: 32212254720 bytes remaining: 0 bytes total: 32212254720
> > bytes progression: 100.00 %
> > transferred: 32212254720 bytes remaining: 0 bytes total: 32212254720
> > bytes progression: 100.00 %
> > create full clone of drive virtio1 (storage-1:vm-8002-disk-2)
> > TASK ERROR: clone failed: File exists. at
> > /usr/share/perl5/PVE/Storage/LunCmd/Iet.pm line 376.
> >
> > after that, "zfs list" on storage shows there's one volume on ZFS:
> >
> > root at storage:/etc/iet# zfs list|grep 80123
> > rpool/vm-80123-disk-2 64K 6.32T 64K -
> >
> > Obviously it was created by PVE. rpool/vm-80123-disk-2 was removed
> > automatically when task failed to complete.
> >
> > And here's what I see in /var/log/syslog on storage when this task
> > fails:
> >
> > Mar 18 17:40:19 storage kernel: [840181.448223] zd240: unknown
> > partition table
> > Mar 18 17:40:19 storage ietd: unable to create logical unit 12 in
> > target 1: 17
> > Mar 18 17:40:21 storage kernel: [840182.870132] iscsi_trgt:
> > iscsi_volume_del(319) 1 b
> >
> >
> > So I guess this has something to do with IET.
> >
> > regards,
> > Mikhail.
> > _______________________________________________
> > pve-user mailing list
> > pve-user at pve.proxmox.com
> > http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
>
> _______________________________________________
> pve-user mailing list
> pve-user at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 213 bytes
Desc: Digitale Signatur von OpenPGP
URL: <http://lists.proxmox.com/pipermail/pve-user/attachments/20160318/1bc308fb/attachment.sig>
More information about the pve-user
mailing list