[PVE-User] Strange ZFS Performance

[Lindsay Mathieson]

Seeing a lot of this in the syslog

Aug 17 12:05:28 vna kernel: [252707.346162] INFO: task zvol:1244 blocked
for more than 120 seconds.
Aug 17 12:05:28 vna kernel: [252707.346191] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 17 12:05:28 vna kernel: [252707.346215] zvol            D
ffff88101e6d3640     0  1244      2 0x00000000
Aug 17 12:05:28 vna kernel: [252707.346219]  ffff88080ec6fb98
0000000000000046 ffff88080f3244a0 ffff88080ec6ffd8
Aug 17 12:05:28 vna kernel: [252707.346223]  ffff88080ec6ffd8
ffff88080ec6ffd8 ffff8810132bd010 ffff88080f3244a0
Aug 17 12:05:28 vna kernel: [252707.346226]  0000000000000001
ffff88100ef20b68 ffff88100ef20a20 ffff88100ef20b70
Aug 17 12:05:28 vna kernel: [252707.346230] Call Trace:
Aug 17 12:05:28 vna kernel: [252707.346237]  [<ffffffff8163cc79>]
schedule+0x29/0x70
Aug 17 12:05:28 vna kernel: [252707.346261]  [<ffffffffa0126965>]
cv_wait_common+0x115/0x130 [spl]
Aug 17 12:05:28 vna kernel: [252707.346264]  [<ffffffff81095240>] ?
wake_up_bit+0x40/0x40
Aug 17 12:05:28 vna kernel: [252707.346271]  [<ffffffffa01269d5>]
__cv_wait+0x15/0x20 [spl]
Aug 17 12:05:28 vna kernel: [252707.346310]  [<ffffffffa0233c73>]
txg_wait_open+0x83/0xd0 [zfs]
Aug 17 12:05:28 vna kernel: [252707.346329]  [<ffffffffa01ef1f0>]
dmu_tx_wait+0x380/0x390 [zfs]
Aug 17 12:05:28 vna kernel: [252707.346346]  [<ffffffffa01ef29a>]
dmu_tx_assign+0x9a/0x520 [zfs]
Aug 17 12:05:28 vna kernel: [252707.346372]  [<ffffffffa0287df5>]
zvol_write+0xf5/0x450 [zfs]
Aug 17 12:05:28 vna kernel: [252707.346378]  [<ffffffffa0122fc6>]
taskq_thread+0x1f6/0x440 [spl]
Aug 17 12:05:28 vna kernel: [252707.346382]  [<ffffffff810a61b0>] ?
try_to_wake_up+0x290/0x290
Aug 17 12:05:28 vna kernel: [252707.346388]  [<ffffffffa0122dd0>] ?
taskq_dispatch+0x170/0x170 [spl]
Aug 17 12:05:28 vna kernel: [252707.346391]  [<ffffffff810948d0>]
kthread+0xc0/0xd0
Aug 17 12:05:28 vna kernel: [252707.346393]  [<ffffffff81094810>] ?
kthread_create_on_node+0x120/0x120
Aug 17 12:05:28 vna kernel: [252707.346396]  [<ffffffff81646518>]
ret_from_fork+0x58/0x90
Aug 17 12:05:28 vna kernel: [252707.346398]  [<ffffffff81094810>] ?
kthread_create_on_node+0x120/0x120
Aug 17 12:05:28 vna kernel: [252707.346465] INFO: task kvm:226112 blocked
for more than 120 seconds.
Aug 17 12:05:28 vna kernel: [252707.346485] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 17 12:05:28 vna kernel: [252707.346509] kvm             D
ffff88101e673640     0 226112      1 0x00000000
Aug 17 12:05:28 vna kernel: [252707.346511]  ffff880b94a13d48
0000000000000086 ffff88047479c4a0 ffff880b94a13fd8
Aug 17 12:05:28 vna kernel: [252707.346515]  ffff880b94a13fd8
ffff880b94a13fd8 ffff8810132badc0 ffff88047479c4a0
Aug 17 12:05:28 vna kernel: [252707.346518]  ffff880b94a13d48
ffff88047479c4a0 ffff88101e673f08 7fffffffffffffff

kernel is 3.10.0-11-pve

Perhaps I should take it to the zfsinlinux list?

On 17 August 2015 at 14:14, Lindsay Mathieson <lindsay.mathieson at gmail.com>
wrote:

>
> On 17 August 2015 at 14:00, Dietmar Maurer <dietmar at proxmox.com> wrote:
>
>> Please can you post exact steps to reproduce the behavior?
>
>
> Trouble is, I'm not sure what causes it to happen. This time round it
> seemed to coincide with cloning a VM to the data pool, so maybe it was the
> stress?
>
> smartctl doesn't reveal any problems in the disks.
>
> Any suggestions for logging I should check?
>
>
> --
> Lindsay
>



-- 
Lindsay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.proxmox.com/pipermail/pve-user/attachments/20150817/6c757da9/attachment.htm>