[PVE-User] Problem with QEMU drive-mirror after cancelling VM disk move

[Fabian Grünbichler]

On March 31, 2020 12:21 pm, Mikhail wrote:
> Hello,
> 
> On one of our clusters we're seeing issues with VM backup task - the
> backup task fails with the following:
> 
> ERROR: Node 'drive-scsi0' is busy: block device is in use by block job:
> mirror
> INFO: aborting backup job
> ERROR: Backup of VM 123 failed - Node 'drive-scsi0' is busy: block
> device is in use by block job: mirror
> 
> I tried digging some information and it appears to be that there's
> "drive-mirror" job in QEMU/KVM for this particular VM is blocking backup
> process. It is clear to me that this problem started couple weeks ago
> when we attempted to change VM's disk underlying storage, however the
> "Move disk" operation was cancelled manually by administrator at the
> time and backup tasks started failing right after that. I'm not sure
> whether this is Proxmox issue or QEMU/KVM, but I suppose that stopping
> and starting VM from within Proxmox will remove this block, however in
> our case keeping this virtual machine up and running is critical and we
> should avoid even 1-2 min downtime.

yes, shutting the VM down and starting it again gets rid of any leftover 
block-jobs for sure.

> The question is how to remove this drive-mirror block online and how to
> avoid this in the future.

you should be able to manually clean the messup using the QMP/monitor 
interface:

`man qemu-qmp-ref` gives a detailed tour, you probably want
`query-block-jobs` and `query-block`, and then, depending on the output
`block-job-cancel` or `block-job-complete`.

the HMP interface accessible via 'qm monitor <VMID>' has slightly 
different commands: `info block -v`, `info block-jobs` and 
`block_job_cancel`/`block_job_complete` ('_' instead of '-').

feel free to post the output of the query/info commands before deciding 
how to proceed. the complete task log of the failed 'move disk' 
operation would also be interesting, if it is still available.