[pve-devel] qemu 4.4 : rbd : snapshot memory, savevm is hanging

[Alexandre DERUMIER]

Something is really wrong,

because guest is also freezing, with a simple snapshot, with cache=none

monitor : snapshot_blkdev_internal drive-virtio0 snap1

or 

rbd command : rbd --image myrbdvolume snap create --snap snap1 

----- Mail original -----
De: "aderumier" <aderumier at odiso.com>
À: "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Lundi 9 Novembre 2015 04:55:16
Objet: Re: [pve-devel] qemu 4.4 : rbd : snapshot memory, savevm is hanging

This occur only with cache=none | rbd_cache=false. 


with cache=writeback, I don't have the hang error. 

----- Mail original ----- 
De: "aderumier" <aderumier at odiso.com> 
À: "pve-devel" <pve-devel at pve.proxmox.com> 
Envoyé: Lundi 9 Novembre 2015 04:20:35 
Objet: Re: [pve-devel] qemu 4.4 : rbd : snapshot memory, savevm is hanging 

mmm, I can reproduce it too with manual snapshot with rbd 


#rbd --image myrbdvolume snap create --snap snap1 

monitor: 

#stop 



----- Mail original ----- 
De: "aderumier" <aderumier at odiso.com> 
À: "pve-devel" <pve-devel at pve.proxmox.com> 
Envoyé: Lundi 9 Novembre 2015 04:05:14 
Objet: Re: [pve-devel] qemu 4.4 : rbd : snapshot memory, savevm is hanging 

Seem to be a bug in rbd block driver snapshot implementation. 

use monitor, I can reproduce the same bug with 

# snapshot_blkdev_internal drive-virtio0 snap1 
# stop 

-> hang 

----- Mail original ----- 
De: "aderumier" <aderumier at odiso.com> 
À: "pve-devel" <pve-devel at pve.proxmox.com> 
Envoyé: Lundi 9 Novembre 2015 03:55:15 
Objet: Re: [pve-devel] qemu 4.4 : rbd : snapshot memory, savevm is hanging 

>>Another way to have a hang. 
>> 
>>Do a snapshot without memory state, 
>> 
>>then in monitor, send "stop" 
>> 
>>I'll hang forever, monitor is not working anymore. 


If I comment disk snapshot, and only do vmstate save 

#qemu_volume_snapshot($vmid, $device, $storecfg, $volid, $snapname); 

It's working fine, no more hang. 


I see that rbd snapshot are done with qemu api now, (It was not the case previously), 
maybe they are a bug somewhere in rbd block driver. 

Or it's a bug in our custom snapshot-drive qmp. (I think we could use blockdev-snapshot-internal-sync from qemu upstream) 


I'll try rbd snapshot using pve-storage api to compare. 



----- Mail original ----- 
De: "aderumier" <aderumier at odiso.com> 
À: "pve-devel" <pve-devel at pve.proxmox.com> 
Envoyé: Lundi 9 Novembre 2015 03:39:27 
Objet: Re: [pve-devel] qemu 4.4 : rbd : snapshot memory, savevm is hanging 

Another way to have a hang. 

Do a snapshot without memory state, 

then in monitor, send "stop" 

I'll hang forever, monitor is not working anymore. 





----- Mail original ----- 
De: "aderumier" <aderumier at odiso.com> 
À: "dietmar" <dietmar at proxmox.com> 
Cc: "pve-devel" <pve-devel at pve.proxmox.com> 
Envoyé: Dimanche 8 Novembre 2015 14:55:17 
Objet: Re: [pve-devel] qemu 4.4 : rbd : snapshot memory, savevm is hanging 

I think this should work: 

QDict *options = NULL; 
options = qdict_new(); 
qdict_put(options, "driver", qstring_from_str('raw')); 
ret = bdrv_open(&snap_state.bs, statefile, NULL, options, bdrv_oflags, drv, ...) 




About rbd, something strange, 

when I try a first state snapshot it's always working, 
but try to launch a second always hang (on vm_stop). 

inside vm_stop, they are some 
bdrv_drain_all(); 
ret = bdrv_flush_all(); 


I just wonder if it couldn't be an old bs pointer not correctly closed. 



----- Mail original ----- 
De: "aderumier" <aderumier at odiso.com> 
À: "dietmar" <dietmar at proxmox.com> 
Cc: "pve-devel" <pve-devel at pve.proxmox.com> 
Envoyé: Dimanche 8 Novembre 2015 14:52:40 
Objet: Re: [pve-devel] qemu 4.4 : rbd : snapshot memory, savevm is hanging 

>>I thought we always use 'raw' format for statefile? 

Oh, yes, you are right. 


----- Mail original ----- 
De: "dietmar" <dietmar at proxmox.com> 
À: "aderumier" <aderumier at odiso.com>, "pve-devel" <pve-devel at pve.proxmox.com> 
Envoyé: Dimanche 8 Novembre 2015 14:18:57 
Objet: Re: [pve-devel] qemu 4.4 : rbd : snapshot memory, savevm is hanging 

> On November 8, 2015 at 1:03 PM Alexandre DERUMIER <aderumier at odiso.com> wrote: 
> 
> 
> >> it's since qemu 2.3, we need to specify format=raw, we are already doing it 
> >> 
> >> for -drive , I'm not sure with vmstate open. 
> >> 
> 
> >>>Oh, I though the PVE::Storage already fixed that ... 
> >Currently we use PVE::QemuServer::qemu_img_format, to generate the format=raw 
> >string in 
> >differente place (-drive , mirror,...) 
> 
> I think we should pass format in a new qmp option, 
> and do like qemu-img 
> 
> 
> QDict *options = NULL; 
> 
> if (fmt) { 
> options = qdict_new(); 
> qdict_put(options, "driver", qstring_from_str(fmt)); 
> } 
> 
> ret = bdrv_open(&snap_state.bs, statefile, NULL, options, bdrv_oflags, drv, 
> &local_err); 

I thought we always use 'raw' format for statefile? 
_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 

_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 
_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel 

_______________________________________________ 
pve-devel mailing list 
pve-devel at pve.proxmox.com 
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel