[pve-devel] Speed up PVE Backup

Fri Mar 4 11:09:36 CET 2016

Hi Alexandre,

Am 02.03.2016 um 15:23 schrieb Alexandre DERUMIER:
>>>We're currently implementing our own backup system which uses the
>>>drive-backup qmp routine to dump qcow2. We don't need the config files.
>>>And this works with ceph and 4MB cluster size.
> 
> How about speed performance (64K,4MB) and stability   vs proxmox vma
> format ?

We (Stefan and I) do not have benchmarks, that are fully comparable.

We tested a 100GB virtio drive from ceph backup upped to NFS:

- with VMA, default 64kb cluster size and without read/write limits:

|      | Snapshot | Suspend |  Stop |
|------+----------+---------+-------|
| none |    09:55 |   09:37 | 09:35 |
| LZO  |    10:24 |   10:26 | 10:20 |
| GZip |    31:52 |   32:22 |       |

- QMP drive-backup (params sync => full, format => qcow2), 4MB cluster
size and 130 mb/s, 215 ops/s read; 90 mb/s, 155 ops/s write limits:

9:15 Minutes.

QMP drive-backup with above limits and default cluster size 64KB took
very long, more than 40 minutes...

> I'm curious, because I still fear to use proxmox feature when I see all
> the stability bugs reports on the forum.
> And I would like to test incremental backup too. (only full backup is
> not possible for me)

We don't have much experience with stability for plain QMP drive-backup
yet, sorry.

Greets,
Timo

> 
> 
> 
> 
> 	
> 
> *Alexandre* *Derumier* 
> *Ingénieur système et stockage*
> 
> *Fixe :* 03 20 68 90 88 
> *Fax :* 03 20 68 90 81
> 
> 45 Bvd du Général Leclerc 59100 Roubaix 
> 12 rue Marivaux 75002 Paris
> 
> <https://twitter.com/OdisoHosting> <https://twitter.com/mindbaz> <https://www.linkedin.com/company/odiso> <http://www.viadeo.com/fr/company/odiso> <https://www.facebook.com/monsiteestlent>
> 
> MonSiteEstLent.com <http://www.monsiteestlent.com/> - Blog dédié à la
> webperformance et la gestion de pics de trafic
> 
> 
> ------------------------------------------------------------------------
> *De: *"Stefan Priebe" <s.priebe at profihost.ag>
> *À: *"pve-devel" <pve-devel at pve.proxmox.com>
> *Envoyé: *Mercredi 2 Mars 2016 08:28:52
> *Objet: *Re: [pve-devel] Speed up PVE Backup
> 
> Am 01.03.2016 um 14:05 schrieb Alexandre DERUMIER:
>>>> Yes it does. We already tested it - it just does not work with VMA. We
>>>> need to dump qcow2 files cause vma is crashing when the clustersize it
>>>> get's is not 64kb.
>>
>> How do you backup to qcow2 ?
> 
> We're currently implementing our own backup system which uses the
> drive-backup qmp routine to dump qcow2. We don't need the config files.
> And this works with ceph and 4MB cluster size.
> 
> Greets,
> Stefan
> 
>> ----- Mail original -----
>> De: "Stefan Priebe" <s.priebe at profihost.ag>
>> À: "pve-devel" <pve-devel at pve.proxmox.com>, "dietmar"
> <dietmar at proxmox.com>
>> Envoyé: Mardi 1 Mars 2016 11:55:21
>> Objet: Re: [pve-devel] Speed up PVE Backup
>>
>> )
>>
>> Am 01.03.2016 um 11:03 schrieb Alexandre DERUMIER:
>>> Hi, qemu devs have send patches to configure backup cluster size:
>>>
>>> http://lists.gnu.org/archive/html/qemu-devel/2016-02/msg06062.html
>>> http://lists.gnu.org/archive/html/qemu-devel/2016-02/msg06064.html
>>>
>>>
>>> Could be great to see if it's improve ceph backup performance.
>>
>> Yes it does. We already tested it - it just does not work with VMA. We
>> need to dump qcow2 files cause vma is crashing when the clustersize it
>> get's is not 64kb.
>>
>>>
>>>
>>> ----- Mail original -----
>>> De: "dietmar" <dietmar at proxmox.com>
>>> À: "aderumier" <aderumier at odiso.com>, "pve-devel"
> <pve-devel at pve.proxmox.com>
>>> Envoyé: Vendredi 19 Février 2016 09:17:14
>>> Objet: Re: [pve-devel] Speed up PVE Backup
>>>
>>>> I wonder how perform the native qemu backup blockjob vs proxmox vma
> backup
>>>> format ?
>>>
>>> We use the qemu backup blockjob, just slightly modified...
>>> _______________________________________________
>>> pve-devel mailing list
>>> pve-devel at pve.proxmox.com
>>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>>>
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>> _______________________________________________
>> pve-devel mailing list
>> pve-devel at pve.proxmox.com
>> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
>>
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 
> 
> _______________________________________________
> pve-devel mailing list
> pve-devel at pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
> 

-- 
Mit freundlichen Grüßen
  Timo Grodzinski
Ihr Profihost Team

-------------------------------
Profihost AG
Expo Plaza 1
30539 Hannover
Deutschland

Tel.: +49 (511) 5151 8181     | Fax.: +49 (511) 5151 8282
URL: http://www.profihost.com | E-Mail: info at profihost.com

Sitz der Gesellschaft: Hannover, USt-IdNr. DE813460827
Registergericht: Amtsgericht Hannover, Register-Nr.: HRB 202350
Vorstand: Cristoph Bluhm, Sebastian Bluhm, Stefan Priebe
Aufsichtsrat: Prof. Dr. iur. Winfried Huck (Vorsitzender)