[pve-devel] Speed up PVE Backup

[Eneko Lacunza]

Hi all,

El 16/02/16 a las 15:52, Stefan Priebe - Profihost AG escribió:
> Am 16.02.2016 um 15:50 schrieb Dmitry Petuhov:
>> 16.02.2016 13:20, Dietmar Maurer wrote:
>>>> Storage Backend is ceph using 2x 10Gbit/s and i'm able to read from it
>>>> with 500-1500MB/s. See below for an example.
>>> The backup process reads 64KB blocks, and it seems this slows down ceph.
>>> This is a known behavior, but I found no solution to speed it up.
>> Just done script to speedup my backups from ceph. It's simply does
>> (actually little more):
>> rbd snap create $SNAP
>> rbd export $SNAP $DUMPDIR/$POOL-$VOLUME-$DATE.raw
>> rbd snap rm $SNAP
>> for every image in selected pools.
>>
>> When exporting to file, it's faster than my temporary HDD can write
>> (about 120MB/s). But exporting to STDOUT ('-' instead of filename, with
>> compression or without it) noticeably decreases speed to qemu's levels
>> (20-30MB/s). That's little strange.
>>
>> This method is incompatible with PVE's backup-restore tools, but good
>> enough for manual disaster recovery from CLI.
> right - that'S working for me too but just at night and not when a
> single user wants RIGHT now a backup incl. config.
Do we have any improvement related to this in the pipeline? Yesterday 
our 9-osd 3-node cluster restored a backup at 6MB/s... it was very 
boring, painfull and expensive to wait for it :) (I decided to buy a new 
server to replace our 7.5-year IBM while waiting ;) )

Our backups are slow too, but we do those during weekend... but usually 
we want to restore fast... :)

Dietmar, I haven't  looked at the backup/restore code, but do you think 
we could do something to read/write to storage in larger chunks than the 
actual 64KB? I'm out of a high work load period and maybe could look at 
this issue this summer.

Thanks
Eneko

-- 
Zuzendari Teknikoa / Director Técnico
Binovo IT Human Project, S.L.
Telf. 943493611
       943324914
Astigarraga bidea 2, planta 6 dcha., ofi. 3-2; 20180 Oiartzun (Gipuzkoa)
www.binovo.es