[pve-devel] Speed up PVE Backup

[Martin Waschbüsch]

Hi Stefan,

> Am 16.02.2016 um 09:22 schrieb Stefan Priebe - Profihost AG <s.priebe at profihost.ag>:
> 
> Hi,
> 
> is there any way to speed up PVE Backups?
> 
> I'm trying to evaluate the optimal method doing backups but they took
> very long.
> 
> I'm trying to use vzdump on top of nfs on top of btrfs using zlib
> compression.
> 
> The target FS it totally idle but the backup is running at a very low speed.
> 
> The output after 15 minutes is:
> INFO: starting new backup job: vzdump 132 --remove 0 --mode snapshot
> --storage vmbackup --node 1234
> INFO: Starting Backup of VM 132 (qemu)
> INFO: status = running
> INFO: update VM 132: -lock backup
> INFO: backup mode: snapshot
> INFO: ionice priority: 7
> INFO: snapshots found (not included into backup)
> INFO: creating archive
> '/mnt/pve/vmbackup/dump/vzdump-qemu-132-2016_02_16-09_05_28.vma'
> INFO: started backup task 'e75cc760-2be6-4731-b65b-f78b2832baf9'
> INFO: status: 0% (79036416/1073741824000), sparse 0% (10665984),
> duration 3, 26/22 MB/s
> INFO: status: 1% (10742530048/1073741824000), sparse 0% (6539657216),
> duration 411, 26/10 MB/s
> INFO: status: 2% (21485322240/1073741824000), sparse 1% (17281466368),
> duration 820, 26/0 MB/s
> 
> This is PVE 3.4 running Qemu 2.4

To me this looks like the compression is the limiting factor? What speed do you get for this NFS mount when just copying an existing file?

Anyway, first of all, if network bandwidth and backup disk space are not limiting factors, lzo compression is *way* faster than gzip.

Having said that, there is a way to speed up gzip compression: I am not sure if that option already exists in PVE 3.4, but new versions of vzdump recognize an option in /etc/vzdump.conf:
pigz: n
where n is the number of threads that pigz (parallel implementation of gzip) should use.
Obviously, you also need to have pigz installed (apt-get install pigz).
This can drastically speed up the backups, but for the cost of much higher load during the backup.

The other thing I did to minimize backup times is ensure that unused diskspace inside KVM guests is zeroed so that there are more sparse blocks. Helps a lot!
Personally, I do this by having images on a ceph cluster which supports using discard (trim), so VMs take care of this themselves.
But there are other options like zerofree, etc.

Just my five cents...

Martin