[PVE-User] [pve-devel] backup : add pigz compressor

[Chidester, Bryce]

On Fri, 2015-07-10 at 07:05 +0200, Dietmar Maurer wrote:
> > user from pve-user mailing report than using pigz (multithread 
> > gzip),
> > improve a lot speed of compression backup.
> > 
> > Maybe could we use add it ? (or maybe replace gzip ?)
> 
> We already have lzop, which is as fast as pgzip and uses less 
> resources.
> So what is the advantage of pgzip? IMHO a parallel lzop would be 
> nice.
> 

I ran a quick, informal "benchmark" to compare lzo, gzip, pigz, and
uncompressed backups of a modest 8GB VM on a host with 8 cores, all
currently at 100% from guest usage. 

None: 50 seconds, 199MB/s, output 963MB
pigz: 20 seconds, 453MB/s, output 266MB
gzip: 57 seconds, 150MB/s, output 254MB
lzop:   9 seconds, 954MB/s, output 362MB

These results demonstrate a clear trade-off between speed and
compression with gzip and lzo. However, pigz offers gzip's compression
ratio at only a modest bump in the time it took to compress it. 

I'm not saying pigz is for everybody or all scenarios or deployments,
but it has worked well for mine. I backup 400-700GB daily on any given
host, and pigz allows me to run those backups relatively quickly while
still getting making the most of the disk space. And impact on the
guests is minimal based on the host graphs, guest reporting, and a lack
of user complaints.


PS: A VM with 88GB of disk, lzop compresses to 16.5GB in 19 minutes 35
seconds (80MB/s). pigz compressed it to 14.96GB in 19 minutes 47
seconds (80MB/s).

PPS: A parallel/multi-threaded lzop would be nice, yes.

-- 

Bryce Chidester
Director of Systems Engineering
Calyptix Security | Simply Powerful Network Security.

www.calyptix.com