[pve-devel] [PVE-User] backup : add pigz compressor

[Alexandre DERUMIER]

>>Aren't there solutions to this already? INotify based sync daemons (or
>>even just a cronjob)?

yes, it could be done like this, but it more difficult if you have a proxmox backups running,
and in paralell rsync running. (if rsync are not fast enough, temp storage could be full).
I would like something like:

proxmox backup job1: backup temp storage , rsync , 
  then 
proxmox backup job2: backup temp storage , rsync .
 ....
sequentially


also,for restore, It's more easy if the backup storage is the one defined in backup job.





Thinking about that, 
the main problem currently, is for writes coming from qemu guest,
on block which are not yet backuped.
In this case, old block need to be write to backup, before being replace. (Correct me if I'm wrong)

This is where latencies are important, because if backup storage is satured, it can freeze ios in the vm.

I don't have digged too much in backup patches, but maybe for theses block, is it possible to write them to
a temp storage (local ssd for example), and merge them to vma archive at the end of the backup ?








----- Mail original -----
De: "Wolfgang Bumiller" <w.bumiller at proxmox.com>
À: "aderumier" <aderumier at odiso.com>
Cc: "dietmar" <dietmar at proxmox.com>, "proxmoxve" <pve-user at pve.proxmox.com>, "pve-devel" <pve-devel at pve.proxmox.com>
Envoyé: Vendredi 10 Juillet 2015 09:33:42
Objet: Re: [pve-devel] [PVE-User]  backup : add pigz compressor

> I would like to add an option, to backup to a temp storage (local ssd for example), then send the backup async (rsync for example), 
> to the destination storage. (big slow storage) 

Aren't there solutions to this already? INotify based sync daemons (or 
even just a cronjob)? 
Although if this was implemented in PVE there could be a status 
message and a progress bar somewhere in the UI I suppose. 
Maybe there's an acceptable already-packaged solution which has a nice 
interface to get the status and/or start/pause the syncing. 

On Fri, Jul 10, 2015 at 07:56:11AM +0200, Alexandre DERUMIER wrote: 
> About backup, 
> 
> I would like to add a new feature : 
> 
> As I m always afraid of problem which can occur with network backup through nfs (network saturated, latencies,...), 
> 
> I would like to add an option, to backup to a temp storage (local ssd for example), then send the backup async (rsync for example), 
> to the destination storage. (big slow storage) 
> 
> What do you think about it ? 
> 
> 
> 
> ----- Mail original ----- 
> De: "aderumier" <aderumier at odiso.com> 
> À: "dietmar" <dietmar at proxmox.com> 
> Cc: "proxmoxve" <pve-user at pve.proxmox.com>, "pve-devel" <pve-devel at pve.proxmox.com> 
> Envoyé: Vendredi 10 Juillet 2015 07:41:35 
> Objet: Re: [PVE-User] [pve-devel] backup : add pigz compressor 
> 
> >>We already have lzop, which is as fast as pgzip and uses less resources. 
> >>So what is the advantage of pgzip? 
> I think better compression than lzop (but slower), but still faster than classic gzip. 
> 
> 
> >>IMHO a parallel lzop would be nice. 
> 
> yeah. don't have find any implementation currently. 
> 
> 
> ----- Mail original ----- 
> De: "dietmar" <dietmar at proxmox.com> 
> À: "aderumier" <aderumier at odiso.com>, "pve-devel" <pve-devel at pve.proxmox.com> 
> Cc: "proxmoxve" <pve-user at pve.proxmox.com>, "Chidester, Bryce" <Bryce.Chidester at calyptix.com> 
> Envoyé: Vendredi 10 Juillet 2015 07:05:07 
> Objet: Re: [pve-devel] backup : add pigz compressor 
> 
> > user from pve-user mailing report than using pigz (multithread gzip), 
> > improve a lot speed of compression backup. 
> > 
> > Maybe could we use add it ? (or maybe replace gzip ?) 
> 
> We already have lzop, which is as fast as pgzip and uses less resources. 
> So what is the advantage of pgzip? IMHO a parallel lzop would be nice. 
> _______________________________________________ 
> pve-user mailing list 
> pve-user at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user 
> 
> _______________________________________________ 
> pve-devel mailing list 
> pve-devel at pve.proxmox.com 
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-devel