[pve-devel] Proxmox-Backup absolute to slow and to hardware intensive

[Detlef Bracker]

Dear,

3 years ago I have written scripts to make the backup fast and I have
reduced the load and cpu for
a better backup to an other server. Now I have test again, how we can
use the proxmox backup,
but for containers it?s absolute to slow, need to much cpu and make an
overhead of load a long time.
So one container with 70 GB needs 5 hours. Thats not usable!

So, small things to change the backup strategy works by us with own
scripts now to NFS very fast!

a) create rsync to a separate folder on same drive. In this folder hold
the copy of the rsync and dont
   delete them after the backup, so next day when the backups runs
again, the rsync is much faster
   and need 10 % of the time as when they must create a complete new
copy. About this the files
   synchronised exactly the same, delete the entries, they equal deleted
on the source.
   OK, yes thats need for containers they will backed up the double
space! This must been a feature
   in the setting in the config-File, so admin can set tempfolder and a
flag for hold the rsync in this
   tempfolder! We make too for every container 2 rsync runnings. The
first in running mode of the
   container and the 2nd in suspended mode of the container!
b) The second step is to create the tar-archives with lzo, but with full
and incremetal backup. So we
   make on the nfs for every container a folder and in this we create
the backups ins lzo-archives. When
   the counter file in this folder not exists or has inside a number of
backups they are runs over a limit,
   we delete only the timestamp - file (-g) and then a fullbackup runs
and the number of the backup will
   been written in this counter and begins from 1 otherwise +1.
c) About this strategy, we can have expl. 15 backups per container, but
only every 7 days a fullbackup and
   the other days a incremental backup. This reduce extremly the
backuptimes, reduce the load on the
   server and need less backup space and we can devide the fullbackups
of containers on diferent days!
   Expl. first big container has no counter file a make a fullbackup on
monday, the other big container
   will backup full on tuesday and so on!

We have reduced the time with this strategy for a lot of containers to
ca. 2 hours for ALL with less using
cpu and load! This backup strategy is much secure, ok need for
containers, they must backup, the
double space! But you see one big container 5 hours and we have 4 big
containers as same, this will
use with proxmox backup strategy 20 hours and with our concept all
together in 2-3 hours for all containers
with less cpu and less serverload and our clients very happy about this!

A other idea is, to make not a rsync and hold the copy! I mean, but I
will test in feature too, make a copy
of a container with inode links only! This works very fast, we used in
our other backup to hold many sets
of backups with small using of place, why only when a file was changed,
then for the old and new file this
need space. When the file was not changed, this need no space! So easy
tu use is, to make not the
rsync, but make a

/usr/bin/nice -n10 /usr/bin/ionice -c2 -n6 cp -al {container folder}
{temp_backup_folder} 2>/dev/null

before create the archive files. The problem I see, you get this not
synchrone in short time, so as with
rsync with 2 runnings of rsync. But this need no space and is very fast
on the fly and when the files changed on the
source, then the server held automaticly 2 files otherwise only the link!


-- 
ACHTUNG: Ihr Anfragetext befindet sich unter unserem Absender!
P.S. ePrivacy in Europa - lesen Sie mehr - read more
<http://blog.1awww.com/2012/05/30/achtung-internet-seiten-betreiber-eprivacy-richtlinien-umzusetzen/>


Mit freundlichen Gruessen
1awww.com - Internet-Service-Provider

Detlef Bracker
Velilla, Calle Club s/n, E 18690 Almunecar, Tel.: +34.6 343 232 61 *
EU-VAT-ID: ESX4516542D

This email and any files transmitted are confidential and intended only
or the person(s) directly addressed. If you are not the intended
recipient, any use, copying, transmission, distribution, or other forms
of dissemination is strictly prohibited. If you have received this email
in error, please notify the sender immediately and permanently delete
this email with any files that may be attached.

Este correo electrónico y, en su caso, cualquier fichero anexo al mismo,
contiene información de carácter confidencial exclusivamente dirigida a
su destinatario o destinatarios. Queda prohibida su divulgación, copia o
distribución a terceros sin la previa autorización escrita de Detlef
Bracker. En caso de no ser usted la persona a la que fuera dirigido este
mensaje y a pesar de ello está continúa leyéndolo, ponemos en su
conocimiento que está cometiendo un acto ilícito en virtud de la
legislación vigente en la actualidad, por lo que deberá dejarlo de leer
automáticamente.

Detlef Bracker no es responsable de su integridad, exactitud, o de lo
que acontezca cuando el correo electrónico circula por las
infraestructuras de comunicaciones electrónicas públicas. En el caso de
haber recibido este correo electrónico por error, se ruega notificar
inmediatamente esta circunstancia mediante reenvío a la dirección
electrónica del remitente.

El correo electrónico vía Internet no permite asegurar la
confidencialidad de los mensajes que se transmiten ni su integridad o
correcta recepción, por lo que Detlef Bracker no asume ninguna
responsabilidad que pueda derivarse de este hecho.

No imprima este correo si no es necesario. Ahorrar papel protege el
medio ambiente.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.proxmox.com/pipermail/pve-devel/attachments/20141119/5d48bc50/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1awww_abs_logo.gif
Type: image/gif
Size: 1457 bytes
Desc: not available
URL: <http://lists.proxmox.com/pipermail/pve-devel/attachments/20141119/5d48bc50/attachment.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.proxmox.com/pipermail/pve-devel/attachments/20141119/5d48bc50/attachment.sig>