[pbs-devel] [PATCH proxmox-backup] docs: explain some technical details about datastores/chunks

[Dominik Csapak]

On 12/10/20 12:42 PM, Lubomir Apostolov wrote:
> Hi,

hi,

> 
> Thank you for the reply.
> 
>> thanks for your message, your questions mean i did not
>> write the documentation clear enough, i try to answer here,
>> but i'll also incorporate that info in a v2
> 
> My questions are more about flaws in design than documentation.

well, all designs come with tradeoffs, i'll try to explain our choices
(and also add that to the docs in the future maybe)

> 
> I should also state that currently PBS is so highly inefficient that from
> our point of view (but we're not alone as you can see it in the forum)
> it's unsuitable in production :
> * Reading the all data every day for a backup is an overkill, which
>    leads to wasted energy and premature hardware failures.

but the only way to get 100% consistent backups
e.g. if an application sets the mtime/ctime
back to the original date, restic/borg would not
backup that file, even when the data changes

> * With big clusters of data, it can't be done on a daily basis,
>    so risk of loosing data is higher

if you have much data that changes every day, no other
backup solution will bring much benefit, as they also have
to read that data

if you have much data that does not change every day,
why do you want to back it up every day?

if your guest has a mixed use-case (some files change every day,
some files only change rarely) you can split them
into different directories and backup them separately

if you really need fast complete backups for big guests
that you do often, vms are the way to go

> 
>> just to clarify, that sentence only relates to 'file-based backups',
>> since for block based backups, we do not care about files at all,
>> just about the 'block image' we try back up.
> 
> Agreed.
> 
>> while something like that *could* work, it is not how we do it.
> 
> So there's no catch, PBS only took the simplest path ?

*could* as in theoretically, if all parts of the software
play well. for example, you wrote about snapshot diffs

those do not exists for all storages, or are not
easiliy accessible. writing backup for every specific
storage technology is really not sensible.

also you now would have to leave the snapshots on your source
storage intact, and in the case of zfs for example, you cannot
rollback without deleting them. in that case you'd have
to read the data to get the diff anyway..

if you want backups based on zfs, you can of course use
zfs send/receive (or pves replication) which is a perfectly
valid choice.

> 
>> instead of relying on storage snapshots (which may
>> not be available, e.g. for '.raw' files)
>> we simply iterate over the content of the block level image,
>> and create the chunks hashes. this normally means that we have
>> to read the whole image, but in case of pve qemu vms,
>> we use (as written in my patch) dirty-bitmaps
>> which keeps track of the changed blocks
>> (only those have to be hashed and backed up)
> 
> If snapshots aren't available, then you have to read all the data, agreed.
> But when snapshots are available, not using them is problematic as
> previously stated. Using dirty-bitmaps means you already implemented
> part of the changed blocks algorithm.

since with qemu backups we already use the dirty-bitmap,
we already only read what changed, or am i missing something here?

> 
>> again, we do not rely on storage snapshots, since those may
>> not be available (e.g. ext4)
> 
> Same comments as above.
> 
>> so, first we iterate over the filesystem/directory, from there,
>> we create a consistent archive format (pxar). this is something
>> like 'tar' but can be created in a streaming fashion (which we
>> need).
>> only over that archive, we create the chunks. so the data that
>> the chunker gets, has no direct relation to any files
> 
> When iterating through filesystem in a tar fashion, you have all the
> filenames you read so far.
> You can map data extents to files precisely.

i do not completely understand what you mean here, but
afaiu, you mean the chunks <-> file mapping?

if yes, then that could work but that comes with *very* big drawbacks: 
chunk size/speed/deduplication

if you have many small chunks, read and write speed will be
absolutely horrible

the readme in proxmox-backup[0] has some calculations of that
but the gist is that for big chunk sizes (we average here to about
4Mb/chunk, which we get with
the archive format and iterating over that) we get
speeds several magnitudes faster than with a small average
chunk size (e.g. 1kb)

if you would combine the files into chunks, deduplication would
suffer greatly, since the changed files gets combined
in a new way for every backup, in contrast if we combine it
consistently across backups, we can reuse many chunks

also the backups are not incremential or differential, but
every snapshot references the complete data of the source
meaning that you can delete every backup snapshot impacting
any other backups. this would not be easily possible
if we would introduce e.g. incremental backups
(since those would depend on a 'full' backup somehow)

also dynamically merging incremental backups while doing
so, costs in backup time and cpu power and is not trivial
to implement in a safe and efficient way

> Also I saw that you can mount pxar archives, which means you
> alread save the mapping inside.
> 

this has nothing to do with backing up really

when mapping, we have the complete list of chunks that
are referenced, and we dynamically look them up if we
need them, but that does not give us any file <=> relation

a simple example:

we have some pxar archive:

|start|file1_metadata|file1_data|file2_metadata|....|end|

now a dynamic chunk can begin anywhere not only on the entry boundaries
for example:

chunk1:
|start|file1_meta

chunk2:
data|file1_data|fil

chunk3:
e2_metadata|file2_

...

chunkX:
_data|end|

so if file1 gets added one byte it could look like

chunk1*:
|start|file1*_meta

chunk2*:
data|file1*_data|fil

chunk3:
e2_metadat|file2_
..

all chunks from chunk3 to chunkX stay the same
and only 2 new chunks get added

(ofc this is a very oversimplified example)

> I hope my point of view is clearer now.

i get where you are coming from, and i hope
i could explain the rationale behind the
design well enough

> 
> Best regards,
> Lubomir Apostolov
> 

kind regards,
Dominik

0: 
https://git.proxmox.com/?p=proxmox-backup.git;a=blob;f=README.rst;h=5c42ad10f58d259606b643df542e2664b08009fe;hb=refs/heads/master