[pbs-devel] Bulk initial sync from remote

[Fabian Grünbichler]

> Sebastian <s.schauenburg at gmail.com> hat am 26.03.2021 16:15 geschrieben:
> 
> Good afternoon everyone,
> 
> is it possible to do an initial bulk sync from a remote? (using external storage media for example)
> E.g. can all files (chunk directory etc.) be blindy copied from one pbs server to the remote pbs server using a external storage medium?

yes. a "blind copy" does risk a certain amount of inconsistency if there are any concurrent actions on the datastore (e.g., if you first copy all the snapshot metadata first, then continue with .chunks, and now a prune + GC run happens and deletes some chunks that you haven't copied yet).

you can avoid that by:
- defining the external medium as datastore, configure a 'local' remote pointing to the same node, and use the sync/pull mechanism instead of a blind copy (that will iterate over snapshots and copy associated chunks together with the snapshot metadata, so you'll never copy orphaned chunks or snapshot metadata without associated chunks). this will incur network/TLS overhead since it works over the API
- do a two-phase rsync or similar, and ensure the datastore is quiet for the final (small) sync

after moving your external disk, you need to manually create the datastore.cfg entry (or create a datastore using the GUI with a different path, and then edit it to point it to your actual path, or copy the contents from your external media into the created directory).

a datastore directory with the .chunks subdir and the backup type directories (by default: vm, ct, host) is self-contained as far as stored backups are concerned. scheduled jobs (prune, verify, GC) are stored outside, so those need to be recreated if you just have the "raw" datastore.

> Use-case: doing a initial full sync from a remote can cost a lot of bandwidth (or time), while incrementals can be small (when aren't a lot of changes).

common use case, should work with the caveats noted above :)