[pve-devel] [PATCH storage v4 09/27] plugin: introduce new_backup_provider() method

[Andreas Rogge]

Am 03.04.25 um 09:24 schrieb Wolfgang Bumiller:
> On Wed, Apr 02, 2025 at 06:16:57PM +0200, Andreas Rogge wrote:
>> Am 02.04.25 um 10:30 schrieb Wolfgang Bumiller:
>>> On Tue, Apr 01, 2025 at 08:21:30PM +0200, Thomas Lamprecht wrote:
> But I do wonder if - to reduce space-requirements for backing up running
> VMs - at some point we might also add the ability for qemu to provide
> some kind of queue containing the offset-length pairs of blocks which
> have been stored in the temporary fleecing image. The provider could
> consume this queue to keep the temporary storage at a minimum by doing
> out-of-order backups.
As we have to be able to restore out-of-order anyway, backing up 
out-of-order is not a problem at all.
However, offset-length in 512-byte offsets might be a bit too 
fine-grained. If we, for example, have a deduplication blocksize of 256k 
on a storage, offset and offset+length would preferably be divisible by 
256k.
While the backup application could easily compute aligned offsets 
itself, read-access to the whole computed region would be required.

> This only makes sense if there are backup systems which can benefit from
> it. (And it should be a simple-enough api extension to add this in the
> future, from what I can tell)

I am not sure if this provides a benefit on our end. However, if it 
makes the backup process more lightweight on the PVE side, it might 
still be worth the effort.

> I guess it makes sense for us to not expect/require random access, as
> any feature like that already imposes limitations on how the data can be
> stored. I'd expect different backup solutions to have different
> limitations in that regard.

Exactly.

> I *believe* `qemu-nbd` should be able to bind all the storage types we
> want to restore to to /dev/nbdXY devices, which would give the provider
> a bunch of block devices to write to in whichever way they want to, so
> the provider would then only need to figure out how to receive the data
> and forward that to the devices.
> We'll need to try.

That's pretty much what I had hoped for.

>>> Which is why the most flexible thing to do is to use a `qemu-img` call
>>> and giving it the paths, or more precisely, the URLs to the disks.
>>
>> I understand how this makes sense. However, if you don't have the data in a
>> format that qemu-img can consume, things become complicated.
> 
> It can also just read data from a stream (but without any smarter
> protocol, this would make holes/unallocated blocks extremely
> inefficient), where the provider would create that stream in whichever
> way they want to.
As the only viable formats to stream into qemu-img are qcow2 and raw and 
(to my understanding) these require the data to be in the right order 
Bareos would have to re-order the data before streaming it into 
qemu-img, which brings us back to staging the image - in which case we 
don't need to stream anymore :)

-- 
Andreas Rogge                             andreas.rogge at bareos.com
   Bareos GmbH & Co. KG                      Phone: +49 221-630693-86
   http://www.bareos.com

   Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
   Komplementär: Bareos Verwaltungs-GmbH
   Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz