[pve-devel] [RFC qemu/storage/qemu-server/container/manager 00/23] backup provider API

[Jonathan Nicklin]

I 100% concur. I am not suggesting any breaking changes; I was just wondering if this work on the API unlocked any new optimizations to make the interactions between the backup client, PBS, and storage more efficient. And also, bbgeek has pinged me to check out the awesome work going on in this space :)

Between your and Dietmar's replies, I see the constraints and potential avenues for improvement. Thanks for your reply!

Respectfully,
-Jonathan

> On Jul 29, 2024, at 4:15 AM, Fiona Ebner <f.ebner at proxmox.com> wrote:
> 
> Hi,
> 
> Am 26.07.24 um 21:47 schrieb Jonathan Nicklin via pve-devel:
>> 
>> Hi Fiona,
>> 
>> Would adding support for offloading incremental difference detection
>> to the underlying storage be feasible with the API updates? The QEMU
>> bitmap strategy works for all storage devices but is far from
>> optimal. If backup coordinated a storage snapshot, the underlying
>> storage could enumerate the differences (or generate a bitmap).
>> 
>> This would allow PBS to connect directly to storage and retrieve
>> incremental differences, which could remove the PVE hosts from the
>> equation. This "storage-direct" approach for backup would improve
>> performance, reduce resources, and support incremental backups in all
>> cases (i.e., power failues, shutdowns, etc.). It would also eliminate
>> the dependency on QEMU bitmaps and the overhead of fleecing.
>> 
>> Theoretically, this should be possible with any shared storage that
>> can enumerate incremental differences between snapshots: Ceph,
>> Blockbridge, iSCSi/ZFS?
>> 
>> Thoughts?
>> 
> 
> The two big advantages of the current mechanism are:
> 
> 1. it's completely storage-agnostic, so you can even use it with raw
> files on a directory storage. It follows in the same spirit as existing
> backup. Prohibiting backup for users when they use certain kinds of
> storages for VMs is not nice.
> 2. it's been battle-tested with PBS and works nicely.
> 
> I don't see why your suggestion can't be implemented in principle.
> Feature requests for (non-incremental) "storage-snapshot" mode backup
> have been around since a while. It was not a priority for development
> yet and is totally different from the current "snapshot" backup mode, so
> will need to be developed from the ground up.
> 
> That said, AFAICS, it's orthogonal to the series here. When an
> implementation like you outlined exists, it can just be added as a new
> backup mechanism for external providers (and PBS).
> 
> See also the related discussion over at:
> https://bugzilla.proxmox.com/show_bug.cgi?id=3233#c19
> 
> Best Regards,
> Fiona
>