[pve-devel] [RFC qemu/storage/qemu-server/container/manager 00/23] backup provider API

[Jonathan Nicklin]

I am by no means a CEPH expert. However, my understanding is that other backup solutions (in the OpenStack world) have used rbd diff to enable incremental backups. I was hoping that would be relevant here. 

Here's the description of `rbd diff`

<rbd-diff>
Dump a list of byte extents in the image that have changed since the specified start snapshot, or since the image was created. Each output line includes the starting offset (in bytes), the length of the region (in bytes), and either ‘zero’ or ‘data’ to indicate whether the region is known to be zeros or may contain other data.
</rbd-diff>

We (Blockbridge) can also enumerate differences between snapshots in the form of extent ranges.

We share the same concerns regarding the consistency of QEMU bitmaps wrt storage. That is why relying on the storage to track differences feels like a more robust solution. 

> On Jul 28, 2024, at 3:55 AM, Dietmar Maurer <dietmar at proxmox.com> wrote:
> 
> 
>> The biggest issue we see reported related to QEMU bitmaps is
>> persistence. The lack of durability results in unpredictable backup
>> behavior at scale. If a host, rack, or data center loses power, you're
>> in for a full backup cycle. Even if several VMs are powered off for
>> some reason, it can be a nuisance. Several storage solutions can
>> generate the incremental difference bitmaps from durable sources,
>> eliminating the issue.
> 
> Several storage solutions provides internal snapshots, but none of them has an API to access the dirty bitmap (please correct me if I am wrong). Or what storage solution do you talk about exactly?
> 
> Storing the dirty bitmap persistently would be relatively easy, but so far we found no way to make sure the bitmap is always up-to-date. 
> We support shared storages, so multiple nodes can access and modify the data without updating the dirty bitmap, which would lead to corrupt backup images...
>