[pbs-devel] [PATCH proxmox-backup v9 17/46] verify: implement chunk verification for stores with s3 backend

[Christian Ebner]

On 7/21/25 3:38 PM, Christian Ebner wrote:
> On 7/21/25 3:36 PM, Hannes Laimer wrote:
>> On Sat Jul 19, 2025 at 2:50 PM CEST, Christian Ebner wrote:
>>> For datastores backed by an S3 compatible object store, rather than
>>> reading the chunks to be verified from the local filesystem, fetch
>>> them via the s3 client from the configured bucket.
>>>
>>
>> Could we somehow take advantage of the hash S3 provides for objects[1]?
>> We can't use our hashes tough, so not if how we would...
>>
>> [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking- 
>> object-integrity.html
>> This is actually on my todo  list, the issue here is that not all s3 
> backends implement the checksuming as required. and for encrypted we 
> would need the crc checksums.
> 
> That is why for now this does not use these values, but rather the 
> regular download and check logic.

Further details and bugtracker issues where referenced in the
fn parse_x_amz_checksum_crc32_header(headers: &HeaderMap) method of 
https://lore.proxmox.com/pbs-devel/20250529143207.694497-13-c.ebner@proxmox.com/ 
which has been dropped since, as dead code.

Including them here for reference again as well:

+    // TODO: Integrity checks via CRC32 or SHA265 currently cannot be 
performed, since not
+    // supported by all S3 object store providers.
+    // See also:
+    // https://tracker.ceph.com/issues/63951
+    // https://tracker.ceph.com/issues/69105
+    // https://www.backblaze.com/docs/cloud-storage-s3-compatible-api