[pbs-devel] Chunk verification speedup discussion, similar speedup opportunities elsewhere

[Adam Kalisz]

On Fri, 2025-07-25 at 14:46 +0200, Fabian Grünbichler wrote:
> On July 25, 2025 1:23 pm, Adam Kalisz wrote:
> > Hi list,
> > 
> > I missed whether the chunk verification speedup when loading chunks
> > got
> > applied or whether it was somehow included in the S3-like storage
> > option change set.
> 
> https://lore.proxmox.com/pbs-devel/20250707132706.2854973-1-d.csapak@proxmox.com/
> 
> hasn't been applied (and probably needs a rebase post-S3 ;))

Yes and the opt-out knob that Thomas mentioned on Friday should be
added:

"So the mentioned series should IMO only be applied if it can be opted
out, or, better in the long term, a pacing/IO bandwidth limit can be
enforced. That is naturally a big amount of work, mostly in evaluating
different options and deciding for what can make sense, that's why I
mentioned that being able to opt-out could be enough, then admins
could at least keep the status quo."

> > In
> > https://forum.proxmox.com/threads/abysmally-slow-restore-from-backup.133602/page-7
> > we have discussed some other opportunities for speedup using
> > similar
> > patterns. People mentioned LXC container restore speed and  host-
> > based
> > backup restore, which looking at the code for the latter seems like
> > a
> > similar async loop pattern would bring some improvement without too
> > much trouble:
> > 
> > https://github.com/proxmox/proxmox-backup/blob/4940514b0f05d6cd6a5f711edfdd47c1fa41b537/proxmox-backup-client/src/main.rs#L1109
> 
> that's not the code path for container/host backups, that's the code
> path for fixed-size indices like vm backups, but just dumping to a
> file instead of via qmrestore.. i.e., nothing that is used by PVE or
> users usually, but can still be improved of course ;)

Ah ok, I just skimmed over it to find patterns that look like the
things Dominik and I improved before elsewhere.

> > Similarly the sync performance between two Proxmox Backup Servers
> > and
> > live-migration got mentioned in various places.
> 
> do you mean live-restore here? live migration has nothing to do with
> PBS..

I threw more things into a single basked as the e-mail was sent to both
lists and meant live migration that seems like it could go faster in
some cases. (But that might be code that originates in the QEMU
project, right?) We could of course have a look at live-restore or even
some things like migration from VMware/ ESXi too (again, unrelated to
PBS).

I just want to confirm if any of this makes sense and where to look
basically so we can have a list of areas for improvement and what code
path that is.

The speed improvements already caused qualitative improvements that
e.g. allow some companies to choose PVE+PBS instead of other options
even in more demanding cases.

Thank you for bringing me up to speed. Have a successful week!