[pbs-devel] [RFC v1 proxmox-backup] fix: #3847 pipe from STDIN to proxmox-backup-client

[Christian Ebner]

Hi, thanks for having a look at this issue, this can be an interesting 
feature.

On 11/21/25 4:41 PM, Robert Obkircher wrote:
> It was requested [0] that the backup client should suport reading data
> from stdin. This could be used to back up files from an unsupported
> remote system via ssh or to store the output of mysqldump/pg_dump
> commands.
> 
> This is currently not supported, because writing fixed indices needs
> to know the size ahead of time.
> 
> This patch adds experimental support for commands like:
>      cat "$FILE" | proxmox-backup-client backup data.img:/dev/stdin
>      proxmox-backup-client backup a.img:<(cmd1) b.img:<(cmd2)
> 
> It uses write_at (pwrite64) instead of mmap, because it seemed easier
> to get a prototype this way, but it would of course also be possible
> to continue using mmap.
> 
> Before I spend more time on this I wanted to ask for some opinions:
> 
> Does it actually make sense to support this at all?

Yes, the requested dumping of contents from e.g. a database into a 
stream which can then be directly be backed up without storing an 
temporary file are a valid usecase I think. Adding a support for this 
justified.

> If so, should it be a separate endpoint/writer or unified?

Using a different endpoint is not required, in my opinion this might 
however be approach a bit differently than what your initial draft does.

I think it would make sense to have the following options for the 
proxmox-backup-client, both of which can be implemented independent from 
each other:
- Allow the input stream to be a generic raw byte stream by using the 
img archive name extension, chunking it using the fixed size chunker and 
upload this as fixed index to the proxmox backup server.
- Allow the user to set the mode based on pxar archive name extension 
(and check the first bytes of the stream for matching pxar magic number) 
to pass in a pre-encoded pxar stream. This would then however also 
require to continuosuly check the consistency of the provided pxar 
stream, aborting if inconsistencies are detected and is more involved.

For both cases the chunking, upload and encryption can still be 
performed using the pre-existing logic.

For the FixedIndexWriter as drafted by your patch, I would however 
suggest to try to approach this a bit differently. This could maybe be 
done by defining a trait which clearly defines the required method 
interfaces first, and then implement this trait for the 
FixedIndexWriterSized and maybe a FixedIndexWriterUnsized?

Both implementations could then reuse as much as possible from the 
pre-existing logic and allow to not interfer with the already existing 
implementation. Also, the mmapped logic for the writer with already 
known size must be maintained.

What do you think?