[pbs-devel] [PATCH proxmox-backup v2 6/6] api: admin: datastore: implement streaming content api call

[Dominik Csapak]

On 10/8/25 9:49 PM, Thomas Lamprecht wrote:
> Am 08.10.25 um 15:43 schrieb Dominik Csapak:
>> this is a new api call that utilizes `proxmox_router::Stream` to provide
>> a streaming interface to querying the datastore content.
>>
>> This can be done when a client requests this api call with the
>> `application/json-seq` Accept header.
>>
>> In contrast to the existing api calls, this one
>> * returns all types of content items (namespaces, groups, snapshots; can
>>    be filtered with a parameter)
>> * iterates over them recursively (with the range that is given with the
>>    parameter)
>>
>> The api call returns the data in the following order:
>> * first all visible namespaces
>> * then for each ns in order
>>    * each group
>>    * each snapshot
>>
>> This is done so that we can have a good way of building a tree view in
>> the ui.
> 
> I guess you did not get around to test some more performance / memory
> usage here? Might be nice to have whatever stats you did compare encoded
> in the commit message here.
> 
> I.e. that part of you and my text from patch 6/6 from the v1:
> 
> Am 03.10.25 um 13:55 schrieb Thomas Lamprecht:
>> Am 03.10.25 um 10:51 schrieb Dominik Csapak:
>>> interesting side node, in my rather large setup with ~600 groups and ~1000
>>> snapshosts per group, streaming this is faster than using the current
>>> `snapshot` api (by a lot):
>>> * `snapshot` api -> ~3 min
>>> * `content` api with streaming -> ~2:11 min
>>> * `content` api without streaming -> ~3 min
>>>
>>> It seems that either collecting such a 'large' api response (~200MiB)
>>> is expensive. My guesses what happens here are either:
>>> * frequent (re)allocation of the resulting vec
>>> * or serde's serializing code
>>
>> You could compare peak (RSS) memory usage of the daemon as side-effect,
>> and/or also use bpftrace to log bigger allocations. While I did use bpftrace
>> lots of times, I did not try this specifically to rust, but I found a
>> shorth'ish article that describes doing just that for rust, and looks like
>> it would not be _that_ much work (and could be a nice tool to have in the
>> belt in the future):
>>
>> https://readyset.io/blog/tracing-large-memory-allocations-in-rust-with-bpftrace

ok so i tried to follow the link, but couldn't find out how to use 
bpftrace in our case, since we use glibcs malloc, which is not in the 
binary itself, and it seems not very obvious to me how to know how the
rust program calls mmap in glibc  (none of the symbols matchin 'mmap'
seem to correlate, and looking for 'alloc' returns too many entries
to be feasible for checking?

i'm not very used to bpftrace though, but i'll keep it on my radar for
the future.

anyway, i measured the peak RSS (directly from procfs) while these api
calls were running:

for an old 'snapshot listing' with the same datastore as mentioned 
before, the peak usage was ~446MiB, while on the newer content
streaming api (while streaming) was ~33MiB (which was about the
same as before and after the api call)

so the memory usage for building the lists in memory takes about
446-32 ~414MiB (for about 600.000 snapshots)


should i send a v3 with that info included? or should i wait
for someone to do a more in depth review of the permissions?
(AFAICS nobody wrote anything about that yet)