[pdm-devel] [PATCH proxmox-datacenter-manager v3 0/6] remote task cache fetching task / better cache backend

[Lukas Wagner]

On  2025-04-18 09:24, Lukas Wagner wrote:
> 
> 
> On  2025-04-17 17:31, Thomas Lamprecht wrote:
>> Am 17.04.25 um 15:22 schrieb Lukas Wagner:
>>> Benchmarking was done using the 'fake-remote' feature. There were 100
>>> remotes, 10 PVE nodes per remote. The task cache contained
>>> about 1.5 million tasks.
>>>                                                before        after
>>> list of active tasks (*):                     ~1.3s          ~30µs
>>> list of 500 tasks, offset 0 (**):             ~1.3s         ~500µs
>>> list of 500 tasks, offset 1 million (***):    ~1.3s         ~200ms
>>> Size on disk:                                 ~500MB        ~200MB
>>>
>>> (*):  Requested by the UI every 3s
>>> (**): Requested by the UI when visiting Remotes > Tasks
>>> (***): E.g. when scrolling towars the bottom of 'Remotes > Tasks'
>>
>> Did you regenerate these for the current revision? Just out of interest
>> whether there is any measurable effect of Wolfgangs proposed changes.
> 
> No, these are still from v1. I'll repeat the benchmark and check if there are any noteworthy
> changes.
> My prediction is that the impact is probably very small, but let's see if there
> are any surprises.
> 

Results are in:
                                              (before)       (v1)         (v3)
list of active tasks (*):                     ~1.3s          ~30µs        ~300µs
list of 500 tasks, offset 0 (**):             ~1.3s         ~500µs       ~1450µs
list of 500 tasks, offset 1 million (***):    ~1.3s         ~200ms        ~175ms
Size on disk:                                 ~500MB        ~200MB        ~200MB

I think the slowdown for the first two scenarios can be explained by the
changes in v2, where I wrapped the body of get_tasks in a tokio::task::spawn_blocking,
which I assume introduces some overhead.

With the third one we might actually see the slight improvements from Wolfgangs suggestions.

I'll include these results in the commit message in v4.

-- 
- Lukas