[pdm-devel] [PATCH proxmox-datacenter-manager v3 0/6] remote task cache fetching task / better cache backend

[Thomas Lamprecht]

Am 17.04.25 um 15:22 schrieb Lukas Wagner:
> Benchmarking was done using the 'fake-remote' feature. There were 100
> remotes, 10 PVE nodes per remote. The task cache contained
> about 1.5 million tasks.
>                                                before        after
> list of active tasks (*):                     ~1.3s          ~30µs
> list of 500 tasks, offset 0 (**):             ~1.3s         ~500µs
> list of 500 tasks, offset 1 million (***):    ~1.3s         ~200ms
> Size on disk:                                 ~500MB        ~200MB
> 
> (*):  Requested by the UI every 3s
> (**): Requested by the UI when visiting Remotes > Tasks
> (***): E.g. when scrolling towars the bottom of 'Remotes > Tasks'

Did you regenerate these for the current revision? Just out of interest
whether there is any measurable effect of Wolfgangs proposed changes.

And FWIW, I'd like to encode these also in the commit message, albeit
with the lore links that b4 adds it's less of a problem, but IMO would
still not really hurt to do. Depending on your answer on the actuality
of above data I could amend that or newly provided measurements into
the commit message on applying though, so definitively no need for a v+1
just for that.

> 
> In the old implementation, the archive file was *always* fully deserialized
> and loaded into RAM, this is the reason why the time needed is pretty
> idential for all scenarios.
> The new implementation reads the archive files only line by line,
> and only 500 tasks were loaded into RAM at the same time. The higher the offset,
> the more archive lines/files we have to scan, which increases the
> time needed to access the data. The tasks are sorted descending
> by starttime, as a result the requests get slower the further you
> go back in history.
> 
> The 'before' times do NOT include the time needed for actually fetching
> the task data.
> 
> This series was preseded by [1], however almost all of the code has changes, which
> is the reason why I send this as a new series.
> 
> [1] https://lore.proxmox.com/pdm-devel/20250128122520.167796-1-l.wagner@proxmox.com/
> 
> Changes since v2:
>   - Change locking approach as suggested by Wolfgang
>   - Incorporated feedback from Wolfang
>      - see patch notes for details
>   - Added some .context/.with_context for better error messages
> 

Looks alright to me now, if Wolfgang sees nothing of w.r.t. the changes
since the last version I'd apply this series tomorrow.