[pdm-devel] [PATCH proxmox-datacenter-manager v3 0/6] remote task cache fetching task / better cache backend

[Lukas Wagner]

On  2025-04-17 17:31, Thomas Lamprecht wrote:
> Am 17.04.25 um 15:22 schrieb Lukas Wagner:
>> Benchmarking was done using the 'fake-remote' feature. There were 100
>> remotes, 10 PVE nodes per remote. The task cache contained
>> about 1.5 million tasks.
>>                                                before        after
>> list of active tasks (*):                     ~1.3s          ~30µs
>> list of 500 tasks, offset 0 (**):             ~1.3s         ~500µs
>> list of 500 tasks, offset 1 million (***):    ~1.3s         ~200ms
>> Size on disk:                                 ~500MB        ~200MB
>>
>> (*):  Requested by the UI every 3s
>> (**): Requested by the UI when visiting Remotes > Tasks
>> (***): E.g. when scrolling towars the bottom of 'Remotes > Tasks'
> 
> Did you regenerate these for the current revision? Just out of interest
> whether there is any measurable effect of Wolfgangs proposed changes.

No, these are still from v1. I'll repeat the benchmark and check if there are any noteworthy
changes.
My prediction is that the impact is probably very small, but let's see if there
are any surprises.

> 
> And FWIW, I'd like to encode these also in the commit message, albeit
> with the lore links that b4 adds it's less of a problem, but IMO would
> still not really hurt to do. Depending on your answer on the actuality
> of above data I could amend that or newly provided measurements into
> the commit message on applying though, so definitively no need for a v+1
> just for that.
> 

Since there is one more thing to fix as reported by Wolfgang, I'd post a v4 with the fix
and amended commit message which includes updated benchmark results. Shouldn't take too
long I hope!

>>
>> In the old implementation, the archive file was *always* fully deserialized
>> and loaded into RAM, this is the reason why the time needed is pretty
>> idential for all scenarios.
>> The new implementation reads the archive files only line by line,
>> and only 500 tasks were loaded into RAM at the same time. The higher the offset,
>> the more archive lines/files we have to scan, which increases the
>> time needed to access the data. The tasks are sorted descending
>> by starttime, as a result the requests get slower the further you
>> go back in history.
>>
>> The 'before' times do NOT include the time needed for actually fetching
>> the task data.
>>
>> This series was preseded by [1], however almost all of the code has changes, which
>> is the reason why I send this as a new series.
>>
>> [1] https://lore.proxmox.com/pdm-devel/20250128122520.167796-1-l.wagner@proxmox.com/
>>
>> Changes since v2:
>>   - Change locking approach as suggested by Wolfgang
>>   - Incorporated feedback from Wolfang
>>      - see patch notes for details
>>   - Added some .context/.with_context for better error messages
>>
> 
> Looks alright to me now, if Wolfgang sees nothing of w.r.t. the changes
> since the last version I'd apply this series tomorrow.

-- 
- Lukas