[pdm-devel] [PATCH proxmox-datacenter-manager v2 00/28] metric collection improvements (concurrency, config, API, CLI)

[Lukas Wagner]

On  2025-02-14 14:06, Lukas Wagner wrote:
> ## To reviewers / open questions:
> - Please review the defaults I've chosen for the settings, especially
>   the ones for the default metric collection interval (10 minutes) as
>   well as max-concurrency (10).
>   I also kindly ask to double-check the naming of the properties.
>   See "pdm-api-types: add CollectionSettings type" for details
> 
> - Please review path and params for new API endpoints (anything public
>   facing that is hard to change later)
> 
> - I've chosen a section-config config now, even though we only have a
>   single section for now. This was done for future-proofing reasons,
>   maybe we want to add support for different setting 'groups' or
>   something, e.g. to have different settings for distinct sets of
>   remotes. Does this make sense?
>   Or should I just stick to a simple config for now? (At moments like
>   these I wish for TOML configs where we could be a bit more flexible...)
> 
> 	collection-settings: default
> 	    max-concurrency 10
> 	    collection-interval 180
> 	    min-interval-offset 0
> 	    max-interval-offset 20
> 	    min-connection-delay 10
> 	    max-connection-delay 100
> 

Currently thinking about generalizing the `max-concurrency` setting into something global
that affects all 'background' polling operations (resource cache/task cache refreshes/
metric fetching).

For actually controlling the concurrency we could maybe have a globally available
semaphore (potential deadlock potential in some cases).
Alternatively, we could think about having a 'background request queue' and
a 'background request scheduler', which does the actual requests on behalf of
the other tasks.

I'll give this a bit more thoughts in the next days/weeks, so maybe don't merge
this in the meanwhile.

-- 
- Lukas