[pdm-devel] [PATCH proxmox-datacenter-manager 00/15] change task cache mechanism from time-based to max-size FIFO

[Lukas Wagner]

This patch series changes the remote task caching behavior from being purely
time-based cache to a FIFO cache-replacement policy with a maximum number of
cached tasks per remote. If the maximum number is exceeded, the oldest tasks
are dropped from the cache.

When calling the remote-tasks API, the latest missing task data which is not
yet in the cache is requested from the remotes. There we limit this to once
every 5 minutes at the moment, with the option for a force-refresh (to be triggered
by a refresh button in the UI). As before, we augment the cached task data
with the currently running tracked tasks which were started by PDM.

Some words about the cache storage implementation:
Note that the storage backend for this cache probably needs some more love in
the future. Right now its just a single JSON file for everything, mainly because this
was the quickest approach to implement to unblock UI development work. 
The problem with the approach is that it does not perform well with setups
with a large number of remotes, since every update to the cache rewrites
the entire cache file when the cache is persisted, causing additional
CPU and IO load.

In the future, we should use a similar mechanism as the task archive in PBS.
I'm not sure if the exact same mechanism can be used due to some different
requirements, but the general direct probably fits quite well.
If we can reuse it 1:1, we have to break it out of (iirc) the WorkerTask struct
to be reusable.
It will probably require some experimentation and benchmarking to find an
ideal approach.
We probably don't want a separate archive per remote, since we do not want to
read hundres/thousands of files when we request an aggregated remote task history
via the API. But having the complete archive in a single file also seems quite
challenging - we need to keep the data sorted, while we also need to
handle task data arriving out of order from different remotes. Keeping
the data sorted when new data arrives leads us to the same problem as with
JSON file, being that we have to rewrite the file over and over again, causing
load and writes.

The good news is that since this is just a cache, we are pretty free to change
the storage implementation without too much trouble; we don't even have to
migrate the old data, since it should not be an issue to simply request the
data from the remotes again. This is the main reason why I've opted
to keep the JSON file for now; I or somebody else can revisit this at a later
time.

proxmox-datacenter-manager:

Lukas Wagner (15):
  pdm-api-types: derive Debug and PartialEq for TaskListItem
  test support: add NamedTempFile helper
  task cache: add basic test for TaskCache
  task cache: remove max-age machanism
  task cache: add FIFO cache replacement policy
  remote tasks: move to dir based module
  task cache: move to its own submodule
  task cache: fetch every 5mins, requesting only missing tasks
  remote tasks: return tasks in stable order
  remote tasks: allow to force-fetch latest tasks
  fake remote: add missing fields to make the debug feature compile
    again
  fake remote: generate fake task data
  task cache: tests: improve test coverage
  remote tasks: fix unused variable warning
  remote-tasks: restrict function visibility

 lib/pdm-api-types/src/lib.rs                  |   2 +-
 server/src/api/pve/mod.rs                     |   4 +-
 server/src/api/remote_tasks.rs                |  18 +-
 server/src/lib.rs                             |   4 +-
 .../{task_cache.rs => remote_tasks/mod.rs}    | 253 ++++--------
 server/src/remote_tasks/task_cache.rs         | 365 ++++++++++++++++++
 server/src/test_support/fake_remote.rs        |  81 +++-
 server/src/test_support/mod.rs                |   4 +
 server/src/test_support/temp.rs               |  33 ++
 9 files changed, 563 insertions(+), 201 deletions(-)
 rename server/src/{task_cache.rs => remote_tasks/mod.rs} (62%)
 create mode 100644 server/src/remote_tasks/task_cache.rs
 create mode 100644 server/src/test_support/temp.rs


Summary over all repositories:
  9 files changed, 563 insertions(+), 201 deletions(-)

-- 
Generated by git-murpp 0.8.0