[pbs-devel] [PATCH v4 proxmox-backup 0/6] closes #3071: maintenance mode for datastore

[Thomas Lamprecht]

On 12.11.21 13:30, Hannes Laimer wrote:
> Adds maintenance mode and tracking of active reading/writing operations.
> The maintenance mode prevents the start of new operations if the type of
> operation they would perform on the datastore would conflict with the
> maintenance type that is currently set. This check is performed when
> lookup_datastore is called. Tasks only call this function once at the
> beginning, therefore updating the maintenance type cannot interfere with
> already running tasks.
> 
> active operations tracking:
> Changed file layout to now also keep track of the pid and the counts of
> operations that that pid started, like this it is possible to not count
> operations that were started by a dead process, since they are also not
> active anymore. Whenever the file is updated, also entries of dead

note that PIDs can be reused, normally one also safes the processes start-time
to check for (that's theoretically also racy, but in practice no one can spin
that many processes up to force reuse in such a short time).


FWIW, your use case here would actually just require the processes start time,
you can then simply check if it's older than the one of the current process,
if it is, it'd be out of date.

> processes are removed. When the file is read, only entries of active
> processes are counted.
> 
> The UI shows a spinner with the count of conflictintg tasks (the tasks
> that were started before the maintenance type was updated) next to it.
> As soon as all conflicting tasks are finished a checkmark appears.
> 

It did not applied cleanly anymore, so I resolved the merge conflicts, fixed
a compile error (operations.clone() cannot work as is) applied Dominik's fixes
and cleanups, and did another quick round of cleanup myself, reducing indentation
depth and line count quite a bit, pushed that out to my internal staff repository
as: hannes/datastore-maintenance-v4-with-cleanups


Anyhow, the main reason I did not went for this now is the manual file format
parser, with serde and now also the shared-memory stuff from Dietmar available
we have way better approaches that keeps us more flexible and normally gets a
more efficient parser plus more friendly working with the data as programmer.

So, please:

* check out my branch (you can squash stuff in, at least mine, if you want)
  the most important thing is that you ensure Dominik's patch that fixes
  tracking for tape is not lost, as without that it won't work as you was
  reminded twice by Dominik, IIRC.
* create an actual struct for the active operations tracking and either
  - derive serialize and de-serialize and just save/load it as json
  - use the shared memory infrastructure, it should cover your use case
    relatively fine and may be slightly cheaper (as we save the parse step
    completely)
* see if we can move that tracking related stuff in its own file, datastore.rs
  is already relatively big..

As said, most of it was ok-ish as is and could have been followed up, but I just
do not want another self-rolled format, that is hard to expand nicely, out there,
especially not as the rust infrastructure we have just makes it so simply to do
it in a nicer way.