[pve-devel] [PATCH common 3/4] limit tasklist also to the maximal pmxcfs status entry size

[Thomas Lamprecht]

We tried to limit the size of the tasklist by including non-running
task only if we have less than 25 entries. A reason, among others,
was that a single status entry in the cfs_status.kvhash is limited to
32 KiB.

The "max. 25 entry" heuristic assumes that entries are small, which
is also the norm.  But on failed tasks, e.g. a Qemu VM with a
problematic command line, is far longer than the usual task entry.

This led to a situation where the last 25 task were bigger than
32KiB, so the ipcc call to the pmxcfs failed with EFBIG.
This aborted then every new task run with fork_worker, and could
render a node partially unusable until "/var/log/pve/tasks/active"
got truncated.
To recreate this issue quite fast do:
---
 # qm create 11109 --args "'$(dd if=/dev/urandom bs=1024 count=1 2>/dev/null | base64 -w 0)'"
 # while true; do qm start 11109; done
---

You should see soon a "ipcc_send_rec failed: File too large"
After this all new task fail, even if they could succeed. pvestatd
also fails to broadcast the tasklist now. To get out of this do:
 # echo > /var/log/pve/tasks/active

To address this add a size counter, which tracks the serialized
length of the task list and limits it to 32KB (not KiB to have some
bytes in reserve for the full serialization of this array).

Current running task will get always included, as previously.

Signed-off-by: Thomas Lamprecht <t.lamprecht at proxmox.com>
---
 src/PVE/RESTEnvironment.pm | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/src/PVE/RESTEnvironment.pm b/src/PVE/RESTEnvironment.pm
index 15eef90..06270d2 100644
--- a/src/PVE/RESTEnvironment.pm
+++ b/src/PVE/RESTEnvironment.pm
@@ -17,6 +17,7 @@ use PVE::SafeSyslog;
 use PVE::Tools;
 use PVE::INotify;
 use PVE::ProcFSTools;
+use JSON;
 
 
 my $rest_env;
@@ -315,12 +316,15 @@ sub active_workers  {
 	    }
 	}
 
-	# we try to reduce the amount of data
-	# list all running tasks and task and a few others
-	# try to limit to 25 tasks
+	# running task get always included, recent finished then get included
+	# until we reach either 25 entries or the serialized list would be
+	# bigger than 32kb (aligned to CFS_MAX_STATUS_SIZE from pmxcfs)
 	my $max = 25 - scalar(@$tlist);
+	my $size = length(encode_json($tlist));
         foreach my $task (@ta) {
 	    last if $max <= 0;
+	    $size += length(encode_json($task));
+	    last if $size > (32*1000);
 	    push @$tlist, $task;
 	    $max--;
 	}
-- 
2.11.0