[pve-devel] [RFC] towards automated integration testing

[Lukas Wagner]

Thank you for the feedback!

On 10/16/23 13:20, Stefan Hanreich wrote:
> On 10/13/23 15:33, Lukas Wagner wrote:
> 
>> - Additionally, it should be easy to run these integration tests locally
>>    on a developer's workstation in order to write new test cases, as well
>>    as troubleshooting and debugging existing test cases. The local
>>    test environment should match the one being used for automated testing
>>    as closely as possible
> This would also include sharing those fixture templates somewhere, do
> you already have an idea on how to accomplish this? PBS sounds like a
> good option for this if I'm not missing something.
> 
Yes, these templates could be stored on some shared storage, e.g. a PBS
instance, or they could also distributed via a .deb/multiple .debs (not 
sure if that is a good idea, since these would become huge pretty
quickly).

It could also be a two-step process: Use one command to get the
latest test templates, restoring them from a remote backup, converting
them to a local VM template. When executing tests, the test runner
could then use linked clones, speeding up the test setup time
quite a bit.

All in all, these templates that can be used in test fixtures should be:
   - easily obtainable for developers, in order to have a fully
     functional test setup on their workstation
   - easily updateable (e.g. installing the latest packages, so that
     the setup-hook does not need to fetch a boatload of packages every
     time)


>> As a main mode of operation, the Systems under Test (SUTs)
>> will be virtualized on top of a Proxmox VE node.
>>
>> This has the following benefits:
>> - it is easy to create various test setups (fixtures), including but not
>>    limited to single Proxmox VE nodes, clusters, Backup servers and
>>    auxiliary services (e.g. an LDAP server for testing LDAP
>>    authentication)
> I can imagine having to setup VMs inside the Test Setup as well for
> doing various tests. Doing this manually every time could be quite
> cumbersome / hard to automate. Do you have a mechanism in mind to deploy
> VMs inside the test system as well? Again, PBS could be an interesting
> option for this imo.
> 
Several options come to mind. We could use a virtualized PBS instance 
with a datastore containing the VM backup as part of the fixture.
We could use some external backup store (so the same 'source' as for the 
templates themselves) - however that means that the systems under test
must have network access to that.
We could also think about using iPXE to boot test VMs, with the
boot image either be provided by some template from the fixture, or by
some external server.
For both approaches, the 'as part of the fixture' approaches seem a bit
nicer, as they are more self-contained.

Also, the vmbd2 thingy that thomas mentioned might be interesting for
this - i've only glanced at it so far though.

As of now it seems that this question will not influence the design
of the test runner much, so it can probably be postponed to a later
stage.

>> In theory, the test runner would also be able to drive tests on real
>> hardware, but of course with some limitations (harder to have a
>> predictable, reproducible environment, etc.)
> 
> Maybe utilizing Aaron's installer for setting up those test systems
> could at least produce somewhat identical setups? Although it is really
> hard managing systems with different storage types, network cards, ... .

In general my biggest concern with 'bare-metal' tests - and to precise,
that does not really have anything to do with being 'bare-metal',
more about testing on something that is harder roll back into
a clean state that can be used for the next test execution, is that
I'm afraid that a setup like this could become quite brittle and a 
maintenance burden. At some point, a test execution might leave
something in an unclean state (e.g. due to a crashed test or missing
something while cleanup), tripping up the following test job.
As an example from personal experience: One test run
might test new packages which introduce a new flag in a configuration
file. If that flag is not cleanup up afterwards, another test job
testing other packages might fail because it now has to
deal with an 'unknown' configuration key.

Maybe ZFS snapshots could help with that, but I'm not sure how that
would work in practice (e.g. due to the kernel being stored on
the EFI partition).

The automated installer *could* certainly help here - however,
right now I don't want to extend the scope of this project too much.
Also, there is also the question if the installation should be refreshed 
after every single test run, increasing the test cycle time/resource 
consumption quite a bit? Or only if 'something' breaks?

That being said, it might also make sense to be able to run the tests
(or more likely, a subset of them, since some will inherently
require a fixture) against an arbitrary PVE instance that is under full 
control of a developer (e.g. a development VM, or, if feeling 
adventurous, the workstation itself). If this is possible, then these 
tests could the fastest way to get feedback while developing, since
there is no need to instantiate a template, update, deploy, etc.
In this case, the test runner's job would only be to run the test 
scripts, without managing fixtures/etc, and then reporting the results
back to the developer.

Essentially, as Thomas already mentioned, one approach to do this would
be to decouple the 'fixture setup' and 'test case execution' part as
much as possible. How that will look in practice will be part of
further research.

> I've seen GitLab using tags for runners that specify certain
> capabilities of systems. Maybe we could also introduce something like
> that here for different bare-metal systems? E.g. a test case specifies
> it needs a system with tag `ZFS` and then you can run / skip the
> respective test case on that system. Managing those tags can introduce
> quite a lot of churn though, so I'm not sure if this would be a good idea.
> 

I have thought about a tag system as well - not necessarily for test
runners, but for test cases. E.g. you could tag tests for the
authentication system with 'auth' - because at least for the local
development cycle it might not make much sense to run tests for
clusters, ceph, etc. while working on the authentication system.

The 'tags' to be executed might then be simply passed to the test
runner.

These tags could also be used to mark the subset of 'simple'
test cases that don't need a special test fixture, as described above...

This could also be extended to a full 'predicate-like' system as Thomas
described.


>> The test script is executed by the test runner; the test outcome is
>> determined by the exit code of the script. Test scripts could be written
> Are you considering capturing output as well? That would make sense when
> using assertions at least, so in case of failures developers have a
> starting point for debugging.
Yup, I'd capture stdout/stderr from all test executables/scripts and
include it in the final test report.
Test output is indeed very useful when determining *why* something went
wrong.

> 
> Would it make sense to allow specifying a expected exit code for tests
> that actually should fail - or do you consider this something that
> should be handled by the test script?

I guess that's a matter of taste. Personally I'd keep the contract
between test runner and test script simple and say 0 == success,
everything else is a failure. If there are any test cases that
expect a failure of some API call, then the script should 'translate'
the exit code.
If we discover that specifying an expected exit actually makes things
easier for us, then adding it should be rather trivial - and easier
than ripping it out the other way round.


> I've refrained from talking about the toml files too much since it's
> probably too early to say something about that, but they look good so
> far from my pov.
> 
> In general this sounds like quite the exciting feature and the RFC looks
> very promising already.

Thanks for your feedback!

> 
> Kind Regards
> Stefan

-- 
- Lukas