meta-dts/docs/testing.md

Testing

Current E2E tests are defined in OSFV repo

They work by running through different workflows from the moment user sees UI to the last step (e.g. reboot when updating). Currently, we only check whether we can get from point A (e.g. start updating) to point B (e.g. printed reboot after successful update).

Planned changes

To improve end-to-end (E2E) testing, we plan to create a dedicated profile for each workflow and platform. This profile will store an ordered list of all Hardware Abstraction Layer (HAL) calls made during the actual execution of the workflow. When running tests on QEMU, we will compare the HAL call sequence generated during the test with the reference profile. The test will pass only if both sequences are identical.

This change should be fairly trivial, we only have to modify tool_wrapper to make sure it appends command we are currently trying to run to a specific file.

Specifics

1. Make tool_wrapper function append <command> <command_args> <return_code> to /tmp/logs/profile.
   • To make debugging easier create /tmp/logs/debug_profile which contains lines in format: <script_file>:<script_line> <command> <command_args> <return_code>
2. Generate profile for single platform and workflow
3. Modify E2E tests
   • Add generated profile to open-source-firmware-validation/dts/profiles/
   • Modify test for this platform and workflow and make it compare real profile with one generated after workflow finishes

Profile generated on real hardware can be accepted if workflow used to generate it was successful e.g. there were no DTS errors, platform works after rebooting, workflow assumptions are fulfilled e.g. firmware version changed to expected one.

Pros:

• Little changes needed to implement

Cons:

• Generating profiles for all workflows and platforms might take a while
• Changes in DTS might force us to regenerate all profiles