diff --git a/docs/testing.md b/docs/testing.md new file mode 100644 index 0000000..d57e646 --- /dev/null +++ b/docs/testing.md @@ -0,0 +1,50 @@ +# Testing + +Current E2E tests are defined in [OSFV +repo](https://github.com/Dasharo/open-source-firmware-validation/blob/develop/dts/dts-e2e.robot) + +They work by running through different workflows from the moment user sees UI to +the last step (e.g. reboot when updating). Currently, we only check whether we +can get from point A (e.g. start updating) to point B (e.g. printed reboot after +successful update). + +## Planned changes + +To improve end-to-end (E2E) testing, we plan to create a dedicated profile for +each workflow and platform. This profile will store an ordered list of all +Hardware Abstraction Layer (HAL) calls made during the actual execution of the +workflow. When running tests on QEMU, we will compare the HAL call sequence +generated during the test with the reference profile. The test will pass only if +both sequences are identical. + +This change should be fairly trivial, we only have to modify +[tool_wrapper](https://github.com/Dasharo/dts-scripts/blob/bcf7871d4e334a47e7f8722793db5537db67a569/include/hal/dts-hal.sh#L65) +to make sure it appends command we are currently trying to run to a specific +file. + +**Specifics** + +1. Make `tool_wrapper` function append ` ` + to `/tmp/logs/profile`. + - To make debugging easier create `/tmp/logs/debug_profile` which contains + lines in format: + `: ` +2. Generate profile for single platform and workflow +3. Modify E2E tests + - Add generated profile to `open-source-firmware-validation/dts/profiles/` + - Modify test for this platform and workflow and make it compare real + profile with one generated after workflow finishes + +Profile generated on real hardware can be accepted if workflow used to generate +it was successful e.g. there were no DTS errors, platform works after rebooting, +workflow assumptions are fulfilled e.g. firmware version changed to expected +one. + +**Pros:** + +* Little changes needed to implement + +**Cons:** + +* Generating profiles for all workflows and platforms might take a while +* Changes in DTS might force us to regenerate all profiles