mirror of
https://github.com/Dasharo/meta-dts.git
synced 2026-03-06 14:51:12 -08:00
51 lines
2.1 KiB
Markdown
51 lines
2.1 KiB
Markdown
|
# Testing
|
||
|
|
|
||
|
|
Current E2E tests are defined in [OSFV
|
||
|
|
repo](https://github.com/Dasharo/open-source-firmware-validation/blob/develop/dts/dts-e2e.robot)
|
||
|
|
|
||
|
|
They work by running through different workflows from the moment user sees UI to
|
||
|
|
the last step (e.g. reboot when updating). Currently, we only check whether we
|
||
|
|
can get from point A (e.g. start updating) to point B (e.g. printed reboot after
|
||
|
|
successful update).
|
||
|
|
|
||
|
|
## Planned changes
|
||
|
|
|
||
|
|
To improve end-to-end (E2E) testing, we plan to create a dedicated profile for
|
||
|
|
each workflow and platform. This profile will store an ordered list of all
|
||
|
|
Hardware Abstraction Layer (HAL) calls made during the actual execution of the
|
||
|
|
workflow. When running tests on QEMU, we will compare the HAL call sequence
|
||
|
|
generated during the test with the reference profile. The test will pass only if
|
||
|
|
both sequences are identical.
|
||
|
|
|
||
|
|
This change should be fairly trivial, we only have to modify
|
||
|
|
[tool_wrapper](https://github.com/Dasharo/dts-scripts/blob/bcf7871d4e334a47e7f8722793db5537db67a569/include/hal/dts-hal.sh#L65)
|
||
|
|
to make sure it appends command we are currently trying to run to a specific
|
||
|
|
file.
|
||
|
|
|
||
|
|
**Specifics**
|
||
|
|
|
||
|
|
1. Make `tool_wrapper` function append `<command> <command_args> <return_code>`
|
||
|
|
to `/tmp/logs/profile`.
|
||
|
|
- To make debugging easier create `/tmp/logs/debug_profile` which contains
|
||
|
|
lines in format:
|
||
|
|
`<script_file>:<script_line> <command> <command_args> <return_code>`
|
||
|
|
2. Generate profile for single platform and workflow
|
||
|
|
3. Modify E2E tests
|
||
|
|
- Add generated profile to `open-source-firmware-validation/dts/profiles/`
|
||
|
|
- Modify test for this platform and workflow and make it compare real
|
||
|
|
profile with one generated after workflow finishes
|
||
|
|
|
||
|
|
Profile generated on real hardware can be accepted if workflow used to generate
|
||
|
|
it was successful e.g. there were no DTS errors, platform works after rebooting,
|
||
|
|
workflow assumptions are fulfilled e.g. firmware version changed to expected
|
||
|
|
one.
|
||
|
|
|
||
|
|
**Pros:**
|
||
|
|
|
||
|
|
* Little changes needed to implement
|
||
|
|
|
||
|
|
**Cons:**
|
||
|
|
|
||
|
|
* Generating profiles for all workflows and platforms might take a while
|
||
|
|
* Changes in DTS might force us to regenerate all profiles
|