mirror of
https://github.com/Dasharo/meta-dts.git
synced 2026-06-13 10:16:12 -07:00
docs: testing.md: add planned changes to DTS tests
Signed-off-by: Michał Iwanicki <michal.iwanicki@3mdeb.com>
This commit is contained in:
@@ -0,0 +1,50 @@
|
||||
# Testing
|
||||
|
||||
Current E2E tests are defined in [OSFV
|
||||
repo](https://github.com/Dasharo/open-source-firmware-validation/blob/develop/dts/dts-e2e.robot)
|
||||
|
||||
They work by running through different workflows from the moment user sees UI to
|
||||
the last step (e.g. reboot when updating). Currently, we only check whether we
|
||||
can get from point A (e.g. start updating) to point B (e.g. printed reboot after
|
||||
successful update).
|
||||
|
||||
## Planned changes
|
||||
|
||||
To improve end-to-end (E2E) testing, we plan to create a dedicated profile for
|
||||
each workflow and platform. This profile will store an ordered list of all
|
||||
Hardware Abstraction Layer (HAL) calls made during the actual execution of the
|
||||
workflow. When running tests on QEMU, we will compare the HAL call sequence
|
||||
generated during the test with the reference profile. The test will pass only if
|
||||
both sequences are identical.
|
||||
|
||||
This change should be fairly trivial, we only have to modify
|
||||
[tool_wrapper](https://github.com/Dasharo/dts-scripts/blob/bcf7871d4e334a47e7f8722793db5537db67a569/include/hal/dts-hal.sh#L65)
|
||||
to make sure it appends command we are currently trying to run to a specific
|
||||
file.
|
||||
|
||||
**Specifics**
|
||||
|
||||
1. Make `tool_wrapper` function append `<command> <command_args> <return_code>`
|
||||
to `/tmp/logs/profile`.
|
||||
- To make debugging easier create `/tmp/logs/debug_profile` which contains
|
||||
lines in format:
|
||||
`<script_file>:<script_line> <command> <command_args> <return_code>`
|
||||
2. Generate profile for single platform and workflow
|
||||
3. Modify E2E tests
|
||||
- Add generated profile to `open-source-firmware-validation/dts/profiles/`
|
||||
- Modify test for this platform and workflow and make it compare real
|
||||
profile with one generated after workflow finishes
|
||||
|
||||
Profile generated on real hardware can be accepted if workflow used to generate
|
||||
it was successful e.g. there were no DTS errors, platform works after rebooting,
|
||||
workflow assumptions are fulfilled e.g. firmware version changed to expected
|
||||
one.
|
||||
|
||||
**Pros:**
|
||||
|
||||
* Little changes needed to implement
|
||||
|
||||
**Cons:**
|
||||
|
||||
* Generating profiles for all workflows and platforms might take a while
|
||||
* Changes in DTS might force us to regenerate all profiles
|
||||
Reference in New Issue
Block a user