Files
Filip Gołaś a1a542fd5f Osfv stable release improvements (#1254)
* osfv_stability_{run,reports}.py: Add

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

osfv_stability_run: add release rules file

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

osfv_stability_reports: add run count

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

osfv_stability_reports: Fix: account for parallelism in total run time

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

osfv_stability_run.py: accept env vars

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

osfv_stability_reports.py: save logs to dirs depending on date and revision

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

osfv_stability_run.py: Save results in paths per run date and rev

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

osfv_stability_reports.py: Parse date, revision, branch and support json output

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

osfv_stability_*.py: Allow configurable LOGS_DIR using env vars

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

osfv_stability_reports.py: Handle broken robot xmls

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

osfv_stability_reports.py: Unpack merged results

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

osfv_stability_reports.py: Add suites to skip config

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Deploy Uefi Shell: Ensure it doesnt break the bootorder

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* os-specific-tasks/201.yml: Remove uefi shell installation

it breaks bootorder and tests deploy it anyway

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Login To OS: Rename to Boot And Login To OS

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Login To Windows: Rename to Boot And Login To Windows

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Login To Windows: Add keyword

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Logint To Booted OS: Add keyword

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Serial User Login Linux: Remove dead code

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Login To Linux: Remove retrials of Login To Linux Via SSH

Login To Linux Via SSH already retries all its code,
so this is redundant and multiplies the wait time

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* fix login ssh timeouts

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Login To Windows Via SSH: Reduce login timeout from 1h@20 tries to 5m@5 tries

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* keywords.robot: Implement login to OS recovery

When the boot order unexpectedly changes and testing via SSH,
the whole regression will start failing.

With this fallback, after we fail to log in to an OS, there
will at least be the hope of recovering the regression
eventually.

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Ensure Custom Bootentry: fix indexing empty list

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* BPS010: Deploy UEFI Shell: Add

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* linux-packages-playbook.yaml: Add systemwide update task

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Login To Windows Via SSH: Fail if login fails

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* TMP TEST: reduce trials for linux login

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* basic-platform-setup.robot: Normalize Boot&Login keyword variants usage in suite

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Set Nextboot Bootentry: Move implementation to lib/custom_boot_entries

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Load OS Credentials: Create keyword for flaky repeated import

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Recover Broken Bootorder: resume login after recover

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* options-lib_dcu.robot: Boot System... Do not assume default_boot is the target

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* lib/flash.robot: Fix missing 'x' in Wait Unil KW Succeeds

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Login To* kwds: Add configurable retries amount for ssh variants

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* capsule-update.robot: Flash rc0 before tests, update instructions

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Revert "TMP TEST: reduce trials for linux login"

This reverts commit d41693da2a3a600beb7a4a26d95f6e7f90a2430a.

* lib/dcu.robot: Remove redundant sleep after flashing

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* lib/network.robot: Send File To DUT: check hashes, rm before send

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* platform-configs: Update INTEL_CBNT_BOOTGUARD_FUSING_SUPPORT state

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* options-lib_dcu.robot: Remove redundant sleep in Set UEFI Option

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* keywords.robot: Load OS Credentials: Remove redundant OS login

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* lib/flash.robot: Flash ME in BTG supported devices

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* lib/flash.robot: Increase timeouts a bit to compensate for large RAM training

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* keywords.robot: Fix Login To Linux Via SSH not doing retries

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* capsule-update.robot: Rework suite to work without additional manual setup steps

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* novacustom-nuc_box.robot: Remove capsule update support: No capsules published

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* flash.robot: Flash FD using internal programmer to avoid FD mismatch bricks

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* regression-scope/configs/devices: Add capsule update env vars

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* capsule-update.robot: Remove manual step from Flash Firmware If Not QEMU

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* keyword & options-lib_dcu: Remove redundant Login To Windows keyword variant

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* capsule-update.robot: Remove redundant serial console test flow

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* release_tests_suite_list*: Add Basic Platform Setup to release tests

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* capsule-update.robot: Restore custom logo checking

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* requirements.txt: Bump osfv-scripts to fix time import

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* lib/dcu.robot: Prevent temp file collisions with uuids

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* msi-pro-690-a-wifi-ddr4_00039.json: change fw version to support custom logo

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* lib/flash.robot: Increase response timeout after flash to 300s

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* keywords.robot: Fix Login To Linux With Root Priviledges doing nothing on serial

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* keywords.robot: Login To Booted OS: Only attempt recover when via pure SSH

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* capsule-update.robot: Restore FUM dialog handling when serial console supported

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* capsule-update.robot: Adjust for execution on both serial and ssh

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* capsule_update_tests.sh: Remove only previous run files of the same name

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* lib/network.robot: Rm target path before moving

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Revert "lib/network.robot: Rm target path before moving"

This reverts commit 3bca42177f528f17eba13095dff0189a932fce8d.

* lib/network.robot: Add sync on the end to prevent interrupting the file transfer

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* capsule-update.robot: Make FUM screen optional, it depends on FW version

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* capsule-update.robot: Fix mismatch in waiting for tianocore string

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* ci//pcengines-apu3_00252.json: Fix FW file apu variant

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* platform-configs/protectli: Reduce repeated DMIDECODE values

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* protectli-vp6650_00002.json: Update sonoff ip

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* keys-and-keywords/ubuntu-keywords.robot: Fix applying log level immediately

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* ci//pcengines-apu3_00252.json: Update device IP address

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* menus.robot: Add warning if OS not found and will try to boot the default disk

Otherwise leads to cryptic fails failing because some disk name was not
found, while the real culprit is the OS not being installed.

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* linux-packages-playbook.yaml: accept gsettings modifications to fail

will fail if Gnome is not installed (server OSes)

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* BPS006: Add workaround for sudo-rs which is incompatible with ansible

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* release_tests_devices.txt: Fix protectli vp6650 filename

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* novacustom-nu_box-155H_00254.json: Update device IP

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* msi-z690-common.robot: Change keyboard model to fit hardware matrix

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* uefi-shell.robot: Use library keywords to perform test

Otherwise it was failing on MSI z690.
It's always better to use common and working
keywords instead of reinventing the wheel.

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* osfv-test-data: Bump version

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* tpm-support.robot: Add missing skips for serial-only tests

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* verified-boot.robot: Run DCU locally to keep DUT from downloading containers

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* platform-configs/pcengines: Update dmidecode values

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* platform-configs/novacustom: Update dmidecode values

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* lib/dcu.robot: Put temp tiles in /tmp to prevent littering if cleanup fails

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* BPS006: Put temp tiles in /tmp to prevent littering if cleanup fails

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* lib/flash.robot: Read Firmware include ME and IFD

We might assume the read firmware contains some IFD
or ME later, not reading them would result in a brick
and doesn't increase the read time that much.

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* SSH Boot System: Always reinit connection to fix 'No open connections' errors

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* cpu-performance.robot: Limit test runs to 3

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* manual_cpu_performance.robot: Limit runs to 3

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* novacustom-v540tnd_00266.json: Update device ip

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* novacustom-nuc_box-155H_00254.json: Update device IP

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Login To Booted OS: Fix infinite recovery loop

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* release_tests_devices.txt: Remove nucbox

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* keywords.robot: Fix ssh_retries being passed as timeout

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* SSH Login keywords: Limit retries to 5m instead of fixed try count

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* release_tests_suite_list.txt: Remove duplicated __init__s

robot crashes otherwise

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* lib/prepare-platform.resource: Unify redundant Prepare Platform definitions

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* util/prepare-platform.robot: Add util test for running Prepare Platform

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* release_tests_suite_list.txt: Use util/prepare-platform instead of module inits

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* dts-lib.robot: Add semiauto test skip for flashing kwd

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* develop_pr_auto_regression.py: Only compare revs if files list not overridden

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* osfv_stability_run.py: Fix logs_base undefined on custom logs dir

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* scripts/ci/regression-scope: Preserve suites order

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* develop_pr_auto_regression.py: Fix orphan robot processes

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* release_tests_rules.json: Add automated tag filter

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* osfv_stability_reports.py: Add per-suite cache

For suites which take a long long time to process,
to not lose the time spent processing it if the
processing is interrupted

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* osfv_stability_reports.py: Differentiate run dates in caches and reports

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* release_tests_devices.tst: Revert using temporary desk setup devices

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* scripts/regression.sh: Fix scope change

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* cpu-performance: Fix FORCE_TIMES_TO_RUN envvar set command

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* Login To Booted OS: Fix continuing on fail to login

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* test_cases.json: fix spelling and such (#1244)

* test_cases.json: fix spelling and such

Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>

test_cases.json: fix

Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>

* test_cases.json: fixname

Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>

* test_cases.json:remaining tests

Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>

* test_cases.json: add WLE

Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>

* check_tests_consistency.py: script lists diffs in test_cases.json and modules

Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>

* dasharo-compatibility/*:fix

Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>

* test_cases.json: fix

Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>

---------

Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>

* test_cases.json: Add missing tests and fix changed_to's (#1246)

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>

* test_cases.json:fix net spd test (#1247)

Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>

* docs/novacustom.md: document v0.9.0 downgrade bug

Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>

* docs/novacustom.md: add link to ec downgrade procedure

Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>

---------

Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
Co-authored-by: Wiktor Mówiński <wiktor.mowinski@3mdeb.com>
Co-authored-by: Michał Kopeć <michal.kopec@3mdeb.com>
2026-03-10 08:53:21 +01:00

89 lines
3.8 KiB
Plaintext

*** Settings ***
Library Collections
Library OperatingSystem
Library Process
Library String
Library Telnet timeout=20 seconds connection_timeout=120 seconds
Library SSHLibrary timeout=90 seconds
Library RequestsLibrary
# TODO: maybe have a single file to include if we need to include the same
# stuff in all test cases
Resource ../variables.robot
Resource ../keywords.robot
Resource ../keys.robot
# TODO:
# - document which setup/teardown keywords to use and what are they doing
# - go threough them and make sure they are doing what the name suggest (not
# exactly the case right now)
Suite Setup Run Keywords
... Prepare Test Suite
... AND
... Skip If not ${NVIDIA_GRAPHICS_CARD_SUPPORT} Nvidia GPU tests not supported
Suite Teardown Run Keyword
... Log Out And Close Connection
Default Tags automated
*** Test Cases ***
NVI001.201 NVIDIA Graphics detect (Ubuntu)
[Documentation] Check whether the NVIDIA graphics card is initialized
... correctly and can be detected by the Linux OS.
... Previous IDs: NVI001.001
Skip If not ${TESTS_IN_UBUNTU_SUPPORT} NVI001.201 not supported
Skip If "${ENV_ID_UBUNTU}" not in ${TESTED_LINUX_DISTROS} NVI001.201 not supported
Power On
Boot System Or From Connected Disk ${ENV_ID_UBUNTU}
Login To Linux
Switch To Root User
${out}= Execute Linux Command lspci | grep -i nvidia | cat
Should Contain Any ${out} 3D controller: NVIDIA Corporation VGA compatible controller: NVIDIA Corporation
Exit From Root User
NVI002.201 NVIDIA Graphics power management (Ubuntu)
[Documentation] Check whether the NVIDIA graphics power management is
... functional and the card powers on only while it's used.
... Previous IDs: NVI002.001
Skip If not ${TESTS_IN_UBUNTU_SUPPORT} NVI002.201 not supported
Skip If "${ENV_ID_UBUNTU}" not in ${TESTED_LINUX_DISTROS} NVI002.201 not supported
Power On
Boot System Or From Connected Disk ${ENV_ID_UBUNTU}
Login To Linux
Switch To Root User
Check NVIDIA Power Management In Linux
Exit From Root User
NVI001.202 NVIDIA Graphics detect (Fedora)
[Documentation] Check whether the NVIDIA graphics card is initialized
... correctly and can be detected by the Linux OS.
Skip If "${ENV_ID_FEDORA}" not in ${TESTED_LINUX_DISTROS} NVI001.202 not supported
Power On
Boot System Or From Connected Disk ${ENV_ID_FEDORA}
Login To Linux
Switch To Root User
${out}= Execute Linux Command lspci | grep -i nvidia | cat
Should Contain Any ${out} 3D controller: NVIDIA Corporation VGA compatible controller: NVIDIA Corporation
Exit From Root User
NVI002.202 NVIDIA Graphics power management (Fedora)
[Documentation] Check whether the NVIDIA graphics power management is
... functional and the card powers on only while it's used.
Skip If "${ENV_ID_FEDORA}" not in ${TESTED_LINUX_DISTROS} NVI001.202 not supported
Power On
Boot System Or From Connected Disk ${ENV_ID_FEDORA}
Login To Linux
Switch To Root User
Check NVIDIA Power Management In Linux
Exit From Root User
NVI001.301 NVIDIA Graphics detect (Windows)
[Documentation] Check whether the NVIDIA graphics card is initialized
... correctly and can be detected by the Windows 11.
... Previous IDs: NVI001.002
Skip If not ${TESTS_IN_WINDOWS_SUPPORT} NVI001.301 not supported
Power On
Boot And Login To Windows
${out}= Get Video Controllers Windows
Should Contain ${out} NVIDIA GeForce