* osfv_stability_{run,reports}.py: Add
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_run: add release rules file
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports: add run count
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports: Fix: account for parallelism in total run time
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_run.py: accept env vars
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports.py: save logs to dirs depending on date and revision
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_run.py: Save results in paths per run date and rev
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports.py: Parse date, revision, branch and support json output
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_*.py: Allow configurable LOGS_DIR using env vars
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports.py: Handle broken robot xmls
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports.py: Unpack merged results
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports.py: Add suites to skip config
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Deploy Uefi Shell: Ensure it doesnt break the bootorder
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* os-specific-tasks/201.yml: Remove uefi shell installation
it breaks bootorder and tests deploy it anyway
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Login To OS: Rename to Boot And Login To OS
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Login To Windows: Rename to Boot And Login To Windows
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Login To Windows: Add keyword
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Logint To Booted OS: Add keyword
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Serial User Login Linux: Remove dead code
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Login To Linux: Remove retrials of Login To Linux Via SSH
Login To Linux Via SSH already retries all its code,
so this is redundant and multiplies the wait time
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* fix login ssh timeouts
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Login To Windows Via SSH: Reduce login timeout from 1h@20 tries to 5m@5 tries
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* keywords.robot: Implement login to OS recovery
When the boot order unexpectedly changes and testing via SSH,
the whole regression will start failing.
With this fallback, after we fail to log in to an OS, there
will at least be the hope of recovering the regression
eventually.
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Ensure Custom Bootentry: fix indexing empty list
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* BPS010: Deploy UEFI Shell: Add
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* linux-packages-playbook.yaml: Add systemwide update task
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Login To Windows Via SSH: Fail if login fails
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* TMP TEST: reduce trials for linux login
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* basic-platform-setup.robot: Normalize Boot&Login keyword variants usage in suite
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Set Nextboot Bootentry: Move implementation to lib/custom_boot_entries
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Load OS Credentials: Create keyword for flaky repeated import
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Recover Broken Bootorder: resume login after recover
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* options-lib_dcu.robot: Boot System... Do not assume default_boot is the target
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/flash.robot: Fix missing 'x' in Wait Unil KW Succeeds
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Login To* kwds: Add configurable retries amount for ssh variants
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* capsule-update.robot: Flash rc0 before tests, update instructions
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Revert "TMP TEST: reduce trials for linux login"
This reverts commit d41693da2a3a600beb7a4a26d95f6e7f90a2430a.
* lib/dcu.robot: Remove redundant sleep after flashing
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/network.robot: Send File To DUT: check hashes, rm before send
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* platform-configs: Update INTEL_CBNT_BOOTGUARD_FUSING_SUPPORT state
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* options-lib_dcu.robot: Remove redundant sleep in Set UEFI Option
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* keywords.robot: Load OS Credentials: Remove redundant OS login
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/flash.robot: Flash ME in BTG supported devices
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/flash.robot: Increase timeouts a bit to compensate for large RAM training
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* keywords.robot: Fix Login To Linux Via SSH not doing retries
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* capsule-update.robot: Rework suite to work without additional manual setup steps
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* novacustom-nuc_box.robot: Remove capsule update support: No capsules published
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* flash.robot: Flash FD using internal programmer to avoid FD mismatch bricks
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* regression-scope/configs/devices: Add capsule update env vars
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* capsule-update.robot: Remove manual step from Flash Firmware If Not QEMU
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* keyword & options-lib_dcu: Remove redundant Login To Windows keyword variant
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* capsule-update.robot: Remove redundant serial console test flow
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* release_tests_suite_list*: Add Basic Platform Setup to release tests
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* capsule-update.robot: Restore custom logo checking
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* requirements.txt: Bump osfv-scripts to fix time import
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/dcu.robot: Prevent temp file collisions with uuids
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* msi-pro-690-a-wifi-ddr4_00039.json: change fw version to support custom logo
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/flash.robot: Increase response timeout after flash to 300s
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* keywords.robot: Fix Login To Linux With Root Priviledges doing nothing on serial
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* keywords.robot: Login To Booted OS: Only attempt recover when via pure SSH
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* capsule-update.robot: Restore FUM dialog handling when serial console supported
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* capsule-update.robot: Adjust for execution on both serial and ssh
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* capsule_update_tests.sh: Remove only previous run files of the same name
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/network.robot: Rm target path before moving
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Revert "lib/network.robot: Rm target path before moving"
This reverts commit 3bca42177f528f17eba13095dff0189a932fce8d.
* lib/network.robot: Add sync on the end to prevent interrupting the file transfer
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* capsule-update.robot: Make FUM screen optional, it depends on FW version
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* capsule-update.robot: Fix mismatch in waiting for tianocore string
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* ci//pcengines-apu3_00252.json: Fix FW file apu variant
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* platform-configs/protectli: Reduce repeated DMIDECODE values
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* protectli-vp6650_00002.json: Update sonoff ip
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* keys-and-keywords/ubuntu-keywords.robot: Fix applying log level immediately
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* ci//pcengines-apu3_00252.json: Update device IP address
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* menus.robot: Add warning if OS not found and will try to boot the default disk
Otherwise leads to cryptic fails failing because some disk name was not
found, while the real culprit is the OS not being installed.
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* linux-packages-playbook.yaml: accept gsettings modifications to fail
will fail if Gnome is not installed (server OSes)
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* BPS006: Add workaround for sudo-rs which is incompatible with ansible
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* release_tests_devices.txt: Fix protectli vp6650 filename
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* novacustom-nu_box-155H_00254.json: Update device IP
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* msi-z690-common.robot: Change keyboard model to fit hardware matrix
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* uefi-shell.robot: Use library keywords to perform test
Otherwise it was failing on MSI z690.
It's always better to use common and working
keywords instead of reinventing the wheel.
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* osfv-test-data: Bump version
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* tpm-support.robot: Add missing skips for serial-only tests
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* verified-boot.robot: Run DCU locally to keep DUT from downloading containers
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* platform-configs/pcengines: Update dmidecode values
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* platform-configs/novacustom: Update dmidecode values
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/dcu.robot: Put temp tiles in /tmp to prevent littering if cleanup fails
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* BPS006: Put temp tiles in /tmp to prevent littering if cleanup fails
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/flash.robot: Read Firmware include ME and IFD
We might assume the read firmware contains some IFD
or ME later, not reading them would result in a brick
and doesn't increase the read time that much.
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* SSH Boot System: Always reinit connection to fix 'No open connections' errors
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* cpu-performance.robot: Limit test runs to 3
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* manual_cpu_performance.robot: Limit runs to 3
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* novacustom-v540tnd_00266.json: Update device ip
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* novacustom-nuc_box-155H_00254.json: Update device IP
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Login To Booted OS: Fix infinite recovery loop
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* release_tests_devices.txt: Remove nucbox
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* keywords.robot: Fix ssh_retries being passed as timeout
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* SSH Login keywords: Limit retries to 5m instead of fixed try count
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* release_tests_suite_list.txt: Remove duplicated __init__s
robot crashes otherwise
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/prepare-platform.resource: Unify redundant Prepare Platform definitions
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* util/prepare-platform.robot: Add util test for running Prepare Platform
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* release_tests_suite_list.txt: Use util/prepare-platform instead of module inits
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* dts-lib.robot: Add semiauto test skip for flashing kwd
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* develop_pr_auto_regression.py: Only compare revs if files list not overridden
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* osfv_stability_run.py: Fix logs_base undefined on custom logs dir
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* scripts/ci/regression-scope: Preserve suites order
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* develop_pr_auto_regression.py: Fix orphan robot processes
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* release_tests_rules.json: Add automated tag filter
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* osfv_stability_reports.py: Add per-suite cache
For suites which take a long long time to process,
to not lose the time spent processing it if the
processing is interrupted
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* osfv_stability_reports.py: Differentiate run dates in caches and reports
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* release_tests_devices.tst: Revert using temporary desk setup devices
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* scripts/regression.sh: Fix scope change
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* cpu-performance: Fix FORCE_TIMES_TO_RUN envvar set command
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Login To Booted OS: Fix continuing on fail to login
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* test_cases.json: fix spelling and such (#1244)
* test_cases.json: fix spelling and such
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
test_cases.json: fix
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
* test_cases.json: fixname
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
* test_cases.json:remaining tests
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
* test_cases.json: add WLE
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
* check_tests_consistency.py: script lists diffs in test_cases.json and modules
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
* dasharo-compatibility/*:fix
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
* test_cases.json: fix
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
---------
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
* test_cases.json: Add missing tests and fix changed_to's (#1246)
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* test_cases.json:fix net spd test (#1247)
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
* docs/novacustom.md: document v0.9.0 downgrade bug
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
* docs/novacustom.md: add link to ec downgrade procedure
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
---------
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
Signed-off-by: Wiktor Mowinski <wiktor.mowinski@3mdeb.com>
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
Co-authored-by: Wiktor Mówiński <wiktor.mowinski@3mdeb.com>
Co-authored-by: Michał Kopeć <michal.kopec@3mdeb.com>
* util/bootentries-fix.robot: Add PoC for adding bootentries
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* options-lib_dcu.robot: Fix faulty comparison of bootid to empty
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/flash.robot: Add bootorder replacement when flashing fw and tesing via ssh
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Execute Reboot Command: Use BOOTED_OS_ID to determine reboot type
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* novacustom-v540tu.robot: Resture Ubuntu as default boot os
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/custom_bootentries.robot: Add library for managing bootorder
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/flash.robot: Migrate smmstore when flashing without access to serial
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* util/bootentries-fix.robot: Clean up the suite
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* util/bootentries-fix.robot: Move to self-tests/bootentries-persistence.robot
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* util/basic-platform-setup: Add BPS005.002 for creating defaultboot entires
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* BPS009: Add suite that adds DEFAULT_BOOT_OS_ID entry
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* bootentries-persistence.robot: Make it depend on basic platform setup done
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/custom_bootentries.robot: Make sure esp are skipped
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* environment-test-ids.py: Extend Windows bootmenu name
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* BPS009: make sure windows bootentry is created
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Execute Shutdown Command: Skip if no POWER_CTRL
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
---------
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* requirements.txt: bump robotframework and telnetlib to add fuzzysearch
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/terminal.robot: Use Telnet.Read Until Fuzzy
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/bios/menus.robot: Add fuzzy search to menu parsing
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* default.robot: Set TELNET_FUZZY_MAX_ERRORS default to 0
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* novacustom-v540tu.robot: Set TELNET_FUZZY_MAX_ERRORS to 1
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* self-tests/serial_stability.robot: Add
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* develop-pr-regression.yml: Ensure clean python environment
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* develop-pr-regression.yml: Ensure clean osfv repo
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* requirements.txt: bump robotframework and telnetlib to add fuzzysearch
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* requirements.txt: bump fuzzysearch
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* platform-configs: Use max_insertions and max_deletions in fuzzy telnet
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Parse Menu Snapshot Into Construction: workaround for random chars
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* menus.robot: Add first_line parameter to menu parsing kwds
Helps to remove unwanted contents from the parsed menu in case
of connection errors
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* pre-commit fixes
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* bump requirements
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* novacustom-v560tnd.robot: Remove power led support
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* verified-boot.robot: Add laptops without battery workaround
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Get Boot Menu Construction: Look for longer string
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* pre-commit fixes
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* fuzzysearch: change max_errors to max_substitutions
setting max_errors, max_insertions and max_deletions, but
not max_substitutions allowed for an edge case to trigger
where the fuzzysearch library was crashing.
Changing the interface like that should prevent unhandled
situations
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* requirements.txt: Use Dasharo/fuzzysearch fork
* pre-commit fix
---------
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
Maciej Pijanowski
Jakub Redmerski
Sebastian Czapla
Jan Prusinowski
Cezary Wieczorkowski