Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
regresion-scope/configs: Add DEVICE_IP
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
regression-scope/configs/devices: Add FW files paths
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
protectli-vp66t0_00004.json: Add sonoff_ip for a good measure
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
novacustom-v540tu_00191.json: Update device ip
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
release_tests_devices.csv: Add novacustom MTL instead of ADL
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
regression-scope/configs/devices: csv->txt
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_run: add release rules file
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports: add run count
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports: Fix: account for parallelism in total run time
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_run.py: accept env vars
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports.py: save logs to dirs depending on date and revision
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_run.py: Save results in paths per run date and rev
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports.py: Parse date, revision, branch and support json output
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_*.py: Allow configurable LOGS_DIR using env vars
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports.py: Handle broken robot xmls
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports.py: Unpack merged results
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
osfv_stability_reports.py: Add suites to skip config
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
scripts/ci/develop_pr_auto_regression: Respect snipeit
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
develop_pr_auto_regression.py: Express max snipeit timeout in seconds not tries
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
develop_pr_auto_regression.py: Cap for exponential backoff + increase to ~5h
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
bash supervisor fixed
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
develop_pr_auto_regression.py: filter out comments from devices file
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
develop_pr_auto_regression.py: remove forced BPS, make it just a test in scope
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
develop_pr_auto_regression.py: Accept max snipeit timeout as env variable
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
develop_pr_auto_regression.py: if only single job was run, return its RC
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* CPT_CPF_STB/201.robot: Add missing test variants
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* CPT_CPF_STB: Fix measurement loop 0s iteration timer bug
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* CPT_CPF_STB/common.robot: Do not gather data for skipped tests
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* test_cases.json: Add missing tests from robot files
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* test_id_checking_lib.py: Only compare test IDs when looking for missing tests
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* .pre-commit-config.yaml: Add missing dependencies of broken robocop hooks
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
---------
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* .gitignore: Add _envvars, _roms directories
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* keywords.robot: Allow for the DUT ssh key to change when trying to connect via SSH in Login To Linux via SSH
* lib/tpm.robot: Check all PCRs before failing in Validate PCRs Against Event Log
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* miniPCIe-slot-verification: Skip LTE_CARD tests if card is not defined
* Stress Test Windows: use Execute Command In Terminal
Otherwise there is no timeout for the keyword and the
tests could hang forever
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* CBMEM001: Add semiauto implementation
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Set Nextboot Bootentry: compare strings as variables to fix utf decode errors
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* novacustom-mtl.robot: Add USB model t oconfig
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* novacustom-v560tne: v1.0.0-rc10 platform configs changes
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
novacustom-v560tne: Add windows support
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
v560tne: temp RTE serial tests
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
v560tne: dmidecode update version
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
v560tne: Restore fedora support flag
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
v560tne: Add USB drives model names for detection tests
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
v560tne: Add DOCKING_STATION_USB_C_CHARGING_SUPPORT
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
v560tne: add qubes support
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
v540tnd: Restore ubuntu default boot os
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
REMOVE platform-configs: restore SSH
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* acpi-driver.robot: Fix typo in deb package name
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* minipcie-slot-verification: Replace hardcoded wifi name with the configs
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Switch To Root User: Add workaround for passwordless root
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* os-specific-tasks/203.yml: Add qubesos ansible config
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* fwupd-support.robot: automate qubes
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* cpu-performance.robot: Add missing phoronix batch mode setup to automated suite
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Login To Windows Via SSH: Allow the key to change across retries
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Get Index Of Matching Option In Menu: do not use fuzzy match
It works a bit differently - the * is not a special character in
fuzzy search
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Execute Shutdown Command: Add linux systemd support
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
Execute Shutdown Command: Add some workaround for no power_led
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Measure Warmboot Time: add workaround for no RTC boot
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* platform-configs: Add RTC_BOOT_SUPPORT
https://github.com/Dasharo/open-source-firmware-validation/issues/1068
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* CBMEM suite: dynamic semiauto skips for no RTC boot support
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* network-interface-after-suspend: Add workaroud for no RTC boot
https://github.com/Dasharo/open-source-firmware-validation/issues/1068
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* linux-packages-playbook.yaml: Remove masking systemd suspend
It makes it impossible to suspend the device in conventional
ways like using the UI or via suspend key combination.
It has more cons that pros
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* cbnt.robot: Fix invalid ibg menu parsing
Due to confusing line wrapping in the menu the
state of the options following to the target
were being parsed
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* SBO: Fix test ids ENV_ID
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* fast-boot.robot: Increase wait time for systemd-analyze
50s was not always enough for bootup to finish.
150s should be better.
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* gpu-performance.robot: stability fixes
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
gpu-performance.robot: Fix string comparison error and x11 invalid display
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
gpu-performance.robot: Detect not running x11
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
gpu-performance.robot: Note that X11 is required
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
gpu-performance.robot: fix check for installed unigine-super
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
gpu-performance.robot: Add comment about display numbering bug
https://github.com/Dasharo/open-source-firmware-validation/issues/1215
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
lib/performance/common: Setup Phoronix: Upload results to openbenchmarking
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
disk-io.robot: Verify correct disk model before checking performance
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
disk-io.robot: move sleep to after skips to reduce unnecessary idle time
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* phoronix-test-suite tests: Cap run count to 3
Using: FORCE_TIMES_TO_RUN env variable
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/docs.robot: Increase time for docking station to init
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* util/fwupd-support-lvfs: stylistic changes
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
util/fwupd-support-lvfs: Remove unused variable
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
util/fwupd-support-lvfs.robot: Add missing newline in fwupd config
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* cbnt.robot: Remove newline characters from cbmem regex
The newline characters are not matched by this keyword.
Its matched line-by-line so newline characters
never appear
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/utc.robot: stability fixes
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
lib/utc.robot: reboot tests add delay to prevent immediate re-login
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
lib/utc: hotplug tests: reorder disconnect/connect to end with connected
Otherwise the docking station would be left disconnected after
the tests causing further automated tests to possibly fail
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
UTC: Add dynamic skips for laptop semiautos
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* fwupd-support: Add --assume-yes to get-devices
First time its executed after a fwupd update it will prompt to send the
results to the vendor. It was not handled before.
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* Execute Reboot Command: Add delay for no-power-ctrl devices
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* EXTERNAL_HEADSET: Change to list of acceptable values
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* BOOTED_OS_ID: Prevent resetting on new suite
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* warm/coldboot tests: Add dynamic semiauto skips if not supported
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* lib/robot.sh: Run all suites at once and split results later
We've made it so that the test suites are run separately so that
the logs can be organized neatly before.
Turns out that makes the state between suites get lost.
It's especially important for SSH tests which need to
always remember which OS is booted to use the correct
SSH credentials.
This commit changes this so that everything is run at once,
but the results are later parsed and split per-suite using
rebot.
An important thing to remember is that testers will often
interrupt the run using ctrl+c to get the test results
early and skip further execution. The change handles that
by trapping the INT signal and running the split
anyway.
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
lib/robot.sh: Move robot to fg instead of waiting
When running robot as a subprocess it wont receive the control
over stdin otherwise. Some semi-auto tests require user input
in terminal which will crash otherwise.
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
lib/robot.sh: fix detecting tags
tests that are fullauto on some and semi on some platforms
need to have both tags so that they are executed no matter
the `-i` param and the skips can take care of whether
the test is automated on given platform or not
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* TEST_TAGS: rename to INCLUDE_TAGS
Theres a builtin ${TEST TAGS}, but for some reason ${TEST_TAGS}
seems to be like an alias to it.
Our variable that keeps track of the tags requested by tester
needs to be changed because of that
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* options-lib_dcu: Set nextboot before Set UEFI Option
Set UEFI Option of options-lib_dcu reads the current
SMM Store, modifies it, and flashes it back.
If we set the Nextboot variable in SMM before reading
the SMM store it will stay after flash and reboot reducing
an unnecessary DEFAULT_BOOT_OS_ID boot.
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* sleep-lib.robot: warn on failed fwts sleep
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
sleep-lib: Add description of s3 test also working for s2idle(soix)
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* keywords.robot: Identify Path To USB: Fix returning TYPE instead of PATH
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* capsule-update.robot: Skip unnecessary setup when on ssh
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* scripts/lib/robot.sh & related: Restore old logs directory structure
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* audio-subsystem.robot: Delete old version of suite
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
* platform-configs/include/protectli-vp66xx.robot: enable redundant boot
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
* coreboot-redundant-boot.robot: work with the new log convention
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
* platform-configs/include/protectli-vp66xx.robot: enable CBnT w/ status menu
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
* lib/bios/menus.robot: Parse Menu Snapshot Into Construction: drop fuzzy search
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
* platform-configs/include/protectli-vp66xx.robot: remove unused variable
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
* coreboot-redundant-boot.robot: RTC BUC Control Bit Should Be: don't convert case
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
* coreboot-redundant-boot.robot: parse log from current boot only
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
---------
Signed-off-by: Filip Gołaś <filip.golas@3mdeb.com>
Signed-off-by: Michał Kopeć <michal.kopec@3mdeb.com>
Co-authored-by: Michał Kopeć <michal.kopec@3mdeb.com>
