* tests: properly build snapd snap
Now we build also the test version of snapd snap in `snap-builds`
workflow job. We copy this into the spread tests. And we use that
snap, which we only instrument instead of copying the snapd deb build.
If the snap is not available, then we build it in spread. On CI, this
happens on arm since the workflow does not build it. It will also
happen when triggering test manually.
* tests: couple of small improvements to test syntax, move WORK_DIR into script scope, use PWD instead of dot notation
---------
Co-authored-by: Philip Meulengracht <the_meulengracht@hotmail.com>
* update gadget.yaml for nested remodel tests
* add new snaps with base: core24
* tests: fixes for uc24 nested tests
This change contains:
. fixes for model
. use of core24 snaps (jq, rsync and sh)
. test fixes
. using legacy scp in snapd-testing-tools remote-pull (to avoid errors)
* use -O when the scp version is -ge than 9
* re-enable hybrid-remodel test on noble
* update wording of remote.pull
* Adding more workers for nested tests
This is to avoid instances are cleaned before the execution finishes
* First list of changes to make nested tests run
* avoid using OVMF with 4M
* Check for cloud init status in uc24
* removed OVMF_SIZE var which is not used anymore
* fix merge issue
* update gadget.yaml for test-snapd-remove-pc-24 snap
* remove mbr and source: shim.efi.signed from gadget.yaml
* restoring source: shim.efi.signed in seed partition
* Fixed the incorrect uuid of the core24 snap
* fixed the snapcraft.yaml for jq-core24
* tests: create test user with useradd in uc24
This change is needed to create the test user in uc24 with home
directory.
This is needed to fix tests in uc24 (edge validation)
* use --create-home and --comment
--gecos is deprecated
Build drivers tree on installation so kernel modules can be found on first boot after installation (otherwise the modules would be available too late, after initial installation of the kernel has happened while seeding).
Also, make sure that the tree is not re-created when seeding, and add a spread test for kernel-modules components.
* dirs: add function to get root of drivers trees
* kernel: do not recreate drivers tree if it already exists
On installation, a drivers tree has already been copied to the data
partition. Do not re-create if found when seeding happens (note though
that on preseeding we will not find the tree and we will still create
it).
* gadget/install: create device tree on installations
Create a device tree on installation so we have it on first boot.
* o/snapstate: fix task message
* tests/nested/manual: add kernel-modules-components test
This test installs a kernel-modules component and verifies that the
shipped kernel module is installed and can be loaded.
* tests/lib: do not error out if cloud-init finished with
'recoverable error' status.
* kernel: have destination as argument when building drivers tree
We need to tackle the generic installation case as opposed to creating
the drivers tree when seeding the system. In that case the destination
tree can vary depending on installation type.
* overlord: pass kernel information to install package
so the drivers tree can be created.
* cmd/snap-bootstrap: build drivers tree on installation from initramfs
* tests: adapt to additional parameters in install.Run
* snap/naming: add ancillary method to split a full component name
into snap and component names.
* gadget/install: fix nosecboot tests
* enable nested tests for uc24
* fix issue related to cloud init
* some test fixes
* Add missing details for tests
* Adding missing details for nested tests
* addin support for ubuntu-24.04 in upload-snapd-to-gce
* updated the password for external users in spread.yaml
* tests:new perf test to install many snaps with --no-wait and check changes
Install different snaps many times based on arch availability. In this
scenario all the snaps are installed in parallel and the test checks
the system can handel all the installs and remains active.
* performance tests reorganization
To avoid executing perf tests when the nested tests are executed, the
tests have been moved under tests/perf
tests/perf/main -> suite with the tests executed in the target device
tests/perf/nested -> suite with the tests executed in a nested
environment
* Update tests/perf/nested/install-many-snaps-no-wait/task.yaml
Co-authored-by: Miguel Pires <miguelpires94@gmail.com>
* Update tests/perf/nested/install-many-snaps-no-wait/task.yaml
Co-authored-by: Miguel Pires <miguelpires94@gmail.com>
* Update tests/perf/main/install-many-snaps-no-wait/task.yaml
Co-authored-by: Miguel Pires <miguelpires94@gmail.com>
* improve logs check and fixes based on review comments
* addressing review comments
---------
Co-authored-by: Miguel Pires <miguelpires94@gmail.com>
* tests: initial changes to run core suite in uc24
Changes needed to run core test suite in uc24
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 1c8efb77e1..1db5214d5f
1db5214d5f Improve the remote docs (#36)
2e4a3153a2 1 more comment
3a0fc57e1e add explanation about why we check for ( Do | Doing )
4cf8e635bf fix os.query test after merge
b89b4f8647 fix artifacts name
d30cee6da0 Merge remote-tracking branch 'upstream/main'
5ef5dcbe8f Tests use artifacts in spread tests (#51)
555c43d2ab Support auto-refresh with Do instead of Doing
96c2b0c19c remove tests support for ubuntu 23.04 (EoL)
74082c0c34 Tests improve remote wait (#49)
5121bfb659 remove support for opensuse leap 15.4 (#48)
30df700d08 Add new systems support (#47)
1f08938925 Support check amazon linux version (#46)
43533bdd97 Change the exit value checking for test formats (#45)
3c88244c04 Update check-test-format to support a dir and a list of files (#44)
510d95f429 add extra check for error in auto-refresh detection function
3289d4031b Try open the log with latin-1 encoding when utf-8 is not working
9db785499f improved how the tools are waiting for system reboot
2a5c4414a3 fix shellcheck errors
5e7b63883d Fixes for osquery and tests pkgs (#43)
4c9145e2ac support reboot waiting for auto-refresh
45768f5188 show changes in unknown status after refresh
8013c30c2a Remove support for ubuntu 22.10
b32b80bf54 Fix remote.rait-for test in bionic
5675c625e9 Enable fedora 38
55f4471957 Support for new oss
f2e88b357c New tool used to query spread json reports
cacd35ede0 utils/spread-shellcheck: explain disabled warnings (#42)
c82afb2dee Support --no-install-recommends parameter when installing dependencies with tests.pkgs
b84eea92e2 spread-shellcheck: fix quotes in environment variables (#41)
ab1e51c29f New comparison in os-query for core systems (#40)
e5ae22a5d4 systemd units can be overwritten
63540b845a Fix error messages in remote pull and push
75e8a426a5 make sure the unit is removed in tests.systemd test
9089ff5c02 Update tests to use the new tests.systemd stop-unit
44ecd5e56a Move tests.systemd stop-units to stop-unit
01a2a83b4b Update tests.systemd to have stop units as systemd.sh
162e93bd35 update tests.systemd CLI options to be the same than retry command
14aa43a405 new feature to re-run failed spread tests (#39)
604cb782db Fix shellcheck in systemd tool
bfc71082c8 Update the tests.systemd to allow parameters waiting for service status
8a2d0a99df Adding quiet tool and removing set +-x from tests.pkgs
d90935d2a4 A comment explaining about the default values for wait-for
3232c5dba7 Add support for ubuntu 23.04
a7164fba07 remove fedora 35 support, add fedora 37 support
89b9eb5301 Update systems supported
92bb6a0664 Include snap-sufix in the snaps.name tool
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 1db5214d5fe91d90b4ffcd4768db8080fcc245ab
* fix core version under test
* adding missing model
* add missing function in nested.sh
* fix keys used for uc24
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 1db5214d5f..dacfd81de9
dacfd81de9 fix is_core functions
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: dacfd81de95e05a9e56d84be45e0611275b083f4
* use pc-kernel from beta channel
* removing file created for workflow tests
* remove more dirs created during automatic merge
* restore perimssions for files in test snap-repair
* restore tools permissions merged incorrectly
* fix wording in test
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from dacfd81de9..b89ec98b23
b89ec98b23 use local variables in os.query tool
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: b89ec98b239dc9ef729b6af68ce1b5028b4eee23
* fix remove test details
* tests: make more robust the nested log checks
Consider the case then the log is not growing but the ssh can be
stablished. In that case the log checks should return.
* remove the log growing check
This is not working in all the cases. Sometimes the los is not growing
and the machine is ready.
* o/assertstate, o/devicestate: add more general function for fetching validation set assertions
* daemon, client: add API routes for creating/removing recovery system
* daemon, o/snapstate: add .snap file extension to snaps from forms
The seed writer will fail to consider files as snaps if their filenames
do not end in .snap.
* tests: test creating a recovery system
* tests: add spread test for offline creation of recovery system
* tests: update offline recovery system test to reboot into new system
* tests/nested/manual/recovery-system-reboot: add variants for factory-reset and install modes
* tests: replace usage of default-recovery-system with default-recovery
* o/devicestate: enable offline creation of recovery system entirely from pre-installed snaps
* daemon, client: test that offline API works without providing snaps or validation sets
* tests/nested/manual/recovery-system-offline: test offline remodel with only pre-installed snaps
* tests/nested/manual/recovery-system-reboot: modify test to create system with new set of essential snaps
* tests: disable shellcheck printf check
* daemon: rename functions for working with form values and add one for working with booleans
* daemon: acquire state lock later in postSystemActionCreateOffline
* daemon: cleanup form files if we fail to make change to create a recovery system
* daemon: rename parseValidationSets to assertionsFromValidationSetStrings for clarity
* client, daemon, tests: add "offline" field to create recovery system JSON api
* daemon: convert TODO about comma-delimited list into explanation of why we use a comma delimited list
* NEWS.md: add mention of create/remove recovery systems API
* tests/nested/manual/recovery-system-offline: explicitly disable network from nested vm
* tests/nested/manual/recovery-system-reboot: do not use new gadget in recovery system for now
* tests/lib/nested.sh: add variable NESTED_FORCE_MS_KEYS to force using microsoft keys
* tests/nested/manual/recovery-system-reboot: add back gadget snap swap to test
* tests/nested/manual/recovery-system-reboot: retry POST to remove since there might be an auto-refresh happening
* new test isntall many snaps
this test installs as many snaps as desired and then checks the snapd
status and main functions
* fix shellcheck about quotes
* check if ssh is ready in nested tests before the checks in serial log
This will save time and also prevent race issues
* new full-run test
* update tests to support pre-built images
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 1c8efb77e1..74082c0c34
74082c0c34 Tests improve remote wait (#49)
5121bfb659 remove support for opensuse leap 15.4 (#48)
30df700d08 Add new systems support (#47)
1f08938925 Support check amazon linux version (#46)
43533bdd97 Change the exit value checking for test formats (#45)
3c88244c04 Update check-test-format to support a dir and a list of files (#44)
510d95f429 add extra check for error in auto-refresh detection function
3289d4031b Try open the log with latin-1 encoding when utf-8 is not working
9db785499f improved how the tools are waiting for system reboot
2a5c4414a3 fix shellcheck errors
5e7b63883d Fixes for osquery and tests pkgs (#43)
4c9145e2ac support reboot waiting for auto-refresh
45768f5188 show changes in unknown status after refresh
8013c30c2a Remove support for ubuntu 22.10
b32b80bf54 Fix remote.rait-for test in bionic
5675c625e9 Enable fedora 38
55f4471957 Support for new oss
f2e88b357c New tool used to query spread json reports
cacd35ede0 utils/spread-shellcheck: explain disabled warnings (#42)
c82afb2dee Support --no-install-recommends parameter when installing dependencies with tests.pkgs
b84eea92e2 spread-shellcheck: fix quotes in environment variables (#41)
ab1e51c29f New comparison in os-query for core systems (#40)
e5ae22a5d4 systemd units can be overwritten
63540b845a Fix error messages in remote pull and push
75e8a426a5 make sure the unit is removed in tests.systemd test
9089ff5c02 Update tests to use the new tests.systemd stop-unit
44ecd5e56a Move tests.systemd stop-units to stop-unit
01a2a83b4b Update tests.systemd to have stop units as systemd.sh
162e93bd35 update tests.systemd CLI options to be the same than retry command
14aa43a405 new feature to re-run failed spread tests (#39)
604cb782db Fix shellcheck in systemd tool
bfc71082c8 Update the tests.systemd to allow parameters waiting for service status
8a2d0a99df Adding quiet tool and removing set +-x from tests.pkgs
d90935d2a4 A comment explaining about the default values for wait-for
3232c5dba7 Add support for ubuntu 23.04
a7164fba07 remove fedora 35 support, add fedora 37 support
89b9eb5301 Update systems supported
92bb6a0664 Include snap-sufix in the snaps.name tool
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 74082c0c345a7707ca360d2177a23ee958c880bd
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 74082c0c34..5ef5dcbe8f
5ef5dcbe8f Tests use artifacts in spread tests (#51)
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 5ef5dcbe8f376e3958019fffcd54fd6add8117a4
* update args order in new test
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 5ef5dcbe8f..b89b4f8647
b89b4f8647 fix artifacts name
d30cee6da0 Merge remote-tracking branch 'upstream/main'
555c43d2ab Support auto-refresh with Do instead of Doing
96c2b0c19c remove tests support for ubuntu 23.04 (EoL)
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: b89b4f86471b76eae9d5ac1b1f55728598da544e
* improbe checks in nested.sh
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from b89b4f8647..4cf8e635bf
4cf8e635bf fix os.query test after merge
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 4cf8e635bf65f974dbcc5b303f890ac9078ed78a
* removing duplicated debian-11
* remove another merge error
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 4cf8e635bf..3a0fc57e1e
3a0fc57e1e add explanation about why we check for ( Do | Doing )
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 3a0fc57e1eaf13b189a83982835cf289b3dfad3c
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 3a0fc57e1e..2e4a3153a2
2e4a3153a2 1 more comment
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 2e4a3153a27529092156fbe723cdfef221467bd4
* increase limit waiting for log changes because check to ntp server breaks
* increase time to wait for log changes
* wait more time waiting for changes in the log
It gets stuck more than 60 seconds waiting for ntp sync
* Revert "daemon: move the closing of snapdListener"
This reverts commit fe9c662b1e.
* New procedure to determine when a nested vm is not booting as expected
The idea is to stop a nested test which:
. The nested service is not active
. The log hasn't changed during the last 60 seconds
By doing this, when a vm fails to boot, the test will fail much faster
than before, reducing the nested tests execution time.
* fix shellcheck errors
* Fix comparison for log sizes
* fix issue retrieving the lines in the file
* add env vars to retry to make sure the evaluation is done while retrying
* fix shellcheck
* fix quotes
* Add check to detect infinit loops
* fix issues and improve checks
* Fix shellcheck errors
* fix misspelling
* make less verbose the waiting for the machine is ready
* Revert "Revert "daemon: move the closing of snapdListener""
This reverts commit b5231f05c18f889439d81d9e01eff5f7bf4a7dc3.
* move retry - 1 operation
* Update tests/lib/nested.sh
Co-authored-by: Maciej Borzecki <maciek.borzecki@gmail.com>
* Update tests/lib/nested.sh
Co-authored-by: Maciej Borzecki <maciek.borzecki@gmail.com>
* tests/lib/nested: fix shellcheck errors
Signed-off-by: Maciej Borzecki <maciej.borzecki@canonical.com>
* tests/core/snapd-maintenance-msg: add details
Signed-off-by: Maciej Borzecki <maciej.borzecki@canonical.com>
---------
Signed-off-by: Maciej Borzecki <maciej.borzecki@canonical.com>
Co-authored-by: Maciej Borzecki <maciek.borzecki@gmail.com>
Co-authored-by: Maciej Borzecki <maciej.borzecki@canonical.com>
* tests: add new performance and load test suite
This is the initial change to support performance tests.
These new tests have to be able to run either in nested vms and in
external devices.
The idea is to run nightly and on demand on nested vms and also as part
of regular runs on specific devices.
The initial test is about connecting and disconnecting interfaces,
covering a specific percentage of those and also test how the system
behaves when all are connected together.
* fix shellcheck
* fix minimal-smoke test
* add details to minimal-smoke test
* use ubuntu-image snap in xenial
* tests: use KERNEL_CHANNEL/NESTED_KERNEL_CHANNEL instead of edge for pc-kernel
* add var to define gadget channel used to create the nested vm
---------
Co-authored-by: Sergio Cazzolato <sergio.cazzolato@canonical.com>
This change adds a dependency between the labeler and the tests
workflows. Then after the labeler is executed, the nested test job
request the labels added to the PR and based on that request it is
determined if the nested tests have to be executed.
Before it would result it being added to install mode as well, when we only set up the minimal set of snaps (gadget, kernel, base, snapd). If the validation-set contained any snaps besides these, it would fail. Wait until run mode, where we set up any non-essential snaps, and after this we expect all snaps to be present in system.
