When a gadget is installed, we compare the previous gadget to detect
removed files and remove them. Preserved files that are removed from
content list are kept.
This is important to be able to remove old boot firmware since some,
like shim, change behavior depending of files that are present.
This does not support directories. If the gadget install directories
with content, the files in the directories will not be removed.
* fix run-tests test to allow to run multiple tests on custom variant
Ths is needed to re-exec failed tests on beta validation
* add acho to show tests to be executed
* fix shellcheck disable=SC2086
* add stderr to spread.log
* tests:new perf test to install many snaps with --no-wait and check changes
Install different snaps many times based on arch availability. In this
scenario all the snaps are installed in parallel and the test checks
the system can handel all the installs and remains active.
* performance tests reorganization
To avoid executing perf tests when the nested tests are executed, the
tests have been moved under tests/perf
tests/perf/main -> suite with the tests executed in the target device
tests/perf/nested -> suite with the tests executed in a nested
environment
* Update tests/perf/nested/install-many-snaps-no-wait/task.yaml
Co-authored-by: Miguel Pires <miguelpires94@gmail.com>
* Update tests/perf/nested/install-many-snaps-no-wait/task.yaml
Co-authored-by: Miguel Pires <miguelpires94@gmail.com>
* Update tests/perf/main/install-many-snaps-no-wait/task.yaml
Co-authored-by: Miguel Pires <miguelpires94@gmail.com>
* improve logs check and fixes based on review comments
* addressing review comments
---------
Co-authored-by: Miguel Pires <miguelpires94@gmail.com>
* tests/nested/manual/recovery-system-reboot: fix test that was being interrupted by a system reboot
In the case of an auto-refresh, the system would reboot. This resulted
in some cryptic failures. Make sure to wait for an auto-refresh to
happen before removing the recovery system, and make sure to wait for
the system to reboot once the auto-refresh has finished.
* tests/nested/manual/recovery-system-reboot: add an extra wait to continue waiting after reboot
* spread.yaml: use lxd edge with shmounts fix
* tests: use snapcraft edge with fix to handle lxd version naming change
* tests: use lxd 5.20/stable with snapcraft 4.x that cannot handle lxd version naming change
* tests: added details for nested/manual/devmode-snaps-can-run-other-snaps
* o/assertstate, o/devicestate: add more general function for fetching validation set assertions
* daemon, client: add API routes for creating/removing recovery system
* daemon, o/snapstate: add .snap file extension to snaps from forms
The seed writer will fail to consider files as snaps if their filenames
do not end in .snap.
* tests: test creating a recovery system
* tests: add spread test for offline creation of recovery system
* tests: update offline recovery system test to reboot into new system
* tests/nested/manual/recovery-system-reboot: add variants for factory-reset and install modes
* tests: replace usage of default-recovery-system with default-recovery
* o/devicestate: enable offline creation of recovery system entirely from pre-installed snaps
* daemon, client: test that offline API works without providing snaps or validation sets
* tests/nested/manual/recovery-system-offline: test offline remodel with only pre-installed snaps
* tests/nested/manual/recovery-system-reboot: modify test to create system with new set of essential snaps
* tests: disable shellcheck printf check
* daemon: rename functions for working with form values and add one for working with booleans
* daemon: acquire state lock later in postSystemActionCreateOffline
* daemon: cleanup form files if we fail to make change to create a recovery system
* daemon: rename parseValidationSets to assertionsFromValidationSetStrings for clarity
* client, daemon, tests: add "offline" field to create recovery system JSON api
* daemon: convert TODO about comma-delimited list into explanation of why we use a comma delimited list
* NEWS.md: add mention of create/remove recovery systems API
* tests/nested/manual/recovery-system-offline: explicitly disable network from nested vm
* tests/nested/manual/recovery-system-reboot: do not use new gadget in recovery system for now
* tests/lib/nested.sh: add variable NESTED_FORCE_MS_KEYS to force using microsoft keys
* tests/nested/manual/recovery-system-reboot: add back gadget snap swap to test
* tests/nested/manual/recovery-system-reboot: retry POST to remove since there might be an auto-refresh happening
* new full-run test
* update tests to support pre-built images
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 1c8efb77e1..74082c0c34
74082c0c34 Tests improve remote wait (#49)
5121bfb659 remove support for opensuse leap 15.4 (#48)
30df700d08 Add new systems support (#47)
1f08938925 Support check amazon linux version (#46)
43533bdd97 Change the exit value checking for test formats (#45)
3c88244c04 Update check-test-format to support a dir and a list of files (#44)
510d95f429 add extra check for error in auto-refresh detection function
3289d4031b Try open the log with latin-1 encoding when utf-8 is not working
9db785499f improved how the tools are waiting for system reboot
2a5c4414a3 fix shellcheck errors
5e7b63883d Fixes for osquery and tests pkgs (#43)
4c9145e2ac support reboot waiting for auto-refresh
45768f5188 show changes in unknown status after refresh
8013c30c2a Remove support for ubuntu 22.10
b32b80bf54 Fix remote.rait-for test in bionic
5675c625e9 Enable fedora 38
55f4471957 Support for new oss
f2e88b357c New tool used to query spread json reports
cacd35ede0 utils/spread-shellcheck: explain disabled warnings (#42)
c82afb2dee Support --no-install-recommends parameter when installing dependencies with tests.pkgs
b84eea92e2 spread-shellcheck: fix quotes in environment variables (#41)
ab1e51c29f New comparison in os-query for core systems (#40)
e5ae22a5d4 systemd units can be overwritten
63540b845a Fix error messages in remote pull and push
75e8a426a5 make sure the unit is removed in tests.systemd test
9089ff5c02 Update tests to use the new tests.systemd stop-unit
44ecd5e56a Move tests.systemd stop-units to stop-unit
01a2a83b4b Update tests.systemd to have stop units as systemd.sh
162e93bd35 update tests.systemd CLI options to be the same than retry command
14aa43a405 new feature to re-run failed spread tests (#39)
604cb782db Fix shellcheck in systemd tool
bfc71082c8 Update the tests.systemd to allow parameters waiting for service status
8a2d0a99df Adding quiet tool and removing set +-x from tests.pkgs
d90935d2a4 A comment explaining about the default values for wait-for
3232c5dba7 Add support for ubuntu 23.04
a7164fba07 remove fedora 35 support, add fedora 37 support
89b9eb5301 Update systems supported
92bb6a0664 Include snap-sufix in the snaps.name tool
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 74082c0c345a7707ca360d2177a23ee958c880bd
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 74082c0c34..5ef5dcbe8f
5ef5dcbe8f Tests use artifacts in spread tests (#51)
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 5ef5dcbe8f376e3958019fffcd54fd6add8117a4
* update args order in new test
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 5ef5dcbe8f..b89b4f8647
b89b4f8647 fix artifacts name
d30cee6da0 Merge remote-tracking branch 'upstream/main'
555c43d2ab Support auto-refresh with Do instead of Doing
96c2b0c19c remove tests support for ubuntu 23.04 (EoL)
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: b89b4f86471b76eae9d5ac1b1f55728598da544e
* improbe checks in nested.sh
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from b89b4f8647..4cf8e635bf
4cf8e635bf fix os.query test after merge
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 4cf8e635bf65f974dbcc5b303f890ac9078ed78a
* removing duplicated debian-11
* remove another merge error
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 4cf8e635bf..3a0fc57e1e
3a0fc57e1e add explanation about why we check for ( Do | Doing )
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 3a0fc57e1eaf13b189a83982835cf289b3dfad3c
* Squashed 'tests/lib/external/snapd-testing-tools/' changes from 3a0fc57e1e..2e4a3153a2
2e4a3153a2 1 more comment
git-subtree-dir: tests/lib/external/snapd-testing-tools
git-subtree-split: 2e4a3153a27529092156fbe723cdfef221467bd4
* increase limit waiting for log changes because check to ntp server breaks
* increase time to wait for log changes
* wait more time waiting for changes in the log
It gets stuck more than 60 seconds waiting for ntp sync
* o/snapstatetest, o/devicestate, overlord: add param to InstallSnap and InstallEssentialSnaps for setting up gadget files
* o/devicestate: allow remodeling on hybrid models
When remodeling on a hybrid model, check to make sure that we have a
seed partition before creating a new recovery system as a part of the
remodel. Creating the new system is skipped if we don't see a seed
partition.
* tests/nested/manual/hybrid-remodel: new test for remodeling on a hybrid model
* o/snapstate/snapstatetest: forward gadget files to InstallSnap for gadget
* o/devicestate: add check for if the system is core-boot in undoCreateRecoverySystem
* gadget: add Info.HasRole method that checks for a role in an Info struct
* o/devicestate: replace manual role lookup with gadget.Info.HasRole incovation
* tests/nested/manual/hybrid-remodel: add TODO about testing system seeds on hybrid systems
* tests/nested/manual/hybrid-remodel: properly wait for reboot
* tests/nested/manual/hybrid-remodel: use change_id to make assertions more specific
* o/devicestate: do not attempt to finalize seed install when there is no seed partition
* tests: put muinstaller use into script for use in multiple tests
* o/devicestate: add comment explaining reason for explicitly checking classic systems
* tests: rename muinstaller.sh to setup_nested_hybrid_system.sh to clarify functionality
* tests/lib/tools/setup_nested_hybrid_system.sh: tell shellcheck where to find sourced files
* tests/lib/tools: rename --fake-disk to --disk in setup_nested_hybrid_system.sh
* tests/nested/manual/muinstaller-real: fix missing usage of $label variable in setup_nested_hybrid_system.sh
* tests/nested/manual/muinstaller-real: fix check for pre-existing gpt partition
* tests/nested/manual/muinstaller-real: use proper quoting on path with escaped chars
We don't need to do this any more, as muinstaller does not need it.
Additionally, this was preventing us from booting into the installed
recovery system, as some files (grubenv) that are part of the seed
reference the seed's label, but they were not updated.
Wait for the kernel change to be fully done before proceeding,
otherwise we can have a race condition and remodel will fail with
message:
error: cannot do offline remodel: cannot remodel device: snap
"pc-kernel" has "refresh-snap" change in progress
Copy the seed on Ubuntu Core installations.
* tests/muinstaller: do not copy seed anymore
This job is done by snapd now.
* o/devicestate: update installer API to use seed.Copy interface rather than a manual copy
* tests/nested/manual/muinstaller-core: update test to check for properly installed seed after installation
* o/devicestate: use temporary db when loading assertions
* o/devicestate: refactor to only open seed once in doInstallFinish
* o/devicestate: consider seed not implementing seed.Copier as an internal error
* o/devicestate: use timings.Measurer from task
* o/devicestate: pack return values from loadSystemAndEssentialSnaps into systemAndEssentialSnaps struct
* o/devicestate: convert loadSystemAndEssentialSnaps return parameter into a pointer
* o/devicestate: add TODO about optionally returning seed from loadSystemAndEssentialSnaps
---------
Co-authored-by: Andrew Phelps <andrew.phelps@canonical.com>
* c/snap-bootstrap: lift requirement for fde-setup hook for single-boot-install
* tests/nested/manual/uc20-install-in-initrd: test various systems that support single-boot install
* c/snap-bootstrap: add test for single-boot install without a fde-setup hook
* tests/nested/manual/uc20-install-in-initrd: use "not" rather than "!" in spread test
* daemon: add offline field to remodeling JSON payload
* client, cmd/snap: add offline boolean option for remodeling to client and cli interface
* daemon: make snap files optional in the "multipart/form-data" remodel api
* tests/nested/manual/remodel-offline: add variants to test different combinations of installed/local snaps/assertions
* client: update comment on RemodelWithLocalSnaps to mention not contacting the store
* c/snap: remove unneeded var declaration
* tests/nested/manual/remodel-offline: clarify section that acks/adds the validation set assertion
* tests/nested/manual/remodel-offline: fix shellcheck warning on test's -a flag
* client: verify that normal (online) remodel does not set offline field in JSON payload
* tests: use load generator to generate load in perf tests
Also it is included a new test to check snap connections for snap
provided interfaces and another to check many installations in parallel.
* fix shellchecks
* fix misspelling
* update parallel install numbers
* Target the perf tests for ubuntu core >=18
This PR makes remodels take into account revision constraints from validation sets on the new model. Additionally, snaps that are marked as invalid in validation sets are checked for in the model.
* a/snapasserts: add methods for extracting more information out of ValidationSets type
* o/assertstate: add ValidationSetsFromModel function for extracting a snapasserts.ValidationSets from an asserts.Model
* o/snapstate: prevent installing/updating a snap from a local file that does not match requested revision
* o/devicestate: consider validation sets during remodeling
* tests/nested/manual: add remodel test that downgrades a snap because of a validation set
* tests/nested/manual: add remodel test that fails to remodel because of an invalid snap in a validation set
* tests/nested/manual: extend offline remodel test to also include a validation set
* tests/lib/assertions: fix timestamps on assertions
* asserts: add Key method to ValidationSet and ModelValidationSet
* o/devicestate: use new Key methods
* o/devicestate: maybe enforce validation sets during doSetModel
* o/devicestate: add test for enforcing validation sets in doSetModel
* a/snapasserts: simplify TestCanBePresent with loop
* tests/lib/assertions: add bluez snap to offline remodel test
* o/devicestate: remove done TODO
* o/snapstate: if remodeling, do not install prereq if link-snap task is present
* tests/nested/manual/remodel-offline: extend test to verify that validation sets are accounted for
* Revert "o/snapstate: if remodeling, do not install prereq if link-snap task is present"
This reverts commit 57c7725a2513df51be7ac1c06c492aaed07a6e3b.
This change is independent and will be included in another PR.
* a/snapasserts: add methods for extracting more information out of ValidationSets type
* o/assertstate: add ValidationSetsFromModel function for extracting a snapasserts.ValidationSets from an asserts.Model
* o/devicestate: add test for ValidationSetsConflictError.Is
* a/snapasserts: move methods after New function
* a/snapasserts: add test for ValidationSets.Revisions to verify ValidationSetsConflictError is returned
* o/assertstate: change ValidationSetsFromModel to take in a DeviceContext, rather than a StoreService
* o/assertstate: rename ValidationSetsModelFlags to ValidationSetsModelOptions
* o/devicestate: add type to export_test to make testing simpler
* tests: add details to new spread tests
* asserts: rename ModelValidationSet.Key and ValidationSet.Key to .SequenceName and add unit tests for them
* o/snapstate: update snap revision mismatch error message to be more clear
* o/devicestate: introduce helper for setting ValidationSets on snapstate.RevisionOptions if Revision is set
* o/devicestate: verify the parameters that fakeSequenceStore receives
* o/devicestate: fix revisions not being respected for essential snaps (and add a test for it)
* o/devicestate: extend TestRemodelUC20EssentialSnapsAlreadyInstalledAndLocal to also exercise case where a validation set requires a revision but the currently installed version is unasserted
* s/seedtest: update retrieveSeq to handle unconstrained sequence forming assertions
* a/snapasserts: add ValidationSets.Sets method
* o/assertstate: add deviceContext to ForgetValidationSet function so that change can happen during remodel
* o/devicestate: attempt to handle rollback of validation sets during failed remodel
* overlord: test for replacing conflicting validation sets during remodel
* o/assertstate: update ForgetValidationSet to take in a DeviceContext and to allow for forcing removal even if the validation set is in use by the model
* o/devicestate: roll back validation set changes on remodel failure
* o/devicestate: make sure that validation sets unrelated to the model survive a remodel
* o/devicestate: rename param in installedSnapRevisionChanged
* o/devicestate: rename field newSnapRevision to newRequiredRevision in modelSnapsForRemodel
* o/devicestate: simplify loops in checkForInvalidSnapsInModel
* o/devicestate: compare validation sets using SequenceName methods
* o/devicestate: fail remodel if we attempt to use an unasserted snap as a specific revision
* tests/nested/manual/remodel-offline: fix test to actually use validation set
* o/devicestate: create helper for creating snapstate.RevisionOptions during remodel
* o/devicestate: name param literals for clarity
* o/devicestate: invert logic to eliminate double negative
* o/devicestate: fix missed inversion of logic
* o/assertstate: update comment on ForgetValidationSetOpts.ForceForget
* overlord, o/devicestate: update remodel test to change models that contain the same validation set
* o/assertstate: test ForceForget functionality in ForgetValidationSet
* o/devicestate: rename function newRevisionOptionsForRemodel to revisionOptionsForRemodel
* o/assertstate, o/devicestate, daemon: remove unneeded DeviceContext param from ForgetValidationSet
* o/devicestate: remove println
* o/devicestate: clarify comment in rollback of adding validation sets
* o/devicestate: rename variable in enforceValidationSetsForRemodel
* o/snapstate: clarify error when attempting to install/refresh local snap with different revision than requested
* o/devicestate: naming consistency
* o/devicestate: simplify error when model is missing snap that is required in validation set
* asserts, overlord, o/devicestate: rename SequenceName to SequenceKey and prefix the series to the string that is returned
* tests: add new performance and load test suite
This is the initial change to support performance tests.
These new tests have to be able to run either in nested vms and in
external devices.
The idea is to run nightly and on demand on nested vms and also as part
of regular runs on specific devices.
The initial test is about connecting and disconnecting interfaces,
covering a specific percentage of those and also test how the system
behaves when all are connected together.
* fix shellcheck
* fix minimal-smoke test
* add details to minimal-smoke test
* use ubuntu-image snap in xenial
