This patch fixes the ZeroDivisionError which can
occur during percantage of test execution reporting.
Signed-off-by: Maciej Perkowski <Maciej.Perkowski@nordicsemi.no>
Use the pytest test framework in the dtlib.py and edtlib.py test
suites (testdtlib.py and testedtlib.py respectively).
The goal here is not to change what is being tested. The existing test
suite is excellent and very thorough.
However, it is made up of executable scripts where all of the tests
are run using a hand-rolled framework in a single function per file.
This is a bit all-or-nothing and prevents various nice features
available in the de-facto standard pytest test framework from being
used.
In particular, pytest can:
- drop into a debugger (pdb) when there is a problem
- accept a pattern which specifies a subset of tests to run
- print very detailed error messages about the actual and expected
results in various traceback formats from brief to very verbose
- gather coverage data for the python scripts being tested (via plugin)
- run tests in parallel (via plugin)
- It's easy in pytest to run tests with temporary directories
using the tmp_path and other fixtures. This us avoid
temporarily dirtying the working tree as is done now.
Moving to pytest lets us leverage all of these things without any loss
in ease of use (in fact, some things are nicer in pytest):
- Any function that starts with "test_" is automatically picked up and
run. No need for rolling up lists of functions into a test suite.
- Tests are written using ordinary Python 'assert'
statements.
- Pytest magic unpacks the AST of failed asserts to print details on
what went wrong in really nice ways. For example, it will show you
exactly what parts of two strings that are expected to be equal
differ.
For the most part, this is a pretty mechanical conversion:
- extract helpers and test cases into separate functions
- insert temporary paths and adjust tests accordingly to not match
file names exactly
- use 'assert CONDITION' instead of 'if not CONDITION: fail()'
There are a few cases where making this happen required slightly
larger changes than that, but they are limited.
Move the checks from check_compliance.py to a new GitHub workflow,
removing hacks that are no longer needed.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
So far 'tests' and 'samples' directories were matched only when they
were first directories in path hierarchy. This doesn't work when running
sanitycheck from directory other than Zephyr source code root.
Match any directory in path hierarchy, similar how file is matched
currently. That way sanitycheck can be executed from outside of Zephyr
source code directory.
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Add '--coverage-basedir BASEDIR' argument, which can be used to specify
source code base directory other than default Zephyr root directory.
This is mainly useful for projects built on top of Zephyr, where
sanitycheck is used for unit testing application code.
Signed-off-by: Marcin Niestroj <m.niestroj@grinn-global.com>
Use --filter=runnable for example to limit the tests being built to
those which actually can run on a device or a emulation platform.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
build_only was being used in different contexts and was confusing, so
clean this up and in some places just set if a test is runnable, a
testcase should always be buildable if the filters are matching.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Inspect the hex file with intelhex, and fail if the hex file has any
contents in the UICR area(s).
family == 'NRF52' still always does --sectoranduicrerase, but this
option is not available on other families.
Add --force command line option to proceed with flashing instead of
failing.
Signed-off-by: Øyvind Rønningstad <oyvind.ronningstad@nordicsemi.no>
Only create the dir and file if we have actual "content" that will
exist in testcase_extra.conf. This is to reduce a bit of
noise/footprint in the sanitycheck output dir.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
If we kill qemu because of a timeout, capture this as the reason instead
of reporting the exit code.
Fixes#28040
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Breathe 4.21.0 brings a few interesting features that improve the
generated documentation:
* A new `separate_member_pages` config option that patches issues in the
Doxygen XML generated when SEPARATE_MEMBER_PAGES option is YES.
* A new rST verbatim mode that allows producing inline elements.
Signed-off-by: Fabio Utzig <fabio.utzig@nordicsemi.no>
This undo's commit dd65f7c38a that changed how we sorted tests to get
better distribution from a qemu running point of view. However we now
have some tests that take a long time to build and we end up loading
build machines we all the long running tests.
Lets try going back to sorting by platform and see how that behaves with
the new CI environment.
Signed-off-by: Kumar Gala <kumar.gala@linaro.org>
Commit 3124c02 ("cmake: flash/debug: refactor runner configuration")
changed the way the hex_file and elf_file inputs in the RunnerConfig
object are created. In particular, they are now host-style paths.
This breaks flashing with openocd on Windows, which doesn't handle that
properly. Fix that by "casting" the internal hex_file and elf_file
attributes to POSIX paths.
Fixes: #28138
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
This code had one purpose only, feed timing information into a test and
was not used by anything else. The custom trace points unfortunatly were
not accurate and this test was delivering informatin that conflicted
with other tests we have due to placement of such trace points in the
architecture and kernel code.
For such measurements we are planning to use the tracing functionality
in a special mode that would be used for metrics without polluting the
architecture and kernel code with additional tracing and timing code.
Furthermore, much of the assembly code used had issues.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Signed-off-by: Daniel Leung <daniel.leung@intel.com>
MWDT toolchain adds additional suffix to sections name in case of
ffunction-sections / fdata-sections are enabled.
As proposed by Andy Ross let's pick a single set of rules
and syntax that work.
Suggested-by: Andy Ross <andy@plausible.org>
Signed-off-by: Eugeniy Paltsev <Eugeniy.Paltsev@synopsys.com>
I've repeatedly seen that people are not aware of the existence of
this configuration option.
I've been using build.pristine=auto daily for years and never had any
problems. I've also asked around on Slack a couple of times over
various points to see if anybody objects to making this change. Nobody
has, so let's just turn it on by default.
Signed-off-by: Martí Bolívar <marti.bolivar@nordicsemi.no>
Tell the EDT instance that properties of the /zephyr,user node should
be generated based on the binding types inferred from the property
content.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
Clean up of devicetree tooling removed generation of information
present in devicetree in nodes that have no compatible, or for extra
properties not defined by a binding. Discussion proposed that these
properties should be allowed, but only in a defined node /zephyr,user.
For that node infer bindings based on the presence of properties.
Signed-off-by: Peter Bigot <peter.bigot@nordicsemi.no>
By default, subprocess.Popen commands are supplied as a list of strings.
Using split() allows to use command with arguments, for example it is
possible to use following:
sanitycheck ... \
--device-serial-pty="set_ambient read_terminal.py" \
...
Signed-off-by: Andrei Emeltchenko <andrei.emeltchenko@intel.com>
