It is possible to get qtest to read fuzzer reproducers from a file
rather than directly from stdio; this is useful when you want to run
QEMU under gdb to debug the failure. Document how to do this, which
was previously only written down in the commit message for
5b18a6bf44 ("chardev: Allow setting file chardev input file on the
command line").
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
Message-id: 20251028165236.3327658-1-peter.maydell@linaro.org
The QAPI_TEST_UPDATE env var can be set when running the QAPI
schema tests to regenerate the reference output. For consistent
naming with the tracetool test, change the env var name to
QEMU_TEST_REGENERATE.
The test is modified to provide a hint about use of the new
env var and it is also added to the developer documentation.document its usage.
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-id: 20250916081638.764020-8-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
When reviewing tracetool patches it is often very unclear what the
expected output will be for the generated backends. Compounding
this is that a default build will only enable the 'log' trace
backend, so developers won't see generated code for other backends
without making a special effort. Some backends are also platform
specific, so can't be enabled in QEMU builds, even though tracetool
could generate the code.
To address this, introduce a test suite for tracetool which is
conceptually similar to the qapi-schema test. It is a simple
python program that runs tracetool and compares the actual output
to historical reference output kept in git. The test directly
emits TAP format logs for ease of integration with meson.
This can be run with
make check-tracetool
to make it easier for developers changing generated output, the
sample expected content can be auto-recreated
QEMU_TEST_REGENERATE=1 make check-tracetool
and the changes reviewed and added to the commit. This will also
assist reviewers interpreting the change.
Developers are reminded of this in the test output on failure:
$ make check-tracetool
1/6 qemu:tracetool / dtrace OK 0.14s
2/6 qemu:tracetool / ftrace FAIL 0.06s exit status 1
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
1..2
ok 1 - ftrace.c
#
not ok 1 - ftrace.h (set QEMU_TEST_REGENERATE=1 to recreate reference output if tracetool generator was intentionally changed)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
3/6 qemu:tracetool / log OK 0.06s
4/6 qemu:tracetool / simple OK 0.06s
5/6 qemu:tracetool / syslog OK 0.06s
6/6 qemu:tracetool / ust OK 0.11s
Summary of Failures:
2/6 qemu:tracetool / ftrace FAIL 0.06s exit status 1
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20250916081638.764020-6-berrange@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Now that all Avocado tests have been converted to or been replaced by
other functional tests, we can delete the remainders of the Avocado
tests from the QEMU source tree.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250414113031.151105-16-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
This file was meant for defining the vocabulary for our testing
efforts, but it did not age well: First, the definitions are not
only about the CI part, but also about testing in general, so most
of the information should rather reside in main.rst instead.
Second, some vocabulary has never been really adopted by the QEMU
project, for example we never really use the word "system testing"
since "system" rather means the system emulator binaries in the
QEMU project (and we also don't do any testing with other components
like libvirt and virt-managers here). It also defines that the qtests
are the "functional" tests in QEMU, which is not really up to date
anymore after the "tests/functional" framework has been introduced
a couple of months ago (FWIW, the qtests could rather be seen as a
mix between unit testing and functional testing).
To solve this problem, move the useful parts of this file into
main.rst and directly into ci.rst, and drop the ones (like "system
testing") that we don't really need anymore.
Message-ID: <20250314085959.1585568-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Since we don't run the Avocado jobs in the CI anymore, rename
these variables to QEMU_JOB_FUNCTIONAL and QEMU_CI_FUNCTIONAL.
Also, there was a mismatch between the documentation and the
implementation of QEMU_CI_AVOCADO_TESTING: While the documentation
said that you had to "Set this variable to have the tests using the
Avocado framework run automatically", you indeed needed to set it
to make the pipelines appear in your dashboard - but they were never
run automatically in forks and had to be triggered manually. Let's
improve this now: No need to hide these pipelines from the users
by default anymore (the functional tests should be stable enough
nowadays), and rather allow the users to run the pipelines auto-
matically with this switch now instead, as was documented.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250414113031.151105-15-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The 'qemu_bin' field is currently set on the class, despite being
accessed as if it were an object instance field with 'self.qemu_bin'.
This is no obvious need to have it as a class field, so move it into
the object instance.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-ID: <20250228102738.3064045-3-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Some tests have a very long runtime and might run into timeout issues
e.g. when QEMU has been compiled with --enable-debug. Add a decorator
for marking them more easily. Rename the corresponding environment
variable to be more in sync with the other QEMU_TEST_ALLOW_* switches
that we already have, and add a paragraph about it in the documentation.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250128152839.184599-2-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Make it clear that the commands have to be run from the folder with the
build, and use the python3 from our pyvenv to make sure that the
pycotap module is available.
Message-ID: <20241112115302.470527-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Support the QEMU_TEST_QMP_BACKDOOR=backdoor.sock env variable as a
way to get a QMP backdoor for debugging a stalled QEMU test. Most
typically this would be used if running the tests directly:
$ QEMU_TEST_QMP_BACKDOOR=backdoor.sock \
QEMU_TEST_QEMU_BINARY=./build/qemu-system-arm \
PYTHONPATH=./python \
./tests/functional/test_arm_tuxrun.py
And then, when the test stalls, in a second shell run:
$ ./scripts/qmp/qmp-shell backdoor.sock
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-18-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-18-alex.bennee@linaro.org>
The build/tests/functional subdirectories are consuming huge amounts
of disk space.
Split the location for scratch files into a 'scratch' sub-directory,
separate from log files, and delete it upon completion of each test.
The new env variable QEMU_TEST_KEEP_SCRATCH can be set to preserve
this scratch dir for debugging access if required.
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-3-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-3-alex.bennee@linaro.org>
Replace the footnotes with inline links whenever the footnote text
consists of nothing but the URL. While at it, make the link texts
consistent in the surrounding areas, for example avoiding usage of
"here" for the link's text.
In the case of acpi-bits.rst this fixes a build failure with Sphinx
8.1.0, because the FOSDEM link was duplicated in the paragraph and the
new version is a lot stricter about unreferenced footnotes.
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Alex Bennée
Peter Maydell
Daniel P. BerrangéandStefan Hajnoczi
Manos PitsidianakisandAlex Bennée
Gustavo RomeroandMichael Tokarev
Aditya GuptaandThomas Huth
Pierrick BouvierandAlex Bennée
Paolo Bonzini
Matheus Tavares BernardinoandThomas Huth