A bug was reported when rtla-timerlat-top tool performs on-threshold
actions, even though no threshold was hit. This is reproduced even if no
threshold is set at all:
$ rtla timerlat top -q -c 0 --on-threshold shell,command='echo BAD'
BAD
Timer Latency
...
The bug is due to incorrect logic in timerlat_top_bpf_main_loop().
The loop uses timerlat_bpf_wait(), the return values of which are:
- > 0 (number of ringbuffer entries): at least 1 CPU hit threshold
- = 0: time out
- < 0: wait was interrupted by a signal
Commit 3138df6f0c ("rtla/timerlat: Exit top main loop on any non-zero
wait_retval") changed the condition for "threshold hit" from
"wait_reval == 1" (exactly 1 CPU hit threshold) to "wait_retval != 0",
to fix a race where multiple CPUs hit the threshold at the same time.
That also made it incorrectly include a signal (< 0), coming from either
duration expired (SIGALRM) or user interrupt (SIGINT).
Check for wait_retval greater than zero in the if condition to cover all
return values correctly.
Fixes: 3138df6f0c ("rtla/timerlat: Exit top main loop on any non-zero wait_retval")
Reported-by: Attila Fazekas <afazekas@redhat.com>
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260713141047.687877-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
rtla started to only link parts of the tools library. It now misses the
ctype information used by all the related string operations. Just add
another single file to make it build again.
Signed-off-by: Bastian Blank <waldi@debian.org>
Fixes: 48209d763c ("rtla: Add libsubcmd dependency")
Link: https://lore.kernel.org/r/ako2S4mzIqWwYuas@steamhammer.waldi.eu.org
[ remove duplicated spaces in commit message ]
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Compiling RTLA 7.1.x with GCC 16 and uClibc as standard library fails
with these errors:
src/common.c: In function ‘set_signals’:
src/common.c:40:17: error: implicit declaration of function ‘alarm’ [-Wimplicit-function-declaration]
40 | alarm(params->duration);
| ^~~~~
src/common.c: In function ‘common_apply_config’:
src/common.c:187:44: error: implicit declaration of function ‘getpid’; did you mean ‘getpt’? [-Wimplicit-function-declaration]
187 | retval = sched_setaffinity(getpid(), sizeof(params->hk_cpu_set),
| ^~~~~~
| getpt
In file included from src/common.c:9:
src/common.c: In function ‘run_tool’:
src/common.c:262:19: error: implicit declaration of function ‘sysconf’; did you mean ‘sscanf’? [-Wimplicit-function-declaration]
262 | nr_cpus = get_nprocs_conf();
| ^~~~~~~~~~~~~~~
src/common.c:262:19: error: ‘_SC_NPROCESSORS_CONF’ undeclared (first use in this function)
262 | nr_cpus = get_nprocs_conf();
| ^~~~~~~~~~~~~~~
src/common.c:262:19: note: each undeclared identifier is reported only once for each function it appears in
src/common.c:370:17: error: implicit declaration of function ‘sleep’ [-Wimplicit-function-declaration]
370 | sleep(1);
| ^~~~~
Restore the missing unistd.h include.
Fixes: 115b06a008 ("tools/rtla: Consolidate nr_cpus usage across all tools")
Signed-off-by: Andreas Ziegler <br025@umbiko.net>
Link: https://lore.kernel.org/r/20260624033342.129970-1-br025@umbiko.net
[ capital letter in patch subject, remove EOL after Fixes ]
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Multiple runtime tests in RTLA rely on the get_workload_pids() shell
helper function to get the PIDs of both kernel and user workloads.
On some systems (e.g. Fedora 43), pgrep matches kernel thread names
including square brackets: "[osnoise/0]"; on other systems (e.g.
RHEL 9.8), brackets are not included: "osnoise/0".
Accept both as valid workload PIDs rather that just the non-bracket form
to make the tests work on all systems.
Fixes: a98dad63cd ("rtla/tests: Add runtime test for -k and -u options")
Reported-by: Crystal Wood <crwood@redhat.com>
Link: https://lore.kernel.org/r/20260604140547.3616495-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
.gitignore includes several entries prone to unwanted matches in
subdirectories. One of them, the recently added "lib/", matches the
recently added directory "tests/scripts/lib/" in addition to the
intended top-level "lib/", which contains object files built from
sources in tools/lib.
Add "/" to all .gitignore entries that are intended to only match
top-level files or directories: rtla, rtla_static, unit_tests,
libsubcmd/.
Remove .gitignore entries that are not needed at all:
- lib/ (contains only object files, ignored by top-level .gitignore
already).
- .txt rtla output files added to .gitignore in commit 02689ae385
("rtla: Add generated output files to gitignore"). Since commit
ad5b50a095 ("rtla/tests: Run runtime tests in temporary directory"),
those are created in a temporary directory, not in tools/tracing/rtla.
Keeping libsubcmd/ as that contains other generated files (headers,
archives, etc.).
Fixes: 48209d763c ("rtla: Add libsubcmd dependency")
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202605291919.eszupseg-lkp@intel.com/
Closes: https://lore.kernel.org/oe-kbuild-all/202605300436.PqQ0Bc8q-lkp@intel.com/
Link: https://lore.kernel.org/r/20260601091835.3118094-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Add a new option, -A/--aligned, that enables timerlat thread alignment
implemented on the kernel-side in commit 4245bf4dc5 ("tracing/osnoise:
Add option to align tlat threads"). The option takes an argument,
representing alignment between timerlat threads in microseconds.
The feature is modeled after the option of the same name in the
cyclictest tool.
Link: https://lore.kernel.org/r/20260527144928.2944472-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
In addition to testing all tool_parse_args() functions, test also all
callbacks used for parsing custom option formats.
The callbacks represent a middle layer between the parsing functions
and utility functions dedicated to checking specific argument formats,
for example, scheduling class and duration. Callback tests are run
before parsing functions to make sure any issue in the former is
reported before it is encountered through the latter.
Tests verify both successful parsing and proper rejection of invalid
inputs (via exit tests). To enable testing static callbacks, a pragma
once guard is added to timerlat.h for safe inclusion by cli_p.h.
Add dependency of UNIT_TESTS_IN on LIBSUBCMD_INCLUDES, as the new test
file tests/unit/cli_opt_callback.c includes cli_p.h which includes
subcmd/parse-options.h.
Link: https://lore.kernel.org/r/20260528103254.2990068-7-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Add a test suite for the _parse_args() function of each tool that checks
the params structures (struct common_params, struct osnoise_params,
struct timerlat_params) returned by them for correctness.
One test case is added per option, as well as a few special cases for
tricky combinations of options. Test cases are ordered the same as the
option arrays and help message to allow easy checking of whether all
options are covered.
This should help clarify what the proper command line behavior of RTLA
is in case there are holes in the documentation and verify that the
intended behavior is implemented correctly.
A few necessary changes to the unit tests were done as part of this
commit:
- Unit tests now also link to libsubcmd and its dependencies.
- A new global variable in_unit_test is added to RTLA's CLI interface,
causing it to skip check for root if running in unit tests. This
allows the CLI unit tests to run as non-root, like existing unit
tests.
There is quite a lot of duplication, some of it is mitigated with macros,
but partially it is intentional so that future changes in behavior are
tracked across tools.
Link: https://lore.kernel.org/r/20260528103254.2990068-6-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Instead of using getopt_long() directly to parse the command line
arguments given to an RTLA tool, use libsubcmd's parse_options().
Utilizing libsubcmd for parsing command line arguments has several
benefits:
- A help message is automatically generated by libsubcmd from the
specification, removing the need of writing it by hand.
- Options are sorted into groups based on which part of tracing (CPU,
thread, auto-analysis, tuning, histogram) they relate to.
- Common parsing patterns for numerical and boolean values now share
code, with the target variable being stored in the option array.
To avoid duplication of the option parsing logic, RTLA-specific
macros defining struct option values are created:
- RTLA_OPT_* for options common to all tools
- OSNOISE_OPT_* and TIMERLAT_OPT_* for options specific to
osnoise/timerlat tools
- HIST_OPT_* macros for options specific to histogram-based tools.
Individual *_parse_args() functions then construct an array out of
these macros that is then passed to libsubcmd's parse_options().
All code specific to command line options parsing is moved out of the
individual tool files into a new file, cli.c, which also contains the
contents of the rtla.c file. A private header, cli_p.h, is added
alongside the public header cli.h, so that unit tests are able to test
statically declared option callbacks.
Minor changes:
- The return value of tool-level help option changes to 129, as this is
the value set by libsubcmd; this is reflected in affected test cases.
The implementation of help for command-level and tracer-level help
is set to 129 as well for consistency, and the change is reflected in
exit value documentation.
- Related to the above, {rtla,osnoise,timerlat}_usage() are marked
__noreturn and exit() is removed from after they are called for
cleaner code.
- The error messages for invalid argument for options --dma-latency and
-E/--entries were corrected, fixing off-by-one in the limits.
Note that unsetting options (using --no-<opt> syntax) is currently not
implemented for options that use custom callbacks. For --irq and
--thread, it will never be implemented, as they conflict with already
existing --no-irq and --no-thread with a different meaning.
Assisted-by: Composer:composer-1.5
Link: https://lore.kernel.org/r/20260528103254.2990068-5-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
In preparation for migrating RTLA to libsubcmd, build libsubcmd from the
appropriate directory next to the RTLA build proper, and link the
resulting object to RTLA.
libsubcmd uses str_error_r() and strlcpy() at several places. To support
these, also link the respective libraries from tools/lib.
For completeness, also add tools/include to include path. This will
allow other userspace functions and macros shipped with the kernel to be
used in RTLA; perf and bpftool, two other users of libsubcmd, already do
that.
To prevent a name conflict, rename RTLA's run_command() function to
run_tool_command(), and replace RTLA's own container_of implementation
with the one in tools/include/linux/container_of.h.
Assisted-by: Composer:composer-1
Link: https://lore.kernel.org/r/20260528103254.2990068-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
In case an action preceding the continue action fails, not only
the continue flag should not be set, it should be unset if it was set
from a previous run of actions_perform().
Add a runtime test to both osnoise and timerlat tools that checks that
this works properly by creating a temporary file.
Link: https://lore.kernel.org/r/20260526102523.2662391-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Create a temporary directory before each test case to serve as working
directory during the duration of the test.
This prevents littering of the original working directory as well as
allows tests to use it to avoid path conflicts.
In order not to break already existing tests, also add a new "testdir"
variable containing the directory where the test file is located. This
is then used to locate artifacts used during testing like BPF programs
and scripts for checking the tracer threads.
Link: https://lore.kernel.org/r/20260526102523.2662391-3-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
In case an action preceding the continue action fails, not only
the continue flag should not be set, it should be unset if it was set
from a previous run of actions_perform().
Add a unit test to check if this is implemented correctly.
Link: https://lore.kernel.org/r/20260526102523.2662391-2-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Currently, actions_perform() only ever sets the continue flag (when
performing the continue action), but never resets it. That leads to
RTLA continuing tracing even if the continue action was not performed in
the current iteration.
For example, the following command:
$ rtla timerlat hist -T 100 --on-threshold shell,command='
echo Spike!
if [ -f /tmp/a ]
then
exit 1
else
touch /tmp/a
fi' --on-threshold continue
should print Spike! at most once, because after hitting the threshold
for the first time, /tmp/a exists, the shell action will fail, and the
continue action is not performed. However, unless /tmp/a exists before
the measurement, it will print Spike! until stopped, as the continue
flag stays set.
Set the continue flag to false in the beginning of actions_perform() to
make RTLA continue only if the action was actually performed.
Fixes: 8d933d5c89 ("rtla/timerlat: Add continue action")
Link: https://lore.kernel.org/r/20260526102523.2662391-1-tglozar@redhat.com
[ correct Fixes tag to include 12 characters of hash ]
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Before, when rtla got a signal, it stopped the main trace but not the
record trace. With "--on-end trace", this can lead to
save_trace_to_file() failing to keep up, especially on a debug kernel.
Plus, it adds post-stoppage noise to the trace file.
Signed-off-by: Crystal Wood <crwood@redhat.com>
Fixes: c73cab9dbe ("rtla/timerlat_hist: Stop timerlat tracer on signal")
Fixes: a4dfce7559 ("rtla/timerlat_top: Stop timerlat tracer on signal")
Fixes: 3aadb65db5 ("rtla/timerlat: Add action on end feature")
Link: https://lore.kernel.org/r/20260512173731.2151841-1-crwood@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Add unit tests covering all functions in the actions module, including
both valid and invalid inputs and all action types, except for
actions_perform(), where only shell and continue actions are tested.
To support testing multiple modules, the unit test build was modified so
that it links the entire rtla-in.o file. For this to work, the main()
function in rtla.c was declared weak, so that the unit test main is able
to override it.
Other included minor changes to unit tests are:
- Make unit test output verbose to show which tests are being run, now
that we have more than 3 tests.
- Add unit_tests file to .gitignore.
- Split unit test sources to one file per test suite, and keep only
main() function in unit_tests.c.
- Fix Makefile dependencies so that "make unit-tests" will rebuild the
binary with the changes in the commit.
Also with the linking the entire rtla-in.o file, it now has rtla's
nr_cpus symbol, so the declaration in utils unit tests is made extern.
Assisted-by: Composer:composer-2-fast
Link: https://lore.kernel.org/r/20260424140244.958495-1-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Add a new script check-cgroup-match.sh that retrieves the cgroup of the
main rtla process and compares it to the cgroup of the rtla workload
threads.
Add a new test based on this script, for both osnoise and timerlat
tools, testing the variant of -C without argument (which sets the cgroup
of the workload to the cgroup of the rtla main process).
Note that this has to be tested in kernel mode to be significant for
timerlat tool, as user workloads inherit the parent rtla process cgroup
even without the option.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-10-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Add runtime test for rtla-timerlat's -k/--kernel-threads and
-u/--user-threads options using get_workload_pids.sh to check whether
the appropriate threads are being created.
The tests are implemented for both top and hist. Additionally, all tests
related to timerlat threads are moved to a separate section in the test
files. The latter is also done for rtla-osnoise tests.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-9-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Cover all options regarding histogram formatting for both
rtla-osnoise-hist and rtla-timerlat-hist tools. All options also have
output checking using positive or negative match, except for
-b/--bucket-size and -E/--entries, which cannot be tested in isolated
due to the output depending on the actual data collected.
Old -E/--entries test for rtla-osnoise was replaced with a new one
equivalent to the timerlat one.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-7-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
rtla-timerlat-top's --aa-only option is currently only tested for return
value.
Extend the tests to also check that only auto-analysis is being done via
a negative match for the "Timer Latency" text in the top header, and
further split the test case into two:
- one test case for --aa-only stopping on threshold
- one test case for --aa-only exiting without threshold being hit
For both cases, the expected output ("analyzing it" or "Max latency was"
respectively) is checked against in addition to the negative match.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-6-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
RTLA runtime tests verify the -c/--cpus options, but do not check
whether the correct affinity is actually applied.
Add a script named check-cpus.sh that retrieves the affinity of all
workload threads and use it to check the -c/--cpus option for both
osnoise and timerlat tools.
Also add missing -c/--cpus test for osnoise.
Reviewed-by: Wander Lairson Costa <wander@redhat.com>
Link: https://lore.kernel.org/r/20260423130558.882022-4-tglozar@redhat.com
Signed-off-by: Tomas Glozar <tglozar@redhat.com>
Tomas Glozar
Bastian BlankandTomas Glozar
Andreas ZieglerandTomas Glozar
Crystal WoodandTomas Glozar