coccicheck prints debug logs to stdout unless a debug file has been set.
This makes it hard to read coccinelle's suggested changes, especially
for someone new to coccicheck.
From this commit, we warn about this behaviour from within the script on
an unset debug file. Explicitly setting the debug file to /dev/null
suppresses the warning while keeping the default.
Signed-off-by: Benjamin Philip <benjamin.philip495@gmail.com>
Signed-off-by: Julia Lawall <julia.lawall@inria.fr>
This commit separates handling unset files and pre-existing files. It
also eliminates a duplicated check for unset files in run_cmd_parmap().
Signed-off-by: Benjamin Philip <benjamin.philip495@gmail.com>
Signed-off-by: Julia Lawall <julia.lawall@inria.fr>
Enhance the coccicheck script to filter *.cocci files based on the
specified MODE (e.g., report, patch). This ensures that only compatible
semantic patch files are executed, preventing errors such as:
"virtual rule report not supported"
This error occurs when a .cocci file does not define a 'virtual <MODE>'
rule, yet is executed in that mode.
For example:
make coccicheck M=drivers/hwtracing/coresight/ MODE=report
In this case, running "secs_to_jiffies.cocci" would trigger the error
because it lacks support for 'report' mode. With this change, such files
are skipped automatically, improving robustness and developer
experience.
Signed-off-by: Songwei Chai <quic_songchai@quicinc.com>
Reviewed-by: Julia Lawall <Julia.Lawall@inria.fr>
Currently, Kbuild always operates in the output directory of the kernel,
even when building external modules. This increases the risk of external
module Makefiles attempting to write to the kernel directory.
This commit switches the working directory to the external module
directory, allowing the removal of the $(KBUILD_EXTMOD)/ prefix from
some build artifacts.
The command for building external modules maintains backward
compatibility, but Makefiles that rely on working in the kernel
directory may break. In such cases, $(objtree) and $(srctree) should
be used to refer to the output and source directories of the kernel.
The appearance of the build log will change as follows:
[Before]
$ make -C /path/to/my/linux M=/path/to/my/externel/module
make: Entering directory '/path/to/my/linux'
CC [M] /path/to/my/externel/module/helloworld.o
MODPOST /path/to/my/externel/module/Module.symvers
CC [M] /path/to/my/externel/module/helloworld.mod.o
CC [M] /path/to/my/externel/module/.module-common.o
LD [M] /path/to/my/externel/module/helloworld.ko
make: Leaving directory '/path/to/my/linux'
[After]
$ make -C /path/to/my/linux M=/path/to/my/externel/module
make: Entering directory '/path/to/my/linux'
make[1]: Entering directory '/path/to/my/externel/module'
CC [M] helloworld.o
MODPOST Module.symvers
CC [M] helloworld.mod.o
CC [M] .module-common.o
LD [M] helloworld.ko
make[1]: Leaving directory '/path/to/my/externel/module'
make: Leaving directory '/path/to/my/linux'
Printing "Entering directory" twice is cumbersome. This will be
addressed later.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nicolas Schier <n.schier@avm.de>
If bash is not located under /bin, coccicheck fails to run. In the real
world, this happens for instance when NixOS is used in the host. Instead,
use /usr/bin/env to locate the executable binary for bash.
Signed-off-by: Jarkko Sakkinen <jarkko@kernel.org>
Tested-by: Deepak R Varma <drv@mailo.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
The latest version of grep claims that egrep is now obsolete so the build
now contains warnings that look like:
egrep: warning: egrep is obsolescent; using grep -E
fix this up by moving the vdso Makefile to use "grep -E" instead.
Cc: Julia Lawall <Julia.Lawall@inria.fr>
Cc: Nicolas Palix <nicolas.palix@imag.fr>
Cc: cocci@inria.fr
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
When LANG is not set to English, the logic which checks the
number of CPUs fail, as the messages can be localized, and
the logic at:
THREADS_PER_CORE=$(lscpu | grep "Thread(s) per core: " | tr -cd "[:digit:]")
will not get the number of threads per core.
This causes the script to not run properly, as it will produce
a warning:
$ make coccicheck COCCI=$PWD/scripts/coccinelle/misc/add_namespace.cocci MODE=report drivers/media/
./scripts/coccicheck: linha 93: [: número excessivo de argumentos
Fix it by forcing LANG=C when calling lscpu.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
The command "make coccicheck C=1 CHECK=scripts/coccicheck" results in the
error:
./scripts/coccicheck: line 65: -1: shift count out of range
This happens because every time the C variable is specified,
the shell arguments need to be "shifted" in order to take only
the last argument, which is the C file to test. These shell arguments
mostly comprise flags that have been set in the Makefile. However,
when coccicheck is specified in the make command as a rule, the
number of shell arguments is zero, thus passing the invalid value -1
to the shift command, resulting in an error.
Modify coccicheck to print correct usage of make coccicheck so as to
avoid the error.
Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
scripts/ld-version.sh was, as its file name implies, originally intended
for the GNU ld version, but is (ab)used for the spatch version too.
Use 'sort -CV' for the version comparison, then coccicheck does not need
to use scripts/ld-version.sh. Fix nsdeps as well.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
Currently, Coccinelle uses at most one thread per core by default in
machines with more than 2 hyperthreads. However, for systems with only 4
hyperthreads, this does not improve performance.
Modify coccicheck to use all available threads in machines with
upto 4 hyperthreads.
Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
While fetching the number of threads per core with lscpu,
the [:digit:] set is used for translation of digits from 0-9.
However, using [:digit:] instead of "[:digit:]" does not seem
to work uniformly for some shell types and configurations
(such as zsh).
Therefore, modify coccicheck to use double quotes around the
[:digit:] set for uniformity and better portability.
Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
By default, coccicheck utilizes all available threads to implement
parallelisation. However, when all available threads are used,
a decrease in performance is noted. The elapsed time is minimum
when at most one thread per core is used.
For example, on benchmarking the semantic patch kfree.cocci for
usb/serial using hyperfine, the outputs obtained for J=5 and J=2
are 1.32 and 1.90 times faster than those for J=10 and J=9
respectively for two separate runs. For the larger drivers/staging
directory, minimium elapsed time is obtained for J=3 which is 1.86
times faster than that for J=12. The optimal J value does not
exceed 6 in any of the test runs. The benchmarks are run on a machine
with 6 cores, with 2 threads per core, i.e, 12 hyperthreads in all.
To improve performance, modify coccicheck to use at most only
one thread per core by default.
Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
Currently, coccicheck fails with only the message "coccicheck failed"
and the error code for the failure. To obtain the error logs,
one needs to specify a debug file using the DEBUG_FILE option.
Modify coccicheck to display error logs when it crashes unless
DEBUG_FILE is set, in which case, the error logs are stored in
the specified debug file.
Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
This patch adds chain mode to the list of available modes in coccicheck.
Signed-off-by: Sumera Priyadarsini <sylphrenadin@gmail.com>
Signed-off-by: Julia Lawall <Julia.Lawall@inria.fr>
If coccicheck fails, it should return an error code distinct from zero
to signal about an internal problem. Current code instead of exiting with
the tool's error code returns the error code of 'echo "coccicheck failed"'
which is almost always equals to zero, thus failing the original intention
of alerting about a problem. This patch fixes the code.
Found by Linux Driver Verification project (linuxtesting.org).
Signed-off-by: Denis Efremov <efremov@linux.com>
Acked-by: Julia Lawall <julia.lawall@lip6.fr>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
The command "make -j8 C=1 CHECK=scripts/coccicheck" produces
lots of "coccicheck failed" error messages.
Julia Lawall explained the Coccinelle behavior as follows:
"The problem on the Coccinelle side is that it uses a subdirectory
with the name of the semantic patch to store standard output and
standard error for the different threads. I didn't want to use a
name with the pid, so that one could easily find this information
while Coccinelle is running. Normally the subdirectory is cleaned
up when Coccinelle completes, so there is only one of them at a time.
Maybe it is best to just add the pid. There is the risk that these
subdirectories will accumulate if Coccinelle crashes in a way such
that they don't get cleaned up, but Coccinelle could print a warning
if it detects this case, rather than failing."
When scripts/coccicheck is used as CHECK tool and -j option is given
to Make, the whole of build process runs in parallel. So, multiple
processes try to get access to the same subdirectory.
I notice spatch creates the subdirectory only when it runs in parallel
(i.e. --jobs <N> is given and <N> is greater than 1).
Setting NPROC=1 is a reasonable solution; spatch does not create the
subdirectory. Besides, ONLINE=1 mode takes a single file input for
each spatch invocation, so there is no reason to parallelize it in
the first place.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Julia Lawall <Julia.Lawall@lip6.fr>
Pull Kbuild misc updates from Masahiro Yamada:
- Clean up and fix RPM package build
- Fix a warning in DEB package build
- Improve coccicheck script
- Improve some semantic patches
* tag 'kbuild-misc-v4.15' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild:
docs: dev-tools: coccinelle: delete out of date wiki reference
coccinelle: orplus: reorganize to improve performance
coccinelle: use exists to improve efficiency
builddeb: Pass the kernel:debarch substvar to dpkg-genchanges
Coccinelle: use false positive annotation
coccinelle: fix verbose message about .cocci file being run
coccinelle: grep Options and Requires fields more precisely
Coccinelle: make DEBUG_FILE option more useful
coccinelle: api: detect identical chip data arrays
coccinelle: Improve setup_timer.cocci matching
Coccinelle: setup_timer: improve messages from setup_timer
kbuild: rpm-pkg: do not force -jN in submake
kbuild: rpm-pkg: keep spec file until make mrproper
kbuild: rpm-pkg: fix jobserver unavailable warning
kbuild: rpm-pkg: replace $RPM_BUILD_ROOT with %{buildroot}
kbuild: rpm-pkg: fix build error when CONFIG_MODULES is disabled
kbuild: rpm-pkg: refactor mkspec with here doc
kbuild: rpm-pkg: clean up mkspec
kbuild: rpm-pkg: install vmlinux.bz2 unconditionally
kbuild: rpm-pkg: remove ppc64 specific image handling
If you run coccicheck with V=1 and COCCI=, you will see a strange
path to the semantic patch file. For example, run the following:
$ make V=1 COCCI=scripts/coccinelle/free/kfree.cocci coccicheck
[ snip ]
The semantic patch that makes this report is available
in scriptcoccinelle/free/kfree.cocci.
Notice "s/" was dropped from "scripts/coccinelle/free/kfree.cocci".
When running coccicheck without O=, $srctree is expanded to ".", which
represents one arbitrary character in the regular expression. Using
sed is not a good choice here. Strip $srctree/ simply without sed.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Currently, the required version for badzero.cocci is picked up from
its "Comments:" line since it contains the word "Requires".
Surprisingly, ld-version.sh can extract the version number from the
string "Requires Coccinelle version 1.0.0-rc20 or later", but this
expectation is fragile. Fix the .cocci file. I removed "-rc20"
because ld-version.sh cannot handle it.
Make the coccicheck script to see exact patterns for "Options:" and
"Requires:" in order to avoid accidental matching to what just happens
to appear in comment lines.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Julia Lawall <julia.lawall@lip6.fr>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Make coccicheck checked for the existence of DEBUG_FILE on each semantic
patch, and bailed if it already existed. This meant that DEBUG_FILE was
useless for checking more than one semantic patch at a time. Now the check
is moved to the start of make coccicheck, and the 2> is changed to a 2>> to
append to the file on each semantic patch. Furthermore, the spatch command
that is run for each semantic patch is also added to the DEBUG_FILE, to
make clear what each stdout trace corresponds to.
Signed-off-by: Julia Lawall <Julia.Lawall@lip6.fr>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Use the current name (in a comment at the beginning of this script) for
the file which was converted to the documentation format "reStructuredText"
in August 2016.
Fixes: 4b9033a334 ("docs: sphinxify coccinelle.txt and add it to dev-tools")
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
Acked-by: Julia Lawall <julia.lawall@lip6.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
Refer to the Documentation/coccinelle.txt and supplemental documentation
on the wiki:
https://bottest.wiki.kernel.org/coccicheck
This page shall always refer to the linux-next iteration of scripts/coccicheck.
v4: only refer to the wiki as supplemental documentation, and also
update Documentation/coccinelle.txt.
Signed-off-by: Luis R. Rodriguez <mcgrof@kernel.org>
Acked-by: Nicolas Palix <nicolas.palix@imag.fr>
Signed-off-by: Michal Marek <mmarek@suse.com>
