GNU Make manual says:
$?
The names of all the prerequisites that are newer than the target,
with spaces between them.
To reflect this, rename any-prereq to newer-prereqs, which is clearer
and more intuitive.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
The incremental build of Linux kernel is pretty slow when lots of
objects are compiled. The rebuild of allmodconfig may take a few
minutes even when none of the objects needs to be rebuilt.
The time-consuming part in the incremental build is the evaluation of
if_changed* macros since they are used in the recipes to compile C and
assembly source files into objects.
I notice the following code in if_changed* is expensive:
$(filter-out $(PHONY) $(wildcard $^),$^)
In the incremental build, every object has its .*.cmd file, which
contains the auto-generated list of included headers. So, $^ are
expanded into the long list of the source file + included headers,
and $(wildcard $^) checks whether they exist.
It may not be clear why this check exists there.
Here is the record of my research.
[1] The first code addition into Kbuild
This code dates back to 2002. It is the pre-git era. So, I copy-pasted
it from the historical git tree.
| commit 4a6db0791528c220655b063cf13fefc8470dbfee (HEAD)
| Author: Kai Germaschewski <kai@tp1.ruhr-uni-bochum.de>
| Date: Mon Jun 17 00:22:37 2002 -0500
|
| kbuild: Handle removed headers
|
| New and old way to handle dependencies would choke when a file
| #include'd by other files was removed, since the dependency on it was
| still recorded, but since it was gone, make has no idea what to do about
| it (and would complain with "No rule to make <file> ...")
|
| We now add targets for all the previously included files, so make will
| just ignore them if they disappear.
|
| diff --git a/Rules.make b/Rules.make
| index 6ef827d3df39..7db5301ea7db 100644
| --- a/Rules.make
| +++ b/Rules.make
| @@ -446,7 +446,7 @@ if_changed = $(if $(strip $? \
| # execute the command and also postprocess generated .d dependencies
| # file
|
| -if_changed_dep = $(if $(strip $? \
| +if_changed_dep = $(if $(strip $? $(filter-out FORCE $(wildcard $^),$^)\
| $(filter-out $(cmd_$(1)),$(cmd_$@))\
| $(filter-out $(cmd_$@),$(cmd_$(1)))),\
| @set -e; \
| diff --git a/scripts/fixdep.c b/scripts/fixdep.c
| index b5d7bee8efc7..db45bd1888c0 100644
| --- a/scripts/fixdep.c
| +++ b/scripts/fixdep.c
| @@ -292,7 +292,7 @@ void parse_dep_file(void *map, size_t len)
| exit(1);
| }
| memcpy(s, m, p-m); s[p-m] = 0;
| - printf("%s: \\\n", target);
| + printf("deps_%s := \\\n", target);
| m = p+1;
|
| clear_config();
| @@ -314,7 +314,8 @@ void parse_dep_file(void *map, size_t len)
| }
| m = p + 1;
| }
| - printf("\n");
| + printf("\n%s: $(deps_%s)\n\n", target, target);
| + printf("$(deps_%s):\n", target);
| }
|
| void print_deps(void)
The "No rule to make <file> ..." error can be solved by passing -MP to
the compiler, but I think the detection of header removal is a good
feature. When a header is removed, all source files that previously
included it should be re-compiled. This makes sure we has correctly
got rid of #include directives of it.
This is also related with the behavior of $?. The GNU Make manual says:
$?
The names of all the prerequisites that are newer than the target,
with spaces between them.
This does not explain whether a non-existent prerequisite is considered
to be newer than the target.
At this point of time, GNU Make 3.7x was used, where the $? did not
include non-existent prerequisites. Therefore,
$(filter-out FORCE $(wildcard $^),$^)
was useful to detect the header removal, and to rebuild the related
objects if it is the case.
[2] Change of $? behavior
Later, the behavior of $? was changed (fixed) to include prerequisites
that did not exist.
First, GNU Make commit 64e16d6c00a5 ("Various changes getting ready for
the release of 3.81.") changed it, but in the release test of 3.81, it
turned out to break the kernel build.
See these:
- http://lists.gnu.org/archive/html/bug-make/2006-03/msg00003.html
- https://savannah.gnu.org/bugs/?16002
- https://savannah.gnu.org/bugs/?16051
Then, GNU Make commit 6d8d9b74d9c5 ("Numerous updates to tests for
issues found on Cygwin and Windows.") reverted it for the 3.81 release
to give Linux kernel time to adjust to the new behavior.
After the 3.81 release, GNU Make commit 7595f38f62af ("Fixed a number
of documentation bugs, plus some build/install issues:") re-added it.
[3] Adjustment to the new $? behavior on Kbuild side
Meanwhile, the kernel build was changed by commit 4f1933620f ("kbuild:
change kbuild to not rely on incorrect GNU make behavior") to adjust to
the new $? behavior.
[4] GNU Make 3.82 released in 2010
GNU Make 3.82 was the first release that integrated the correct $?
behavior. At this point, Kbuild dealt with GNU Make versions with
different $? behaviors.
3.81 or older:
$? does not contain any non-existent prerequisite.
$(filter-out $(PHONY) $(wildcard $^),$^) was useful to detect
removed include headers.
3.82 or newer:
$? contains non-existent prerequisites. When a header is removed,
it appears in $?. $(filter-out $(PHONY) $(wildcard $^),$^) became
a redundant check.
With the correct $? behavior, we could have dropped the expensive
check for 3.82 or later, but we did not. (Maybe nobody noticed this
optimization.)
[5] The .SECONDARY special target trips up $?
Some time later, I noticed $? did not work as expected under some
circumstances. As above, $? should contain non-existent prerequisites,
but the ones specified as SECONDARY do not appear in $?.
I asked this in GNU Make ML, and it seems a bug:
https://lists.gnu.org/archive/html/bug-make/2019-01/msg00001.html
Since commit 8e9b61b293 ("kbuild: move .SECONDARY special target to
Kbuild.include"), all files, including headers listed in .*.cmd files,
are treated as secondary.
So, we are back into the incorrect $? behavior.
If we Kbuild want to react to the header removal, we need to keep
$(filter-out $(PHONY) $(wildcard $^),$^) but this makes the rebuild
so slow.
[Summary]
- I believe noticing the header removal and recompiling related objects
is a nice feature for the build system.
- If $? worked correctly, $(filter-out $(PHONY),$?) would be enough
to detect the header removal.
- Currently, $? does not work correctly when used with .SECONDARY,
and Kbuild is hit by this bug.
- I filed a bug report for this, but not fixed yet as of writing.
- Currently, the header removal is detected by the following expensive
code:
$(filter-out $(PHONY) $(wildcard $^),$^)
- I do not want to revert commit 8e9b61b293 ("kbuild: move
.SECONDARY special target to Kbuild.include"). Specifying
.SECONDARY globally is clean, and it matches to the Kbuild policy.
This commit proactively removes the expensive check since it makes the
incremental build faster. A downside is Kbuild will no longer be able
to notice the header removal.
You can confirm it by the full-build followed by a header removal, and
then re-build.
$ make defconfig all
[ full build ]
$ rm include/linux/device.h
$ make
CALL scripts/checksyscalls.sh
CALL scripts/atomic/check-atomics.sh
DESCEND objtool
CHK include/generated/compile.h
Kernel: arch/x86/boot/bzImage is ready (#11)
Building modules, stage 2.
MODPOST 12 modules
Previously, Kbuild noticed a missing header and emits a build error.
Now, Kbuild is fine with it. This is an unusual corner-case, not a big
deal. Once the $? bug is fixed in GNU Make, everything will work fine.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Commit 40df759e2b ("kbuild: Fix build with binutils <= 2.19")
introduced ar-option and KBUILD_ARFLAGS to deal with old binutils.
According to Documentation/process/changes.rst, the current minimal
supported version of binutils is 2.21 so you can assume the 'D' option
is always supported. Not only GNU ar but also llvm-ar supports it.
With the 'D' option hard-coded, there is no more user of ar-option
or KBUILD_ARFLAGS.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Tested-by: Nick Desaulniers <ndesaulniers@google.com>
Commit 415008af32 ("docs-rst: convert lsm from DocBook to ReST")
removed the last users of this macro.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Pull Kbuild updates from Masahiro Yamada:
- remove headers_{install,check}_all targets
- remove unreasonable 'depends on !UML' from CONFIG_SAMPLES
- re-implement 'make headers_install' more cleanly
- add new header-test-y syntax to compile-test headers
- compile-test exported headers to ensure they are compilable in
user-space
- compile-test headers under include/ to ensure they are self-contained
- remove -Waggregate-return, -Wno-uninitialized, -Wno-unused-value
flags
- add -Werror=unknown-warning-option for Clang
- add 128-bit built-in types support to genksyms
- fix missed rebuild of modules.builtin
- propagate 'No space left on device' error in fixdep to Make
- allow Clang to use its integrated assembler
- improve some coccinelle scripts
- add a new flag KBUILD_ABS_SRCTREE to request Kbuild to use absolute
path for $(srctree).
- do not ignore errors when compression utility is missing
- misc cleanups
* tag 'kbuild-v5.3' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild: (49 commits)
kbuild: use -- separater intead of $(filter-out ...) for cc-cross-prefix
kbuild: Inform user to pass ARCH= for make mrproper
kbuild: fix compression errors getting ignored
kbuild: add a flag to force absolute path for srctree
kbuild: replace KBUILD_SRCTREE with boolean building_out_of_srctree
kbuild: remove src and obj from the top Makefile
scripts/tags.sh: remove unused environment variables from comments
scripts/tags.sh: drop SUBARCH support for ARM
kbuild: compile-test kernel headers to ensure they are self-contained
kheaders: include only headers into kheaders_data.tar.xz
kheaders: remove meaningless -R option of 'ls'
kbuild: support header-test-pattern-y
kbuild: do not create wrappers for header-test-y
kbuild: compile-test exported headers to ensure they are self-contained
init/Kconfig: add CONFIG_CC_CAN_LINK
kallsyms: exclude kasan local symbols on s390
kbuild: add more hints about SUBDIRS replacement
coccinelle: api/stream_open: treat all wait_.*() calls as blocking
coccinelle: put_device: Add a cast to an expression for an assignment
coccinelle: put_device: Adjust a message construction
...
arch/mips/Makefile passes prefixes that start with '-' to cc-cross-prefix
when $(tool-archpref) evaluates to the empty string.
They are filtered-out before the $(shell ...) invocation. Otherwise,
'command -v' would be confused.
$ command -v -linux-gcc
bash: command: -l: invalid option
command: usage: command [-pVv] command [arg ...]
Since commit 913ab9780f ("kbuild: use more portable 'command -v' for
cc-cross-prefix"), cc-cross-prefix throws away the stderr output, so
the console is not polluted in any way.
This is not a big deal in practice, but I see a slightly better taste
in adding '--' to teach it that '-linux-gcc' is an argument instead of
a command option.
This will cause extra forking of subshell, but it will not be noticeable
performance regression.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
The string returned by $(filter-out ...) does not contain any leading
or trailing spaces.
With the previous commit, 'any-prereq' no longer contains any
excessive spaces.
Nor does 'cmd-check' since it expands to a $(filter-out ...) call.
So, only the space that matters is the one between 'any-prereq'
and 'cmd-check'.
By removing it from the code, we can save $(strip ...) evaluation.
This refactoring is possible because $(any-prereq)$(cmd-check) is only
passed to the first argument of $(if ...), so we are only interested
in whether or not it is empty.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
The string returned by $(filter-out ...) does not contain any leading
or trailing spaces.
So, only the space that matters is the one between
$(filter-out $(PHONY),$?)
and
$(filter-out $(PHONY) $(wildcard $^),$^)
By removing it from the code, we can save $(strip ...) evaluation.
This refactoring is possible because $(any-prereq) is only passed to
the first argument of $(if ...), so we are only interested in whether
or not it is empty.
This is also the prerequisite for the next commit.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
I prefer 'cmd-check' for consistency.
We have 'echo-cmd', 'cmd', 'cmd_and_fixdep', etc. in this file.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Now that hdr-inst is used only in the top Makefile, move it there
from scripts/Kbuild.include.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
The kbuild documentation clearly shows that the documents
there are written at different times: some use markdown,
some use their own peculiar logic to split sections.
Convert everything to ReST without affecting too much
the author's style and avoiding adding uneeded markups.
The conversion is actually:
- add blank lines and identation in order to identify paragraphs;
- fix tables markups;
- add some lists markups;
- mark literal blocks;
- adjust title markups.
At its new index.rst, let's add a :orphan: while this is not linked to
the main index.rst file, in order to avoid build warnings.
Signed-off-by: Mauro Carvalho Chehab <mchehab+samsung@kernel.org>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
To print the pathname that will be used by shell in the current
environment, 'command -v' is a standardized way. [1]
'which' is also often used in scripts, but it is less portable.
When I worked on commit bd55f96fa9 ("kbuild: refactor cc-cross-prefix
implementation"), I was eager to use 'command -v' but it did not work.
(The reason is explained below.)
I kept 'which' as before but got rid of '> /dev/null 2>&1' as I
thought it was no longer needed. Sorry, I was wrong.
It works well on my Ubuntu machine, but Alexey Brodkin reports noisy
warnings on CentOS7 when 'which' fails to find the given command in
the PATH environment.
$ which foo
which: no foo in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
Given that behavior of 'which' depends on system (and it may not be
installed by default), I want to try 'command -v' once again.
The specification [1] clearly describes the behavior of 'command -v'
when the given command is not found:
Otherwise, no output shall be written and the exit status shall reflect
that the name was not found.
However, we need a little magic to use 'command -v' from Make.
$(shell ...) passes the argument to a subshell for execution, and
returns the standard output of the command.
Here is a trick. GNU Make may optimize this by executing the command
directly instead of forking a subshell, if no shell special characters
are found in the command and omitting the subshell will not change the
behavior.
In this case, no shell special character is used. So, Make will try
to run it directly. However, 'command' is a shell-builtin command,
then Make would fail to find it in the PATH environment:
$ make ARCH=m68k defconfig
make: command: Command not found
make: command: Command not found
make: command: Command not found
In fact, Make has a table of shell-builtin commands because it must
ask the shell to execute them.
Until recently, 'command' was missing in the table.
This issue was fixed by the following commit:
| commit 1af314465e5dfe3e8baa839a32a72e83c04f26ef
| Author: Paul Smith <psmith@gnu.org>
| Date: Sun Nov 12 18:10:28 2017 -0500
|
| * job.c: Add "command" as a known shell built-in.
|
| This is not a POSIX shell built-in but it's common in UNIX shells.
| Reported by Nick Bowler <nbowler@draconx.ca>.
Because the latest release is GNU Make 4.2.1 in 2016, this commit is
not included in any released versions. (But some distributions may
have back-ported it.)
We need to trick Make to spawn a subshell. There are various ways to
do so:
1) Use a shell special character '~' as dummy
$(shell : ~; command -v $(c)gcc)
2) Use a variable reference that always expands to the empty string
(suggested by David Laight)
$(shell command$${x:+} -v $(c)gcc)
3) Use redirect
$(shell command -v $(c)gcc 2>/dev/null)
I chose 3) to not confuse people. The stderr would not be polluted
anyway, but it will provide extra safety, and is easy to understand.
Tested on Make 3.81, 3.82, 4.0, 4.1, 4.2, 4.2.1
[1] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/command.html
Fixes: bd55f96fa9 ("kbuild: refactor cc-cross-prefix implementation")
Cc: linux-stable <stable@vger.kernel.org> # 5.1
Reported-by: Alexey Brodkin <abrodkin@synopsys.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Tested-by: Alexey Brodkin <abrodkin@synopsys.com>
Add SPDX license identifiers to all Make/Kconfig files which:
- Have no license information of any form
These files fall under the project license, GPL v2 only. The resulting SPDX
license identifier is:
GPL-2.0
Reported-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
If you want to see if your linker supports a certain flag, then ask the
linker directly with ld-option (not the compiler with cc-ldoption).
Checking for linker flag support is an antipattern that complicates the
usage of various linkers other than bfd via -fuse-ld={bfd|gold|lld}.
Cc: clang-built-linux@googlegroups.com
Suggested-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Nick Desaulniers <ndesaulniers@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
The 'addtree' and 'flags' in scripts/Kbuild.include are so compilecated
and ugly.
As I mentioned in [1], Kbuild should stop automatic prefixing of header
search path options.
I fixed up (almost) all Makefiles in the kernel. Now 'addtree' and
'flags' have been removed.
Kbuild still caters to add $(srctree)/$(src) and $(objtree)/$(obj)
to the header search path for O= building, but never touches extra
compiler options from ccflags-y etc.
[1]: https://patchwork.kernel.org/patch/9632347/
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
There is no more direct user of this macro; it is only used by
cc-ifversion.
Calling this macro is not efficient since it invokes the compiler to
get the compiler version. CONFIG_GCC_VERSION is already calculated in
the Kconfig stage, so Makefile can reuse it.
Here is a note about the slight difference between cc-version and
CONFIG_GCC_VERSION:
When using Clang, cc-version is evaluated to '0402' because Clang
defines __GNUC__ and __GNUC__MINOR__, and looks like GCC 4.2 in the
version point of view. On the other hand, CONFIG_GCC_VERSION=0
when $(CC) is clang.
There are currently two users of cc-ifversion:
arch/mips/loongson64/Platform
arch/powerpc/Makefile
They are not affected by this change.
The format of cc-version is <major><minor>, while CONFIG_GCC_VERSION
<major><minor><patch>. I adjusted cc-ifversion for the difference of
the number of digits.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
- $(word 1, <text>) is equivalent to $(firstword <text>)
- hardcode "gcc" instead of $(CC)
- minimize the shell script part
A little more notes in case $(filter-out -%, ...) is not clear.
arch/mips/Makefile passes prefixes depending on the configuration.
CROSS_COMPILE := $(call cc-cross-prefix, $(tool-archpref)-linux- \
$(tool-archpref)-linux-gnu- $(tool-archpref)-unknown-linux-gnu-)
In the Kconfig stage (e.g. when you run 'make defconfig'), neither
CONFIG_32BIT nor CONFIG_64BIT is defined. So, $(tool-archpref) is
empty. As a result, "-linux -linux-gnu- -unknown-linux-gnu" is passed
into cc-cross-prefix. The command 'which' assumes arguments starting
with a hyphen as command options, then emits the following messages:
Illegal option -l
Illegal option -l
Illegal option -u
I think it is strange to define CROSS_COMPILE depending on the CONFIG
options since you need to feed $(CC) to Kconfig, but it is how MIPS
Makefile currently works. Anyway, it would not hurt to filter-out
invalid strings beforehand.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
In Kbuild, if_changed and friends must have FORCE as a prerequisite.
Hence, $(filter-out FORCE,$^) or $(filter-out $(PHONY),$^) is a common
idiom to get the names of all the prerequisites except phony targets.
Add real-prereqs as a shorthand.
Note:
We cannot replace $(filter %.o,$^) in cmd_link_multi-m because $^ may
include auto-generated dependencies from the .*.cmd file when a single
object module is changed into a multi object module. Refer to commit
69ea912fda ("kbuild: remove unneeded link_multi_deps"). I added some
comment to avoid accidental breakage.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Rob Herring <robh@kernel.org>
Commit eea199b445 ("kbuild: remove unnecessary LEX_PREFIX and
YACC_PREFIX") removed the last users of this macro.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
You do not have to use define ... endef for filechk_* rules.
For simple cases, the use of assignment looks cleaner, IMHO.
I updated the usage for scripts/Kbuild.include in case somebody
misunderstands the 'define ... endif' is the requirement.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Heiko Carstens <heiko.carstens@de.ibm.com>
filechk_* rules often consist of multiple 'echo' lines. They must be
surrounded with { } or ( ) to work correctly. Otherwise, only the
string from the last 'echo' would be written into the target.
Let's take care of that in the 'filechk' in scripts/Kbuild.include
to clean up filechk_* rules.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
In commit 54a702f705 ("kbuild: mark $(targets) as .SECONDARY and
remove .PRECIOUS markers"), I missed one important feature of the
.SECONDARY target:
.SECONDARY with no prerequisites causes all targets to be
treated as secondary.
... which agrees with the policy of Kbuild.
Let's move it to scripts/Kbuild.include, with no prerequisites.
Note:
If an intermediate file is generated by $(call if_changed,...), you
still need to add it to "targets" so its .*.cmd file is included.
The arm/arm64 crypto files are generated by $(call cmd,shipped),
so they do not need to be added to "targets", but need to be added
to "clean-files" so "make clean" can properly clean them away.
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
