Commit
c79f816311 ("ARM: 9283/1: permit non-nested kernel mode NEON in softirq context")
relaxed the rules around the use of SIMD instructions in kernel mode on
ARM, to allow such use when serving a softirq. To avoid having to
preserve/restore kernel mode NEON state when such a softirq is taken,
softirqs are now disabled when using the NEON from task context.
However, the fact that the softirq API does not allow unmasking of
softirqs with interrupts disabled was overlooked, resulting in a WARN()
in some cases, as reported by Guenter:
WARNING: CPU: 0 PID: 1145 at kernel/softirq.c:369 __local_bh_enable_ip+0x118/0x194
Call trace:
unwind_backtrace from show_stack+0x10/0x14
show_stack from dump_stack_lvl+0x7c/0xac
dump_stack_lvl from __warn+0x7c/0x1b8
__warn from warn_slowpath_fmt+0x19c/0x1a4
warn_slowpath_fmt from __local_bh_enable_ip+0x118/0x194
__local_bh_enable_ip from crc_t10dif_arch+0xd4/0xe8
crc_t10dif_arch from crc_t10dif_wrapper+0x14/0x1c
crc_t10dif_wrapper from crc_main_test+0x178/0x360
crc_main_test from kunit_try_run_case+0x78/0x1e0
kunit_try_run_case from kunit_generic_run_threadfn_adapter+0x1c/0x34
kunit_generic_run_threadfn_adapter from kthread+0x118/0x254
kthread from ret_from_fork+0x14/0x28
While disabling softirqs is not really needed when running with IRQs
disabled (given that the only way a softirq can be delivered
asynchrously is over the back of an IRQ), let's not complicate this
logic more than needed, and simply disallow use of the NEON in kernel
mode when IRQs are disabled.
Another approach might be to only disable and re-enable softirqs if IRQs
are enabled, but other than the test case above, there are no clear use
cases for doing non-trivial arithmetic processing (hence using an
accelerated SIMD implementation) with IRQs disabled.
Reported-by: Guenter Roeck <linux@roeck-us.net>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Link: https://lore.kernel.org/all/389b899f-893c-4855-9e30-d8920a5d6f91@roeck-us.net
Reviewed-by: Eric Biggers <ebiggers@kernel.org>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
Now that the minimum supported binutils version is 2.25, we no longer
need a workaround for binutils older than 2.24 for accessing VFP control
registers from assembler.
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
VFP_bounce() is invoked from within vfp_support_entry() and may send a
signal. Sending a signal uses spinlock_t which becomes a sleeping lock
on PREEMPT_RT and must not be acquired within a preempt-disabled
section.
Move the vfp_raise_sigfpe() block outside of the vfp_state_hold() section.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
vfp_entry() is invoked from exception handler and is fully preemptible.
It uses local_bh_disable() to remain uninterrupted while checking the
VFP state.
This is not working on PREEMPT_RT because local_bh_disable()
synchronizes the relevant section but the context remains fully
preemptible.
Use vfp_state_hold() for uninterrupted access.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
vfp_sync_hwstate() uses preempt_disable() followed by local_bh_disable()
to ensure that it won't get interrupted while checking the VFP state.
This harms PREEMPT_RT because softirq handling can get preempted and
local_bh_disable() synchronizes the related section with a sleeping lock
which does not work with disabled preemption.
Use the vfp_state_hold() to synchronize the access.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
kernel_neon_begin() uses local_bh_disable() to ensure exclusive access
to the VFP unit. This is broken on PREEMPT_RT because a BH disabled
section remains preemptible on PREEMPT_RT.
Introduce vfp_state_hold() which uses local_bh_disable() and
preempt_disable() on PREEMPT_RT. Since softirqs are processed always in
thread context, disabling preemption is enough to ensure that the
current context won't get interrupted by something that is using the
VFP. Use it in kernel_neon_begin().
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
Pull ARM updates from Russell King:
- lots of build cleanups from Arnd spread throughout the arch/arm tree
- replace strlcpy() with the preferred strscpy()
- use sign_extend32() in the module linker
- drop handle_irq() machine descriptor method
* tag 'for-linus' of git://git.armlinux.org.uk/~rmk/linux-arm:
ARM: 9315/1: fiq: include asm/mach/irq.h for prototypes
ARM: 9314/1: tcm: move tcm_init() prototype to asm/tcm.h
ARM: 9313/1: vdso: add missing prototypes
ARM: 9312/1: vfp: include asm/neon.h in vfpmodule.c
ARM: 9311/1: decompressor: move function prototypes to misc.h
ARM: 9310/1: xip-kernel: add __inflate_kernel_data prototype
ARM: 9309/1: add missing syscall prototypes
ARM: 9308/1: move setup functions to header
ARM: 9307/1: nommu: include asm/idmap.h
ARM: 9306/1: cacheflush: avoid __flush_anon_page() missing-prototype warning
ARM: 9305/1: add clear/copy_user_highpage declarations
ARM: 9304/1: add prototype for function called only from asm
ARM: 9303/1: kprobes: avoid missing-declaration warnings
ARM: 9302/1: traps: hide unused functions on NOMMU
ARM: 9301/1: dma-mapping: hide unused dma_contiguous_early_fixup function
ARM: 9300/1: Replace all non-returning strlcpy with strscpy
ARM: 9299/1: module: use sign_extend32() to extend the signedness
ARM: 9298/1: Drop custom mdesc->handle_irq()
Two functions defined here need the declaration in a header
to avoid W=1 warnings:
arch/arm/vfp/vfpmodule.c:735:6: error: no previous prototype for 'kernel_neon_begin' [-Werror=missing-prototypes]
arch/arm/vfp/vfpmodule.c:768:6: error: no previous prototype for 'kernel_neon_end' [-Werror=missing-prototypes]
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
Now that the VFP support code has been reimplemented as a C function
that takes a struct pt_regs pointer and an opcode, we can use the
existing undef_hook framework to deal with undef exceptions triggered by
VFP instructions instead of having special handling in assembler.
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
En/disabling softirqs from asm code turned out to be trickier than
expected, so vfp_support_entry now returns by tail calling
__local_enable_bh_ip() and passing the same arguments that a C call to
local_bh_enable() would pass. However, this is slightly hacky, as we
don't want to carry our own implementation of local_bh_enable().
So let's bite the bullet, and get rid of the asm logic in
vfp_support_entry that reasons about whether or not to save and/or
reload the VFP state, and about whether or not an FP exception is
pending, and only keep the VFP loading logic as a function that is
callable from C.
Replicate the removed logic in vfp_entry(), and use the exact same
reasoning as in the asm code. To emphasize the correspondence, retain
some of the asm comments in the C version as well.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Acked-by: Linus Walleij <linus.walleij@linaro.org>
Feroceon CPUs have a non-standard implementation of VFP which reports
synchronous VFP exceptions using the async VFP flag. This requires a
workaround which is difficult to reconcile with other implementations,
making it tricky to support both versions in a single image.
Since this is a v5 CPU, it is not supported by armhf and so the
likelihood that anybody is using this with recent distros/kernels and
rely on the VFP at the same time is extremely low. So let's just disable
VFP support on these cores, so we can remove the workaround.
This will help future development to support v5 and v6 CPUs with a
single kernel image.
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Acked-by: Nicolas Pitre <nico@fluxnic.net>
Acked-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
VFP 'bouncing' occurs when the VFP unit cannot complete the execution of
a VFP instruction, either because it is not implemented at all, or
because the values of the arguments are out of range for the hardware
implementation, and the software needs to step in to complete the
operation.
To give some insight in how much certain programs rely on this bouncing,
record the emulation of a VFP instruction in perf's emulation-faults
counter.
This can be used like so
perf stat -e emulation-faults ./testfloat -all2
and the output will be something like
Performance counter stats for './testfloat -all2':
259,277 emulation-faults:u
6.846432176 seconds time elapsed
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Commit c76c6c4ecb ("ARM: 9294/2: vfp: Fix broken softirq handling
with instrumentation enabled") updated the VFP exception entry logic to
go via a C function, so that we get the compiler's version of
local_bh_disable(), which may be instrumented, and isn't generally
callable from assembler.
However, this assumes that passing an alternative 'success' return
address works in C as it does in asm, and this is only the case if the C
calls in question are tail calls, as otherwise, the stack will need some
unwinding as well.
I have already sent patches to the list that replace most of the asm
logic with C code, and so it is preferable to have a minimal fix that
addresses the issue and can be backported along with the commit that it
fixes to v6.3 from v6.4. Hopefully, we can land the C conversion for v6.5.
So instead of passing the 'success' return address as a function
argument, pass the stack address from where to pop it so that both LR
and SP have the expected value.
Fixes: c76c6c4ecb ("ARM: 9294/2: vfp: Fix broken softirq handling with ...")
Reported-by: syzbot+d4b00edc2d0c910d4bf4@syzkaller.appspotmail.com
Tested-by: syzbot+d4b00edc2d0c910d4bf4@syzkaller.appspotmail.com
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Tested-by: Andrew Lunn <andrew@lunn.ch>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Tested-by: Andre Przywara <andre.przywara@arm.com>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
Commit 62b95a7b44 ("ARM: 9282/1: vfp: Manipulate task VFP state with
softirqs disabled") replaced the en/disable preemption calls inside the
VFP state handling code with en/disabling of soft IRQs, which is
necessary to allow kernel use of the VFP/SIMD unit when handling a soft
IRQ.
Unfortunately, when lockdep is enabled (or other instrumentation that
enables TRACE_IRQFLAGS), the disable path implemented in asm fails to
perform the lockdep and RCU related bookkeeping, resulting in spurious
warnings and other badness.
Set let's rework the VFP entry code a little bit so we can make the
local_bh_disable() call from C, with all the instrumentations that
happen to have been configured. Calling local_bh_enable() can be done
from asm, as it is a simple wrapper around __local_bh_enable_ip(), which
is always a callable function.
Link: https://lore.kernel.org/all/ZBBYCSZUJOWBg1s8@localhost.localdomain/
Fixes: 62b95a7b44 ("ARM: 9282/1: vfp: Manipulate task VFP state with softirqs disabled")
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
In preparation for reimplementing the do_vfp()->vfp_support_entry()
handover in C code, switch to using R3 to pass the 'success' return
address, rather than R9, as it cannot be used for parameter passing.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
Instead of dereferencing thread_info in do_vfp, pass the thread_info
pointer to vfp_support_entry via R1. That way, we only use a single
caller save register, which makes it easier to convert do_vfp to C code
in a subsequent patch.
Note that, unlike the CPU number, which can change due to preemption,
passing the thread_info pointer can safely be done with preemption
enabled.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
We currently only permit kernel mode NEON in process context, to avoid
the need to preserve/restore the NEON register file when taking an
exception while running in the kernel.
Like we did on arm64, we can relax this restriction substantially, by
permitting kernel mode NEON from softirq context, while ensuring that
softirq processing is disabled when the NEON is being used in task
context. This guarantees that only NEON context belonging to user space
needs to be preserved and restored, which is already taken care of.
This is especially relevant for network encryption, where incoming
frames are typically handled in softirq context, and deferring software
decryption to a kernel thread or falling back to C code are both
undesirable from a performance PoV.
Tested-by: Martin Willi <martin@strongswan.org>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
In a subsequent patch, we will relax the kernel mode NEON policy, and
permit kernel mode NEON to be used not only from task context, as is
permitted today, but also from softirq context.
Given that softirqs may trigger over the back of any IRQ unless they are
explicitly disabled, we need to address the resulting races in the VFP
state handling, by disabling softirq processing in two distinct but
related cases:
- kernel mode NEON will leave the FPU disabled after it completes, so
any kernel code sequence that enables the FPU and subsequently accesses
its registers needs to disable softirqs until it completes;
- kernel_neon_begin() will preserve the userland VFP state in memory,
and if it interrupts the ordinary VFP state preserve sequence, the
latter will resume execution with the VFP registers corrupted, and
happily continue saving them to memory.
Given that disabling softirqs also disables preemption, we can replace
the existing preempt_disable/enable occurrences in the VFP state
handling asm code with new macros that dis/enable softirqs instead.
In the VFP state handling C code, add local_bh_disable/enable() calls
in those places where the VFP state is preserved.
One thing to keep in mind is that, once we allow NEON use in softirq
context, the result of any such interruption is that the FPEXC_EN bit in
the FPEXC register will be cleared, and vfp_current_hw_state[cpu] will
be NULL. This means that any sequence that [conditionally] clears
FPEXC_EN and/or sets vfp_current_hw_state[cpu] to NULL does not need to
run with softirqs disabled, as the result will be the same. Furthermore,
the handling of THREAD_NOTIFY_SWITCH is guaranteed to run with IRQs
disabled, and so it does not need protection from softirq interruptions
either.
Tested-by: Martin Willi <martin@strongswan.org>
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
Int8 matrix multiplication (FEAT_AA32I8MM) is a feature present in AArch32 state for Armv8 and is represented by ISAR6.I8MM identification register.
This feature denotes the presence of VSMMLA, VSUDOT, VUMMLA, VUSMMLA and
VUSDOT instructions and hence adding a hwcap will enable the userspace
to check it before trying to use those instructions.
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Amit Daniel Kachhap <amit.kachhap@arm.com>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
Advanced SIMD BFloat16 (FEAT_AA32BF16) is a feature present in AArch32
state for Armv8 and is represented by ISAR6.BF16 identification
register.
This feature denotes the presence of VCVT, VCVTB, VCVTT, VDOT, VFMAB,
VFMAT and VMMLA instructions and hence adding a hwcap will enable the
userspace to check it before trying to use those instructions.
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Amit Daniel Kachhap <amit.kachhap@arm.com>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
Floating-point half-precision multiplication (FHM) is a feature present
in AArch32 state for Armv8 and is represented by ISAR6.FHM identification register.
This feature denotes the presence of VFMAL and VMFSL instructions and
hence adding a hwcap will enable the userspace to check it before
trying to use those instructions.
Reviewed-by: Linus Walleij <linus.walleij@linaro.org>
Signed-off-by: Amit Daniel Kachhap <amit.kachhap@arm.com>
Signed-off-by: Russell King (Oracle) <rmk+kernel@armlinux.org.uk>
