Pull RCU updates from Paul McKenney:
- Documentation updates
- Miscellaneous fixes
- Callback-offload updates, perhaps most notably a new
RCU_NOCB_CPU_DEFAULT_ALL Kconfig option that causes all CPUs to be
offloaded at boot time, regardless of kernel boot parameters.
This is useful to battery-powered systems such as ChromeOS and
Android. In addition, a new RCU_NOCB_CPU_CB_BOOST kernel boot
parameter prevents offloaded callbacks from interfering with
real-time workloads and with energy-efficiency mechanisms
- Polled grace-period updates, perhaps most notably making these APIs
account for both normal and expedited grace periods
- Tasks RCU updates, perhaps most notably reducing the CPU overhead of
RCU tasks trace grace periods by more than a factor of two on a
system with 15,000 tasks.
The reduction is expected to increase with the number of tasks, so it
seems reasonable to hypothesize that a system with 150,000 tasks
might see a 20-fold reduction in CPU overhead
- Torture-test updates
- Updates that merge RCU's dyntick-idle tracking into context tracking,
thus reducing the overhead of transitioning to kernel mode from
either idle or nohz_full userspace execution for kernels that track
context independently of RCU.
This is expected to be helpful primarily for kernels built with
CONFIG_NO_HZ_FULL=y
* tag 'rcu.2022.07.26a' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu: (98 commits)
rcu: Add irqs-disabled indicator to expedited RCU CPU stall warnings
rcu: Diagnose extended sync_rcu_do_polled_gp() loops
rcu: Put panic_on_rcu_stall() after expedited RCU CPU stall warnings
rcutorture: Test polled expedited grace-period primitives
rcu: Add polled expedited grace-period primitives
rcutorture: Verify that polled GP API sees synchronous grace periods
rcu: Make Tiny RCU grace periods visible to polled APIs
rcu: Make polled grace-period API account for expedited grace periods
rcu: Switch polled grace-period APIs to ->gp_seq_polled
rcu/nocb: Avoid polling when my_rdp->nocb_head_rdp list is empty
rcu/nocb: Add option to opt rcuo kthreads out of RT priority
rcu: Add nocb_cb_kthread check to rcu_is_callbacks_kthread()
rcu/nocb: Add an option to offload all CPUs on boot
rcu/nocb: Fix NOCB kthreads spawn failure with rcu_nocb_rdp_deoffload() direct call
rcu/nocb: Invert rcu_state.barrier_mutex VS hotplug lock locking order
rcu/nocb: Add/del rdp to iterate from rcuog itself
rcu/tree: Add comment to describe GP-done condition in fqs loop
rcu: Initialize first_gp_fqs at declaration in rcu_gp_fqs()
rcu/kvfree: Remove useless monitor_todo flag
rcu: Cleanup RCU urgency state for offline CPU
...
For some reason (ancient assembler?) the following build error is
reported by the kisskb:
kisskb/src/arch/xtensa/kernel/entry.S: Error: unknown pseudo-op: `.bss':
=> 2176
Change abbreviated '.bss' to the full '.section .bss, "aw"' to fix this
error.
Reported-by: Geert Uytterhoeven <geert@linux-m68k.org>
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
When call0 userspace ABI support by probing is enabled instructions that
cause illegal instruction exception when PS.WOE is clear are retried
with PS.WOE set before calling c-level exception handler. Record user pc
at which PS.WOE was set in the fast exception handler and clear PS.WOE
in the c-level exception handler if we get there from the same address.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
Don't use numeric labels for long jumps, use named local labels instead.
Avoid conditional label definition.
No functional changes.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
When fast_store_prohibited needs to go to the C-level exception handler
it leaves TLB entry that caused page fault in the TLB. If the faulting
task gets switched to a different CPU and completes page table update
there the TLB entry will get out of sync with the page table which may
cause a livelock on access to that page.
Invalidate faulting TLB entry on a slow path exit from the
fast_store_prohibited.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
Current coprocessor support on xtensa only works correctly on
uniprocessor configurations. Make it work on SMP too and keep it lazy.
Make coprocessor_owner array per-CPU and move it to struct exc_table for
easy access from the fast_coprocessor exception handler. Allow task to
have live coprocessors only on single CPU, record this CPU number in the
struct thread_info::cp_owner_cpu. Change struct thread_info::cpenable
meaning to be 'coprocessors live on cp_owner_cpu'.
Introduce C-level coprocessor exception handler that flushes and
releases live coprocessors of the task taking 'coprocessor disabled'
exception and call it from the fast_coprocessor handler when the task
has live coprocessors on other CPU.
Make coprocessor_flush_all and coprocessor_release_all work correctly
when called from any CPU by sending IPI to the cp_owner_cpu. Add
function coprocessor_flush_release_all to do flush followed by release
atomically. Add function local_coprocessors_flush_release_all to flush
and release all coprocessors on the local CPU and use it to flush
coprocessor contexts from the CPU that goes offline.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
Put user exit context tracking call on the common kernel entry/exit path
(function calls are impossible at earlier kernel entry stages because
PS.EXCM is not cleared yet). Put user entry context tracking call on the
user exit path. Syscalls go through this common code too, so nothing
specific needs to be done for them.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
Using plain register names is prone to errors when code is changed and
new calls are added between the register load and use. Change plain
register names to abi_* names in the call-heavy part of the kernel exit
code to clearly indicate what's supposed to be preserved and what's not.
Re-align code while at it.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
Context tracking call must be done after hardirq tracking call,
otherwise lockdep_assert_irqs_disabled called from rcu_eqs_exit gives
a warning. To avoid context tracking logic duplication for IRQ/exception
entry paths move trace_hardirqs_off call back to common entry code.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
Don't use numeric labels for complex branching logic. Mark each branch
with named local label and use them. Rearrange exit back to kernel mode
to avoid conditional label definition.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
NMI exit path to userspace should neither check TIF_DB_DISABLED nor call
check_tlb_sanity because NMI shouldn't touch anything related to
userspace. Drop kernel/userspace check in NMI exit path.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
PT_SIZE is used by the xtensa port to designate kernel exception frame
size. In preparation for struct pt_regs size change rename PT_SIZE to
PT_KERNEL_SIZE for clarity and change its definition to always cover
only the kernel exception frame.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
There have historically been two big uses of do_exit. The first is
it's design use to be the guts of the exit(2) system call. The second
use is to terminate a task after something catastrophic has happened
like a NULL pointer in kernel code. The function make_task_dead
has been added to accomidate the second use.
The call to do_exit in Linvalidmask is clearly not a normal userspace
exit. As failure handling there are two possible ways to go.
If userspace can trigger the issue force_exit_sig should be called.
Otherwise make_task_dead probably from the implementation of die
is appropriate.
Replace the call of do_exit in Linvalidmask with make_task_dead as
I don't know xtensa and especially xtensa assembly language well
enough to do anything else.
Link: https://lkml.kernel.org/r/YdUmN7n4W5YETUhW@zeniv-ca.linux.org.uk
Suggested-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
There's no need in window overflow/underflow/alloca exception handlers
or window spill code when neither kernel nor userspace support windowed
registers. Don't build or link it.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
xtensa core may be configured without register windows support, don't
use register window specific opcodes in that case. Use window register
specific opcodes to initialize hardware or reset core to a known state
regardless of the chosen ABI.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
Replace hardcoded register and opcode names with ABI-agnostic macros.
Add register save/restore code where necessary. Conditionalize windowed
only or call0 only code. Add stack initialization matching _switch_to
epilogue to copy_thread.
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
The result of the s32ex opcode is recorded in the ATOMCTL special
register and must be retrieved with the getex opcode. Context switch
between s32ex and getex may trash the ATOMCTL register and result in
duplicate update or missing update of the atomic variable.
Add atomctl8 field to the struct thread_info and use getex to swap
ATOMCTL bit 8 as a part of context switch.
Clear exclusive access monitor on kernel entry.
Cc: stable@vger.kernel.org
Fixes: f7c34874f0 ("xtensa: add exclusive atomics support")
Signed-off-by: Max Filippov <jcmvbkbc@gmail.com>
Linus Torvalds
Frederic WeisbeckerandPaul E. McKenney
Max Filippov
Eric W. Biederman
Jens Axboe
Mike RapoportandLinus Torvalds
Chris PackhamandMax Filippov