You've already forked linux-apfs
mirror of
https://github.com/linux-apfs/linux-apfs.git
synced 2026-05-01 15:00:59 -07:00
Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull RCU updates from Ingo Molnar:
"The sole purpose of these changes is to shrink and simplify the RCU
code base, which has suffered from creeping bloat over the past couple
of years. The end result is a net removal of ~2700 lines of code:
79 files changed, 1496 insertions(+), 4211 deletions(-)
Plus there's a marked reduction in the Kconfig space complexity as
well, here's the number of matches on 'grep RCU' in the .config:
before after
x86-defconfig 17 15
x86-allmodconfig 33 20"
* 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (86 commits)
rcu: Remove RCU CPU stall warnings from Tiny RCU
rcu: Remove event tracing from Tiny RCU
rcu: Move RCU debug Kconfig options to kernel/rcu
rcu: Move RCU non-debug Kconfig options to kernel/rcu
rcu: Eliminate NOCBs CPU-state Kconfig options
rcu: Remove debugfs tracing
srcu: Remove Classic SRCU
srcu: Fix rcutorture-statistics typo
rcu: Remove SPARSE_RCU_POINTER Kconfig option
rcu: Remove the now-obsolete PROVE_RCU_REPEATEDLY Kconfig option
rcu: Remove typecheck() from RCU locking wrapper functions
rcu: Remove #ifdef moving rcu_end_inkernel_boot from rcupdate.h
rcu: Remove nohz_full full-system-idle state machine
rcu: Remove the RCU_KTHREAD_PRIO Kconfig option
rcu: Remove *_SLOW_* Kconfig options
srcu: Use rnp->lock wrappers to replace explicit memory barriers
rcu: Move rnp->lock wrappers for SRCU use
rcu: Convert rnp->lock wrappers to macros for SRCU use
rcu: Refactor #includes from include/linux/rcupdate.h
bcm47xx: Fix build regression
...
This commit is contained in:
Linus Torvalds
@@ -28,8 +28,6 @@ stallwarn.txt
|
||||
- RCU CPU stall warnings (module parameter rcu_cpu_stall_suppress)
|
||||
torture.txt
|
||||
- RCU Torture Test Operation (CONFIG_RCU_TORTURE_TEST)
|
||||
trace.txt
|
||||
- CONFIG_RCU_TRACE debugfs files and formats
|
||||
UP.txt
|
||||
- RCU on Uniprocessor Systems
|
||||
whatisRCU.txt
|
||||
|
||||
@@ -559,9 +559,7 @@ The <tt>rcu_access_pointer()</tt> on line 6 is similar to
|
||||
For <tt>remove_gp_synchronous()</tt>, as long as all modifications
|
||||
to <tt>gp</tt> are carried out while holding <tt>gp_lock</tt>,
|
||||
the above optimizations are harmless.
|
||||
However,
|
||||
with <tt>CONFIG_SPARSE_RCU_POINTER=y</tt>,
|
||||
<tt>sparse</tt> will complain if you
|
||||
However, <tt>sparse</tt> will complain if you
|
||||
define <tt>gp</tt> with <tt>__rcu</tt> and then
|
||||
access it without using
|
||||
either <tt>rcu_access_pointer()</tt> or <tt>rcu_dereference()</tt>.
|
||||
@@ -1849,7 +1847,8 @@ mass storage, or user patience, whichever comes first.
|
||||
If the nesting is not visible to the compiler, as is the case with
|
||||
mutually recursive functions each in its own translation unit,
|
||||
stack overflow will result.
|
||||
If the nesting takes the form of loops, either the control variable
|
||||
If the nesting takes the form of loops, perhaps in the guise of tail
|
||||
recursion, either the control variable
|
||||
will overflow or (in the Linux kernel) you will get an RCU CPU stall warning.
|
||||
Nevertheless, this class of RCU implementations is one
|
||||
of the most composable constructs in existence.
|
||||
@@ -1977,9 +1976,8 @@ guard against mishaps and misuse:
|
||||
and <tt>rcu_dereference()</tt>, perhaps (incorrectly)
|
||||
substituting a simple assignment.
|
||||
To catch this sort of error, a given RCU-protected pointer may be
|
||||
tagged with <tt>__rcu</tt>, after which running sparse
|
||||
with <tt>CONFIG_SPARSE_RCU_POINTER=y</tt> will complain
|
||||
about simple-assignment accesses to that pointer.
|
||||
tagged with <tt>__rcu</tt>, after which sparse
|
||||
will complain about simple-assignment accesses to that pointer.
|
||||
Arnd Bergmann made me aware of this requirement, and also
|
||||
supplied the needed
|
||||
<a href="https://lwn.net/Articles/376011/">patch series</a>.
|
||||
@@ -2036,7 +2034,7 @@ guard against mishaps and misuse:
|
||||
some other synchronization mechanism, for example, reference
|
||||
counting.
|
||||
<li> In kernels built with <tt>CONFIG_RCU_TRACE=y</tt>, RCU-related
|
||||
information is provided via both debugfs and event tracing.
|
||||
information is provided via event tracing.
|
||||
<li> Open-coded use of <tt>rcu_assign_pointer()</tt> and
|
||||
<tt>rcu_dereference()</tt> to create typical linked
|
||||
data structures can be surprisingly error-prone.
|
||||
@@ -2519,11 +2517,7 @@ It is similarly socially unacceptable to interrupt an
|
||||
<tt>nohz_full</tt> CPU running in userspace.
|
||||
RCU must therefore track <tt>nohz_full</tt> userspace
|
||||
execution.
|
||||
And in
|
||||
<a href="https://lwn.net/Articles/558284/"><tt>CONFIG_NO_HZ_FULL_SYSIDLE=y</tt></a>
|
||||
kernels, RCU must separately track idle CPUs on the one hand and
|
||||
CPUs that are either idle or executing in userspace on the other.
|
||||
In both cases, RCU must be able to sample state at two points in
|
||||
RCU must therefore be able to sample state at two points in
|
||||
time, and be able to determine whether or not some other CPU spent
|
||||
any time idle and/or executing in userspace.
|
||||
|
||||
@@ -2935,6 +2929,20 @@ The reason that this is possible is that SRCU is insensitive
|
||||
to whether or not a CPU is online, which means that <tt>srcu_barrier()</tt>
|
||||
need not exclude CPU-hotplug operations.
|
||||
|
||||
<p>
|
||||
SRCU also differs from other RCU flavors in that SRCU's expedited and
|
||||
non-expedited grace periods are implemented by the same mechanism.
|
||||
This means that in the current SRCU implementation, expediting a
|
||||
future grace period has the side effect of expediting all prior
|
||||
grace periods that have not yet completed.
|
||||
(But please note that this is a property of the current implementation,
|
||||
not necessarily of future implementations.)
|
||||
In addition, if SRCU has been idle for longer than the interval
|
||||
specified by the <tt>srcutree.exp_holdoff</tt> kernel boot parameter
|
||||
(25 microseconds by default),
|
||||
and if a <tt>synchronize_srcu()</tt> invocation ends this idle period,
|
||||
that invocation will be automatically expedited.
|
||||
|
||||
<p>
|
||||
As of v4.12, SRCU's callbacks are maintained per-CPU, eliminating
|
||||
a locking bottleneck present in prior kernel versions.
|
||||
|
||||
@@ -413,11 +413,11 @@ over a rather long period of time, but improvements are always welcome!
|
||||
read-side critical sections. It is the responsibility of the
|
||||
RCU update-side primitives to deal with this.
|
||||
|
||||
17. Use CONFIG_PROVE_RCU, CONFIG_DEBUG_OBJECTS_RCU_HEAD, and the
|
||||
__rcu sparse checks (enabled by CONFIG_SPARSE_RCU_POINTER) to
|
||||
validate your RCU code. These can help find problems as follows:
|
||||
17. Use CONFIG_PROVE_LOCKING, CONFIG_DEBUG_OBJECTS_RCU_HEAD, and the
|
||||
__rcu sparse checks to validate your RCU code. These can help
|
||||
find problems as follows:
|
||||
|
||||
CONFIG_PROVE_RCU: check that accesses to RCU-protected data
|
||||
CONFIG_PROVE_LOCKING: check that accesses to RCU-protected data
|
||||
structures are carried out under the proper RCU
|
||||
read-side critical section, while holding the right
|
||||
combination of locks, or whatever other conditions
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -3238,21 +3238,17 @@
|
||||
|
||||
rcutree.gp_cleanup_delay= [KNL]
|
||||
Set the number of jiffies to delay each step of
|
||||
RCU grace-period cleanup. This only has effect
|
||||
when CONFIG_RCU_TORTURE_TEST_SLOW_CLEANUP is set.
|
||||
RCU grace-period cleanup.
|
||||
|
||||
rcutree.gp_init_delay= [KNL]
|
||||
Set the number of jiffies to delay each step of
|
||||
RCU grace-period initialization. This only has
|
||||
effect when CONFIG_RCU_TORTURE_TEST_SLOW_INIT
|
||||
is set.
|
||||
RCU grace-period initialization.
|
||||
|
||||
rcutree.gp_preinit_delay= [KNL]
|
||||
Set the number of jiffies to delay each step of
|
||||
RCU grace-period pre-initialization, that is,
|
||||
the propagation of recent CPU-hotplug changes up
|
||||
the rcu_node combining tree. This only has effect
|
||||
when CONFIG_RCU_TORTURE_TEST_SLOW_PREINIT is set.
|
||||
the rcu_node combining tree.
|
||||
|
||||
rcutree.rcu_fanout_exact= [KNL]
|
||||
Disable autobalancing of the rcu_node combining
|
||||
@@ -3328,6 +3324,17 @@
|
||||
This wake_up() will be accompanied by a
|
||||
WARN_ONCE() splat and an ftrace_dump().
|
||||
|
||||
rcuperf.gp_async= [KNL]
|
||||
Measure performance of asynchronous
|
||||
grace-period primitives such as call_rcu().
|
||||
|
||||
rcuperf.gp_async_max= [KNL]
|
||||
Specify the maximum number of outstanding
|
||||
callbacks per writer thread. When a writer
|
||||
thread exceeds this limit, it invokes the
|
||||
corresponding flavor of rcu_barrier() to allow
|
||||
previously posted callbacks to drain.
|
||||
|
||||
rcuperf.gp_exp= [KNL]
|
||||
Measure performance of expedited synchronous
|
||||
grace-period primitives.
|
||||
@@ -3355,17 +3362,22 @@
|
||||
rcuperf.perf_runnable= [BOOT]
|
||||
Start rcuperf running at boot time.
|
||||
|
||||
rcuperf.perf_type= [KNL]
|
||||
Specify the RCU implementation to test.
|
||||
|
||||
rcuperf.shutdown= [KNL]
|
||||
Shut the system down after performance tests
|
||||
complete. This is useful for hands-off automated
|
||||
testing.
|
||||
|
||||
rcuperf.perf_type= [KNL]
|
||||
Specify the RCU implementation to test.
|
||||
|
||||
rcuperf.verbose= [KNL]
|
||||
Enable additional printk() statements.
|
||||
|
||||
rcuperf.writer_holdoff= [KNL]
|
||||
Write-side holdoff between grace periods,
|
||||
in microseconds. The default of zero says
|
||||
no holdoff.
|
||||
|
||||
rcutorture.cbflood_inter_holdoff= [KNL]
|
||||
Set holdoff time (jiffies) between successive
|
||||
callback-flood tests.
|
||||
@@ -3803,6 +3815,15 @@
|
||||
spia_pedr=
|
||||
spia_peddr=
|
||||
|
||||
srcutree.counter_wrap_check [KNL]
|
||||
Specifies how frequently to check for
|
||||
grace-period sequence counter wrap for the
|
||||
srcu_data structure's ->srcu_gp_seq_needed field.
|
||||
The greater the number of bits set in this kernel
|
||||
parameter, the less frequently counter wrap will
|
||||
be checked for. Note that the bottom two bits
|
||||
are ignored.
|
||||
|
||||
srcutree.exp_holdoff [KNL]
|
||||
Specifies how many nanoseconds must elapse
|
||||
since the end of the last SRCU grace period for
|
||||
|
||||
@@ -303,6 +303,11 @@ defined which accomplish this::
|
||||
void smp_mb__before_atomic(void);
|
||||
void smp_mb__after_atomic(void);
|
||||
|
||||
Preceding a non-value-returning read-modify-write atomic operation with
|
||||
smp_mb__before_atomic() and following it with smp_mb__after_atomic()
|
||||
provides the same full ordering that is provided by value-returning
|
||||
read-modify-write atomic operations.
|
||||
|
||||
For example, smp_mb__before_atomic() can be used like so::
|
||||
|
||||
obj->dead = 1;
|
||||
|
||||
@@ -103,9 +103,3 @@ have already built it.
|
||||
|
||||
The optional make variable CF can be used to pass arguments to sparse. The
|
||||
build system passes -Wbitwise to sparse automatically.
|
||||
|
||||
Checking RCU annotations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
RCU annotations are not checked by default. To enable RCU annotation
|
||||
checks, include -DCONFIG_SPARSE_RCU_POINTER in your CF flags.
|
||||
|
||||
@@ -109,13 +109,12 @@ SCHED_SOFTIRQ: Do all of the following:
|
||||
on that CPU. If a thread that expects to run on the de-jittered
|
||||
CPU awakens, the scheduler will send an IPI that can result in
|
||||
a subsequent SCHED_SOFTIRQ.
|
||||
2. Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
|
||||
CONFIG_NO_HZ_FULL=y, and, in addition, ensure that the CPU
|
||||
to be de-jittered is marked as an adaptive-ticks CPU using the
|
||||
"nohz_full=" boot parameter. This reduces the number of
|
||||
scheduler-clock interrupts that the de-jittered CPU receives,
|
||||
minimizing its chances of being selected to do the load balancing
|
||||
work that runs in SCHED_SOFTIRQ context.
|
||||
2. CONFIG_NO_HZ_FULL=y and ensure that the CPU to be de-jittered
|
||||
is marked as an adaptive-ticks CPU using the "nohz_full="
|
||||
boot parameter. This reduces the number of scheduler-clock
|
||||
interrupts that the de-jittered CPU receives, minimizing its
|
||||
chances of being selected to do the load balancing work that
|
||||
runs in SCHED_SOFTIRQ context.
|
||||
3. To the extent possible, keep the CPU out of the kernel when it
|
||||
is non-idle, for example, by avoiding system calls and by
|
||||
forcing both kernel threads and interrupts to execute elsewhere.
|
||||
@@ -135,11 +134,10 @@ HRTIMER_SOFTIRQ: Do all of the following:
|
||||
RCU_SOFTIRQ: Do at least one of the following:
|
||||
1. Offload callbacks and keep the CPU in either dyntick-idle or
|
||||
adaptive-ticks state by doing all of the following:
|
||||
a. Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
|
||||
CONFIG_NO_HZ_FULL=y, and, in addition ensure that the CPU
|
||||
to be de-jittered is marked as an adaptive-ticks CPU using
|
||||
the "nohz_full=" boot parameter. Bind the rcuo kthreads
|
||||
to housekeeping CPUs, which can tolerate OS jitter.
|
||||
a. CONFIG_NO_HZ_FULL=y and ensure that the CPU to be
|
||||
de-jittered is marked as an adaptive-ticks CPU using the
|
||||
"nohz_full=" boot parameter. Bind the rcuo kthreads to
|
||||
housekeeping CPUs, which can tolerate OS jitter.
|
||||
b. To the extent possible, keep the CPU out of the kernel
|
||||
when it is non-idle, for example, by avoiding system
|
||||
calls and by forcing both kernel threads and interrupts
|
||||
@@ -236,11 +234,10 @@ To reduce its OS jitter, do at least one of the following:
|
||||
is feasible only if your workload never requires RCU priority
|
||||
boosting, for example, if you ensure frequent idle time on all
|
||||
CPUs that might execute within the kernel.
|
||||
3. Build with CONFIG_RCU_NOCB_CPU=y and CONFIG_RCU_NOCB_CPU_ALL=y,
|
||||
which offloads all RCU callbacks to kthreads that can be moved
|
||||
off of CPUs susceptible to OS jitter. This approach prevents the
|
||||
rcuc/%u kthreads from having any work to do, so that they are
|
||||
never awakened.
|
||||
3. Build with CONFIG_RCU_NOCB_CPU=y and boot with the rcu_nocbs=
|
||||
boot parameter offloading RCU callbacks from all CPUs susceptible
|
||||
to OS jitter. This approach prevents the rcuc/%u kthreads from
|
||||
having any work to do, so that they are never awakened.
|
||||
4. Ensure that the CPU never enters the kernel, and, in particular,
|
||||
avoid initiating any CPU hotplug operations on this CPU. This is
|
||||
another way of preventing any callbacks from being queued on the
|
||||
|
||||
@@ -27,7 +27,7 @@ The purpose of this document is twofold:
|
||||
(2) to provide a guide as to how to use the barriers that are available.
|
||||
|
||||
Note that an architecture can provide more than the minimum requirement
|
||||
for any particular barrier, but if the architecure provides less than
|
||||
for any particular barrier, but if the architecture provides less than
|
||||
that, that architecture is incorrect.
|
||||
|
||||
Note also that it is possible that a barrier may be a no-op for an
|
||||
|
||||
@@ -194,32 +194,9 @@ that the RCU callbacks are processed in a timely fashion.
|
||||
|
||||
Another approach is to offload RCU callback processing to "rcuo" kthreads
|
||||
using the CONFIG_RCU_NOCB_CPU=y Kconfig option. The specific CPUs to
|
||||
offload may be selected via several methods:
|
||||
|
||||
1. One of three mutually exclusive Kconfig options specify a
|
||||
build-time default for the CPUs to offload:
|
||||
|
||||
a. The CONFIG_RCU_NOCB_CPU_NONE=y Kconfig option results in
|
||||
no CPUs being offloaded.
|
||||
|
||||
b. The CONFIG_RCU_NOCB_CPU_ZERO=y Kconfig option causes
|
||||
CPU 0 to be offloaded.
|
||||
|
||||
c. The CONFIG_RCU_NOCB_CPU_ALL=y Kconfig option causes all
|
||||
CPUs to be offloaded. Note that the callbacks will be
|
||||
offloaded to "rcuo" kthreads, and that those kthreads
|
||||
will in fact run on some CPU. However, this approach
|
||||
gives fine-grained control on exactly which CPUs the
|
||||
callbacks run on, along with their scheduling priority
|
||||
(including the default of SCHED_OTHER), and it further
|
||||
allows this control to be varied dynamically at runtime.
|
||||
|
||||
2. The "rcu_nocbs=" kernel boot parameter, which takes a comma-separated
|
||||
list of CPUs and CPU ranges, for example, "1,3-5" selects CPUs 1,
|
||||
3, 4, and 5. The specified CPUs will be offloaded in addition to
|
||||
any CPUs specified as offloaded by CONFIG_RCU_NOCB_CPU_ZERO=y or
|
||||
CONFIG_RCU_NOCB_CPU_ALL=y. This means that the "rcu_nocbs=" boot
|
||||
parameter has no effect for kernels built with RCU_NOCB_CPU_ALL=y.
|
||||
offload may be selected using The "rcu_nocbs=" kernel boot parameter,
|
||||
which takes a comma-separated list of CPUs and CPU ranges, for example,
|
||||
"1,3-5" selects CPUs 1, 3, 4, and 5.
|
||||
|
||||
The offloaded CPUs will never queue RCU callbacks, and therefore RCU
|
||||
never prevents offloaded CPUs from entering either dyntick-idle mode
|
||||
|
||||
@@ -8,6 +8,7 @@
|
||||
#ifndef __BCM47XX_NVRAM_H
|
||||
#define __BCM47XX_NVRAM_H
|
||||
|
||||
#include <linux/errno.h>
|
||||
#include <linux/types.h>
|
||||
#include <linux/kernel.h>
|
||||
#include <linux/vmalloc.h>
|
||||
|
||||
@@ -17,11 +17,7 @@
|
||||
# define __release(x) __context__(x,-1)
|
||||
# define __cond_lock(x,c) ((c) ? ({ __acquire(x); 1; }) : 0)
|
||||
# define __percpu __attribute__((noderef, address_space(3)))
|
||||
#ifdef CONFIG_SPARSE_RCU_POINTER
|
||||
# define __rcu __attribute__((noderef, address_space(4)))
|
||||
#else /* CONFIG_SPARSE_RCU_POINTER */
|
||||
# define __rcu
|
||||
#endif /* CONFIG_SPARSE_RCU_POINTER */
|
||||
# define __private __attribute__((noderef))
|
||||
extern void __chk_user_ptr(const volatile void __user *);
|
||||
extern void __chk_io_ptr(const volatile void __iomem *);
|
||||
|
||||
@@ -7,6 +7,10 @@
|
||||
* unlimited scalability while maintaining a constant level of contention
|
||||
* on the root node.
|
||||
*
|
||||
* This seemingly RCU-private file must be available to SRCU users
|
||||
* because the size of the TREE SRCU srcu_struct structure depends
|
||||
* on these definitions.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation; either version 2 of the License, or
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
/*
|
||||
* RCU segmented callback lists
|
||||
*
|
||||
* This seemingly RCU-private file must be available to SRCU users
|
||||
* because the size of the TREE SRCU srcu_struct structure depends
|
||||
* on these definitions.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation; either version 2 of the License, or
|
||||
|
||||
+18
-300
@@ -34,104 +34,15 @@
|
||||
#define __LINUX_RCUPDATE_H
|
||||
|
||||
#include <linux/types.h>
|
||||
#include <linux/cache.h>
|
||||
#include <linux/spinlock.h>
|
||||
#include <linux/threads.h>
|
||||
#include <linux/cpumask.h>
|
||||
#include <linux/seqlock.h>
|
||||
#include <linux/lockdep.h>
|
||||
#include <linux/debugobjects.h>
|
||||
#include <linux/bug.h>
|
||||
#include <linux/compiler.h>
|
||||
#include <linux/ktime.h>
|
||||
#include <linux/atomic.h>
|
||||
#include <linux/irqflags.h>
|
||||
#include <linux/preempt.h>
|
||||
#include <linux/bottom_half.h>
|
||||
#include <linux/lockdep.h>
|
||||
#include <asm/processor.h>
|
||||
#include <linux/cpumask.h>
|
||||
|
||||
#include <asm/barrier.h>
|
||||
|
||||
#ifndef CONFIG_TINY_RCU
|
||||
extern int rcu_expedited; /* for sysctl */
|
||||
extern int rcu_normal; /* also for sysctl */
|
||||
#endif /* #ifndef CONFIG_TINY_RCU */
|
||||
|
||||
#ifdef CONFIG_TINY_RCU
|
||||
/* Tiny RCU doesn't expedite, as its purpose in life is instead to be tiny. */
|
||||
static inline bool rcu_gp_is_normal(void) /* Internal RCU use. */
|
||||
{
|
||||
return true;
|
||||
}
|
||||
static inline bool rcu_gp_is_expedited(void) /* Internal RCU use. */
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
static inline void rcu_expedite_gp(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void rcu_unexpedite_gp(void)
|
||||
{
|
||||
}
|
||||
#else /* #ifdef CONFIG_TINY_RCU */
|
||||
bool rcu_gp_is_normal(void); /* Internal RCU use. */
|
||||
bool rcu_gp_is_expedited(void); /* Internal RCU use. */
|
||||
void rcu_expedite_gp(void);
|
||||
void rcu_unexpedite_gp(void);
|
||||
#endif /* #else #ifdef CONFIG_TINY_RCU */
|
||||
|
||||
enum rcutorture_type {
|
||||
RCU_FLAVOR,
|
||||
RCU_BH_FLAVOR,
|
||||
RCU_SCHED_FLAVOR,
|
||||
RCU_TASKS_FLAVOR,
|
||||
SRCU_FLAVOR,
|
||||
INVALID_RCU_FLAVOR
|
||||
};
|
||||
|
||||
#if defined(CONFIG_TREE_RCU) || defined(CONFIG_PREEMPT_RCU)
|
||||
void rcutorture_get_gp_data(enum rcutorture_type test_type, int *flags,
|
||||
unsigned long *gpnum, unsigned long *completed);
|
||||
void rcutorture_record_test_transition(void);
|
||||
void rcutorture_record_progress(unsigned long vernum);
|
||||
void do_trace_rcu_torture_read(const char *rcutorturename,
|
||||
struct rcu_head *rhp,
|
||||
unsigned long secs,
|
||||
unsigned long c_old,
|
||||
unsigned long c);
|
||||
bool rcu_irq_enter_disabled(void);
|
||||
#else
|
||||
static inline void rcutorture_get_gp_data(enum rcutorture_type test_type,
|
||||
int *flags,
|
||||
unsigned long *gpnum,
|
||||
unsigned long *completed)
|
||||
{
|
||||
*flags = 0;
|
||||
*gpnum = 0;
|
||||
*completed = 0;
|
||||
}
|
||||
static inline void rcutorture_record_test_transition(void)
|
||||
{
|
||||
}
|
||||
static inline void rcutorture_record_progress(unsigned long vernum)
|
||||
{
|
||||
}
|
||||
static inline bool rcu_irq_enter_disabled(void)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
#ifdef CONFIG_RCU_TRACE
|
||||
void do_trace_rcu_torture_read(const char *rcutorturename,
|
||||
struct rcu_head *rhp,
|
||||
unsigned long secs,
|
||||
unsigned long c_old,
|
||||
unsigned long c);
|
||||
#else
|
||||
#define do_trace_rcu_torture_read(rcutorturename, rhp, secs, c_old, c) \
|
||||
do { } while (0)
|
||||
#endif
|
||||
#endif
|
||||
|
||||
#define UINT_CMP_GE(a, b) (UINT_MAX / 2 >= (a) - (b))
|
||||
#define UINT_CMP_LT(a, b) (UINT_MAX / 2 < (a) - (b))
|
||||
#define ULONG_CMP_GE(a, b) (ULONG_MAX / 2 >= (a) - (b))
|
||||
#define ULONG_CMP_LT(a, b) (ULONG_MAX / 2 < (a) - (b))
|
||||
#define ulong2long(a) (*(long *)(&(a)))
|
||||
@@ -139,115 +50,14 @@ void do_trace_rcu_torture_read(const char *rcutorturename,
|
||||
/* Exported common interfaces */
|
||||
|
||||
#ifdef CONFIG_PREEMPT_RCU
|
||||
|
||||
/**
|
||||
* call_rcu() - Queue an RCU callback for invocation after a grace period.
|
||||
* @head: structure to be used for queueing the RCU updates.
|
||||
* @func: actual callback function to be invoked after the grace period
|
||||
*
|
||||
* The callback function will be invoked some time after a full grace
|
||||
* period elapses, in other words after all pre-existing RCU read-side
|
||||
* critical sections have completed. However, the callback function
|
||||
* might well execute concurrently with RCU read-side critical sections
|
||||
* that started after call_rcu() was invoked. RCU read-side critical
|
||||
* sections are delimited by rcu_read_lock() and rcu_read_unlock(),
|
||||
* and may be nested.
|
||||
*
|
||||
* Note that all CPUs must agree that the grace period extended beyond
|
||||
* all pre-existing RCU read-side critical section. On systems with more
|
||||
* than one CPU, this means that when "func()" is invoked, each CPU is
|
||||
* guaranteed to have executed a full memory barrier since the end of its
|
||||
* last RCU read-side critical section whose beginning preceded the call
|
||||
* to call_rcu(). It also means that each CPU executing an RCU read-side
|
||||
* critical section that continues beyond the start of "func()" must have
|
||||
* executed a memory barrier after the call_rcu() but before the beginning
|
||||
* of that RCU read-side critical section. Note that these guarantees
|
||||
* include CPUs that are offline, idle, or executing in user mode, as
|
||||
* well as CPUs that are executing in the kernel.
|
||||
*
|
||||
* Furthermore, if CPU A invoked call_rcu() and CPU B invoked the
|
||||
* resulting RCU callback function "func()", then both CPU A and CPU B are
|
||||
* guaranteed to execute a full memory barrier during the time interval
|
||||
* between the call to call_rcu() and the invocation of "func()" -- even
|
||||
* if CPU A and CPU B are the same CPU (but again only if the system has
|
||||
* more than one CPU).
|
||||
*/
|
||||
void call_rcu(struct rcu_head *head,
|
||||
rcu_callback_t func);
|
||||
|
||||
void call_rcu(struct rcu_head *head, rcu_callback_t func);
|
||||
#else /* #ifdef CONFIG_PREEMPT_RCU */
|
||||
|
||||
/* In classic RCU, call_rcu() is just call_rcu_sched(). */
|
||||
#define call_rcu call_rcu_sched
|
||||
|
||||
#endif /* #else #ifdef CONFIG_PREEMPT_RCU */
|
||||
|
||||
/**
|
||||
* call_rcu_bh() - Queue an RCU for invocation after a quicker grace period.
|
||||
* @head: structure to be used for queueing the RCU updates.
|
||||
* @func: actual callback function to be invoked after the grace period
|
||||
*
|
||||
* The callback function will be invoked some time after a full grace
|
||||
* period elapses, in other words after all currently executing RCU
|
||||
* read-side critical sections have completed. call_rcu_bh() assumes
|
||||
* that the read-side critical sections end on completion of a softirq
|
||||
* handler. This means that read-side critical sections in process
|
||||
* context must not be interrupted by softirqs. This interface is to be
|
||||
* used when most of the read-side critical sections are in softirq context.
|
||||
* RCU read-side critical sections are delimited by :
|
||||
* - rcu_read_lock() and rcu_read_unlock(), if in interrupt context.
|
||||
* OR
|
||||
* - rcu_read_lock_bh() and rcu_read_unlock_bh(), if in process context.
|
||||
* These may be nested.
|
||||
*
|
||||
* See the description of call_rcu() for more detailed information on
|
||||
* memory ordering guarantees.
|
||||
*/
|
||||
void call_rcu_bh(struct rcu_head *head,
|
||||
rcu_callback_t func);
|
||||
|
||||
/**
|
||||
* call_rcu_sched() - Queue an RCU for invocation after sched grace period.
|
||||
* @head: structure to be used for queueing the RCU updates.
|
||||
* @func: actual callback function to be invoked after the grace period
|
||||
*
|
||||
* The callback function will be invoked some time after a full grace
|
||||
* period elapses, in other words after all currently executing RCU
|
||||
* read-side critical sections have completed. call_rcu_sched() assumes
|
||||
* that the read-side critical sections end on enabling of preemption
|
||||
* or on voluntary preemption.
|
||||
* RCU read-side critical sections are delimited by :
|
||||
* - rcu_read_lock_sched() and rcu_read_unlock_sched(),
|
||||
* OR
|
||||
* anything that disables preemption.
|
||||
* These may be nested.
|
||||
*
|
||||
* See the description of call_rcu() for more detailed information on
|
||||
* memory ordering guarantees.
|
||||
*/
|
||||
void call_rcu_sched(struct rcu_head *head,
|
||||
rcu_callback_t func);
|
||||
|
||||
void call_rcu_bh(struct rcu_head *head, rcu_callback_t func);
|
||||
void call_rcu_sched(struct rcu_head *head, rcu_callback_t func);
|
||||
void synchronize_sched(void);
|
||||
|
||||
/**
|
||||
* call_rcu_tasks() - Queue an RCU for invocation task-based grace period
|
||||
* @head: structure to be used for queueing the RCU updates.
|
||||
* @func: actual callback function to be invoked after the grace period
|
||||
*
|
||||
* The callback function will be invoked some time after a full grace
|
||||
* period elapses, in other words after all currently executing RCU
|
||||
* read-side critical sections have completed. call_rcu_tasks() assumes
|
||||
* that the read-side critical sections end at a voluntary context
|
||||
* switch (not a preemption!), entry into idle, or transition to usermode
|
||||
* execution. As such, there are no read-side primitives analogous to
|
||||
* rcu_read_lock() and rcu_read_unlock() because this primitive is intended
|
||||
* to determine that all tasks have passed through a safe state, not so
|
||||
* much for data-strcuture synchronization.
|
||||
*
|
||||
* See the description of call_rcu() for more detailed information on
|
||||
* memory ordering guarantees.
|
||||
*/
|
||||
void call_rcu_tasks(struct rcu_head *head, rcu_callback_t func);
|
||||
void synchronize_rcu_tasks(void);
|
||||
void rcu_barrier_tasks(void);
|
||||
@@ -301,22 +111,12 @@ void rcu_check_callbacks(int user);
|
||||
void rcu_report_dead(unsigned int cpu);
|
||||
void rcu_cpu_starting(unsigned int cpu);
|
||||
|
||||
#ifndef CONFIG_TINY_RCU
|
||||
void rcu_end_inkernel_boot(void);
|
||||
#else /* #ifndef CONFIG_TINY_RCU */
|
||||
static inline void rcu_end_inkernel_boot(void) { }
|
||||
#endif /* #ifndef CONFIG_TINY_RCU */
|
||||
|
||||
#ifdef CONFIG_RCU_STALL_COMMON
|
||||
void rcu_sysrq_start(void);
|
||||
void rcu_sysrq_end(void);
|
||||
#else /* #ifdef CONFIG_RCU_STALL_COMMON */
|
||||
static inline void rcu_sysrq_start(void)
|
||||
{
|
||||
}
|
||||
static inline void rcu_sysrq_end(void)
|
||||
{
|
||||
}
|
||||
static inline void rcu_sysrq_start(void) { }
|
||||
static inline void rcu_sysrq_end(void) { }
|
||||
#endif /* #else #ifdef CONFIG_RCU_STALL_COMMON */
|
||||
|
||||
#ifdef CONFIG_NO_HZ_FULL
|
||||
@@ -330,9 +130,7 @@ static inline void rcu_user_exit(void) { }
|
||||
#ifdef CONFIG_RCU_NOCB_CPU
|
||||
void rcu_init_nohz(void);
|
||||
#else /* #ifdef CONFIG_RCU_NOCB_CPU */
|
||||
static inline void rcu_init_nohz(void)
|
||||
{
|
||||
}
|
||||
static inline void rcu_init_nohz(void) { }
|
||||
#endif /* #else #ifdef CONFIG_RCU_NOCB_CPU */
|
||||
|
||||
/**
|
||||
@@ -397,10 +195,6 @@ do { \
|
||||
rcu_note_voluntary_context_switch(current); \
|
||||
} while (0)
|
||||
|
||||
#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) || defined(CONFIG_SMP)
|
||||
bool __rcu_is_watching(void);
|
||||
#endif /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) || defined(CONFIG_SMP) */
|
||||
|
||||
/*
|
||||
* Infrastructure to implement the synchronize_() primitives in
|
||||
* TREE_RCU and rcu_barrier_() primitives in TINY_RCU.
|
||||
@@ -414,10 +208,6 @@ bool __rcu_is_watching(void);
|
||||
#error "Unknown RCU implementation specified to kernel configuration"
|
||||
#endif
|
||||
|
||||
#define RCU_SCHEDULER_INACTIVE 0
|
||||
#define RCU_SCHEDULER_INIT 1
|
||||
#define RCU_SCHEDULER_RUNNING 2
|
||||
|
||||
/*
|
||||
* init_rcu_head_on_stack()/destroy_rcu_head_on_stack() are needed for dynamic
|
||||
* initialization and destruction of rcu_head on the stack. rcu_head structures
|
||||
@@ -430,30 +220,16 @@ void destroy_rcu_head(struct rcu_head *head);
|
||||
void init_rcu_head_on_stack(struct rcu_head *head);
|
||||
void destroy_rcu_head_on_stack(struct rcu_head *head);
|
||||
#else /* !CONFIG_DEBUG_OBJECTS_RCU_HEAD */
|
||||
static inline void init_rcu_head(struct rcu_head *head)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void destroy_rcu_head(struct rcu_head *head)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void init_rcu_head_on_stack(struct rcu_head *head)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void destroy_rcu_head_on_stack(struct rcu_head *head)
|
||||
{
|
||||
}
|
||||
static inline void init_rcu_head(struct rcu_head *head) { }
|
||||
static inline void destroy_rcu_head(struct rcu_head *head) { }
|
||||
static inline void init_rcu_head_on_stack(struct rcu_head *head) { }
|
||||
static inline void destroy_rcu_head_on_stack(struct rcu_head *head) { }
|
||||
#endif /* #else !CONFIG_DEBUG_OBJECTS_RCU_HEAD */
|
||||
|
||||
#if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU)
|
||||
bool rcu_lockdep_current_cpu_online(void);
|
||||
#else /* #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU) */
|
||||
static inline bool rcu_lockdep_current_cpu_online(void)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
static inline bool rcu_lockdep_current_cpu_online(void) { return true; }
|
||||
#endif /* #else #if defined(CONFIG_HOTPLUG_CPU) && defined(CONFIG_PROVE_RCU) */
|
||||
|
||||
#ifdef CONFIG_DEBUG_LOCK_ALLOC
|
||||
@@ -473,18 +249,8 @@ extern struct lockdep_map rcu_bh_lock_map;
|
||||
extern struct lockdep_map rcu_sched_lock_map;
|
||||
extern struct lockdep_map rcu_callback_map;
|
||||
int debug_lockdep_rcu_enabled(void);
|
||||
|
||||
int rcu_read_lock_held(void);
|
||||
int rcu_read_lock_bh_held(void);
|
||||
|
||||
/**
|
||||
* rcu_read_lock_sched_held() - might we be in RCU-sched read-side critical section?
|
||||
*
|
||||
* If CONFIG_DEBUG_LOCK_ALLOC is selected, returns nonzero iff in an
|
||||
* RCU-sched read-side critical section. In absence of
|
||||
* CONFIG_DEBUG_LOCK_ALLOC, this assumes we are in an RCU-sched read-side
|
||||
* critical section unless it can prove otherwise.
|
||||
*/
|
||||
int rcu_read_lock_sched_held(void);
|
||||
|
||||
#else /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
|
||||
@@ -531,9 +297,7 @@ static inline void rcu_preempt_sleep_check(void)
|
||||
"Illegal context switch in RCU read-side critical section");
|
||||
}
|
||||
#else /* #ifdef CONFIG_PROVE_RCU */
|
||||
static inline void rcu_preempt_sleep_check(void)
|
||||
{
|
||||
}
|
||||
static inline void rcu_preempt_sleep_check(void) { }
|
||||
#endif /* #else #ifdef CONFIG_PROVE_RCU */
|
||||
|
||||
#define rcu_sleep_check() \
|
||||
@@ -1084,52 +848,6 @@ static inline notrace void rcu_read_unlock_sched_notrace(void)
|
||||
#define kfree_rcu(ptr, rcu_head) \
|
||||
__kfree_rcu(&((ptr)->rcu_head), offsetof(typeof(*(ptr)), rcu_head))
|
||||
|
||||
#ifdef CONFIG_TINY_RCU
|
||||
static inline int rcu_needs_cpu(u64 basemono, u64 *nextevt)
|
||||
{
|
||||
*nextevt = KTIME_MAX;
|
||||
return 0;
|
||||
}
|
||||
#endif /* #ifdef CONFIG_TINY_RCU */
|
||||
|
||||
#if defined(CONFIG_RCU_NOCB_CPU_ALL)
|
||||
static inline bool rcu_is_nocb_cpu(int cpu) { return true; }
|
||||
#elif defined(CONFIG_RCU_NOCB_CPU)
|
||||
bool rcu_is_nocb_cpu(int cpu);
|
||||
#else
|
||||
static inline bool rcu_is_nocb_cpu(int cpu) { return false; }
|
||||
#endif
|
||||
|
||||
|
||||
/* Only for use by adaptive-ticks code. */
|
||||
#ifdef CONFIG_NO_HZ_FULL_SYSIDLE
|
||||
bool rcu_sys_is_idle(void);
|
||||
void rcu_sysidle_force_exit(void);
|
||||
#else /* #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
|
||||
|
||||
static inline bool rcu_sys_is_idle(void)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
static inline void rcu_sysidle_force_exit(void)
|
||||
{
|
||||
}
|
||||
|
||||
#endif /* #else #ifdef CONFIG_NO_HZ_FULL_SYSIDLE */
|
||||
|
||||
|
||||
/*
|
||||
* Dump the ftrace buffer, but only one time per callsite per boot.
|
||||
*/
|
||||
#define rcu_ftrace_dump(oops_dump_mode) \
|
||||
do { \
|
||||
static atomic_t ___rfd_beenhere = ATOMIC_INIT(0); \
|
||||
\
|
||||
if (!atomic_read(&___rfd_beenhere) && \
|
||||
!atomic_xchg(&___rfd_beenhere, 1)) \
|
||||
ftrace_dump(oops_dump_mode); \
|
||||
} while (0)
|
||||
|
||||
/*
|
||||
* Place this after a lock-acquisition primitive to guarantee that
|
||||
|
||||
+24
-147
@@ -25,7 +25,7 @@
|
||||
#ifndef __LINUX_TINY_H
|
||||
#define __LINUX_TINY_H
|
||||
|
||||
#include <linux/cache.h>
|
||||
#include <linux/ktime.h>
|
||||
|
||||
struct rcu_dynticks;
|
||||
static inline int rcu_dynticks_snap(struct rcu_dynticks *rdtp)
|
||||
@@ -33,10 +33,8 @@ static inline int rcu_dynticks_snap(struct rcu_dynticks *rdtp)
|
||||
return 0;
|
||||
}
|
||||
|
||||
static inline bool rcu_eqs_special_set(int cpu)
|
||||
{
|
||||
return false; /* Never flag non-existent other CPUs! */
|
||||
}
|
||||
/* Never flag non-existent other CPUs! */
|
||||
static inline bool rcu_eqs_special_set(int cpu) { return false; }
|
||||
|
||||
static inline unsigned long get_state_synchronize_rcu(void)
|
||||
{
|
||||
@@ -98,159 +96,38 @@ static inline void kfree_call_rcu(struct rcu_head *head,
|
||||
rcu_note_voluntary_context_switch_lite(current); \
|
||||
} while (0)
|
||||
|
||||
static inline int rcu_needs_cpu(u64 basemono, u64 *nextevt)
|
||||
{
|
||||
*nextevt = KTIME_MAX;
|
||||
return 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* Take advantage of the fact that there is only one CPU, which
|
||||
* allows us to ignore virtualization-based context switches.
|
||||
*/
|
||||
static inline void rcu_virt_note_context_switch(int cpu)
|
||||
{
|
||||
}
|
||||
|
||||
/*
|
||||
* Return the number of grace periods started.
|
||||
*/
|
||||
static inline unsigned long rcu_batches_started(void)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* Return the number of bottom-half grace periods started.
|
||||
*/
|
||||
static inline unsigned long rcu_batches_started_bh(void)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* Return the number of sched grace periods started.
|
||||
*/
|
||||
static inline unsigned long rcu_batches_started_sched(void)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* Return the number of grace periods completed.
|
||||
*/
|
||||
static inline unsigned long rcu_batches_completed(void)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* Return the number of bottom-half grace periods completed.
|
||||
*/
|
||||
static inline unsigned long rcu_batches_completed_bh(void)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* Return the number of sched grace periods completed.
|
||||
*/
|
||||
static inline unsigned long rcu_batches_completed_sched(void)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* Return the number of expedited grace periods completed.
|
||||
*/
|
||||
static inline unsigned long rcu_exp_batches_completed(void)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
/*
|
||||
* Return the number of expedited sched grace periods completed.
|
||||
*/
|
||||
static inline unsigned long rcu_exp_batches_completed_sched(void)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
static inline void rcu_force_quiescent_state(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void rcu_bh_force_quiescent_state(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void rcu_sched_force_quiescent_state(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void show_rcu_gp_kthreads(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void rcu_cpu_stall_reset(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void rcu_idle_enter(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void rcu_idle_exit(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void rcu_irq_enter(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void rcu_irq_exit_irqson(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void rcu_irq_enter_irqson(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void rcu_irq_exit(void)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void exit_rcu(void)
|
||||
{
|
||||
}
|
||||
static inline void rcu_virt_note_context_switch(int cpu) { }
|
||||
static inline void rcu_cpu_stall_reset(void) { }
|
||||
static inline void rcu_idle_enter(void) { }
|
||||
static inline void rcu_idle_exit(void) { }
|
||||
static inline void rcu_irq_enter(void) { }
|
||||
static inline bool rcu_irq_enter_disabled(void) { return false; }
|
||||
static inline void rcu_irq_exit_irqson(void) { }
|
||||
static inline void rcu_irq_enter_irqson(void) { }
|
||||
static inline void rcu_irq_exit(void) { }
|
||||
static inline void exit_rcu(void) { }
|
||||
|
||||
#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU)
|
||||
extern int rcu_scheduler_active __read_mostly;
|
||||
void rcu_scheduler_starting(void);
|
||||
#else /* #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU) */
|
||||
static inline void rcu_scheduler_starting(void)
|
||||
{
|
||||
}
|
||||
static inline void rcu_scheduler_starting(void) { }
|
||||
#endif /* #else #if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_SRCU) */
|
||||
static inline void rcu_end_inkernel_boot(void) { }
|
||||
static inline bool rcu_is_watching(void) { return true; }
|
||||
|
||||
#if defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE)
|
||||
|
||||
static inline bool rcu_is_watching(void)
|
||||
{
|
||||
return __rcu_is_watching();
|
||||
}
|
||||
|
||||
#else /* defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) */
|
||||
|
||||
static inline bool rcu_is_watching(void)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
#endif /* #else defined(CONFIG_DEBUG_LOCK_ALLOC) || defined(CONFIG_RCU_TRACE) */
|
||||
|
||||
static inline void rcu_request_urgent_qs_task(struct task_struct *t)
|
||||
{
|
||||
}
|
||||
|
||||
static inline void rcu_all_qs(void)
|
||||
{
|
||||
barrier(); /* Avoid RCU read-side critical sections leaking across. */
|
||||
}
|
||||
/* Avoid RCU read-side critical sections leaking across. */
|
||||
static inline void rcu_all_qs(void) { barrier(); }
|
||||
|
||||
/* RCUtree hotplug events */
|
||||
#define rcutree_prepare_cpu NULL
|
||||
|
||||
+2
-19
@@ -79,37 +79,20 @@ void cond_synchronize_rcu(unsigned long oldstate);
|
||||
unsigned long get_state_synchronize_sched(void);
|
||||
void cond_synchronize_sched(unsigned long oldstate);
|
||||
|
||||
extern unsigned long rcutorture_testseq;
|
||||
extern unsigned long rcutorture_vernum;
|
||||
unsigned long rcu_batches_started(void);
|
||||
unsigned long rcu_batches_started_bh(void);
|
||||
unsigned long rcu_batches_started_sched(void);
|
||||
unsigned long rcu_batches_completed(void);
|
||||
unsigned long rcu_batches_completed_bh(void);
|
||||
unsigned long rcu_batches_completed_sched(void);
|
||||
unsigned long rcu_exp_batches_completed(void);
|
||||
unsigned long rcu_exp_batches_completed_sched(void);
|
||||
void show_rcu_gp_kthreads(void);
|
||||
|
||||
void rcu_force_quiescent_state(void);
|
||||
void rcu_bh_force_quiescent_state(void);
|
||||
void rcu_sched_force_quiescent_state(void);
|
||||
|
||||
void rcu_idle_enter(void);
|
||||
void rcu_idle_exit(void);
|
||||
void rcu_irq_enter(void);
|
||||
void rcu_irq_exit(void);
|
||||
void rcu_irq_enter_irqson(void);
|
||||
void rcu_irq_exit_irqson(void);
|
||||
bool rcu_irq_enter_disabled(void);
|
||||
|
||||
void exit_rcu(void);
|
||||
|
||||
void rcu_scheduler_starting(void);
|
||||
extern int rcu_scheduler_active __read_mostly;
|
||||
|
||||
void rcu_end_inkernel_boot(void);
|
||||
bool rcu_is_watching(void);
|
||||
void rcu_request_urgent_qs_task(struct task_struct *t);
|
||||
|
||||
void rcu_all_qs(void);
|
||||
|
||||
/* RCUtree hotplug events */
|
||||
|
||||
@@ -369,6 +369,26 @@ static __always_inline int spin_trylock_irq(spinlock_t *lock)
|
||||
raw_spin_trylock_irqsave(spinlock_check(lock), flags); \
|
||||
})
|
||||
|
||||
/**
|
||||
* spin_unlock_wait - Interpose between successive critical sections
|
||||
* @lock: the spinlock whose critical sections are to be interposed.
|
||||
*
|
||||
* Semantically this is equivalent to a spin_lock() immediately
|
||||
* followed by a spin_unlock(). However, most architectures have
|
||||
* more efficient implementations in which the spin_unlock_wait()
|
||||
* cannot block concurrent lock acquisition, and in some cases
|
||||
* where spin_unlock_wait() does not write to the lock variable.
|
||||
* Nevertheless, spin_unlock_wait() can have high overhead, so if
|
||||
* you feel the need to use it, please check to see if there is
|
||||
* a better way to get your job done.
|
||||
*
|
||||
* The ordering guarantees provided by spin_unlock_wait() are:
|
||||
*
|
||||
* 1. All accesses preceding the spin_unlock_wait() happen before
|
||||
* any accesses in later critical sections for this same lock.
|
||||
* 2. All accesses following the spin_unlock_wait() happen after
|
||||
* any accesses in earlier critical sections for this same lock.
|
||||
*/
|
||||
static __always_inline void spin_unlock_wait(spinlock_t *lock)
|
||||
{
|
||||
raw_spin_unlock_wait(&lock->rlock);
|
||||
|
||||
+4
-21
@@ -60,32 +60,15 @@ int init_srcu_struct(struct srcu_struct *sp);
|
||||
#include <linux/srcutiny.h>
|
||||
#elif defined(CONFIG_TREE_SRCU)
|
||||
#include <linux/srcutree.h>
|
||||
#elif defined(CONFIG_CLASSIC_SRCU)
|
||||
#include <linux/srcuclassic.h>
|
||||
#else
|
||||
#elif defined(CONFIG_SRCU)
|
||||
#error "Unknown SRCU implementation specified to kernel configuration"
|
||||
#else
|
||||
/* Dummy definition for things like notifiers. Actual use gets link error. */
|
||||
struct srcu_struct { };
|
||||
#endif
|
||||
|
||||
/**
|
||||
* call_srcu() - Queue a callback for invocation after an SRCU grace period
|
||||
* @sp: srcu_struct in queue the callback
|
||||
* @head: structure to be used for queueing the SRCU callback.
|
||||
* @func: function to be invoked after the SRCU grace period
|
||||
*
|
||||
* The callback function will be invoked some time after a full SRCU
|
||||
* grace period elapses, in other words after all pre-existing SRCU
|
||||
* read-side critical sections have completed. However, the callback
|
||||
* function might well execute concurrently with other SRCU read-side
|
||||
* critical sections that started after call_srcu() was invoked. SRCU
|
||||
* read-side critical sections are delimited by srcu_read_lock() and
|
||||
* srcu_read_unlock(), and may be nested.
|
||||
*
|
||||
* The callback will be invoked from process context, but must nevertheless
|
||||
* be fast and must not block.
|
||||
*/
|
||||
void call_srcu(struct srcu_struct *sp, struct rcu_head *head,
|
||||
void (*func)(struct rcu_head *head));
|
||||
|
||||
void cleanup_srcu_struct(struct srcu_struct *sp);
|
||||
int __srcu_read_lock(struct srcu_struct *sp) __acquires(sp);
|
||||
void __srcu_read_unlock(struct srcu_struct *sp, int idx) __releases(sp);
|
||||
|
||||
@@ -1,115 +0,0 @@
|
||||
/*
|
||||
* Sleepable Read-Copy Update mechanism for mutual exclusion,
|
||||
* classic v4.11 variant.
|
||||
*
|
||||
* This program is free software; you can redistribute it and/or modify
|
||||
* it under the terms of the GNU General Public License as published by
|
||||
* the Free Software Foundation; either version 2 of the License, or
|
||||
* (at your option) any later version.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful,
|
||||
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* You should have received a copy of the GNU General Public License
|
||||
* along with this program; if not, you can access it online at
|
||||
* http://www.gnu.org/licenses/gpl-2.0.html.
|
||||
*
|
||||
* Copyright (C) IBM Corporation, 2017
|
||||
*
|
||||
* Author: Paul McKenney <paulmck@us.ibm.com>
|
||||
*/
|
||||
|
||||
#ifndef _LINUX_SRCU_CLASSIC_H
|
||||
#define _LINUX_SRCU_CLASSIC_H
|
||||
|
||||
struct srcu_array {
|
||||
unsigned long lock_count[2];
|
||||
unsigned long unlock_count[2];
|
||||
};
|
||||
|
||||
struct rcu_batch {
|
||||
struct rcu_head *head, **tail;
|
||||
};
|
||||
|
||||
#define RCU_BATCH_INIT(name) { NULL, &(name.head) }
|
||||
|
||||
struct srcu_struct {
|
||||
unsigned long completed;
|
||||
struct srcu_array __percpu *per_cpu_ref;
|
||||
spinlock_t queue_lock; /* protect ->batch_queue, ->running */
|
||||
bool running;
|
||||
/* callbacks just queued */
|
||||
struct rcu_batch batch_queue;
|
||||
/* callbacks try to do the first check_zero */
|
||||
struct rcu_batch batch_check0;
|
||||
/* callbacks done with the first check_zero and the flip */
|
||||
struct rcu_batch batch_check1;
|
||||
struct rcu_batch batch_done;
|
||||
struct delayed_work work;
|
||||
#ifdef CONFIG_DEBUG_LOCK_ALLOC
|
||||
struct lockdep_map dep_map;
|
||||
#endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */
|
||||
};
|
||||
|
||||
void process_srcu(struct work_struct *work);
|
||||
|
||||
#define __SRCU_STRUCT_INIT(name) \
|
||||
{ \
|
||||
.completed = -300, \
|
||||
.per_cpu_ref = &name##_srcu_array, \
|
||||
.queue_lock = __SPIN_LOCK_UNLOCKED(name.queue_lock), \
|
||||
.running = false, \
|
||||
.batch_queue = RCU_BATCH_INIT(name.batch_queue), \
|
||||
.batch_check0 = RCU_BATCH_INIT(name.batch_check0), \
|
||||
.batch_check1 = RCU_BATCH_INIT(name.batch_check1), \
|
||||
.batch_done = RCU_BATCH_INIT(name.batch_done), \
|
||||
.work = __DELAYED_WORK_INITIALIZER(name.work, process_srcu, 0),\
|
||||
__SRCU_DEP_MAP_INIT(name) \
|
||||
}
|
||||
|
||||
/*
|
||||
* Define and initialize a srcu struct at build time.
|
||||
* Do -not- call init_srcu_struct() nor cleanup_srcu_struct() on it.
|
||||
*
|
||||
* Note that although DEFINE_STATIC_SRCU() hides the name from other
|
||||
* files, the per-CPU variable rules nevertheless require that the
|
||||
* chosen name be globally unique. These rules also prohibit use of
|
||||
* DEFINE_STATIC_SRCU() within a function. If these rules are too
|
||||
* restrictive, declare the srcu_struct manually. For example, in
|
||||
* each file:
|
||||
*
|
||||
* static struct srcu_struct my_srcu;
|
||||
*
|
||||
* Then, before the first use of each my_srcu, manually initialize it:
|
||||
*
|
||||
* init_srcu_struct(&my_srcu);
|
||||
*
|
||||
* See include/linux/percpu-defs.h for the rules on per-CPU variables.
|
||||
*/
|
||||
#define __DEFINE_SRCU(name, is_static) \
|
||||
static DEFINE_PER_CPU(struct srcu_array, name##_srcu_array);\
|
||||
is_static struct srcu_struct name = __SRCU_STRUCT_INIT(name)
|
||||
#define DEFINE_SRCU(name) __DEFINE_SRCU(name, /* not static */)
|
||||
#define DEFINE_STATIC_SRCU(name) __DEFINE_SRCU(name, static)
|
||||
|
||||
void synchronize_srcu_expedited(struct srcu_struct *sp);
|
||||
void srcu_barrier(struct srcu_struct *sp);
|
||||
unsigned long srcu_batches_completed(struct srcu_struct *sp);
|
||||
|
||||
static inline void srcutorture_get_gp_data(enum rcutorture_type test_type,
|
||||
struct srcu_struct *sp, int *flags,
|
||||
unsigned long *gpnum,
|
||||
unsigned long *completed)
|
||||
{
|
||||
if (test_type != SRCU_FLAVOR)
|
||||
return;
|
||||
*flags = 0;
|
||||
*completed = sp->completed;
|
||||
*gpnum = *completed;
|
||||
if (sp->batch_queue.head || sp->batch_check0.head || sp->batch_check0.head)
|
||||
(*gpnum)++;
|
||||
}
|
||||
|
||||
#endif
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user