You've already forked linux-apfs
mirror of
https://github.com/linux-apfs/linux-apfs.git
synced 2026-05-01 15:00:59 -07:00
Merge branch 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull locking updates from Ingo Molnar: - Lots of tidying up changes all across the map for Linux's formal memory/locking-model tooling, by Alan Stern, Akira Yokosawa, Andrea Parri, Paul E. McKenney and SeongJae Park. Notable changes beyond an overall update in the tooling itself is the tidying up of spin_is_locked() semantics, which spills over into the kernel proper as well. - qspinlock improvements: the locking algorithm now guarantees forward progress whereas the previous implementation in mainline could starve threads indefinitely in cmpxchg() loops. Also other related cleanups to the qspinlock code (Will Deacon) - misc smaller improvements, cleanups and fixes all across the locking subsystem * 'locking-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (51 commits) locking/rwsem: Simplify the is-owner-spinnable checks tools/memory-model: Add reference for 'Simplifying ARM concurrency' tools/memory-model: Update ASPLOS information MAINTAINERS, tools/memory-model: Update e-mail address for Andrea Parri tools/memory-model: Fix coding style in 'lock.cat' tools/memory-model: Remove out-of-date comments and code from lock.cat tools/memory-model: Improve mixed-access checking in lock.cat tools/memory-model: Improve comments in lock.cat tools/memory-model: Remove duplicated code from lock.cat tools/memory-model: Flag "cumulativity" and "propagation" tests tools/memory-model: Add model support for spin_is_locked() tools/memory-model: Add scripts to test memory model tools/memory-model: Fix coding style in 'linux-kernel.def' tools/memory-model: Model 'smp_store_mb()' tools/memory-order: Update the cheat-sheet to show that smp_mb__after_atomic() orders later RMW operations tools/memory-order: Improve key for SELF and SV tools/memory-model: Fix cheat sheet typo tools/memory-model: Update required version of herdtools7 tools/memory-model: Redefine rb in terms of rcu-fence tools/memory-model: Rename link and rcu-path to rcu-link and rb ...
This commit is contained in:
Linus Torvalds
@@ -111,7 +111,6 @@ If the compiler can prove that do_something() does not store to the
|
||||
variable a, then the compiler is within its rights transforming this to
|
||||
the following::
|
||||
|
||||
tmp = a;
|
||||
if (a > 0)
|
||||
for (;;)
|
||||
do_something();
|
||||
@@ -119,7 +118,7 @@ the following::
|
||||
If you don't want the compiler to do this (and you probably don't), then
|
||||
you should use something like the following::
|
||||
|
||||
while (READ_ONCE(a) < 0)
|
||||
while (READ_ONCE(a) > 0)
|
||||
do_something();
|
||||
|
||||
Alternatively, you could place a barrier() call in the loop.
|
||||
@@ -467,10 +466,12 @@ Like the above, except that these routines return a boolean which
|
||||
indicates whether the changed bit was set _BEFORE_ the atomic bit
|
||||
operation.
|
||||
|
||||
WARNING! It is incredibly important that the value be a boolean,
|
||||
ie. "0" or "1". Do not try to be fancy and save a few instructions by
|
||||
declaring the above to return "long" and just returning something like
|
||||
"old_val & mask" because that will not work.
|
||||
|
||||
.. warning::
|
||||
It is incredibly important that the value be a boolean, ie. "0" or "1".
|
||||
Do not try to be fancy and save a few instructions by declaring the
|
||||
above to return "long" and just returning something like "old_val &
|
||||
mask" because that will not work.
|
||||
|
||||
For one thing, this return value gets truncated to int in many code
|
||||
paths using these interfaces, so on 64-bit if the bit is set in the
|
||||
|
||||
@@ -1920,9 +1920,6 @@ There are some more advanced barrier functions:
|
||||
/* assign ownership */
|
||||
desc->status = DEVICE_OWN;
|
||||
|
||||
/* force memory to sync before notifying device via MMIO */
|
||||
wmb();
|
||||
|
||||
/* notify device of new descriptors */
|
||||
writel(DESC_NOTIFY, doorbell);
|
||||
}
|
||||
@@ -1930,11 +1927,15 @@ There are some more advanced barrier functions:
|
||||
The dma_rmb() allows us guarantee the device has released ownership
|
||||
before we read the data from the descriptor, and the dma_wmb() allows
|
||||
us to guarantee the data is written to the descriptor before the device
|
||||
can see it now has ownership. The wmb() is needed to guarantee that the
|
||||
cache coherent memory writes have completed before attempting a write to
|
||||
the cache incoherent MMIO region.
|
||||
can see it now has ownership. Note that, when using writel(), a prior
|
||||
wmb() is not needed to guarantee that the cache coherent memory writes
|
||||
have completed before writing to the MMIO region. The cheaper
|
||||
writel_relaxed() does not provide this guarantee and must not be used
|
||||
here.
|
||||
|
||||
See Documentation/DMA-API.txt for more information on consistent memory.
|
||||
See the subsection "Kernel I/O barrier effects" for more information on
|
||||
relaxed I/O accessors and the Documentation/DMA-API.txt file for more
|
||||
information on consistent memory.
|
||||
|
||||
|
||||
MMIO WRITE BARRIER
|
||||
|
||||
@@ -36,6 +36,9 @@ Documentation/memory-barriers.txt
|
||||
부분도 있고, 의도하진 않았지만 사람에 의해 쓰였다보니 불완전한 부분도 있습니다.
|
||||
이 문서는 리눅스에서 제공하는 다양한 메모리 배리어들을 사용하기 위한
|
||||
안내서입니다만, 뭔가 이상하다 싶으면 (그런게 많을 겁니다) 질문을 부탁드립니다.
|
||||
일부 이상한 점들은 공식적인 메모리 일관성 모델과 tools/memory-model/ 에 있는
|
||||
관련 문서를 참고해서 해결될 수 있을 겁니다. 그러나, 이 메모리 모델조차도 그
|
||||
관리자들의 의견의 집합으로 봐야지, 절대 옳은 예언자로 신봉해선 안될 겁니다.
|
||||
|
||||
다시 말하지만, 이 문서는 리눅스가 하드웨어에 기대하는 사항에 대한 명세서가
|
||||
아닙니다.
|
||||
@@ -77,7 +80,7 @@ Documentation/memory-barriers.txt
|
||||
|
||||
- 메모리 배리어의 종류.
|
||||
- 메모리 배리어에 대해 가정해선 안될 것.
|
||||
- 데이터 의존성 배리어.
|
||||
- 데이터 의존성 배리어 (역사적).
|
||||
- 컨트롤 의존성.
|
||||
- SMP 배리어 짝맞추기.
|
||||
- 메모리 배리어 시퀀스의 예.
|
||||
@@ -255,17 +258,20 @@ CPU 에게 기대할 수 있는 최소한의 보장사항 몇가지가 있습니
|
||||
(*) 어떤 CPU 든, 의존성이 존재하는 메모리 액세스들은 해당 CPU 자신에게
|
||||
있어서는 순서대로 메모리 시스템에 수행 요청됩니다. 즉, 다음에 대해서:
|
||||
|
||||
Q = READ_ONCE(P); smp_read_barrier_depends(); D = READ_ONCE(*Q);
|
||||
Q = READ_ONCE(P); D = READ_ONCE(*Q);
|
||||
|
||||
CPU 는 다음과 같은 메모리 오퍼레이션 시퀀스를 수행 요청합니다:
|
||||
|
||||
Q = LOAD P, D = LOAD *Q
|
||||
|
||||
그리고 그 시퀀스 내에서의 순서는 항상 지켜집니다. 대부분의 시스템에서
|
||||
smp_read_barrier_depends() 는 아무일도 안하지만 DEC Alpha 에서는
|
||||
명시적으로 사용되어야 합니다. 보통의 경우에는 smp_read_barrier_depends()
|
||||
를 직접 사용하는 대신 rcu_dereference() 같은 것들을 사용해야 함을
|
||||
알아두세요.
|
||||
그리고 그 시퀀스 내에서의 순서는 항상 지켜집니다. 하지만, DEC Alpha 에서
|
||||
READ_ONCE() 는 메모리 배리어 명령도 내게 되어 있어서, DEC Alpha CPU 는
|
||||
다음과 같은 메모리 오퍼레이션들을 내놓게 됩니다:
|
||||
|
||||
Q = LOAD P, MEMORY_BARRIER, D = LOAD *Q, MEMORY_BARRIER
|
||||
|
||||
DEC Alpha 에서 수행되든 아니든, READ_ONCE() 는 컴파일러로부터의 악영향
|
||||
또한 제거합니다.
|
||||
|
||||
(*) 특정 CPU 내에서 겹치는 영역의 메모리에 행해지는 로드와 스토어 들은 해당
|
||||
CPU 안에서는 순서가 바뀌지 않은 것으로 보여집니다. 즉, 다음에 대해서:
|
||||
@@ -421,8 +427,8 @@ CPU 에게 기대할 수 있는 최소한의 보장사항 몇가지가 있습니
|
||||
데이터 의존성 배리어는 읽기 배리어의 보다 완화된 형태입니다. 두개의 로드
|
||||
오퍼레이션이 있고 두번째 것이 첫번째 것의 결과에 의존하고 있을 때(예:
|
||||
두번째 로드가 참조할 주소를 첫번째 로드가 읽는 경우), 두번째 로드가 읽어올
|
||||
데이터는 첫번째 로드에 의해 그 주소가 얻어지기 전에 업데이트 되어 있음을
|
||||
보장하기 위해서 데이터 의존성 배리어가 필요할 수 있습니다.
|
||||
데이터는 첫번째 로드에 의해 그 주소가 얻어진 뒤에 업데이트 됨을 보장하기
|
||||
위해서 데이터 의존성 배리어가 필요할 수 있습니다.
|
||||
|
||||
데이터 의존성 배리어는 상호 의존적인 로드 오퍼레이션들 사이의 부분적 순서
|
||||
세우기입니다; 스토어 오퍼레이션들이나 독립적인 로드들, 또는 중복되는
|
||||
@@ -570,8 +576,14 @@ ACQUIRE 는 해당 오퍼레이션의 로드 부분에만 적용되고 RELEASE
|
||||
Documentation/DMA-API.txt
|
||||
|
||||
|
||||
데이터 의존성 배리어
|
||||
--------------------
|
||||
데이터 의존성 배리어 (역사적)
|
||||
-----------------------------
|
||||
|
||||
리눅스 커널 v4.15 기준으로, smp_read_barrier_depends() 가 READ_ONCE() 에
|
||||
추가되었는데, 이는 이 섹션에 주의를 기울여야 하는 사람들은 DEC Alpha 아키텍쳐
|
||||
전용 코드를 만드는 사람들과 READ_ONCE() 자체를 만드는 사람들 뿐임을 의미합니다.
|
||||
그런 분들을 위해, 그리고 역사에 관심 있는 분들을 위해, 여기 데이터 의존성
|
||||
배리어에 대한 이야기를 적습니다.
|
||||
|
||||
데이터 의존성 배리어의 사용에 있어 지켜야 하는 사항들은 약간 미묘하고, 데이터
|
||||
의존성 배리어가 사용되어야 하는 상황도 항상 명백하지는 않습니다. 설명을 위해
|
||||
@@ -1787,7 +1799,7 @@ CPU 메모리 배리어
|
||||
범용 mb() smp_mb()
|
||||
쓰기 wmb() smp_wmb()
|
||||
읽기 rmb() smp_rmb()
|
||||
데이터 의존성 read_barrier_depends() smp_read_barrier_depends()
|
||||
데이터 의존성 READ_ONCE()
|
||||
|
||||
|
||||
데이터 의존성 배리어를 제외한 모든 메모리 배리어는 컴파일러 배리어를
|
||||
@@ -2796,8 +2808,9 @@ CPU 2 는 C/D 를 갖습니다)가 병렬로 연결되어 있는 시스템을
|
||||
|
||||
|
||||
여기에 개입하기 위해선, 데이터 의존성 배리어나 읽기 배리어를 로드 오퍼레이션들
|
||||
사이에 넣어야 합니다. 이렇게 함으로써 캐시가 다음 요청을 처리하기 전에 일관성
|
||||
큐를 처리하도록 강제하게 됩니다.
|
||||
사이에 넣어야 합니다 (v4.15 부터는 READ_ONCE() 매크로에 의해 무조건적으로
|
||||
그렇게 됩니다). 이렇게 함으로써 캐시가 다음 요청을 처리하기 전에 일관성 큐를
|
||||
처리하도록 강제하게 됩니다.
|
||||
|
||||
CPU 1 CPU 2 COMMENT
|
||||
=============== =============== =======================================
|
||||
@@ -2826,7 +2839,10 @@ CPU 2 는 C/D 를 갖습니다)가 병렬로 연결되어 있는 시스템을
|
||||
다른 CPU 들도 분할된 캐시를 가지고 있을 수 있지만, 그런 CPU 들은 평범한 메모리
|
||||
액세스를 위해서도 이 분할된 캐시들 사이의 조정을 해야만 합니다. Alpha 는 가장
|
||||
약한 메모리 순서 시맨틱 (semantic) 을 선택함으로써 메모리 배리어가 명시적으로
|
||||
사용되지 않았을 때에는 그런 조정이 필요하지 않게 했습니다.
|
||||
사용되지 않았을 때에는 그런 조정이 필요하지 않게 했으며, 이는 Alpha 가 당시에
|
||||
더 높은 CPU 클락 속도를 가질 수 있게 했습니다. 하지만, (다시 말하건대, v4.15
|
||||
이후부터는) Alpha 아키텍쳐 전용 코드와 READ_ONCE() 매크로 내부에서를 제외하고는
|
||||
smp_read_barrier_depends() 가 사용되지 않아야 함을 알아두시기 바랍니다.
|
||||
|
||||
|
||||
캐시 일관성 VS DMA
|
||||
@@ -2988,7 +3004,9 @@ Alpha CPU 의 일부 버전은 분할된 데이터 캐시를 가지고 있어서
|
||||
메모리 일관성 시스템과 함께 두개의 캐시를 동기화 시켜서, 포인터 변경과 새로운
|
||||
데이터의 발견을 올바른 순서로 일어나게 하기 때문입니다.
|
||||
|
||||
리눅스 커널의 메모리 배리어 모델은 Alpha 에 기초해서 정의되었습니다.
|
||||
리눅스 커널의 메모리 배리어 모델은 Alpha 에 기초해서 정의되었습니다만, v4.15
|
||||
부터는 리눅스 커널이 READ_ONCE() 내에 smp_read_barrier_depends() 를 추가해서
|
||||
Alpha 의 메모리 모델로의 영향력이 크게 줄어들긴 했습니다.
|
||||
|
||||
위의 "캐시 일관성" 서브섹션을 참고하세요.
|
||||
|
||||
|
||||
+2
-1
@@ -8212,7 +8212,7 @@ F: drivers/misc/lkdtm/*
|
||||
|
||||
LINUX KERNEL MEMORY CONSISTENCY MODEL (LKMM)
|
||||
M: Alan Stern <stern@rowland.harvard.edu>
|
||||
M: Andrea Parri <parri.andrea@gmail.com>
|
||||
M: Andrea Parri <andrea.parri@amarulasolutions.com>
|
||||
M: Will Deacon <will.deacon@arm.com>
|
||||
M: Peter Zijlstra <peterz@infradead.org>
|
||||
M: Boqun Feng <boqun.feng@gmail.com>
|
||||
@@ -8319,6 +8319,7 @@ F: Documentation/admin-guide/LSM/LoadPin.rst
|
||||
LOCKING PRIMITIVES
|
||||
M: Peter Zijlstra <peterz@infradead.org>
|
||||
M: Ingo Molnar <mingo@redhat.com>
|
||||
M: Will Deacon <will.deacon@arm.com>
|
||||
L: linux-kernel@vger.kernel.org
|
||||
T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git locking/core
|
||||
S: Maintained
|
||||
|
||||
@@ -122,11 +122,6 @@ static inline int arch_spin_value_unlocked(arch_spinlock_t lock)
|
||||
|
||||
static inline int arch_spin_is_locked(arch_spinlock_t *lock)
|
||||
{
|
||||
/*
|
||||
* Ensure prior spin_lock operations to other locks have completed
|
||||
* on this CPU before we test whether "lock" is locked.
|
||||
*/
|
||||
smp_mb(); /* ^^^ */
|
||||
return !arch_spin_value_unlocked(READ_ONCE(*lock));
|
||||
}
|
||||
|
||||
|
||||
@@ -7,6 +7,14 @@
|
||||
#include <asm-generic/qspinlock_types.h>
|
||||
#include <asm/paravirt.h>
|
||||
|
||||
#define _Q_PENDING_LOOPS (1 << 9)
|
||||
|
||||
#ifdef CONFIG_PARAVIRT_SPINLOCKS
|
||||
extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
|
||||
extern void __pv_init_lock_hash(void);
|
||||
extern void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
|
||||
extern void __raw_callee_save___pv_queued_spin_unlock(struct qspinlock *lock);
|
||||
|
||||
#define queued_spin_unlock queued_spin_unlock
|
||||
/**
|
||||
* queued_spin_unlock - release a queued spinlock
|
||||
@@ -16,15 +24,9 @@
|
||||
*/
|
||||
static inline void native_queued_spin_unlock(struct qspinlock *lock)
|
||||
{
|
||||
smp_store_release((u8 *)lock, 0);
|
||||
smp_store_release(&lock->locked, 0);
|
||||
}
|
||||
|
||||
#ifdef CONFIG_PARAVIRT_SPINLOCKS
|
||||
extern void native_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
|
||||
extern void __pv_init_lock_hash(void);
|
||||
extern void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val);
|
||||
extern void __raw_callee_save___pv_queued_spin_unlock(struct qspinlock *lock);
|
||||
|
||||
static inline void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
|
||||
{
|
||||
pv_queued_spin_lock_slowpath(lock, val);
|
||||
@@ -40,11 +42,6 @@ static inline bool vcpu_is_preempted(long cpu)
|
||||
{
|
||||
return pv_vcpu_is_preempted(cpu);
|
||||
}
|
||||
#else
|
||||
static inline void queued_spin_unlock(struct qspinlock *lock)
|
||||
{
|
||||
native_queued_spin_unlock(lock);
|
||||
}
|
||||
#endif
|
||||
|
||||
#ifdef CONFIG_PARAVIRT
|
||||
|
||||
@@ -22,8 +22,7 @@ PV_CALLEE_SAVE_REGS_THUNK(__pv_queued_spin_unlock_slowpath);
|
||||
*
|
||||
* void __pv_queued_spin_unlock(struct qspinlock *lock)
|
||||
* {
|
||||
* struct __qspinlock *l = (void *)lock;
|
||||
* u8 lockval = cmpxchg(&l->locked, _Q_LOCKED_VAL, 0);
|
||||
* u8 lockval = cmpxchg(&lock->locked, _Q_LOCKED_VAL, 0);
|
||||
*
|
||||
* if (likely(lockval == _Q_LOCKED_VAL))
|
||||
* return;
|
||||
|
||||
@@ -25,6 +25,7 @@ typedef atomic64_t atomic_long_t;
|
||||
|
||||
#define ATOMIC_LONG_INIT(i) ATOMIC64_INIT(i)
|
||||
#define ATOMIC_LONG_PFX(x) atomic64 ## x
|
||||
#define ATOMIC_LONG_TYPE s64
|
||||
|
||||
#else
|
||||
|
||||
@@ -32,6 +33,7 @@ typedef atomic_t atomic_long_t;
|
||||
|
||||
#define ATOMIC_LONG_INIT(i) ATOMIC_INIT(i)
|
||||
#define ATOMIC_LONG_PFX(x) atomic ## x
|
||||
#define ATOMIC_LONG_TYPE int
|
||||
|
||||
#endif
|
||||
|
||||
@@ -90,6 +92,21 @@ ATOMIC_LONG_ADD_SUB_OP(sub, _release)
|
||||
#define atomic_long_cmpxchg(l, old, new) \
|
||||
(ATOMIC_LONG_PFX(_cmpxchg)((ATOMIC_LONG_PFX(_t) *)(l), (old), (new)))
|
||||
|
||||
|
||||
#define atomic_long_try_cmpxchg_relaxed(l, old, new) \
|
||||
(ATOMIC_LONG_PFX(_try_cmpxchg_relaxed)((ATOMIC_LONG_PFX(_t) *)(l), \
|
||||
(ATOMIC_LONG_TYPE *)(old), (ATOMIC_LONG_TYPE)(new)))
|
||||
#define atomic_long_try_cmpxchg_acquire(l, old, new) \
|
||||
(ATOMIC_LONG_PFX(_try_cmpxchg_acquire)((ATOMIC_LONG_PFX(_t) *)(l), \
|
||||
(ATOMIC_LONG_TYPE *)(old), (ATOMIC_LONG_TYPE)(new)))
|
||||
#define atomic_long_try_cmpxchg_release(l, old, new) \
|
||||
(ATOMIC_LONG_PFX(_try_cmpxchg_release)((ATOMIC_LONG_PFX(_t) *)(l), \
|
||||
(ATOMIC_LONG_TYPE *)(old), (ATOMIC_LONG_TYPE)(new)))
|
||||
#define atomic_long_try_cmpxchg(l, old, new) \
|
||||
(ATOMIC_LONG_PFX(_try_cmpxchg)((ATOMIC_LONG_PFX(_t) *)(l), \
|
||||
(ATOMIC_LONG_TYPE *)(old), (ATOMIC_LONG_TYPE)(new)))
|
||||
|
||||
|
||||
#define atomic_long_xchg_relaxed(v, new) \
|
||||
(ATOMIC_LONG_PFX(_xchg_relaxed)((ATOMIC_LONG_PFX(_t) *)(v), (new)))
|
||||
#define atomic_long_xchg_acquire(v, new) \
|
||||
@@ -244,6 +261,8 @@ static inline long atomic_long_add_unless(atomic_long_t *l, long a, long u)
|
||||
#define atomic_long_inc_not_zero(l) \
|
||||
ATOMIC_LONG_PFX(_inc_not_zero)((ATOMIC_LONG_PFX(_t) *)(l))
|
||||
|
||||
#define atomic_long_cond_read_relaxed(v, c) \
|
||||
ATOMIC_LONG_PFX(_cond_read_relaxed)((ATOMIC_LONG_PFX(_t) *)(v), (c))
|
||||
#define atomic_long_cond_read_acquire(v, c) \
|
||||
ATOMIC_LONG_PFX(_cond_read_acquire)((ATOMIC_LONG_PFX(_t) *)(v), (c))
|
||||
|
||||
|
||||
@@ -221,18 +221,17 @@ do { \
|
||||
#endif
|
||||
|
||||
/**
|
||||
* smp_cond_load_acquire() - (Spin) wait for cond with ACQUIRE ordering
|
||||
* smp_cond_load_relaxed() - (Spin) wait for cond with no ordering guarantees
|
||||
* @ptr: pointer to the variable to wait on
|
||||
* @cond: boolean expression to wait for
|
||||
*
|
||||
* Equivalent to using smp_load_acquire() on the condition variable but employs
|
||||
* the control dependency of the wait to reduce the barrier on many platforms.
|
||||
* Equivalent to using READ_ONCE() on the condition variable.
|
||||
*
|
||||
* Due to C lacking lambda expressions we load the value of *ptr into a
|
||||
* pre-named variable @VAL to be used in @cond.
|
||||
*/
|
||||
#ifndef smp_cond_load_acquire
|
||||
#define smp_cond_load_acquire(ptr, cond_expr) ({ \
|
||||
#ifndef smp_cond_load_relaxed
|
||||
#define smp_cond_load_relaxed(ptr, cond_expr) ({ \
|
||||
typeof(ptr) __PTR = (ptr); \
|
||||
typeof(*ptr) VAL; \
|
||||
for (;;) { \
|
||||
@@ -241,10 +240,26 @@ do { \
|
||||
break; \
|
||||
cpu_relax(); \
|
||||
} \
|
||||
smp_acquire__after_ctrl_dep(); \
|
||||
VAL; \
|
||||
})
|
||||
#endif
|
||||
|
||||
/**
|
||||
* smp_cond_load_acquire() - (Spin) wait for cond with ACQUIRE ordering
|
||||
* @ptr: pointer to the variable to wait on
|
||||
* @cond: boolean expression to wait for
|
||||
*
|
||||
* Equivalent to using smp_load_acquire() on the condition variable but employs
|
||||
* the control dependency of the wait to reduce the barrier on many platforms.
|
||||
*/
|
||||
#ifndef smp_cond_load_acquire
|
||||
#define smp_cond_load_acquire(ptr, cond_expr) ({ \
|
||||
typeof(*ptr) _val; \
|
||||
_val = smp_cond_load_relaxed(ptr, cond_expr); \
|
||||
smp_acquire__after_ctrl_dep(); \
|
||||
_val; \
|
||||
})
|
||||
#endif
|
||||
|
||||
#endif /* !__ASSEMBLY__ */
|
||||
#endif /* __ASM_GENERIC_BARRIER_H */
|
||||
|
||||
@@ -26,7 +26,6 @@
|
||||
* @lock: Pointer to queued spinlock structure
|
||||
* Return: 1 if it is locked, 0 otherwise
|
||||
*/
|
||||
#ifndef queued_spin_is_locked
|
||||
static __always_inline int queued_spin_is_locked(struct qspinlock *lock)
|
||||
{
|
||||
/*
|
||||
@@ -35,7 +34,6 @@ static __always_inline int queued_spin_is_locked(struct qspinlock *lock)
|
||||
*/
|
||||
return atomic_read(&lock->val);
|
||||
}
|
||||
#endif
|
||||
|
||||
/**
|
||||
* queued_spin_value_unlocked - is the spinlock structure unlocked?
|
||||
@@ -100,7 +98,7 @@ static __always_inline void queued_spin_unlock(struct qspinlock *lock)
|
||||
/*
|
||||
* unlock() needs release semantics:
|
||||
*/
|
||||
(void)atomic_sub_return_release(_Q_LOCKED_VAL, &lock->val);
|
||||
smp_store_release(&lock->locked, 0);
|
||||
}
|
||||
#endif
|
||||
|
||||
|
||||
@@ -29,13 +29,41 @@
|
||||
#endif
|
||||
|
||||
typedef struct qspinlock {
|
||||
atomic_t val;
|
||||
union {
|
||||
atomic_t val;
|
||||
|
||||
/*
|
||||
* By using the whole 2nd least significant byte for the
|
||||
* pending bit, we can allow better optimization of the lock
|
||||
* acquisition for the pending bit holder.
|
||||
*/
|
||||
#ifdef __LITTLE_ENDIAN
|
||||
struct {
|
||||
u8 locked;
|
||||
u8 pending;
|
||||
};
|
||||
struct {
|
||||
u16 locked_pending;
|
||||
u16 tail;
|
||||
};
|
||||
#else
|
||||
struct {
|
||||
u16 tail;
|
||||
u16 locked_pending;
|
||||
};
|
||||
struct {
|
||||
u8 reserved[2];
|
||||
u8 pending;
|
||||
u8 locked;
|
||||
};
|
||||
#endif
|
||||
};
|
||||
} arch_spinlock_t;
|
||||
|
||||
/*
|
||||
* Initializier
|
||||
*/
|
||||
#define __ARCH_SPIN_LOCK_UNLOCKED { ATOMIC_INIT(0) }
|
||||
#define __ARCH_SPIN_LOCK_UNLOCKED { .val = ATOMIC_INIT(0) }
|
||||
|
||||
/*
|
||||
* Bitfields in the atomic value:
|
||||
|
||||
@@ -654,6 +654,7 @@ static inline int atomic_dec_if_positive(atomic_t *v)
|
||||
}
|
||||
#endif
|
||||
|
||||
#define atomic_cond_read_relaxed(v, c) smp_cond_load_relaxed(&(v)->counter, (c))
|
||||
#define atomic_cond_read_acquire(v, c) smp_cond_load_acquire(&(v)->counter, (c))
|
||||
|
||||
#ifdef CONFIG_GENERIC_ATOMIC64
|
||||
@@ -1075,6 +1076,7 @@ static inline long long atomic64_fetch_andnot_release(long long i, atomic64_t *v
|
||||
}
|
||||
#endif
|
||||
|
||||
#define atomic64_cond_read_relaxed(v, c) smp_cond_load_relaxed(&(v)->counter, (c))
|
||||
#define atomic64_cond_read_acquire(v, c) smp_cond_load_acquire(&(v)->counter, (c))
|
||||
|
||||
#include <asm-generic/atomic-long.h>
|
||||
|
||||
@@ -29,7 +29,7 @@
|
||||
|
||||
#ifdef CONFIG_TASK_DELAY_ACCT
|
||||
struct task_delay_info {
|
||||
spinlock_t lock;
|
||||
raw_spinlock_t lock;
|
||||
unsigned int flags; /* Private per-task flags */
|
||||
|
||||
/* For each stat XXX, add following, aligned appropriately
|
||||
|
||||
@@ -146,9 +146,6 @@ extern void __mutex_init(struct mutex *lock, const char *name,
|
||||
*/
|
||||
static inline bool mutex_is_locked(struct mutex *lock)
|
||||
{
|
||||
/*
|
||||
* XXX think about spin_is_locked
|
||||
*/
|
||||
return __mutex_owner(lock) != NULL;
|
||||
}
|
||||
|
||||
|
||||
@@ -380,6 +380,24 @@ static __always_inline int spin_trylock_irq(spinlock_t *lock)
|
||||
raw_spin_trylock_irqsave(spinlock_check(lock), flags); \
|
||||
})
|
||||
|
||||
/**
|
||||
* spin_is_locked() - Check whether a spinlock is locked.
|
||||
* @lock: Pointer to the spinlock.
|
||||
*
|
||||
* This function is NOT required to provide any memory ordering
|
||||
* guarantees; it could be used for debugging purposes or, when
|
||||
* additional synchronization is needed, accompanied with other
|
||||
* constructs (memory barriers) enforcing the synchronization.
|
||||
*
|
||||
* Returns: 1 if @lock is locked, 0 otherwise.
|
||||
*
|
||||
* Note that the function only tells you that the spinlock is
|
||||
* seen to be locked, not that it is locked on your CPU.
|
||||
*
|
||||
* Further, on CONFIG_SMP=n builds with CONFIG_DEBUG_SPINLOCK=n,
|
||||
* the return value is always 0 (see include/linux/spinlock_up.h).
|
||||
* Therefore you should not rely heavily on the return value.
|
||||
*/
|
||||
static __always_inline int spin_is_locked(spinlock_t *lock)
|
||||
{
|
||||
return raw_spin_is_locked(&lock->rlock);
|
||||
|
||||
+9
-8
@@ -44,23 +44,24 @@ void __delayacct_tsk_init(struct task_struct *tsk)
|
||||
{
|
||||
tsk->delays = kmem_cache_zalloc(delayacct_cache, GFP_KERNEL);
|
||||
if (tsk->delays)
|
||||
spin_lock_init(&tsk->delays->lock);
|
||||
raw_spin_lock_init(&tsk->delays->lock);
|
||||
}
|
||||
|
||||
/*
|
||||
* Finish delay accounting for a statistic using its timestamps (@start),
|
||||
* accumalator (@total) and @count
|
||||
*/
|
||||
static void delayacct_end(spinlock_t *lock, u64 *start, u64 *total, u32 *count)
|
||||
static void delayacct_end(raw_spinlock_t *lock, u64 *start, u64 *total,
|
||||
u32 *count)
|
||||
{
|
||||
s64 ns = ktime_get_ns() - *start;
|
||||
unsigned long flags;
|
||||
|
||||
if (ns > 0) {
|
||||
spin_lock_irqsave(lock, flags);
|
||||
raw_spin_lock_irqsave(lock, flags);
|
||||
*total += ns;
|
||||
(*count)++;
|
||||
spin_unlock_irqrestore(lock, flags);
|
||||
raw_spin_unlock_irqrestore(lock, flags);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -127,7 +128,7 @@ int __delayacct_add_tsk(struct taskstats *d, struct task_struct *tsk)
|
||||
|
||||
/* zero XXX_total, non-zero XXX_count implies XXX stat overflowed */
|
||||
|
||||
spin_lock_irqsave(&tsk->delays->lock, flags);
|
||||
raw_spin_lock_irqsave(&tsk->delays->lock, flags);
|
||||
tmp = d->blkio_delay_total + tsk->delays->blkio_delay;
|
||||
d->blkio_delay_total = (tmp < d->blkio_delay_total) ? 0 : tmp;
|
||||
tmp = d->swapin_delay_total + tsk->delays->swapin_delay;
|
||||
@@ -137,7 +138,7 @@ int __delayacct_add_tsk(struct taskstats *d, struct task_struct *tsk)
|
||||
d->blkio_count += tsk->delays->blkio_count;
|
||||
d->swapin_count += tsk->delays->swapin_count;
|
||||
d->freepages_count += tsk->delays->freepages_count;
|
||||
spin_unlock_irqrestore(&tsk->delays->lock, flags);
|
||||
raw_spin_unlock_irqrestore(&tsk->delays->lock, flags);
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -147,10 +148,10 @@ __u64 __delayacct_blkio_ticks(struct task_struct *tsk)
|
||||
__u64 ret;
|
||||
unsigned long flags;
|
||||
|
||||
spin_lock_irqsave(&tsk->delays->lock, flags);
|
||||
raw_spin_lock_irqsave(&tsk->delays->lock, flags);
|
||||
ret = nsec_to_clock_t(tsk->delays->blkio_delay +
|
||||
tsk->delays->swapin_delay);
|
||||
spin_unlock_irqrestore(&tsk->delays->lock, flags);
|
||||
raw_spin_unlock_irqrestore(&tsk->delays->lock, flags);
|
||||
return ret;
|
||||
}
|
||||
|
||||
|
||||
+20
-50
@@ -561,20 +561,24 @@ static void print_lock(struct held_lock *hlock)
|
||||
printk(KERN_CONT ", at: %pS\n", (void *)hlock->acquire_ip);
|
||||
}
|
||||
|
||||
static void lockdep_print_held_locks(struct task_struct *curr)
|
||||
static void lockdep_print_held_locks(struct task_struct *p)
|
||||
{
|
||||
int i, depth = curr->lockdep_depth;
|
||||
int i, depth = READ_ONCE(p->lockdep_depth);
|
||||
|
||||
if (!depth) {
|
||||
printk("no locks held by %s/%d.\n", curr->comm, task_pid_nr(curr));
|
||||
if (!depth)
|
||||
printk("no locks held by %s/%d.\n", p->comm, task_pid_nr(p));
|
||||
else
|
||||
printk("%d lock%s held by %s/%d:\n", depth,
|
||||
depth > 1 ? "s" : "", p->comm, task_pid_nr(p));
|
||||
/*
|
||||
* It's not reliable to print a task's held locks if it's not sleeping
|
||||
* and it's not the current task.
|
||||
*/
|
||||
if (p->state == TASK_RUNNING && p != current)
|
||||
return;
|
||||
}
|
||||
printk("%d lock%s held by %s/%d:\n",
|
||||
depth, depth > 1 ? "s" : "", curr->comm, task_pid_nr(curr));
|
||||
|
||||
for (i = 0; i < depth; i++) {
|
||||
printk(" #%d: ", i);
|
||||
print_lock(curr->held_locks + i);
|
||||
print_lock(p->held_locks + i);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4451,8 +4455,6 @@ EXPORT_SYMBOL_GPL(debug_check_no_locks_held);
|
||||
void debug_show_all_locks(void)
|
||||
{
|
||||
struct task_struct *g, *p;
|
||||
int count = 10;
|
||||
int unlock = 1;
|
||||
|
||||
if (unlikely(!debug_locks)) {
|
||||
pr_warn("INFO: lockdep is turned off.\n");
|
||||
@@ -4460,50 +4462,18 @@ void debug_show_all_locks(void)
|
||||
}
|
||||
pr_warn("\nShowing all locks held in the system:\n");
|
||||
|
||||
/*
|
||||
* Here we try to get the tasklist_lock as hard as possible,
|
||||
* if not successful after 2 seconds we ignore it (but keep
|
||||
* trying). This is to enable a debug printout even if a
|
||||
* tasklist_lock-holding task deadlocks or crashes.
|
||||
*/
|
||||
retry:
|
||||
if (!read_trylock(&tasklist_lock)) {
|
||||
if (count == 10)
|
||||
pr_warn("hm, tasklist_lock locked, retrying... ");
|
||||
if (count) {
|
||||
count--;
|
||||
pr_cont(" #%d", 10-count);
|
||||
mdelay(200);
|
||||
goto retry;
|
||||
}
|
||||
pr_cont(" ignoring it.\n");
|
||||
unlock = 0;
|
||||
} else {
|
||||
if (count != 10)
|
||||
pr_cont(" locked it.\n");
|
||||
}
|
||||
|
||||
do_each_thread(g, p) {
|
||||
/*
|
||||
* It's not reliable to print a task's held locks
|
||||
* if it's not sleeping (or if it's not the current
|
||||
* task):
|
||||
*/
|
||||
if (p->state == TASK_RUNNING && p != current)
|
||||
rcu_read_lock();
|
||||
for_each_process_thread(g, p) {
|
||||
if (!p->lockdep_depth)
|
||||
continue;
|
||||
if (p->lockdep_depth)
|
||||
lockdep_print_held_locks(p);
|
||||
if (!unlock)
|
||||
if (read_trylock(&tasklist_lock))
|
||||
unlock = 1;
|
||||
lockdep_print_held_locks(p);
|
||||
touch_nmi_watchdog();
|
||||
} while_each_thread(g, p);
|
||||
touch_all_softlockup_watchdogs();
|
||||
}
|
||||
rcu_read_unlock();
|
||||
|
||||
pr_warn("\n");
|
||||
pr_warn("=============================================\n\n");
|
||||
|
||||
if (unlock)
|
||||
read_unlock(&tasklist_lock);
|
||||
}
|
||||
EXPORT_SYMBOL_GPL(debug_show_all_locks);
|
||||
#endif
|
||||
|
||||
@@ -23,13 +23,15 @@ struct mcs_spinlock {
|
||||
|
||||
#ifndef arch_mcs_spin_lock_contended
|
||||
/*
|
||||
* Using smp_load_acquire() provides a memory barrier that ensures
|
||||
* subsequent operations happen after the lock is acquired.
|
||||
* Using smp_cond_load_acquire() provides the acquire semantics
|
||||
* required so that subsequent operations happen after the
|
||||
* lock is acquired. Additionally, some architectures such as
|
||||
* ARM64 would like to do spin-waiting instead of purely
|
||||
* spinning, and smp_cond_load_acquire() provides that behavior.
|
||||
*/
|
||||
#define arch_mcs_spin_lock_contended(l) \
|
||||
do { \
|
||||
while (!(smp_load_acquire(l))) \
|
||||
cpu_relax(); \
|
||||
smp_cond_load_acquire(l, VAL); \
|
||||
} while (0)
|
||||
#endif
|
||||
|
||||
|
||||
@@ -139,8 +139,9 @@ static inline bool __mutex_trylock(struct mutex *lock)
|
||||
static __always_inline bool __mutex_trylock_fast(struct mutex *lock)
|
||||
{
|
||||
unsigned long curr = (unsigned long)current;
|
||||
unsigned long zero = 0UL;
|
||||
|
||||
if (!atomic_long_cmpxchg_acquire(&lock->owner, 0UL, curr))
|
||||
if (atomic_long_try_cmpxchg_acquire(&lock->owner, &zero, curr))
|
||||
return true;
|
||||
|
||||
return false;
|
||||
|
||||
+121
-130
@@ -12,11 +12,11 @@
|
||||
* GNU General Public License for more details.
|
||||
*
|
||||
* (C) Copyright 2013-2015 Hewlett-Packard Development Company, L.P.
|
||||
* (C) Copyright 2013-2014 Red Hat, Inc.
|
||||
* (C) Copyright 2013-2014,2018 Red Hat, Inc.
|
||||
* (C) Copyright 2015 Intel Corp.
|
||||
* (C) Copyright 2015 Hewlett-Packard Enterprise Development LP
|
||||
*
|
||||
* Authors: Waiman Long <waiman.long@hpe.com>
|
||||
* Authors: Waiman Long <longman@redhat.com>
|
||||
* Peter Zijlstra <peterz@infradead.org>
|
||||
*/
|
||||
|
||||
@@ -32,6 +32,11 @@
|
||||
#include <asm/byteorder.h>
|
||||
#include <asm/qspinlock.h>
|
||||
|
||||
/*
|
||||
* Include queued spinlock statistics code
|
||||
*/
|
||||
#include "qspinlock_stat.h"
|
||||
|
||||
/*
|
||||
* The basic principle of a queue-based spinlock can best be understood
|
||||
* by studying a classic queue-based spinlock implementation called the
|
||||
@@ -76,6 +81,18 @@
|
||||
#define MAX_NODES 4
|
||||
#endif
|
||||
|
||||
/*
|
||||
* The pending bit spinning loop count.
|
||||
* This heuristic is used to limit the number of lockword accesses
|
||||
* made by atomic_cond_read_relaxed when waiting for the lock to
|
||||
* transition out of the "== _Q_PENDING_VAL" state. We don't spin
|
||||
* indefinitely because there's no guarantee that we'll make forward
|
||||
* progress.
|
||||
*/
|
||||
#ifndef _Q_PENDING_LOOPS
|
||||
#define _Q_PENDING_LOOPS 1
|
||||
#endif
|
||||
|
||||
/*
|
||||
* Per-CPU queue node structures; we can never have more than 4 nested
|
||||
* contexts: task, softirq, hardirq, nmi.
|
||||
@@ -114,41 +131,18 @@ static inline __pure struct mcs_spinlock *decode_tail(u32 tail)
|
||||
|
||||
#define _Q_LOCKED_PENDING_MASK (_Q_LOCKED_MASK | _Q_PENDING_MASK)
|
||||
|
||||
/*
|
||||
* By using the whole 2nd least significant byte for the pending bit, we
|
||||
* can allow better optimization of the lock acquisition for the pending
|
||||
* bit holder.
|
||||
*
|
||||
* This internal structure is also used by the set_locked function which
|
||||
* is not restricted to _Q_PENDING_BITS == 8.
|
||||
*/
|
||||
struct __qspinlock {
|
||||
union {
|
||||
atomic_t val;
|
||||
#ifdef __LITTLE_ENDIAN
|
||||
struct {
|
||||
u8 locked;
|
||||
u8 pending;
|
||||
};
|
||||
struct {
|
||||
u16 locked_pending;
|
||||
u16 tail;
|
||||
};
|
||||
#else
|
||||
struct {
|
||||
u16 tail;
|
||||
u16 locked_pending;
|
||||
};
|
||||
struct {
|
||||
u8 reserved[2];
|
||||
u8 pending;
|
||||
u8 locked;
|
||||
};
|
||||
#endif
|
||||
};
|
||||
};
|
||||
|
||||
#if _Q_PENDING_BITS == 8
|
||||
/**
|
||||
* clear_pending - clear the pending bit.
|
||||
* @lock: Pointer to queued spinlock structure
|
||||
*
|
||||
* *,1,* -> *,0,*
|
||||
*/
|
||||
static __always_inline void clear_pending(struct qspinlock *lock)
|
||||
{
|
||||
WRITE_ONCE(lock->pending, 0);
|
||||
}
|
||||
|
||||
/**
|
||||
* clear_pending_set_locked - take ownership and clear the pending bit.
|
||||
* @lock: Pointer to queued spinlock structure
|
||||
@@ -159,9 +153,7 @@ struct __qspinlock {
|
||||
*/
|
||||
static __always_inline void clear_pending_set_locked(struct qspinlock *lock)
|
||||
{
|
||||
struct __qspinlock *l = (void *)lock;
|
||||
|
||||
WRITE_ONCE(l->locked_pending, _Q_LOCKED_VAL);
|
||||
WRITE_ONCE(lock->locked_pending, _Q_LOCKED_VAL);
|
||||
}
|
||||
|
||||
/*
|
||||
@@ -176,18 +168,27 @@ static __always_inline void clear_pending_set_locked(struct qspinlock *lock)
|
||||
*/
|
||||
static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail)
|
||||
{
|
||||
struct __qspinlock *l = (void *)lock;
|
||||
|
||||
/*
|
||||
* Use release semantics to make sure that the MCS node is properly
|
||||
* initialized before changing the tail code.
|
||||
* We can use relaxed semantics since the caller ensures that the
|
||||
* MCS node is properly initialized before updating the tail.
|
||||
*/
|
||||
return (u32)xchg_release(&l->tail,
|
||||
return (u32)xchg_relaxed(&lock->tail,
|
||||
tail >> _Q_TAIL_OFFSET) << _Q_TAIL_OFFSET;
|
||||
}
|
||||
|
||||
#else /* _Q_PENDING_BITS == 8 */
|
||||
|
||||
/**
|
||||
* clear_pending - clear the pending bit.
|
||||
* @lock: Pointer to queued spinlock structure
|
||||
*
|
||||
* *,1,* -> *,0,*
|
||||
*/
|
||||
static __always_inline void clear_pending(struct qspinlock *lock)
|
||||
{
|
||||
atomic_andnot(_Q_PENDING_VAL, &lock->val);
|
||||
}
|
||||
|
||||
/**
|
||||
* clear_pending_set_locked - take ownership and clear the pending bit.
|
||||
* @lock: Pointer to queued spinlock structure
|
||||
@@ -216,10 +217,11 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail)
|
||||
for (;;) {
|
||||
new = (val & _Q_LOCKED_PENDING_MASK) | tail;
|
||||
/*
|
||||
* Use release semantics to make sure that the MCS node is
|
||||
* properly initialized before changing the tail code.
|
||||
* We can use relaxed semantics since the caller ensures that
|
||||
* the MCS node is properly initialized before updating the
|
||||
* tail.
|
||||
*/
|
||||
old = atomic_cmpxchg_release(&lock->val, val, new);
|
||||
old = atomic_cmpxchg_relaxed(&lock->val, val, new);
|
||||
if (old == val)
|
||||
break;
|
||||
|
||||
@@ -237,9 +239,7 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail)
|
||||
*/
|
||||
static __always_inline void set_locked(struct qspinlock *lock)
|
||||
{
|
||||
struct __qspinlock *l = (void *)lock;
|
||||
|
||||
WRITE_ONCE(l->locked, _Q_LOCKED_VAL);
|
||||
WRITE_ONCE(lock->locked, _Q_LOCKED_VAL);
|
||||
}
|
||||
|
||||
|
||||
@@ -294,86 +294,83 @@ static __always_inline u32 __pv_wait_head_or_lock(struct qspinlock *lock,
|
||||
void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
|
||||
{
|
||||
struct mcs_spinlock *prev, *next, *node;
|
||||
u32 new, old, tail;
|
||||
u32 old, tail;
|
||||
int idx;
|
||||
|
||||
BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS));
|
||||
|
||||
if (pv_enabled())
|
||||
goto queue;
|
||||
goto pv_queue;
|
||||
|
||||
if (virt_spin_lock(lock))
|
||||
return;
|
||||
|
||||
/*
|
||||
* wait for in-progress pending->locked hand-overs
|
||||
* Wait for in-progress pending->locked hand-overs with a bounded
|
||||
* number of spins so that we guarantee forward progress.
|
||||
*
|
||||
* 0,1,0 -> 0,0,1
|
||||
*/
|
||||
if (val == _Q_PENDING_VAL) {
|
||||
while ((val = atomic_read(&lock->val)) == _Q_PENDING_VAL)
|
||||
cpu_relax();
|
||||
int cnt = _Q_PENDING_LOOPS;
|
||||
val = atomic_cond_read_relaxed(&lock->val,
|
||||
(VAL != _Q_PENDING_VAL) || !cnt--);
|
||||
}
|
||||
|
||||
/*
|
||||
* If we observe any contention; queue.
|
||||
*/
|
||||
if (val & ~_Q_LOCKED_MASK)
|
||||
goto queue;
|
||||
|
||||
/*
|
||||
* trylock || pending
|
||||
*
|
||||
* 0,0,0 -> 0,0,1 ; trylock
|
||||
* 0,0,1 -> 0,1,1 ; pending
|
||||
*/
|
||||
for (;;) {
|
||||
val = atomic_fetch_or_acquire(_Q_PENDING_VAL, &lock->val);
|
||||
if (!(val & ~_Q_LOCKED_MASK)) {
|
||||
/*
|
||||
* If we observe any contention; queue.
|
||||
* We're pending, wait for the owner to go away.
|
||||
*
|
||||
* *,1,1 -> *,1,0
|
||||
*
|
||||
* this wait loop must be a load-acquire such that we match the
|
||||
* store-release that clears the locked bit and create lock
|
||||
* sequentiality; this is because not all
|
||||
* clear_pending_set_locked() implementations imply full
|
||||
* barriers.
|
||||
*/
|
||||
if (val & ~_Q_LOCKED_MASK)
|
||||
goto queue;
|
||||
|
||||
new = _Q_LOCKED_VAL;
|
||||
if (val == new)
|
||||
new |= _Q_PENDING_VAL;
|
||||
if (val & _Q_LOCKED_MASK) {
|
||||
atomic_cond_read_acquire(&lock->val,
|
||||
!(VAL & _Q_LOCKED_MASK));
|
||||
}
|
||||
|
||||
/*
|
||||
* Acquire semantic is required here as the function may
|
||||
* return immediately if the lock was free.
|
||||
* take ownership and clear the pending bit.
|
||||
*
|
||||
* *,1,0 -> *,0,1
|
||||
*/
|
||||
old = atomic_cmpxchg_acquire(&lock->val, val, new);
|
||||
if (old == val)
|
||||
break;
|
||||
|
||||
val = old;
|
||||
clear_pending_set_locked(lock);
|
||||
qstat_inc(qstat_lock_pending, true);
|
||||
return;
|
||||
}
|
||||
|
||||
/*
|
||||
* we won the trylock
|
||||
* If pending was clear but there are waiters in the queue, then
|
||||
* we need to undo our setting of pending before we queue ourselves.
|
||||
*/
|
||||
if (new == _Q_LOCKED_VAL)
|
||||
return;
|
||||
|
||||
/*
|
||||
* we're pending, wait for the owner to go away.
|
||||
*
|
||||
* *,1,1 -> *,1,0
|
||||
*
|
||||
* this wait loop must be a load-acquire such that we match the
|
||||
* store-release that clears the locked bit and create lock
|
||||
* sequentiality; this is because not all clear_pending_set_locked()
|
||||
* implementations imply full barriers.
|
||||
*/
|
||||
smp_cond_load_acquire(&lock->val.counter, !(VAL & _Q_LOCKED_MASK));
|
||||
|
||||
/*
|
||||
* take ownership and clear the pending bit.
|
||||
*
|
||||
* *,1,0 -> *,0,1
|
||||
*/
|
||||
clear_pending_set_locked(lock);
|
||||
return;
|
||||
if (!(val & _Q_PENDING_MASK))
|
||||
clear_pending(lock);
|
||||
|
||||
/*
|
||||
* End of pending bit optimistic spinning and beginning of MCS
|
||||
* queuing.
|
||||
*/
|
||||
queue:
|
||||
qstat_inc(qstat_lock_slowpath, true);
|
||||
pv_queue:
|
||||
node = this_cpu_ptr(&mcs_nodes[0]);
|
||||
idx = node->count++;
|
||||
tail = encode_tail(smp_processor_id(), idx);
|
||||
@@ -400,12 +397,18 @@ queue:
|
||||
goto release;
|
||||
|
||||
/*
|
||||
* Ensure that the initialisation of @node is complete before we
|
||||
* publish the updated tail via xchg_tail() and potentially link
|
||||
* @node into the waitqueue via WRITE_ONCE(prev->next, node) below.
|
||||
*/
|
||||
smp_wmb();
|
||||
|
||||
/*
|
||||
* Publish the updated tail.
|
||||
* We have already touched the queueing cacheline; don't bother with
|
||||
* pending stuff.
|
||||
*
|
||||
* p,*,* -> n,*,*
|
||||
*
|
||||
* RELEASE, such that the stores to @node must be complete.
|
||||
*/
|
||||
old = xchg_tail(lock, tail);
|
||||
next = NULL;
|
||||
@@ -417,14 +420,8 @@ queue:
|
||||
if (old & _Q_TAIL_MASK) {
|
||||
prev = decode_tail(old);
|
||||
|
||||
/*
|
||||
* We must ensure that the stores to @node are observed before
|
||||
* the write to prev->next. The address dependency from
|
||||
* xchg_tail is not sufficient to ensure this because the read
|
||||
* component of xchg_tail is unordered with respect to the
|
||||
* initialisation of @node.
|
||||
*/
|
||||
smp_store_release(&prev->next, node);
|
||||
/* Link @node into the waitqueue. */
|
||||
WRITE_ONCE(prev->next, node);
|
||||
|
||||
pv_wait_node(node, prev);
|
||||
arch_mcs_spin_lock_contended(&node->locked);
|
||||
@@ -453,8 +450,8 @@ queue:
|
||||
*
|
||||
* The PV pv_wait_head_or_lock function, if active, will acquire
|
||||
* the lock and return a non-zero value. So we have to skip the
|
||||
* smp_cond_load_acquire() call. As the next PV queue head hasn't been
|
||||
* designated yet, there is no way for the locked value to become
|
||||
* atomic_cond_read_acquire() call. As the next PV queue head hasn't
|
||||
* been designated yet, there is no way for the locked value to become
|
||||
* _Q_SLOW_VAL. So both the set_locked() and the
|
||||
* atomic_cmpxchg_relaxed() calls will be safe.
|
||||
*
|
||||
@@ -464,44 +461,38 @@ queue:
|
||||
if ((val = pv_wait_head_or_lock(lock, node)))
|
||||
goto locked;
|
||||
|
||||
val = smp_cond_load_acquire(&lock->val.counter, !(VAL & _Q_LOCKED_PENDING_MASK));
|
||||
val = atomic_cond_read_acquire(&lock->val, !(VAL & _Q_LOCKED_PENDING_MASK));
|
||||
|
||||
locked:
|
||||
/*
|
||||
* claim the lock:
|
||||
*
|
||||
* n,0,0 -> 0,0,1 : lock, uncontended
|
||||
* *,0,0 -> *,0,1 : lock, contended
|
||||
* *,*,0 -> *,*,1 : lock, contended
|
||||
*
|
||||
* If the queue head is the only one in the queue (lock value == tail),
|
||||
* clear the tail code and grab the lock. Otherwise, we only need
|
||||
* to grab the lock.
|
||||
* If the queue head is the only one in the queue (lock value == tail)
|
||||
* and nobody is pending, clear the tail code and grab the lock.
|
||||
* Otherwise, we only need to grab the lock.
|
||||
*/
|
||||
for (;;) {
|
||||
/* In the PV case we might already have _Q_LOCKED_VAL set */
|
||||
if ((val & _Q_TAIL_MASK) != tail) {
|
||||
set_locked(lock);
|
||||
break;
|
||||
}
|
||||
/*
|
||||
* The smp_cond_load_acquire() call above has provided the
|
||||
* necessary acquire semantics required for locking. At most
|
||||
* two iterations of this loop may be ran.
|
||||
*/
|
||||
old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL);
|
||||
if (old == val)
|
||||
goto release; /* No contention */
|
||||
|
||||
val = old;
|
||||
}
|
||||
/*
|
||||
* In the PV case we might already have _Q_LOCKED_VAL set.
|
||||
*
|
||||
* The atomic_cond_read_acquire() call above has provided the
|
||||
* necessary acquire semantics required for locking.
|
||||
*/
|
||||
if (((val & _Q_TAIL_MASK) == tail) &&
|
||||
atomic_try_cmpxchg_relaxed(&lock->val, &val, _Q_LOCKED_VAL))
|
||||
goto release; /* No contention */
|
||||
|
||||
/* Either somebody is queued behind us or _Q_PENDING_VAL is set */
|
||||
set_locked(lock);
|
||||
|
||||
/*
|
||||
* contended path; wait for next if not observed yet, release.
|
||||
*/
|
||||
if (!next) {
|
||||
while (!(next = READ_ONCE(node->next)))
|
||||
cpu_relax();
|
||||
}
|
||||
if (!next)
|
||||
next = smp_cond_load_relaxed(&node->next, (VAL));
|
||||
|
||||
arch_mcs_spin_unlock_contended(&next->locked);
|
||||
pv_kick_node(lock, next);
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user