dasharo/linux
0
0
Fork 0
mirror of https://github.com/Dasharo/linux.git synced 2026-03-06 15:25:10 -08:00
Code Issues Packages Projects Releases Wiki Activity

Merge tag 'riscv-for-linus-6.13-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux

Browse Source 
Pull RISC-v updates from Palmer Dabbelt:

 - Support for pointer masking in userspace

 - Support for probing vector misaligned access performance

 - Support for qspinlock on systems with Zacas and Zabha

* tag 'riscv-for-linus-6.13-mw1' of git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux: (38 commits)
  RISC-V: Remove unnecessary include from compat.h
  riscv: Fix default misaligned access trap
  riscv: Add qspinlock support
  dt-bindings: riscv: Add Ziccrse ISA extension description
  riscv: Add ISA extension parsing for Ziccrse
  asm-generic: ticket-lock: Add separate ticket-lock.h
  asm-generic: ticket-lock: Reuse arch_spinlock_t of qspinlock
  riscv: Implement xchg8/16() using Zabha
  riscv: Implement arch_cmpxchg128() using Zacas
  riscv: Improve zacas fully-ordered cmpxchg()
  riscv: Implement cmpxchg8/16() using Zabha
  dt-bindings: riscv: Add Zabha ISA extension description
  riscv: Implement cmpxchg32/64() using Zacas
  riscv: Do not fail to build on byte/halfword operations with Zawrs
  riscv: Move cpufeature.h macros into their own header
  KVM: riscv: selftests: Add Smnpm and Ssnpm to get-reg-list test
  RISC-V: KVM: Allow Smnpm and Ssnpm extensions for guests
  riscv: hwprobe: Export the Supm ISA extension
  riscv: selftests: Add a pointer masking test
  riscv: Allow ptrace control of the tagged address ABI
  ...
This commit is contained in:
Linus Torvalds
2024-11-27 11:19:09 -08:00
parent c946969775 8d4f1e05ff
commit 91dbbe6c9f
53 changed files with 1790 additions and 283 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
Documentation/arch/riscv/hwprobe.rst
View File

@@ -239,6 +239,9 @@ The following keys are defined:
ratified in commit 98918c844281 ("Merge pull request #1217 from
riscv/zawrs") of riscv-isa-manual.
* :c:macro:`RISCV_HWPROBE_EXT_SUPM`: The Supm extension is supported as
defined in version 1.0 of the RISC-V Pointer Masking extensions.
* :c:macro:`RISCV_HWPROBE_KEY_CPUPERF_0`: Deprecated. Returns similar values to
:c:macro:`RISCV_HWPROBE_KEY_MISALIGNED_SCALAR_PERF`, but the key was
mistakenly classified as a bitmask rather than a value.
@@ -274,3 +277,19 @@ The following keys are defined:
represent the highest userspace virtual address usable.
* :c:macro:`RISCV_HWPROBE_KEY_TIME_CSR_FREQ`: Frequency (in Hz) of `time CSR`.
* :c:macro:`RISCV_HWPROBE_KEY_MISALIGNED_VECTOR_PERF`: An enum value describing the
performance of misaligned vector accesses on the selected set of processors.
* :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_UNKNOWN`: The performance of misaligned
vector accesses is unknown.
* :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_SLOW`: 32-bit misaligned accesses using vector
registers are slower than the equivalent quantity of byte accesses via vector registers.
Misaligned accesses may be supported directly in hardware, or trapped and emulated by software.
* :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_FAST`: 32-bit misaligned accesses using vector
registers are faster than the equivalent quantity of byte accesses via vector registers.
* :c:macro:`RISCV_HWPROBE_MISALIGNED_VECTOR_UNSUPPORTED`: Misaligned vector accesses are
not supported at all and will generate a misaligned address fault.

16
Documentation/arch/riscv/uabi.rst
View File

@@ -68,3 +68,19 @@ Misaligned accesses
Misaligned scalar accesses are supported in userspace, but they may perform
poorly. Misaligned vector accesses are only supported if the Zicclsm extension
is supported.
Pointer masking
---------------
Support for pointer masking in userspace (the Supm extension) is provided via
the ``PR_SET_TAGGED_ADDR_CTRL`` and ``PR_GET_TAGGED_ADDR_CTRL`` ``prctl()``
operations. Pointer masking is disabled by default. To enable it, userspace
must call ``PR_SET_TAGGED_ADDR_CTRL`` with the ``PR_PMLEN`` field set to the
number of mask/tag bits needed by the application. ``PR_PMLEN`` is interpreted
as a lower bound; if the kernel is unable to satisfy the request, the
``PR_SET_TAGGED_ADDR_CTRL`` operation will fail. The actual number of tag bits
is returned in ``PR_PMLEN`` by the ``PR_GET_TAGGED_ADDR_CTRL`` operation.
Additionally, when pointer masking is enabled (``PR_PMLEN`` is greater than 0),
a tagged address ABI is supported, with the same interface and behavior as
documented for AArch64 (Documentation/arch/arm64/tagged-address-abi.rst).

30
Documentation/devicetree/bindings/riscv/extensions.yaml
View File

@@ -128,6 +128,18 @@ properties:
changes to interrupts as frozen at commit ccbddab ("Merge pull
request #42 from riscv/jhauser-2023-RC4") of riscv-aia.
- const: smmpm
description: |
The standard Smmpm extension for M-mode pointer masking as
ratified at commit d70011dde6c2 ("Update to ratified state")
of riscv-j-extension.
- const: smnpm
description: |
The standard Smnpm extension for next-mode pointer masking as
ratified at commit d70011dde6c2 ("Update to ratified state")
of riscv-j-extension.
- const: smstateen
description: |
The standard Smstateen extension for controlling access to CSRs
@@ -147,6 +159,12 @@ properties:
and mode-based filtering as ratified at commit 01d1df0 ("Add ability
to manually trigger workflow. (#2)") of riscv-count-overflow.
- const: ssnpm
description: |
The standard Ssnpm extension for next-mode pointer masking as
ratified at commit d70011dde6c2 ("Update to ratified state")
of riscv-j-extension.
- const: sstc
description: |
The standard Sstc supervisor-level extension for time compare as
@@ -178,6 +196,12 @@ properties:
as ratified at commit 4a69197e5617 ("Update to ratified state") of
riscv-svvptc.
- const: zabha
description: |
The Zabha extension for Byte and Halfword Atomic Memory Operations
as ratified at commit 49f49c842ff9 ("Update to Rafified state") of
riscv-zabha.
- const: zacas
description: |
The Zacas extension for Atomic Compare-and-Swap (CAS) instructions
@@ -290,6 +314,12 @@ properties:
in commit 64074bc ("Update version numbers for Zfh/Zfinx") of
riscv-isa-manual.
- const: ziccrse
description:
The standard Ziccrse extension which provides forward progress
guarantee on LR/SC sequences, as ratified in commit b1d806605f87
("Updated to ratified state.") of the riscv profiles specification.
- const: zk
description:
The standard Zk Standard Scalar cryptography extension as ratified

2
Documentation/features/locking/queued-spinlocks/arch-support.txt
View File

@@ -20,7 +20,7 @@
| openrisc: | ok |
| parisc: | TODO |
| powerpc: | ok |
| riscv: | TODO |
| riscv: | ok |
| s390: | TODO |
| sh: | TODO |
| sparc: | ok |

138
arch/riscv/Kconfig
View File

@@ -83,6 +83,7 @@ config RISCV
select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP
select ARCH_WANTS_NO_INSTR
select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE
select ARCH_WEAK_RELEASE_ACQUIRE if ARCH_USE_QUEUED_SPINLOCKS
select BINFMT_FLAT_NO_DATA_START_OFFSET if !MMU
select BUILDTIME_TABLE_SORT if MMU
select CLINT_TIMER if RISCV_M_MODE
@@ -116,6 +117,7 @@ config RISCV
select GENERIC_VDSO_TIME_NS if HAVE_GENERIC_VDSO
select HARDIRQS_SW_RESEND
select HAS_IOPORT if MMU
select HAVE_ALIGNED_STRUCT_PAGE
select HAVE_ARCH_AUDITSYSCALL
select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP
select HAVE_ARCH_HUGE_VMAP if MMU && 64BIT
@@ -507,6 +509,39 @@ config NODES_SHIFT
Specify the maximum number of NUMA Nodes available on the target
system. Increases memory reserved to accommodate various tables.
choice
prompt "RISC-V spinlock type"
default RISCV_COMBO_SPINLOCKS
config RISCV_TICKET_SPINLOCKS
bool "Using ticket spinlock"
config RISCV_QUEUED_SPINLOCKS
bool "Using queued spinlock"
depends on SMP && MMU && NONPORTABLE
select ARCH_USE_QUEUED_SPINLOCKS
help
The queued spinlock implementation requires the forward progress
guarantee of cmpxchg()/xchg() atomic operations: CAS with Zabha or
LR/SC with Ziccrse provide such guarantee.
Select this if and only if Zabha or Ziccrse is available on your
platform, RISCV_QUEUED_SPINLOCKS must not be selected for platforms
without one of those extensions.
If unsure, select RISCV_COMBO_SPINLOCKS, which will use qspinlocks
when supported and otherwise ticket spinlocks.
config RISCV_COMBO_SPINLOCKS
bool "Using combo spinlock"
depends on SMP && MMU
select ARCH_USE_QUEUED_SPINLOCKS
help
Embed both queued spinlock and ticket lock so that the spinlock
implementation can be chosen at runtime.
endchoice
config RISCV_ALTERNATIVE
bool
depends on !XIP_KERNEL
@@ -532,6 +567,17 @@ config RISCV_ISA_C
If you don't know what to do here, say Y.
config RISCV_ISA_SUPM
bool "Supm extension for userspace pointer masking"
depends on 64BIT
default y
help
Add support for pointer masking in userspace (Supm) when the
underlying hardware extension (Smnpm or Ssnpm) is detected at boot.
If this option is disabled, userspace will be unable to use
the prctl(PR_{SET,GET}_TAGGED_ADDR_CTRL) API.
config RISCV_ISA_SVNAPOT
bool "Svnapot extension support for supervisor mode NAPOT pages"
depends on 64BIT && MMU
@@ -633,6 +679,40 @@ config RISCV_ISA_ZAWRS
use of these instructions in the kernel when the Zawrs extension is
detected at boot.
config TOOLCHAIN_HAS_ZABHA
bool
default y
depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zabha)
depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zabha)
depends on AS_HAS_OPTION_ARCH
config RISCV_ISA_ZABHA
bool "Zabha extension support for atomic byte/halfword operations"
depends on TOOLCHAIN_HAS_ZABHA
depends on RISCV_ALTERNATIVE
default y
help
Enable the use of the Zabha ISA-extension to implement kernel
byte/halfword atomic memory operations when it is detected at boot.
If you don't know what to do here, say Y.
config TOOLCHAIN_HAS_ZACAS
bool
default y
depends on !64BIT || $(cc-option,-mabi=lp64 -march=rv64ima_zacas)
depends on !32BIT || $(cc-option,-mabi=ilp32 -march=rv32ima_zacas)
depends on AS_HAS_OPTION_ARCH
config RISCV_ISA_ZACAS
bool "Zacas extension support for atomic CAS"
depends on TOOLCHAIN_HAS_ZACAS
depends on RISCV_ALTERNATIVE
default y
help
Enable the use of the Zacas ISA-extension to implement kernel atomic
cmpxchg operations when it is detected at boot.
If you don't know what to do here, say Y.
config TOOLCHAIN_HAS_ZBB
@@ -786,10 +866,24 @@ config THREAD_SIZE_ORDER
config RISCV_MISALIGNED
bool
help
Embed support for detecting and emulating misaligned
scalar or vector loads and stores.
config RISCV_SCALAR_MISALIGNED
bool
select RISCV_MISALIGNED
select SYSCTL_ARCH_UNALIGN_ALLOW
help
Embed support for emulating misaligned loads and stores.
config RISCV_VECTOR_MISALIGNED
bool
select RISCV_MISALIGNED
depends on RISCV_ISA_V
help
Enable detecting support for vector misaligned loads and stores.
choice
prompt "Unaligned Accesses Support"
default RISCV_PROBE_UNALIGNED_ACCESS
@@ -801,7 +895,7 @@ choice
config RISCV_PROBE_UNALIGNED_ACCESS
bool "Probe for hardware unaligned access support"
select RISCV_MISALIGNED
select RISCV_SCALAR_MISALIGNED
help
During boot, the kernel will run a series of tests to determine the
speed of unaligned accesses. This probing will dynamically determine
@@ -812,7 +906,7 @@ config RISCV_PROBE_UNALIGNED_ACCESS
config RISCV_EMULATED_UNALIGNED_ACCESS
bool "Emulate unaligned access where system support is missing"
select RISCV_MISALIGNED
select RISCV_SCALAR_MISALIGNED
help
If unaligned memory accesses trap into the kernel as they are not
supported by the system, the kernel will emulate the unaligned
@@ -841,6 +935,46 @@ config RISCV_EFFICIENT_UNALIGNED_ACCESS
endchoice
choice
prompt "Vector unaligned Accesses Support"
depends on RISCV_ISA_V
default RISCV_PROBE_VECTOR_UNALIGNED_ACCESS
help
This determines the level of support for vector unaligned accesses. This
information is used by the kernel to perform optimizations. It is also
exposed to user space via the hwprobe syscall. The hardware will be
probed at boot by default.
config RISCV_PROBE_VECTOR_UNALIGNED_ACCESS
bool "Probe speed of vector unaligned accesses"
select RISCV_VECTOR_MISALIGNED
depends on RISCV_ISA_V
help
During boot, the kernel will run a series of tests to determine the
speed of vector unaligned accesses if they are supported. This probing
will dynamically determine the speed of vector unaligned accesses on
the underlying system if they are supported.
config RISCV_SLOW_VECTOR_UNALIGNED_ACCESS
bool "Assume the system supports slow vector unaligned memory accesses"
depends on NONPORTABLE
help
Assume that the system supports slow vector unaligned memory accesses. The
kernel and userspace programs may not be able to run at all on systems
that do not support unaligned memory accesses.
config RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS
bool "Assume the system supports fast vector unaligned memory accesses"
depends on NONPORTABLE
help
Assume that the system supports fast vector unaligned memory accesses. When
enabled, this option improves the performance of the kernel on such
systems. However, the kernel and userspace programs will run much more
slowly, or will not be able to run at all, on systems that do not
support efficient unaligned memory accesses.
endchoice
source "arch/riscv/Kconfig.vendor"
endmenu # "Platform type"

6
arch/riscv/Makefile
View File

@@ -82,6 +82,12 @@ else
riscv-march-$(CONFIG_TOOLCHAIN_NEEDS_EXPLICIT_ZICSR_ZIFENCEI) := $(riscv-march-y)_zicsr_zifencei
endif
# Check if the toolchain supports Zacas
riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZACAS) := $(riscv-march-y)_zacas
# Check if the toolchain supports Zabha
riscv-march-$(CONFIG_TOOLCHAIN_HAS_ZABHA) := $(riscv-march-y)_zabha
# Remove F,D,V from isa string for all. Keep extensions between "fd" and "v" by
# matching non-v and non-multi-letter extensions out with the filter ([^v_]*)
KBUILD_CFLAGS += -march=$(shell echo $(riscv-march-y) | sed -E 's/(rv32ima|rv64ima)fd([^v_]*)v?/\1\2/')

1
arch/riscv/configs/defconfig
View File

@@ -167,6 +167,7 @@ CONFIG_PINCTRL_SOPHGO_CV1800B=y
CONFIG_PINCTRL_SOPHGO_CV1812H=y
CONFIG_PINCTRL_SOPHGO_SG2000=y
CONFIG_PINCTRL_SOPHGO_SG2002=y
CONFIG_GPIO_DWAPB=y
CONFIG_GPIO_SIFIVE=y
CONFIG_POWER_RESET_GPIO_RESTART=y
CONFIG_SENSORS_SFCTEMP=m

4
arch/riscv/include/asm/Kbuild
View File

@@ -6,10 +6,12 @@ generic-y += early_ioremap.h
generic-y += flat.h
generic-y += kvm_para.h
generic-y += mmzone.h
generic-y += mcs_spinlock.h
generic-y += parport.h
generic-y += spinlock.h
generic-y += spinlock_types.h
generic-y += ticket_spinlock.h
generic-y += qrwlock.h
generic-y += qrwlock_types.h
generic-y += qspinlock.h
generic-y += user.h
generic-y += vmlinux.lds.h

284
arch/riscv/include/asm/cmpxchg.h
View File

@@ -12,30 +12,43 @@
#include <asm/fence.h>
#include <asm/hwcap.h>
#include <asm/insn-def.h>
#include <asm/cpufeature-macros.h>
#define __arch_xchg_masked(sc_sfx, prepend, append, r, p, n) \
({ \
u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \
ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \
ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \
<< __s; \
ulong __newx = (ulong)(n) << __s; \
ulong __retx; \
ulong __rc; \
\
__asm__ __volatile__ ( \
prepend \
"0: lr.w %0, %2\n" \
" and %1, %0, %z4\n" \
" or %1, %1, %z3\n" \
" sc.w" sc_sfx " %1, %1, %2\n" \
" bnez %1, 0b\n" \
append \
: "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \
: "rJ" (__newx), "rJ" (~__mask) \
: "memory"); \
\
r = (__typeof__(*(p)))((__retx & __mask) >> __s); \
#define __arch_xchg_masked(sc_sfx, swap_sfx, prepend, sc_append, \
swap_append, r, p, n) \
({ \
if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \
riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA)) { \
__asm__ __volatile__ ( \
prepend \
" amoswap" swap_sfx " %0, %z2, %1\n" \
swap_append \
: "=&r" (r), "+A" (*(p)) \
: "rJ" (n) \
: "memory"); \
} else { \
u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \
ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \
ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \
<< __s; \
ulong __newx = (ulong)(n) << __s; \
ulong __retx; \
ulong __rc; \
\
__asm__ __volatile__ ( \
prepend \
"0: lr.w %0, %2\n" \
" and %1, %0, %z4\n" \
" or %1, %1, %z3\n" \
" sc.w" sc_sfx " %1, %1, %2\n" \
" bnez %1, 0b\n" \
sc_append \
: "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \
: "rJ" (__newx), "rJ" (~__mask) \
: "memory"); \
\
r = (__typeof__(*(p)))((__retx & __mask) >> __s); \
} \
})
#define __arch_xchg(sfx, prepend, append, r, p, n) \
@@ -58,8 +71,13 @@
\
switch (sizeof(*__ptr)) { \
case 1: \
__arch_xchg_masked(sc_sfx, ".b" swap_sfx, \
prepend, sc_append, swap_append, \
__ret, __ptr, __new); \
break; \
case 2: \
__arch_xchg_masked(sc_sfx, prepend, sc_append, \
__arch_xchg_masked(sc_sfx, ".h" swap_sfx, \
prepend, sc_append, swap_append, \
__ret, __ptr, __new); \
break; \
case 4: \
@@ -106,55 +124,90 @@
* store NEW in MEM. Return the initial value in MEM. Success is
* indicated by comparing RETURN with OLD.
*/
#define __arch_cmpxchg_masked(sc_sfx, prepend, append, r, p, o, n) \
({ \
u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \
ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \
ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \
<< __s; \
ulong __newx = (ulong)(n) << __s; \
ulong __oldx = (ulong)(o) << __s; \
ulong __retx; \
ulong __rc; \
\
__asm__ __volatile__ ( \
prepend \
"0: lr.w %0, %2\n" \
" and %1, %0, %z5\n" \
" bne %1, %z3, 1f\n" \
" and %1, %0, %z6\n" \
" or %1, %1, %z4\n" \
" sc.w" sc_sfx " %1, %1, %2\n" \
" bnez %1, 0b\n" \
append \
"1:\n" \
: "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \
: "rJ" ((long)__oldx), "rJ" (__newx), \
"rJ" (__mask), "rJ" (~__mask) \
: "memory"); \
\
r = (__typeof__(*(p)))((__retx & __mask) >> __s); \
#define __arch_cmpxchg_masked(sc_sfx, cas_sfx, \
sc_prepend, sc_append, \
cas_prepend, cas_append, \
r, p, o, n) \
({ \
if (IS_ENABLED(CONFIG_RISCV_ISA_ZABHA) && \
IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \
riscv_has_extension_unlikely(RISCV_ISA_EXT_ZABHA) && \
riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \
r = o; \
\
__asm__ __volatile__ ( \
cas_prepend \
" amocas" cas_sfx " %0, %z2, %1\n" \
cas_append \
: "+&r" (r), "+A" (*(p)) \
: "rJ" (n) \
: "memory"); \
} else { \
u32 *__ptr32b = (u32 *)((ulong)(p) & ~0x3); \
ulong __s = ((ulong)(p) & (0x4 - sizeof(*p))) * BITS_PER_BYTE; \
ulong __mask = GENMASK(((sizeof(*p)) * BITS_PER_BYTE) - 1, 0) \
<< __s; \
ulong __newx = (ulong)(n) << __s; \
ulong __oldx = (ulong)(o) << __s; \
ulong __retx; \
ulong __rc; \
\
__asm__ __volatile__ ( \
sc_prepend \
"0: lr.w %0, %2\n" \
" and %1, %0, %z5\n" \
" bne %1, %z3, 1f\n" \
" and %1, %0, %z6\n" \
" or %1, %1, %z4\n" \
" sc.w" sc_sfx " %1, %1, %2\n" \
" bnez %1, 0b\n" \
sc_append \
"1:\n" \
: "=&r" (__retx), "=&r" (__rc), "+A" (*(__ptr32b)) \
: "rJ" ((long)__oldx), "rJ" (__newx), \
"rJ" (__mask), "rJ" (~__mask) \
: "memory"); \
\
r = (__typeof__(*(p)))((__retx & __mask) >> __s); \
} \
})
#define __arch_cmpxchg(lr_sfx, sc_sfx, prepend, append, r, p, co, o, n) \
#define __arch_cmpxchg(lr_sfx, sc_sfx, cas_sfx, \
sc_prepend, sc_append, \
cas_prepend, cas_append, \
r, p, co, o, n) \
({ \
register unsigned int __rc; \
if (IS_ENABLED(CONFIG_RISCV_ISA_ZACAS) && \
riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)) { \
r = o; \
\
__asm__ __volatile__ ( \
prepend \
"0: lr" lr_sfx " %0, %2\n" \
" bne %0, %z3, 1f\n" \
" sc" sc_sfx " %1, %z4, %2\n" \
" bnez %1, 0b\n" \
append \
"1:\n" \
: "=&r" (r), "=&r" (__rc), "+A" (*(p)) \
: "rJ" (co o), "rJ" (n) \
: "memory"); \
__asm__ __volatile__ ( \
cas_prepend \
" amocas" cas_sfx " %0, %z2, %1\n" \
cas_append \
: "+&r" (r), "+A" (*(p)) \
: "rJ" (n) \
: "memory"); \
} else { \
register unsigned int __rc; \
\
__asm__ __volatile__ ( \
sc_prepend \
"0: lr" lr_sfx " %0, %2\n" \
" bne %0, %z3, 1f\n" \
" sc" sc_sfx " %1, %z4, %2\n" \
" bnez %1, 0b\n" \
sc_append \
"1:\n" \
: "=&r" (r), "=&r" (__rc), "+A" (*(p)) \
: "rJ" (co o), "rJ" (n) \
: "memory"); \
} \
})
#define _arch_cmpxchg(ptr, old, new, sc_sfx, prepend, append) \
#define _arch_cmpxchg(ptr, old, new, sc_sfx, cas_sfx, \
sc_prepend, sc_append, \
cas_prepend, cas_append) \
({ \
__typeof__(ptr) __ptr = (ptr); \
__typeof__(*(__ptr)) __old = (old); \
@@ -163,17 +216,28 @@
\
switch (sizeof(*__ptr)) { \
case 1: \
__arch_cmpxchg_masked(sc_sfx, ".b" cas_sfx, \
sc_prepend, sc_append, \
cas_prepend, cas_append, \
__ret, __ptr, __old, __new); \
break; \
case 2: \
__arch_cmpxchg_masked(sc_sfx, prepend, append, \
__ret, __ptr, __old, __new); \
__arch_cmpxchg_masked(sc_sfx, ".h" cas_sfx, \
sc_prepend, sc_append, \
cas_prepend, cas_append, \
__ret, __ptr, __old, __new); \
break; \
case 4: \
__arch_cmpxchg(".w", ".w" sc_sfx, prepend, append, \
__ret, __ptr, (long), __old, __new); \
__arch_cmpxchg(".w", ".w" sc_sfx, ".w" cas_sfx, \
sc_prepend, sc_append, \
cas_prepend, cas_append, \
__ret, __ptr, (long), __old, __new); \
break; \
case 8: \
__arch_cmpxchg(".d", ".d" sc_sfx, prepend, append, \
__ret, __ptr, /**/, __old, __new); \
__arch_cmpxchg(".d", ".d" sc_sfx, ".d" cas_sfx, \
sc_prepend, sc_append, \
cas_prepend, cas_append, \
__ret, __ptr, /**/, __old, __new); \
break; \
default: \
BUILD_BUG(); \
@@ -181,17 +245,40 @@
(__typeof__(*(__ptr)))__ret; \
})
/*
* These macros are here to improve the readability of the arch_cmpxchg_XXX()
* macros.
*/
#define SC_SFX(x) x
#define CAS_SFX(x) x
#define SC_PREPEND(x) x
#define SC_APPEND(x) x
#define CAS_PREPEND(x) x
#define CAS_APPEND(x) x
#define arch_cmpxchg_relaxed(ptr, o, n) \
_arch_cmpxchg((ptr), (o), (n), "", "", "")
_arch_cmpxchg((ptr), (o), (n), \
SC_SFX(""), CAS_SFX(""), \
SC_PREPEND(""), SC_APPEND(""), \
CAS_PREPEND(""), CAS_APPEND(""))
#define arch_cmpxchg_acquire(ptr, o, n) \
_arch_cmpxchg((ptr), (o), (n), "", "", RISCV_ACQUIRE_BARRIER)
_arch_cmpxchg((ptr), (o), (n), \
SC_SFX(""), CAS_SFX(""), \
SC_PREPEND(""), SC_APPEND(RISCV_ACQUIRE_BARRIER), \
CAS_PREPEND(""), CAS_APPEND(RISCV_ACQUIRE_BARRIER))
#define arch_cmpxchg_release(ptr, o, n) \
_arch_cmpxchg((ptr), (o), (n), "", RISCV_RELEASE_BARRIER, "")
_arch_cmpxchg((ptr), (o), (n), \
SC_SFX(""), CAS_SFX(""), \
SC_PREPEND(RISCV_RELEASE_BARRIER), SC_APPEND(""), \
CAS_PREPEND(RISCV_RELEASE_BARRIER), CAS_APPEND(""))
#define arch_cmpxchg(ptr, o, n) \
_arch_cmpxchg((ptr), (o), (n), ".rl", "", " fence rw, rw\n")
_arch_cmpxchg((ptr), (o), (n), \
SC_SFX(".rl"), CAS_SFX(".aqrl"), \
SC_PREPEND(""), SC_APPEND(RISCV_FULL_BARRIER), \
CAS_PREPEND(""), CAS_APPEND(""))
#define arch_cmpxchg_local(ptr, o, n) \
arch_cmpxchg_relaxed((ptr), (o), (n))
@@ -226,6 +313,44 @@
arch_cmpxchg_release((ptr), (o), (n)); \
})
#if defined(CONFIG_64BIT) && defined(CONFIG_RISCV_ISA_ZACAS)
#define system_has_cmpxchg128() riscv_has_extension_unlikely(RISCV_ISA_EXT_ZACAS)
union __u128_halves {
u128 full;
struct {
u64 low, high;
};
};
#define __arch_cmpxchg128(p, o, n, cas_sfx) \
({ \
__typeof__(*(p)) __o = (o); \
union __u128_halves __hn = { .full = (n) }; \
union __u128_halves __ho = { .full = (__o) }; \
register unsigned long t1 asm ("t1") = __hn.low; \
register unsigned long t2 asm ("t2") = __hn.high; \
register unsigned long t3 asm ("t3") = __ho.low; \
register unsigned long t4 asm ("t4") = __ho.high; \
\
__asm__ __volatile__ ( \
" amocas.q" cas_sfx " %0, %z3, %2" \
: "+&r" (t3), "+&r" (t4), "+A" (*(p)) \
: "rJ" (t1), "rJ" (t2) \
: "memory"); \
\
((u128)t4 << 64) | t3; \
})
#define arch_cmpxchg128(ptr, o, n) \
__arch_cmpxchg128((ptr), (o), (n), ".aqrl")
#define arch_cmpxchg128_local(ptr, o, n) \
__arch_cmpxchg128((ptr), (o), (n), "")
#endif /* CONFIG_64BIT && CONFIG_RISCV_ISA_ZACAS */
#ifdef CONFIG_RISCV_ISA_ZAWRS
/*
* Despite wrs.nto being "WRS-with-no-timeout", in the absence of changes to
@@ -245,6 +370,11 @@ static __always_inline void __cmpwait(volatile void *ptr,
: : : : no_zawrs);
switch (size) {
case 1:
fallthrough;
case 2:
/* RISC-V doesn't have lr instructions on byte and half-word. */
goto no_zawrs;
case 4:
asm volatile(
" lr.w %0, %1\n"

1
arch/riscv/include/asm/compat.h
View File

@@ -9,7 +9,6 @@
*/
#include <linux/types.h>
#include <linux/sched.h>
#include <linux/sched/task_stack.h>
#include <asm-generic/compat.h>
static inline int is_compat_task(void)

66
arch/riscv/include/asm/cpufeature-macros.h Normal file
View File

@@ -0,0 +1,66 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
* Copyright 2022-2024 Rivos, Inc
*/
#ifndef _ASM_CPUFEATURE_MACROS_H
#define _ASM_CPUFEATURE_MACROS_H
#include <asm/hwcap.h>
#include <asm/alternative-macros.h>
#define STANDARD_EXT 0
bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned int bit);
#define riscv_isa_extension_available(isa_bitmap, ext) \
__riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext)
static __always_inline bool __riscv_has_extension_likely(const unsigned long vendor,
const unsigned long ext)
{
asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1)
:
: [vendor] "i" (vendor), [ext] "i" (ext)
:
: l_no);
return true;
l_no:
return false;
}
static __always_inline bool __riscv_has_extension_unlikely(const unsigned long vendor,
const unsigned long ext)
{
asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1)
:
: [vendor] "i" (vendor), [ext] "i" (ext)
:
: l_yes);
return false;
l_yes:
return true;
}
static __always_inline bool riscv_has_extension_unlikely(const unsigned long ext)
{
compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX");
if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE))
return __riscv_has_extension_unlikely(STANDARD_EXT, ext);
return __riscv_isa_extension_available(NULL, ext);
}
static __always_inline bool riscv_has_extension_likely(const unsigned long ext)
{
compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX");
if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE))
return __riscv_has_extension_likely(STANDARD_EXT, ext);
return __riscv_isa_extension_available(NULL, ext);
}
#endif /* _ASM_CPUFEATURE_MACROS_H */

73
arch/riscv/include/asm/cpufeature.h
View File

@@ -8,9 +8,12 @@
#include <linux/bitmap.h>
#include <linux/jump_label.h>
#include <linux/workqueue.h>
#include <linux/kconfig.h>
#include <linux/percpu-defs.h>
#include <linux/threads.h>
#include <asm/hwcap.h>
#include <asm/alternative-macros.h>
#include <asm/errno.h>
#include <asm/cpufeature-macros.h>
/*
* These are probed via a device_initcall(), via either the SBI or directly
@@ -31,7 +34,7 @@ DECLARE_PER_CPU(struct riscv_cpuinfo, riscv_cpuinfo);
/* Per-cpu ISA extensions. */
extern struct riscv_isainfo hart_isa[NR_CPUS];
void riscv_user_isa_enable(void);
void __init riscv_user_isa_enable(void);
#define _RISCV_ISA_EXT_DATA(_name, _id, _subset_exts, _subset_exts_size, _validate) { \
.name = #_name, \
@@ -58,8 +61,9 @@ void riscv_user_isa_enable(void);
#define __RISCV_ISA_EXT_SUPERSET_VALIDATE(_name, _id, _sub_exts, _validate) \
_RISCV_ISA_EXT_DATA(_name, _id, _sub_exts, ARRAY_SIZE(_sub_exts), _validate)
#if defined(CONFIG_RISCV_MISALIGNED)
bool check_unaligned_access_emulated_all_cpus(void);
#if defined(CONFIG_RISCV_SCALAR_MISALIGNED)
void check_unaligned_access_emulated(struct work_struct *work __always_unused);
void unaligned_emulation_finish(void);
bool unaligned_ctl_available(void);
DECLARE_PER_CPU(long, misaligned_access_speed);
@@ -70,6 +74,12 @@ static inline bool unaligned_ctl_available(void)
}
#endif
bool check_vector_unaligned_access_emulated_all_cpus(void);
#if defined(CONFIG_RISCV_VECTOR_MISALIGNED)
void check_vector_unaligned_access_emulated(struct work_struct *work __always_unused);
DECLARE_PER_CPU(long, vector_misaligned_access);
#endif
#if defined(CONFIG_RISCV_PROBE_UNALIGNED_ACCESS)
DECLARE_STATIC_KEY_FALSE(fast_unaligned_access_speed_key);
@@ -103,61 +113,6 @@ extern const size_t riscv_isa_ext_count;
extern bool riscv_isa_fallback;
unsigned long riscv_isa_extension_base(const unsigned long *isa_bitmap);
#define STANDARD_EXT 0
bool __riscv_isa_extension_available(const unsigned long *isa_bitmap, unsigned int bit);
#define riscv_isa_extension_available(isa_bitmap, ext) \
__riscv_isa_extension_available(isa_bitmap, RISCV_ISA_EXT_##ext)
static __always_inline bool __riscv_has_extension_likely(const unsigned long vendor,
const unsigned long ext)
{
asm goto(ALTERNATIVE("j %l[l_no]", "nop", %[vendor], %[ext], 1)
:
: [vendor] "i" (vendor), [ext] "i" (ext)
:
: l_no);
return true;
l_no:
return false;
}
static __always_inline bool __riscv_has_extension_unlikely(const unsigned long vendor,
const unsigned long ext)
{
asm goto(ALTERNATIVE("nop", "j %l[l_yes]", %[vendor], %[ext], 1)
:
: [vendor] "i" (vendor), [ext] "i" (ext)
:
: l_yes);
return false;
l_yes:
return true;
}
static __always_inline bool riscv_has_extension_unlikely(const unsigned long ext)
{
compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX");
if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE))
return __riscv_has_extension_unlikely(STANDARD_EXT, ext);
return __riscv_isa_extension_available(NULL, ext);
}
static __always_inline bool riscv_has_extension_likely(const unsigned long ext)
{
compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX");
if (IS_ENABLED(CONFIG_RISCV_ALTERNATIVE))
return __riscv_has_extension_likely(STANDARD_EXT, ext);
return __riscv_isa_extension_available(NULL, ext);
}
static __always_inline bool riscv_cpu_has_extension_likely(int cpu, const unsigned long ext)
{
compiletime_assert(ext < RISCV_ISA_EXT_MAX, "ext must be < RISCV_ISA_EXT_MAX");

16
arch/riscv/include/asm/csr.h
View File

@@ -119,6 +119,10 @@
/* HSTATUS flags */
#ifdef CONFIG_64BIT
#define HSTATUS_HUPMM _AC(0x3000000000000, UL)
#define HSTATUS_HUPMM_PMLEN_0 _AC(0x0000000000000, UL)
#define HSTATUS_HUPMM_PMLEN_7 _AC(0x2000000000000, UL)
#define HSTATUS_HUPMM_PMLEN_16 _AC(0x3000000000000, UL)
#define HSTATUS_VSXL _AC(0x300000000, UL)
#define HSTATUS_VSXL_SHIFT 32
#endif
@@ -195,6 +199,10 @@
/* xENVCFG flags */
#define ENVCFG_STCE (_AC(1, ULL) << 63)
#define ENVCFG_PBMTE (_AC(1, ULL) << 62)
#define ENVCFG_PMM (_AC(0x3, ULL) << 32)
#define ENVCFG_PMM_PMLEN_0 (_AC(0x0, ULL) << 32)
#define ENVCFG_PMM_PMLEN_7 (_AC(0x2, ULL) << 32)
#define ENVCFG_PMM_PMLEN_16 (_AC(0x3, ULL) << 32)
#define ENVCFG_CBZE (_AC(1, UL) << 7)
#define ENVCFG_CBCFE (_AC(1, UL) << 6)
#define ENVCFG_CBIE_SHIFT 4
@@ -216,6 +224,12 @@
#define SMSTATEEN0_SSTATEEN0_SHIFT 63
#define SMSTATEEN0_SSTATEEN0 (_ULL(1) << SMSTATEEN0_SSTATEEN0_SHIFT)
/* mseccfg bits */
#define MSECCFG_PMM ENVCFG_PMM
#define MSECCFG_PMM_PMLEN_0 ENVCFG_PMM_PMLEN_0
#define MSECCFG_PMM_PMLEN_7 ENVCFG_PMM_PMLEN_7
#define MSECCFG_PMM_PMLEN_16 ENVCFG_PMM_PMLEN_16
/* symbolic CSR names: */
#define CSR_CYCLE 0xc00
#define CSR_TIME 0xc01
@@ -382,6 +396,8 @@
#define CSR_MIP 0x344
#define CSR_PMPCFG0 0x3a0
#define CSR_PMPADDR0 0x3b0
#define CSR_MSECCFG 0x747
#define CSR_MSECCFGH 0x757
#define CSR_MVENDORID 0xf11
#define CSR_MARCHID 0xf12
#define CSR_MIMPID 0xf13

1
arch/riscv/include/asm/entry-common.h
View File

@@ -33,6 +33,7 @@ static inline int handle_misaligned_load(struct pt_regs *regs)
{
return -1;
}
static inline int handle_misaligned_store(struct pt_regs *regs)
{
return -1;

7
arch/riscv/include/asm/hwcap.h
View File

@@ -93,6 +93,11 @@
#define RISCV_ISA_EXT_ZCMOP 84
#define RISCV_ISA_EXT_ZAWRS 85
#define RISCV_ISA_EXT_SVVPTC 86
#define RISCV_ISA_EXT_SMMPM 87
#define RISCV_ISA_EXT_SMNPM 88
#define RISCV_ISA_EXT_SSNPM 89
#define RISCV_ISA_EXT_ZABHA 90
#define RISCV_ISA_EXT_ZICCRSE 91
#define RISCV_ISA_EXT_XLINUXENVCFG 127
@@ -101,8 +106,10 @@
#ifdef CONFIG_RISCV_M_MODE
#define RISCV_ISA_EXT_SxAIA RISCV_ISA_EXT_SMAIA
#define RISCV_ISA_EXT_SUPM RISCV_ISA_EXT_SMNPM
#else
#define RISCV_ISA_EXT_SxAIA RISCV_ISA_EXT_SSAIA
#define RISCV_ISA_EXT_SUPM RISCV_ISA_EXT_SSNPM
#endif
#endif /* _ASM_RISCV_HWCAP_H */

2
arch/riscv/include/asm/hwprobe.h
View File

@@ -8,7 +8,7 @@
#include <uapi/asm/hwprobe.h>
#define RISCV_HWPROBE_MAX_KEY 9
#define RISCV_HWPROBE_MAX_KEY 10
static inline bool riscv_hwprobe_key_is_valid(__s64 key)
{

7
arch/riscv/include/asm/mmu.h
View File

@@ -25,9 +25,16 @@ typedef struct {
#ifdef CONFIG_BINFMT_ELF_FDPIC
unsigned long exec_fdpic_loadmap;
unsigned long interp_fdpic_loadmap;
#endif
unsigned long flags;
#ifdef CONFIG_RISCV_ISA_SUPM
u8 pmlen;
#endif
} mm_context_t;
/* Lock the pointer masking mode because this mm is multithreaded */
#define MM_CONTEXT_LOCK_PMLEN 0
#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK)
#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK)

13
arch/riscv/include/asm/mmu_context.h
View File

@@ -20,6 +20,9 @@ void switch_mm(struct mm_struct *prev, struct mm_struct *next,
static inline void activate_mm(struct mm_struct *prev,
struct mm_struct *next)
{
#ifdef CONFIG_RISCV_ISA_SUPM
next->context.pmlen = 0;
#endif
switch_mm(prev, next, NULL);
}
@@ -30,11 +33,21 @@ static inline int init_new_context(struct task_struct *tsk,
#ifdef CONFIG_MMU
atomic_long_set(&mm->context.id, 0);
#endif
if (IS_ENABLED(CONFIG_RISCV_ISA_SUPM))
clear_bit(MM_CONTEXT_LOCK_PMLEN, &mm->context.flags);
return 0;
}
DECLARE_STATIC_KEY_FALSE(use_asid_allocator);
#ifdef CONFIG_RISCV_ISA_SUPM
#define mm_untag_mask mm_untag_mask
static inline unsigned long mm_untag_mask(struct mm_struct *mm)
{
return -1UL >> mm->context.pmlen;
}
#endif
#include <asm-generic/mmu_context.h>
#endif /* _ASM_RISCV_MMU_CONTEXT_H */

9
arch/riscv/include/asm/processor.h
View File

@@ -102,6 +102,7 @@ struct thread_struct {
unsigned long s[12]; /* s[0]: frame pointer */
struct __riscv_d_ext_state fstate;
unsigned long bad_cause;
unsigned long envcfg;
u32 riscv_v_flags;
u32 vstate_ctrl;
struct __riscv_v_ext_state vstate;
@@ -177,6 +178,14 @@ extern int set_unalign_ctl(struct task_struct *tsk, unsigned int val);
#define RISCV_SET_ICACHE_FLUSH_CTX(arg1, arg2) riscv_set_icache_flush_ctx(arg1, arg2)
extern int riscv_set_icache_flush_ctx(unsigned long ctx, unsigned long per_thread);
#ifdef CONFIG_RISCV_ISA_SUPM
/* PR_{SET,GET}_TAGGED_ADDR_CTRL prctl */
long set_tagged_addr_ctrl(struct task_struct *task, unsigned long arg);
long get_tagged_addr_ctrl(struct task_struct *task);
#define SET_TAGGED_ADDR_CTRL(arg) set_tagged_addr_ctrl(current, arg)
#define GET_TAGGED_ADDR_CTRL() get_tagged_addr_ctrl(current)
#endif
#endif /* __ASSEMBLY__ */
#endif /* _ASM_RISCV_PROCESSOR_H */

47
arch/riscv/include/asm/spinlock.h Normal file
View File

@@ -0,0 +1,47 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef __ASM_RISCV_SPINLOCK_H
#define __ASM_RISCV_SPINLOCK_H
#ifdef CONFIG_RISCV_COMBO_SPINLOCKS
#define _Q_PENDING_LOOPS (1 << 9)
#define __no_arch_spinlock_redefine
#include <asm/ticket_spinlock.h>
#include <asm/qspinlock.h>
#include <asm/jump_label.h>
/*
* TODO: Use an alternative instead of a static key when we are able to parse
* the extensions string earlier in the boot process.
*/
DECLARE_STATIC_KEY_TRUE(qspinlock_key);
#define SPINLOCK_BASE_DECLARE(op, type, type_lock) \
static __always_inline type arch_spin_##op(type_lock lock) \
{ \
if (static_branch_unlikely(&qspinlock_key)) \
return queued_spin_##op(lock); \
return ticket_spin_##op(lock); \
}
SPINLOCK_BASE_DECLARE(lock, void, arch_spinlock_t *)
SPINLOCK_BASE_DECLARE(unlock, void, arch_spinlock_t *)
SPINLOCK_BASE_DECLARE(is_locked, int, arch_spinlock_t *)
SPINLOCK_BASE_DECLARE(is_contended, int, arch_spinlock_t *)
SPINLOCK_BASE_DECLARE(trylock, bool, arch_spinlock_t *)
SPINLOCK_BASE_DECLARE(value_unlocked, int, arch_spinlock_t)
#elif defined(CONFIG_RISCV_QUEUED_SPINLOCKS)
#include <asm/qspinlock.h>
#else
#include <asm/ticket_spinlock.h>
#endif
#include <asm/qrwlock.h>
#endif /* __ASM_RISCV_SPINLOCK_H */

Some files were not shown because too many files have changed in this diff Show More