mirror of
https://github.com/suyu-emu/horinux.git
synced 2026-09-21 14:15:21 -07:00
Merge remote-tracking branch 'arm64/for-next/sme' into kvmarm-master/next
Merge arm64's SME branch to resolve conflicts with the WFxT branch. Signed-off-by: Marc Zyngier <maz@kernel.org>
This commit is contained in:
Marc Zyngier
@@ -264,6 +264,39 @@ HWCAP2_MTE3
|
||||
Functionality implied by ID_AA64PFR1_EL1.MTE == 0b0011, as described
|
||||
by Documentation/arm64/memory-tagging-extension.rst.
|
||||
|
||||
HWCAP2_SME
|
||||
|
||||
Functionality implied by ID_AA64PFR1_EL1.SME == 0b0001, as described
|
||||
by Documentation/arm64/sme.rst.
|
||||
|
||||
HWCAP2_SME_I16I64
|
||||
|
||||
Functionality implied by ID_AA64SMFR0_EL1.I16I64 == 0b1111.
|
||||
|
||||
HWCAP2_SME_F64F64
|
||||
|
||||
Functionality implied by ID_AA64SMFR0_EL1.F64F64 == 0b1.
|
||||
|
||||
HWCAP2_SME_I8I32
|
||||
|
||||
Functionality implied by ID_AA64SMFR0_EL1.I8I32 == 0b1111.
|
||||
|
||||
HWCAP2_SME_F16F32
|
||||
|
||||
Functionality implied by ID_AA64SMFR0_EL1.F16F32 == 0b1.
|
||||
|
||||
HWCAP2_SME_B16F32
|
||||
|
||||
Functionality implied by ID_AA64SMFR0_EL1.B16F32 == 0b1.
|
||||
|
||||
HWCAP2_SME_F32F32
|
||||
|
||||
Functionality implied by ID_AA64SMFR0_EL1.F32F32 == 0b1.
|
||||
|
||||
HWCAP2_SME_FA64
|
||||
|
||||
Functionality implied by ID_AA64SMFR0_EL1.FA64 == 0b1.
|
||||
|
||||
4. Unused AT_HWCAP bits
|
||||
-----------------------
|
||||
|
||||
|
||||
@@ -21,6 +21,7 @@ ARM64 Architecture
|
||||
perf
|
||||
pointer-authentication
|
||||
silicon-errata
|
||||
sme
|
||||
sve
|
||||
tagged-address-abi
|
||||
tagged-pointers
|
||||
|
||||
@@ -0,0 +1,428 @@
|
||||
===================================================
|
||||
Scalable Matrix Extension support for AArch64 Linux
|
||||
===================================================
|
||||
|
||||
This document outlines briefly the interface provided to userspace by Linux in
|
||||
order to support use of the ARM Scalable Matrix Extension (SME).
|
||||
|
||||
This is an outline of the most important features and issues only and not
|
||||
intended to be exhaustive. It should be read in conjunction with the SVE
|
||||
documentation in sve.rst which provides details on the Streaming SVE mode
|
||||
included in SME.
|
||||
|
||||
This document does not aim to describe the SME architecture or programmer's
|
||||
model. To aid understanding, a minimal description of relevant programmer's
|
||||
model features for SME is included in Appendix A.
|
||||
|
||||
|
||||
1. General
|
||||
-----------
|
||||
|
||||
* PSTATE.SM, PSTATE.ZA, the streaming mode vector length, the ZA
|
||||
register state and TPIDR2_EL0 are tracked per thread.
|
||||
|
||||
* The presence of SME is reported to userspace via HWCAP2_SME in the aux vector
|
||||
AT_HWCAP2 entry. Presence of this flag implies the presence of the SME
|
||||
instructions and registers, and the Linux-specific system interfaces
|
||||
described in this document. SME is reported in /proc/cpuinfo as "sme".
|
||||
|
||||
* Support for the execution of SME instructions in userspace can also be
|
||||
detected by reading the CPU ID register ID_AA64PFR1_EL1 using an MRS
|
||||
instruction, and checking that the value of the SME field is nonzero. [3]
|
||||
|
||||
It does not guarantee the presence of the system interfaces described in the
|
||||
following sections: software that needs to verify that those interfaces are
|
||||
present must check for HWCAP2_SME instead.
|
||||
|
||||
* There are a number of optional SME features, presence of these is reported
|
||||
through AT_HWCAP2 through:
|
||||
|
||||
HWCAP2_SME_I16I64
|
||||
HWCAP2_SME_F64F64
|
||||
HWCAP2_SME_I8I32
|
||||
HWCAP2_SME_F16F32
|
||||
HWCAP2_SME_B16F32
|
||||
HWCAP2_SME_F32F32
|
||||
HWCAP2_SME_FA64
|
||||
|
||||
This list may be extended over time as the SME architecture evolves.
|
||||
|
||||
These extensions are also reported via the CPU ID register ID_AA64SMFR0_EL1,
|
||||
which userspace can read using an MRS instruction. See elf_hwcaps.txt and
|
||||
cpu-feature-registers.txt for details.
|
||||
|
||||
* Debuggers should restrict themselves to interacting with the target via the
|
||||
NT_ARM_SVE, NT_ARM_SSVE and NT_ARM_ZA regsets. The recommended way
|
||||
of detecting support for these regsets is to connect to a target process
|
||||
first and then attempt a
|
||||
|
||||
ptrace(PTRACE_GETREGSET, pid, NT_ARM_<regset>, &iov).
|
||||
|
||||
* Whenever ZA register values are exchanged in memory between userspace and
|
||||
the kernel, the register value is encoded in memory as a series of horizontal
|
||||
vectors from 0 to VL/8-1 stored in the same endianness invariant format as is
|
||||
used for SVE vectors.
|
||||
|
||||
* On thread creation TPIDR2_EL0 is preserved unless CLONE_SETTLS is specified,
|
||||
in which case it is set to 0.
|
||||
|
||||
2. Vector lengths
|
||||
------------------
|
||||
|
||||
SME defines a second vector length similar to the SVE vector length which is
|
||||
controls the size of the streaming mode SVE vectors and the ZA matrix array.
|
||||
The ZA matrix is square with each side having as many bytes as a streaming
|
||||
mode SVE vector.
|
||||
|
||||
|
||||
3. Sharing of streaming and non-streaming mode SVE state
|
||||
---------------------------------------------------------
|
||||
|
||||
It is implementation defined which if any parts of the SVE state are shared
|
||||
between streaming and non-streaming modes. When switching between modes
|
||||
via software interfaces such as ptrace if no register content is provided as
|
||||
part of switching no state will be assumed to be shared and everything will
|
||||
be zeroed.
|
||||
|
||||
|
||||
4. System call behaviour
|
||||
-------------------------
|
||||
|
||||
* On syscall PSTATE.ZA is preserved, if PSTATE.ZA==1 then the contents of the
|
||||
ZA matrix are preserved.
|
||||
|
||||
* On syscall PSTATE.SM will be cleared and the SVE registers will be handled
|
||||
as per the standard SVE ABI.
|
||||
|
||||
* Neither the SVE registers nor ZA are used to pass arguments to or receive
|
||||
results from any syscall.
|
||||
|
||||
* On process creation (eg, clone()) the newly created process will have
|
||||
PSTATE.SM cleared.
|
||||
|
||||
* All other SME state of a thread, including the currently configured vector
|
||||
length, the state of the PR_SME_VL_INHERIT flag, and the deferred vector
|
||||
length (if any), is preserved across all syscalls, subject to the specific
|
||||
exceptions for execve() described in section 6.
|
||||
|
||||
|
||||
5. Signal handling
|
||||
-------------------
|
||||
|
||||
* Signal handlers are invoked with streaming mode and ZA disabled.
|
||||
|
||||
* A new signal frame record za_context encodes the ZA register contents on
|
||||
signal delivery. [1]
|
||||
|
||||
* The signal frame record for ZA always contains basic metadata, in particular
|
||||
the thread's vector length (in za_context.vl).
|
||||
|
||||
* The ZA matrix may or may not be included in the record, depending on
|
||||
the value of PSTATE.ZA. The registers are present if and only if:
|
||||
za_context.head.size >= ZA_SIG_CONTEXT_SIZE(sve_vq_from_vl(za_context.vl))
|
||||
in which case PSTATE.ZA == 1.
|
||||
|
||||
* If matrix data is present, the remainder of the record has a vl-dependent
|
||||
size and layout. Macros ZA_SIG_* are defined [1] to facilitate access to
|
||||
them.
|
||||
|
||||
* The matrix is stored as a series of horizontal vectors in the same format as
|
||||
is used for SVE vectors.
|
||||
|
||||
* If the ZA context is too big to fit in sigcontext.__reserved[], then extra
|
||||
space is allocated on the stack, an extra_context record is written in
|
||||
__reserved[] referencing this space. za_context is then written in the
|
||||
extra space. Refer to [1] for further details about this mechanism.
|
||||
|
||||
|
||||
5. Signal return
|
||||
-----------------
|
||||
|
||||
When returning from a signal handler:
|
||||
|
||||
* If there is no za_context record in the signal frame, or if the record is
|
||||
present but contains no register data as described in the previous section,
|
||||
then ZA is disabled.
|
||||
|
||||
* If za_context is present in the signal frame and contains matrix data then
|
||||
PSTATE.ZA is set to 1 and ZA is populated with the specified data.
|
||||
|
||||
* The vector length cannot be changed via signal return. If za_context.vl in
|
||||
the signal frame does not match the current vector length, the signal return
|
||||
attempt is treated as illegal, resulting in a forced SIGSEGV.
|
||||
|
||||
|
||||
6. prctl extensions
|
||||
--------------------
|
||||
|
||||
Some new prctl() calls are added to allow programs to manage the SME vector
|
||||
length:
|
||||
|
||||
prctl(PR_SME_SET_VL, unsigned long arg)
|
||||
|
||||
Sets the vector length of the calling thread and related flags, where
|
||||
arg == vl | flags. Other threads of the calling process are unaffected.
|
||||
|
||||
vl is the desired vector length, where sve_vl_valid(vl) must be true.
|
||||
|
||||
flags:
|
||||
|
||||
PR_SME_VL_INHERIT
|
||||
|
||||
Inherit the current vector length across execve(). Otherwise, the
|
||||
vector length is reset to the system default at execve(). (See
|
||||
Section 9.)
|
||||
|
||||
PR_SME_SET_VL_ONEXEC
|
||||
|
||||
Defer the requested vector length change until the next execve()
|
||||
performed by this thread.
|
||||
|
||||
The effect is equivalent to implicit execution of the following
|
||||
call immediately after the next execve() (if any) by the thread:
|
||||
|
||||
prctl(PR_SME_SET_VL, arg & ~PR_SME_SET_VL_ONEXEC)
|
||||
|
||||
This allows launching of a new program with a different vector
|
||||
length, while avoiding runtime side effects in the caller.
|
||||
|
||||
Without PR_SME_SET_VL_ONEXEC, the requested change takes effect
|
||||
immediately.
|
||||
|
||||
|
||||
Return value: a nonnegative on success, or a negative value on error:
|
||||
EINVAL: SME not supported, invalid vector length requested, or
|
||||
invalid flags.
|
||||
|
||||
|
||||
On success:
|
||||
|
||||
* Either the calling thread's vector length or the deferred vector length
|
||||
to be applied at the next execve() by the thread (dependent on whether
|
||||
PR_SME_SET_VL_ONEXEC is present in arg), is set to the largest value
|
||||
supported by the system that is less than or equal to vl. If vl ==
|
||||
SVE_VL_MAX, the value set will be the largest value supported by the
|
||||
system.
|
||||
|
||||
* Any previously outstanding deferred vector length change in the calling
|
||||
thread is cancelled.
|
||||
|
||||
* The returned value describes the resulting configuration, encoded as for
|
||||
PR_SME_GET_VL. The vector length reported in this value is the new
|
||||
current vector length for this thread if PR_SME_SET_VL_ONEXEC was not
|
||||
present in arg; otherwise, the reported vector length is the deferred
|
||||
vector length that will be applied at the next execve() by the calling
|
||||
thread.
|
||||
|
||||
* Changing the vector length causes all of ZA, P0..P15, FFR and all bits of
|
||||
Z0..Z31 except for Z0 bits [127:0] .. Z31 bits [127:0] to become
|
||||
unspecified, including both streaming and non-streaming SVE state.
|
||||
Calling PR_SME_SET_VL with vl equal to the thread's current vector
|
||||
length, or calling PR_SME_SET_VL with the PR_SVE_SET_VL_ONEXEC flag,
|
||||
does not constitute a change to the vector length for this purpose.
|
||||
|
||||
* Changing the vector length causes PSTATE.ZA and PSTATE.SM to be cleared.
|
||||
Calling PR_SME_SET_VL with vl equal to the thread's current vector
|
||||
length, or calling PR_SME_SET_VL with the PR_SVE_SET_VL_ONEXEC flag,
|
||||
does not constitute a change to the vector length for this purpose.
|
||||
|
||||
|
||||
prctl(PR_SME_GET_VL)
|
||||
|
||||
Gets the vector length of the calling thread.
|
||||
|
||||
The following flag may be OR-ed into the result:
|
||||
|
||||
PR_SME_VL_INHERIT
|
||||
|
||||
Vector length will be inherited across execve().
|
||||
|
||||
There is no way to determine whether there is an outstanding deferred
|
||||
vector length change (which would only normally be the case between a
|
||||
fork() or vfork() and the corresponding execve() in typical use).
|
||||
|
||||
To extract the vector length from the result, bitwise and it with
|
||||
PR_SME_VL_LEN_MASK.
|
||||
|
||||
Return value: a nonnegative value on success, or a negative value on error:
|
||||
EINVAL: SME not supported.
|
||||
|
||||
|
||||
7. ptrace extensions
|
||||
---------------------
|
||||
|
||||
* A new regset NT_ARM_SSVE is defined for access to streaming mode SVE
|
||||
state via PTRACE_GETREGSET and PTRACE_SETREGSET, this is documented in
|
||||
sve.rst.
|
||||
|
||||
* A new regset NT_ARM_ZA is defined for ZA state for access to ZA state via
|
||||
PTRACE_GETREGSET and PTRACE_SETREGSET.
|
||||
|
||||
Refer to [2] for definitions.
|
||||
|
||||
The regset data starts with struct user_za_header, containing:
|
||||
|
||||
size
|
||||
|
||||
Size of the complete regset, in bytes.
|
||||
This depends on vl and possibly on other things in the future.
|
||||
|
||||
If a call to PTRACE_GETREGSET requests less data than the value of
|
||||
size, the caller can allocate a larger buffer and retry in order to
|
||||
read the complete regset.
|
||||
|
||||
max_size
|
||||
|
||||
Maximum size in bytes that the regset can grow to for the target
|
||||
thread. The regset won't grow bigger than this even if the target
|
||||
thread changes its vector length etc.
|
||||
|
||||
vl
|
||||
|
||||
Target thread's current streaming vector length, in bytes.
|
||||
|
||||
max_vl
|
||||
|
||||
Maximum possible streaming vector length for the target thread.
|
||||
|
||||
flags
|
||||
|
||||
Zero or more of the following flags, which have the same
|
||||
meaning and behaviour as the corresponding PR_SET_VL_* flags:
|
||||
|
||||
SME_PT_VL_INHERIT
|
||||
|
||||
SME_PT_VL_ONEXEC (SETREGSET only).
|
||||
|
||||
* The effects of changing the vector length and/or flags are equivalent to
|
||||
those documented for PR_SME_SET_VL.
|
||||
|
||||
The caller must make a further GETREGSET call if it needs to know what VL is
|
||||
actually set by SETREGSET, unless is it known in advance that the requested
|
||||
VL is supported.
|
||||
|
||||
* The size and layout of the payload depends on the header fields. The
|
||||
SME_PT_ZA_*() macros are provided to facilitate access to the data.
|
||||
|
||||
* In either case, for SETREGSET it is permissible to omit the payload, in which
|
||||
case the vector length and flags are changed and PSTATE.ZA is set to 0
|
||||
(along with any consequences of those changes). If a payload is provided
|
||||
then PSTATE.ZA will be set to 1.
|
||||
|
||||
* For SETREGSET, if the requested VL is not supported, the effect will be the
|
||||
same as if the payload were omitted, except that an EIO error is reported.
|
||||
No attempt is made to translate the payload data to the correct layout
|
||||
for the vector length actually set. It is up to the caller to translate the
|
||||
payload layout for the actual VL and retry.
|
||||
|
||||
* The effect of writing a partial, incomplete payload is unspecified.
|
||||
|
||||
|
||||
8. ELF coredump extensions
|
||||
---------------------------
|
||||
|
||||
* NT_ARM_SSVE notes will be added to each coredump for
|
||||
each thread of the dumped process. The contents will be equivalent to the
|
||||
data that would have been read if a PTRACE_GETREGSET of the corresponding
|
||||
type were executed for each thread when the coredump was generated.
|
||||
|
||||
* A NT_ARM_ZA note will be added to each coredump for each thread of the
|
||||
dumped process. The contents will be equivalent to the data that would have
|
||||
been read if a PTRACE_GETREGSET of NT_ARM_ZA were executed for each thread
|
||||
when the coredump was generated.
|
||||
|
||||
|
||||
9. System runtime configuration
|
||||
--------------------------------
|
||||
|
||||
* To mitigate the ABI impact of expansion of the signal frame, a policy
|
||||
mechanism is provided for administrators, distro maintainers and developers
|
||||
to set the default vector length for userspace processes:
|
||||
|
||||
/proc/sys/abi/sme_default_vector_length
|
||||
|
||||
Writing the text representation of an integer to this file sets the system
|
||||
default vector length to the specified value, unless the value is greater
|
||||
than the maximum vector length supported by the system in which case the
|
||||
default vector length is set to that maximum.
|
||||
|
||||
The result can be determined by reopening the file and reading its
|
||||
contents.
|
||||
|
||||
At boot, the default vector length is initially set to 32 or the maximum
|
||||
supported vector length, whichever is smaller and supported. This
|
||||
determines the initial vector length of the init process (PID 1).
|
||||
|
||||
Reading this file returns the current system default vector length.
|
||||
|
||||
* At every execve() call, the new vector length of the new process is set to
|
||||
the system default vector length, unless
|
||||
|
||||
* PR_SME_VL_INHERIT (or equivalently SME_PT_VL_INHERIT) is set for the
|
||||
calling thread, or
|
||||
|
||||
* a deferred vector length change is pending, established via the
|
||||
PR_SME_SET_VL_ONEXEC flag (or SME_PT_VL_ONEXEC).
|
||||
|
||||
* Modifying the system default vector length does not affect the vector length
|
||||
of any existing process or thread that does not make an execve() call.
|
||||
|
||||
|
||||
Appendix A. SME programmer's model (informative)
|
||||
=================================================
|
||||
|
||||
This section provides a minimal description of the additions made by SVE to the
|
||||
ARMv8-A programmer's model that are relevant to this document.
|
||||
|
||||
Note: This section is for information only and not intended to be complete or
|
||||
to replace any architectural specification.
|
||||
|
||||
A.1. Registers
|
||||
---------------
|
||||
|
||||
In A64 state, SME adds the following:
|
||||
|
||||
* A new mode, streaming mode, in which a subset of the normal FPSIMD and SVE
|
||||
features are available. When supported EL0 software may enter and leave
|
||||
streaming mode at any time.
|
||||
|
||||
For best system performance it is strongly encouraged for software to enable
|
||||
streaming mode only when it is actively being used.
|
||||
|
||||
* A new vector length controlling the size of ZA and the Z registers when in
|
||||
streaming mode, separately to the vector length used for SVE when not in
|
||||
streaming mode. There is no requirement that either the currently selected
|
||||
vector length or the set of vector lengths supported for the two modes in
|
||||
a given system have any relationship. The streaming mode vector length
|
||||
is referred to as SVL.
|
||||
|
||||
* A new ZA matrix register. This is a square matrix of SVLxSVL bits. Most
|
||||
operations on ZA require that streaming mode be enabled but ZA can be
|
||||
enabled without streaming mode in order to load, save and retain data.
|
||||
|
||||
For best system performance it is strongly encouraged for software to enable
|
||||
ZA only when it is actively being used.
|
||||
|
||||
* Two new 1 bit fields in PSTATE which may be controlled via the SMSTART and
|
||||
SMSTOP instructions or by access to the SVCR system register:
|
||||
|
||||
* PSTATE.ZA, if this is 1 then the ZA matrix is accessible and has valid
|
||||
data while if it is 0 then ZA can not be accessed. When PSTATE.ZA is
|
||||
changed from 0 to 1 all bits in ZA are cleared.
|
||||
|
||||
* PSTATE.SM, if this is 1 then the PE is in streaming mode. When the value
|
||||
of PSTATE.SM is changed then it is implementation defined if the subset
|
||||
of the floating point register bits valid in both modes may be retained.
|
||||
Any other bits will be cleared.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
[1] arch/arm64/include/uapi/asm/sigcontext.h
|
||||
AArch64 Linux signal ABI definitions
|
||||
|
||||
[2] arch/arm64/include/uapi/asm/ptrace.h
|
||||
AArch64 Linux ptrace ABI definitions
|
||||
|
||||
[3] Documentation/arm64/cpu-feature-registers.rst
|
||||
+60
-10
@@ -7,7 +7,9 @@ Author: Dave Martin <Dave.Martin@arm.com>
|
||||
Date: 4 August 2017
|
||||
|
||||
This document outlines briefly the interface provided to userspace by Linux in
|
||||
order to support use of the ARM Scalable Vector Extension (SVE).
|
||||
order to support use of the ARM Scalable Vector Extension (SVE), including
|
||||
interactions with Streaming SVE mode added by the Scalable Matrix Extension
|
||||
(SME).
|
||||
|
||||
This is an outline of the most important features and issues only and not
|
||||
intended to be exhaustive.
|
||||
@@ -23,6 +25,10 @@ model features for SVE is included in Appendix A.
|
||||
* SVE registers Z0..Z31, P0..P15 and FFR and the current vector length VL, are
|
||||
tracked per-thread.
|
||||
|
||||
* In streaming mode FFR is not accessible unless HWCAP2_SME_FA64 is present
|
||||
in the system, when it is not supported and these interfaces are used to
|
||||
access streaming mode FFR is read and written as zero.
|
||||
|
||||
* The presence of SVE is reported to userspace via HWCAP_SVE in the aux vector
|
||||
AT_HWCAP entry. Presence of this flag implies the presence of the SVE
|
||||
instructions and registers, and the Linux-specific system interfaces
|
||||
@@ -53,10 +59,19 @@ model features for SVE is included in Appendix A.
|
||||
which userspace can read using an MRS instruction. See elf_hwcaps.txt and
|
||||
cpu-feature-registers.txt for details.
|
||||
|
||||
* On hardware that supports the SME extensions, HWCAP2_SME will also be
|
||||
reported in the AT_HWCAP2 aux vector entry. Among other things SME adds
|
||||
streaming mode which provides a subset of the SVE feature set using a
|
||||
separate SME vector length and the same Z/V registers. See sme.rst
|
||||
for more details.
|
||||
|
||||
* Debuggers should restrict themselves to interacting with the target via the
|
||||
NT_ARM_SVE regset. The recommended way of detecting support for this regset
|
||||
is to connect to a target process first and then attempt a
|
||||
ptrace(PTRACE_GETREGSET, pid, NT_ARM_SVE, &iov).
|
||||
ptrace(PTRACE_GETREGSET, pid, NT_ARM_SVE, &iov). Note that when SME is
|
||||
present and streaming SVE mode is in use the FPSIMD subset of registers
|
||||
will be read via NT_ARM_SVE and NT_ARM_SVE writes will exit streaming mode
|
||||
in the target.
|
||||
|
||||
* Whenever SVE scalable register values (Zn, Pn, FFR) are exchanged in memory
|
||||
between userspace and the kernel, the register value is encoded in memory in
|
||||
@@ -126,6 +141,11 @@ the SVE instruction set architecture.
|
||||
are only present in fpsimd_context. For convenience, the content of V0..V31
|
||||
is duplicated between sve_context and fpsimd_context.
|
||||
|
||||
* The record contains a flag field which includes a flag SVE_SIG_FLAG_SM which
|
||||
if set indicates that the thread is in streaming mode and the vector length
|
||||
and register data (if present) describe the streaming SVE data and vector
|
||||
length.
|
||||
|
||||
* The signal frame record for SVE always contains basic metadata, in particular
|
||||
the thread's vector length (in sve_context.vl).
|
||||
|
||||
@@ -170,6 +190,11 @@ When returning from a signal handler:
|
||||
the signal frame does not match the current vector length, the signal return
|
||||
attempt is treated as illegal, resulting in a forced SIGSEGV.
|
||||
|
||||
* It is permitted to enter or leave streaming mode by setting or clearing
|
||||
the SVE_SIG_FLAG_SM flag but applications should take care to ensure that
|
||||
when doing so sve_context.vl and any register data are appropriate for the
|
||||
vector length in the new mode.
|
||||
|
||||
|
||||
6. prctl extensions
|
||||
--------------------
|
||||
@@ -265,8 +290,14 @@ prctl(PR_SVE_GET_VL)
|
||||
7. ptrace extensions
|
||||
---------------------
|
||||
|
||||
* A new regset NT_ARM_SVE is defined for use with PTRACE_GETREGSET and
|
||||
PTRACE_SETREGSET.
|
||||
* New regsets NT_ARM_SVE and NT_ARM_SSVE are defined for use with
|
||||
PTRACE_GETREGSET and PTRACE_SETREGSET. NT_ARM_SSVE describes the
|
||||
streaming mode SVE registers and NT_ARM_SVE describes the
|
||||
non-streaming mode SVE registers.
|
||||
|
||||
In this description a register set is referred to as being "live" when
|
||||
the target is in the appropriate streaming or non-streaming mode and is
|
||||
using data beyond the subset shared with the FPSIMD Vn registers.
|
||||
|
||||
Refer to [2] for definitions.
|
||||
|
||||
@@ -297,7 +328,7 @@ The regset data starts with struct user_sve_header, containing:
|
||||
|
||||
flags
|
||||
|
||||
either
|
||||
at most one of
|
||||
|
||||
SVE_PT_REGS_FPSIMD
|
||||
|
||||
@@ -331,6 +362,10 @@ The regset data starts with struct user_sve_header, containing:
|
||||
|
||||
SVE_PT_VL_ONEXEC (SETREGSET only).
|
||||
|
||||
If neither FPSIMD nor SVE flags are provided then no register
|
||||
payload is available, this is only possible when SME is implemented.
|
||||
|
||||
|
||||
* The effects of changing the vector length and/or flags are equivalent to
|
||||
those documented for PR_SVE_SET_VL.
|
||||
|
||||
@@ -346,6 +381,13 @@ The regset data starts with struct user_sve_header, containing:
|
||||
case only the vector length and flags are changed (along with any
|
||||
consequences of those changes).
|
||||
|
||||
* In systems supporting SME when in streaming mode a GETREGSET for
|
||||
NT_REG_SVE will return only the user_sve_header with no register data,
|
||||
similarly a GETREGSET for NT_REG_SSVE will not return any register data
|
||||
when not in streaming mode.
|
||||
|
||||
* A GETREGSET for NT_ARM_SSVE will never return SVE_PT_REGS_FPSIMD.
|
||||
|
||||
* For SETREGSET, if an SVE_PT_REGS_SVE payload is present and the
|
||||
requested VL is not supported, the effect will be the same as if the
|
||||
payload were omitted, except that an EIO error is reported. No
|
||||
@@ -355,17 +397,25 @@ The regset data starts with struct user_sve_header, containing:
|
||||
unspecified. It is up to the caller to translate the payload layout
|
||||
for the actual VL and retry.
|
||||
|
||||
* Where SME is implemented it is not possible to GETREGSET the register
|
||||
state for normal SVE when in streaming mode, nor the streaming mode
|
||||
register state when in normal mode, regardless of the implementation defined
|
||||
behaviour of the hardware for sharing data between the two modes.
|
||||
|
||||
* Any SETREGSET of NT_ARM_SVE will exit streaming mode if the target was in
|
||||
streaming mode and any SETREGSET of NT_ARM_SSVE will enter streaming mode
|
||||
if the target was not in streaming mode.
|
||||
|
||||
* The effect of writing a partial, incomplete payload is unspecified.
|
||||
|
||||
|
||||
8. ELF coredump extensions
|
||||
---------------------------
|
||||
|
||||
* A NT_ARM_SVE note will be added to each coredump for each thread of the
|
||||
dumped process. The contents will be equivalent to the data that would have
|
||||
been read if a PTRACE_GETREGSET of NT_ARM_SVE were executed for each thread
|
||||
when the coredump was generated.
|
||||
|
||||
* NT_ARM_SVE and NT_ARM_SSVE notes will be added to each coredump for
|
||||
each thread of the dumped process. The contents will be equivalent to the
|
||||
data that would have been read if a PTRACE_GETREGSET of the corresponding
|
||||
type were executed for each thread when the coredump was generated.
|
||||
|
||||
9. System runtime configuration
|
||||
--------------------------------
|
||||
|
||||
@@ -1948,6 +1948,17 @@ config ARM64_SVE
|
||||
booting the kernel. If unsure and you are not observing these
|
||||
symptoms, you should assume that it is safe to say Y.
|
||||
|
||||
config ARM64_SME
|
||||
bool "ARM Scalable Matrix Extension support"
|
||||
default y
|
||||
depends on ARM64_SVE
|
||||
help
|
||||
The Scalable Matrix Extension (SME) is an extension to the AArch64
|
||||
execution state which utilises a substantial subset of the SVE
|
||||
instruction set, together with the addition of new architectural
|
||||
register state capable of holding two dimensional matrix tiles to
|
||||
enable various matrix operations.
|
||||
|
||||
config ARM64_MODULE_PLTS
|
||||
bool "Use PLTs to allow module memory to spill over into vmalloc area"
|
||||
depends on MODULES
|
||||
|
||||
@@ -58,11 +58,15 @@ struct cpuinfo_arm64 {
|
||||
u64 reg_id_aa64pfr0;
|
||||
u64 reg_id_aa64pfr1;
|
||||
u64 reg_id_aa64zfr0;
|
||||
u64 reg_id_aa64smfr0;
|
||||
|
||||
struct cpuinfo_32bit aarch32;
|
||||
|
||||
/* pseudo-ZCR for recording maximum ZCR_EL1 LEN value: */
|
||||
u64 reg_zcr;
|
||||
|
||||
/* pseudo-SMCR for recording maximum SMCR_EL1 LEN value: */
|
||||
u64 reg_smcr;
|
||||
};
|
||||
|
||||
DECLARE_PER_CPU(struct cpuinfo_arm64, cpu_data);
|
||||
|
||||
@@ -622,6 +622,13 @@ static inline bool id_aa64pfr0_sve(u64 pfr0)
|
||||
return val > 0;
|
||||
}
|
||||
|
||||
static inline bool id_aa64pfr1_sme(u64 pfr1)
|
||||
{
|
||||
u32 val = cpuid_feature_extract_unsigned_field(pfr1, ID_AA64PFR1_SME_SHIFT);
|
||||
|
||||
return val > 0;
|
||||
}
|
||||
|
||||
static inline bool id_aa64pfr1_mte(u64 pfr1)
|
||||
{
|
||||
u32 val = cpuid_feature_extract_unsigned_field(pfr1, ID_AA64PFR1_MTE_SHIFT);
|
||||
@@ -759,6 +766,23 @@ static __always_inline bool system_supports_sve(void)
|
||||
cpus_have_const_cap(ARM64_SVE);
|
||||
}
|
||||
|
||||
static __always_inline bool system_supports_sme(void)
|
||||
{
|
||||
return IS_ENABLED(CONFIG_ARM64_SME) &&
|
||||
cpus_have_const_cap(ARM64_SME);
|
||||
}
|
||||
|
||||
static __always_inline bool system_supports_fa64(void)
|
||||
{
|
||||
return IS_ENABLED(CONFIG_ARM64_SME) &&
|
||||
cpus_have_const_cap(ARM64_SME_FA64);
|
||||
}
|
||||
|
||||
static __always_inline bool system_supports_tpidr2(void)
|
||||
{
|
||||
return system_supports_sme();
|
||||
}
|
||||
|
||||
static __always_inline bool system_supports_cnp(void)
|
||||
{
|
||||
return IS_ENABLED(CONFIG_ARM64_CNP) &&
|
||||
|
||||
@@ -143,6 +143,50 @@
|
||||
.Lskip_sve_\@:
|
||||
.endm
|
||||
|
||||
/* SME register access and priority mapping */
|
||||
.macro __init_el2_nvhe_sme
|
||||
mrs x1, id_aa64pfr1_el1
|
||||
ubfx x1, x1, #ID_AA64PFR1_SME_SHIFT, #4
|
||||
cbz x1, .Lskip_sme_\@
|
||||
|
||||
bic x0, x0, #CPTR_EL2_TSM // Also disable SME traps
|
||||
msr cptr_el2, x0 // Disable copro. traps to EL2
|
||||
isb
|
||||
|
||||
mrs x1, sctlr_el2
|
||||
orr x1, x1, #SCTLR_ELx_ENTP2 // Disable TPIDR2 traps
|
||||
msr sctlr_el2, x1
|
||||
isb
|
||||
|
||||
mov x1, #0 // SMCR controls
|
||||
|
||||
mrs_s x2, SYS_ID_AA64SMFR0_EL1
|
||||
ubfx x2, x2, #ID_AA64SMFR0_FA64_SHIFT, #1 // Full FP in SM?
|
||||
cbz x2, .Lskip_sme_fa64_\@
|
||||
|
||||
orr x1, x1, SMCR_ELx_FA64_MASK
|
||||
.Lskip_sme_fa64_\@:
|
||||
|
||||
orr x1, x1, #SMCR_ELx_LEN_MASK // Enable full SME vector
|
||||
msr_s SYS_SMCR_EL2, x1 // length for EL1.
|
||||
|
||||
mrs_s x1, SYS_SMIDR_EL1 // Priority mapping supported?
|
||||
ubfx x1, x1, #SYS_SMIDR_EL1_SMPS_SHIFT, #1
|
||||
cbz x1, .Lskip_sme_\@
|
||||
|
||||
msr_s SYS_SMPRIMAP_EL2, xzr // Make all priorities equal
|
||||
|
||||
mrs x1, id_aa64mmfr1_el1 // HCRX_EL2 present?
|
||||
ubfx x1, x1, #ID_AA64MMFR1_HCX_SHIFT, #4
|
||||
cbz x1, .Lskip_sme_\@
|
||||
|
||||
mrs_s x1, SYS_HCRX_EL2
|
||||
orr x1, x1, #HCRX_EL2_SMPME_MASK // Enable priority mapping
|
||||
msr_s SYS_HCRX_EL2, x1
|
||||
|
||||
.Lskip_sme_\@:
|
||||
.endm
|
||||
|
||||
/* Disable any fine grained traps */
|
||||
.macro __init_el2_fgt
|
||||
mrs x1, id_aa64mmfr0_el1
|
||||
@@ -153,15 +197,26 @@
|
||||
mrs x1, id_aa64dfr0_el1
|
||||
ubfx x1, x1, #ID_AA64DFR0_PMSVER_SHIFT, #4
|
||||
cmp x1, #3
|
||||
b.lt .Lset_fgt_\@
|
||||
b.lt .Lset_debug_fgt_\@
|
||||
/* Disable PMSNEVFR_EL1 read and write traps */
|
||||
orr x0, x0, #(1 << 62)
|
||||
|
||||
.Lset_fgt_\@:
|
||||
.Lset_debug_fgt_\@:
|
||||
msr_s SYS_HDFGRTR_EL2, x0
|
||||
msr_s SYS_HDFGWTR_EL2, x0
|
||||
msr_s SYS_HFGRTR_EL2, xzr
|
||||
msr_s SYS_HFGWTR_EL2, xzr
|
||||
|
||||
mov x0, xzr
|
||||
mrs x1, id_aa64pfr1_el1
|
||||
ubfx x1, x1, #ID_AA64PFR1_SME_SHIFT, #4
|
||||
cbz x1, .Lset_fgt_\@
|
||||
|
||||
/* Disable nVHE traps of TPIDR2 and SMPRI */
|
||||
orr x0, x0, #HFGxTR_EL2_nSMPRI_EL1_MASK
|
||||
orr x0, x0, #HFGxTR_EL2_nTPIDR2_EL0_MASK
|
||||
|
||||
.Lset_fgt_\@:
|
||||
msr_s SYS_HFGRTR_EL2, x0
|
||||
msr_s SYS_HFGWTR_EL2, x0
|
||||
msr_s SYS_HFGITR_EL2, xzr
|
||||
|
||||
mrs x1, id_aa64pfr0_el1 // AMU traps UNDEF without AMU
|
||||
@@ -196,6 +251,7 @@
|
||||
__init_el2_nvhe_idregs
|
||||
__init_el2_nvhe_cptr
|
||||
__init_el2_nvhe_sve
|
||||
__init_el2_nvhe_sme
|
||||
__init_el2_fgt
|
||||
__init_el2_nvhe_prepare_eret
|
||||
.endm
|
||||
|
||||
@@ -37,7 +37,8 @@
|
||||
#define ESR_ELx_EC_ERET (0x1a) /* EL2 only */
|
||||
/* Unallocated EC: 0x1B */
|
||||
#define ESR_ELx_EC_FPAC (0x1C) /* EL1 and above */
|
||||
/* Unallocated EC: 0x1D - 0x1E */
|
||||
#define ESR_ELx_EC_SME (0x1D)
|
||||
/* Unallocated EC: 0x1E */
|
||||
#define ESR_ELx_EC_IMP_DEF (0x1f) /* EL3 only */
|
||||
#define ESR_ELx_EC_IABT_LOW (0x20)
|
||||
#define ESR_ELx_EC_IABT_CUR (0x21)
|
||||
@@ -75,6 +76,7 @@
|
||||
#define ESR_ELx_IL_SHIFT (25)
|
||||
#define ESR_ELx_IL (UL(1) << ESR_ELx_IL_SHIFT)
|
||||
#define ESR_ELx_ISS_MASK (ESR_ELx_IL - 1)
|
||||
#define ESR_ELx_ISS(esr) ((esr) & ESR_ELx_ISS_MASK)
|
||||
|
||||
/* ISS field definitions shared by different classes */
|
||||
#define ESR_ELx_WNR_SHIFT (6)
|
||||
@@ -327,6 +329,15 @@
|
||||
#define ESR_ELx_CP15_32_ISS_SYS_CNTFRQ (ESR_ELx_CP15_32_ISS_SYS_VAL(0, 0, 14, 0) |\
|
||||
ESR_ELx_CP15_32_ISS_DIR_READ)
|
||||
|
||||
/*
|
||||
* ISS values for SME traps
|
||||
*/
|
||||
|
||||
#define ESR_ELx_SME_ISS_SME_DISABLED 0
|
||||
#define ESR_ELx_SME_ISS_ILL 1
|
||||
#define ESR_ELx_SME_ISS_SM_DISABLED 2
|
||||
#define ESR_ELx_SME_ISS_ZA_DISABLED 3
|
||||
|
||||
#ifndef __ASSEMBLY__
|
||||
#include <asm/types.h>
|
||||
|
||||
|
||||
@@ -64,6 +64,7 @@ void do_debug_exception(unsigned long addr_if_watchpoint, unsigned int esr,
|
||||
struct pt_regs *regs);
|
||||
void do_fpsimd_acc(unsigned int esr, struct pt_regs *regs);
|
||||
void do_sve_acc(unsigned int esr, struct pt_regs *regs);
|
||||
void do_sme_acc(unsigned int esr, struct pt_regs *regs);
|
||||
void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs);
|
||||
void do_sysinstr(unsigned int esr, struct pt_regs *regs);
|
||||
void do_sp_pc_abort(unsigned long addr, unsigned int esr, struct pt_regs *regs);
|
||||
|
||||
@@ -46,11 +46,23 @@ extern void fpsimd_restore_current_state(void);
|
||||
extern void fpsimd_update_current_state(struct user_fpsimd_state const *state);
|
||||
|
||||
extern void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *state,
|
||||
void *sve_state, unsigned int sve_vl);
|
||||
void *sve_state, unsigned int sve_vl,
|
||||
void *za_state, unsigned int sme_vl,
|
||||
u64 *svcr);
|
||||
|
||||
extern void fpsimd_flush_task_state(struct task_struct *target);
|
||||
extern void fpsimd_save_and_flush_cpu_state(void);
|
||||
|
||||
static inline bool thread_sm_enabled(struct thread_struct *thread)
|
||||
{
|
||||
return system_supports_sme() && (thread->svcr & SYS_SVCR_EL0_SM_MASK);
|
||||
}
|
||||
|
||||
static inline bool thread_za_enabled(struct thread_struct *thread)
|
||||
{
|
||||
return system_supports_sme() && (thread->svcr & SYS_SVCR_EL0_ZA_MASK);
|
||||
}
|
||||
|
||||
/* Maximum VL that SVE/SME VL-agnostic software can transparently support */
|
||||
#define VL_ARCH_MAX 0x100
|
||||
|
||||
@@ -62,7 +74,14 @@ static inline size_t sve_ffr_offset(int vl)
|
||||
|
||||
static inline void *sve_pffr(struct thread_struct *thread)
|
||||
{
|
||||
return (char *)thread->sve_state + sve_ffr_offset(thread_get_sve_vl(thread));
|
||||
unsigned int vl;
|
||||
|
||||
if (system_supports_sme() && thread_sm_enabled(thread))
|
||||
vl = thread_get_sme_vl(thread);
|
||||
else
|
||||
vl = thread_get_sve_vl(thread);
|
||||
|
||||
return (char *)thread->sve_state + sve_ffr_offset(vl);
|
||||
}
|
||||
|
||||
extern void sve_save_state(void *state, u32 *pfpsr, int save_ffr);
|
||||
@@ -71,11 +90,17 @@ extern void sve_load_state(void const *state, u32 const *pfpsr,
|
||||
extern void sve_flush_live(bool flush_ffr, unsigned long vq_minus_1);
|
||||
extern unsigned int sve_get_vl(void);
|
||||
extern void sve_set_vq(unsigned long vq_minus_1);
|
||||
extern void sme_set_vq(unsigned long vq_minus_1);
|
||||
extern void za_save_state(void *state);
|
||||
extern void za_load_state(void const *state);
|
||||
|
||||
struct arm64_cpu_capabilities;
|
||||
extern void sve_kernel_enable(const struct arm64_cpu_capabilities *__unused);
|
||||
extern void sme_kernel_enable(const struct arm64_cpu_capabilities *__unused);
|
||||
extern void fa64_kernel_enable(const struct arm64_cpu_capabilities *__unused);
|
||||
|
||||
extern u64 read_zcr_features(void);
|
||||
extern u64 read_smcr_features(void);
|
||||
|
||||
/*
|
||||
* Helpers to translate bit indices in sve_vq_map to VQ values (and
|
||||
@@ -119,6 +144,7 @@ struct vl_info {
|
||||
extern void sve_alloc(struct task_struct *task);
|
||||
extern void fpsimd_release_task(struct task_struct *task);
|
||||
extern void fpsimd_sync_to_sve(struct task_struct *task);
|
||||
extern void fpsimd_force_sync_to_sve(struct task_struct *task);
|
||||
extern void sve_sync_to_fpsimd(struct task_struct *task);
|
||||
extern void sve_sync_from_fpsimd_zeropad(struct task_struct *task);
|
||||
|
||||
@@ -170,6 +196,12 @@ static inline void write_vl(enum vec_type type, u64 val)
|
||||
tmp = read_sysreg_s(SYS_ZCR_EL1) & ~ZCR_ELx_LEN_MASK;
|
||||
write_sysreg_s(tmp | val, SYS_ZCR_EL1);
|
||||
break;
|
||||
#endif
|
||||
#ifdef CONFIG_ARM64_SME
|
||||
case ARM64_VEC_SME:
|
||||
tmp = read_sysreg_s(SYS_SMCR_EL1) & ~SMCR_ELx_LEN_MASK;
|
||||
write_sysreg_s(tmp | val, SYS_SMCR_EL1);
|
||||
break;
|
||||
#endif
|
||||
default:
|
||||
WARN_ON_ONCE(1);
|
||||
@@ -208,6 +240,8 @@ static inline bool sve_vq_available(unsigned int vq)
|
||||
return vq_available(ARM64_VEC_SVE, vq);
|
||||
}
|
||||
|
||||
size_t sve_state_size(struct task_struct const *task);
|
||||
|
||||
#else /* ! CONFIG_ARM64_SVE */
|
||||
|
||||
static inline void sve_alloc(struct task_struct *task) { }
|
||||
@@ -247,8 +281,93 @@ static inline void vec_update_vq_map(enum vec_type t) { }
|
||||
static inline int vec_verify_vq_map(enum vec_type t) { return 0; }
|
||||
static inline void sve_setup(void) { }
|
||||
|
||||
static inline size_t sve_state_size(struct task_struct const *task)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
#endif /* ! CONFIG_ARM64_SVE */
|
||||
|
||||
#ifdef CONFIG_ARM64_SME
|
||||
|
||||
static inline void sme_user_disable(void)
|
||||
{
|
||||
sysreg_clear_set(cpacr_el1, CPACR_EL1_SMEN_EL0EN, 0);
|
||||
}
|
||||
|
||||
static inline void sme_user_enable(void)
|
||||
{
|
||||
sysreg_clear_set(cpacr_el1, 0, CPACR_EL1_SMEN_EL0EN);
|
||||
}
|
||||
|
||||
static inline void sme_smstart_sm(void)
|
||||
{
|
||||
asm volatile(__msr_s(SYS_SVCR_SMSTART_SM_EL0, "xzr"));
|
||||
}
|
||||
|
||||
static inline void sme_smstop_sm(void)
|
||||
{
|
||||
asm volatile(__msr_s(SYS_SVCR_SMSTOP_SM_EL0, "xzr"));
|
||||
}
|
||||
|
||||
static inline void sme_smstop(void)
|
||||
{
|
||||
asm volatile(__msr_s(SYS_SVCR_SMSTOP_SMZA_EL0, "xzr"));
|
||||
}
|
||||
|
||||
extern void __init sme_setup(void);
|
||||
|
||||
static inline int sme_max_vl(void)
|
||||
{
|
||||
return vec_max_vl(ARM64_VEC_SME);
|
||||
}
|
||||
|
||||
static inline int sme_max_virtualisable_vl(void)
|
||||
{
|
||||
return vec_max_virtualisable_vl(ARM64_VEC_SME);
|
||||
}
|
||||
|
||||
extern void sme_alloc(struct task_struct *task);
|
||||
extern unsigned int sme_get_vl(void);
|
||||
extern int sme_set_current_vl(unsigned long arg);
|
||||
extern int sme_get_current_vl(void);
|
||||
|
||||
/*
|
||||
* Return how many bytes of memory are required to store the full SME
|
||||
* specific state (currently just ZA) for task, given task's currently
|
||||
* configured vector length.
|
||||
*/
|
||||
static inline size_t za_state_size(struct task_struct const *task)
|
||||
{
|
||||
unsigned int vl = task_get_sme_vl(task);
|
||||
|
||||
return ZA_SIG_REGS_SIZE(sve_vq_from_vl(vl));
|
||||
}
|
||||
|
||||
#else
|
||||
|
||||
static inline void sme_user_disable(void) { BUILD_BUG(); }
|
||||
static inline void sme_user_enable(void) { BUILD_BUG(); }
|
||||
|
||||
static inline void sme_smstart_sm(void) { }
|
||||
static inline void sme_smstop_sm(void) { }
|
||||
static inline void sme_smstop(void) { }
|
||||
|
||||
static inline void sme_alloc(struct task_struct *task) { }
|
||||
static inline void sme_setup(void) { }
|
||||
static inline unsigned int sme_get_vl(void) { return 0; }
|
||||
static inline int sme_max_vl(void) { return 0; }
|
||||
static inline int sme_max_virtualisable_vl(void) { return 0; }
|
||||
static inline int sme_set_current_vl(unsigned long arg) { return -EINVAL; }
|
||||
static inline int sme_get_current_vl(void) { return -EINVAL; }
|
||||
|
||||
static inline size_t za_state_size(struct task_struct const *task)
|
||||
{
|
||||
return 0;
|
||||
}
|
||||
|
||||
#endif /* ! CONFIG_ARM64_SME */
|
||||
|
||||
/* For use by EFI runtime services calls only */
|
||||
extern void __efi_fpsimd_begin(void);
|
||||
extern void __efi_fpsimd_end(void);
|
||||
|
||||
@@ -93,6 +93,12 @@
|
||||
.endif
|
||||
.endm
|
||||
|
||||
.macro _sme_check_wv v
|
||||
.if (\v) < 12 || (\v) > 15
|
||||
.error "Bad vector select register \v."
|
||||
.endif
|
||||
.endm
|
||||
|
||||
/* SVE instruction encodings for non-SVE-capable assemblers */
|
||||
/* (pre binutils 2.28, all kernel capable clang versions support SVE) */
|
||||
|
||||
@@ -174,6 +180,54 @@
|
||||
| (\np)
|
||||
.endm
|
||||
|
||||
/* SME instruction encodings for non-SME-capable assemblers */
|
||||
/* (pre binutils 2.38/LLVM 13) */
|
||||
|
||||
/* RDSVL X\nx, #\imm */
|
||||
.macro _sme_rdsvl nx, imm
|
||||
_check_general_reg \nx
|
||||
_check_num (\imm), -0x20, 0x1f
|
||||
.inst 0x04bf5800 \
|
||||
| (\nx) \
|
||||
| (((\imm) & 0x3f) << 5)
|
||||
.endm
|
||||
|
||||
/*
|
||||
* STR (vector from ZA array):
|
||||
* STR ZA[\nw, #\offset], [X\nxbase, #\offset, MUL VL]
|
||||
*/
|
||||
.macro _sme_str_zav nw, nxbase, offset=0
|
||||
_sme_check_wv \nw
|
||||
_check_general_reg \nxbase
|
||||
_check_num (\offset), -0x100, 0xff
|
||||
.inst 0xe1200000 \
|
||||
| (((\nw) & 3) << 13) \
|
||||
| ((\nxbase) << 5) \
|
||||
| ((\offset) & 7)
|
||||
.endm
|
||||
|
||||
/*
|
||||
* LDR (vector to ZA array):
|
||||
* LDR ZA[\nw, #\offset], [X\nxbase, #\offset, MUL VL]
|
||||
*/
|
||||
.macro _sme_ldr_zav nw, nxbase, offset=0
|
||||
_sme_check_wv \nw
|
||||
_check_general_reg \nxbase
|
||||
_check_num (\offset), -0x100, 0xff
|
||||
.inst 0xe1000000 \
|
||||
| (((\nw) & 3) << 13) \
|
||||
| ((\nxbase) << 5) \
|
||||
| ((\offset) & 7)
|
||||
.endm
|
||||
|
||||
/*
|
||||
* Zero the entire ZA array
|
||||
* ZERO ZA
|
||||
*/
|
||||
.macro zero_za
|
||||
.inst 0xc00800ff
|
||||
.endm
|
||||
|
||||
.macro __for from:req, to:req
|
||||
.if (\from) == (\to)
|
||||
_for__body %\from
|
||||
@@ -208,6 +262,17 @@
|
||||
921:
|
||||
.endm
|
||||
|
||||
/* Update SMCR_EL1.LEN with the new VQ */
|
||||
.macro sme_load_vq xvqminus1, xtmp, xtmp2
|
||||
mrs_s \xtmp, SYS_SMCR_EL1
|
||||
bic \xtmp2, \xtmp, SMCR_ELx_LEN_MASK
|
||||
orr \xtmp2, \xtmp2, \xvqminus1
|
||||
cmp \xtmp2, \xtmp
|
||||
b.eq 921f
|
||||
msr_s SYS_SMCR_EL1, \xtmp2 //self-synchronising
|
||||
921:
|
||||
.endm
|
||||
|
||||
/* Preserve the first 128-bits of Znz and zero the rest. */
|
||||
.macro _sve_flush_z nz
|
||||
_sve_check_zreg \nz
|
||||
@@ -254,3 +319,25 @@
|
||||
ldr w\nxtmp, [\xpfpsr, #4]
|
||||
msr fpcr, x\nxtmp
|
||||
.endm
|
||||
|
||||
.macro sme_save_za nxbase, xvl, nw
|
||||
mov w\nw, #0
|
||||
|
||||
423:
|
||||
_sme_str_zav \nw, \nxbase
|
||||
add x\nxbase, x\nxbase, \xvl
|
||||
add x\nw, x\nw, #1
|
||||
cmp \xvl, x\nw
|
||||
bne 423b
|
||||
.endm
|
||||
|
||||
.macro sme_load_za nxbase, xvl, nw
|
||||
mov w\nw, #0
|
||||
|
||||
423:
|
||||
_sme_ldr_zav \nw, \nxbase
|
||||
add x\nxbase, x\nxbase, \xvl
|
||||
add x\nw, x\nw, #1
|
||||
cmp \xvl, x\nw
|
||||
bne 423b
|
||||
.endm
|
||||
|
||||
@@ -109,6 +109,14 @@
|
||||
#define KERNEL_HWCAP_AFP __khwcap2_feature(AFP)
|
||||
#define KERNEL_HWCAP_RPRES __khwcap2_feature(RPRES)
|
||||
#define KERNEL_HWCAP_MTE3 __khwcap2_feature(MTE3)
|
||||
#define KERNEL_HWCAP_SME __khwcap2_feature(SME)
|
||||
#define KERNEL_HWCAP_SME_I16I64 __khwcap2_feature(SME_I16I64)
|
||||
#define KERNEL_HWCAP_SME_F64F64 __khwcap2_feature(SME_F64F64)
|
||||
#define KERNEL_HWCAP_SME_I8I32 __khwcap2_feature(SME_I8I32)
|
||||
#define KERNEL_HWCAP_SME_F16F32 __khwcap2_feature(SME_F16F32)
|
||||
#define KERNEL_HWCAP_SME_B16F32 __khwcap2_feature(SME_B16F32)
|
||||
#define KERNEL_HWCAP_SME_F32F32 __khwcap2_feature(SME_F32F32)
|
||||
#define KERNEL_HWCAP_SME_FA64 __khwcap2_feature(SME_FA64)
|
||||
|
||||
/*
|
||||
* This yields a mask that user programs can use to figure out what
|
||||
|
||||
@@ -279,6 +279,7 @@
|
||||
#define CPTR_EL2_TCPAC (1U << 31)
|
||||
#define CPTR_EL2_TAM (1 << 30)
|
||||
#define CPTR_EL2_TTA (1 << 20)
|
||||
#define CPTR_EL2_TSM (1 << 12)
|
||||
#define CPTR_EL2_TFP (1 << CPTR_EL2_TFP_SHIFT)
|
||||
#define CPTR_EL2_TZ (1 << 8)
|
||||
#define CPTR_NVHE_EL2_RES1 0x000032ff /* known RES1 bits in CPTR_EL2 (nVHE) */
|
||||
|
||||
@@ -295,8 +295,11 @@ struct vcpu_reset_state {
|
||||
|
||||
struct kvm_vcpu_arch {
|
||||
struct kvm_cpu_context ctxt;
|
||||
|
||||
/* Guest floating point state */
|
||||
void *sve_state;
|
||||
unsigned int sve_max_vl;
|
||||
u64 svcr;
|
||||
|
||||
/* Stage 2 paging state used by the hardware on next switch */
|
||||
struct kvm_s2_mmu *hw_mmu;
|
||||
@@ -451,6 +454,7 @@ struct kvm_vcpu_arch {
|
||||
#define KVM_ARM64_DEBUG_STATE_SAVE_TRBE (1 << 13) /* Save TRBE context if active */
|
||||
#define KVM_ARM64_FP_FOREIGN_FPSTATE (1 << 14)
|
||||
#define KVM_ARM64_ON_UNSUPPORTED_CPU (1 << 15) /* Physical CPU not in supported_cpus */
|
||||
#define KVM_ARM64_HOST_SME_ENABLED (1 << 16) /* SME enabled for EL0 */
|
||||
|
||||
#define KVM_GUESTDBG_VALID_MASK (KVM_GUESTDBG_ENABLE | \
|
||||
KVM_GUESTDBG_USE_SW_BP | \
|
||||
|
||||
@@ -118,6 +118,7 @@ struct debug_info {
|
||||
|
||||
enum vec_type {
|
||||
ARM64_VEC_SVE = 0,
|
||||
ARM64_VEC_SME,
|
||||
ARM64_VEC_MAX,
|
||||
};
|
||||
|
||||
@@ -153,6 +154,7 @@ struct thread_struct {
|
||||
|
||||
unsigned int fpsimd_cpu;
|
||||
void *sve_state; /* SVE registers, if any */
|
||||
void *za_state; /* ZA register, if any */
|
||||
unsigned int vl[ARM64_VEC_MAX]; /* vector length */
|
||||
unsigned int vl_onexec[ARM64_VEC_MAX]; /* vl after next exec */
|
||||
unsigned long fault_address; /* fault info */
|
||||
@@ -168,6 +170,8 @@ struct thread_struct {
|
||||
u64 mte_ctrl;
|
||||
#endif
|
||||
u64 sctlr_user;
|
||||
u64 svcr;
|
||||
u64 tpidr2_el0;
|
||||
};
|
||||
|
||||
static inline unsigned int thread_get_vl(struct thread_struct *thread,
|
||||
@@ -181,6 +185,19 @@ static inline unsigned int thread_get_sve_vl(struct thread_struct *thread)
|
||||
return thread_get_vl(thread, ARM64_VEC_SVE);
|
||||
}
|
||||
|
||||
static inline unsigned int thread_get_sme_vl(struct thread_struct *thread)
|
||||
{
|
||||
return thread_get_vl(thread, ARM64_VEC_SME);
|
||||
}
|
||||
|
||||
static inline unsigned int thread_get_cur_vl(struct thread_struct *thread)
|
||||
{
|
||||
if (system_supports_sme() && (thread->svcr & SYS_SVCR_EL0_SM_MASK))
|
||||
return thread_get_sme_vl(thread);
|
||||
else
|
||||
return thread_get_sve_vl(thread);
|
||||
}
|
||||
|
||||
unsigned int task_get_vl(const struct task_struct *task, enum vec_type type);
|
||||
void task_set_vl(struct task_struct *task, enum vec_type type,
|
||||
unsigned long vl);
|
||||
@@ -194,6 +211,11 @@ static inline unsigned int task_get_sve_vl(const struct task_struct *task)
|
||||
return task_get_vl(task, ARM64_VEC_SVE);
|
||||
}
|
||||
|
||||
static inline unsigned int task_get_sme_vl(const struct task_struct *task)
|
||||
{
|
||||
return task_get_vl(task, ARM64_VEC_SME);
|
||||
}
|
||||
|
||||
static inline void task_set_sve_vl(struct task_struct *task, unsigned long vl)
|
||||
{
|
||||
task_set_vl(task, ARM64_VEC_SVE, vl);
|
||||
@@ -354,9 +376,11 @@ extern void __init minsigstksz_setup(void);
|
||||
*/
|
||||
#include <asm/fpsimd.h>
|
||||
|
||||
/* Userspace interface for PR_SVE_{SET,GET}_VL prctl()s: */
|
||||
/* Userspace interface for PR_S[MV]E_{SET,GET}_VL prctl()s: */
|
||||
#define SVE_SET_VL(arg) sve_set_current_vl(arg)
|
||||
#define SVE_GET_VL() sve_get_current_vl()
|
||||
#define SME_SET_VL(arg) sme_set_current_vl(arg)
|
||||
#define SME_GET_VL() sme_get_current_vl()
|
||||
|
||||
/* PR_PAC_RESET_KEYS prctl */
|
||||
#define PAC_RESET_KEYS(tsk, arg) ptrauth_prctl_reset_keys(tsk, arg)
|
||||
|
||||
@@ -118,6 +118,10 @@
|
||||
* System registers, organised loosely by encoding but grouped together
|
||||
* where the architected name contains an index. e.g. ID_MMFR<n>_EL1.
|
||||
*/
|
||||
#define SYS_SVCR_SMSTOP_SM_EL0 sys_reg(0, 3, 4, 2, 3)
|
||||
#define SYS_SVCR_SMSTART_SM_EL0 sys_reg(0, 3, 4, 3, 3)
|
||||
#define SYS_SVCR_SMSTOP_SMZA_EL0 sys_reg(0, 3, 4, 6, 3)
|
||||
|
||||
#define SYS_OSDTRRX_EL1 sys_reg(2, 0, 0, 0, 2)
|
||||
#define SYS_MDCCINT_EL1 sys_reg(2, 0, 0, 2, 0)
|
||||
#define SYS_MDSCR_EL1 sys_reg(2, 0, 0, 2, 2)
|
||||
@@ -181,6 +185,7 @@
|
||||
#define SYS_ID_AA64PFR0_EL1 sys_reg(3, 0, 0, 4, 0)
|
||||
#define SYS_ID_AA64PFR1_EL1 sys_reg(3, 0, 0, 4, 1)
|
||||
#define SYS_ID_AA64ZFR0_EL1 sys_reg(3, 0, 0, 4, 4)
|
||||
#define SYS_ID_AA64SMFR0_EL1 sys_reg(3, 0, 0, 4, 5)
|
||||
|
||||
#define SYS_ID_AA64DFR0_EL1 sys_reg(3, 0, 0, 5, 0)
|
||||
#define SYS_ID_AA64DFR1_EL1 sys_reg(3, 0, 0, 5, 1)
|
||||
@@ -204,6 +209,8 @@
|
||||
|
||||
#define SYS_ZCR_EL1 sys_reg(3, 0, 1, 2, 0)
|
||||
#define SYS_TRFCR_EL1 sys_reg(3, 0, 1, 2, 1)
|
||||
#define SYS_SMPRI_EL1 sys_reg(3, 0, 1, 2, 4)
|
||||
#define SYS_SMCR_EL1 sys_reg(3, 0, 1, 2, 6)
|
||||
|
||||
#define SYS_TTBR0_EL1 sys_reg(3, 0, 2, 0, 0)
|
||||
#define SYS_TTBR1_EL1 sys_reg(3, 0, 2, 0, 1)
|
||||
@@ -396,6 +403,8 @@
|
||||
#define TRBIDR_ALIGN_MASK GENMASK(3, 0)
|
||||
#define TRBIDR_ALIGN_SHIFT 0
|
||||
|
||||
#define SMPRI_EL1_PRIORITY_MASK 0xf
|
||||
|
||||
#define SYS_PMINTENSET_EL1 sys_reg(3, 0, 9, 14, 1)
|
||||
#define SYS_PMINTENCLR_EL1 sys_reg(3, 0, 9, 14, 2)
|
||||
|
||||
@@ -451,8 +460,13 @@
|
||||
#define SYS_CCSIDR_EL1 sys_reg(3, 1, 0, 0, 0)
|
||||
#define SYS_CLIDR_EL1 sys_reg(3, 1, 0, 0, 1)
|
||||
#define SYS_GMID_EL1 sys_reg(3, 1, 0, 0, 4)
|
||||
#define SYS_SMIDR_EL1 sys_reg(3, 1, 0, 0, 6)
|
||||
#define SYS_AIDR_EL1 sys_reg(3, 1, 0, 0, 7)
|
||||
|
||||
#define SYS_SMIDR_EL1_IMPLEMENTER_SHIFT 24
|
||||
#define SYS_SMIDR_EL1_SMPS_SHIFT 15
|
||||
#define SYS_SMIDR_EL1_AFFINITY_SHIFT 0
|
||||
|
||||
#define SYS_CSSELR_EL1 sys_reg(3, 2, 0, 0, 0)
|
||||
|
||||
#define SYS_CTR_EL0 sys_reg(3, 3, 0, 0, 1)
|
||||
@@ -461,6 +475,10 @@
|
||||
#define SYS_RNDR_EL0 sys_reg(3, 3, 2, 4, 0)
|
||||
#define SYS_RNDRRS_EL0 sys_reg(3, 3, 2, 4, 1)
|
||||
|
||||
#define SYS_SVCR_EL0 sys_reg(3, 3, 4, 2, 2)
|
||||
#define SYS_SVCR_EL0_ZA_MASK 2
|
||||
#define SYS_SVCR_EL0_SM_MASK 1
|
||||
|
||||
#define SYS_PMCR_EL0 sys_reg(3, 3, 9, 12, 0)
|
||||
#define SYS_PMCNTENSET_EL0 sys_reg(3, 3, 9, 12, 1)
|
||||
#define SYS_PMCNTENCLR_EL0 sys_reg(3, 3, 9, 12, 2)
|
||||
@@ -477,6 +495,7 @@
|
||||
|
||||
#define SYS_TPIDR_EL0 sys_reg(3, 3, 13, 0, 2)
|
||||
#define SYS_TPIDRRO_EL0 sys_reg(3, 3, 13, 0, 3)
|
||||
#define SYS_TPIDR2_EL0 sys_reg(3, 3, 13, 0, 5)
|
||||
|
||||
#define SYS_SCXTNUM_EL0 sys_reg(3, 3, 13, 0, 7)
|
||||
|
||||
@@ -546,6 +565,9 @@
|
||||
#define SYS_HFGITR_EL2 sys_reg(3, 4, 1, 1, 6)
|
||||
#define SYS_ZCR_EL2 sys_reg(3, 4, 1, 2, 0)
|
||||
#define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1)
|
||||
#define SYS_HCRX_EL2 sys_reg(3, 4, 1, 2, 2)
|
||||
#define SYS_SMPRIMAP_EL2 sys_reg(3, 4, 1, 2, 5)
|
||||
#define SYS_SMCR_EL2 sys_reg(3, 4, 1, 2, 6)
|
||||
#define SYS_DACR32_EL2 sys_reg(3, 4, 3, 0, 0)
|
||||
#define SYS_HDFGRTR_EL2 sys_reg(3, 4, 3, 1, 4)
|
||||
#define SYS_HDFGWTR_EL2 sys_reg(3, 4, 3, 1, 5)
|
||||
@@ -605,6 +627,7 @@
|
||||
#define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0)
|
||||
#define SYS_CPACR_EL12 sys_reg(3, 5, 1, 0, 2)
|
||||
#define SYS_ZCR_EL12 sys_reg(3, 5, 1, 2, 0)
|
||||
#define SYS_SMCR_EL12 sys_reg(3, 5, 1, 2, 6)
|
||||
#define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0)
|
||||
#define SYS_TTBR1_EL12 sys_reg(3, 5, 2, 0, 1)
|
||||
#define SYS_TCR_EL12 sys_reg(3, 5, 2, 0, 2)
|
||||
@@ -628,6 +651,7 @@
|
||||
#define SYS_CNTV_CVAL_EL02 sys_reg(3, 5, 14, 3, 2)
|
||||
|
||||
/* Common SCTLR_ELx flags. */
|
||||
#define SCTLR_ELx_ENTP2 (BIT(60))
|
||||
#define SCTLR_ELx_DSSBS (BIT(44))
|
||||
#define SCTLR_ELx_ATA (BIT(43))
|
||||
|
||||
@@ -836,6 +860,7 @@
|
||||
#define ID_AA64PFR0_ELx_32BIT_64BIT 0x2
|
||||
|
||||
/* id_aa64pfr1 */
|
||||
#define ID_AA64PFR1_SME_SHIFT 24
|
||||
#define ID_AA64PFR1_MPAMFRAC_SHIFT 16
|
||||
#define ID_AA64PFR1_RASFRAC_SHIFT 12
|
||||
#define ID_AA64PFR1_MTE_SHIFT 8
|
||||
@@ -846,6 +871,7 @@
|
||||
#define ID_AA64PFR1_SSBS_PSTATE_ONLY 1
|
||||
#define ID_AA64PFR1_SSBS_PSTATE_INSNS 2
|
||||
#define ID_AA64PFR1_BT_BTI 0x1
|
||||
#define ID_AA64PFR1_SME 1
|
||||
|
||||
#define ID_AA64PFR1_MTE_NI 0x0
|
||||
#define ID_AA64PFR1_MTE_EL0 0x1
|
||||
@@ -874,6 +900,23 @@
|
||||
#define ID_AA64ZFR0_AES_PMULL 0x2
|
||||
#define ID_AA64ZFR0_SVEVER_SVE2 0x1
|
||||
|
||||
/* id_aa64smfr0 */
|
||||
#define ID_AA64SMFR0_FA64_SHIFT 63
|
||||
#define ID_AA64SMFR0_I16I64_SHIFT 52
|
||||
#define ID_AA64SMFR0_F64F64_SHIFT 48
|
||||
#define ID_AA64SMFR0_I8I32_SHIFT 36
|
||||
#define ID_AA64SMFR0_F16F32_SHIFT 35
|
||||
#define ID_AA64SMFR0_B16F32_SHIFT 34
|
||||
#define ID_AA64SMFR0_F32F32_SHIFT 32
|
||||
|
||||
#define ID_AA64SMFR0_FA64 0x1
|
||||
#define ID_AA64SMFR0_I16I64 0x4
|
||||
#define ID_AA64SMFR0_F64F64 0x1
|
||||
#define ID_AA64SMFR0_I8I32 0x4
|
||||
#define ID_AA64SMFR0_F16F32 0x1
|
||||
#define ID_AA64SMFR0_B16F32 0x1
|
||||
#define ID_AA64SMFR0_F32F32 0x1
|
||||
|
||||
/* id_aa64mmfr0 */
|
||||
#define ID_AA64MMFR0_ECV_SHIFT 60
|
||||
#define ID_AA64MMFR0_FGT_SHIFT 56
|
||||
@@ -926,6 +969,7 @@
|
||||
|
||||
/* id_aa64mmfr1 */
|
||||
#define ID_AA64MMFR1_ECBHB_SHIFT 60
|
||||
#define ID_AA64MMFR1_HCX_SHIFT 40
|
||||
#define ID_AA64MMFR1_AFP_SHIFT 44
|
||||
#define ID_AA64MMFR1_ETS_SHIFT 36
|
||||
#define ID_AA64MMFR1_TWED_SHIFT 32
|
||||
@@ -1119,9 +1163,24 @@
|
||||
#define ZCR_ELx_LEN_SIZE 9
|
||||
#define ZCR_ELx_LEN_MASK 0x1ff
|
||||
|
||||
#define SMCR_ELx_FA64_SHIFT 31
|
||||
#define SMCR_ELx_FA64_MASK (1 << SMCR_ELx_FA64_SHIFT)
|
||||
|
||||
/*
|
||||
* The SMCR_ELx_LEN_* definitions intentionally include bits [8:4] which
|
||||
* are reserved by the SME architecture for future expansion of the LEN
|
||||
* field, with compatible semantics.
|
||||
*/
|
||||
#define SMCR_ELx_LEN_SHIFT 0
|
||||
#define SMCR_ELx_LEN_SIZE 9
|
||||
#define SMCR_ELx_LEN_MASK 0x1ff
|
||||
|
||||
#define CPACR_EL1_FPEN_EL1EN (BIT(20)) /* enable EL1 access */
|
||||
#define CPACR_EL1_FPEN_EL0EN (BIT(21)) /* enable EL0 access, if EL1EN set */
|
||||
|
||||
#define CPACR_EL1_SMEN_EL1EN (BIT(24)) /* enable EL1 access */
|
||||
#define CPACR_EL1_SMEN_EL0EN (BIT(25)) /* enable EL0 access, if EL1EN set */
|
||||
|
||||
#define CPACR_EL1_ZEN_EL1EN (BIT(16)) /* enable EL1 access */
|
||||
#define CPACR_EL1_ZEN_EL0EN (BIT(17)) /* enable EL0 access, if EL1EN set */
|
||||
|
||||
@@ -1170,6 +1229,8 @@
|
||||
#define TRFCR_ELx_ExTRE BIT(1)
|
||||
#define TRFCR_ELx_E0TRE BIT(0)
|
||||
|
||||
/* HCRX_EL2 definitions */
|
||||
#define HCRX_EL2_SMPME_MASK (1 << 5)
|
||||
|
||||
/* GIC Hypervisor interface registers */
|
||||
/* ICH_MISR_EL2 bit definitions */
|
||||
@@ -1233,6 +1294,12 @@
|
||||
#define ICH_VTR_TDS_SHIFT 19
|
||||
#define ICH_VTR_TDS_MASK (1 << ICH_VTR_TDS_SHIFT)
|
||||
|
||||
/* HFG[WR]TR_EL2 bit definitions */
|
||||
#define HFGxTR_EL2_nTPIDR2_EL0_SHIFT 55
|
||||
#define HFGxTR_EL2_nTPIDR2_EL0_MASK BIT_MASK(HFGxTR_EL2_nTPIDR2_EL0_SHIFT)
|
||||
#define HFGxTR_EL2_nSMPRI_EL1_SHIFT 54
|
||||
#define HFGxTR_EL2_nSMPRI_EL1_MASK BIT_MASK(HFGxTR_EL2_nSMPRI_EL1_SHIFT)
|
||||
|
||||
#define ARM64_FEATURE_FIELD_BITS 4
|
||||
|
||||
/* Create a mask for the feature bits of the specified feature. */
|
||||
|
||||
@@ -82,6 +82,8 @@ int arch_dup_task_struct(struct task_struct *dst,
|
||||
#define TIF_SVE_VL_INHERIT 24 /* Inherit SVE vl_onexec across exec */
|
||||
#define TIF_SSBD 25 /* Wants SSB mitigation */
|
||||
#define TIF_TAGGED_ADDR 26 /* Allow tagged user addresses */
|
||||
#define TIF_SME 27 /* SME in use */
|
||||
#define TIF_SME_VL_INHERIT 28 /* Inherit SME vl_onexec across exec */
|
||||
|
||||
#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)
|
||||
#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)
|
||||
|
||||
@@ -79,5 +79,13 @@
|
||||
#define HWCAP2_AFP (1 << 20)
|
||||
#define HWCAP2_RPRES (1 << 21)
|
||||
#define HWCAP2_MTE3 (1 << 22)
|
||||
#define HWCAP2_SME (1 << 23)
|
||||
#define HWCAP2_SME_I16I64 (1 << 24)
|
||||
#define HWCAP2_SME_F64F64 (1 << 25)
|
||||
#define HWCAP2_SME_I8I32 (1 << 26)
|
||||
#define HWCAP2_SME_F16F32 (1 << 27)
|
||||
#define HWCAP2_SME_B16F32 (1 << 28)
|
||||
#define HWCAP2_SME_F32F32 (1 << 29)
|
||||
#define HWCAP2_SME_FA64 (1 << 30)
|
||||
|
||||
#endif /* _UAPI__ASM_HWCAP_H */
|
||||
|
||||
@@ -109,7 +109,7 @@ struct user_hwdebug_state {
|
||||
} dbg_regs[16];
|
||||
};
|
||||
|
||||
/* SVE/FP/SIMD state (NT_ARM_SVE) */
|
||||
/* SVE/FP/SIMD state (NT_ARM_SVE & NT_ARM_SSVE) */
|
||||
|
||||
struct user_sve_header {
|
||||
__u32 size; /* total meaningful regset content in bytes */
|
||||
@@ -220,6 +220,7 @@ struct user_sve_header {
|
||||
(SVE_PT_SVE_PREG_OFFSET(vq, __SVE_NUM_PREGS) - \
|
||||
SVE_PT_SVE_PREGS_OFFSET(vq))
|
||||
|
||||
/* For streaming mode SVE (SSVE) FFR must be read and written as zero */
|
||||
#define SVE_PT_SVE_FFR_OFFSET(vq) \
|
||||
(SVE_PT_REGS_OFFSET + __SVE_FFR_OFFSET(vq))
|
||||
|
||||
@@ -240,10 +241,12 @@ struct user_sve_header {
|
||||
- SVE_PT_SVE_OFFSET + (__SVE_VQ_BYTES - 1)) \
|
||||
/ __SVE_VQ_BYTES * __SVE_VQ_BYTES)
|
||||
|
||||
#define SVE_PT_SIZE(vq, flags) \
|
||||
(((flags) & SVE_PT_REGS_MASK) == SVE_PT_REGS_SVE ? \
|
||||
SVE_PT_SVE_OFFSET + SVE_PT_SVE_SIZE(vq, flags) \
|
||||
: SVE_PT_FPSIMD_OFFSET + SVE_PT_FPSIMD_SIZE(vq, flags))
|
||||
#define SVE_PT_SIZE(vq, flags) \
|
||||
(((flags) & SVE_PT_REGS_MASK) == SVE_PT_REGS_SVE ? \
|
||||
SVE_PT_SVE_OFFSET + SVE_PT_SVE_SIZE(vq, flags) \
|
||||
: ((((flags) & SVE_PT_REGS_MASK) == SVE_PT_REGS_FPSIMD ? \
|
||||
SVE_PT_FPSIMD_OFFSET + SVE_PT_FPSIMD_SIZE(vq, flags) \
|
||||
: SVE_PT_REGS_OFFSET)))
|
||||
|
||||
/* pointer authentication masks (NT_ARM_PAC_MASK) */
|
||||
|
||||
@@ -265,6 +268,62 @@ struct user_pac_generic_keys {
|
||||
__uint128_t apgakey;
|
||||
};
|
||||
|
||||
/* ZA state (NT_ARM_ZA) */
|
||||
|
||||
struct user_za_header {
|
||||
__u32 size; /* total meaningful regset content in bytes */
|
||||
__u32 max_size; /* maxmium possible size for this thread */
|
||||
__u16 vl; /* current vector length */
|
||||
__u16 max_vl; /* maximum possible vector length */
|
||||
__u16 flags;
|
||||
__u16 __reserved;
|
||||
};
|
||||
|
||||
/*
|
||||
* Common ZA_PT_* flags:
|
||||
* These must be kept in sync with prctl interface in <linux/prctl.h>
|
||||
*/
|
||||
#define ZA_PT_VL_INHERIT ((1 << 17) /* PR_SME_VL_INHERIT */ >> 16)
|
||||
#define ZA_PT_VL_ONEXEC ((1 << 18) /* PR_SME_SET_VL_ONEXEC */ >> 16)
|
||||
|
||||
|
||||
/*
|
||||
* The remainder of the ZA state follows struct user_za_header. The
|
||||
* total size of the ZA state (including header) depends on the
|
||||
* metadata in the header: ZA_PT_SIZE(vq, flags) gives the total size
|
||||
* of the state in bytes, including the header.
|
||||
*
|
||||
* Refer to <asm/sigcontext.h> for details of how to pass the correct
|
||||
* "vq" argument to these macros.
|
||||
*/
|
||||
|
||||
/* Offset from the start of struct user_za_header to the register data */
|
||||
#define ZA_PT_ZA_OFFSET \
|
||||
((sizeof(struct user_za_header) + (__SVE_VQ_BYTES - 1)) \
|
||||
/ __SVE_VQ_BYTES * __SVE_VQ_BYTES)
|
||||
|
||||
/*
|
||||
* The payload starts at offset ZA_PT_ZA_OFFSET, and is of size
|
||||
* ZA_PT_ZA_SIZE(vq, flags).
|
||||
*
|
||||
* The ZA array is stored as a sequence of horizontal vectors ZAV of SVL/8
|
||||
* bytes each, starting from vector 0.
|
||||
*
|
||||
* Additional data might be appended in the future.
|
||||
*
|
||||
* The ZA matrix is represented in memory in an endianness-invariant layout
|
||||
* which differs from the layout used for the FPSIMD V-registers on big-endian
|
||||
* systems: see sigcontext.h for more explanation.
|
||||
*/
|
||||
|
||||
#define ZA_PT_ZAV_OFFSET(vq, n) \
|
||||
(ZA_PT_ZA_OFFSET + ((vq * __SVE_VQ_BYTES) * n))
|
||||
|
||||
#define ZA_PT_ZA_SIZE(vq) ((vq * __SVE_VQ_BYTES) * (vq * __SVE_VQ_BYTES))
|
||||
|
||||
#define ZA_PT_SIZE(vq) \
|
||||
(ZA_PT_ZA_OFFSET + ZA_PT_ZA_SIZE(vq))
|
||||
|
||||
#endif /* __ASSEMBLY__ */
|
||||
|
||||
#endif /* _UAPI__ASM_PTRACE_H */
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user