Merge tag 'v6.7-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6

Pull crypto updates from Herbert Xu:
 "API:
   - Add virtual-address based lskcipher interface
   - Optimise ahash/shash performance in light of costly indirect calls
   - Remove ahash alignmask attribute

  Algorithms:
   - Improve AES/XTS performance of 6-way unrolling for ppc
   - Remove some uses of obsolete algorithms (md4, md5, sha1)
   - Add FIPS 202 SHA-3 support in pkcs1pad
   - Add fast path for single-page messages in adiantum
   - Remove zlib-deflate

  Drivers:
   - Add support for S4 in meson RNG driver
   - Add STM32MP13x support in stm32
   - Add hwrng interface support in qcom-rng
   - Add support for deflate algorithm in hisilicon/zip"

* tag 'v6.7-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6: (283 commits)
  crypto: adiantum - flush destination page before unmapping
  crypto: testmgr - move pkcs1pad(rsa,sha3-*) to correct place
  Documentation/module-signing.txt: bring up to date
  module: enable automatic module signing with FIPS 202 SHA-3
  crypto: asymmetric_keys - allow FIPS 202 SHA-3 signatures
  crypto: rsa-pkcs1pad - Add FIPS 202 SHA-3 support
  crypto: FIPS 202 SHA-3 register in hash info for IMA
  x509: Add OIDs for FIPS 202 SHA-3 hash and signatures
  crypto: ahash - optimize performance when wrapping shash
  crypto: ahash - check for shash type instead of not ahash type
  crypto: hash - move "ahash wrapping shash" functions to ahash.c
  crypto: talitos - stop using crypto_ahash::init
  crypto: chelsio - stop using crypto_ahash::init
  crypto: ahash - improve file comment
  crypto: ahash - remove struct ahash_request_priv
  crypto: ahash - remove crypto_ahash_alignmask
  crypto: gcm - stop using alignmask of ahash
  crypto: chacha20poly1305 - stop using alignmask of ahash
  crypto: ccm - stop using alignmask of ahash
  net: ipv6: stop checking crypto_ahash_alignmask
  ...
This commit is contained in:
Linus Torvalds
2023-11-02 16:15:30 -10:00
275 changed files with 10690 additions and 3351 deletions

View File

@@ -1,4 +1,4 @@
What: /sys/kernel/debug/qat_<device>_<BDF>/qat/fw_counters
What: /sys/kernel/debug/qat_<device>_<BDF>/fw_counters
Date: November 2023
KernelVersion: 6.6
Contact: qat-linux@intel.com
@@ -59,3 +59,25 @@ Description: (RO) Read returns the device health status.
The driver does not monitor for Heartbeat. It is left for a user
to poll the status periodically.
What: /sys/kernel/debug/qat_<device>_<BDF>/pm_status
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description: (RO) Read returns power management information specific to the
QAT device.
This attribute is only available for qat_4xxx devices.
What: /sys/kernel/debug/qat_<device>_<BDF>/cnv_errors
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description: (RO) Read returns, for each Acceleration Engine (AE), the number
of errors and the type of the last error detected by the device
when performing verified compression.
Reported counters::
<N>: Number of Compress and Verify (CnV) errors and type
of the last CnV error detected by Acceleration
Engine N.

View File

@@ -29,6 +29,8 @@ Description: (RW) Reports the current configuration of the QAT device.
services
* asym;sym: identical to sym;asym
* dc: the device is configured for running compression services
* dcc: identical to dc but enables the dc chaining feature,
hash then compression. If this is not required chose dc
* sym: the device is configured for running symmetric crypto
services
* asym: the device is configured for running asymmetric crypto
@@ -93,3 +95,49 @@ Description: (RW) This configuration option provides a way to force the device i
0
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat/rp2srv
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description:
(RW) This attribute provides a way for a user to query a
specific ring pair for the type of service that it is currently
configured for.
When written to, the value is cached and used to perform the
read operation. Allowed values are in the range 0 to N-1, where
N is the max number of ring pairs supported by a device. This
can be queried using the attribute qat/num_rps.
A read returns the service associated to the ring pair queried.
The values are:
* dc: the ring pair is configured for running compression services
* sym: the ring pair is configured for running symmetric crypto
services
* asym: the ring pair is configured for running asymmetric crypto
services
Example usage::
# echo 1 > /sys/bus/pci/devices/<BDF>/qat/rp2srv
# cat /sys/bus/pci/devices/<BDF>/qat/rp2srv
sym
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat/num_rps
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description:
(RO) Returns the number of ring pairs that a single device has.
Example usage::
# cat /sys/bus/pci/devices/<BDF>/qat/num_rps
64
This attribute is only available for qat_4xxx devices.

View File

@@ -0,0 +1,41 @@
What: /sys/bus/pci/devices/<BDF>/qat_ras/errors_correctable
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description: (RO) Reports the number of correctable errors detected by the device.
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat_ras/errors_nonfatal
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description: (RO) Reports the number of non fatal errors detected by the device.
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat_ras/errors_fatal
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description: (RO) Reports the number of fatal errors detected by the device.
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat_ras/reset_error_counters
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description: (WO) Write to resets all error counters of a device.
The following example reports how to reset the counters::
# echo 1 > /sys/bus/pci/devices/<BDF>/qat_ras/reset_error_counters
# cat /sys/bus/pci/devices/<BDF>/qat_ras/errors_correctable
0
# cat /sys/bus/pci/devices/<BDF>/qat_ras/errors_nonfatal
0
# cat /sys/bus/pci/devices/<BDF>/qat_ras/errors_fatal
0
This attribute is only available for qat_4xxx devices.

View File

@@ -0,0 +1,226 @@
What: /sys/bus/pci/devices/<BDF>/qat_rl/sla_op
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description:
(WO) This attribute is used to perform an operation on an SLA.
The supported operations are: add, update, rm, rm_all, and get.
Input values must be filled through the associated attribute in
this group before a write to this file.
If the operation completes successfully, the associated
attributes will be updated.
The associated attributes are: cir, pir, srv, rp, and id.
Supported operations:
* add: Creates a new SLA with the provided inputs from user.
* Inputs: cir, pir, srv, and rp
* Output: id
* get: Returns the configuration of the specified SLA in id attribute
* Inputs: id
* Outputs: cir, pir, srv, and rp
* update: Updates the SLA with new values set in the following attributes
* Inputs: id, cir, and pir
* rm: Removes the specified SLA in the id attribute.
* Inputs: id
* rm_all: Removes all the configured SLAs.
* Inputs: None
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat_rl/rp
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description:
(RW) When read, reports the current assigned ring pairs for the
queried SLA.
When wrote to, configures the ring pairs associated to a new SLA.
The value is a 64-bit bit mask and is written/displayed in hex.
Each bit of this mask represents a single ring pair i.e.,
bit 1 == ring pair id 0; bit 3 == ring pair id 2.
Selected ring pairs must to be assigned to a single service,
i.e. the one provided with the srv attribute. The service
assigned to a certain ring pair can be checked by querying
the attribute qat/rp2srv.
The maximum number of ring pairs is 4 per SLA.
Applicability in sla_op:
* WRITE: add operation
* READ: get operation
Example usage::
## Read
# echo 4 > /sys/bus/pci/devices/<BDF>/qat_rl/id
# cat /sys/bus/pci/devices/<BDF>/qat_rl/rp
0x5
## Write
# echo 0x5 > /sys/bus/pci/devices/<BDF>/qat_rl/rp
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat_rl/id
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description:
(RW) If written to, the value is used to retrieve a particular
SLA and operate on it.
This is valid only for the following operations: update, rm,
and get.
A read of this attribute is only guaranteed to have correct data
after creation of an SLA.
Applicability in sla_op:
* WRITE: rm and update operations
* READ: add and get operations
Example usage::
## Read
## Set attributes e.g. cir, pir, srv, etc
# echo "add" > /sys/bus/pci/devices/<BDF>/qat_rl/sla_op
# cat /sys/bus/pci/devices/<BDF>/qat_rl/id
4
## Write
# echo 7 > /sys/bus/pci/devices/<BDF>/qat_rl/id
# echo "get" > /sys/bus/pci/devices/<BDF>/qat_rl/sla_op
# cat /sys/bus/pci/devices/<BDF>/qat_rl/rp
0x5 ## ring pair ID 0 and ring pair ID 2
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat_rl/cir
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description:
(RW) Committed information rate (CIR). Rate guaranteed to be
achieved by a particular SLA. The value is expressed in
permille scale, i.e. 1000 refers to the maximum device
throughput for a selected service.
After sending a "get" to sla_op, this will be populated with the
CIR for that queried SLA.
Write to this file before sending an "add/update" sla_op, to set
the SLA to the specified value.
Applicability in sla_op:
* WRITE: add and update operations
* READ: get operation
Example usage::
## Write
# echo 500 > /sys/bus/pci/devices/<BDF>/qat_rl/cir
# echo "add" /sys/bus/pci/devices/<BDF>/qat_rl/sla_op
## Read
# echo 4 > /sys/bus/pci/devices/<BDF>/qat_rl/id
# echo "get" > /sys/bus/pci/devices/<BDF>/qat_rl/sla_op
# cat /sys/bus/pci/devices/<BDF>/qat_rl/cir
500
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat_rl/pir
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description:
(RW) Peak information rate (PIR). The maximum rate that can be
achieved by that particular SLA. An SLA can reach a value
between CIR and PIR when the device is not fully utilized by
requests from other users (assigned to different SLAs).
After sending a "get" to sla_op, this will be populated with the
PIR for that queried SLA.
Write to this file before sending an "add/update" sla_op, to set
the SLA to the specified value.
Applicability in sla_op:
* WRITE: add and update operations
* READ: get operation
Example usage::
## Write
# echo 750 > /sys/bus/pci/devices/<BDF>/qat_rl/pir
# echo "add" > /sys/bus/pci/devices/<BDF>/qat_rl/sla_op
## Read
# echo 4 > /sys/bus/pci/devices/<BDF>/qat_rl/id
# echo "get" > /sys/bus/pci/devices/<BDF>/qat_rl/sla_op
# cat /sys/bus/pci/devices/<BDF>/qat_rl/pir
750
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat_rl/srv
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description:
(RW) Service (SRV). Represents the service (sym, asym, dc)
associated to an SLA.
Can be written to or queried to set/show the SRV type for an SLA.
The SRV attribute is used to specify the SRV type before adding
an SLA. After an SLA is configured, reports the service
associated to that SLA.
Applicability in sla_op:
* WRITE: add and update operations
* READ: get operation
Example usage::
## Write
# echo "dc" > /sys/bus/pci/devices/<BDF>/qat_rl/srv
# echo "add" > /sys/bus/pci/devices/<BDF>/qat_rl/sla_op
# cat /sys/bus/pci/devices/<BDF>/qat_rl/id
4
## Read
# echo 4 > /sys/bus/pci/devices/<BDF>/qat_rl/id
# echo "get" > /sys/bus/pci/devices/<BDF>/qat_rl/sla_op
# cat /sys/bus/pci/devices/<BDF>/qat_rl/srv
dc
This attribute is only available for qat_4xxx devices.
What: /sys/bus/pci/devices/<BDF>/qat_rl/cap_rem
Date: January 2024
KernelVersion: 6.7
Contact: qat-linux@intel.com
Description:
(RW) This file will return the remaining capability for a
particular service/sla. This is the remaining value that a new
SLA can be set to or a current SLA can be increased with.
Example usage::
# echo "asym" > /sys/bus/pci/devices/<BDF>/qat_rl/cap_rem
# cat /sys/bus/pci/devices/<BDF>/qat_rl/cap_rem
250
# echo 250 > /sys/bus/pci/devices/<BDF>/qat_rl/cir
# echo "add" > /sys/bus/pci/devices/<BDF>/qat_rl/sla_op
# cat /sys/bus/pci/devices/<BDF>/qat_rl/cap_rem
0
This attribute is only available for qat_4xxx devices.

View File

@@ -28,10 +28,10 @@ trusted userspace bits.
This facility uses X.509 ITU-T standard certificates to encode the public keys
involved. The signatures are not themselves encoded in any industrial standard
type. The facility currently only supports the RSA public key encryption
standard (though it is pluggable and permits others to be used). The possible
hash algorithms that can be used are SHA-1, SHA-224, SHA-256, SHA-384, and
SHA-512 (the algorithm is selected by data in the signature).
type. The built-in facility currently only supports the RSA & NIST P-384 ECDSA
public key signing standard (though it is pluggable and permits others to be
used). The possible hash algorithms that can be used are SHA-2 and SHA-3 of
sizes 256, 384, and 512 (the algorithm is selected by data in the signature).
==========================
@@ -81,11 +81,12 @@ This has a number of options available:
sign the modules with:
=============================== ==========================================
``CONFIG_MODULE_SIG_SHA1`` :menuselection:`Sign modules with SHA-1`
``CONFIG_MODULE_SIG_SHA224`` :menuselection:`Sign modules with SHA-224`
``CONFIG_MODULE_SIG_SHA256`` :menuselection:`Sign modules with SHA-256`
``CONFIG_MODULE_SIG_SHA384`` :menuselection:`Sign modules with SHA-384`
``CONFIG_MODULE_SIG_SHA512`` :menuselection:`Sign modules with SHA-512`
``CONFIG_MODULE_SIG_SHA3_256`` :menuselection:`Sign modules with SHA3-256`
``CONFIG_MODULE_SIG_SHA3_384`` :menuselection:`Sign modules with SHA3-384`
``CONFIG_MODULE_SIG_SHA3_512`` :menuselection:`Sign modules with SHA3-512`
=============================== ==========================================
The algorithm selected here will also be built into the kernel (rather
@@ -145,6 +146,10 @@ into vmlinux) using parameters in the::
file (which is also generated if it does not already exist).
One can select between RSA (``MODULE_SIG_KEY_TYPE_RSA``) and ECDSA
(``MODULE_SIG_KEY_TYPE_ECDSA``) to generate either RSA 4k or NIST
P-384 keypair.
It is strongly recommended that you provide your own x509.genkey file.
Most notably, in the x509.genkey file, the req_distinguished_name section

View File

@@ -235,6 +235,4 @@ Specifics Of Asynchronous HASH Transformation
Some of the drivers will want to use the Generic ScatterWalk in case the
implementation needs to be fed separate chunks of the scatterlist which
contains the input data. The buffer containing the resulting hash will
always be properly aligned to .cra_alignmask so there is no need to
worry about this.
contains the input data.

View File

@@ -4,7 +4,7 @@
$id: http://devicetree.org/schemas/crypto/fsl-imx-sahara.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: Freescale SAHARA Cryptographic Accelerator included in some i.MX chips
title: Freescale SAHARA Cryptographic Accelerator
maintainers:
- Steffen Trumtrar <s.trumtrar@pengutronix.de>
@@ -19,19 +19,56 @@ properties:
maxItems: 1
interrupts:
maxItems: 1
items:
- description: SAHARA Interrupt for Host 0
- description: SAHARA Interrupt for Host 1
minItems: 1
clocks:
items:
- description: Sahara IPG clock
- description: Sahara AHB clock
clock-names:
items:
- const: ipg
- const: ahb
required:
- compatible
- reg
- interrupts
- clocks
- clock-names
allOf:
- if:
properties:
compatible:
contains:
enum:
- fsl,imx53-sahara
then:
properties:
interrupts:
minItems: 2
maxItems: 2
else:
properties:
interrupts:
maxItems: 1
additionalProperties: false
examples:
- |
#include <dt-bindings/clock/imx27-clock.h>
crypto@10025000 {
compatible = "fsl,imx27-sahara";
reg = < 0x10025000 0x800>;
reg = <0x10025000 0x800>;
interrupts = <75>;
clocks = <&clks IMX27_CLK_SAHARA_IPG_GATE>,
<&clks IMX27_CLK_SAHARA_AHB_GATE>;
clock-names = "ipg", "ahb";
};

View File

@@ -13,6 +13,7 @@ properties:
compatible:
items:
- enum:
- qcom,sa8775p-inline-crypto-engine
- qcom,sm8450-inline-crypto-engine
- qcom,sm8550-inline-crypto-engine
- const: qcom,inline-crypto-engine

View File

@@ -11,9 +11,17 @@ maintainers:
properties:
compatible:
enum:
- qcom,prng # 8916 etc.
- qcom,prng-ee # 8996 and later using EE
oneOf:
- enum:
- qcom,prng # 8916 etc.
- qcom,prng-ee # 8996 and later using EE
- items:
- enum:
- qcom,sa8775p-trng
- qcom,sc7280-trng
- qcom,sm8450-trng
- qcom,sm8550-trng
- const: qcom,trng
reg:
maxItems: 1
@@ -28,8 +36,18 @@ properties:
required:
- compatible
- reg
- clocks
- clock-names
allOf:
- if:
not:
properties:
compatible:
contains:
const: qcom,trng
then:
required:
- clocks
- clock-names
additionalProperties: false

View File

@@ -14,6 +14,7 @@ properties:
compatible:
enum:
- amlogic,meson-rng
- amlogic,meson-s4-rng
reg:
maxItems: 1

View File

@@ -15,7 +15,9 @@ maintainers:
properties:
compatible:
const: st,stm32-rng
enum:
- st,stm32-rng
- st,stm32mp13-rng
reg:
maxItems: 1
@@ -30,11 +32,27 @@ properties:
type: boolean
description: If set enable the clock detection management
st,rng-lock-conf:
type: boolean
description: If set, the RNG configuration in RNG_CR, RNG_HTCR and
RNG_NSCR will be locked.
required:
- compatible
- reg
- clocks
allOf:
- if:
properties:
compatible:
contains:
enum:
- st,stm32-rng
then:
properties:
st,rng-lock-conf: false
additionalProperties: false
examples:

View File

@@ -908,7 +908,7 @@ F: drivers/crypto/ccp/
F: include/linux/ccp.h
AMD CRYPTOGRAPHIC COPROCESSOR (CCP) DRIVER - SEV SUPPORT
M: Brijesh Singh <brijesh.singh@amd.com>
M: Ashish Kalra <ashish.kalra@amd.com>
M: Tom Lendacky <thomas.lendacky@amd.com>
L: linux-crypto@vger.kernel.org
S: Supported

View File

@@ -34,6 +34,14 @@ static int nhpoly1305_neon_update(struct shash_desc *desc,
return 0;
}
static int nhpoly1305_neon_digest(struct shash_desc *desc,
const u8 *src, unsigned int srclen, u8 *out)
{
return crypto_nhpoly1305_init(desc) ?:
nhpoly1305_neon_update(desc, src, srclen) ?:
crypto_nhpoly1305_final(desc, out);
}
static struct shash_alg nhpoly1305_alg = {
.base.cra_name = "nhpoly1305",
.base.cra_driver_name = "nhpoly1305-neon",
@@ -44,6 +52,7 @@ static struct shash_alg nhpoly1305_alg = {
.init = crypto_nhpoly1305_init,
.update = nhpoly1305_neon_update,
.final = crypto_nhpoly1305_final,
.digest = nhpoly1305_neon_digest,
.setkey = crypto_nhpoly1305_setkey,
.descsize = sizeof(struct nhpoly1305_state),
};

View File

@@ -34,6 +34,14 @@ static int nhpoly1305_neon_update(struct shash_desc *desc,
return 0;
}
static int nhpoly1305_neon_digest(struct shash_desc *desc,
const u8 *src, unsigned int srclen, u8 *out)
{
return crypto_nhpoly1305_init(desc) ?:
nhpoly1305_neon_update(desc, src, srclen) ?:
crypto_nhpoly1305_final(desc, out);
}
static struct shash_alg nhpoly1305_alg = {
.base.cra_name = "nhpoly1305",
.base.cra_driver_name = "nhpoly1305-neon",
@@ -44,6 +52,7 @@ static struct shash_alg nhpoly1305_alg = {
.init = crypto_nhpoly1305_init,
.update = nhpoly1305_neon_update,
.final = crypto_nhpoly1305_final,
.digest = nhpoly1305_neon_digest,
.setkey = crypto_nhpoly1305_setkey,
.descsize = sizeof(struct nhpoly1305_state),
};

View File

@@ -62,10 +62,10 @@
.endm
/*
* int sha1_ce_transform(struct sha1_ce_state *sst, u8 const *src,
* int blocks)
* int __sha1_ce_transform(struct sha1_ce_state *sst, u8 const *src,
* int blocks)
*/
SYM_FUNC_START(sha1_ce_transform)
SYM_FUNC_START(__sha1_ce_transform)
/* load round constants */
loadrc k0.4s, 0x5a827999, w6
loadrc k1.4s, 0x6ed9eba1, w6
@@ -147,4 +147,4 @@ CPU_LE( rev32 v11.16b, v11.16b )
str dgb, [x0, #16]
mov w0, w2
ret
SYM_FUNC_END(sha1_ce_transform)
SYM_FUNC_END(__sha1_ce_transform)

View File

@@ -29,18 +29,19 @@ struct sha1_ce_state {
extern const u32 sha1_ce_offsetof_count;
extern const u32 sha1_ce_offsetof_finalize;
asmlinkage int sha1_ce_transform(struct sha1_ce_state *sst, u8 const *src,
int blocks);
asmlinkage int __sha1_ce_transform(struct sha1_ce_state *sst, u8 const *src,
int blocks);
static void __sha1_ce_transform(struct sha1_state *sst, u8 const *src,
int blocks)
static void sha1_ce_transform(struct sha1_state *sst, u8 const *src,
int blocks)
{
while (blocks) {
int rem;
kernel_neon_begin();
rem = sha1_ce_transform(container_of(sst, struct sha1_ce_state,
sst), src, blocks);
rem = __sha1_ce_transform(container_of(sst,
struct sha1_ce_state,
sst), src, blocks);
kernel_neon_end();
src += (blocks - rem) * SHA1_BLOCK_SIZE;
blocks = rem;
@@ -59,7 +60,7 @@ static int sha1_ce_update(struct shash_desc *desc, const u8 *data,
return crypto_sha1_update(desc, data, len);
sctx->finalize = 0;
sha1_base_do_update(desc, data, len, __sha1_ce_transform);
sha1_base_do_update(desc, data, len, sha1_ce_transform);
return 0;
}
@@ -79,9 +80,9 @@ static int sha1_ce_finup(struct shash_desc *desc, const u8 *data,
*/
sctx->finalize = finalize;
sha1_base_do_update(desc, data, len, __sha1_ce_transform);
sha1_base_do_update(desc, data, len, sha1_ce_transform);
if (!finalize)
sha1_base_do_finalize(desc, __sha1_ce_transform);
sha1_base_do_finalize(desc, sha1_ce_transform);
return sha1_base_finish(desc, out);
}
@@ -93,7 +94,7 @@ static int sha1_ce_final(struct shash_desc *desc, u8 *out)
return crypto_sha1_finup(desc, NULL, 0, out);
sctx->finalize = 0;
sha1_base_do_finalize(desc, __sha1_ce_transform);
sha1_base_do_finalize(desc, sha1_ce_transform);
return sha1_base_finish(desc, out);
}

View File

@@ -71,11 +71,11 @@
.word 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2
/*
* void sha2_ce_transform(struct sha256_ce_state *sst, u8 const *src,
* int blocks)
* int __sha256_ce_transform(struct sha256_ce_state *sst, u8 const *src,
* int blocks)
*/
.text
SYM_FUNC_START(sha2_ce_transform)
SYM_FUNC_START(__sha256_ce_transform)
/* load round constants */
adr_l x8, .Lsha2_rcon
ld1 { v0.4s- v3.4s}, [x8], #64
@@ -154,4 +154,4 @@ CPU_LE( rev32 v19.16b, v19.16b )
3: st1 {dgav.4s, dgbv.4s}, [x0]
mov w0, w2
ret
SYM_FUNC_END(sha2_ce_transform)
SYM_FUNC_END(__sha256_ce_transform)

View File

@@ -30,18 +30,19 @@ struct sha256_ce_state {
extern const u32 sha256_ce_offsetof_count;
extern const u32 sha256_ce_offsetof_finalize;
asmlinkage int sha2_ce_transform(struct sha256_ce_state *sst, u8 const *src,
int blocks);
asmlinkage int __sha256_ce_transform(struct sha256_ce_state *sst, u8 const *src,
int blocks);
static void __sha2_ce_transform(struct sha256_state *sst, u8 const *src,
static void sha256_ce_transform(struct sha256_state *sst, u8 const *src,
int blocks)
{
while (blocks) {
int rem;
kernel_neon_begin();
rem = sha2_ce_transform(container_of(sst, struct sha256_ce_state,
sst), src, blocks);
rem = __sha256_ce_transform(container_of(sst,
struct sha256_ce_state,
sst), src, blocks);
kernel_neon_end();
src += (blocks - rem) * SHA256_BLOCK_SIZE;
blocks = rem;
@@ -55,8 +56,8 @@ const u32 sha256_ce_offsetof_finalize = offsetof(struct sha256_ce_state,
asmlinkage void sha256_block_data_order(u32 *digest, u8 const *src, int blocks);
static void __sha256_block_data_order(struct sha256_state *sst, u8 const *src,
int blocks)
static void sha256_arm64_transform(struct sha256_state *sst, u8 const *src,
int blocks)
{
sha256_block_data_order(sst->state, src, blocks);
}
@@ -68,10 +69,10 @@ static int sha256_ce_update(struct shash_desc *desc, const u8 *data,
if (!crypto_simd_usable())
return sha256_base_do_update(desc, data, len,
__sha256_block_data_order);
sha256_arm64_transform);
sctx->finalize = 0;
sha256_base_do_update(desc, data, len, __sha2_ce_transform);
sha256_base_do_update(desc, data, len, sha256_ce_transform);
return 0;
}
@@ -85,8 +86,8 @@ static int sha256_ce_finup(struct shash_desc *desc, const u8 *data,
if (!crypto_simd_usable()) {
if (len)
sha256_base_do_update(desc, data, len,
__sha256_block_data_order);
sha256_base_do_finalize(desc, __sha256_block_data_order);
sha256_arm64_transform);
sha256_base_do_finalize(desc, sha256_arm64_transform);
return sha256_base_finish(desc, out);
}
@@ -96,9 +97,9 @@ static int sha256_ce_finup(struct shash_desc *desc, const u8 *data,
*/
sctx->finalize = finalize;
sha256_base_do_update(desc, data, len, __sha2_ce_transform);
sha256_base_do_update(desc, data, len, sha256_ce_transform);
if (!finalize)
sha256_base_do_finalize(desc, __sha2_ce_transform);
sha256_base_do_finalize(desc, sha256_ce_transform);
return sha256_base_finish(desc, out);
}
@@ -107,15 +108,22 @@ static int sha256_ce_final(struct shash_desc *desc, u8 *out)
struct sha256_ce_state *sctx = shash_desc_ctx(desc);
if (!crypto_simd_usable()) {
sha256_base_do_finalize(desc, __sha256_block_data_order);
sha256_base_do_finalize(desc, sha256_arm64_transform);
return sha256_base_finish(desc, out);
}
sctx->finalize = 0;
sha256_base_do_finalize(desc, __sha2_ce_transform);
sha256_base_do_finalize(desc, sha256_ce_transform);
return sha256_base_finish(desc, out);
}
static int sha256_ce_digest(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
sha256_base_init(desc);
return sha256_ce_finup(desc, data, len, out);
}
static int sha256_ce_export(struct shash_desc *desc, void *out)
{
struct sha256_ce_state *sctx = shash_desc_ctx(desc);
@@ -155,6 +163,7 @@ static struct shash_alg algs[] = { {
.update = sha256_ce_update,
.final = sha256_ce_final,
.finup = sha256_ce_finup,
.digest = sha256_ce_digest,
.export = sha256_ce_export,
.import = sha256_ce_import,
.descsize = sizeof(struct sha256_ce_state),

View File

@@ -27,8 +27,8 @@ asmlinkage void sha256_block_data_order(u32 *digest, const void *data,
unsigned int num_blks);
EXPORT_SYMBOL(sha256_block_data_order);
static void __sha256_block_data_order(struct sha256_state *sst, u8 const *src,
int blocks)
static void sha256_arm64_transform(struct sha256_state *sst, u8 const *src,
int blocks)
{
sha256_block_data_order(sst->state, src, blocks);
}
@@ -36,8 +36,8 @@ static void __sha256_block_data_order(struct sha256_state *sst, u8 const *src,
asmlinkage void sha256_block_neon(u32 *digest, const void *data,
unsigned int num_blks);
static void __sha256_block_neon(struct sha256_state *sst, u8 const *src,
int blocks)
static void sha256_neon_transform(struct sha256_state *sst, u8 const *src,
int blocks)
{
sha256_block_neon(sst->state, src, blocks);
}
@@ -45,17 +45,15 @@ static void __sha256_block_neon(struct sha256_state *sst, u8 const *src,
static int crypto_sha256_arm64_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
return sha256_base_do_update(desc, data, len,
__sha256_block_data_order);
return sha256_base_do_update(desc, data, len, sha256_arm64_transform);
}
static int crypto_sha256_arm64_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
if (len)
sha256_base_do_update(desc, data, len,
__sha256_block_data_order);
sha256_base_do_finalize(desc, __sha256_block_data_order);
sha256_base_do_update(desc, data, len, sha256_arm64_transform);
sha256_base_do_finalize(desc, sha256_arm64_transform);
return sha256_base_finish(desc, out);
}
@@ -98,7 +96,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data,
if (!crypto_simd_usable())
return sha256_base_do_update(desc, data, len,
__sha256_block_data_order);
sha256_arm64_transform);
while (len > 0) {
unsigned int chunk = len;
@@ -114,7 +112,7 @@ static int sha256_update_neon(struct shash_desc *desc, const u8 *data,
sctx->count % SHA256_BLOCK_SIZE;
kernel_neon_begin();
sha256_base_do_update(desc, data, chunk, __sha256_block_neon);
sha256_base_do_update(desc, data, chunk, sha256_neon_transform);
kernel_neon_end();
data += chunk;
len -= chunk;
@@ -128,13 +126,13 @@ static int sha256_finup_neon(struct shash_desc *desc, const u8 *data,
if (!crypto_simd_usable()) {
if (len)
sha256_base_do_update(desc, data, len,
__sha256_block_data_order);
sha256_base_do_finalize(desc, __sha256_block_data_order);
sha256_arm64_transform);
sha256_base_do_finalize(desc, sha256_arm64_transform);
} else {
if (len)
sha256_update_neon(desc, data, len);
kernel_neon_begin();
sha256_base_do_finalize(desc, __sha256_block_neon);
sha256_base_do_finalize(desc, sha256_neon_transform);
kernel_neon_end();
}
return sha256_base_finish(desc, out);

View File

@@ -102,11 +102,11 @@
.endm
/*
* void sha512_ce_transform(struct sha512_state *sst, u8 const *src,
* int blocks)
* int __sha512_ce_transform(struct sha512_state *sst, u8 const *src,
* int blocks)
*/
.text
SYM_FUNC_START(sha512_ce_transform)
SYM_FUNC_START(__sha512_ce_transform)
/* load state */
ld1 {v8.2d-v11.2d}, [x0]
@@ -203,4 +203,4 @@ CPU_LE( rev64 v19.16b, v19.16b )
3: st1 {v8.2d-v11.2d}, [x0]
mov w0, w2
ret
SYM_FUNC_END(sha512_ce_transform)
SYM_FUNC_END(__sha512_ce_transform)

Some files were not shown because too many files have changed in this diff Show More