apfs/linux-apfs
0
0
Fork 0
mirror of https://github.com/linux-apfs/linux-apfs.git synced 2026-05-01 15:00:59 -07:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'next' of git://git.infradead.org/users/vkoul/slave-dma

Browse Source 
Pull slave-dmaengine changes from Vinod Koul:
 "This brings for slave dmaengine:

   - Change dma notification flag to DMA_COMPLETE from DMA_SUCCESS as
     dmaengine can only transfer and not verify validaty of dma
     transfers

   - Bunch of fixes across drivers:

      - cppi41 driver fixes from Daniel

      - 8 channel freescale dma engine support and updated bindings from
        Hongbo

      - msx-dma fixes and cleanup by Markus

   - DMAengine updates from Dan:

      - Bartlomiej and Dan finalized a rework of the dma address unmap
        implementation.

      - In the course of testing 1/ a collection of enhancements to
        dmatest fell out.  Notably basic performance statistics, and
        fixed / enhanced test control through new module parameters
        'run', 'wait', 'noverify', and 'verbose'.  Thanks to Andriy and
        Linus [Walleij] for their review.

      - Testing the raid related corner cases of 1/ triggered bugs in
        the recently added 16-source operation support in the ioatdma
        driver.

      - Some minor fixes / cleanups to mv_xor and ioatdma"

* 'next' of git://git.infradead.org/users/vkoul/slave-dma: (99 commits)
  dma: mv_xor: Fix mis-usage of mmio 'base' and 'high_base' registers
  dma: mv_xor: Remove unneeded NULL address check
  ioat: fix ioat3_irq_reinit
  ioat: kill msix_single_vector support
  raid6test: add new corner case for ioatdma driver
  ioatdma: clean up sed pool kmem_cache
  ioatdma: fix selection of 16 vs 8 source path
  ioatdma: fix sed pool selection
  ioatdma: Fix bug in selftest after removal of DMA_MEMSET.
  dmatest: verbose mode
  dmatest: convert to dmaengine_unmap_data
  dmatest: add a 'wait' parameter
  dmatest: add basic performance metrics
  dmatest: add support for skipping verification and random data setup
  dmatest: use pseudo random numbers
  dmatest: support xor-only, or pq-only channels in tests
  dmatest: restore ability to start test at module load and init
  dmatest: cleanup redundant "dmatest: " prefixes
  dmatest: replace stored results mechanism, with uniform messages
  Revert "dmatest: append verify result to results"
  ...
This commit is contained in:
Linus Torvalds
2013-11-20 13:20:24 -08:00
parent 5a1efc6e68 df12a3178d
commit e6d69a60b7
71 changed files with 1994 additions and 2287 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/devicetree/bindings/dma/atmel-dma.txt
+1 -1
View File
@@ -28,7 +28,7 @@ The three cells in order are:
dependent: dependent:
- bit 7-0: peripheral identifier for the hardware handshaking interface. The - bit 7-0: peripheral identifier for the hardware handshaking interface. The
identifier can be different for tx and rx. identifier can be different for tx and rx.
- bit 11-8: FIFO configuration. 0 for half FIFO, 1 for ALAP, 1 for ASAP. - bit 11-8: FIFO configuration. 0 for half FIFO, 1 for ALAP, 2 for ASAP.
Example: Example:
Documentation/devicetree/bindings/powerpc/fsl/dma.txt
+101 -37
View File
@@ -1,33 +1,30 @@
* Freescale 83xx DMA Controller * Freescale DMA Controllers
Freescale PowerPC 83xx have on chip general purpose DMA controllers. ** Freescale Elo DMA Controller
This is a little-endian 4-channel DMA controller, used in Freescale mpc83xx
series chips such as mpc8315, mpc8349, mpc8379 etc.
Required properties: Required properties:
- compatible : compatible list, contains 2 entries, first is - compatible : must include "fsl,elo-dma"
"fsl,CHIP-dma", where CHIP is the processor - reg : DMA General Status Register, i.e. DGSR which contains
(mpc8349, mpc8360, etc.) and the second is status for all the 4 DMA channels
"fsl,elo-dma" - ranges : describes the mapping between the address space of the
- reg : <registers mapping for DMA general status reg> DMA channels and the address space of the DMA controller
- ranges : Should be defined as specified in 1) to describe the
DMA controller channels.
- cell-index : controller index. 0 for controller @ 0x8100 - cell-index : controller index. 0 for controller @ 0x8100
- interrupts : <interrupt mapping for DMA IRQ> - interrupts : interrupt specifier for DMA IRQ
- interrupt-parent : optional, if needed for interrupt mapping - interrupt-parent : optional, if needed for interrupt mapping
- DMA channel nodes: - DMA channel nodes:
- compatible : compatible list, contains 2 entries, first is - compatible : must include "fsl,elo-dma-channel"
"fsl,CHIP-dma-channel", where CHIP is the processor However, see note below.
(mpc8349, mpc8350, etc.) and the second is - reg : DMA channel specific registers
"fsl,elo-dma-channel". However, see note below. - cell-index : DMA channel index starts at 0.
- reg : <registers mapping for channel>
- cell-index : dma channel index starts at 0.
Optional properties: Optional properties:
- interrupts : <interrupt mapping for DMA channel IRQ> - interrupts : interrupt specifier for DMA channel IRQ
(on 83xx this is expected to be identical to (on 83xx this is expected to be identical to
the interrupts property of the parent node) the interrupts property of the parent node)
- interrupt-parent : optional, if needed for interrupt mapping - interrupt-parent : optional, if needed for interrupt mapping
Example: Example:
@@ -70,30 +67,27 @@ Example:
}; };
}; };
* Freescale 85xx/86xx DMA Controller ** Freescale EloPlus DMA Controller
This is a 4-channel DMA controller with extended addresses and chaining,
Freescale PowerPC 85xx/86xx have on chip general purpose DMA controllers. mainly used in Freescale mpc85xx/86xx, Pxxx and BSC series chips, such as
mpc8540, mpc8641 p4080, bsc9131 etc.
Required properties: Required properties:
- compatible : compatible list, contains 2 entries, first is - compatible : must include "fsl,eloplus-dma"
"fsl,CHIP-dma", where CHIP is the processor - reg : DMA General Status Register, i.e. DGSR which contains
(mpc8540, mpc8540, etc.) and the second is status for all the 4 DMA channels
"fsl,eloplus-dma"
- reg : <registers mapping for DMA general status reg>
- cell-index : controller index. 0 for controller @ 0x21000, - cell-index : controller index. 0 for controller @ 0x21000,
1 for controller @ 0xc000 1 for controller @ 0xc000
- ranges : Should be defined as specified in 1) to describe the - ranges : describes the mapping between the address space of the
DMA controller channels. DMA channels and the address space of the DMA controller
- DMA channel nodes: - DMA channel nodes:
- compatible : compatible list, contains 2 entries, first is - compatible : must include "fsl,eloplus-dma-channel"
"fsl,CHIP-dma-channel", where CHIP is the processor However, see note below.
(mpc8540, mpc8560, etc.) and the second is - cell-index : DMA channel index starts at 0.
"fsl,eloplus-dma-channel". However, see note below. - reg : DMA channel specific registers
- cell-index : dma channel index starts at 0. - interrupts : interrupt specifier for DMA channel IRQ
- reg : <registers mapping for channel>
- interrupts : <interrupt mapping for DMA channel IRQ>
- interrupt-parent : optional, if needed for interrupt mapping - interrupt-parent : optional, if needed for interrupt mapping
Example: Example:
@@ -134,6 +128,76 @@ Example:
}; };
}; };
** Freescale Elo3 DMA Controller
DMA controller which has same function as EloPlus except that Elo3 has 8
channels while EloPlus has only 4, it is used in Freescale Txxx and Bxxx
series chips, such as t1040, t4240, b4860.
Required properties:
- compatible : must include "fsl,elo3-dma"
- reg : contains two entries for DMA General Status Registers,
i.e. DGSR0 which includes status for channel 1~4, and
DGSR1 for channel 5~8
- ranges : describes the mapping between the address space of the
DMA channels and the address space of the DMA controller
- DMA channel nodes:
- compatible : must include "fsl,eloplus-dma-channel"
- reg : DMA channel specific registers
- interrupts : interrupt specifier for DMA channel IRQ
- interrupt-parent : optional, if needed for interrupt mapping
Example:
dma@100300 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "fsl,elo3-dma";
reg = <0x100300 0x4>,
<0x100600 0x4>;
ranges = <0x0 0x100100 0x500>;
dma-channel@0 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x0 0x80>;
interrupts = <28 2 0 0>;
};
dma-channel@80 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x80 0x80>;
interrupts = <29 2 0 0>;
};
dma-channel@100 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x100 0x80>;
interrupts = <30 2 0 0>;
};
dma-channel@180 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x180 0x80>;
interrupts = <31 2 0 0>;
};
dma-channel@300 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x300 0x80>;
interrupts = <76 2 0 0>;
};
dma-channel@380 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x380 0x80>;
interrupts = <77 2 0 0>;
};
dma-channel@400 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x400 0x80>;
interrupts = <78 2 0 0>;
};
dma-channel@480 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x480 0x80>;
interrupts = <79 2 0 0>;
};
};
Note on DMA channel compatible properties: The compatible property must say Note on DMA channel compatible properties: The compatible property must say
"fsl,elo-dma-channel" or "fsl,eloplus-dma-channel" to be used by the Elo DMA "fsl,elo-dma-channel" or "fsl,eloplus-dma-channel" to be used by the Elo DMA
driver (fsldma). Any DMA channel used by fsldma cannot be used by another driver (fsldma). Any DMA channel used by fsldma cannot be used by another
Documentation/dmatest.txt
+39 -29
View File
@@ -15,39 +15,48 @@ be built as module or inside kernel. Let's consider those cases.
Part 2 - When dmatest is built as a module... Part 2 - When dmatest is built as a module...
After mounting debugfs and loading the module, the /sys/kernel/debug/dmatest
folder with nodes will be created. There are two important files located. First
is the 'run' node that controls run and stop phases of the test, and the second
one, 'results', is used to get the test case results.
Note that in this case test will not run on load automatically.
Example of usage: Example of usage:
% modprobe dmatest channel=dma0chan0 timeout=2000 iterations=1 run=1
...or:
% modprobe dmatest
% echo dma0chan0 > /sys/module/dmatest/parameters/channel % echo dma0chan0 > /sys/module/dmatest/parameters/channel
% echo 2000 > /sys/module/dmatest/parameters/timeout % echo 2000 > /sys/module/dmatest/parameters/timeout
% echo 1 > /sys/module/dmatest/parameters/iterations % echo 1 > /sys/module/dmatest/parameters/iterations
% echo 1 > /sys/kernel/debug/dmatest/run % echo 1 > /sys/module/dmatest/parameters/run
...or on the kernel command line:
dmatest.channel=dma0chan0 dmatest.timeout=2000 dmatest.iterations=1 dmatest.run=1
Hint: available channel list could be extracted by running the following Hint: available channel list could be extracted by running the following
command: command:
% ls -1 /sys/class/dma/ % ls -1 /sys/class/dma/
After a while you will start to get messages about current status or error like Once started a message like "dmatest: Started 1 threads using dma0chan0" is
in the original code. emitted. After that only test failure messages are reported until the test
stops.
Note that running a new test will not stop any in progress test. Note that running a new test will not stop any in progress test.
The following command should return actual state of the test. The following command returns the state of the test.
% cat /sys/kernel/debug/dmatest/run % cat /sys/module/dmatest/parameters/run
To wait for test done the user may perform a busy loop that checks the state. To wait for test completion userpace can poll 'run' until it is false, or use
the wait parameter. Specifying 'wait=1' when loading the module causes module
initialization to pause until a test run has completed, while reading
/sys/module/dmatest/parameters/wait waits for any running test to complete
before returning. For example, the following scripts wait for 42 tests
to complete before exiting. Note that if 'iterations' is set to 'infinite' then
waiting is disabled.
% while [ $(cat /sys/kernel/debug/dmatest/run) = "Y" ] Example:
> do % modprobe dmatest run=1 iterations=42 wait=1
> echo -n "." % modprobe -r dmatest
> sleep 1 ...or:
> done % modprobe dmatest run=1 iterations=42
> echo % cat /sys/module/dmatest/parameters/wait
% modprobe -r dmatest
Part 3 - When built-in in the kernel... Part 3 - When built-in in the kernel...
@@ -62,21 +71,22 @@ case. You always could check them at run-time by running
Part 4 - Gathering the test results Part 4 - Gathering the test results
The module provides a storage for the test results in the memory. The gathered Test results are printed to the kernel log buffer with the format:
data could be used after test is done.
The special file 'results' in the debugfs represents gathered data of the in "dmatest: result <channel>: <test id>: '<error msg>' with src_off=<val> dst_off=<val> len=<val> (<err code>)"
progress test. The messages collected are printed to the kernel log as well.
Example of output: Example of output:
% cat /sys/kernel/debug/dmatest/results % dmesg | tail -n 1
dma0chan0-copy0: #1: No errors with src_off=0x7bf dst_off=0x8ad len=0x3fea (0) dmatest: result dma0chan0-copy0: #1: No errors with src_off=0x7bf dst_off=0x8ad len=0x3fea (0)
The message format is unified across the different types of errors. A number in The message format is unified across the different types of errors. A number in
the parens represents additional information, e.g. error code, error counter, the parens represents additional information, e.g. error code, error counter,
or status. or status. A test thread also emits a summary line at completion listing the
number of tests executed, number that failed, and a result code.
Comparison between buffers is stored to the dedicated structure. Example:
% dmesg | tail -n 1
dmatest: dma0chan0-copy0: summary 1 test, 0 failures 1000 iops 100000 KB/s (0)
Note that the verify result is now accessible only via file 'results' in the The details of a data miscompare error are also emitted, but do not follow the
debugfs. above format.
arch/arm/common/edma.c
+2 -2
View File
@@ -404,7 +404,7 @@ static irqreturn_t dma_irq_handler(int irq, void *data)
BIT(slot)); BIT(slot));
if (edma_cc[ctlr]->intr_data[channel].callback) if (edma_cc[ctlr]->intr_data[channel].callback)
edma_cc[ctlr]->intr_data[channel].callback( edma_cc[ctlr]->intr_data[channel].callback(
channel, DMA_COMPLETE, channel, EDMA_DMA_COMPLETE,
edma_cc[ctlr]->intr_data[channel].data); edma_cc[ctlr]->intr_data[channel].data);
} }
} while (sh_ipr); } while (sh_ipr);
@@ -459,7 +459,7 @@ static irqreturn_t dma_ccerr_handler(int irq, void *data)
callback) { callback) {
edma_cc[ctlr]->intr_data[k]. edma_cc[ctlr]->intr_data[k].
callback(k, callback(k,
DMA_CC_ERROR, EDMA_DMA_CC_ERROR,
edma_cc[ctlr]->intr_data edma_cc[ctlr]->intr_data
[k].data); [k].data);
} }
arch/arm/include/asm/hardware/iop3xx-adma.h
-30
View File
@@ -393,36 +393,6 @@ static inline int iop_chan_zero_sum_slot_count(size_t len, int src_cnt,
return slot_cnt; return slot_cnt;
} }
static inline int iop_desc_is_pq(struct iop_adma_desc_slot *desc)
{
return 0;
}
static inline u32 iop_desc_get_dest_addr(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan)
{
union iop3xx_desc hw_desc = { .ptr = desc->hw_desc, };
switch (chan->device->id) {
case DMA0_ID:
case DMA1_ID:
return hw_desc.dma->dest_addr;
case AAU_ID:
return hw_desc.aau->dest_addr;
default:
BUG();
}
return 0;
}
static inline u32 iop_desc_get_qdest_addr(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan)
{
BUG();
return 0;
}
static inline u32 iop_desc_get_byte_count(struct iop_adma_desc_slot *desc, static inline u32 iop_desc_get_byte_count(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan) struct iop_adma_chan *chan)
{ {
arch/arm/include/asm/hardware/iop_adma.h
-4
View File
@@ -82,8 +82,6 @@ struct iop_adma_chan {
* @slot_cnt: total slots used in an transaction (group of operations) * @slot_cnt: total slots used in an transaction (group of operations)
* @slots_per_op: number of slots per operation * @slots_per_op: number of slots per operation
* @idx: pool index * @idx: pool index
* @unmap_src_cnt: number of xor sources
* @unmap_len: transaction bytecount
* @tx_list: list of descriptors that are associated with one operation * @tx_list: list of descriptors that are associated with one operation
* @async_tx: support for the async_tx api * @async_tx: support for the async_tx api
* @group_list: list of slots that make up a multi-descriptor transaction * @group_list: list of slots that make up a multi-descriptor transaction
@@ -99,8 +97,6 @@ struct iop_adma_desc_slot {
u16 slot_cnt; u16 slot_cnt;
u16 slots_per_op; u16 slots_per_op;
u16 idx; u16 idx;
u16 unmap_src_cnt;
size_t unmap_len;
struct list_head tx_list; struct list_head tx_list;
struct dma_async_tx_descriptor async_tx; struct dma_async_tx_descriptor async_tx;
union { union {
arch/arm/mach-iop13xx/include/mach/adma.h
-26
View File
@@ -218,20 +218,6 @@ iop_chan_xor_slot_count(size_t len, int src_cnt, int *slots_per_op)
#define iop_chan_pq_slot_count iop_chan_xor_slot_count #define iop_chan_pq_slot_count iop_chan_xor_slot_count
#define iop_chan_pq_zero_sum_slot_count iop_chan_xor_slot_count #define iop_chan_pq_zero_sum_slot_count iop_chan_xor_slot_count
static inline u32 iop_desc_get_dest_addr(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan)
{
struct iop13xx_adma_desc_hw *hw_desc = desc->hw_desc;
return hw_desc->dest_addr;
}
static inline u32 iop_desc_get_qdest_addr(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan)
{
struct iop13xx_adma_desc_hw *hw_desc = desc->hw_desc;
return hw_desc->q_dest_addr;
}
static inline u32 iop_desc_get_byte_count(struct iop_adma_desc_slot *desc, static inline u32 iop_desc_get_byte_count(struct iop_adma_desc_slot *desc,
struct iop_adma_chan *chan) struct iop_adma_chan *chan)
{ {
@@ -350,18 +336,6 @@ iop_desc_init_pq(struct iop_adma_desc_slot *desc, int src_cnt,
hw_desc->desc_ctrl = u_desc_ctrl.value; hw_desc->desc_ctrl = u_desc_ctrl.value;
} }
static inline int iop_desc_is_pq(struct iop_adma_desc_slot *desc)
{
struct iop13xx_adma_desc_hw *hw_desc = desc->hw_desc;
union {
u32 value;
struct iop13xx_adma_desc_ctrl field;
} u_desc_ctrl;
u_desc_ctrl.value = hw_desc->desc_ctrl;
return u_desc_ctrl.field.pq_xfer_en;
}
static inline void static inline void
iop_desc_init_pq_zero_sum(struct iop_adma_desc_slot *desc, int src_cnt, iop_desc_init_pq_zero_sum(struct iop_adma_desc_slot *desc, int src_cnt,
unsigned long flags) unsigned long flags)
arch/powerpc/boot/dts/fsl/b4si-post.dtsi
+2 -2
View File
@@ -223,13 +223,13 @@
reg = <0xe2000 0x1000>; reg = <0xe2000 0x1000>;
}; };
/include/ "qoriq-dma-0.dtsi" /include/ "elo3-dma-0.dtsi"
dma@100300 { dma@100300 {
fsl,iommu-parent = <&pamu0>; fsl,iommu-parent = <&pamu0>;
fsl,liodn-reg = <&guts 0x580>; /* DMA1LIODNR */ fsl,liodn-reg = <&guts 0x580>; /* DMA1LIODNR */
}; };
/include/ "qoriq-dma-1.dtsi" /include/ "elo3-dma-1.dtsi"
dma@101300 { dma@101300 {
fsl,iommu-parent = <&pamu0>; fsl,iommu-parent = <&pamu0>;
fsl,liodn-reg = <&guts 0x584>; /* DMA2LIODNR */ fsl,liodn-reg = <&guts 0x584>; /* DMA2LIODNR */
arch/powerpc/boot/dts/fsl/elo3-dma-0.dtsi
+82
View File
@@ -0,0 +1,82 @@
/*
* QorIQ Elo3 DMA device tree stub [ controller @ offset 0x100000 ]
*
* Copyright 2013 Freescale Semiconductor Inc.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Freescale Semiconductor nor the
* names of its contributors may be used to endorse or promote products
* derived from this software without specific prior written permission.
*
*
* ALTERNATIVELY, this software may be distributed under the terms of the
* GNU General Public License ("GPL") as published by the Free Software
* Foundation, either version 2 of that License or (at your option) any
* later version.
*
* THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
dma0: dma@100300 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "fsl,elo3-dma";
reg = <0x100300 0x4>,
<0x100600 0x4>;
ranges = <0x0 0x100100 0x500>;
dma-channel@0 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x0 0x80>;
interrupts = <28 2 0 0>;
};
dma-channel@80 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x80 0x80>;
interrupts = <29 2 0 0>;
};
dma-channel@100 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x100 0x80>;
interrupts = <30 2 0 0>;
};
dma-channel@180 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x180 0x80>;
interrupts = <31 2 0 0>;
};
dma-channel@300 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x300 0x80>;
interrupts = <76 2 0 0>;
};
dma-channel@380 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x380 0x80>;
interrupts = <77 2 0 0>;
};
dma-channel@400 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x400 0x80>;
interrupts = <78 2 0 0>;
};
dma-channel@480 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x480 0x80>;
interrupts = <79 2 0 0>;
};
};
arch/powerpc/boot/dts/fsl/elo3-dma-1.dtsi
+82
View File
@@ -0,0 +1,82 @@
/*
* QorIQ Elo3 DMA device tree stub [ controller @ offset 0x101000 ]
*
* Copyright 2013 Freescale Semiconductor Inc.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Freescale Semiconductor nor the
* names of its contributors may be used to endorse or promote products
* derived from this software without specific prior written permission.
*
*
* ALTERNATIVELY, this software may be distributed under the terms of the
* GNU General Public License ("GPL") as published by the Free Software
* Foundation, either version 2 of that License or (at your option) any
* later version.
*
* THIS SOFTWARE IS PROVIDED BY Freescale Semiconductor ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL Freescale Semiconductor BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
dma1: dma@101300 {
#address-cells = <1>;
#size-cells = <1>;
compatible = "fsl,elo3-dma";
reg = <0x101300 0x4>,
<0x101600 0x4>;
ranges = <0x0 0x101100 0x500>;
dma-channel@0 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x0 0x80>;
interrupts = <32 2 0 0>;
};
dma-channel@80 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x80 0x80>;
interrupts = <33 2 0 0>;
};
dma-channel@100 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x100 0x80>;
interrupts = <34 2 0 0>;
};
dma-channel@180 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x180 0x80>;
interrupts = <35 2 0 0>;
};
dma-channel@300 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x300 0x80>;
interrupts = <80 2 0 0>;
};
dma-channel@380 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x380 0x80>;
interrupts = <81 2 0 0>;
};
dma-channel@400 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x400 0x80>;
interrupts = <82 2 0 0>;
};
dma-channel@480 {
compatible = "fsl,eloplus-dma-channel";
reg = <0x480 0x80>;
interrupts = <83 2 0 0>;
};
};
arch/powerpc/boot/dts/fsl/t4240si-post.dtsi
+2 -2
View File
@@ -387,8 +387,8 @@
reg = <0xea000 0x4000>; reg = <0xea000 0x4000>;
}; };
/include/ "qoriq-dma-0.dtsi" /include/ "elo3-dma-0.dtsi"
/include/ "qoriq-dma-1.dtsi" /include/ "elo3-dma-1.dtsi"
/include/ "qoriq-espi-0.dtsi" /include/ "qoriq-espi-0.dtsi"
spi@110000 { spi@110000 {
crypto/async_tx/async_memcpy.c
+19 -14
View File
@@ -50,33 +50,36 @@ async_memcpy(struct page *dest, struct page *src, unsigned int dest_offset,
&dest, 1, &src, 1, len); &dest, 1, &src, 1, len);
struct dma_device *device = chan ? chan->device : NULL; struct dma_device *device = chan ? chan->device : NULL;
struct dma_async_tx_descriptor *tx = NULL; struct dma_async_tx_descriptor *tx = NULL;
struct dmaengine_unmap_data *unmap = NULL;
if (device && is_dma_copy_aligned(device, src_offset, dest_offset, len)) { if (device)
dma_addr_t dma_dest, dma_src; unmap = dmaengine_get_unmap_data(device->dev, 2, GFP_NOIO);
if (unmap && is_dma_copy_aligned(device, src_offset, dest_offset, len)) {
unsigned long dma_prep_flags = 0; unsigned long dma_prep_flags = 0;
if (submit->cb_fn) if (submit->cb_fn)
dma_prep_flags |= DMA_PREP_INTERRUPT; dma_prep_flags |= DMA_PREP_INTERRUPT;
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_prep_flags |= DMA_PREP_FENCE; dma_prep_flags |= DMA_PREP_FENCE;
dma_dest = dma_map_page(device->dev, dest, dest_offset, len,
DMA_FROM_DEVICE);
dma_src = dma_map_page(device->dev, src, src_offset, len, unmap->to_cnt = 1;
DMA_TO_DEVICE); unmap->addr[0] = dma_map_page(device->dev, src, src_offset, len,
DMA_TO_DEVICE);
unmap->from_cnt = 1;
unmap->addr[1] = dma_map_page(device->dev, dest, dest_offset, len,
DMA_FROM_DEVICE);
unmap->len = len;
tx = device->device_prep_dma_memcpy(chan, dma_dest, dma_src, tx = device->device_prep_dma_memcpy(chan, unmap->addr[1],
len, dma_prep_flags); unmap->addr[0], len,
if (!tx) { dma_prep_flags);
dma_unmap_page(device->dev, dma_dest, len,
DMA_FROM_DEVICE);
dma_unmap_page(device->dev, dma_src, len,
DMA_TO_DEVICE);
}
} }
if (tx) { if (tx) {
pr_debug("%s: (async) len: %zu\n", __func__, len); pr_debug("%s: (async) len: %zu\n", __func__, len);
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
} else { } else {
void *dest_buf, *src_buf; void *dest_buf, *src_buf;
@@ -96,6 +99,8 @@ async_memcpy(struct page *dest, struct page *src, unsigned int dest_offset,
async_tx_sync_epilog(submit); async_tx_sync_epilog(submit);
} }
dmaengine_unmap_put(unmap);
return tx; return tx;
} }
EXPORT_SYMBOL_GPL(async_memcpy); EXPORT_SYMBOL_GPL(async_memcpy);
crypto/async_tx/async_pq.c
+99 -75
View File
@@ -46,49 +46,24 @@ static struct page *pq_scribble_page;
* do_async_gen_syndrome - asynchronously calculate P and/or Q * do_async_gen_syndrome - asynchronously calculate P and/or Q
*/ */
static __async_inline struct dma_async_tx_descriptor * static __async_inline struct dma_async_tx_descriptor *
do_async_gen_syndrome(struct dma_chan *chan, struct page **blocks, do_async_gen_syndrome(struct dma_chan *chan,
const unsigned char *scfs, unsigned int offset, int disks, const unsigned char *scfs, int disks,
size_t len, dma_addr_t *dma_src, struct dmaengine_unmap_data *unmap,
enum dma_ctrl_flags dma_flags,
struct async_submit_ctl *submit) struct async_submit_ctl *submit)
{ {
struct dma_async_tx_descriptor *tx = NULL; struct dma_async_tx_descriptor *tx = NULL;
struct dma_device *dma = chan->device; struct dma_device *dma = chan->device;
enum dma_ctrl_flags dma_flags = 0;
enum async_tx_flags flags_orig = submit->flags; enum async_tx_flags flags_orig = submit->flags;
dma_async_tx_callback cb_fn_orig = submit->cb_fn; dma_async_tx_callback cb_fn_orig = submit->cb_fn;
dma_async_tx_callback cb_param_orig = submit->cb_param; dma_async_tx_callback cb_param_orig = submit->cb_param;
int src_cnt = disks - 2; int src_cnt = disks - 2;
unsigned char coefs[src_cnt];
unsigned short pq_src_cnt; unsigned short pq_src_cnt;
dma_addr_t dma_dest[2]; dma_addr_t dma_dest[2];
int src_off = 0; int src_off = 0;
int idx;
int i;
/* DMAs use destinations as sources, so use BIDIRECTIONAL mapping */ if (submit->flags & ASYNC_TX_FENCE)
if (P(blocks, disks)) dma_flags |= DMA_PREP_FENCE;
dma_dest[0] = dma_map_page(dma->dev, P(blocks, disks), offset,
len, DMA_BIDIRECTIONAL);
else
dma_flags |= DMA_PREP_PQ_DISABLE_P;
if (Q(blocks, disks))
dma_dest[1] = dma_map_page(dma->dev, Q(blocks, disks), offset,
len, DMA_BIDIRECTIONAL);
else
dma_flags |= DMA_PREP_PQ_DISABLE_Q;
/* convert source addresses being careful to collapse 'empty'
* sources and update the coefficients accordingly
*/
for (i = 0, idx = 0; i < src_cnt; i++) {
if (blocks[i] == NULL)
continue;
dma_src[idx] = dma_map_page(dma->dev, blocks[i], offset, len,
DMA_TO_DEVICE);
coefs[idx] = scfs[i];
idx++;
}
src_cnt = idx;
while (src_cnt > 0) { while (src_cnt > 0) {
submit->flags = flags_orig; submit->flags = flags_orig;
@@ -100,28 +75,25 @@ do_async_gen_syndrome(struct dma_chan *chan, struct page **blocks,
if (src_cnt > pq_src_cnt) { if (src_cnt > pq_src_cnt) {
submit->flags &= ~ASYNC_TX_ACK; submit->flags &= ~ASYNC_TX_ACK;
submit->flags |= ASYNC_TX_FENCE; submit->flags |= ASYNC_TX_FENCE;
dma_flags |= DMA_COMPL_SKIP_DEST_UNMAP;
submit->cb_fn = NULL; submit->cb_fn = NULL;
submit->cb_param = NULL; submit->cb_param = NULL;
} else { } else {
dma_flags &= ~DMA_COMPL_SKIP_DEST_UNMAP;
submit->cb_fn = cb_fn_orig; submit->cb_fn = cb_fn_orig;
submit->cb_param = cb_param_orig; submit->cb_param = cb_param_orig;
if (cb_fn_orig) if (cb_fn_orig)
dma_flags |= DMA_PREP_INTERRUPT; dma_flags |= DMA_PREP_INTERRUPT;
} }
if (submit->flags & ASYNC_TX_FENCE)
dma_flags |= DMA_PREP_FENCE;
/* Since we have clobbered the src_list we are committed /* Drivers force forward progress in case they can not provide
* to doing this asynchronously. Drivers force forward * a descriptor
* progress in case they can not provide a descriptor
*/ */
for (;;) { for (;;) {
dma_dest[0] = unmap->addr[disks - 2];
dma_dest[1] = unmap->addr[disks - 1];
tx = dma->device_prep_dma_pq(chan, dma_dest, tx = dma->device_prep_dma_pq(chan, dma_dest,
&dma_src[src_off], &unmap->addr[src_off],
pq_src_cnt, pq_src_cnt,
&coefs[src_off], len, &scfs[src_off], unmap->len,
dma_flags); dma_flags);
if (likely(tx)) if (likely(tx))
break; break;
@@ -129,6 +101,7 @@ do_async_gen_syndrome(struct dma_chan *chan, struct page **blocks,
dma_async_issue_pending(chan); dma_async_issue_pending(chan);
} }
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
submit->depend_tx = tx; submit->depend_tx = tx;
@@ -188,10 +161,6 @@ do_sync_gen_syndrome(struct page **blocks, unsigned int offset, int disks,
* set to NULL those buffers will be replaced with the raid6_zero_page * set to NULL those buffers will be replaced with the raid6_zero_page
* in the synchronous path and omitted in the hardware-asynchronous * in the synchronous path and omitted in the hardware-asynchronous
* path. * path.
*
* 'blocks' note: if submit->scribble is NULL then the contents of
* 'blocks' may be overwritten to perform address conversions
* (dma_map_page() or page_address()).
*/ */
struct dma_async_tx_descriptor * struct dma_async_tx_descriptor *
async_gen_syndrome(struct page **blocks, unsigned int offset, int disks, async_gen_syndrome(struct page **blocks, unsigned int offset, int disks,
@@ -202,26 +171,69 @@ async_gen_syndrome(struct page **blocks, unsigned int offset, int disks,
&P(blocks, disks), 2, &P(blocks, disks), 2,
blocks, src_cnt, len); blocks, src_cnt, len);
struct dma_device *device = chan ? chan->device : NULL; struct dma_device *device = chan ? chan->device : NULL;
dma_addr_t *dma_src = NULL; struct dmaengine_unmap_data *unmap = NULL;
BUG_ON(disks > 255 || !(P(blocks, disks) || Q(blocks, disks))); BUG_ON(disks > 255 || !(P(blocks, disks) || Q(blocks, disks)));
if (submit->scribble) if (device)
dma_src = submit->scribble; unmap = dmaengine_get_unmap_data(device->dev, disks, GFP_NOIO);
else if (sizeof(dma_addr_t) <= sizeof(struct page *))
dma_src = (dma_addr_t *) blocks;
if (dma_src && device && if (unmap &&
(src_cnt <= dma_maxpq(device, 0) || (src_cnt <= dma_maxpq(device, 0) ||
dma_maxpq(device, DMA_PREP_CONTINUE) > 0) && dma_maxpq(device, DMA_PREP_CONTINUE) > 0) &&
is_dma_pq_aligned(device, offset, 0, len)) { is_dma_pq_aligned(device, offset, 0, len)) {
struct dma_async_tx_descriptor *tx;
enum dma_ctrl_flags dma_flags = 0;
unsigned char coefs[src_cnt];
int i, j;
/* run the p+q asynchronously */ /* run the p+q asynchronously */
pr_debug("%s: (async) disks: %d len: %zu\n", pr_debug("%s: (async) disks: %d len: %zu\n",
__func__, disks, len); __func__, disks, len);
return do_async_gen_syndrome(chan, blocks, raid6_gfexp, offset,
disks, len, dma_src, submit); /* convert source addresses being careful to collapse 'empty'
* sources and update the coefficients accordingly
*/
unmap->len = len;
for (i = 0, j = 0; i < src_cnt; i++) {
if (blocks[i] == NULL)
continue;
unmap->addr[j] = dma_map_page(device->dev, blocks[i], offset,
len, DMA_TO_DEVICE);
coefs[j] = raid6_gfexp[i];
unmap->to_cnt++;
j++;
}
/*
* DMAs use destinations as sources,
* so use BIDIRECTIONAL mapping
*/
unmap->bidi_cnt++;
if (P(blocks, disks))
unmap->addr[j++] = dma_map_page(device->dev, P(blocks, disks),
offset, len, DMA_BIDIRECTIONAL);
else {
unmap->addr[j++] = 0;
dma_flags |= DMA_PREP_PQ_DISABLE_P;
}
unmap->bidi_cnt++;
if (Q(blocks, disks))
unmap->addr[j++] = dma_map_page(device->dev, Q(blocks, disks),
offset, len, DMA_BIDIRECTIONAL);
else {
unmap->addr[j++] = 0;
dma_flags |= DMA_PREP_PQ_DISABLE_Q;
}
tx = do_async_gen_syndrome(chan, coefs, j, unmap, dma_flags, submit);
dmaengine_unmap_put(unmap);
return tx;
} }
dmaengine_unmap_put(unmap);
/* run the pq synchronously */ /* run the pq synchronously */
pr_debug("%s: (sync) disks: %d len: %zu\n", __func__, disks, len); pr_debug("%s: (sync) disks: %d len: %zu\n", __func__, disks, len);
@@ -277,50 +289,60 @@ async_syndrome_val(struct page **blocks, unsigned int offset, int disks,
struct dma_async_tx_descriptor *tx; struct dma_async_tx_descriptor *tx;
unsigned char coefs[disks-2]; unsigned char coefs[disks-2];
enum dma_ctrl_flags dma_flags = submit->cb_fn ? DMA_PREP_INTERRUPT : 0; enum dma_ctrl_flags dma_flags = submit->cb_fn ? DMA_PREP_INTERRUPT : 0;
dma_addr_t *dma_src = NULL; struct dmaengine_unmap_data *unmap = NULL;
int src_cnt = 0;
BUG_ON(disks < 4); BUG_ON(disks < 4);
if (submit->scribble) if (device)
dma_src = submit->scribble; unmap = dmaengine_get_unmap_data(device->dev, disks, GFP_NOIO);
else if (sizeof(dma_addr_t) <= sizeof(struct page *))
dma_src = (dma_addr_t *) blocks;
if (dma_src && device && disks <= dma_maxpq(device, 0) && if (unmap && disks <= dma_maxpq(device, 0) &&
is_dma_pq_aligned(device, offset, 0, len)) { is_dma_pq_aligned(device, offset, 0, len)) {
struct device *dev = device->dev; struct device *dev = device->dev;
dma_addr_t *pq = &dma_src[disks-2]; dma_addr_t pq[2];
int i; int i, j = 0, src_cnt = 0;
pr_debug("%s: (async) disks: %d len: %zu\n", pr_debug("%s: (async) disks: %d len: %zu\n",
__func__, disks, len); __func__, disks, len);
if (!P(blocks, disks))
unmap->len = len;
for (i = 0; i < disks-2; i++)
if (likely(blocks[i])) {
unmap->addr[j] = dma_map_page(dev, blocks[i],
offset, len,
DMA_TO_DEVICE);
coefs[j] = raid6_gfexp[i];
unmap->to_cnt++;
src_cnt++;
j++;
}
if (!P(blocks, disks)) {
pq[0] = 0;
dma_flags |= DMA_PREP_PQ_DISABLE_P; dma_flags |= DMA_PREP_PQ_DISABLE_P;
else } else {
pq[0] = dma_map_page(dev, P(blocks, disks), pq[0] = dma_map_page(dev, P(blocks, disks),
offset, len, offset, len,
DMA_TO_DEVICE); DMA_TO_DEVICE);
if (!Q(blocks, disks)) unmap->addr[j++] = pq[0];
unmap->to_cnt++;
}
if (!Q(blocks, disks)) {
pq[1] = 0;
dma_flags |= DMA_PREP_PQ_DISABLE_Q; dma_flags |= DMA_PREP_PQ_DISABLE_Q;
else } else {
pq[1] = dma_map_page(dev, Q(blocks, disks), pq[1] = dma_map_page(dev, Q(blocks, disks),
offset, len, offset, len,
DMA_TO_DEVICE); DMA_TO_DEVICE);
unmap->addr[j++] = pq[1];
unmap->to_cnt++;
}
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_flags |= DMA_PREP_FENCE; dma_flags |= DMA_PREP_FENCE;
for (i = 0; i < disks-2; i++)
if (likely(blocks[i])) {
dma_src[src_cnt] = dma_map_page(dev, blocks[i],
offset, len,
DMA_TO_DEVICE);
coefs[src_cnt] = raid6_gfexp[i];
src_cnt++;
}
for (;;) { for (;;) {
tx = device->device_prep_dma_pq_val(chan, pq, dma_src, tx = device->device_prep_dma_pq_val(chan, pq,
unmap->addr,
src_cnt, src_cnt,
coefs, coefs,
len, pqres, len, pqres,
@@ -330,6 +352,8 @@ async_syndrome_val(struct page **blocks, unsigned int offset, int disks,
async_tx_quiesce(&submit->depend_tx); async_tx_quiesce(&submit->depend_tx);
dma_async_issue_pending(chan); dma_async_issue_pending(chan);
} }
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
return tx; return tx;
crypto/async_tx/async_raid6_recov.c
+43 -18
View File
@@ -26,6 +26,7 @@
#include <linux/dma-mapping.h> #include <linux/dma-mapping.h>
#include <linux/raid/pq.h> #include <linux/raid/pq.h>
#include <linux/async_tx.h> #include <linux/async_tx.h>
#include <linux/dmaengine.h>
static struct dma_async_tx_descriptor * static struct dma_async_tx_descriptor *
async_sum_product(struct page *dest, struct page **srcs, unsigned char *coef, async_sum_product(struct page *dest, struct page **srcs, unsigned char *coef,
@@ -34,35 +35,45 @@ async_sum_product(struct page *dest, struct page **srcs, unsigned char *coef,
struct dma_chan *chan = async_tx_find_channel(submit, DMA_PQ, struct dma_chan *chan = async_tx_find_channel(submit, DMA_PQ,
&dest, 1, srcs, 2, len); &dest, 1, srcs, 2, len);
struct dma_device *dma = chan ? chan->device : NULL; struct dma_device *dma = chan ? chan->device : NULL;
struct dmaengine_unmap_data *unmap = NULL;
const u8 *amul, *bmul; const u8 *amul, *bmul;
u8 ax, bx; u8 ax, bx;
u8 *a, *b, *c; u8 *a, *b, *c;
if (dma) { if (dma)
dma_addr_t dma_dest[2]; unmap = dmaengine_get_unmap_data(dma->dev, 3, GFP_NOIO);
dma_addr_t dma_src[2];
if (unmap) {
struct device *dev = dma->dev; struct device *dev = dma->dev;
dma_addr_t pq[2];
struct dma_async_tx_descriptor *tx; struct dma_async_tx_descriptor *tx;
enum dma_ctrl_flags dma_flags = DMA_PREP_PQ_DISABLE_P; enum dma_ctrl_flags dma_flags = DMA_PREP_PQ_DISABLE_P;
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_flags |= DMA_PREP_FENCE; dma_flags |= DMA_PREP_FENCE;
dma_dest[1] = dma_map_page(dev, dest, 0, len, DMA_BIDIRECTIONAL); unmap->addr[0] = dma_map_page(dev, srcs[0], 0, len, DMA_TO_DEVICE);
dma_src[0] = dma_map_page(dev, srcs[0], 0, len, DMA_TO_DEVICE); unmap->addr[1] = dma_map_page(dev, srcs[1], 0, len, DMA_TO_DEVICE);
dma_src[1] = dma_map_page(dev, srcs[1], 0, len, DMA_TO_DEVICE); unmap->to_cnt = 2;
tx = dma->device_prep_dma_pq(chan, dma_dest, dma_src, 2, coef,
unmap->addr[2] = dma_map_page(dev, dest, 0, len, DMA_BIDIRECTIONAL);
unmap->bidi_cnt = 1;
/* engine only looks at Q, but expects it to follow P */
pq[1] = unmap->addr[2];
unmap->len = len;
tx = dma->device_prep_dma_pq(chan, pq, unmap->addr, 2, coef,
len, dma_flags); len, dma_flags);
if (tx) { if (tx) {
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
dmaengine_unmap_put(unmap);
return tx; return tx;
} }
/* could not get a descriptor, unmap and fall through to /* could not get a descriptor, unmap and fall through to
* the synchronous path * the synchronous path
*/ */
dma_unmap_page(dev, dma_dest[1], len, DMA_BIDIRECTIONAL); dmaengine_unmap_put(unmap);
dma_unmap_page(dev, dma_src[0], len, DMA_TO_DEVICE);
dma_unmap_page(dev, dma_src[1], len, DMA_TO_DEVICE);
} }
/* run the operation synchronously */ /* run the operation synchronously */
@@ -89,23 +100,38 @@ async_mult(struct page *dest, struct page *src, u8 coef, size_t len,
struct dma_chan *chan = async_tx_find_channel(submit, DMA_PQ, struct dma_chan *chan = async_tx_find_channel(submit, DMA_PQ,
&dest, 1, &src, 1, len); &dest, 1, &src, 1, len);
struct dma_device *dma = chan ? chan->device : NULL; struct dma_device *dma = chan ? chan->device : NULL;
struct dmaengine_unmap_data *unmap = NULL;
const u8 *qmul; /* Q multiplier table */ const u8 *qmul; /* Q multiplier table */
u8 *d, *s; u8 *d, *s;
if (dma) { if (dma)
unmap = dmaengine_get_unmap_data(dma->dev, 3, GFP_NOIO);
if (unmap) {
dma_addr_t dma_dest[2]; dma_addr_t dma_dest[2];
dma_addr_t dma_src[1];
struct device *dev = dma->dev; struct device *dev = dma->dev;
struct dma_async_tx_descriptor *tx; struct dma_async_tx_descriptor *tx;
enum dma_ctrl_flags dma_flags = DMA_PREP_PQ_DISABLE_P; enum dma_ctrl_flags dma_flags = DMA_PREP_PQ_DISABLE_P;
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_flags |= DMA_PREP_FENCE; dma_flags |= DMA_PREP_FENCE;
dma_dest[1] = dma_map_page(dev, dest, 0, len, DMA_BIDIRECTIONAL); unmap->addr[0] = dma_map_page(dev, src, 0, len, DMA_TO_DEVICE);
dma_src[0] = dma_map_page(dev, src, 0, len, DMA_TO_DEVICE); unmap->to_cnt++;
tx = dma->device_prep_dma_pq(chan, dma_dest, dma_src, 1, &coef, unmap->addr[1] = dma_map_page(dev, dest, 0, len, DMA_BIDIRECTIONAL);
len, dma_flags); dma_dest[1] = unmap->addr[1];
unmap->bidi_cnt++;
unmap->len = len;
/* this looks funny, but the engine looks for Q at
* dma_dest[1] and ignores dma_dest[0] as a dest
* due to DMA_PREP_PQ_DISABLE_P
*/
tx = dma->device_prep_dma_pq(chan, dma_dest, unmap->addr,
1, &coef, len, dma_flags);
if (tx) { if (tx) {
dma_set_unmap(tx, unmap);
dmaengine_unmap_put(unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
return tx; return tx;
} }
@@ -113,8 +139,7 @@ async_mult(struct page *dest, struct page *src, u8 coef, size_t len,
/* could not get a descriptor, unmap and fall through to /* could not get a descriptor, unmap and fall through to
* the synchronous path * the synchronous path
*/ */
dma_unmap_page(dev, dma_dest[1], len, DMA_BIDIRECTIONAL); dmaengine_unmap_put(unmap);
dma_unmap_page(dev, dma_src[0], len, DMA_TO_DEVICE);
} }
/* no channel available, or failed to allocate a descriptor, so /* no channel available, or failed to allocate a descriptor, so
crypto/async_tx/async_tx.c
+2 -2
View File
@@ -128,7 +128,7 @@ async_tx_channel_switch(struct dma_async_tx_descriptor *depend_tx,
} }
device->device_issue_pending(chan); device->device_issue_pending(chan);
} else { } else {
if (dma_wait_for_async_tx(depend_tx) != DMA_SUCCESS) if (dma_wait_for_async_tx(depend_tx) != DMA_COMPLETE)
panic("%s: DMA error waiting for depend_tx\n", panic("%s: DMA error waiting for depend_tx\n",
__func__); __func__);
tx->tx_submit(tx); tx->tx_submit(tx);
@@ -280,7 +280,7 @@ void async_tx_quiesce(struct dma_async_tx_descriptor **tx)
* we are referring to the correct operation * we are referring to the correct operation
*/ */
BUG_ON(async_tx_test_ack(*tx)); BUG_ON(async_tx_test_ack(*tx));
if (dma_wait_for_async_tx(*tx) != DMA_SUCCESS) if (dma_wait_for_async_tx(*tx) != DMA_COMPLETE)
panic("%s: DMA error waiting for transaction\n", panic("%s: DMA error waiting for transaction\n",
__func__); __func__);
async_tx_ack(*tx); async_tx_ack(*tx);
crypto/async_tx/async_xor.c
+66 -57
View File
@@ -33,48 +33,31 @@
/* do_async_xor - dma map the pages and perform the xor with an engine */ /* do_async_xor - dma map the pages and perform the xor with an engine */
static __async_inline struct dma_async_tx_descriptor * static __async_inline struct dma_async_tx_descriptor *
do_async_xor(struct dma_chan *chan, struct page *dest, struct page **src_list, do_async_xor(struct dma_chan *chan, struct dmaengine_unmap_data *unmap,
unsigned int offset, int src_cnt, size_t len, dma_addr_t *dma_src,
struct async_submit_ctl *submit) struct async_submit_ctl *submit)
{ {
struct dma_device *dma = chan->device; struct dma_device *dma = chan->device;
struct dma_async_tx_descriptor *tx = NULL; struct dma_async_tx_descriptor *tx = NULL;
int src_off = 0;
int i;
dma_async_tx_callback cb_fn_orig = submit->cb_fn; dma_async_tx_callback cb_fn_orig = submit->cb_fn;
void *cb_param_orig = submit->cb_param; void *cb_param_orig = submit->cb_param;
enum async_tx_flags flags_orig = submit->flags; enum async_tx_flags flags_orig = submit->flags;
enum dma_ctrl_flags dma_flags; enum dma_ctrl_flags dma_flags = 0;
int xor_src_cnt = 0; int src_cnt = unmap->to_cnt;
dma_addr_t dma_dest; int xor_src_cnt;
dma_addr_t dma_dest = unmap->addr[unmap->to_cnt];
/* map the dest bidrectional in case it is re-used as a source */ dma_addr_t *src_list = unmap->addr;
dma_dest = dma_map_page(dma->dev, dest, offset, len, DMA_BIDIRECTIONAL);
for (i = 0; i < src_cnt; i++) {
/* only map the dest once */
if (!src_list[i])
continue;
if (unlikely(src_list[i] == dest)) {
dma_src[xor_src_cnt++] = dma_dest;
continue;
}
dma_src[xor_src_cnt++] = dma_map_page(dma->dev, src_list[i], offset,
len, DMA_TO_DEVICE);
}
src_cnt = xor_src_cnt;
while (src_cnt) { while (src_cnt) {
dma_addr_t tmp;
submit->flags = flags_orig; submit->flags = flags_orig;
dma_flags = 0;
xor_src_cnt = min(src_cnt, (int)dma->max_xor); xor_src_cnt = min(src_cnt, (int)dma->max_xor);
/* if we are submitting additional xors, leave the chain open, /* if we are submitting additional xors, leave the chain open
* clear the callback parameters, and leave the destination * and clear the callback parameters
* buffer mapped
*/ */
if (src_cnt > xor_src_cnt) { if (src_cnt > xor_src_cnt) {
submit->flags &= ~ASYNC_TX_ACK; submit->flags &= ~ASYNC_TX_ACK;
submit->flags |= ASYNC_TX_FENCE; submit->flags |= ASYNC_TX_FENCE;
dma_flags = DMA_COMPL_SKIP_DEST_UNMAP;
submit->cb_fn = NULL; submit->cb_fn = NULL;
submit->cb_param = NULL; submit->cb_param = NULL;
} else { } else {
@@ -85,12 +68,18 @@ do_async_xor(struct dma_chan *chan, struct page *dest, struct page **src_list,
dma_flags |= DMA_PREP_INTERRUPT; dma_flags |= DMA_PREP_INTERRUPT;
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_flags |= DMA_PREP_FENCE; dma_flags |= DMA_PREP_FENCE;
/* Since we have clobbered the src_list we are committed
* to doing this asynchronously. Drivers force forward progress /* Drivers force forward progress in case they can not provide a
* in case they can not provide a descriptor * descriptor
*/ */
tx = dma->device_prep_dma_xor(chan, dma_dest, &dma_src[src_off], tmp = src_list[0];
xor_src_cnt, len, dma_flags); if (src_list > unmap->addr)
src_list[0] = dma_dest;
tx = dma->device_prep_dma_xor(chan, dma_dest, src_list,
xor_src_cnt, unmap->len,
dma_flags);
src_list[0] = tmp;
if (unlikely(!tx)) if (unlikely(!tx))
async_tx_quiesce(&submit->depend_tx); async_tx_quiesce(&submit->depend_tx);
@@ -99,22 +88,21 @@ do_async_xor(struct dma_chan *chan, struct page *dest, struct page **src_list,
while (unlikely(!tx)) { while (unlikely(!tx)) {
dma_async_issue_pending(chan); dma_async_issue_pending(chan);
tx = dma->device_prep_dma_xor(chan, dma_dest, tx = dma->device_prep_dma_xor(chan, dma_dest,
&dma_src[src_off], src_list,
xor_src_cnt, len, xor_src_cnt, unmap->len,
dma_flags); dma_flags);
} }
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
submit->depend_tx = tx; submit->depend_tx = tx;
if (src_cnt > xor_src_cnt) { if (src_cnt > xor_src_cnt) {
/* drop completed sources */ /* drop completed sources */
src_cnt -= xor_src_cnt; src_cnt -= xor_src_cnt;
src_off += xor_src_cnt;
/* use the intermediate result a source */ /* use the intermediate result a source */
dma_src[--src_off] = dma_dest;
src_cnt++; src_cnt++;
src_list += xor_src_cnt - 1;
} else } else
break; break;
} }
@@ -189,22 +177,40 @@ async_xor(struct page *dest, struct page **src_list, unsigned int offset,
struct dma_chan *chan = async_tx_find_channel(submit, DMA_XOR, struct dma_chan *chan = async_tx_find_channel(submit, DMA_XOR,
&dest, 1, src_list, &dest, 1, src_list,
src_cnt, len); src_cnt, len);
dma_addr_t *dma_src = NULL; struct dma_device *device = chan ? chan->device : NULL;
struct dmaengine_unmap_data *unmap = NULL;
BUG_ON(src_cnt <= 1); BUG_ON(src_cnt <= 1);
if (submit->scribble) if (device)
dma_src = submit->scribble; unmap = dmaengine_get_unmap_data(device->dev, src_cnt+1, GFP_NOIO);
else if (sizeof(dma_addr_t) <= sizeof(struct page *))
dma_src = (dma_addr_t *) src_list; if (unmap && is_dma_xor_aligned(device, offset, 0, len)) {
struct dma_async_tx_descriptor *tx;
int i, j;
if (dma_src && chan && is_dma_xor_aligned(chan->device, offset, 0, len)) {
/* run the xor asynchronously */ /* run the xor asynchronously */
pr_debug("%s (async): len: %zu\n", __func__, len); pr_debug("%s (async): len: %zu\n", __func__, len);
return do_async_xor(chan, dest, src_list, offset, src_cnt, len, unmap->len = len;
dma_src, submit); for (i = 0, j = 0; i < src_cnt; i++) {
if (!src_list[i])
continue;
unmap->to_cnt++;
unmap->addr[j++] = dma_map_page(device->dev, src_list[i],
offset, len, DMA_TO_DEVICE);
}
/* map it bidirectional as it may be re-used as a source */
unmap->addr[j] = dma_map_page(device->dev, dest, offset, len,
DMA_BIDIRECTIONAL);
unmap->bidi_cnt = 1;
tx = do_async_xor(chan, unmap, submit);
dmaengine_unmap_put(unmap);
return tx;
} else { } else {
dmaengine_unmap_put(unmap);
/* run the xor synchronously */ /* run the xor synchronously */
pr_debug("%s (sync): len: %zu\n", __func__, len); pr_debug("%s (sync): len: %zu\n", __func__, len);
WARN_ONCE(chan, "%s: no space for dma address conversion\n", WARN_ONCE(chan, "%s: no space for dma address conversion\n",
@@ -268,16 +274,14 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset,
struct dma_chan *chan = xor_val_chan(submit, dest, src_list, src_cnt, len); struct dma_chan *chan = xor_val_chan(submit, dest, src_list, src_cnt, len);
struct dma_device *device = chan ? chan->device : NULL; struct dma_device *device = chan ? chan->device : NULL;
struct dma_async_tx_descriptor *tx = NULL; struct dma_async_tx_descriptor *tx = NULL;
dma_addr_t *dma_src = NULL; struct dmaengine_unmap_data *unmap = NULL;
BUG_ON(src_cnt <= 1); BUG_ON(src_cnt <= 1);
if (submit->scribble) if (device)
dma_src = submit->scribble; unmap = dmaengine_get_unmap_data(device->dev, src_cnt, GFP_NOIO);
else if (sizeof(dma_addr_t) <= sizeof(struct page *))
dma_src = (dma_addr_t *) src_list;
if (dma_src && device && src_cnt <= device->max_xor && if (unmap && src_cnt <= device->max_xor &&
is_dma_xor_aligned(device, offset, 0, len)) { is_dma_xor_aligned(device, offset, 0, len)) {
unsigned long dma_prep_flags = 0; unsigned long dma_prep_flags = 0;
int i; int i;
@@ -288,11 +292,15 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset,
dma_prep_flags |= DMA_PREP_INTERRUPT; dma_prep_flags |= DMA_PREP_INTERRUPT;
if (submit->flags & ASYNC_TX_FENCE) if (submit->flags & ASYNC_TX_FENCE)
dma_prep_flags |= DMA_PREP_FENCE; dma_prep_flags |= DMA_PREP_FENCE;
for (i = 0; i < src_cnt; i++)
dma_src[i] = dma_map_page(device->dev, src_list[i],
offset, len, DMA_TO_DEVICE);
tx = device->device_prep_dma_xor_val(chan, dma_src, src_cnt, for (i = 0; i < src_cnt; i++) {
unmap->addr[i] = dma_map_page(device->dev, src_list[i],
offset, len, DMA_TO_DEVICE);
unmap->to_cnt++;
}
unmap->len = len;
tx = device->device_prep_dma_xor_val(chan, unmap->addr, src_cnt,
len, result, len, result,
dma_prep_flags); dma_prep_flags);
if (unlikely(!tx)) { if (unlikely(!tx)) {
@@ -301,11 +309,11 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset,
while (!tx) { while (!tx) {
dma_async_issue_pending(chan); dma_async_issue_pending(chan);
tx = device->device_prep_dma_xor_val(chan, tx = device->device_prep_dma_xor_val(chan,
dma_src, src_cnt, len, result, unmap->addr, src_cnt, len, result,
dma_prep_flags); dma_prep_flags);
} }
} }
dma_set_unmap(tx, unmap);
async_tx_submit(chan, tx, submit); async_tx_submit(chan, tx, submit);
} else { } else {
enum async_tx_flags flags_orig = submit->flags; enum async_tx_flags flags_orig = submit->flags;
@@ -327,6 +335,7 @@ async_xor_val(struct page *dest, struct page **src_list, unsigned int offset,
async_tx_sync_epilog(submit); async_tx_sync_epilog(submit);
submit->flags = flags_orig; submit->flags = flags_orig;
} }
dmaengine_unmap_put(unmap);
return tx; return tx;
} }
crypto/async_tx/raid6test.c
+9 -1
View File
@@ -28,7 +28,7 @@
#undef pr #undef pr
#define pr(fmt, args...) pr_info("raid6test: " fmt, ##args) #define pr(fmt, args...) pr_info("raid6test: " fmt, ##args)
#define NDISKS 16 /* Including P and Q */ #define NDISKS 64 /* Including P and Q */
static struct page *dataptrs[NDISKS]; static struct page *dataptrs[NDISKS];
static addr_conv_t addr_conv[NDISKS]; static addr_conv_t addr_conv[NDISKS];
@@ -219,6 +219,14 @@ static int raid6_test(void)
err += test(11, &tests); err += test(11, &tests);
err += test(12, &tests); err += test(12, &tests);
} }
/* the 24 disk case is special for ioatdma as it is the boudary point
* at which it needs to switch from 8-source ops to 16-source
* ops for continuation (assumes DMA_HAS_PQ_CONTINUE is not set)
*/
if (NDISKS > 24)
err += test(24, &tests);
err += test(NDISKS, &tests); err += test(NDISKS, &tests);
pr("\n"); pr("\n");
drivers/ata/pata_arasan_cf.c
+1 -2
View File
@@ -396,8 +396,7 @@ dma_xfer(struct arasan_cf_dev *acdev, dma_addr_t src, dma_addr_t dest, u32 len)
struct dma_async_tx_descriptor *tx; struct dma_async_tx_descriptor *tx;
struct dma_chan *chan = acdev->dma_chan; struct dma_chan *chan = acdev->dma_chan;
dma_cookie_t cookie; dma_cookie_t cookie;
unsigned long flags = DMA_PREP_INTERRUPT | DMA_COMPL_SKIP_SRC_UNMAP | unsigned long flags = DMA_PREP_INTERRUPT;
DMA_COMPL_SKIP_DEST_UNMAP;
int ret = 0; int ret = 0;
tx = chan->device->device_prep_dma_memcpy(chan, dest, src, len, flags); tx = chan->device->device_prep_dma_memcpy(chan, dest, src, len, flags);
drivers/dma/Kconfig
+5 -4
View File
@@ -89,14 +89,15 @@ config AT_HDMAC
Support the Atmel AHB DMA controller. Support the Atmel AHB DMA controller.
config FSL_DMA config FSL_DMA
tristate "Freescale Elo and Elo Plus DMA support" tristate "Freescale Elo series DMA support"
depends on FSL_SOC depends on FSL_SOC
select DMA_ENGINE select DMA_ENGINE
select ASYNC_TX_ENABLE_CHANNEL_SWITCH select ASYNC_TX_ENABLE_CHANNEL_SWITCH
---help--- ---help---
Enable support for the Freescale Elo and Elo Plus DMA controllers. Enable support for the Freescale Elo series DMA controllers.
The Elo is the DMA controller on some 82xx and 83xx parts, and the The Elo is the DMA controller on some mpc82xx and mpc83xx parts, the
Elo Plus is the DMA controller on 85xx and 86xx parts. EloPlus is on mpc85xx and mpc86xx and Pxxx parts, and the Elo3 is on
some Txxx and Bxxx parts.
config MPC512X_DMA config MPC512X_DMA
tristate "Freescale MPC512x built-in DMA engine support" tristate "Freescale MPC512x built-in DMA engine support"
drivers/dma/amba-pl08x.c
+4 -35
View File
@@ -1164,42 +1164,12 @@ static void pl08x_free_txd(struct pl08x_driver_data *pl08x,
kfree(txd); kfree(txd);
} }
static void pl08x_unmap_buffers(struct pl08x_txd *txd)
{
struct device *dev = txd->vd.tx.chan->device->dev;
struct pl08x_sg *dsg;
if (!(txd->vd.tx.flags & DMA_COMPL_SKIP_SRC_UNMAP)) {
if (txd->vd.tx.flags & DMA_COMPL_SRC_UNMAP_SINGLE)
list_for_each_entry(dsg, &txd->dsg_list, node)
dma_unmap_single(dev, dsg->src_addr, dsg->len,
DMA_TO_DEVICE);
else {
list_for_each_entry(dsg, &txd->dsg_list, node)
dma_unmap_page(dev, dsg->src_addr, dsg->len,
DMA_TO_DEVICE);
}
}
if (!(txd->vd.tx.flags & DMA_COMPL_SKIP_DEST_UNMAP)) {
if (txd->vd.tx.flags & DMA_COMPL_DEST_UNMAP_SINGLE)
list_for_each_entry(dsg, &txd->dsg_list, node)
dma_unmap_single(dev, dsg->dst_addr, dsg->len,
DMA_FROM_DEVICE);
else
list_for_each_entry(dsg, &txd->dsg_list, node)
dma_unmap_page(dev, dsg->dst_addr, dsg->len,
DMA_FROM_DEVICE);
}
}
static void pl08x_desc_free(struct virt_dma_desc *vd) static void pl08x_desc_free(struct virt_dma_desc *vd)
{ {
struct pl08x_txd *txd = to_pl08x_txd(&vd->tx); struct pl08x_txd *txd = to_pl08x_txd(&vd->tx);
struct pl08x_dma_chan *plchan = to_pl08x_chan(vd->tx.chan); struct pl08x_dma_chan *plchan = to_pl08x_chan(vd->tx.chan);
if (!plchan->slave) dma_descriptor_unmap(txd);
pl08x_unmap_buffers(txd);
if (!txd->done) if (!txd->done)
pl08x_release_mux(plchan); pl08x_release_mux(plchan);
@@ -1252,7 +1222,7 @@ static enum dma_status pl08x_dma_tx_status(struct dma_chan *chan,
size_t bytes = 0; size_t bytes = 0;
ret = dma_cookie_status(chan, cookie, txstate); ret = dma_cookie_status(chan, cookie, txstate);
if (ret == DMA_SUCCESS) if (ret == DMA_COMPLETE)
return ret; return ret;
/* /*
@@ -1267,7 +1237,7 @@ static enum dma_status pl08x_dma_tx_status(struct dma_chan *chan,
spin_lock_irqsave(&plchan->vc.lock, flags); spin_lock_irqsave(&plchan->vc.lock, flags);
ret = dma_cookie_status(chan, cookie, txstate); ret = dma_cookie_status(chan, cookie, txstate);
if (ret != DMA_SUCCESS) { if (ret != DMA_COMPLETE) {
vd = vchan_find_desc(&plchan->vc, cookie); vd = vchan_find_desc(&plchan->vc, cookie);
if (vd) { if (vd) {
/* On the issued list, so hasn't been processed yet */ /* On the issued list, so hasn't been processed yet */
@@ -2138,8 +2108,7 @@ static int pl08x_probe(struct amba_device *adev, const struct amba_id *id)
writel(0x000000FF, pl08x->base + PL080_ERR_CLEAR); writel(0x000000FF, pl08x->base + PL080_ERR_CLEAR);
writel(0x000000FF, pl08x->base + PL080_TC_CLEAR); writel(0x000000FF, pl08x->base + PL080_TC_CLEAR);
ret = request_irq(adev->irq[0], pl08x_irq, IRQF_DISABLED, ret = request_irq(adev->irq[0], pl08x_irq, 0, DRIVER_NAME, pl08x);
DRIVER_NAME, pl08x);
if (ret) { if (ret) {
dev_err(&adev->dev, "%s failed to request interrupt %d\n", dev_err(&adev->dev, "%s failed to request interrupt %d\n",
__func__, adev->irq[0]); __func__, adev->irq[0]);

Some files were not shown because too many files have changed in this diff Show More