Fix a MIPS `dma_alloc_coherent' regression from commit bc3ec75de5
("dma-mapping: merge direct and noncoherent ops") that causes a cached
allocation to be returned on noncoherent cache systems.
This is due to an inverted check now used in the MIPS implementation of
`arch_dma_alloc' on the result from `dma_direct_alloc_pages' before
doing the cached-to-uncached mapping of the allocation address obtained.
The mapping has to be done for a non-NULL rather than NULL result,
because a NULL result means the allocation has failed.
Invert the check for correct operation then.
Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Signed-off-by: Paul Burton <paul.burton@mips.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Fixes: bc3ec75de5 ("dma-mapping: merge direct and noncoherent ops")
Patchwork: https://patchwork.linux-mips.org/patch/20965/
Pull MIPS updates from Paul Burton:
- kexec support for the generic MIPS platform when running on a CPU
including the MIPS Coherence Manager & related hardware.
- Improvements to the definition of memory barriers used around MMIO
accesses, and fixes in their use.
- Switch to CONFIG_NO_BOOTMEM from Mike Rapoport, finally dropping
reliance on the old bootmem code.
- A number of fixes & improvements for Loongson 3 systems.
- DT & config updates for the Microsemi Ocelot platform.
- Workaround to enable USB power on the Netgear WNDR3400v3.
- Various cleanups & fixes.
* tag 'mips_4.20' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux: (51 commits)
MIPS: Cleanup DSP ASE detection
MIPS: dts: Change upper case to lower case
MIPS: generic: Add Network, SPI and I2C to ocelot_defconfig
MIPS: Loongson-3: Fix BRIDGE irq delivery problem
MIPS: Loongson-3: Fix CPU UART irq delivery problem
MIPS: Remove unused PREF, PREFE & PREFX macros
MIPS: lib: Use kernel_pref & user_pref in memcpy()
MIPS: Remove unused CAT macro
MIPS: Add kernel_pref & user_pref helpers
MIPS: Remove unused TTABLE macro
MIPS: Remove unused PIC macros
MIPS: Remove unused MOVN & MOVZ macros
MIPS: Provide actually relaxed MMIO accessors
MIPS: Enforce strong ordering for MMIO accessors
MIPS: Correct `mmiowb' barrier for `wbflush' platforms
MIPS: Define MMIO ordering barriers
MIPS: mscc: add PCB120 to the ocelot fitImage
MIPS: mscc: add DT for Ocelot PCB120
MIPS: memset: Limit excessive `noreorder' assembly mode use
MIPS: memset: Fix CPU_DADDI_WORKAROUNDS `small_fixup' regression
...
The only functional differences (modulo a few missing fixes in the arch
code) is that architectures without coherent caches need a hook to
convert a virtual or dma address into a pfn, given that we don't have
the kernel linear mapping available for the otherwise easy virt_to_page
call. As a side effect we can support mmap of the per-device coherent
area even on architectures not providing the callback, and we make
previous dangerous default methods dma_common_mmap actually save for
non-coherent architectures by rejecting it without the right helper.
In addition to that we need a hook so that some architectures can
override the protection bits when mmaping a dma coherent allocations.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Paul Burton <paul.burton@mips.com> # MIPS parts
All the cache maintainance is already stubbed out when not enabled,
but merging the two allows us to nicely handle the case where
cache maintainance is required for some devices, but not others.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Paul Burton <paul.burton@mips.com> # MIPS parts
Various architectures support both coherent and non-coherent dma on a
per-device basis. Move the dma_noncoherent flag from the mips archdata
field to struct device proper to prepare the infrastructure for reuse on
other architectures.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Paul Burton <paul.burton@mips.com>
Acked-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
While both option select a form of conditional dma coherence they don't
actually share any code in the implementation, so untangle them.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Paul Burton <paul.burton@mips.com>
Pull MIPS fixes from Paul Burton:
- Fix microMIPS build failures by adding a .insn directive to the
barrier_before_unreachable() asm statement in order to convince the
toolchain that the asm statement is a valid branch target rather
than a bogus attempt to switch ISA.
- Clean up our declarations of TLB functions that we overwrite with
generated code in order to prevent the compiler making assumptions
about alignment that cause microMIPS kernels built with GCC 7 &
above to die early during boot.
- Fix up a regression for MIPS32 kernels which slipped into the main
MIPS pull for 4.19, causing CONFIG_32BIT=y kernels to contain
inappropriate MIPS64 instructions.
- Extend our existing workaround for MIPSr6 builds that end up using
the __multi3 intrinsic to GCC 7 & below, rather than just GCC 7.
* tag 'mips_4.19_2' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux:
MIPS: lib: Provide MIPS64r6 __multi3() for GCC < 7
MIPS: Workaround GCC __builtin_unreachable reordering bug
compiler.h: Allow arch-specific asm/compiler.h
MIPS: Avoid move psuedo-instruction whilst using MIPS_ISA_LEVEL
MIPS: Consistently declare TLB functions
MIPS: Export tlbmiss_handler_setup_pgd near its definition
Pull MIPS updates from Paul Burton:
"Here are the main MIPS changes for 4.19.
An overview of the general architecture changes:
- Massive DMA ops refactoring from Christoph Hellwig (huzzah for
deleting crufty code!).
- We introduce NT_MIPS_DSP & NT_MIPS_FP_MODE ELF notes &
corresponding regsets to expose DSP ASE & floating point mode state
respectively, both for live debugging & core dumps.
- We better optimize our code by hard-coding cpu_has_* macros at
compile time where their values are known due to the ISA revision
that the kernel build is targeting.
- The EJTAG exception handler now better handles SMP systems, where
it was previously possible for CPUs to clobber a register value
saved by another CPU.
- Our implementation of memset() gained a couple of fixes for MIPSr6
systems to return correct values in some cases where stores fault.
- We now implement ioremap_wc() using the uncached-accelerated cache
coherency attribute where supported, which is detected during boot,
and fall back to plain uncached access where necessary. The
MIPS-specific (and unused in tree) ioremap_uncached_accelerated() &
ioremap_cacheable_cow() are removed.
- The prctl(PR_SET_FP_MODE, ...) syscall is better supported for SMP
systems by reworking the way we ensure remote CPUs that may be
running threads within the affected process switch mode.
- Systems using the MIPS Coherence Manager will now set the
MIPS_IC_SNOOPS_REMOTE flag to avoid some unnecessary cache
maintenance overhead when flushing the icache.
- A few fixes were made for building with clang/LLVM, which now
sucessfully builds kernels for many of our platforms.
- Miscellaneous cleanups all over.
And some platform-specific changes:
- ar7 gained stubs for a few clock API functions to fix build
failures for some drivers.
- ath79 gained support for a few new SoCs, a few fixes & better
gpio-keys support.
- Ci20 now exposes its SPI bus using the spi-gpio driver.
- The generic platform can now auto-detect a suitable value for
PHYS_OFFSET based upon the memory map described by the device tree,
allowing us to avoid wasting memory on page book-keeping for
systems where RAM starts at a non-zero physical address.
- Ingenic systems using the jz4740 platform code now link their
vmlinuz higher to allow for kernels of a realistic size.
- Loongson32 now builds the kernel targeting MIPSr1 rather than
MIPSr2 to avoid CPU errata.
- Loongson64 gains a couple of fixes, a workaround for a write
buffering issue & support for the Loongson 3A R3.1 CPU.
- Malta now uses the piix4-poweroff driver to handle powering down.
- Microsemi Ocelot gained support for its SPI bus & NOR flash, its
second MDIO bus and can now be supported by a FIT/.itb image.
- Octeon saw a bunch of header cleanups which remove a lot of
duplicate or unused code"
* tag 'mips_4.19' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux: (123 commits)
MIPS: Remove remnants of UASM_ISA
MIPS: netlogic: xlr: Remove erroneous check in nlm_fmn_send()
MIPS: VDSO: Force link endianness
MIPS: Always specify -EB or -EL when using clang
MIPS: Use dins to simplify __write_64bit_c0_split()
MIPS: Use read-write output operand in __write_64bit_c0_split()
MIPS: Avoid using array as parameter to write_c0_kpgd()
MIPS: vdso: Allow clang's --target flag in VDSO cflags
MIPS: genvdso: Remove GOT checks
MIPS: Remove obsolete MIPS checks for DST node "chosen@0"
MIPS: generic: Remove input symbols from defconfig
MIPS: Delete unused code in linux32.c
MIPS: Remove unused sys_32_mmap2
MIPS: Remove nabi_no_regargs
mips: dts: mscc: enable spi and NOR flash support on ocelot PCB123
mips: dts: mscc: Add spi on Ocelot
MIPS: Loongson: Merge load addresses
MIPS: Loongson: Set Loongson32 to MIPS32R1
MIPS: mscc: ocelot: add interrupt controller properties to GPIO controller
MIPS: generic: Select MIPS_AUTO_PFN_OFFSET
...
Since at least the beginning of the git era we've declared our TLB
exception handling functions inconsistently. They're actually functions,
but we declare them as arrays of u32 where each u32 is an encoded
instruction. This has always been the case for arch/mips/mm/tlbex.c, and
has also been true for arch/mips/kernel/traps.c since commit
86a1708a9d ("MIPS: Make tlb exception handler definitions and
declarations match.") which aimed for consistency but did so by
consistently making the our C code inconsistent with our assembly.
This is all usually harmless, but when using GCC 7 or newer to build a
kernel targeting microMIPS (ie. CONFIG_CPU_MICROMIPS=y) it becomes
problematic. With microMIPS bit 0 of the program counter indicates the
ISA mode. When bit 0 is zero instructions are decoded using the standard
MIPS32 or MIPS64 ISA. When bit 0 is one instructions are decoded using
microMIPS. This means that function pointers become odd - their least
significant bit is one for microMIPS code. We work around this in cases
where we need to access code using loads & stores with our
msk_isa16_mode() macro which simply clears bit 0 of the value it is
given:
#define msk_isa16_mode(x) ((x) & ~0x1)
For example we do this for our TLB load handler in
build_r4000_tlb_load_handler():
u32 *p = (u32 *)msk_isa16_mode((ulong)handle_tlbl);
We then write code to p, expecting it to be suitably aligned (our LEAF
macro aligns functions on 4 byte boundaries, so (ulong)handle_tlbl will
give a value one greater than a multiple of 4 - ie. the start of a
function on a 4 byte boundary, with the ISA mode bit 0 set).
This worked fine up to GCC 6, but GCC 7 & onwards is smart enough to
presume that handle_tlbl which we declared as an array of u32s must be
aligned sufficiently that bit 0 of its address will never be set, and as
a result optimize out msk_isa16_mode(). This leads to p having an
address with bit 0 set, and when we go on to attempt to store code at
that address we take an address error exception due to the unaligned
memory access.
This leads to an exception prior to the kernel having configured its own
exception handlers, so we jump to whatever handlers the bootloader
configured. In the case of QEMU this results in a silent hang, since it
has no useful general exception vector.
Fix this by consistently declaring our TLB-related functions as
functions. For handle_tlbl(), handle_tlbs() & handle_tlbm() we do this
in asm/tlbex.h & we make use of the existing declaration of
tlbmiss_handler_setup_pgd() in asm/mmu_context.h. Our TLB handler
generation code in arch/mips/mm/tlbex.c is adjusted to deal with these
definitions, in most cases simply by casting the function pointers to
u32 pointers.
This allows us to include asm/mmu_context.h in arch/mips/mm/tlbex.c to
get the definitions of tlbmiss_handler_setup_pgd & pgd_current, removing
some needless duplication. Consistently using msk_isa16_mode() on
function pointers means we no longer need the
tlbmiss_handler_setup_pgd_start symbol so that is removed entirely.
Now that we're declaring our functions as functions GCC stops optimizing
out msk_isa16_mode() & a microMIPS kernel built with either GCC 7.3.0 or
8.1.0 boots successfully.
Signed-off-by: Paul Burton <paul.burton@mips.com>
We export tlbmiss_handler_setup_pgd in arch/mips/mm/tlbex.c close to a
declaration of it, rather than close to its definition as is standard.
We've supported exporting symbols in assembly code since commit
22823ab419 ("EXPORT_SYMBOL() for asm"), so move the export to follow
the function's (stub) definition.
Signed-off-by: Paul Burton <paul.burton@mips.com>
Commit 33679a5037 ("MIPS: uasm: Remove needless ISA abstraction")
removed use of the MIPS_ISA preprocessor macro, but left a couple of
unused definitions of it behind.
Remove the dead code.
Signed-off-by: Paul Burton <paul.burton@mips.com>
Passing an array (swapper_pg_dir) as the argument to write_c0_kpgd() in
setup_pw() will become problematic if we modify __write_64bit_c0_split()
to cast its val argument to unsigned long long, because for 32-bit
kernel builds the size of a pointer will differ from the size of an
unsigned long long. This would fall foul of gcc's pointer-to-int-cast
diagnostic.
Cast the value to a long, which should be the same width as the pointer
that we ultimately want & will be sign extended if required to the
unsigned long long that __write_64bit_c0_split() ultimately needs.
Signed-off-by: Paul Burton <paul.burton@mips.com>
Converting an address between cached & uncached (typically addresses in
(c)kseg0 & (c)kseg1 or 2 xkphys regions) should not depend upon
PHYS_OFFSET in any way - we're converting from a virtual address in one
unmapped region to a virtual address in another unmapped region.
For some reason our CAC_ADDR() & UNCAC_ADDR() macros make use of
PAGE_OFFSET, which typically includes PHYS_OFFSET. This means that
platforms with a non-zero PHYS_OFFSET typically have to workaround
miscalculation by these 2 macros by also defining UNCAC_BASE to a value
that isn't really correct.
It appears that an attempt has previously been made to address this with
commit 3f4579252aa1 ("MIPS: make CAC_ADDR and UNCAC_ADDR account for
PHYS_OFFSET") which was later undone by commit ed3ce16c3d ("Revert
"MIPS: make CAC_ADDR and UNCAC_ADDR account for PHYS_OFFSET"") which
also introduced the ar7 workaround. That attempt at a fix was roughly
equivalent, but essentially caused the CAC_ADDR() & UNCAC_ADDR() macros
to cancel out PHYS_OFFSET by adding & then subtracting it again. In his
revert Leonid is correct that using PHYS_OFFSET makes no sense in the
context of these macros, but appears to have missed its inclusion via
PAGE_OFFSET which means PHYS_OFFSET actually had an effect after the
revert rather than before it.
Here we fix this by modifying CAC_ADDR() & UNCAC_ADDR() to stop using
PAGE_OFFSET (& thus PHYS_OFFSET), instead using __pa() & __va() along
with UNCAC_BASE.
For UNCAC_ADDR(), __pa() will convert a cached address to a physical
address which we can simply use as an offset from UNCAC_BASE to obtain
an address in the uncached region.
For CAC_ADDR() we can undo the effect of UNCAC_ADDR() by subtracting
UNCAC_BASE and using __va() on the result.
With this change made, remove definitions of UNCAC_BASE from the ar7 &
pic32 platforms which appear to have defined them only to workaround
this problem.
Signed-off-by: Paul Burton <paul.burton@mips.com>
References: 3f4579252aa1 ("MIPS: make CAC_ADDR and UNCAC_ADDR account for PHYS_OFFSET")
References: ed3ce16c3d ("Revert "MIPS: make CAC_ADDR and UNCAC_ADDR account for PHYS_OFFSET"")
Patchwork: https://patchwork.linux-mips.org/patch/20046/
Cc: Florian Fainelli <f.fainelli@gmail.com>
Cc: James Hogan <jhogan@kernel.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: linux-mips@linux-mips.org
Cc: Vladimir Kondratiev <vladimir.kondratiev@intel.com>
mips_swiotlb_ops differs from the generic swiotlb_dma_ops only in that
it contains a mb() barrier after each operations that maps or syncs
dma memory to the device.
The dma operations are defined to not be memory barriers, but instead
the write* operations to kick the DMA off are supposed to contain them.
For mips this handled by war_io_reorder_wmb(), which evaluates to the
stronger wmb() instead of the pure compiler barrier barrier() for
just those platforms that use swiotlb, so I think we are covered
properly.
[paul.burton@mips.com:
- Include linux/swiotlb.h to fix build failures for configs with
CONFIG_SWIOTLB=y.]
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Paul Burton <paul.burton@mips.com>
Patchwork: https://patchwork.linux-mips.org/patch/20038/
Cc: David Daney <ddaney@caviumnetworks.com>
Cc: Huacai Chen <chenhc@lemote.com>
Cc: linux-mips@linux-mips.org
Cc: iommu@lists.linux-foundation.org
Cc: linux-kernel@vger.kernel.org
We currently attempt to check whether a physical address range provided
to __ioremap() may be in use by the page allocator by examining the
value of PageReserved for each page in the region - lowmem pages not
marked reserved are presumed to be in use by the page allocator, and
requests to ioremap them fail.
The way we check this has been broken since commit 92923ca3aa ("mm:
meminit: only set page reserved in the memblock region"), because
memblock will typically not have any knowledge of non-RAM pages and
therefore those pages will not have the PageReserved flag set. Thus when
we attempt to ioremap a region outside of RAM we incorrectly fail
believing that the region is RAM that may be in use.
In most cases ioremap() on MIPS will take a fast-path to use the
unmapped kseg1 or xkphys virtual address spaces and never hit this path,
so the only way to hit it is for a MIPS32 system to attempt to ioremap()
an address range in lowmem with flags other than _CACHE_UNCACHED.
Perhaps the most straightforward way to do this is using
ioremap_uncached_accelerated(), which is how the problem was discovered.
Fix this by making use of walk_system_ram_range() to test the address
range provided to __ioremap() against only RAM pages, rather than all
lowmem pages. This means that if we have a lowmem I/O region, which is
very common for MIPS systems, we're free to ioremap() address ranges
within it. A nice bonus is that the test is no longer limited to lowmem.
The approach here matches the way x86 performed the same test after
commit c81c8a1eee ("x86, ioremap: Speed up check for RAM pages") until
x86 moved towards a slightly more complicated check using walk_mem_res()
for unrelated reasons with commit 0e4c12b45a ("x86/mm, resource: Use
PAGE_KERNEL protection for ioremap of memory pages").
Signed-off-by: Paul Burton <paul.burton@mips.com>
Reported-by: Serge Semin <fancer.lancer@gmail.com>
Tested-by: Serge Semin <fancer.lancer@gmail.com>
Fixes: 92923ca3aa ("mm: meminit: only set page reserved in the memblock region")
Cc: James Hogan <jhogan@kernel.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: linux-mips@linux-mips.org
Cc: stable@vger.kernel.org # v4.2+
Patchwork: https://patchwork.linux-mips.org/patch/19786/
In systems that include a MIPS Coherency Manager, the icache always
fills from a cache which is coherent across all CPUs. In I6400 & I6500
systems the icache fills from the dcache which is coherent across all
CPUs. In all other CM-based systems the icache fills from the L2 cache
which is shared between all cores.
This means that an icache will always see stores from remote CPUs
without needing to write them back any further than that L2, which is
what the cpu_icache_snoops_remote_store feature is used to test. In
order for it to return 1 without needing a per-platform override (which
is what Malta has relied upon so far) set the MIPS_IC_SNOOPS_REMOTE flag
when a CM is present.
Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16200/
Maciej W. Rozycki
Mike Rapoport
Linus Torvalds
Alexandre Belloni
Christoph Hellwig
Souptick Joarder
Paul Burton
Paul Burton
Rickard Strandqvist