ANDROID: arm64: Work around Cortex-A510 erratum 2454944

Cortex-A510 erratum 2454944 may cause clean cache lines to be
erroneously written back to memory, breaking the assumptions we rely on
for non-coherent DMA. Try to mitigate this by implementing special DMA
ops that do their best to avoid cacheable aliases via a combination of
bounce-buffering and manipulating the linear map directly, to minimise
the chance of DMA-mapped pages being speculated back into caches.

The other main concern is initial entry, where cache lines covering the
kernel image might potentially become affected between being cleaned by
the bootloader and the kernel being called, which might require additional
cache maintenance from the bootloader to be safe in that regard too.
Cortex-A510 supports S2FWB, so KVM should be unaffected.

For the workaround to be applied, it needs to be explicitly requested
through dedicated arm64_noalias_setup_dma_ops callback.

Bug: 223346425
(cherry picked from commit 683efc5fc6eeb653caf85c33a2fb92a33c8faa75
 https://git.gitlab.arm.com/linux-arm/linux-rm.git arm64/2454944-dev)
Change-Id: If76b97dc39c278edb80f9b750129975ab2ac563e
Signed-off-by: Robin Murphy <robin.murphy@arm.com>
[BM: Stripping-down the original solution by removing support for
     cpu capabilities and ammending relevant bits, with the final
     version being reduced to dedicated DMA ops with dependencies on
     rodata_full being enabled (CONFIG_RODATA_FULL_DEFAULT_ENABLED),
     swiotlb late init and disabling lazy tlb flushing.
     Also, as a consequence, reducing debugging support.]
Signed-off-by: Beata Michalska <beata.michalska@arm.com>

arch/arm64/Kconfig

@@ -729,6 +729,29 @@ config ARM64_ERRATUM_2067961
 
 	  If unsure, say Y.
 
+config ARM64_ERRATUM_2454944
+	bool "Cortex-A510: 2454944: Unmodified cache line might be written back to memory"
+	select ARCH_HAS_TEARDOWN_DMA_OPS
+	select RODATA_FULL_DEFAULT_ENABLED
+	help
+	  This option adds the workaround for ARM Cortex-A510 erratum 2454944.
+
+	  Affected Cortex-A510 core might write unmodified cache lines back to
+	  memory, which breaks the assumptions upon which software coherency
+	  management for non-coherent DMA relies. If a cache line is
+	  speculatively fetched while a non-coherent device is writing directly
+	  to DRAM, and subsequently written back by natural eviction, data
+	  written by the device in the intervening period can be lost.
+
+	  The workaround is to enforce as far as reasonably possible that all
+	  non-coherent DMA transfers are bounced and/or remapped to minimise
+	  the chance that any Cacheable alias exists through which speculative
+	  cache fills could occur. To further improve effectiveness of
+	  the workaround, lazy TLB flushing should be disabled.
+
+	  This is quite involved and has unavoidable performance impact on
+	  affected systems.
+
 config ARM64_ERRATUM_2457168
 	bool "Cortex-A510: 2457168: workaround for AMEVCNTR01 incrementing incorrectly"
 	depends on ARM64_AMU_EXTN

arch/arm64/include/asm/dma-mapping-noalias.h

@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2022-2023 ARM Ltd.
+ */
+#ifndef __ASM_DMA_MAPPING_NOALIAS_H
+#define __ASM_DMA_MAPPING_NOALIAS_H
+
+#ifdef CONFIG_ARM64_ERRATUM_2454944
+void arm64_noalias_setup_dma_ops(struct device *dev);
+#else
+static inline void arm64_noalias_setup_dma_ops(struct device *dev)
+{
+}
+#endif
+#endif /* __ASM_DMA_MAPPING_NOALIAS_H */

arch/arm64/mm/Makefile

@@ -13,3 +13,5 @@ KASAN_SANITIZE_physaddr.o	+= n
 
 obj-$(CONFIG_KASAN)		+= kasan_init.o
 KASAN_SANITIZE_kasan_init.o	:= n
+
+obj-$(CONFIG_ARM64_ERRATUM_2454944) += dma-mapping-noalias.o