From 1833ce36b35426504c64600c94f322437ea44bb2 Mon Sep 17 00:00:00 2001 From: Breno Leitao Date: Thu, 16 Jul 2026 06:42:18 -0700 Subject: [PATCH 01/28] mm: memcg: initialize *locked in memcg1_oom_prepare() stub mem_cgroup_oom() passes an uninitialized "locked" to memcg1_oom_prepare() and reads it back in memcg1_oom_finish(): bool locked, ret; ... if (!memcg1_oom_prepare(memcg, &locked)) return false; ret = mem_cgroup_out_of_memory(memcg, mask, order); memcg1_oom_finish(memcg, locked); This relies on memcg1_oom_prepare() setting *locked whenever it returns true. The CONFIG_MEMCG_V1=y version does, but the stub used when CONFIG_MEMCG_V1=n returns true without touching *locked, so memcg1_oom_finish() consumes an uninitialized value. On a memcg OOM this is reported by UBSAN: UBSAN: invalid-load in mm/memcontrol.c:1932:27 load of value 0 is not a valid value for type 'bool' (aka '_Bool') Initialize *locked to false in the stub; with cgroup v1 compiled out there is no OOM lock to take. Link: https://lore.kernel.org/20260716-memcg-oom-uninit-locked-v2-1-63631d878eb4@debian.org Fixes: e93d4166b40a ("mm: memcg: put cgroup v1-specific code under a config option") Signed-off-by: Breno Leitao Reviewed-by: Joshua Hahn Acked-by: Johannes Weiner Reviewed-by: SeongJae Park Acked-by: Shakeel Butt Cc: Michal Hocko Cc: Muchun Song Cc: Roman Gushchin Cc: Shakeel Butt Cc: Signed-off-by: Andrew Morton --- mm/memcontrol-v1.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol-v1.h b/mm/memcontrol-v1.h index f92f81108d5e..4fa6e2bc8413 100644 --- a/mm/memcontrol-v1.h +++ b/mm/memcontrol-v1.h @@ -107,7 +107,11 @@ static inline void memcg1_remove_from_trees(struct mem_cgroup *memcg) {} static inline void memcg1_soft_limit_reset(struct mem_cgroup *memcg) {} static inline void memcg1_css_offline(struct mem_cgroup *memcg) {} -static inline bool memcg1_oom_prepare(struct mem_cgroup *memcg, bool *locked) { return true; } +static inline bool memcg1_oom_prepare(struct mem_cgroup *memcg, bool *locked) +{ + *locked = false; + return true; +} static inline void memcg1_oom_finish(struct mem_cgroup *memcg, bool locked) {} static inline void memcg1_oom_recover(struct mem_cgroup *memcg) {} From 4bd0c3515a041d9aa1558c4e8ea0efdb56509f9a Mon Sep 17 00:00:00 2001 From: Nico Pache Date: Fri, 17 Jul 2026 00:44:59 -0600 Subject: [PATCH 02/28] mm: decrement MTHP_STAT_NR_ANON in free_zone_device_folio() Patch series "mm: fix PMD level mTHP accounting bugs", v2. While running selftests I noticed the PMD level per-mTHP stats (nr_anon) remained elevated after each run. After further investigation I noticed this accounting error occurs for both the migration.private_anon_htlb_test and the HMM tests. In the HMM case this is due to folio_add_new_anon_rmap() incrementing the mTHP stats, but never containing a corresponding decrement in free_zone_device_folio(). We solve this by making sure to decrement the counter when freeing device memory. In the migration case, we are incrementing this counter without first checking whether this folio is a hugetlb folio, which relies on a separate accounting system. We solve this by adding the proper hugetlb check before incrementing this counter. With these changes in place, the two tests no longer cause elevated PMD level accounting issues. This patch (of 2): When a zone device folio is mapped as anonymous, folio_add_new_anon_rmap() increments MTHP_STAT_NR_ANON. The corresponding decrement lives in __free_pages_prepare() in page_alloc.c, but zone device folios are freed via free_zone_device_folio() which never calls __free_pages_prepare(). This causes nr_anon to remain permanently elevated after zone device folios are freed. Add the missing mod_mthp_stat() decrement to free_zone_device_folio() so that the counter is properly balanced. Link: https://lore.kernel.org/20260717064502.1980173-1-npache@redhat.com Link: https://lore.kernel.org/20260717064502.1980173-2-npache@redhat.com Fixes: 5d65c8d758f2 ("mm: count the number of anonymous THPs per size") Co-developed-by: David Hildenbrand Signed-off-by: David Hildenbrand Signed-off-by: Nico Pache Reviewed-by: Zi Yan Cc: Alistair Popple Cc: Barry Song Cc: Byungchul Park Cc: Gregory Price Cc: "Huang, Ying" Cc: Joshua Hahn Cc: Lorenzo Stoakes Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Oscar Salvador Cc: Rakie Kim Cc: Baolin Wang Signed-off-by: Andrew Morton --- mm/memremap.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/memremap.c b/mm/memremap.c index 81766d822400..accba23aef28 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -425,6 +425,7 @@ void free_zone_device_folio(struct folio *folio) mem_cgroup_uncharge(folio); if (folio_test_anon(folio)) { + mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON, -1); for (i = 0; i < nr; i++) __ClearPageAnonExclusive(folio_page(folio, i)); } From d09a8fd521476950079ee01a1408e6f5b7a0aa3e Mon Sep 17 00:00:00 2001 From: Nico Pache Date: Fri, 17 Jul 2026 00:45:00 -0600 Subject: [PATCH 03/28] mm/migrate: exclude hugetlb folios from MTHP_STAT_NR_ANON accounting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit __folio_migrate_mapping() increments MTHP_STAT_NR_ANON for the destination folio when `folio_test_anon(folio) && folio_test_large(folio)` is true. However, hugetlb folios satisfy both conditions despite having a completely separate accounting system — they use hugetlb_add_anon_rmap() which does not touch mTHP stats, and their free path also bypasses the mTHP decrement in __free_pages_prepare(). This causes MTHP_STAT_NR_ANON to be incremented on each hugetlb migration without a corresponding decrement, permanently inflating the nr_anon counter. Add a !folio_test_hugetlb() check to __folio_migrate_mapping() so that only actual mTHP folios are counted. Link: https://lore.kernel.org/20260717064502.1980173-3-npache@redhat.com Fixes: 5d65c8d758f2 ("mm: count the number of anonymous THPs per size") Co-developed-by: David Hildenbrand Signed-off-by: David Hildenbrand Signed-off-by: Nico Pache Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Cc: Alistair Popple Cc: Barry Song Cc: Byungchul Park Cc: Gregory Price Cc: "Huang, Ying" Cc: Joshua Hahn Cc: Lorenzo Stoakes Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Oscar Salvador Cc: Rakie Kim Signed-off-by: Andrew Morton --- mm/migrate.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index d9b23909d716..dd15a84b2a52 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -590,7 +590,8 @@ static int __folio_migrate_mapping(struct address_space *mapping, /* No turning back from here */ newfolio->index = folio->index; newfolio->mapping = folio->mapping; - if (folio_test_anon(folio) && folio_test_large(folio)) + if (folio_test_anon(folio) && folio_test_large(folio) && + !folio_test_hugetlb(folio)) mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON, 1); if (folio_test_swapbacked(folio)) __folio_set_swapbacked(newfolio); From 45ebe4540817cb540a59e4dc04014ac5dddc9990 Mon Sep 17 00:00:00 2001 From: Jiakai Xu Date: Thu, 16 Jul 2026 11:53:25 +0000 Subject: [PATCH 04/28] riscv/mm: use physical alignment for vmemmap_start_pfn RISC-V computes vmemmap_start_pfn by rounding phys_ram_base down to VMEMMAP_ADDR_ALIGN. That alignment must therefore be expressed in the physical-address domain. Commit 476849b0fba4 ("riscv/mm: align vmemmap to maximal folio size") attempted to account for the maximal folio alignment by feeding MAX_FOLIO_VMEMMAP_ALIGN directly into VMEMMAP_ADDR_ALIGN. However, MAX_FOLIO_VMEMMAP_ALIGN is measured in bytes of struct page storage, whereas VMEMMAP_ADDR_ALIGN is used to align a physical address. The mask-based compound_info encoding requires pfn_to_page(0) to be naturally aligned to MAX_FOLIO_VMEMMAP_ALIGN. Commit 9f94db4c7eaa ("mm/sparse: check memmap alignment for compound_info_has_mask()") added a check for that requirement and exposed the unit mismatch on systems such as QEMU virt, where the DRAM base is not aligned to MAX_FOLIO_NR_PAGES * PAGE_SIZE. Here is the log: [ 0.000000][ C0] ------------[ cut here ]------------ [ 0.000000][ C0] WARNING: mm/sparse.c:365 at sparse_init+0x58a/0x6fe, CPU#0: swapper/0 [ 0.000000][ C0] Modules linked in: [ 0.000000][ C0] CPU: 0 UID: 0 PID: 0 Comm: swapper Not tainted 7.2.0-rc3-g1d8304bdd65f #2 PREEMPT [ 0.000000][ C0] Hardware name: riscv-virtio,qemu (DT) [ 0.000000][ C0] epc : sparse_init+0x58a/0x6fe [ 0.000000][ C0] ra : sparse_init+0x58a/0x6fe [ 0.000000][ C0] epc : ffffffff86851c88 ra : ffffffff86851c88 sp : ffffffff88807a30 [ 0.000000][ C0] gp : ffffffff8a3bf240 tp : ffffffff88842080 t0 : ff600000ffab6000 [ 0.000000][ C0] t1 : 000000017fab6000 t2 : 65203a6573726363 s0 : ffffffff88807bc0 [ 0.000000][ C0] s1 : 000000000e000000 a0 : 0000000000000007 a1 : 0000000000000000 [ 0.000000][ C0] a2 : 0000000000000002 a3 : ffffffff86851c88 a4 : 0000000000000000 [ 0.000000][ C0] a5 : ffffffff88843080 a6 : 0000000000000003 a7 : 0000000000000000 [ 0.000000][ C0] s2 : ff60000000000000 s3 : 0040000000000000 s4 : 0004000000000000 [ 0.000000][ C0] s5 : ffffffff8a4d92e0 s6 : ff600000ffab55e0 s7 : ffffffff88384d00 [ 0.000000][ C0] s8 : 0000000000000003 s9 : ffffffff88384cc1 s10: ffffffff88384cc0 [ 0.000000][ C0] s11: ffffffff8a4daae0 t3 : ffffffff915e8b20 t4 : ffffffff915e8b20 [ 0.000000][ C0] t5 : ffffffff915e8b20 t6 : ffffffff915e8bc8 ssp : 0000000000000000 [ 0.000000][ C0] status: 0000000200000100 badaddr: ffffffff86851c88 cause: 0000000000000003 [ 0.000000][ C0] [] sparse_init+0x58a/0x6fe [ 0.000000][ C0] [] mm_core_init_early+0x116/0x1e30 [ 0.000000][ C0] [] start_kernel+0xd2/0x848 Convert MAX_FOLIO_VMEMMAP_ALIGN to the equivalent physical alignment before using it in VMEMMAP_ADDR_ALIGN. This keeps the existing round_down() logic while making the resulting vmemmap base satisfy the mask-alignment requirement. Link: https://lore.kernel.org/20260716115326.3466926-1-xujiakai2025@iscas.ac.cn Fixes: 476849b0fba4 ("riscv/mm: align vmemmap to maximal folio size") Signed-off-by: Jiakai Xu Reviewed-by: Kiryl Shutsemau (Meta) Cc: Albert Ou Cc: Alexandre Ghiti Cc: David Hildenbrand Cc: Guo Ren Cc: Mike Rapoport Cc: Muchun Song Cc: Nam Cao Cc: Palmer Dabbelt Cc: Vishal Moola (Oracle) Assisted-by: YuanSheng:DeepSeek-V4-Flash Cc: Signed-off-by: Andrew Morton --- arch/riscv/mm/init.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 3e450890be07..422efa11824b 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -63,7 +63,8 @@ EXPORT_SYMBOL(phys_ram_base); #ifdef CONFIG_SPARSEMEM_VMEMMAP #define VMEMMAP_ADDR_ALIGN max(1ULL << SECTION_SIZE_BITS, \ - MAX_FOLIO_VMEMMAP_ALIGN) + PFN_PHYS(MAX_FOLIO_VMEMMAP_ALIGN / \ + sizeof(struct page))) unsigned long vmemmap_start_pfn __ro_after_init; EXPORT_SYMBOL(vmemmap_start_pfn); From e923bd21058ea02fd0dcd3549d151d143fd036e5 Mon Sep 17 00:00:00 2001 From: "Kiryl Shutsemau (Meta)" Date: Thu, 16 Jul 2026 10:54:24 +0100 Subject: [PATCH 05/28] mm/huge_memory: unlock i_mmap_rwsem before releasing after-split folios __folio_split() keeps dereferencing the mapping after the split: shmem_uncharge(mapping->host) and remap_page() while the folios are still frozen/locked, and i_mmap_unlock_read(mapping) at the very end, after the after-split folios have been unlocked and freed. Nothing holds an inode reference across that. The split relies on @folio -- which the beyond-EOF drop loop never removes, as it starts at folio_next(folio) -- staying locked and in the page cache to hold off eviction. But the unlock loop unlocks @folio before i_mmap_unlock_read() runs. If the caller's @lock_at is a tail beyond EOF, as memory_failure() passes when splitting a poisoned tail of a shmem THP that reaches past i_size during truncation, it too is gone from the page cache; so once @folio is unlocked no locked, in-cache folio pins the inode, and a concurrent final iput() can evict and RCU-free it before i_mmap_unlock_read() touches i_mmap_rwsem: BUG: KASAN: slab-use-after-free in __up_read+0x634/0x790 i_mmap_unlock_read include/linux/fs.h:537 [inline] __folio_split+0x732/0x1640 mm/huge_memory.c:4100 try_to_split_thp_page+0xab/0x390 mm/memory-failure.c:1675 memory_failure+0x1394/0x26e0 mm/memory-failure.c:2470 Freed by task 4601: shmem_free_in_core_inode+0x54/0xb0 mm/shmem.c:5177 evict+0x57f/0xac0 fs/inode.c:870 Do every mapping dereference while @folio still pins the inode: drop i_mmap_rwsem right after remap_page(), before the loop that unlocks and frees the after-split folios, and clear @mapping so the exit path does not unlock it again. shmem_uncharge() and remap_page() already run before that point, so after this nothing past the unlock loop touches the inode or the mapping. This is now a rule the split depends on, alongside keeping @folio frozen until the page cache is updated: no inode or mapping dereference once the after-split folios start being unlocked. Link: https://lore.kernel.org/20260716095424.471052-1-kirill@shutemov.name Fixes: baa355fd3314 ("thp: file pages support for split_huge_page()") Signed-off-by: Kiryl Shutsemau (Meta) Reported-by: Hao Zhang Closes: https://lore.kernel.org/linux-mm/20260710071344.GA106129@zh-pc Co-developed-by: Hao Zhang Signed-off-by: Hao Zhang Acked-by: David Hildenbrand (Arm) Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Reviewed-by: Miaohe Lin Cc: Baolin Wang Cc: Barry Song Cc: Dev Jain Cc: Lance Yang Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Naoya Horiguchi Cc: Nico Pache Cc: Ryan Roberts Cc: Signed-off-by: Andrew Morton --- mm/huge_memory.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 032702a4637b..58cabe6af33d 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4113,6 +4113,18 @@ fail: remap_page(folio, 1 << old_order, ttu_flags); + /* + * Drop the mapping while the inode is still pinned. @folio stays + * locked and present in the page cache until the loop below, so + * eviction cannot free the inode yet; @lock_at is not enough, it may + * be a tail beyond EOF that the split already dropped from the page + * cache. Nothing past this point may touch the inode or the mapping. + */ + if (mapping) { + i_mmap_unlock_read(mapping); + mapping = NULL; + } + /* * Unlock all after-split folios except the one containing * @lock_at page. If @folio is not split, it will be kept locked. From 598356522b3a7c337fbf643561ee9b2ba868840e Mon Sep 17 00:00:00 2001 From: Burak Emir Date: Sat, 18 Jul 2026 17:13:03 +0200 Subject: [PATCH 06/28] MAINTAINERS: update address for Burak Emir Update MAINTAINERS and .mailmap to point to my gmail.com address: burak.emir@gmail.com. Link: https://lore.kernel.org/20260718151303.2649-1-burak.emir@gmail.com Signed-off-by: Burak Emir Cc: Alice Ryhl Cc: Jakub Kacinski Cc: Martin Kepplinger Cc: Yury Norov (NVIDIA) Signed-off-by: Andrew Morton --- .mailmap | 1 + MAINTAINERS | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.mailmap b/.mailmap index a8be42f87e02..de68ca3cbd23 100644 --- a/.mailmap +++ b/.mailmap @@ -176,6 +176,7 @@ Brian Cain Brian King Brian Silverman Bryan Tan +Burak Emir Cai Huoqing Casey Connolly Casey Connolly diff --git a/MAINTAINERS b/MAINTAINERS index 2f9472c1a090..39c9413355f3 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4634,7 +4634,7 @@ F: rust/helpers/cpumask.c BITMAP API [RUST] M: Alice Ryhl -M: Burak Emir +M: Burak Emir R: Yury Norov S: Maintained F: lib/find_bit_benchmark_rust.rs From 1fcac73551425c6924cca5cc29f10a5caf80907b Mon Sep 17 00:00:00 2001 From: Peter Collingbourne Date: Sat, 18 Jul 2026 10:29:23 -0700 Subject: [PATCH 07/28] arm64, mailmap: update email address for Peter Collingbourne I am no longer at Google. Link: https://lore.kernel.org/20260718172923.8297-1-peter@pcc.me.uk Signed-off-by: Peter Collingbourne Cc: Catalin Marinas Cc: Ian Rogers Cc: Jakub Kacinski Cc: Martin Kepplinger Cc: Nick Desaulniers Cc: Will Deacon Signed-off-by: Andrew Morton --- .mailmap | 1 + arch/arm64/kernel/pi/relocate.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.mailmap b/.mailmap index de68ca3cbd23..c9c3a6b2b1f5 100644 --- a/.mailmap +++ b/.mailmap @@ -696,6 +696,7 @@ Paulo Alcantara Paulo Alcantara Pavankumar Kondeti Peter A Jonsson +Peter Collingbourne Peter Hilber Peter Oruba Peter Oruba diff --git a/arch/arm64/kernel/pi/relocate.c b/arch/arm64/kernel/pi/relocate.c index 2407d2696398..82592f3a5c1c 100644 --- a/arch/arm64/kernel/pi/relocate.c +++ b/arch/arm64/kernel/pi/relocate.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only // Copyright 2023 Google LLC // Authors: Ard Biesheuvel -// Peter Collingbourne +// Peter Collingbourne #include #include From 8c7372c5b155fff151a111f03ec0fb87c4c063fe Mon Sep 17 00:00:00 2001 From: "Nico Pache (Red Hat)" Date: Tue, 21 Jul 2026 04:18:11 -0600 Subject: [PATCH 08/28] MAINTAINERS: update Nico Pache's email address Switch my entry in MAINTAINERS and .mailmap to my @linux.dev email address Link: https://lore.kernel.org/20260721101811.115954-1-nico.pache@linux.dev Signed-off-by: Nico Pache Signed-off-by: Nico Pache (Red Hat) Acked-by: David Hildenbrand (Arm) Signed-off-by: Andrew Morton --- .mailmap | 1 + MAINTAINERS | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.mailmap b/.mailmap index c9c3a6b2b1f5..ca6dc2575802 100644 --- a/.mailmap +++ b/.mailmap @@ -649,6 +649,7 @@ Nicholas Piggin Nicholas Piggin Nicholas Piggin Nicolas Ferre +Nico Pache Nicolas Pitre Nicolas Pitre Nicolas Saenz Julienne diff --git a/MAINTAINERS b/MAINTAINERS index 39c9413355f3..48b2baa1541f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17253,7 +17253,7 @@ M: Lorenzo Stoakes R: Zi Yan R: Baolin Wang R: Liam R. Howlett -R: Nico Pache +R: Nico Pache R: Ryan Roberts R: Dev Jain R: Barry Song From dc37771a43d4a8762f05060c28463b0c20e6dc9b Mon Sep 17 00:00:00 2001 From: Richard Chang Date: Mon, 20 Jul 2026 04:41:03 +0000 Subject: [PATCH 09/28] mm: vmscan: abort proactive reclaim early when freezing for suspend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Proactive reclaim (triggered via memory.reclaim or node sysfs) checks for pending signals in its outer loop in user_proactive_reclaim(). However, the inner reclaim loops—specifically scanning cgroups in shrink_many() and evicting/aging folios in try_to_shrink_lruvec()—can run for a long time before returning to the outer loop, especially on systems with many cgroups or large memory sizes. During system suspend, the PM freezer attempts to freeze all tasks by sending fake signals (setting TIF_SIGPENDING). Because the inner loops do not check for pending signals, the proactive reclaim task can remain stuck in kernel space for seconds, failing to enter the refrigerator in a timely manner. This leads to suspend failures due to freeze timeouts, a behavior observed on Android devices. This latency issue is specific to proactive reclaim because of its large, user-defined reclaim targets (could be gigabytes). Since commit 287d5fedb377 ("mm: memcg: use larger batches for proactive reclaim"), proactive reclaim uses larger decaying batch sizes (starting at 1/4 of the remaining target) to maintain throughput. This keeps the task in the inner reclaim loop for extended periods. In contrast, reactive reclaim (global/memcg) uses small targets (SWAP_CLUSTER_MAX, typically 32 pages), allowing it to return to the outer loop and check signals frequently. To fix this, add a signal_pending() check to should_abort_scan() for proactive reclaim paths. Since should_abort_scan() is called within the inner scanning and eviction loops, this allows proactive reclaim to abort early and return to the outer loop in user_proactive_reclaim(). Additionally, return -ERESTARTSYS instead of -EINTR in user_proactive_reclaim(). When interrupted by system suspend, returning -ERESTARTSYS allows the task to enter the refrigerator and automatically restart the syscall upon resume, making the freezer transparent to userspace. For real signals, the signal layer will either restart the syscall (if SA_RESTART is set) or return -EINTR to userspace. This fix specifically targets Multi-Gen LRU (MGLRU). Classic LRU's scan targets per iteration are strictly bounded by get_scan_count(), which ensures it returns to the outer loop more frequently. The check in should_abort_scan() is limited to proactive reclaim (sc->proactive) to avoid inadvertently affecting reactive reclaim paths, and is wrapped in unlikely() as it is a slow path. Link: https://lore.kernel.org/20260720044103.905191-1-richardycc@google.com Fixes: 287d5fedb377 ("mm: memcg: use larger batches for proactive reclaim") Fixes: 94968384dde1 ("memcg: introduce per-memcg reclaim interface") Suggested-by: Michal Hocko Suggested-by: Oleg Nesterov Signed-off-by: Richard Chang Acked-by: Michal Hocko Cc: Axel Rasmussen Cc: Barry Song Cc: David Hildenbrand Cc: Johannes Weiner Cc: Kairui Song Cc: Lorenzo Stoakes Cc: Martin Liu Cc: Minchan Kim Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: T.J. Mercier Cc: Wei Xu Cc: Yuanchu Xie Signed-off-by: Andrew Morton --- mm/vmscan.c | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 1a142c58700d..56708d1d2dfd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4926,6 +4926,9 @@ static bool should_abort_scan(struct lruvec *lruvec, struct scan_control *sc) int i; enum zone_watermarks mark; + if (unlikely(sc->proactive && signal_pending(current))) + return true; + if (sc->nr_reclaimed >= max(sc->nr_to_reclaim, compact_gap(sc->order))) return true; @@ -7906,8 +7909,15 @@ int user_proactive_reclaim(char *buf, unsigned long batch_size = (nr_to_reclaim - nr_reclaimed) / 4; unsigned long reclaimed; + /* + * Return -ERESTARTSYS to allow the freezer to interrupt the + * task. The syscall will be transparently restarted upon + * resume. For real signals, it either restarts the syscall + * (if SA_RESTART is set) or is converted to -EINTR by the + * signal layer. + */ if (signal_pending(current)) - return -EINTR; + return -ERESTARTSYS; /* * This is the final attempt, drain percpu lru caches in the From 0b45f6927a14914ff685fe0e6f9d11232a1e03df Mon Sep 17 00:00:00 2001 From: Link Lin Date: Tue, 21 Jul 2026 00:55:33 +0000 Subject: [PATCH 10/28] mm/page_reporting: use system_freezable_wq to fix UAF during suspend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit During PM freeze (e.g. S3 suspend or S4 hibernation), device drivers like virtio_balloon reset their underlying virtio devices and delete their virtqueues via vdev->config->del_vqs(). However, page reporting work (page_reporting_process) was scheduled on the global system_wq. Because system_wq lacks the WQ_FREEZABLE flag, the PM freezer skips it, leaving page_reporting_process active during suspend. If pages are freed into the buddy allocator while suspending (for example, when core MM invokes the balloon shrinker during S4 hibernation image saving), page reporting triggers virtballoon_free_page_report() on deleted virtqueues, resulting in a Use-After-Free / General Protection Fault: [ 196.795226] general protection fault, probably for non-canonical address 0xaa1436fe70dae6df: 0000 [#1] SMP NOPTI [ 196.825967] Workqueue: events page_reporting_process [ 196.831038] RIP: 0010:virtqueue_add_split+0x233/0x4c0 [virtio_ring] [ 196.927073] virtballoon_free_page_report+0x3a/0xe0 [virtio_balloon] [ 196.946943] page_reporting_process+0x370/0x4f0 Fix this by switching page reporting work to system_freezable_wq. This ensures that the PM freezer pauses page_reporting_process before device drivers destroy their reporting virtqueues. Because the reporting worker is frozen, memory reclamation/freeing (e.g. via shrinker execution) can safely return pages to MM during freeze without triggering unfrozen reporting work on deleted virtqueues. This aligns with the driver's existing design. The comment in virtballoon_freeze() states: /* * The workqueue is already frozen by the PM core before this * function is called. */ Testing: I have verified these fixes using Google’s virtualization infrastructure by running continuous suspend/resume iterations (40+ cycles) while churning memory using stress-ng (`stress-ng --vm 4 --vm-bytes 60% --timeout 1`) to constantly create free pages for the buddy allocator. We also set the `page_reporting_order` parameter to 0 to make the page reporting worker highly sensitive, forcing it to pick up any 4K free pages. This confirmed that the UAF crashes are no longer reproducible. Link: https://lore.kernel.org/20260721005603.1710551-1-linkl@google.com Fixes: 36e66c554b5c ("mm: introduce Reported pages") Signed-off-by: Link Lin Suggested-by: David Hildenbrand (Arm) Suggested-by: Michael S. Tsirkin Acked-by: David Rientjes Acked-by: David Hildenbrand (Arm) Acked-by: Michael S. Tsirkin Cc: Alexander Duyck Cc: Greg Thelen Cc: James Houghton Cc: Jason Wang Cc: Jiaqi Yan Cc: Vlastimil Babka Cc: Xuan Zhuo Cc: Signed-off-by: Andrew Morton --- mm/page_reporting.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/mm/page_reporting.c b/mm/page_reporting.c index 942e84b6908a..3e30731b940e 100644 --- a/mm/page_reporting.c +++ b/mm/page_reporting.c @@ -80,7 +80,8 @@ __page_reporting_request(struct page_reporting_dev_info *prdev) * now we are limiting this to running no more than once every * couple of seconds. */ - schedule_delayed_work(&prdev->work, PAGE_REPORTING_DELAY); + queue_delayed_work(system_freezable_wq, &prdev->work, + PAGE_REPORTING_DELAY); } /* notify prdev of free page reporting request */ @@ -340,7 +341,8 @@ err_out: */ state = atomic_cmpxchg(&prdev->state, state, PAGE_REPORTING_IDLE); if (state == PAGE_REPORTING_REQUESTED) - schedule_delayed_work(&prdev->work, PAGE_REPORTING_DELAY); + queue_delayed_work(system_freezable_wq, &prdev->work, + PAGE_REPORTING_DELAY); } static DEFINE_MUTEX(page_reporting_mutex); From 8f6f9fd93cd7a5dd607ad5cd910476dd68fff3ed Mon Sep 17 00:00:00 2001 From: Chris Gellermann Date: Wed, 22 Jul 2026 15:02:45 +0200 Subject: [PATCH 11/28] selftests/clone3: fix wild pointer access of getline due to missing init Patch series "selftests: Add missing initalization of pointer passed to getline", v2. This patch (of 2): Clone3_set_tid uses getline(&line, ...) in a loop to read the child's process status. The code expects that getline allocates the buffer for the line on the first loop iteration. According to the Open Group Spec[1], char *line has to be null pointer for this: > ssize_t getline(char **restrict lineptr, ...); > If *lineptr is a null pointer or if the object pointed to by *lineptr > is of insufficient size, an object shall be allocated as if by malloc() > or the object shall be reallocated as if by realloc()[...]. However, char *line is only declared, leading to an undefined value that is potentially non-null. In an example run with Musl v1.2.6, the realloc call[2] of getdelim, which implements getline, triggers a segfault: ./run_kselftest.sh --test clone3:clone3_set_tid [ 1366.165898] kselftest: Running tests in clone3 ... [ 1367.799244] clone3_set_tid[811]: unhandled signal 11 code 0x1 at 0x0000000000000000 in libc.so[68184,3fbf69f000+4c000] [ 1367.802808] CPU: 0 UID: 0 PID: 811 Comm: clone3_set_tid Not tainted .. [ 1367.804188] epc: 0x0000003fbf6b0184 [ 1367.804188] ra : 0x0000003fbf6d4664 [ 1367.804188] sp : 0x0000003fce5f2e40 [ 1367.805314] gp : 0x0000002aaab0dfb8 [ 1367.805314] tp : 0x0000003fbf6f14a8 [ 1367.805314] t0 : 0x0000003fbf63d000 ... Looking at the realloc implementation, Musl mallocs for a null pointer memory. But for a non-null pointer, it assumes it's passed a valid pointer to the heap and tries to access its meta-data. This leads to the segfault we see: void *realloc(void *p, size_t n) { if (!p) return malloc(n); if (size_overflows(n)) return 0; struct meta *g = get_meta(p); ... } Fix this by properly initializing the line pointer to NULL. Link: https://lore.kernel.org/20260722130246.2135563-1-christian.gellermann@codasip.com Link: https://lore.kernel.org/20260722130246.2135563-2-christian.gellermann@codasip.com Link: https://pubs.opengroup.org/onlinepubs/9799919799/functions/getline.html [1] Link: https://git.musl-libc.org/cgit/musl/tree/src/stdio/getdelim.c#n38 [2] Fixes: 41585bbeeef9 ("selftests: add tests for clone3() with *set_tid") Signed-off-by: Chris Gellermann Acked-by: David Hildenbrand (arm) Reviewed-by: Lorenzo Stoakes Cc: Christian Brauner Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Signed-off-by: Andrew Morton --- tools/testing/selftests/clone3/clone3_set_tid.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/clone3/clone3_set_tid.c b/tools/testing/selftests/clone3/clone3_set_tid.c index 5c944aee6b41..485efa7c9eed 100644 --- a/tools/testing/selftests/clone3/clone3_set_tid.c +++ b/tools/testing/selftests/clone3/clone3_set_tid.c @@ -141,7 +141,7 @@ int main(int argc, char *argv[]) { FILE *f; char buf; - char *line; + char *line = NULL; int status; int ret = -1; size_t len = 0; From 9f1d75a4ce04095afdb63d8e540092ff8151dacf Mon Sep 17 00:00:00 2001 From: Chris Gellermann Date: Wed, 22 Jul 2026 15:02:46 +0200 Subject: [PATCH 12/28] selftests/mm: fix potential wild pointer access of getline due to missing init This is another occurrence of using getline where the code assumes that getline allocates memory to store the line, but the pointer passed to it is uninitialized and potentially a non-null pointer. This violates the Open Group Spec[1] and caused a segfault in a similar situation in selftest/clone3/clone3_set_tid. Fix it by initializing the line pointer to NULL. The issue has been found by simply grepping through the selftest code after running into the issue in clone3_set_tid. Whether it segfaults in its current state is unknown to me. But it's good to be addressed due to defensive reasons. Link: https://lore.kernel.org/20260722130246.2135563-3-christian.gellermann@codasip.com Link: https://pubs.opengroup.org/onlinepubs/9799919799/functions/getline.html [1] Fixes: 26b4224d9961 ("selftests: expanding more mlock selftest") Signed-off-by: Chris Gellermann Acked-by: David Hildenbrand (arm) Reviewed-by: Lorenzo Stoakes Cc: Christian Brauner Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Shuah Khan Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Signed-off-by: Andrew Morton --- tools/testing/selftests/mm/mlock-random-test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/mm/mlock-random-test.c b/tools/testing/selftests/mm/mlock-random-test.c index 9d349c151360..16294bc7dae6 100644 --- a/tools/testing/selftests/mm/mlock-random-test.c +++ b/tools/testing/selftests/mm/mlock-random-test.c @@ -84,7 +84,7 @@ int get_proc_locked_vm_size(void) int get_proc_page_size(unsigned long addr) { FILE *smaps; - char *line; + char *line = NULL; unsigned long mmupage_size = 0; size_t size; From d3c87ab2ea5d1d12a5b945b74d8a71ed64d16e44 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 23 Jul 2026 16:16:31 +0100 Subject: [PATCH 13/28] mm/vmalloc: acquire init_mm lock on huge vmap to avoid ptdump UAF Patch series "mm: fix UAF caused by race between ptdump and vmap pgtable freeing", v6. Kernel page table walkers fall into two broad categories - those ranges where no exclusion is required via walk_kernel_page_table_range_lockless() and those where exclusion is required via walk_kernel_page_table_range() or walk_page_range_debug(). The former category is used only by arm64 arch code operating on ranges it both wholly owns and does not concurrently write. The latter category consists of kernel page table walkers operating on ranges that are wholly owned (but which need exclusion against concurrent writers). The lock used for exclusion is the mmap lock, and for kernel ranges this is the mmap lock on init_mm. ptdump is a special case being both the only user of walk_page_range_debug(), and the only case in which it walks ranges it does not own. This presents a problem, as page tables may be freed under ptdump. And indeed there is a use-after-free bug in the kernel as a result, which this series addresses. vmap promotes page tables to huge leaf entries where possible, freeing the lower page table when it does. It does this with no meaningful locks held against concurrent ptdump walks. As a result, use-after-free can currently occur. This series addresses the issue by having the vmap huge promotion logic acquire the mmap read lock while both setting the huge page table entry and freeing the prior leaf page table. The ptdump code already acquires the mmap write lock, so by doing so we ensure that the ptdump walker only ever observes either the huge page table entry or the existing page table entry, and nothing is freed underneath it. A mitigation for this issue was already applied for arm64 in commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), which this series has to deal with carefully. This mitigation resolves the issue by acquiring the mmap read lock on init_mm on vmap page table free if a ptdump is in progress. However the fix in this series would cause a deadlock if we were to simply apply it for arm64 without also reverting the change. This is because vmap may acquire the read lock before ptdump attempts to acquire the write lock, which then gets queued, and rwsem starvation rules mean that the (unacknowledged) nested mmap read lock in the arm64 code would also block, meaning the original read lock is never released and thus deadlock. This series works around this by #ifndef CONFIG_ARM64'ing the mmap read lock in vmap logic, then partially reverting commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), keeping the enablement of huge vmap support, and removing the ifdeffery with the partial revert patch. There are related issues that are also addressed in this series: * x86 page attribute logic, specifically Change Page Attributes (CPA), implements a feature whereby huge ranges can be collapsed into huge leaf entries. This can similarly cause a UAF when done in parallel with a ptdump walk, so similarly acquire the init_mm mmap lock to avoid this. * The CPA logic allows concurrent page table manipulation and CPA collapse, meaning the former risks accessing a page table the latter frees. Fix this by acquiring mmap write lock on init_mm across the whole CPA collapse operation and read lock on the page table manipulation. * x86 and arm64 permit walks of non-kernel mm's (both allowing efi mm walks, and in x86's case arbitrary mm's), so we ensure kernel mappings remain stable by locking the init_mm as well as the mm being walked. The ordering of patches is established for both strict dependencies (the arm64 partial revert in particular has to be done after the vmap changes) and logical ones (the non-kernel mm fix only makes sense once the vmap/CPA fixes are in place). This patch (of 3): Currently there is a nasty race between ptdump and vmap when attempting to map a huge P4D, PUD or PMD entry: * ptdump walks kernel page table ranges it doesn't own. * When vmap maps ranges it tries to promotes existing ones to huge page tables in vmap_try_huge_[p4d,pud,pmd]() at P4D, PUD and PMD level, freeing the lower page table in [p4d,pud,pmd]_free_[pud,pmd,pte]_page() when it succeeds. Both of these things can happen at the same time and as a result ptdump can access a freed page table, resulting in a use-after-free and memory corruption. This is possible because while ptdump_walk_pgd() holds both the mem hotplug lock and the mmap write lock before invoking walk_page_range_debug(), vmap takes no relevant locks at all. Fix this by holding the mmap read lock in vmap_try_huge_*() when freeing page tables. The read lock is sufficient: ptdump is the only walker that must be excluded and it holds the mmap write lock. Other holders of the read lock may run concurrently, but each exclusively owns the range it operates on and cannot reach the page tables freed here. We also hold the lock while assigning the huge page table entry, which means page table walkers observe only the huge or non-huge page table entry. We use a trylock to prevent ptdump from blocking vmap making forward progress. This is fine because it's an optimisation in any case, and thus the vmap can safely proceed regardless. All other kernel page table walkers that touch vmalloc ranges either exclusively own the memory walked or acquire the mmap lock, so this correctly excludes those walkers. One wrinkle here is commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), which addresses the issue for arm64 only by explicitly acquiring the mmap read lock on kernel page table freeing should a concurrent ptdump be in progress. This is problematic as vmap may acquire the mmap read lock prior to ptdump attempting to acquire an mmap write lock, leading to a deadlock when the mmap read lock is slept upon on page table freeing due to rwsem anti-starvation. We work around this by predicating the mmap lock being taken on !CONFIG_ARM64 for the time being. With this patch applied, a follow up will partially revert commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump") and at that stage remove the arm64 ifdeffery. We also update walk_page_range_debug() to assert the mmap write lock unconditionally and update the comment here to reflect this change. The issue has existed as long as ptdump was available and vmap freed page tables when promoting to a huge leaf entry, that is, since commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table") for huge ioremap, and commit 121e6f3258fe ("mm/vmalloc: hugepage vmalloc mappings") for huge vmalloc. Since the former is the earlier of the two we choose that for our Fixes tag. We also define a guard class for mmap_read_trylock() so we can use cleanup.h to make the scope handling cleaner in the implementation. This patch is based on work by David Carlier (linked), with gratitude! Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-0-8cc77dcc0018@kernel.org Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-1-8cc77dcc0018@kernel.org Fixes: b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table") Signed-off-by: Lorenzo Stoakes (ARM) Reported-by: syzbot+fd95a72470f5a44e464c@syzkaller.appspotmail.com Closes: https://lore.kernel.org/all/6a287988.39669fcc.33b062.00a0.GAE@google.com/T/ Link: https://lore.kernel.org/linux-mm/20260706203128.162335-1-devnexen@gmail.com/ Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Dev Jain Acked-by: David Hildenbrand (Arm) Reviewed-by: Kiryl Shutsemau Cc: Cc: Andy Lutomirski Cc: "Borah, Chaitanya Kumar" Cc: "Borislav Petkov (AMD)" Cc: Catalin Marinas Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Liam R. Howlett Cc: Michal Hocko Cc: Peter Zijlstra Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Toshi Kani Cc: "Uladzislau Rezki (Sony)" Cc: Vlastimil Babka Cc: Will Deacon Signed-off-by: Andrew Morton --- include/linux/mmap_lock.h | 1 + mm/pagewalk.c | 22 ++++++++++-------- mm/vmalloc.c | 49 ++++++++++++++++++++++++++++++++------- 3 files changed, 53 insertions(+), 19 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 04b8f61ece5d..6b5c2390cc30 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -621,6 +621,7 @@ static inline void mmap_read_unlock(struct mm_struct *mm) DEFINE_GUARD(mmap_read_lock, struct mm_struct *, mmap_read_lock(_T), mmap_read_unlock(_T)) +DEFINE_GUARD_COND(mmap_read_lock, _try, mmap_read_trylock(_T)) static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) { diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 3ae2586ff45b..bbcfd68d0907 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -678,6 +678,8 @@ int walk_kernel_page_table_range_lockless(unsigned long start, unsigned long end * will also not lock the PTEs for the pte_entry() callback. * * This is for debugging purposes ONLY. + * + * The mmap write lock must be held. */ int walk_page_range_debug(struct mm_struct *mm, unsigned long start, unsigned long end, const struct mm_walk_ops *ops, @@ -691,6 +693,16 @@ int walk_page_range_debug(struct mm_struct *mm, unsigned long start, .no_vma = true }; + /* + * When walking userland page tables, an mmap write lock must be held to + * account for munmap() downgrading to an mmap read lock when tearing + * down page tables. + * + * When walking kernel page tables, an mmap write lock must also be held + * to account for page table freeing on vmap huge page mapping. + */ + mmap_assert_write_locked(mm); + /* For convenience, we allow traversal of kernel mappings. */ if (mm == &init_mm) return walk_kernel_page_table_range(start, end, ops, @@ -700,16 +712,6 @@ int walk_page_range_debug(struct mm_struct *mm, unsigned long start, if (!check_ops_safe(ops)) return -EINVAL; - /* - * The mmap lock protects the page walker from changes to the page - * tables during the walk. However a read lock is insufficient to - * protect those areas which don't have a VMA as munmap() detaches - * the VMAs before downgrading to a read lock and actually tearing - * down PTEs/page tables. In which case, the mmap write lock should - * be held. - */ - mmap_assert_write_locked(mm); - return walk_pgd_range(start, end, &walk); } diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 1afca3568b9b..d5c4d2bb770b 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -43,6 +43,7 @@ #include #include #include +#include #define CREATE_TRACE_POINTS #include @@ -158,10 +159,24 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, if (!IS_ALIGNED(phys_addr, PMD_SIZE)) return 0; - if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) - return 0; + if (!pmd_present(*pmd)) + return pmd_set_huge(pmd, phys_addr, prot); - return pmd_set_huge(pmd, phys_addr, prot); + /* + * Acquire the mmap read lock to exclude ptdump, which walks + * kernel page tables it does not own under the mmap write lock. + * + * Concurrent read lock holders are safe: each exclusively owns + * the range it operates on and cannot reach this page table. + */ +#ifndef CONFIG_ARM64 + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) +#endif + { + if (!pmd_free_pte_page(pmd, addr)) + return 0; + return pmd_set_huge(pmd, phys_addr, prot); + } } static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, @@ -210,10 +225,18 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, if (!IS_ALIGNED(phys_addr, PUD_SIZE)) return 0; - if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) - return 0; + if (!pud_present(*pud)) + return pud_set_huge(pud, phys_addr, prot); - return pud_set_huge(pud, phys_addr, prot); + /* See comment in vmap_try_huge_pmd(). */ +#ifndef CONFIG_ARM64 + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) +#endif + { + if (!pud_free_pmd_page(pud, addr)) + return 0; + return pud_set_huge(pud, phys_addr, prot); + } } static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, @@ -262,10 +285,18 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, if (!IS_ALIGNED(phys_addr, P4D_SIZE)) return 0; - if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) - return 0; + if (!p4d_present(*p4d)) + return p4d_set_huge(p4d, phys_addr, prot); - return p4d_set_huge(p4d, phys_addr, prot); + /* See comment in vmap_try_huge_pmd(). */ +#ifndef CONFIG_ARM64 + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) +#endif + { + if (!p4d_free_pud_page(p4d, addr)) + return 0; + return p4d_set_huge(p4d, phys_addr, prot); + } } static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, From 2460a5817105d5b21606d6fa86fc27ecb447c982 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 23 Jul 2026 16:16:34 +0100 Subject: [PATCH 14/28] mm/ptdump: always stabilise against page table freeing using init_mm Previous commits have established the invariant that kernel page table freeing is performed while an mmap read lock on init_mm is held, which fixes races between ptdump and kernel page table freeing over init_mm. However, x86 and arm64 can perform a ptdump over an mm other than init_mm via ptdump_walk_pgd() and since kernel memory ranges are shared across non-kernel mm's, this means that the race still exists for these cases. Fix this by acquiring a nested mmap write lock for init_mm in ptdump_walk_pgd(). This is safe as we take this after mmap write locking the mm, and nothing acquires the init_mm lock first before locking an arbitrary mm, so no deadlock is possible. Also update walk_page_range_debug() to assert that init_mm is write locked, add a comment explaining why and remove some redundant code, and eliminate the unnecessary and confusing invocation of walk_kernel_page_table_range(). We can safely remove the non-NULL check for walk.mm, as the mmap lock asserts would NULL pointer deref if it was (and of course no callers do this). The first point at which ptdump can race kernel page table freeing is commit b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table"), so we target this in the Fixes tag. Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-4-8cc77dcc0018@kernel.org Fixes: b6bdb7517c3d ("mm/vmalloc: add interfaces to free unmapped page table") Signed-off-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Acked-by: David Hildenbrand (Arm) Reviewed-by: Kiryl Shutsemau Cc: Cc: Andy Lutomirski Cc: "Borah, Chaitanya Kumar" Cc: "Borislav Petkov (AMD)" Cc: Catalin Marinas Cc: Dave Hansen Cc: David Carlier Cc: Dev Jain Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Liam R. Howlett Cc: Michal Hocko Cc: Peter Zijlstra Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Toshi Kani Cc: "Uladzislau Rezki (Sony)" Cc: Vlastimil Babka Cc: Will Deacon Signed-off-by: Andrew Morton --- mm/pagewalk.c | 14 +++++++++----- mm/ptdump.c | 7 +++++++ 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index bbcfd68d0907..5d87c632a255 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -702,12 +702,16 @@ int walk_page_range_debug(struct mm_struct *mm, unsigned long start, * to account for page table freeing on vmap huge page mapping. */ mmap_assert_write_locked(mm); + /* + * x86, arm64 ptdump allow walks of efi mm's and x86 ptdump allows walks + * of arbitrary mm's. + * + * However, they both must also hold the init_mm lock to account for + * concurrent kernel page table freeing. + */ + mmap_assert_write_locked(&init_mm); - /* For convenience, we allow traversal of kernel mappings. */ - if (mm == &init_mm) - return walk_kernel_page_table_range(start, end, ops, - pgd, private); - if (start >= end || !walk.mm) + if (start >= end) return -EINVAL; if (!check_ops_safe(ops)) return -EINVAL; diff --git a/mm/ptdump.c b/mm/ptdump.c index 973020000096..5851096e6f65 100644 --- a/mm/ptdump.c +++ b/mm/ptdump.c @@ -178,11 +178,18 @@ void ptdump_walk_pgd(struct ptdump_state *st, struct mm_struct *mm, pgd_t *pgd) get_online_mems(); mmap_write_lock(mm); + /* To stabilise kernel page tables we must hold the init_mm lock too. */ + if (mm != &init_mm) + mmap_write_lock_nested(&init_mm, SINGLE_DEPTH_NESTING); + while (range->start != range->end) { walk_page_range_debug(mm, range->start, range->end, &ptdump_ops, pgd, st); range++; } + + if (mm != &init_mm) + mmap_write_unlock(&init_mm); mmap_write_unlock(mm); put_online_mems(); From 0f9c62f23a6fc32a063f1206fc1aa303737b47de Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 23 Jul 2026 16:16:35 +0100 Subject: [PATCH 15/28] arm64: remove redundant concurrent ptdump UAF mitigation This partially reverts commit fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump"), retaining vmalloc-huge support but eliminating the now redundant mitigation against a race between huge vmap page table freeing and ptdump, as this issue has now been fixed at core. We also simultaneously remove the arm64 if-deffery when acquiring the mmap read lock upon vmap huge page table promotion as it is no longer required. Note that this patch relies on the preceding vmalloc patch, and should not be backported alone. Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-5-8cc77dcc0018@kernel.org Fixes: fa93b45fd397 ("arm64: Enable vmalloc-huge with ptdump") Signed-off-by: Lorenzo Stoakes (ARM) Reviewed-by: Dev Jain Acked-by: Mike Rapoport (Microsoft) Acked-by: Kiryl Shutsemau (Meta) Acked-by: Will Deacon Reviewed-by: David Hildenbrand (Arm) Cc: Cc: Andy Lutomirski Cc: "Borah, Chaitanya Kumar" Cc: "Borislav Petkov (AMD)" Cc: Catalin Marinas Cc: Dave Hansen Cc: David Carlier Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Liam R. Howlett Cc: Michal Hocko Cc: Peter Zijlstra Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Toshi Kani Cc: "Uladzislau Rezki (Sony)" Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- arch/arm64/include/asm/ptdump.h | 2 -- arch/arm64/mm/mmu.c | 43 +++------------------------------ arch/arm64/mm/ptdump.c | 11 ++------- mm/vmalloc.c | 15 +++--------- 4 files changed, 9 insertions(+), 62 deletions(-) diff --git a/arch/arm64/include/asm/ptdump.h b/arch/arm64/include/asm/ptdump.h index 5b374a6ab34a..50a195eda8ed 100644 --- a/arch/arm64/include/asm/ptdump.h +++ b/arch/arm64/include/asm/ptdump.h @@ -7,8 +7,6 @@ #include -DECLARE_STATIC_KEY_FALSE(arm64_ptdump_lock_key); - #ifdef CONFIG_PTDUMP #include diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 18a8b0d3714e..d4de88770ecf 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -49,8 +49,6 @@ #define NO_CONT_MAPPINGS BIT(1) #define NO_EXEC_MAPPINGS BIT(2) /* assumes FEAT_HPDS is not used */ -DEFINE_STATIC_KEY_FALSE(arm64_ptdump_lock_key); - u64 kimage_voffset __ro_after_init; EXPORT_SYMBOL(kimage_voffset); @@ -1864,8 +1862,7 @@ int pmd_clear_huge(pmd_t *pmdp) return 1; } -static int __pmd_free_pte_page(pmd_t *pmdp, unsigned long addr, - bool acquire_mmap_lock) +int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr) { pte_t *table; pmd_t pmd; @@ -1877,25 +1874,13 @@ static int __pmd_free_pte_page(pmd_t *pmdp, unsigned long addr, return 1; } - /* See comment in pud_free_pmd_page for static key logic */ table = pte_offset_kernel(pmdp, addr); pmd_clear(pmdp); __flush_tlb_kernel_pgtable(addr); - if (static_branch_unlikely(&arm64_ptdump_lock_key) && acquire_mmap_lock) { - mmap_read_lock(&init_mm); - mmap_read_unlock(&init_mm); - } - pte_free_kernel(NULL, table); return 1; } -int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr) -{ - /* If ptdump is walking the pagetables, acquire init_mm.mmap_lock */ - return __pmd_free_pte_page(pmdp, addr, /* acquire_mmap_lock = */ true); -} - int pud_free_pmd_page(pud_t *pudp, unsigned long addr) { pmd_t *table; @@ -1911,36 +1896,16 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr) } table = pmd_offset(pudp, addr); - - /* - * Our objective is to prevent ptdump from reading a PMD table which has - * been freed. In this race, if pud_free_pmd_page observes the key on - * (which got flipped by ptdump) then the mmap lock sequence here will, - * as a result of the mmap write lock/unlock sequence in ptdump, give - * us the correct synchronization. If not, this means that ptdump has - * yet not started walking the pagetables - the sequence of barriers - * issued by __flush_tlb_kernel_pgtable() guarantees that ptdump will - * observe an empty PUD. - */ - pud_clear(pudp); - __flush_tlb_kernel_pgtable(addr); - if (static_branch_unlikely(&arm64_ptdump_lock_key)) { - mmap_read_lock(&init_mm); - mmap_read_unlock(&init_mm); - } - pmdp = table; next = addr; end = addr + PUD_SIZE; do { if (pmd_present(pmdp_get(pmdp))) - /* - * PMD has been isolated, so ptdump won't see it. No - * need to acquire init_mm.mmap_lock. - */ - __pmd_free_pte_page(pmdp, next, /* acquire_mmap_lock = */ false); + pmd_free_pte_page(pmdp, next); } while (pmdp++, next += PMD_SIZE, next != end); + pud_clear(pudp); + __flush_tlb_kernel_pgtable(addr); pmd_free(NULL, table); return 1; } diff --git a/arch/arm64/mm/ptdump.c b/arch/arm64/mm/ptdump.c index 1c20144700d7..5a76c59b5ada 100644 --- a/arch/arm64/mm/ptdump.c +++ b/arch/arm64/mm/ptdump.c @@ -283,13 +283,6 @@ void note_page_flush(struct ptdump_state *pt_st) note_page(pt_st, 0, -1, pte_val(pte_zero)); } -static void arm64_ptdump_walk_pgd(struct ptdump_state *st, struct mm_struct *mm) -{ - static_branch_inc(&arm64_ptdump_lock_key); - ptdump_walk_pgd(st, mm, NULL); - static_branch_dec(&arm64_ptdump_lock_key); -} - void ptdump_walk(struct seq_file *s, struct ptdump_info *info) { unsigned long end = ~0UL; @@ -318,7 +311,7 @@ void ptdump_walk(struct seq_file *s, struct ptdump_info *info) } }; - arm64_ptdump_walk_pgd(&st.ptdump, info->mm); + ptdump_walk_pgd(&st.ptdump, info->mm, NULL); } static void __init ptdump_initialize(void) @@ -360,7 +353,7 @@ bool ptdump_check_wx(void) } }; - arm64_ptdump_walk_pgd(&st.ptdump, &init_mm); + ptdump_walk_pgd(&st.ptdump, &init_mm, NULL); if (st.wx_pages || st.uxn_pages) { pr_warn("Checked W+X mappings: FAILED, %lu W+X pages found, %lu non-UXN pages found\n", diff --git a/mm/vmalloc.c b/mm/vmalloc.c index d5c4d2bb770b..f4fa227a8d7f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -169,10 +169,7 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, * Concurrent read lock holders are safe: each exclusively owns * the range it operates on and cannot reach this page table. */ -#ifndef CONFIG_ARM64 - scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) -#endif - { + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) { if (!pmd_free_pte_page(pmd, addr)) return 0; return pmd_set_huge(pmd, phys_addr, prot); @@ -229,10 +226,7 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, return pud_set_huge(pud, phys_addr, prot); /* See comment in vmap_try_huge_pmd(). */ -#ifndef CONFIG_ARM64 - scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) -#endif - { + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) { if (!pud_free_pmd_page(pud, addr)) return 0; return pud_set_huge(pud, phys_addr, prot); @@ -289,10 +283,7 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, return p4d_set_huge(p4d, phys_addr, prot); /* See comment in vmap_try_huge_pmd(). */ -#ifndef CONFIG_ARM64 - scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) -#endif - { + scoped_cond_guard(mmap_read_lock_try, return 0, &init_mm) { if (!p4d_free_pud_page(p4d, addr)) return 0; return p4d_set_huge(p4d, phys_addr, prot); From 22972069f297d39e7fee3fe800f17d8181f3949a Mon Sep 17 00:00:00 2001 From: Zhiling Zou Date: Thu, 23 Jul 2026 00:48:13 +0800 Subject: [PATCH 16/28] mm/page_table_check: skip special zero mappings page_table_check_set() and page_table_check_clear() account mappings based on PageAnon(). Shared zero-page PTEs and huge zero PMDs are special mappings, but page_table_check can still account them as file-backed pages. An unprivileged process can populate enough zero mappings to overflow file_map_count and hit the existing BUG_ON(). The PTE path can do this with the shared zero page, and the PMD path can do the same with huge zero mappings. Skip special zero mappings in the user page-table accounting paths. Keep the PTE-side pte_special() check, and identify huge zero PMDs from the mapped folio instead of pmd_special(). That covers architectures where pmd_special() is a no-op without adding huge_zero_pfn checks to the generic counter helpers. Link: https://lore.kernel.org/cover.1784717203.git.zhilinz@nebusec.ai Link: https://lore.kernel.org/e94478e4fb7912fb7e8ebebed5ce85d00dc9a69d.1784717203.git.zhilinz@nebusec.ai Fixes: df4e817b7108 ("mm: page table check") Signed-off-by: Zhiling Zou Signed-off-by: Ren Wei Reported-by: Vega Cc: Pasha Tatashin Assisted-by: Codex:gpt-5.4 Cc: Signed-off-by: Andrew Morton --- mm/page_table_check.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 53a8997ec043..2403f5a11410 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -151,18 +151,29 @@ void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr, if (&init_mm == mm) return; - if (pte_user_accessible_page(mm, addr, pte)) + if (pte_user_accessible_page(mm, addr, pte) && !pte_special(pte)) page_table_check_clear(pte_pfn(pte), PAGE_SIZE >> PAGE_SHIFT); } EXPORT_SYMBOL(__page_table_check_pte_clear); +static inline bool page_table_check_huge_zero_pmd(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + if (!pfn_valid(pfn)) + return false; + + return is_huge_zero_folio(page_folio(pfn_to_page(pfn))); +} + void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr, pmd_t pmd) { if (&init_mm == mm) return; - if (pmd_user_accessible_page(mm, addr, pmd)) + if (pmd_user_accessible_page(mm, addr, pmd) && + !page_table_check_huge_zero_pmd(pmd)) page_table_check_clear(pmd_pfn(pmd), PMD_SIZE >> PAGE_SHIFT); } EXPORT_SYMBOL(__page_table_check_pmd_clear); @@ -208,7 +219,7 @@ void __page_table_check_ptes_set(struct mm_struct *mm, unsigned long addr, for (i = 0; i < nr; i++) __page_table_check_pte_clear(mm, addr + PAGE_SIZE * i, ptep_get(ptep + i)); - if (pte_user_accessible_page(mm, addr, pte)) + if (pte_user_accessible_page(mm, addr, pte) && !pte_special(pte)) page_table_check_set(pte_pfn(pte), nr, pte_write(pte)); } EXPORT_SYMBOL(__page_table_check_ptes_set); @@ -238,7 +249,8 @@ void __page_table_check_pmds_set(struct mm_struct *mm, unsigned long addr, for (i = 0; i < nr; i++) __page_table_check_pmd_clear(mm, addr + PMD_SIZE * i, *(pmdp + i)); - if (pmd_user_accessible_page(mm, addr, pmd)) + if (pmd_user_accessible_page(mm, addr, pmd) && + !page_table_check_huge_zero_pmd(pmd)) page_table_check_set(pmd_pfn(pmd), stride * nr, pmd_write(pmd)); } EXPORT_SYMBOL(__page_table_check_pmds_set); From aa4f036339bf36ea61859837b9875fc284f0c60b Mon Sep 17 00:00:00 2001 From: Matt Fleming Date: Sat, 25 Jul 2026 11:14:19 +0100 Subject: [PATCH 17/28] mm/huge_memory: initialise workingset state before folio split xas_try_split() adds __GFP_ACCOUNT for page-cache xa_nodes, but __folio_split() leaves the xa_state's xa_lru unset. That lets a live, memcg-charged xa_node exist without being linked into the mapping's shadow_nodes list_lru; when reclaim later walks the list_lru it trips VM_WARN_ON(!css_is_dying()). Use mapping_set_update() to install both the workingset update callback and the shadow_nodes list_lru on the xa_state. Link: https://lore.kernel.org/20260725101419.3938406-1-matt@readmodwrite.com Fixes: 58729c04cf10 ("mm/huge_memory: add buddy allocator like (non-uniform) folio_split()") Signed-off-by: Matt Fleming Reported-by: syzbot+c5b060ce82921a2fd500@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=c5b060ce82921a2fd500 Reviewed-by: Zi Yan Acked-by: David Hildenbrand (Arm) Cc: Baolin Wang Cc: Barry Song Cc: Dave Chinner Cc: Dev Jain Cc: Kairui Song Cc: Lance Yang Cc: Liam Howlett Cc: Lorenzo Stoakes Cc: Matthew Wilcox (Oracle) Cc: Muchun Song Cc: Nico Pache Cc: Roman Gushchin Cc: Ryan Roberts Cc: Shakeel Butt Cc: Signed-off-by: Andrew Morton --- mm/huge_memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 58cabe6af33d..8c1f35df0e91 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -4033,7 +4033,7 @@ static int __folio_split(struct folio *folio, unsigned int new_order, gfp_t gfp; mapping = folio->mapping; - min_order = mapping_min_folio_order(folio->mapping); + min_order = mapping_min_folio_order(mapping); if (new_order < min_order) { ret = -EINVAL; goto out; @@ -4047,6 +4047,8 @@ static int __folio_split(struct folio *folio, unsigned int new_order, goto out; } + mapping_set_update(&xas, mapping); + if (split_type == SPLIT_TYPE_UNIFORM) { xas_set_order(&xas, folio->index, new_order); xas_split_alloc(&xas, folio, old_order, gfp); From 53aec7e23cace344767860c498cbad7ad0a88495 Mon Sep 17 00:00:00 2001 From: liyouhong Date: Sun, 26 Jul 2026 09:48:15 +0800 Subject: [PATCH 18/28] mm/damon/ops-common: putback folios on invalid migrate nid damon_pa_migrate() and damos_va_migrate() isolate folios into a local list and then call damon_migrate_pages(). When target_nid is invalid (including the scheme default NUMA_NO_NODE / -1), damon_migrate_pages() returns early without putting the folios back to the LRU. Callers then discard the list head while those folios remain isolated with an extra reference taken by folio_isolate_lru(). The pages stay off the LRU for as long as the mapping exists (anon active+inactive counts drop while RSS does not), and the leftover references can pin the pages after the mapping is gone. Put the folios back on the invalid-nid path so ignored migration requests still return them to the LRU. Link: https://lore.kernel.org/20260726014815.1280757-1-dayou5941@163.com Fixes: 7e6c3130690a ("mm/damon/ops-common: ignore migration request to invalid nodes") Assisted-by: Cursor:grok-4.5 Reviewed-by: SJ Park Signed-off-by: liyouhong Cc: Signed-off-by: Andrew Morton --- mm/damon/ops-common.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index 6bdd1cfd3863..9c178c175832 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -392,8 +392,15 @@ unsigned long damon_migrate_pages(struct list_head *folio_list, int target_nid) return nr_migrated; if (target_nid < 0 || target_nid >= MAX_NUMNODES || - !node_state(target_nid, N_MEMORY)) + !node_state(target_nid, N_MEMORY)) { + while (!list_empty(folio_list)) { + struct folio *folio = lru_to_folio(folio_list); + + list_del(&folio->lru); + folio_putback_lru(folio); + } return nr_migrated; + } noreclaim_flag = memalloc_noreclaim_save(); From d4db0b9e162c630cefd74783f2b81ab85d2e46c5 Mon Sep 17 00:00:00 2001 From: Sourav Panda Date: Sun, 26 Jul 2026 07:29:34 +0000 Subject: [PATCH 19/28] mm/hugetlb_cma: dix null nodemask dereference in hugetlb_cma_alloc_frozen_folio alloc_buddy_hugetlb_folio_with_mpol() can pass a NULL nodemask to alloc_fresh_hugetlb_folio() as a fallback to allocate from all nodes. If order is gigantic, alloc_fresh_hugetlb_folio() propagates the NULL nodemask down to hugetlb_cma_alloc_frozen_folio() via alloc_gigantic_frozen_folio(). hugetlb_cma_alloc_frozen_folio() blindly dereferences the nodemask in node_isset(nid, *nodemask) and for_each_node_mask(node, *nodemask), leading to a null pointer dereference kernel panic. Fix this by checking if nodemask is NULL in hugetlb_cma_alloc_frozen_folio() and defaulting it to node_states[N_MEMORY]. This allows hugetlb_cma allocations to fall back to any node with memory, keeping behavior consistent with alloc_contig_frozen_pages() and alloc_buddy_frozen_folio(). From a userspace perspective, this bug allows an unprivileged user to crash the kernel (trigger a panic) by requesting a gigantic hugepage allocation with MPOL_PREFERRED_MANY on a system where CMA is only configured on a subset of NUMA nodes. This can be reproduced by booting a VM with two NUMA nodes, restricting CMA to Node 1 (e.g., hugetlb_cma=1:1G default_hugepagesz=1G hugepagesz=1G hugepages=0), and running a program that allocates a 1GB hugepage area without reserving, restricts allocation to Node 0 using mbind() with MPOL_PREFERRED_MANY, and triggers a page fault: void *ptr = mmap(NULL, 1UL << 30, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_HUGE_1GB | MAP_NORESERVE, -1, 0); unsigned long nodemask = 1; /* Node 0 */ mbind(ptr, 1UL << 30, MPOL_PREFERRED_MANY, &nodemask, sizeof(nodemask) * 8, 0); memset(ptr, 0, 1UL << 30); /* Trigger fault */ This results in a NULL pointer dereference: BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page Oops: Oops: 0000 [#1] SMP NOPTI RIP: 0010:hugetlb_cma_alloc_frozen_folio+0x75/0x120 Call Trace: only_alloc_fresh_hugetlb_folio.isra.0+0x2c/0x160 alloc_surplus_hugetlb_folio+0x6d/0x100 alloc_hugetlb_folio+0x3c5/0x660 hugetlb_no_page+0x3d9/0x650 Link: https://lore.kernel.org/20260726072935.3513996-1-souravpanda@google.com Fixes: eb02f14c4a2b ("mm/hugetlb: allow overcommitting gigantic hugepages") Signed-off-by: Sourav Panda Cc: Kefeng Wang Cc: Shakeel Butt Cc: Usama Arif Cc: David Hildenbrand Cc: Frank van der Linden Cc: Greg Thelen Cc: Muchun Song Cc: Oscar Salvador Cc: Suren Baghdasaryan Cc: Signed-off-by: Andrew Morton --- mm/hugetlb_cma.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c index 39344d6c78d8..5744de0ceeb7 100644 --- a/mm/hugetlb_cma.c +++ b/mm/hugetlb_cma.c @@ -34,7 +34,10 @@ struct folio *hugetlb_cma_alloc_frozen_folio(int order, gfp_t gfp_mask, if (!hugetlb_cma_size) return NULL; - if (hugetlb_cma[nid]) + if (!nodemask) + nodemask = &node_states[N_MEMORY]; + + if (hugetlb_cma[nid] && node_isset(nid, *nodemask)) page = cma_alloc_frozen_compound(hugetlb_cma[nid], order); if (!page && !(gfp_mask & __GFP_THISNODE)) { From f28d46d81aaa89154ab9ae1a1218bb52173d04d5 Mon Sep 17 00:00:00 2001 From: Ramin Moussavi Date: Mon, 27 Jul 2026 23:58:23 +0200 Subject: [PATCH 20/28] microblaze: restore the page alignment of swapper_pg_dir microblaze handles TLB misses in software, and the handler builds the address of the L1 entry by ORing the index into the page directory base instead of adding it (hw_exception_handler.S): bsrli r5, r3, PGDIR_SHIFT - 2 andi r5, r5, PAGE_SIZE - 4 /* Assume pgdir aligned on 4K boundary, no need for "andi r4,r4,0xfffff003" */ or r4, r4, r5 lwi r4, r4, 0 /* Get L1 entry */ The index is masked to the low 12 bits, so the OR only works if those bits of the base are zero -- which is exactly the assumption the comment states and the reason the masking of the base can be skipped. swapper_pg_dir had no alignment directive of its own. It was aligned because it followed empty_zero_page in head.S, and that one carried the .align 12: .section .data .global empty_zero_page .align 12 empty_zero_page: .space PAGE_SIZE .global swapper_pg_dir swapper_pg_dir: .space PAGE_SIZE Commit 6215d9f4470f ("arch, mm: consolidate empty_zero_page") removed empty_zero_page from head.S, and with it the .align 12 that -- despite sitting next to empty_zero_page -- was what page aligned swapper_pg_dir. Since then swapper_pg_dir lands wherever .data happens to put it, its low bits are no longer zero, and every kernel TLB miss ORs the index into a base with a nonzero offset. The resulting L1 lookups read the wrong words, no valid translation is ever installed, and the kernel spins in exceptions long before it can print anything. On qemu-system-microblazeel (petalogix-s3adsp1800) the console stays completely silent at 100% CPU; there is no oops and no guest error reported by qemu, which makes this awkward to diagnose. Give swapper_pg_dir the alignment it requires, rather than relying on a neighbour to provide it. microblaze has no noMMU variant left in mainline -- CONFIG_MMU is def_bool y and mmu_defconfig is the only defconfig -- so this is not a corner case: every mainline microblaze kernel since v7.1-rc1 fails to boot, including the v7.1 release. v7.0: swapper_pg_dir = 0xc05fd000 (aligned) v7.1-rc1: swapper_pg_dir = 0xc0603140 (offset 320) v7.1-rc1 + this fix: swapper_pg_dir = 0xc0604000 (aligned) next-20260726: swapper_pg_dir = 0xc0615140 (offset 320) next-20260726 + this fix: swapper_pg_dir = 0xc0616000 (aligned) Verified on qemu-system-microblazeel (petalogix-s3adsp1800) with mmu_defconfig and microblazeel gcc 12.5.0: v7.1-rc1 and next-20260726 both print nothing at all without the fix, and both boot to userspace with it. Link: https://lore.kernel.org/20260727215823.1422701-1-ramin.moussavi@yacoub.de Fixes: 6215d9f4470f ("arch, mm: consolidate empty_zero_page") Signed-off-by: Ramin Moussavi Cc: Michal Simek Cc: Mike Rapoport Cc: Signed-off-by: Andrew Morton --- arch/microblaze/kernel/head.S | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/microblaze/kernel/head.S b/arch/microblaze/kernel/head.S index 808019c3b7ac..9bd3e513c89b 100644 --- a/arch/microblaze/kernel/head.S +++ b/arch/microblaze/kernel/head.S @@ -39,6 +39,8 @@ #include .section .data +/* The MMU requires a page aligned page directory. */ +.align 12 .global swapper_pg_dir swapper_pg_dir: .space PAGE_SIZE From ddf221b26fa162790f111911352c7e38c2a8931f Mon Sep 17 00:00:00 2001 From: Hugh Dickins Date: Mon, 27 Jul 2026 22:24:14 -0700 Subject: [PATCH 21/28] mm/filemap: __filemap_add_folio() restore index before retrying In __filemap_add_folio()'s split-a-conflict loop, xas_set_order() is applied repeatedly: each application modifies xas.xa_index, rounding it down according to the split_order attempted at that stage: and if all goes as intended, it eventually (or immediately) converges on an xas_try_split() to the required folio_order, with xas.xa_index now the same as index: then xas_store() puts the new folio into the xarray there. But if a new node was needed, and GFP_NOWAIT allocation did not get one, the lock is dropped, xas_nomem() used to allocate, and sequence retried. If (that part of) the xarray is unchanged when the lock is reacquired, no problem. But what if the conflict was meanwhile resolved by another thread (perhaps even doing the same thing, inserting a folio at that same index)? Isn't there a danger of now putting our folio into the xarray at an intermediate rounded-down index? With !folio_contains() bug to follow, when CONFIG_DEBUG_VM=y is checking for that. Fix this with an xas_set_order() to restore the original xas.xa_index at the bottom of the loop, so the retry does a full re-evaluation after reacquiring the lock, and cannot reach xas_store() with the wrong index. Production was suffering from rare SIGILLs and SIGSEGVs, executable text found a page away from where it belonged, !folio_contains() bug hit when debug enabled: symptoms not seen since this patch went in. Link: https://lore.kernel.org/562fbfa6-dd6d-0b6a-2461-ed2ff1173bc8@google.com Fixes: 200a89c159a7 ("mm/filemap: use xas_try_split() in __filemap_add_folio()") Signed-off-by: Hugh Dickins Acked-by: Kiryl Shutsemau (Meta) Reviewed-by: Matthew Wilcox (Oracle) Reviewed-by: Zi Yan Cc: Chris J Arges Cc: David Hildenbrand Cc: Jan Kara Cc: Kairui Song Cc: Signed-off-by: Andrew Morton --- mm/filemap.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 58eb9d240643..d721986d5f46 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -931,6 +931,12 @@ unlock: if (!xas_nomem(&xas, gfp)) break; + + /* + * Lock has been dropped: start again with the original index + * and order (but now with the memory reserved by xas_nomem()). + */ + xas_set_order(&xas, index, forder); } if (xas_error(&xas)) From cd7d85c76ff03832a7f99ec7b34069602199f2c3 Mon Sep 17 00:00:00 2001 From: SJ Park Date: Tue, 28 Jul 2026 07:04:03 -0700 Subject: [PATCH 22/28] mm/damon: adjust isolated pages stat for DAMOS_MIGRATE_{HOT,COLD} Callers of migrate_pages() should adjust NR_MIGRATED_{ANON,FILE} for isolations and putback of the folios. That for migration succeeded folios is done by migrate_pages(), in migrate_folio_done(). That for MR_DEMOTION reason is an exception though. DAMOS_MIGRATE_{HOT,COLD} call migrate_pages() but mistakenly not doing the stat adjustment. As a result, use of DAMOS_MIGRATE_{HOT,COLD} could corrupt the stat. It could confuse too_many_isolated(), make compaction and reclaim to behave in unexpected ways. The stat corruption can be reproduced and confirmed using DAMON user-space tool [1] on NUMA systems, like below. $ numactl --hardware available: 2 nodes (0-1) [...] $ sudo ./damo start --damos_action migrate_hot 1 $ sudo cat /proc/sys/vm/stat_refresh $ sudo dmesg [...] [ 80.215554] vmstat_refresh: nr_isolated_anon -5578 [ 80.216842] vmstat_refresh: nr_isolated_file -34400 This issue was discovered [2] by Sashiko. Link: https://lore.kernel.org/20260728140404.94476-1-sj@kernel.org Link: https://github.com/damonitor/damo [1] Link: https://lore.kernel.org/20260726164356.87940-1-sj@kernel.org [2] Fixes: b51820ebea65 ("mm/damon/paddr: introduce DAMOS_MIGRATE_COLD action for demotion") Signed-off-by: SJ Park Cc: Honggyu Kim Cc: Hyeongtak Ji Cc: # 6.11.x Signed-off-by: Andrew Morton --- mm/damon/ops-common.c | 4 ++++ mm/damon/paddr.c | 2 ++ mm/damon/vaddr.c | 3 ++- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/damon/ops-common.c b/mm/damon/ops-common.c index 9c178c175832..0bcad6b1e5b9 100644 --- a/mm/damon/ops-common.c +++ b/mm/damon/ops-common.c @@ -375,6 +375,8 @@ keep: while (!list_empty(folio_list)) { folio = lru_to_folio(folio_list); list_del(&folio->lru); + node_stat_sub_folio(folio, NR_ISOLATED_ANON + + folio_is_file_lru(folio)); folio_putback_lru(folio); } @@ -397,6 +399,8 @@ unsigned long damon_migrate_pages(struct list_head *folio_list, int target_nid) struct folio *folio = lru_to_folio(folio_list); list_del(&folio->lru); + node_stat_sub_folio(folio, NR_ISOLATED_ANON + + folio_is_file_lru(folio)); folio_putback_lru(folio); } return nr_migrated; diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c index 5c2da45f988c..f7613ce279a2 100644 --- a/mm/damon/paddr.c +++ b/mm/damon/paddr.c @@ -350,6 +350,8 @@ static unsigned long damon_pa_migrate(struct damon_region *r, if (!folio_isolate_lru(folio)) goto put_folio; + node_stat_add_folio(folio, NR_ISOLATED_ANON + + folio_is_file_lru(folio)); list_add(&folio->lru, &folio_list); put_folio: addr += folio_size(folio); diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index e73ec1ce016e..2c1c1952c008 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -649,7 +649,8 @@ static void damos_va_migrate_dests_add(struct folio *folio, isolate: if (!folio_isolate_lru(folio)) return; - + node_stat_add_folio(folio, NR_ISOLATED_ANON + + folio_is_file_lru(folio)); list_add(&folio->lru, &migration_lists[i]); } From eafb1f6f5011ce688eb62bb574e08cda87cac13f Mon Sep 17 00:00:00 2001 From: Brendan Jackman Date: Wed, 29 Jul 2026 12:01:48 +0000 Subject: [PATCH 23/28] MAINTAINERS: update address for Brendan Jackman Switch my entry in MAINTAINERS and .mailmap to my @linux.dev email address Link: https://lore.kernel.org/20260729-email-change-v1-1-666ae7c2b7fc@google.com Signed-off-by: Brendan Jackman Signed-off-by: Brendan Jackman Acked-by: Mike Rapoport (Microsoft) Acked-by: Lorenzo Stoakes (ARM) Acked-by: Zi Yan Cc: David Hildenbrand Cc: Johannes Weiner Cc: Liam R. Howlett Cc: Michal Hocko Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- .mailmap | 1 + MAINTAINERS | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.mailmap b/.mailmap index ca6dc2575802..7324e1af59c2 100644 --- a/.mailmap +++ b/.mailmap @@ -170,6 +170,7 @@ Boris Brezillon Boris Brezillon Boris Brezillon Brendan Higgins +Brendan Jackman Brian Avery Brian Cain Brian Cain diff --git a/MAINTAINERS b/MAINTAINERS index 48b2baa1541f..d28e9f59e921 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17160,7 +17160,7 @@ M: Andrew Morton M: Vlastimil Babka R: Suren Baghdasaryan R: Michal Hocko -R: Brendan Jackman +R: Brendan Jackman R: Johannes Weiner R: Zi Yan L: linux-mm@kvack.org From addb6093816660c57a3759343a1a6cae1ae69919 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 30 Jul 2026 11:55:47 +0100 Subject: [PATCH 24/28] mm/huge_memory: fix huge_zero_pfn race Patch series "mm/huge_memory: fix huge_zero_pfn race", v2. There is a subtle race in the reference-counted huge_zero_folio implementation. The fast path atomic logic fails to account for the fact that the shrinker (which drops the final huge_zero_refcount pin) can overwrite huge_zero_pfn with the ~0UL sentinel value in shrink_huge_zero_folio_scan() after a racing get_huge_zero_folio() installed a valid value there. This results in huge_zero_folio being correctly set but huge_zero_pfn being set incorrectly and thus is_huge_zero_pfn() and consequently is_huge_zero_pmd() will misidentify the huge zero folio as being an ordinary THP folio. This can result in the huge zero folio being split and otherwise treated incorrectly. The solution to this is very subtle as there is an atomic fast path, and thus ordering in weakly ordered architectures has to be treated very carefully. The first commit fixes the issue by introducing a spinlock around huge_zero_[pfn, folio, refcount] write, with careful consideration paid to load/store ordering in the fast path. It is placed first and kept as small as possible so that it can be backported on its own. The second commit is a pure cleanup which reworks the CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic to better separate the persistent logic from the dynamically allocated one. This patch (of 2): If !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, the huge_zero_folio is refcounted by huge_zero_refcount and returned by mm_get_huge_zero_folio(). When the caller is done with the huge zero page, its reference count is decremented. Only a shrinker can set the reference count to zero. A race can unfortunately occur between a shrinker decrementing the reference count to zero and a concurrent page fault. This is because shrink_huge_zero_folio_scan() might, if very unlucky, be preempted between setting huge_zero_refcount to zero and writing an invalid value. During this time get_huge_zero_folio() could write to huge_zero_pfn before shrink_huge_zero_folio_scan() resumes. In this event the huge zero folio will be persistently misidentified causing the THP code path to be entered inappropriately for the huge zero folio: CPU 0 CPU 1 =======================================|================================= shrink_huge_zero_folio_scan() | atomic_cmpxchg() sets refcount to 0 | xchg() sets huge_zero_folio to NULL | get_huge_zero_folio() | | atomic_inc_not_zero() -> zero preempted for a long time | Allocate new huge zero folio | | Write valid huge_zero_folio v | Write valid huge_zero_pfn Overwrite huge_zero_pfn with ~0UL <--- Invalid overwrite! This results in is_huge_zero_pfn() and is_huge_zero_pmd() incorrectly returning false for a huge zero page which could result in issues like the huge zero folio being incorrectly split. Note that the issue is with huge_zero_pfn not huge_zero_folio, as get_huge_zero_folio() uses cmpxchg() gated on huge_zero_folio being NULL with a retry loop and shrink_huge_zero_folio_scan() uses xchg() to set huge_zero_folio. Fix the issue by introducing a spinlock, huge_zero_lock, to prevent concurrent write of huge_zero_folio, huge_zero_pfn and huge_zero_refcount. There needs to be significant care taken here to ensure correctness: The fast path in get_huge_zero_folio() uses atomic_inc_not_zero(), which is outside of the critical section, and means huge zero allocation is gated on zero huge_zero_refcount. The fast path doesn't use huge_zero_lock, so the critical section is irrelevant to it. So invariants are required - huge_zero_refcount MUST: * Only be set in the huge_zero_lock critical section to ensure serialisation of huge_zero_pfn, huge_zero_folio and huge_zero_refcount writes. * Be set non-zero only AFTER huge_zero_[pfn, folio] are set to valid values so installation of the huge zero folio on read page fault ensures concurrent is_huge_zero_*() calls correctly identify the huge zero folio. * Be set zero only BEFORE huge_zero_[pfn, folio] are set to NULL and ~0UL respectively, and atomically. Establish these by: * Only setting huge_zero_refcount to zero or an absolute value in the huge_zero_lock critical section in get_huge_zero_folio() and shrink_huge_zero_folio_scan(), and always updating atomically there and elsewhere. * Using atomic_set_release(&huge_zero_refcount) in get_huge_zero_folio() after huge_zero_[pfn, folio] are set. This is paired with atomic_inc_not_zero() to ensure atomic_inc_not_zero() only observes a non-zero value if huge_zero_[pfn, folio] are set. * Using atomic_cmpxchg() in shrink_huge_zero_folio_scan() (as before) to ensure that it is set zero only when equal to 1 and set atomically. * atomic_cmpxchg() being fully ordered ensures this is done prior to huge_zero_[folio, pfn] being set to NULL and ~0UL respectively. Eliminate the retry loop in get_huge_zero_folio() as the atomic_cmpxchg() in shrink_huge_zero_folio_scan() is now performed under the lock, and replace with an equally locked atomic_inc() to set the reference count should the caller be raced on huge zero folio installation. folio_put() naturally implies a full memory barrier so its ordering is maintained correctly. The huge zero folio also cannot be released except when the shrinker does so as it is non-LRU and non-rmappable. Note that only the huge zero shrinker (via shrink_huge_zero_folio_scan()) can actually set huge_zero_refcount to zero, which is the count of mm's which have at least one huge zero folio installed plus one shrinker pin. Additionally convert a BUG_ON() to a VM_WARN_ON_ONCE(). Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-0-c5d8a41b317f@kernel.org Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-1-c5d8a41b317f@kernel.org Fixes: 3b77e8c8cde5 ("mm/thp: make is_huge_zero_pmd() safe and quicker") Signed-off-by: Lorenzo Stoakes (ARM) Reported-by: Hengbin Zhang Closes: https://lore.kernel.org/linux-mm/20260727154001.4102341-1-uqbarz@gmail.com/ Suggested-by: David Hildenbrand (Arm) Acked-by: David Hildenbrand (Arm) Cc: Baolin Wang Cc: Barry Song Cc: Dev Jain Cc: Hannes Reinecke Cc: Hugh Dickins Cc: Kiryl Shutsemau Cc: Lance Yang Cc: Liam R. Howlett Cc: Nico Pache Cc: Pankaj Raghav Cc: Ryan Roberts Cc: Yang Shi Cc: Zi Yan Cc: Signed-off-by: Andrew Morton --- mm/huge_memory.c | 49 +++++++++++++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 17 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8c1f35df0e91..bd9d01902562 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -41,6 +41,7 @@ #include #include #include +#include #include #include "internal.h" @@ -78,6 +79,7 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, static bool split_underused_thp = true; static atomic_t huge_zero_refcount; +static DEFINE_SPINLOCK(huge_zero_lock); struct folio *huge_zero_folio __read_mostly; unsigned long huge_zero_pfn __read_mostly = ~0UL; unsigned long huge_anon_orders_always __read_mostly; @@ -224,7 +226,8 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, static bool get_huge_zero_folio(void) { struct folio *zero_folio; -retry: + + /* Paired with atomic_set_release(). */ if (likely(atomic_inc_not_zero(&huge_zero_refcount))) return true; @@ -237,17 +240,22 @@ retry: } /* Ensure zero folio won't have large_rmappable flag set. */ folio_clear_large_rmappable(zero_folio); - preempt_disable(); - if (cmpxchg(&huge_zero_folio, NULL, zero_folio)) { - preempt_enable(); - folio_put(zero_folio); - goto retry; - } - WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio)); - /* We take additional reference here. It will be put back by shrinker */ - atomic_set(&huge_zero_refcount, 2); - preempt_enable(); + /* Paired with critical section in shrink_huge_zero_folio_scan(). */ + spin_lock(&huge_zero_lock); + if (huge_zero_folio) { + /* Somebody else already installed it. */ + atomic_inc(&huge_zero_refcount); + spin_unlock(&huge_zero_lock); + folio_put(zero_folio); + return true; + } + WRITE_ONCE(huge_zero_folio, zero_folio); + WRITE_ONCE(huge_zero_pfn, folio_pfn(zero_folio)); + /* Paired with atomic_inc_not_zero(). +1 for shrinker pin. */ + atomic_set_release(&huge_zero_refcount, 2); + spin_unlock(&huge_zero_lock); + count_vm_event(THP_ZERO_PAGE_ALLOC); return true; } @@ -297,15 +305,22 @@ static unsigned long shrink_huge_zero_folio_count(struct shrinker *shrink, static unsigned long shrink_huge_zero_folio_scan(struct shrinker *shrink, struct shrink_control *sc) { - if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) == 1) { - struct folio *zero_folio = xchg(&huge_zero_folio, NULL); - BUG_ON(zero_folio == NULL); + struct folio *zero_folio; + + /* Paired with critical section in get_huge_zero_folio(). */ + scoped_guard(spinlock, &huge_zero_lock) { + /* Paired with atomic_inc_not_zero() in get_huge_zero_folio(). */ + if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) != 1) + return 0; + + zero_folio = huge_zero_folio; + VM_WARN_ON_ONCE(!zero_folio); + WRITE_ONCE(huge_zero_folio, NULL); WRITE_ONCE(huge_zero_pfn, ~0UL); - folio_put(zero_folio); - return HPAGE_PMD_NR; } - return 0; + folio_put(zero_folio); + return HPAGE_PMD_NR; } static struct shrinker *huge_zero_folio_shrinker; From 7c82770ec6cc297941d083c49b7465a07e6f49fd Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 30 Jul 2026 11:55:48 +0100 Subject: [PATCH 25/28] mm/huge_memory: separate out CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic Rather than mixing the refcounted and non-refcounted CONFIG_PERSISTENT_HUGE_ZERO_FOLIO logic, separate the two out cleanly so it is clear what happens when this configuration option is set and what happens when it is not. Introduce HUGE_ZERO_UNSET_PFN to abstract the ~0UL assignment, only introduce the refcount, lock and shrinker if !CONFIG_PERSISTENT_HUGE_ZERO_FOLIO, abstract initialisation and teardown, abstract the huge zero folio allocation from refcounting. Also change a BUG_ON() to WARN_ON_ONCE() while we're at it. No functional change intended. Link: https://lore.kernel.org/20260730-fix-refcounted-huge-zero-v2-2-c5d8a41b317f@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Fixes: 3b77e8c8cde5 ("mm/thp: make is_huge_zero_pmd() safe and quicker") Cc: Baolin Wang Cc: Barry Song Cc: David Hildenbrand (Arm) Cc: Dev Jain Cc: Hannes Reinecke Cc: Hengbin Zhang Cc: Hugh Dickins Cc: Kiryl Shutsemau Cc: Lance Yang Cc: Liam R. Howlett Cc: Nico Pache Cc: Pankaj Raghav Cc: Ryan Roberts Cc: Yang Shi Cc: Zi Yan Cc: Signed-off-by: Andrew Morton --- mm/huge_memory.c | 160 ++++++++++++++++++++++++++++------------------- 1 file changed, 94 insertions(+), 66 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index bd9d01902562..5d94fa4c74fd 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -78,10 +78,15 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, struct shrink_control *sc); static bool split_underused_thp = true; +#define HUGE_ZERO_UNSET_PFN (~0UL) +struct folio *huge_zero_folio __read_mostly; +unsigned long huge_zero_pfn __read_mostly = HUGE_ZERO_UNSET_PFN; +#ifndef CONFIG_PERSISTENT_HUGE_ZERO_FOLIO static atomic_t huge_zero_refcount; static DEFINE_SPINLOCK(huge_zero_lock); -struct folio *huge_zero_folio __read_mostly; -unsigned long huge_zero_pfn __read_mostly = ~0UL; +static struct shrinker *huge_zero_folio_shrinker; +#endif + unsigned long huge_anon_orders_always __read_mostly; unsigned long huge_anon_orders_madvise __read_mostly; unsigned long huge_anon_orders_inherit __read_mostly; @@ -223,6 +228,47 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma, return orders; } +static struct folio *alloc_huge_zero_folio(void) +{ + struct folio *zero_folio; + + zero_folio = folio_alloc((GFP_TRANSHUGE | __GFP_ZERO | __GFP_ZEROTAGS) & + ~__GFP_MOVABLE, + HPAGE_PMD_ORDER); + if (!zero_folio) { + count_vm_event(THP_ZERO_PAGE_ALLOC_FAILED); + return NULL; + } + folio_clear_large_rmappable(zero_folio); /* Explicitly not rmappable. */ + return zero_folio; +} + +#ifdef CONFIG_PERSISTENT_HUGE_ZERO_FOLIO +static int __init huge_zero_init(void) +{ + huge_zero_folio = alloc_huge_zero_folio(); + if (!huge_zero_folio) { + pr_warn("Allocating persistent huge zero folio failed\n"); + } else { + huge_zero_pfn = folio_pfn(huge_zero_folio); + count_vm_event(THP_ZERO_PAGE_ALLOC); + } + return 0; +} + +static void __init huge_zero_shrinker_exit(void) +{ +} + +struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) +{ + return huge_zero_folio; +} + +void mm_put_huge_zero_folio(struct mm_struct *mm) +{ +} +#else static bool get_huge_zero_folio(void) { struct folio *zero_folio; @@ -231,15 +277,9 @@ static bool get_huge_zero_folio(void) if (likely(atomic_inc_not_zero(&huge_zero_refcount))) return true; - zero_folio = folio_alloc((GFP_TRANSHUGE | __GFP_ZERO | __GFP_ZEROTAGS) & - ~__GFP_MOVABLE, - HPAGE_PMD_ORDER); - if (!zero_folio) { - count_vm_event(THP_ZERO_PAGE_ALLOC_FAILED); + zero_folio = alloc_huge_zero_folio(); + if (unlikely(!zero_folio)) return false; - } - /* Ensure zero folio won't have large_rmappable flag set. */ - folio_clear_large_rmappable(zero_folio); /* Paired with critical section in shrink_huge_zero_folio_scan(). */ spin_lock(&huge_zero_lock); @@ -266,33 +306,7 @@ static void put_huge_zero_folio(void) * Counter should never go to zero here. Only shrinker can put * last reference. */ - BUG_ON(atomic_dec_and_test(&huge_zero_refcount)); -} - -struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) -{ - if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO)) - return huge_zero_folio; - - if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) - return READ_ONCE(huge_zero_folio); - - if (!get_huge_zero_folio()) - return NULL; - - if (mm_flags_test_and_set(MMF_HUGE_ZERO_FOLIO, mm)) - put_huge_zero_folio(); - - return READ_ONCE(huge_zero_folio); -} - -void mm_put_huge_zero_folio(struct mm_struct *mm) -{ - if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO)) - return; - - if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) - put_huge_zero_folio(); + WARN_ON_ONCE(atomic_dec_and_test(&huge_zero_refcount)); } static unsigned long shrink_huge_zero_folio_count(struct shrinker *shrink, @@ -316,14 +330,53 @@ static unsigned long shrink_huge_zero_folio_scan(struct shrinker *shrink, zero_folio = huge_zero_folio; VM_WARN_ON_ONCE(!zero_folio); WRITE_ONCE(huge_zero_folio, NULL); - WRITE_ONCE(huge_zero_pfn, ~0UL); + WRITE_ONCE(huge_zero_pfn, HUGE_ZERO_UNSET_PFN); } folio_put(zero_folio); return HPAGE_PMD_NR; } -static struct shrinker *huge_zero_folio_shrinker; +static int __init huge_zero_init(void) +{ + huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero"); + if (!huge_zero_folio_shrinker) { + shrinker_free(deferred_split_shrinker); + list_lru_destroy(&deferred_split_lru); + return -ENOMEM; + } + + huge_zero_folio_shrinker->count_objects = shrink_huge_zero_folio_count; + huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan; + shrinker_register(huge_zero_folio_shrinker); + return 0; +} + +static void __init huge_zero_shrinker_exit(void) +{ + shrinker_free(huge_zero_folio_shrinker); +} + +struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) +{ + if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) + return READ_ONCE(huge_zero_folio); + + if (!get_huge_zero_folio()) + return NULL; + + if (mm_flags_test_and_set(MMF_HUGE_ZERO_FOLIO, mm)) + put_huge_zero_folio(); + + return READ_ONCE(huge_zero_folio); +} + +void mm_put_huge_zero_folio(struct mm_struct *mm) +{ + if (mm_flags_test(MMF_HUGE_ZERO_FOLIO, mm)) + put_huge_zero_folio(); +} +#endif /* CONFIG_PERSISTENT_HUGE_ZERO_FOLIO */ #ifdef CONFIG_SYSFS static ssize_t enabled_show(struct kobject *kobj, @@ -987,39 +1040,14 @@ static int __init thp_shrinker_init(void) deferred_split_shrinker->scan_objects = deferred_split_scan; shrinker_register(deferred_split_shrinker); - if (IS_ENABLED(CONFIG_PERSISTENT_HUGE_ZERO_FOLIO)) { - /* - * Bump the reference of the huge_zero_folio and do not - * initialize the shrinker. - * - * huge_zero_folio will always be NULL on failure. We assume - * that get_huge_zero_folio() will most likely not fail as - * thp_shrinker_init() is invoked early on during boot. - */ - if (!get_huge_zero_folio()) - pr_warn("Allocating persistent huge zero folio failed\n"); - return 0; - } - - huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero"); - if (!huge_zero_folio_shrinker) { - shrinker_free(deferred_split_shrinker); - list_lru_destroy(&deferred_split_lru); - return -ENOMEM; - } - - huge_zero_folio_shrinker->count_objects = shrink_huge_zero_folio_count; - huge_zero_folio_shrinker->scan_objects = shrink_huge_zero_folio_scan; - shrinker_register(huge_zero_folio_shrinker); - - return 0; + return huge_zero_init(); } static void __init thp_shrinker_exit(void) { - shrinker_free(huge_zero_folio_shrinker); shrinker_free(deferred_split_shrinker); list_lru_destroy(&deferred_split_lru); + huge_zero_shrinker_exit(); } static int __init hugepage_init(void) From 3569aedb862c24b66c757567e2dc2292ec9ff738 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 23 Jul 2026 16:16:32 +0100 Subject: [PATCH 26/28] x86/mm/pat: acquire init_mm write lock on collapse to avoid UAF x86 implements page attribute modification using its Change Page Attributes (CPA) mechanism. This tracks properties of ranges such as cache mode through x86 page attributes, and as part of that logic manipulates kernel page tables. Since commit 41d88484c71c ("x86/mm/pat: restore large ROX pages after fragmentation") ranges of kernel page table entries can be collapsed into huge page table entries as part of this logic. As part of this collapse, it frees the page tables which the collapsed entries previously pointed to, and it does so without any relevant locks being held to preclude concurrent kernel page table walkers. The only way this code can be reached is if CPA_COLLAPSE is specified, and this is only set in set_memory_rox() via: set_memory_rox() -> change_page_attr_set_clr() -> cpa_flush() -> cpa_collapse_large_pages() Notable users of this are execmem and bpf when manipulating executable mappings. However, this is problematic for ptdump as it walks ranges it does not own and thus runs the risk of a use-after-free on page tables freed underneath it. In addition, concurrent CPA collapse operations are possible which can also cause races. Resolve the issue by acquiring the mmap write lock on init_mm across the whole operation. It is safe to acquire a sleeping lock as all the callers invoke set_memory_rox() from process context and in any case, change_page_attr_set_clr() calls vm_unmap_alias() which ultimately takes a mutex, disallowing atomic context here. Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-2-8cc77dcc0018@kernel.org Fixes: 41d88484c71c ("x86/mm/pat: restore large ROX pages after fragmentation") Signed-off-by: Lorenzo Stoakes (ARM) Reviewed-by: Mike Rapoport (Microsoft) Reviewed-by: Kiryl Shutsemau (Meta) Reviewed-by: David Hildenbrand (Arm) Reviewed-by: Dave Hansen Reviewed-by: Will Deacon Reviewed-by: David Carlier Cc: Cc: Andy Lutomirski Cc: "Borah, Chaitanya Kumar" Cc: "Borislav Petkov (AMD)" Cc: Catalin Marinas Cc: Dev Jain Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Liam R. Howlett Cc: Michal Hocko Cc: Peter Zijlstra Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Toshi Kani Cc: "Uladzislau Rezki (Sony)" Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- arch/x86/mm/pat/set_memory.c | 15 ++++++++++++++- include/linux/mmap_lock.h | 2 ++ 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index d023a40a1e03..d1e63f7d267f 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -22,6 +22,7 @@ #include #include #include +#include #include #include @@ -410,7 +411,7 @@ static void __cpa_flush_tlb(void *data) static int collapse_large_pages(unsigned long addr, struct list_head *pgtables); -static void cpa_collapse_large_pages(struct cpa_data *cpa) +static void __cpa_collapse_large_pages(struct cpa_data *cpa) { unsigned long start, addr, end; struct ptdesc *ptdesc, *tmp; @@ -442,6 +443,18 @@ static void cpa_collapse_large_pages(struct cpa_data *cpa) } } +static void cpa_collapse_large_pages(struct cpa_data *cpa) +{ + /* + * Take the mmap write lock on init_mm to: + * - Avoid a use-after-free if raced by ptdump (which takes its own + * write lock on init_mm). + * - Serialise concurrent CPA walkers. + */ + scoped_guard(mmap_write_lock, &init_mm) + __cpa_collapse_large_pages(cpa); +} + static void cpa_flush(struct cpa_data *cpa, int cache) { unsigned int i; diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 6b5c2390cc30..047f5f5e2c34 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -621,6 +621,8 @@ static inline void mmap_read_unlock(struct mm_struct *mm) DEFINE_GUARD(mmap_read_lock, struct mm_struct *, mmap_read_lock(_T), mmap_read_unlock(_T)) +DEFINE_GUARD(mmap_write_lock, struct mm_struct *, + mmap_write_lock(_T), mmap_write_unlock(_T)) DEFINE_GUARD_COND(mmap_read_lock, _try, mmap_read_trylock(_T)) static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) From eb17a9ab606f2c9a99d4b9519a921f7b606ed9e5 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 23 Jul 2026 16:16:33 +0100 Subject: [PATCH 27/28] x86/mm/pat: acquire init_mm read lock on attribute change to avoid UAF A previous commit protected us against races between ptdump and CPA collapse, however one still exists between attribute changes and collapse as reported by Denis V. Lunev (linked). When an attribute change arises, a lockless page table walker obtains a PTE entry, which is later written to via set_pte_atomic(): ... -> change_page_attr_set_clr() -> __change_page_attr_set_clr() -> __change_page_attr() -> _lookup_address_cpa() -> lookup_address_in_pgd_attr() -> [ lockless page table walker ] -> set_pte_atomic() There is nothing preventing a concurrent CPA collapse which can free the PTE that was retrieved here, resulting in a use-after-free. With the mmap write lock taken on init_mm over CPA collapse, we can now resolve this race by acquiring an mmap read lock on init_mm over __change_page_attr_set_clr(). This locks across the whole operation over which the walk and the PTE entry write occurs, solving the race. It is safe to do this here, as no spinlocks are held upon entry to __change_page_attr_set_clr(). However, the lock must not be held over an allocation, as allocation can trigger reclaim and shrinkers may call into CPA recursively, making deadlocks possible (init_mm -> ... -> fs_reclaim -> init_mm). A page table is allocated when a huge page needs to be split: -> change_page_attr_set_clr() -> __change_page_attr_set_clr() -> __change_page_attr() -> split_large_page() [ pagetable_alloc() ] -> __split_large_page() Avoid deadlocks by dropping the mmap lock across pagetable_alloc() in split_large_page() and track whether this is needed by adding a new 'init_mm_read_locked' flag to struct cpa_data. This is safe as __split_large_page() (called with locks re-established) revalidates that the page table entry is the same as it was prior to the locks being dropped and __change_page_attr() repeats the entire page table walk whenever a split occurs, so concurrent split and collapse are accounted for. Concurrent ptdump is also safe as the lock is only dropped over page table allocation during which time the page table has not yet been modified. The CPA_COLLAPSE flag is only set by set_memory_rox(), which exclusively operates upon vmalloc ranges, and on x86 only within the module mapping space. This is important, because some callers directly invoke __change_page_attr_set_clr(), bypassing this lock. However, none of these operate within the module mapping space. * cpa_process_alias() - a recursive helper called by __change_page_attr_set_clr(). * __set_memory_enc_pgtable() - operates on the direct mapping and (via __vmbus_establish_gpadl()) the vmalloc mapping space. * __set_pages_[n]p() - called by set_direct_map_[invalid, default, valid]_noflush(), __kernel_map_pages() - operates on the direct map. * kernel_[un]map_pages_in_pgd() - operates on EFI ranges. This work is based upon Denis V. Lunev's excellent analysis of the bug with gratitude. Link: https://lore.kernel.org/all/20260626163213.2284080-1-den@openvz.org/ Link: https://lore.kernel.org/20260723-series-vmap-race-fix-v6-3-8cc77dcc0018@kernel.org Fixes: 41d88484c71c ("x86/mm/pat: restore large ROX pages after fragmentation") Signed-off-by: Lorenzo Stoakes (ARM) Cc: Cc: Andy Lutomirski Cc: "Borah, Chaitanya Kumar" Cc: "Borislav Petkov (AMD)" Cc: Catalin Marinas Cc: Dave Hansen Cc: David Carlier Cc: David Hildenbrand (Arm) Cc: Dev Jain Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Kiryl Shutsemau Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport (Microsoft) Cc: Peter Zijlstra Cc: Ryan Roberts Cc: Shakeel Butt Cc: Suren Baghdasaryan Cc: Toshi Kani Cc: "Uladzislau Rezki (Sony)" Cc: Vlastimil Babka Cc: Will Deacon Signed-off-by: Andrew Morton --- arch/x86/mm/pat/set_memory.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index d1e63f7d267f..618f72096480 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -50,7 +50,8 @@ struct cpa_data { unsigned int flags; unsigned int force_split : 1, force_static_prot : 1, - force_flush_all : 1; + force_flush_all : 1, + init_mm_read_locked : 1; struct page **pages; }; @@ -1250,7 +1251,11 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte, if (!debug_pagealloc_enabled()) spin_unlock(&cpa_lock); + if (cpa->init_mm_read_locked) + mmap_read_unlock(&init_mm); ptdesc = pagetable_alloc(GFP_KERNEL, 0); + if (cpa->init_mm_read_locked) + mmap_read_lock(&init_mm); if (!debug_pagealloc_enabled()) spin_lock(&cpa_lock); if (!ptdesc) @@ -2122,7 +2127,11 @@ static int change_page_attr_set_clr(unsigned long *addr, int numpages, cpa.curpage = 0; cpa.force_split = force_split; - ret = __change_page_attr_set_clr(&cpa, 1); + /* Avoid race with concurrent CPA collapse. */ + cpa.init_mm_read_locked = true; + scoped_guard(mmap_read_lock, &init_mm) + ret = __change_page_attr_set_clr(&cpa, 1); + cpa.init_mm_read_locked = false; /* * Check whether we really changed something: From 4bbc16994e86b98bd8daa73afd27b77c7da42266 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Tue, 21 Jul 2026 13:14:52 +0100 Subject: [PATCH 28/28] x86/mm/pat: allocate split page tables as kernel page tables When splitting a large page in CPA in __split_large_page() we allocate a PTE directly without going through the standard page table allocation routines such as pte_alloc_one_kernel(). This means the page table constructor is never called nor is the page table marked as a kernel page table. The former results in the folio associated with the page table not being marked as a page table (__pagetable_ctor() is never called thus neither is __folio_set_pgtable()) nor are statistics updated to reflect it (lruvec_stat_add_folio() is never called). The latter issue of failing to mark the page table as a kernel page table (ptdesc_set_kernel() is never called) is far more problematic. Since commit 5ba2f0a15564 ("mm: introduce deferred freeing for kernel page tables") kernel page table freeing has been batched and since the subsequent commit e37d5a2d60a3 ("iommu/sva: invalidate stale IOTLB entries for kernel address space") IOTLB cache entries for kernel page tables have been invalidated upon being freed. Since split page tables are freed without this invalidation, the IOTLB can contain stale entries for them. Resolve the issue by using the ordinary PTE allocation API at split time. This results in these kernel page tables invoking a page table constructor, and thus requires a page table destructor. Since we cannot assume one is always present (early allocated direct map page tables are not marked as such), we conditionally call pagetable_dtor_free() if the PG_table folio flag for the ptdesc is set, otherwise we free the page table via pagetable_free(). Regardless of which path is taken page tables marked as kernel page tables, which now includes split page tables, take the correct route through pagetable_free_kernel(). There is a user-visible side effect in that split page tables will appear in nr_page_table_pages in /proc/vmstat (as do other kernel page tables allocated after early boot), however this is a positive change. This issue started being markedly problematic after commit 5ba2f0a15564 ("mm: introduce deferred freeing for kernel page tables") so choose this as the Fixes target. Link: https://lore.kernel.org/20260721-fix-cpa-kernel-pagetables-v2-1-2b255deed710@kernel.org Fixes: 5ba2f0a15564 ("mm: introduce deferred freeing for kernel page tables") Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: Vishal Moola Cc: Andy Lutomirski Cc: Baolu Lu Cc: "Borislav Petkov (AMD)" Cc: Dave Hansen Cc: David Hildenbrand Cc: "H. Peter Anvin" Cc: Ingo Molnar Cc: Jason Gunthorpe Cc: Kevin Tian Cc: Kiryl Shutsemau Cc: Mike Rapoport Cc: Peter Zijlstra Cc: Signed-off-by: Andrew Morton --- arch/x86/mm/pat/set_memory.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 618f72096480..422ce7fba00c 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -440,7 +440,15 @@ static void __cpa_collapse_large_pages(struct cpa_data *cpa) list_for_each_entry_safe(ptdesc, tmp, &pgtables, pt_list) { list_del(&ptdesc->pt_list); - pagetable_free(ptdesc); + /* + * Only early alloc'd direct map should not be flagged PG_table + * here and those shouldn't be collapsed. However be abundantly + * cautious and handle the !PG_table case too. + */ + if (PageTable((ptdesc_page(ptdesc)))) + pagetable_dtor_free(ptdesc); + else + pagetable_free(ptdesc); } } @@ -1139,11 +1147,10 @@ set: static int __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, - struct ptdesc *ptdesc) + pte_t *pbase) { unsigned long lpaddr, lpinc, ref_pfn, pfn, pfninc = 1; - struct page *base = ptdesc_page(ptdesc); - pte_t *pbase = (pte_t *)page_address(base); + struct page *base = virt_to_page(pbase); unsigned int i, level; pgprot_t ref_prot; bool nx, rw; @@ -1247,22 +1254,22 @@ __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address, static int split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address) { - struct ptdesc *ptdesc; + pte_t *pte; if (!debug_pagealloc_enabled()) spin_unlock(&cpa_lock); if (cpa->init_mm_read_locked) mmap_read_unlock(&init_mm); - ptdesc = pagetable_alloc(GFP_KERNEL, 0); + pte = pte_alloc_one_kernel(&init_mm); if (cpa->init_mm_read_locked) mmap_read_lock(&init_mm); if (!debug_pagealloc_enabled()) spin_lock(&cpa_lock); - if (!ptdesc) + if (!pte) return -ENOMEM; - if (__split_large_page(cpa, kpte, address, ptdesc)) - pagetable_free(ptdesc); + if (__split_large_page(cpa, kpte, address, pte)) + pte_free_kernel(&init_mm, pte); return 0; }