Files
Lorenzo Stoakes (ARM)andAndrew Morton 3a509464b7 mm: propagate VMA virtual page offset on map, remap, split + merge
We must correctly update VMA virtual page offset state on all VMA
operations that would result in it changing, with special attention given
to remapping.

We cover most cases by simply updating vma_set_range() to do so (with a
new virtual page offset parameter), but also notably must update the
merging and mapping logic to propagate this parameter correctly.

The remap logic remains the same - we may update the virtual page offset
if the VMA is unfaulted, but now this applies to MAP_PRIVATE file-backed
mappings too, so we update the code to reflect this.

Note that we use __linear_virt_page_index() upon remap as the VMA may be
shared, in order that we update the field consistently regardless of VMA
type.

Also while we're here, replace a VM_BUG_ON_VMA() with a
VM_WARN_ON_ONCE_VMA().

We also introduce vma_anon_pgoff_addr(), vma_start_anon_pgoff(), and
vma_end_anon_pgoff() which differ from their virtual page offset
equivalents in that a shared file-backed mapping returns its file page
offset otherwise the virtual page offset is used.

This means we don't predicate merges for shared file-backed mappings on
virtual page offset.

We simply ensure state is correctly propagated here, so no functional
changes are intended.

Finally, we update insert_vm_struct() to correctly set the virtual page
offset on insertion of a VMA.

Also update VMA userland tests to reflect this change.

Link: https://lore.kernel.org/20260720-b4-scalable-cow-virt-pgoff-v2-6-2d549757a76f@kernel.org
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Gregory Price <gourry@gourry.net>
Cc: Harry Yoo <harry@kernel.org>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Jann Horn <jannh@google.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Kees Cook <kees@kernel.org>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Peter Xu <peterx@redhat.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Rik van Riel <riel@surriel.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: xu xin <xu.xin16@zte.com.cn>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2026-07-30 19:49:22 -07:00

124 lines
2.6 KiB
C

// SPDX-License-Identifier: GPL-2.0-or-later
#include "shared.h"
bool fail_prealloc;
unsigned long mmap_min_addr = CONFIG_DEFAULT_MMAP_MIN_ADDR;
unsigned long dac_mmap_min_addr = CONFIG_DEFAULT_MMAP_MIN_ADDR;
unsigned long stack_guard_gap = 256UL<<PAGE_SHIFT;
const struct vm_operations_struct vma_dummy_vm_ops;
struct anon_vma dummy_anon_vma;
struct task_struct __current;
struct vm_area_struct *alloc_vma(struct mm_struct *mm,
unsigned long start, unsigned long end,
pgoff_t pgoff, vma_flags_t vma_flags)
{
struct vm_area_struct *vma = vm_area_alloc(mm);
if (vma == NULL)
return NULL;
vma->vm_start = start;
vma->vm_end = end;
vma_set_pgoff(vma, pgoff);
vma_set_virt_pgoff(vma, start >> PAGE_SHIFT);
vma->flags = vma_flags;
vma_assert_detached(vma);
return vma;
}
void detach_free_vma(struct vm_area_struct *vma)
{
vma_mark_detached(vma);
vm_area_free(vma);
}
struct vm_area_struct *alloc_and_link_vma(struct mm_struct *mm,
unsigned long start, unsigned long end,
pgoff_t pgoff, vma_flags_t vma_flags)
{
struct vm_area_struct *vma = alloc_vma(mm, start, end, pgoff, vma_flags);
if (vma == NULL)
return NULL;
if (attach_vma(mm, vma)) {
detach_free_vma(vma);
return NULL;
}
/*
* Reset this counter which we use to track whether writes have
* begun. Linking to the tree will have caused this to be incremented,
* which means we will get a false positive otherwise.
*/
vma->vm_lock_seq = UINT_MAX;
return vma;
}
void reset_dummy_anon_vma(void)
{
dummy_anon_vma.was_cloned = false;
dummy_anon_vma.was_unlinked = false;
}
int cleanup_mm(struct mm_struct *mm, struct vma_iterator *vmi)
{
struct vm_area_struct *vma;
int count = 0;
fail_prealloc = false;
reset_dummy_anon_vma();
vma_iter_set(vmi, 0);
for_each_vma(*vmi, vma) {
detach_free_vma(vma);
count++;
}
mtree_destroy(&mm->mm_mt);
mm->map_count = 0;
return count;
}
bool vma_write_started(struct vm_area_struct *vma)
{
int seq = vma->vm_lock_seq;
/* We reset after each check. */
vma->vm_lock_seq = UINT_MAX;
/* The vma_start_write() stub simply increments this value. */
return seq > -1;
}
void __vma_set_dummy_anon_vma(struct vm_area_struct *vma,
struct anon_vma_chain *avc, struct anon_vma *anon_vma)
{
vma->anon_vma = anon_vma;
INIT_LIST_HEAD(&vma->anon_vma_chain);
list_add(&avc->same_vma, &vma->anon_vma_chain);
avc->anon_vma = vma->anon_vma;
}
void vma_set_dummy_anon_vma(struct vm_area_struct *vma,
struct anon_vma_chain *avc)
{
__vma_set_dummy_anon_vma(vma, avc, &dummy_anon_vma);
}
struct task_struct *get_current(void)
{
return &__current;
}
unsigned long rlimit(unsigned int limit)
{
return (unsigned long)-1;
}