linuxmsm/laptops-kernel
0
0
Fork 0
mirror of https://github.com/linux-msm/laptops-kernel.git synced 2026-08-13 14:19:53 -07:00
Code Issues Packages Projects Releases Wiki Activity

mm/hmm: add hmm_range_fault_unlocked_timeout() for mmap lock-drop support

Browse Source 
hmm_range_fault() requires the caller to hold the mmap read lock for the
duration of the call.  This is incompatible with mappings whose fault
handler may release the mmap lock, notably userfaultfd-managed regions,
where handle_mm_fault() can return VM_FAULT_RETRY or VM_FAULT_COMPLETED
after dropping the lock.  Drivers that need to populate device page tables
for such mappings have no way to do so today.

Add hmm_range_fault_unlocked_timeout() for callers that do not need to
hold mmap_lock across any work outside the HMM fault itself.  The helper
takes mmap_read_lock_killable() internally, calls the common HMM fault
implementation, and releases the lock before returning if it is still
held.  The timeout is specified in jiffies; passing 0 retries
indefinitely, while a non-zero timeout makes the helper return -EBUSY when
the retry budget expires.  The retry deadline is set before refreshing the
notifier sequence and acquiring mmap_lock, so contended mmap_lock
acquisition is included in the retry budget.  After acquiring mmap_lock,
the helper also rejects unstable address spaces before walking page
tables.

When handle_mm_fault() drops mmap_lock, or when the range is invalidated,
hmm_range_fault_unlocked_timeout() refreshes range->notifier_seq and
retries the walk internally.  If the lock was dropped, the retry deadline
is also restarted because a lock-dropping fault handler made progress. 
Ordinary -EBUSY retries keep the existing deadline, preserving the
caller's timeout policy for repeated mmu-notifier invalidations.

The caller only needs to perform the usual post-success
mmu_interval_read_retry() check while holding its update lock before
consuming the pfns.  If mmap_lock acquisition is interrupted or a fatal
signal is pending during retry handling, -EINTR is returned instead.

The common implementation conditionally sets FAULT_FLAG_ALLOW_RETRY and
FAULT_FLAG_KILLABLE only for hmm_range_fault_unlocked_timeout().  The
existing hmm_range_fault() path still passes no locked state, does not
allow handle_mm_fault() to drop mmap_lock, and remains a thin wrapper
preserving the existing API contract for current callers.

The previous refactor that moved page fault handling out of the page-table
walk callbacks is what makes this change small.  Faults now run after
walk_page_range() has unwound, with only mmap_lock held, so dropping it
does not interact with the walker's pte spinlock or hugetlb_vma_lock. 
Hugetlb regions therefore participate in the unlocked path uniformly with
PTE- and PMD-level mappings; no special case is required.

Documentation/mm/hmm.rst is updated with a description of the new API and
the recommended caller pattern.

Link: https://lore.kernel.org/20260723-hmm-v10-v11-2-c55b003a4b61@gmail.com
Signed-off-by: Stanislav Kinsburskii <skinsburskii@gmail.com>
Cc: Danilo Krummrich <dakr@kernel.org>
Cc: Dave Airlie <airlied@gmail.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Dexuan Cui <decui@microsoft.com>
Cc: Haiyang Zhang <haiyangz@microsoft.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: K. Y. Srinivasan <kys@microsoft.com>
Cc: Leon Romanovsky <leon@kernel.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lizhi Hou <lizhi.hou@amd.com>
Cc: Long Li <longli@microsoft.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Lyude <lyude@redhat.com>
Cc: Maarten Lankhorst <maarten.lankhorst@linux.intel.com>
Cc: Maxime Ripard <mripard@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Oded Gabbay <ogabbay@kernel.org>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Thomas Zimemrmann <tzimmermann@suse.de>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Liu <wei.liu@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
This commit is contained in:
Stanislav Kinsburskii
2026-07-30 19:48:38 -07:00
committed by Andrew Morton
parent 6855c8437c
commit 31207fa02d
3 changed files with 197 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/mm/hmm.rst
+56 -23
View File
@@ -156,42 +156,57 @@ During the ops->invalidate() callback the device driver must perform the
update action to the range (mark range read only, or fully unmap, etc.). The
device must complete the update before the driver callback returns.
When the device driver wants to populate a range of virtual addresses, it can
use::
When the device driver wants to populate a range of virtual addresses, the
normal interface is::
int hmm_range_fault(struct hmm_range *range);
int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
unsigned long timeout);
It will trigger a page fault on missing or read-only entries if write access is
requested (see below). Page faults use the generic mm page fault code path just
like a CPU page fault. The usage pattern is::
like a CPU page fault.
The caller must not hold ``mmap_read_lock`` before the call.
``hmm_range_fault_unlocked_timeout()`` takes the mmap read lock internally and
allows ``handle_mm_fault()`` to drop it during fault handling. This is required
for VMAs whose fault handlers may release the mmap lock, for example regions
managed by ``userfaultfd``.
If the mmap lock is dropped or the range is invalidated, the function refreshes
``range->notifier_seq`` and restarts the walk internally. ``-EINTR`` is returned
if mmap lock acquisition is interrupted or a fatal signal is pending during
retry handling.
The timeout is specified in jiffies; passing ``0`` means retry indefinitely. The
timeout exists to preserve caller policy for repeated mmu-notifier invalidation
and is checked between retry attempts. HMM does not interrupt page fault
handling when the timeout expires, but returns ``-EBUSY`` if the retry budget is
exhausted before a stable range is obtained.
The usage pattern is::
int driver_populate_range(...)
{
struct hmm_range range;
unsigned long timeout;
...
timeout = msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT);
range.notifier = &interval_sub;
range.start = ...;
range.end = ...;
range.hmm_pfns = ...;
if (!mmget_not_zero(interval_sub->notifier.mm))
if (!mmget_not_zero(interval_sub.mm))
return -EFAULT;
again:
range.notifier_seq = mmu_interval_read_begin(&interval_sub);
mmap_read_lock(mm);
ret = hmm_range_fault(&range);
if (ret) {
mmap_read_unlock(mm);
if (ret == -EBUSY)
goto again;
return ret;
}
mmap_read_unlock(mm);
ret = hmm_range_fault_unlocked_timeout(&range, timeout);
if (ret)
goto out_put;
take_lock(driver->update);
if (mmu_interval_read_retry(&ni, range.notifier_seq) {
if (mmu_interval_read_retry(range.notifier, range.notifier_seq)) {
release_lock(driver->update);
goto again;
}
@@ -200,13 +215,31 @@ like a CPU page fault. The usage pattern is::
* under the update lock */
release_lock(driver->update);
return 0;
ret = 0;
out_put:
mmput(interval_sub.mm);
return ret;
}
The driver->update lock is the same lock that the driver takes inside its
invalidate() callback. That lock must be held before calling
mmu_interval_read_retry() to avoid any race with a concurrent CPU page table
update.
update. The retry check must use the same notifier and sequence number stored
in ``range`` by ``hmm_range_fault_unlocked_timeout()``.
Holding the mmap lock across HMM faults
=======================================
Most callers should use ``hmm_range_fault_unlocked_timeout()``. If a driver
really needs to hold the mmap lock across work outside HMM, it can use::
int hmm_range_fault(struct hmm_range *range);
The mmap lock must be held by the caller and will remain held on return. This
interface cannot support VMAs whose fault handlers need to drop the mmap lock.
New callers should prefer ``hmm_range_fault_unlocked_timeout()`` unless they
have a specific requirement to keep the mmap lock held across the call.
Leverage default_flags and pfn_flags_mask
=========================================
@@ -221,8 +254,8 @@ permission, it sets::
range->default_flags = HMM_PFN_REQ_FAULT;
range->pfn_flags_mask = 0;
and calls hmm_range_fault() as described above. This will fill fault all pages
in the range with at least read permission.
and calls the HMM range fault helper as described above. This will fault
all pages in the range with at least read permission.
Now let's say the driver wants to do the same except for one page in the range for
which it wants to have write permission. Now driver set::
@@ -236,9 +269,9 @@ address == range->start + (index_of_write << PAGE_SHIFT) it will fault with
write permission i.e., if the CPU pte does not have write permission set then HMM
will call handle_mm_fault().
After hmm_range_fault completes the flag bits are set to the current state of
the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be set if the page is
writable.
After the HMM range fault helper completes the flag bits are set to the
current state of the page tables, ie HMM_PFN_VALID | HMM_PFN_WRITE will be
set if the page is writable.
Represent and manage device memory from core kernel point of view
include/linux/hmm.h
+2
View File
@@ -123,6 +123,8 @@ struct hmm_range {
* Please see Documentation/mm/hmm.rst for how to use the range API.
*/
int hmm_range_fault(struct hmm_range *range);
int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
unsigned long timeout);
/*
* HMM_RANGE_DEFAULT_TIMEOUT - default timeout (ms) when waiting for a range
mm/hmm.c
+139 -29
View File
@@ -17,6 +17,7 @@
#include <linux/slab.h>
#include <linux/sched.h>
#include <linux/mmzone.h>
#include <linux/oom.h>
#include <linux/pagemap.h>
#include <linux/leafops.h>
#include <linux/hugetlb.h>
@@ -32,6 +33,7 @@
struct hmm_vma_walk {
struct hmm_range *range;
bool *locked;
unsigned long last;
unsigned long end;
unsigned int required_fault;
@@ -44,6 +46,14 @@ struct hmm_vma_walk {
*/
#define HMM_FAULT_PENDING -EAGAIN
/*
* Internal sentinel returned by hmm_do_fault() when handle_mm_fault()
* completes a page fault with the mmap lock dropped. hmm_do_fault() sets
* *locked = false; the outer loop consumes the sentinel and never propagates
* it to the caller.
*/
#define HMM_FAULT_UNLOCKED -ENOLCK
enum {
HMM_NEED_FAULT = 1 << 0,
HMM_NEED_WRITE_FAULT = 1 << 1,
@@ -73,9 +83,9 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end,
*
* Called by the walk callbacks when they discover that part of the range
* needs a page fault. The callback records what to fault and returns
* HMM_FAULT_PENDING; the outer loop in hmm_range_fault() drops back out of
* walk_page_range() and invokes handle_mm_fault() from a context where no
* page-table or hugetlb_vma_lock is held.
* HMM_FAULT_PENDING; the outer loop in hmm_range_fault_locked() drops
* back out of walk_page_range() and invokes handle_mm_fault() from a context
* where no page-table or hugetlb_vma_lock is held.
*/
static int hmm_record_fault(unsigned long addr, unsigned long end,
unsigned int required_fault,
@@ -624,7 +634,7 @@ static const struct mm_walk_ops hmm_walk_ops = {
/*
* hmm_do_fault - fault in a range recorded by a walk callback
*
* Called from the outer loop in hmm_range_fault() after a callback
* Called from the outer loop in hmm_range_fault_locked() after a callback
* returned HMM_FAULT_PENDING. At this point we hold only mmap_lock;
* the page-table spinlock and any hugetlb_vma_lock acquired by the walk
* framework have already been released by the unwind.
@@ -641,6 +651,9 @@ static int hmm_do_fault(struct mm_struct *mm,
unsigned int fault_flags = FAULT_FLAG_REMOTE;
struct vm_area_struct *vma;
if (hmm_vma_walk->locked)
fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
vma = vma_lookup(mm, addr);
if (!vma)
return -EFAULT;
@@ -651,37 +664,34 @@ static int hmm_do_fault(struct mm_struct *mm,
fault_flags |= FAULT_FLAG_WRITE;
}
for (; addr < end; addr += PAGE_SIZE)
if (handle_mm_fault(vma, addr, fault_flags, NULL) &
VM_FAULT_ERROR)
return -EFAULT;
for (; addr < end; addr += PAGE_SIZE) {
vm_fault_t ret;
ret = handle_mm_fault(vma, addr, fault_flags, NULL);
if (ret & (VM_FAULT_COMPLETED | VM_FAULT_RETRY)) {
*hmm_vma_walk->locked = false;
return HMM_FAULT_UNLOCKED;
}
if (ret & VM_FAULT_ERROR) {
int err = vm_fault_to_errno(ret, 0);
if (WARN_ON(!err))
err = -EINVAL;
return err;
}
}
return -EBUSY;
}
/**
* hmm_range_fault - try to fault some address in a virtual address range
* @range: argument structure
*
* Returns 0 on success or one of the following error codes:
*
* -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
* (e.g., device file vma).
* -ENOMEM: Out of memory.
* -EPERM: Invalid permission (e.g., asking for write and range is read
* only).
* -EBUSY: The range has been invalidated and the caller needs to wait for
* the invalidation to finish.
* -EFAULT: A page was requested to be valid and could not be made valid
* ie it has no backing VMA or it is illegal to access
*
* This is similar to get_user_pages(), except that it can read the page tables
* without mutating them (ie causing faults).
*/
int hmm_range_fault(struct hmm_range *range)
static int hmm_range_fault_locked(struct hmm_range *range, bool *locked)
{
struct hmm_vma_walk hmm_vma_walk = {
.range = range,
.locked = locked,
.last = range->start,
};
struct mm_struct *mm = range->notifier->mm;
@@ -704,8 +714,14 @@ int hmm_range_fault(struct hmm_range *range)
* returns -EBUSY so the loop re-walks and picks up the
* now-present entries.
*/
if (ret == HMM_FAULT_PENDING)
if (ret == HMM_FAULT_PENDING) {
ret = hmm_do_fault(mm, &hmm_vma_walk);
if (ret == HMM_FAULT_UNLOCKED) {
if (fatal_signal_pending(current))
return -EINTR;
return -EBUSY;
}
}
/*
* When -EBUSY is returned the loop restarts with
* hmm_vma_walk.last set to an address that has not been stored
@@ -715,8 +731,102 @@ int hmm_range_fault(struct hmm_range *range)
} while (ret == -EBUSY);
return ret;
}
/**
* hmm_range_fault - try to fault some address in a virtual address range
* @range: argument structure
*
* Returns 0 on success or one of the following error codes:
*
* -EINVAL: Invalid arguments or mm or virtual address is in an invalid vma
* (e.g., device file vma).
* -ENOMEM: Out of memory.
* -EPERM: Invalid permission (e.g., asking for write and range is read
* only).
* -EBUSY: The range has been invalidated and the caller needs to wait for
* the invalidation to finish.
* -EFAULT: A page was requested to be valid and could not be made valid
* ie it has no backing VMA or it is illegal to access
*
* This is similar to get_user_pages(), except that it can read the page tables
* without mutating them (ie causing faults).
*
* The mmap lock must be held by the caller and will remain held on return.
* New users should prefer hmm_range_fault_unlocked_timeout() unless they
* specifically need to keep the mmap lock held across the call. This helper
* cannot support VMAs whose fault handlers need to drop the mmap lock.
*/
int hmm_range_fault(struct hmm_range *range)
{
return hmm_range_fault_locked(range, NULL);
}
EXPORT_SYMBOL(hmm_range_fault);
/**
* hmm_range_fault_unlocked_timeout - fault in a range with a retry timeout
* @range: argument structure
* @timeout: timeout in jiffies for internal -EBUSY retries, or 0 to retry
* indefinitely
*
* The caller must not hold the mmap lock. The function takes the mmap read
* lock internally and allows handle_mm_fault() to drop it during faults. If
* the mmap lock is dropped or the range is invalidated, the function refreshes
* range->notifier_seq and restarts the walk internally.
*
* Passing 0 for @timeout retries indefinitely. A non-zero @timeout is a caller
* policy limit for repeated mmu-notifier invalidation retries. HMM does not
* interrupt page fault handling when the timeout expires, but returns -EBUSY
* if the retry budget is exhausted before a stable range is obtained.
*
* Returns 0 on success or one of the error codes documented for
* hmm_range_fault(). -EINTR is returned if mmap_lock acquisition is
* interrupted or a fatal signal is pending during retry handling.
*/
int hmm_range_fault_unlocked_timeout(struct hmm_range *range,
unsigned long timeout)
{
struct mm_struct *mm = range->notifier->mm;
unsigned long deadline = 0;
bool locked = false;
int ret;
do {
/*
* If the previous fault dropped mmap_lock, then the fault
* handler made progress. Restart the retry timeout in that
* case, but keep the existing deadline for ordinary -EBUSY
* retries.
*/
if (timeout && !locked)
deadline = jiffies + timeout;
range->notifier_seq =
mmu_interval_read_begin(range->notifier);
ret = mmap_read_lock_killable(mm);
if (ret)
return ret;
if (check_stable_address_space(mm)) {
mmap_read_unlock(mm);
return -EFAULT;
}
if (timeout && time_after(jiffies, deadline)) {
mmap_read_unlock(mm);
return -EBUSY;
}
locked = true;
ret = hmm_range_fault_locked(range, &locked);
if (locked)
mmap_read_unlock(mm);
} while (ret == -EBUSY);
return ret;
}
EXPORT_SYMBOL(hmm_range_fault_unlocked_timeout);
/**
* hmm_dma_map_alloc - Allocate HMM map structure
* @dev: device to allocate structure for