When e.g. an application thread takes a page fault on an mmapped file, MM calls
`memmap.Mappable.Translate()` to obtain the corresponding host FD range that
should be mapped into the application's address space. It passes both the range
that *must* be mapped (e.g. the faulting page) as `required`, and the maximum
range that *may* be mapped (the previously-unfaulted part of the corresponding
VMA) as `optional`, such that file implementations can map more than `required`
to avoid future page faults.
Prior to this CL, `tmpfs.regularFile.Translate()` always returned translations
up to `optional`, under the assumption that allocating larger ranges from
`pgalloc.MemoryFile` has negligible incremental cost. This behavior dates to
the introduction of `memmap.Mappable.Translate()` (cl/182882705) and thus
predates the implementation of tmpfs size limits (cl/442686814). Now that the
latter exists, unconditionally translating - and therefore allocating pages -
up to `optional` can result in hitting tmpfs size limits prematurely.
Thus: Constrain optional translations returned by
`tmpfs.regularFile.Translate()`, applying the same logic as
`gofer.maxFillRange()`.
PiperOrigin-RevId: 713134287
When using overlayfs with tmpfs as the upper layer (common case), depending on
the application, a lot of whiteouts can be created. This leads to a lot of
memory allocation because new dentry and inode structs need to be allocated for
each whiteout. With this change, we at least avoid the inode allocations.
This is analogous with what Linux does. See fs/overlayfs/ovl_entry.h:ovl_fs's
field `whiteout` with comment "Shared whiteout cache".
PiperOrigin-RevId: 704897805
For vfs.FileDescriptions for which FileDescriptionOptions.UseDentryMetadata is
true, memmap.MappingIdentity.Device/InodeID() => FileDescription.Stat() =>
FilesystemImpl.StatAt() takes fsimpl locks for path traversal, which violates
the lock ordering and is unnecessary since no path is being traversed. Fix this
by carving out a special case where FilesystemImpl.Stat() (and
FileDescriptionImpl.Stat()) are required to meet the lock ordering requirements
of memmap.MappingIdentity.Device/InodeID(), and implement that special case by
skipping path traversal (and gofer revalidation) locks when not required.
PiperOrigin-RevId: 698608924
This lock was introduced in cl/696713993 and only protects parent/name for all
dentries, which is all that's required by InotifyWithParent().
PiperOrigin-RevId: 697016824
- Add type parameter Filesystem to vfs/genericfstree, which is required to
provide `ancestryMu sync.RWMutex`, and add such a RWMutex to all FSImpls that
use genericfstree.
- Modify genericfstree.PrependPath() and genericfstree.IsDescendant() to use
ancestryMu to ensure atomicity. For callers of genericfstree.PrependPath(),
this means that (broader) FSImpl locks no longer need to be held during the
call. For callers of genericfstree.IsDescendant(), this means that we can
remove documentation warnings about its non-atomicity.
- Minor cleanup: Remove useless variable `start`, which is always 0, from
MM.ReadMaps/SmapsDataInto().
PiperOrigin-RevId: 696713993
This CL addresses the following major issues:
- When an application releases memory to the sentry, the sentry unconditionally
releases that memory to the host, rather than allowing it to be reused for
future allocations, in order to ensure that new allocations are uniformly
decommitted (use no memory): cl/145016083. In most cases, this should have
relatively little performance impact; since releasing memory from the
application to the OS is expensive even outside of gVisor, application memory
allocators optimizing for performance already limit the rate at which they
release memory to the OS. However, in applications that involve frequent
process creation and exit (e.g. build systems), this practice prevents reuse
of memory deallocated by exiting processes for memory allocated by new
processes, resulting in both performance degradation and a spike in memory
usage (since the sentry may not have released all deallocated memory to the
host by the time new allocations occur).
- gVisor's historical approach to application THP is based on THP being enabled
on a per-memfd basis, using the MFD_HUGEPAGE flag not merged into the
upstream Linux kernel
(https://patchwork.kernel.org/project/linux-mm/patch/c140f56a-1aa3-f7ae-b7d1-93da7d5a3572@google.com/).
Thus, on vanilla Linux kernels, gVisor cannot use THP for application memory
without requiring the system to enable THP for all tmpfs files and memfds (by
setting /sys/kernel/mm/transparent_hugepage/shmem_enabled to "always" or
"force").
- Both MM and the application page allocator (pgalloc) are agnostic as to
whether the underlying memory file will be THP-backed. Instead, both attempt
to align hugepage-sized and larger allocations to hugepage boundaries, such
that if the memory file happens to support THP then such allocations will be
appropriately aligned to use THP. This is suboptimal since many allocations
do not benefit from THP, resulting in memory underutilization.
These issues are especially relevant to platforms based on hardware
virtualization, where acquiring memory from the host is significantly more
expensive due to EPT/NPT fault overhead; when effective, THP reduces the
frequency with which said cost is incurred by a factor of 512, and page reuse
avoids incurring it at all.
Thus:
- Instead of inferring whether THP use is desired from allocation size,
indicate this explicitly as AllocOpts.Huge, and only set it to true for
allocations for non-stack private anonymous mappings.
- Add AllocateCallerIndirectCommit, a new possible value for AllocOpts.Mode
that indicates that the caller will commit all pages in the allocation. In
such cases, pgalloc can reuse deallocated pages without risking increased
memory usage, internally referred to as "recycling".
AllocateCallerIndirectCommit is used primarily for page faults on a
THP-backed region. (It is also used for single-page allocations on non-THP
backed regions, but due to expansion of faults to mm.privateAllocUnit-aligned
ranges, this is relatively uncommon.)
- Allow different chunks in pgalloc.MemoryFile's backing file to have varying
THP-ness, indicated to the host using MADV_HUGEPAGE/NOHUGEPAGE.
- Split pgalloc.MemoryFile's existing page metadata set into two sets tracking
deallocated pages for small/huge-page-backed regions respectively; two sets
tracking in-use pages for small/huge-page-backed regions respectively; and a
fifth set tracking memory accounting state.
- Add MemoryFileOpts.DisableMemoryAccounting; this is primarily intended for
pgalloc tests, but may also be applicable to disk-backed MemoryFiles.
Cleanup:
- Remove MemoryFile.usageSwapped; the UpdateUsage() optimization it enabled,
described in updateUsageLocked(), was based on the condition that
MemoryFile.mu would be locked throughout the call to updateUsageLocked(),
which was invalidated by cl/337865250.
- Remove MemoryFileOpts.ManualZeroing, which is unused.
- Rename "reclaiming" to "releasing"; the former is confusing since "reclaim"
in Linux has a significantly different meaning (essentially "eviction" in
pgalloc), and the latter seems to be conventional in user-mode memory
allocators.
Using THP for application memory requires setting
/sys/kernel/mm/transparent_hugepage/shmem_enabled to "advise", in order to
allow runsc to request THP from the kernel.
After this CL, pgalloc.MemoryFile still releases memory to the host as fast as
possible, limiting the effectiveness of page recycling. A following CL adds
optional memory release throttling to improve this.
Performance outcomes vary by workload and platform. (In all of the below,
"baseline" is without this CL, "expt" is with this CL, and "expt2" is with this
CL + reclaim throttling (cl/575046398).)
For systrap in GKE: As noted, this change is required to enable application THP
without forcing it on all host shmem users. In conjunction with recycling
(which has a relatively small effect on systrap since it does not use hardware
virtualization), THP use slightly improves performance, although whether this
is measurable is case-dependent. On an idle VM, with shmem_enabled = "advise":
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
│ baseline │ expt │ expt2 │
│ sec/op │ sec/op vs base │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-16 39.09 ± 4% 38.84 ± 5% ~ (p=0.947 n=30) 38.84 ± 3% ~ (p=0.854 n=30)
BuildABSL/page_cache.dirty/filesystem.bindfs-16 37.83 ± 3% 36.58 ± 4% ~ (p=0.057 n=30) 36.83 ± 5% ~ (p=0.314 n=30)
BuildABSL/page_cache.clean/filesystem.tmpfs-16 39.34 ± 3% 38.59 ± 4% ~ (p=0.350 n=30) 38.58 ± 4% ~ (p=0.300 n=30)
BuildABSL/page_cache.dirty/filesystem.tmpfs-16 37.83 ± 3% 36.08 ± 4% -4.64% (p=0.026 n=30) 36.58 ± 4% ~ (p=0.123 n=30)
BuildABSL/page_cache.clean/filesystem.rootfs-16 39.59 ± 4% 38.83 ± 3% ~ (p=0.485 n=30) 40.09 ± 5% ~ (p=0.971 n=30)
BuildABSL/page_cache.dirty/filesystem.rootfs-16 36.83 ± 3% 38.08 ± 5% ~ (p=0.307 n=30) 38.08 ± 1% ~ (p=0.242 n=30)
BuildABSL/page_cache.clean/filesystem.fusefs-16 38.34 ± 3% 37.59 ± 5% ~ (p=0.752 n=30) 38.59 ± 3% ~ (p=0.982 n=30)
BuildABSL/page_cache.dirty/filesystem.fusefs-16 37.58 ± 4% 38.08 ± 5% ~ (p=0.708 n=30) 36.08 ± 6% ~ (p=0.127 n=30)
BuildGRPC/page_cache.clean/filesystem.bindfs-16 212.7 ± 2% 211.0 ± 1% ~ (p=0.138 n=30) 211.2 ± 1% ~ (p=0.458 n=30)
BuildGRPC/page_cache.dirty/filesystem.bindfs-16 210.0 ± 1% 210.0 ± 1% ~ (p=0.542 n=30) 209.7 ± 1% ~ (p=0.665 n=30)
BuildGRPC/page_cache.clean/filesystem.rootfs-16 210.5 ± 1% 210.0 ± 1% ~ (p=0.423 n=30) 210.0 ± 1% ~ (p=0.142 n=30)
BuildGRPC/page_cache.dirty/filesystem.rootfs-16 210.2 ± 1% 209.0 ± 1% ~ (p=0.219 n=30) 209.5 ± 1% ~ (p=0.230 n=30)
geomean 67.62 66.97 -0.96% 67.12 -0.74%
```
The KVM platform benefits significantly from reduced nested page faults due to
huge pages, and to a lesser extent due to recycling:
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz
│ baseline │ expt │ expt2 │
│ sec/op │ sec/op vs base │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-12 43.11 ± 2% 39.35 ± 3% -8.71% (p=0.000 n=20) 38.10 ± 4% -11.63% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.bindfs-12 42.35 ± 3% 39.09 ± 4% -7.69% (p=0.000 n=20+19) 39.09 ± 5% -7.69% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.tmpfs-12 42.35 ± 3% 38.34 ± 5% -9.46% (p=0.000 n=20) 38.59 ± 3% -8.87% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.tmpfs-12 42.09 ± 1% 37.59 ± 4% -10.70% (p=0.000 n=20) 38.09 ± 4% -9.51% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.rootfs-12 42.85 ± 3% 38.84 ± 3% -9.35% (p=0.000 n=20) 39.09 ± 3% -8.77% (p=0.000 n=20+17)
BuildABSL/page_cache.dirty/filesystem.rootfs-12 41.85 ± 2% 39.59 ± 6% -5.40% (p=0.000 n=20+19) 38.09 ± 3% -9.00% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.fusefs-12 42.60 ± 2% 38.34 ± 2% -10.00% (p=0.000 n=20) 39.59 ± 3% -7.06% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.fusefs-12 42.09 ± 4% 39.09 ± 3% -7.13% (p=0.000 n=20) 38.09 ± 3% -9.52% (p=0.000 n=20+19)
BuildGRPC/page_cache.clean/filesystem.bindfs-12 207.7 ± 1% 206.4 ± 0% -0.60% (p=0.018 n=20) 205.9 ± 1% -0.85% (p=0.001 n=20+19)
BuildGRPC/page_cache.dirty/filesystem.bindfs-12 206.9 ± 1% 206.9 ± 1% ~ (p=0.121 n=20) 204.4 ± 1% -1.22% (p=0.004 n=20+19)
BuildGRPC/page_cache.clean/filesystem.rootfs-12 207.7 ± 1% 204.9 ± 1% -1.33% (p=0.004 n=20) 203.9 ± 0% -1.81% (p=0.000 n=20+19)
BuildGRPC/page_cache.dirty/filesystem.rootfs-12 206.9 ± 1% 204.9 ± 0% -0.97% (p=0.004 n=20+19) 203.9 ± 0% -1.45% (p=0.000 n=20+19)
geomean 71.97 67.63 -6.03% 67.28 -6.52%
```
PiperOrigin-RevId: 647771821
The work done in c087777e37 ("Plumb restore context to afterLoad()") makes
pgalloc.MemoryFileProvider redundant as structs can now easily restore
pgalloc.MemoryFile in stateify's afterLoad() method. This allows structs to
have a pgalloc.MemoryFile field and use that directly, instead of going through
the provided interface.
This cleans up a lot of code and also should be more performant (avoids an
interface method call on many hot paths).
PiperOrigin-RevId: 615258927
Earlier we were always restoring pma.file to mm.mfp.MemoryFile(). However,
d8eb29ed6f ("Add support for saving PMAs referencing tmpfs filestore files.")
added support for saving PMAs that reference "private" pgalloc.MemoryFiles that
are different from mm.mfp.MemoryFile().
We achieve the correct restore by:
- Adding a "RestoreID" field to pgalloc.MemoryFile. Private MemoryFiles set
this with a vfs.RestoreID.String(). Non-private MemoryFile does not set it.
- MemoryFile struct is not savable by itself, but pma.file field is saved as a
string. We store the RestoreID string there.
- On restore, if RestoreID is "", then restore using CtxMemoryFile. If it has a
non-empty RestoreID, then restore using CtxMemoryFileMap.
- Cleanup: vfs.CtxFilesystemMemoryFileMap was moved to pgalloc.CtxMemoryFileMap
so we can now provide a pgalloc.MemoryFileMapFromContext() method which
cleans up some code. Also the key to this map (MemoryFileOpts.RestoreID)
belongs to pgalloc, so it seems like the right place to have this context.
PiperOrigin-RevId: 614903073
This allows for external information to be passed to restore code.
Similar to c087777e37 ("Plumb restore context to afterLoad()").
Updates #1956.
PiperOrigin-RevId: 614125262
Tmpfs with file-backed are widely used:
1. Via --overlay2 flag. The default is root:self so the root mount uses this.
2. EmptyDir mounts with default medium are created as tmpfs with file backend.
This change unblocks (1) use case from being used with checkpoint/restore.
For (2), checkpoint/restore is not yet supported in multicontainers.
Most notably, this allows checkpoint/restore to work with default runsc flags.
PiperOrigin-RevId: 586291915
It is an idea of running codespell as part of our presubmit checks.
Before enabling it for new changes, let's fix what it has found.
Signed-off-by: Andrei Vagin <avagin@gmail.com>
It is not ideal to access a mount's Flags directly, gVisor will access a mount's flags via MountOptions() which locks mount's mu to avoid data race.
PiperOrigin-RevId: 576059331
On machines with really large RAM, the default tmpfs size limit (50% of RAM
size) can still be really large. So tmpfs permits really large fallocate
requests. In gVisor, there are 2 problems:
1. Large fallocate(2) may crash the system via watchdog timeout.
2. fallocate(2) is not interruptible by signal.
In Linux, fallocate iteratively allocates folios, while checking for
interrupts. See mm/shmem.c:shmem_fallocate().
To solve these problems, this patch changes tmpfs Allocate() to work in
4 GiB chunks and checks for interrupts between chunks, which also pets the
watchdog. We also drop tmpfs.regularFile.dataMu between chunks to prevent other
goroutines waiting on dataMu from triggering a watchdog timeout.
Note that this still may not fix watchdog crashes caused by other goroutines
waiting on tmpfs.inode.mu during a large fallocate(2).
Reported-by: syzbot+269e5666da366a3ef2f6@syzkaller.appspotmail.com
Suggested-by: Jamie Liu <jamieliu@google.com>
PiperOrigin-RevId: 570551795
IsDescendant gives the VFS layer an easy way to check if a dentry is a
descendant of another, which is important for some mounting procedures.
This method does not take any locks when accessing the parent, so parent
fields need to switch to be atomic to avoid data races.
PiperOrigin-RevId: 570454446
- Update the remaining platform.File references in comments and
documentation. platform.File has been moved in memmap by commit
f347a578b7 ("Move platform.File in memmap").
- Remove redundant bit operations in statTo in tmpfs. STATX_SIZE and
STATX_BLOCKS has been set at the beginning of the function, there
is no need to do it again for regular files.
- Change "\tmp" to "/tmp" in pathname, as '\' is not a valid path
delimiter in Linux. "/tmp" looks better.
Signed-off-by: Tiwei Bie <tiwei.btw@antgroup.com>
gVisor bot
Jamie LiuandgVisor bot
Ayush RanjanandgVisor bot
Jing Chen
Fabricio VoznikaandgVisor bot
Jing ChenandgVisor bot
Andrei Vagin
Lucas ManningandgVisor bot
Tiwei Bie