This support is only needed when the gofer mount in question is writable.
By default, the rootfs has an overlayfs applied, so the gofer lower layer is
not writabled. But if you are using --overlay2=none, then this change should
allow you to save sandbox with open FDs to deleted files in rootfs.
Updates #11425
PiperOrigin-RevId: 733021267
auth.K{U/G}ID has type uint32. When uid = auth.NoID = math.MaxUint32, which is
intended to represent -1, and it is casted to int, it becomes 4294967295 on
64-bit systems as int is 8 bytes in size. So the == -1 check fails.
Fixed the bug by explicitly setting syscall args to -1 when uid.Ok() == false.
Similarly, fix the bug in runsc/fsgofer.
PiperOrigin-RevId: 728258705
This fixes save/restore of mmapped gofer.specialFileFDs (usually obtained via
mount option "disable_file_handle_sharing") for which mappings have actually
been used (mm.pmas have been obtained).
PiperOrigin-RevId: 700465816
For vfs.FileDescriptions for which FileDescriptionOptions.UseDentryMetadata is
true, memmap.MappingIdentity.Device/InodeID() => FileDescription.Stat() =>
FilesystemImpl.StatAt() takes fsimpl locks for path traversal, which violates
the lock ordering and is unnecessary since no path is being traversed. Fix this
by carving out a special case where FilesystemImpl.Stat() (and
FileDescriptionImpl.Stat()) are required to meet the lock ordering requirements
of memmap.MappingIdentity.Device/InodeID(), and implement that special case by
skipping path traversal (and gofer revalidation) locks when not required.
PiperOrigin-RevId: 698608924
This lock was introduced in cl/696713993 and only protects parent/name for all
dentries, which is all that's required by InotifyWithParent().
PiperOrigin-RevId: 697016824
- Add type parameter Filesystem to vfs/genericfstree, which is required to
provide `ancestryMu sync.RWMutex`, and add such a RWMutex to all FSImpls that
use genericfstree.
- Modify genericfstree.PrependPath() and genericfstree.IsDescendant() to use
ancestryMu to ensure atomicity. For callers of genericfstree.PrependPath(),
this means that (broader) FSImpl locks no longer need to be held during the
call. For callers of genericfstree.IsDescendant(), this means that we can
remove documentation warnings about its non-atomicity.
- Minor cleanup: Remove useless variable `start`, which is always 0, from
MM.ReadMaps/SmapsDataInto().
PiperOrigin-RevId: 696713993
As of writing, gVisor does not support fchmodat2(2). Without it, there is no
way to change permissions for a symlink file. Note that fchmod(2) on an O_PATH
FD fails with EBADF. Also fchmodat(2) (which is supported) does not take any
flags and AT_SYMLINK_NOFOLLOW is needed to change permissions on a symlink.
But if in the future we were to support fchmodat2(2), goferfs would return
EBADF because it would try to change mode using fchmod(2) on a O_PATH FD.
Updated goferfs to explicitly return EOPNOTSUPP (consistent with Linux).
Some other minor fixes:
- The comment about why sockets needed special handling for changing mode was
incorrect. Sockets too have a O_PATH control FD so fchmod(2) on it would fail
with EBADF. This was the real reason.
- The comment about AT_SYMLINK_NOFOLLOW not being supported is stale. Support
for fchmodat2(2) along with AT_SYMLINK_NOFOLLOW and AT_EMPTY_PATH flags was
added in Linux 6.6. Removed the comment for now. We should probably use
fchmodat2(2) when it is available because it is safer.
- When --directfs=true and the application tries to change mode of a mountpoint
socket, gofer client makes an RPC to the fsgofer server to do that work.
fsgofer attempts the mode change using fchmodat(2). However, the fsgofer's
seccomp filters did not allow fchmodat(2) in --directfs=true mode. Added
fchmodat(2) to fsgofer seccomp filters unconditionally.
Updates #10385
PiperOrigin-RevId: 687385378
In goferfs, the control FD (in gofer client in directfs mode) and host FD in
fsgofer server is an O_PATH FD for socket and symlink files.
fgetxattr(2) fails with EBADF for O_PATH fds. So use lgetxattr(2) instead. This
is a path-based syscall, so it should work for symlinks and sockets. Since the
gofer client can not make path-based syscalls, it falls back to lisafs.
Fixes#11049
Updates #10385
PiperOrigin-RevId: 687160973
memmap.Mappable.Translate() is passed a hostarch.AccessType indicating what
permissions are *immediately* required; it returns permissions in
memmap.Translation.Perms that are granted *to MM* until invalidation. MM
ensures that the permissions granted to the application are the intersection of
those granted by Translate, and those granted by VMA permissions; see
determination of pma.effectivePerms in
mm.MemoryManager.getPMAsInternalLocked(). This mechanism is used to avoid
marking pages dirty in the sentry's page cache for gofer-backed files until
PROT_WRITE pages are actually written to; see gofer.dentry.Translate(). In most
other cases, granting all supported permissions (to MM) up-front avoids a
redundant page fault for pages that are touched first for reading, and later
for writing.
Also:
- Prevent PROT_WRITE mappings of erofs files at mmap()/mprotect() time, rather
than raising SIGBUS when writing to such mappings.
- Map nvproxy.frontendFD with PlatformEffectPopulate. This is the original goal
of this CL; however, before this rest of this CL, MM.MMap() =>
MM.populateVMAAndUnlock() => MM.getPMAsLocked(at=hostarch.NoAccess) =>
MM.getPMAsInternalLocked(at=hostarch.NoAccess) =>
nvproxy.frontendFD.Translate(at=hostarch.NoAccess) returns Translations with
no permissions, causing MM.mapASLocked() to no-op.
PiperOrigin-RevId: 679360594
These support the relatively common use case of removing all segments in a
given range (unconditionally) but doing something with them before they're
removed. This is always more compact, and may be slightly faster in some cases
(every replaced loop calls Isolate per iteration, while RemoveRangeWith avoids
redundant split checks between segments), at the cost of a direct function
call.
Also slightly optimize Set.LowerBoundSegmentSplitBefore() and
Set.UpperBoundSegmentSplitAfter() by inlining LowerBoundSegment and
UpperBoundSegment respectively; in the cases where Find() returns a
GapIterator, the segment that is returned doesn't need to be split since it
doesn't contain min/max respectively.
PiperOrigin-RevId: 675824581
This CL addresses the following major issues:
- When an application releases memory to the sentry, the sentry unconditionally
releases that memory to the host, rather than allowing it to be reused for
future allocations, in order to ensure that new allocations are uniformly
decommitted (use no memory): cl/145016083. In most cases, this should have
relatively little performance impact; since releasing memory from the
application to the OS is expensive even outside of gVisor, application memory
allocators optimizing for performance already limit the rate at which they
release memory to the OS. However, in applications that involve frequent
process creation and exit (e.g. build systems), this practice prevents reuse
of memory deallocated by exiting processes for memory allocated by new
processes, resulting in both performance degradation and a spike in memory
usage (since the sentry may not have released all deallocated memory to the
host by the time new allocations occur).
- gVisor's historical approach to application THP is based on THP being enabled
on a per-memfd basis, using the MFD_HUGEPAGE flag not merged into the
upstream Linux kernel
(https://patchwork.kernel.org/project/linux-mm/patch/c140f56a-1aa3-f7ae-b7d1-93da7d5a3572@google.com/).
Thus, on vanilla Linux kernels, gVisor cannot use THP for application memory
without requiring the system to enable THP for all tmpfs files and memfds (by
setting /sys/kernel/mm/transparent_hugepage/shmem_enabled to "always" or
"force").
- Both MM and the application page allocator (pgalloc) are agnostic as to
whether the underlying memory file will be THP-backed. Instead, both attempt
to align hugepage-sized and larger allocations to hugepage boundaries, such
that if the memory file happens to support THP then such allocations will be
appropriately aligned to use THP. This is suboptimal since many allocations
do not benefit from THP, resulting in memory underutilization.
These issues are especially relevant to platforms based on hardware
virtualization, where acquiring memory from the host is significantly more
expensive due to EPT/NPT fault overhead; when effective, THP reduces the
frequency with which said cost is incurred by a factor of 512, and page reuse
avoids incurring it at all.
Thus:
- Instead of inferring whether THP use is desired from allocation size,
indicate this explicitly as AllocOpts.Huge, and only set it to true for
allocations for non-stack private anonymous mappings.
- Add AllocateCallerIndirectCommit, a new possible value for AllocOpts.Mode
that indicates that the caller will commit all pages in the allocation. In
such cases, pgalloc can reuse deallocated pages without risking increased
memory usage, internally referred to as "recycling".
AllocateCallerIndirectCommit is used primarily for page faults on a
THP-backed region. (It is also used for single-page allocations on non-THP
backed regions, but due to expansion of faults to mm.privateAllocUnit-aligned
ranges, this is relatively uncommon.)
- Allow different chunks in pgalloc.MemoryFile's backing file to have varying
THP-ness, indicated to the host using MADV_HUGEPAGE/NOHUGEPAGE.
- Split pgalloc.MemoryFile's existing page metadata set into two sets tracking
deallocated pages for small/huge-page-backed regions respectively; two sets
tracking in-use pages for small/huge-page-backed regions respectively; and a
fifth set tracking memory accounting state.
- Add MemoryFileOpts.DisableMemoryAccounting; this is primarily intended for
pgalloc tests, but may also be applicable to disk-backed MemoryFiles.
Cleanup:
- Remove MemoryFile.usageSwapped; the UpdateUsage() optimization it enabled,
described in updateUsageLocked(), was based on the condition that
MemoryFile.mu would be locked throughout the call to updateUsageLocked(),
which was invalidated by cl/337865250.
- Remove MemoryFileOpts.ManualZeroing, which is unused.
- Rename "reclaiming" to "releasing"; the former is confusing since "reclaim"
in Linux has a significantly different meaning (essentially "eviction" in
pgalloc), and the latter seems to be conventional in user-mode memory
allocators.
Using THP for application memory requires setting
/sys/kernel/mm/transparent_hugepage/shmem_enabled to "advise", in order to
allow runsc to request THP from the kernel.
After this CL, pgalloc.MemoryFile still releases memory to the host as fast as
possible, limiting the effectiveness of page recycling. A following CL adds
optional memory release throttling to improve this.
Performance outcomes vary by workload and platform. (In all of the below,
"baseline" is without this CL, "expt" is with this CL, and "expt2" is with this
CL + reclaim throttling (cl/575046398).)
For systrap in GKE: As noted, this change is required to enable application THP
without forcing it on all host shmem users. In conjunction with recycling
(which has a relatively small effect on systrap since it does not use hardware
virtualization), THP use slightly improves performance, although whether this
is measurable is case-dependent. On an idle VM, with shmem_enabled = "advise":
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
│ baseline │ expt │ expt2 │
│ sec/op │ sec/op vs base │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-16 39.09 ± 4% 38.84 ± 5% ~ (p=0.947 n=30) 38.84 ± 3% ~ (p=0.854 n=30)
BuildABSL/page_cache.dirty/filesystem.bindfs-16 37.83 ± 3% 36.58 ± 4% ~ (p=0.057 n=30) 36.83 ± 5% ~ (p=0.314 n=30)
BuildABSL/page_cache.clean/filesystem.tmpfs-16 39.34 ± 3% 38.59 ± 4% ~ (p=0.350 n=30) 38.58 ± 4% ~ (p=0.300 n=30)
BuildABSL/page_cache.dirty/filesystem.tmpfs-16 37.83 ± 3% 36.08 ± 4% -4.64% (p=0.026 n=30) 36.58 ± 4% ~ (p=0.123 n=30)
BuildABSL/page_cache.clean/filesystem.rootfs-16 39.59 ± 4% 38.83 ± 3% ~ (p=0.485 n=30) 40.09 ± 5% ~ (p=0.971 n=30)
BuildABSL/page_cache.dirty/filesystem.rootfs-16 36.83 ± 3% 38.08 ± 5% ~ (p=0.307 n=30) 38.08 ± 1% ~ (p=0.242 n=30)
BuildABSL/page_cache.clean/filesystem.fusefs-16 38.34 ± 3% 37.59 ± 5% ~ (p=0.752 n=30) 38.59 ± 3% ~ (p=0.982 n=30)
BuildABSL/page_cache.dirty/filesystem.fusefs-16 37.58 ± 4% 38.08 ± 5% ~ (p=0.708 n=30) 36.08 ± 6% ~ (p=0.127 n=30)
BuildGRPC/page_cache.clean/filesystem.bindfs-16 212.7 ± 2% 211.0 ± 1% ~ (p=0.138 n=30) 211.2 ± 1% ~ (p=0.458 n=30)
BuildGRPC/page_cache.dirty/filesystem.bindfs-16 210.0 ± 1% 210.0 ± 1% ~ (p=0.542 n=30) 209.7 ± 1% ~ (p=0.665 n=30)
BuildGRPC/page_cache.clean/filesystem.rootfs-16 210.5 ± 1% 210.0 ± 1% ~ (p=0.423 n=30) 210.0 ± 1% ~ (p=0.142 n=30)
BuildGRPC/page_cache.dirty/filesystem.rootfs-16 210.2 ± 1% 209.0 ± 1% ~ (p=0.219 n=30) 209.5 ± 1% ~ (p=0.230 n=30)
geomean 67.62 66.97 -0.96% 67.12 -0.74%
```
The KVM platform benefits significantly from reduced nested page faults due to
huge pages, and to a lesser extent due to recycling:
```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz
│ baseline │ expt │ expt2 │
│ sec/op │ sec/op vs base │ sec/op vs base │
BuildABSL/page_cache.clean/filesystem.bindfs-12 43.11 ± 2% 39.35 ± 3% -8.71% (p=0.000 n=20) 38.10 ± 4% -11.63% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.bindfs-12 42.35 ± 3% 39.09 ± 4% -7.69% (p=0.000 n=20+19) 39.09 ± 5% -7.69% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.tmpfs-12 42.35 ± 3% 38.34 ± 5% -9.46% (p=0.000 n=20) 38.59 ± 3% -8.87% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.tmpfs-12 42.09 ± 1% 37.59 ± 4% -10.70% (p=0.000 n=20) 38.09 ± 4% -9.51% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.rootfs-12 42.85 ± 3% 38.84 ± 3% -9.35% (p=0.000 n=20) 39.09 ± 3% -8.77% (p=0.000 n=20+17)
BuildABSL/page_cache.dirty/filesystem.rootfs-12 41.85 ± 2% 39.59 ± 6% -5.40% (p=0.000 n=20+19) 38.09 ± 3% -9.00% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.fusefs-12 42.60 ± 2% 38.34 ± 2% -10.00% (p=0.000 n=20) 39.59 ± 3% -7.06% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.fusefs-12 42.09 ± 4% 39.09 ± 3% -7.13% (p=0.000 n=20) 38.09 ± 3% -9.52% (p=0.000 n=20+19)
BuildGRPC/page_cache.clean/filesystem.bindfs-12 207.7 ± 1% 206.4 ± 0% -0.60% (p=0.018 n=20) 205.9 ± 1% -0.85% (p=0.001 n=20+19)
BuildGRPC/page_cache.dirty/filesystem.bindfs-12 206.9 ± 1% 206.9 ± 1% ~ (p=0.121 n=20) 204.4 ± 1% -1.22% (p=0.004 n=20+19)
BuildGRPC/page_cache.clean/filesystem.rootfs-12 207.7 ± 1% 204.9 ± 1% -1.33% (p=0.004 n=20) 203.9 ± 0% -1.81% (p=0.000 n=20+19)
BuildGRPC/page_cache.dirty/filesystem.rootfs-12 206.9 ± 1% 204.9 ± 0% -0.97% (p=0.004 n=20+19) 203.9 ± 0% -1.45% (p=0.000 n=20+19)
geomean 71.97 67.63 -6.03% 67.28 -6.52%
```
PiperOrigin-RevId: 647771821
Mount points are created during container initialization and may
not exist if the container is restored using the original image.
Mark these directories and recreate them upon restore.
PiperOrigin-RevId: 644514801
Jamie LiuandgVisor bot
Ayush RanjanandgVisor bot
xianzhe-databricks
cweld510
Nicolas LacasseandgVisor bot
Fabricio VoznikaandgVisor bot