Add explicit huge page and memory recycling support to pgalloc.MemoryFile.

This CL addresses the following major issues:

- When an application releases memory to the sentry, the sentry unconditionally
  releases that memory to the host, rather than allowing it to be reused for
  future allocations, in order to ensure that new allocations are uniformly
  decommitted (use no memory): cl/145016083. In most cases, this should have
  relatively little performance impact; since releasing memory from the
  application to the OS is expensive even outside of gVisor, application memory
  allocators optimizing for performance already limit the rate at which they
  release memory to the OS. However, in applications that involve frequent
  process creation and exit (e.g. build systems), this practice prevents reuse
  of memory deallocated by exiting processes for memory allocated by new
  processes, resulting in both performance degradation and a spike in memory
  usage (since the sentry may not have released all deallocated memory to the
  host by the time new allocations occur).

- gVisor's historical approach to application THP is based on THP being enabled
  on a per-memfd basis, using the MFD_HUGEPAGE flag not merged into the
  upstream Linux kernel
  (https://patchwork.kernel.org/project/linux-mm/patch/c140f56a-1aa3-f7ae-b7d1-93da7d5a3572@google.com/).
  Thus, on vanilla Linux kernels, gVisor cannot use THP for application memory
  without requiring the system to enable THP for all tmpfs files and memfds (by
  setting /sys/kernel/mm/transparent_hugepage/shmem_enabled to "always" or
  "force").

- Both MM and the application page allocator (pgalloc) are agnostic as to
  whether the underlying memory file will be THP-backed. Instead, both attempt
  to align hugepage-sized and larger allocations to hugepage boundaries, such
  that if the memory file happens to support THP then such allocations will be
  appropriately aligned to use THP. This is suboptimal since many allocations
  do not benefit from THP, resulting in memory underutilization.

These issues are especially relevant to platforms based on hardware
virtualization, where acquiring memory from the host is significantly more
expensive due to EPT/NPT fault overhead; when effective, THP reduces the
frequency with which said cost is incurred by a factor of 512, and page reuse
avoids incurring it at all.

Thus:

- Instead of inferring whether THP use is desired from allocation size,
  indicate this explicitly as AllocOpts.Huge, and only set it to true for
  allocations for non-stack private anonymous mappings.

- Add AllocateCallerIndirectCommit, a new possible value for AllocOpts.Mode
  that indicates that the caller will commit all pages in the allocation. In
  such cases, pgalloc can reuse deallocated pages without risking increased
  memory usage, internally referred to as "recycling".
  AllocateCallerIndirectCommit is used primarily for page faults on a
  THP-backed region. (It is also used for single-page allocations on non-THP
  backed regions, but due to expansion of faults to mm.privateAllocUnit-aligned
  ranges, this is relatively uncommon.)

- Allow different chunks in pgalloc.MemoryFile's backing file to have varying
  THP-ness, indicated to the host using MADV_HUGEPAGE/NOHUGEPAGE.

- Split pgalloc.MemoryFile's existing page metadata set into two sets tracking
  deallocated pages for small/huge-page-backed regions respectively; two sets
  tracking in-use pages for small/huge-page-backed regions respectively; and a
  fifth set tracking memory accounting state.

- Add MemoryFileOpts.DisableMemoryAccounting; this is primarily intended for
  pgalloc tests, but may also be applicable to disk-backed MemoryFiles.

Cleanup:

- Remove MemoryFile.usageSwapped; the UpdateUsage() optimization it enabled,
  described in updateUsageLocked(), was based on the condition that
  MemoryFile.mu would be locked throughout the call to updateUsageLocked(),
  which was invalidated by cl/337865250.

- Remove MemoryFileOpts.ManualZeroing, which is unused.

- Rename "reclaiming" to "releasing"; the former is confusing since "reclaim"
  in Linux has a significantly different meaning (essentially "eviction" in
  pgalloc), and the latter seems to be conventional in user-mode memory
  allocators.

Using THP for application memory requires setting
/sys/kernel/mm/transparent_hugepage/shmem_enabled to "advise", in order to
allow runsc to request THP from the kernel.

After this CL, pgalloc.MemoryFile still releases memory to the host as fast as
possible, limiting the effectiveness of page recycling. A following CL adds
optional memory release throttling to improve this.

Performance outcomes vary by workload and platform. (In all of the below,
"baseline" is without this CL, "expt" is with this CL, and "expt2" is with this
CL + reclaim throttling (cl/575046398).)

For systrap in GKE: As noted, this change is required to enable application THP
without forcing it on all host shmem users. In conjunction with recycling
(which has a relatively small effect on systrap since it does not use hardware
virtualization), THP use slightly improves performance, although whether this
is measurable is case-dependent. On an idle VM, with shmem_enabled = "advise":

```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) CPU @ 2.80GHz
                                                │  baseline  │               expt                │               expt2               │
                                                │   sec/op   │   sec/op    vs base               │   sec/op    vs base               │
BuildABSL/page_cache.clean/filesystem.bindfs-16   39.09 ± 4%   38.84 ± 5%       ~ (p=0.947 n=30)   38.84 ± 3%       ~ (p=0.854 n=30)
BuildABSL/page_cache.dirty/filesystem.bindfs-16   37.83 ± 3%   36.58 ± 4%       ~ (p=0.057 n=30)   36.83 ± 5%       ~ (p=0.314 n=30)
BuildABSL/page_cache.clean/filesystem.tmpfs-16    39.34 ± 3%   38.59 ± 4%       ~ (p=0.350 n=30)   38.58 ± 4%       ~ (p=0.300 n=30)
BuildABSL/page_cache.dirty/filesystem.tmpfs-16    37.83 ± 3%   36.08 ± 4%  -4.64% (p=0.026 n=30)   36.58 ± 4%       ~ (p=0.123 n=30)
BuildABSL/page_cache.clean/filesystem.rootfs-16   39.59 ± 4%   38.83 ± 3%       ~ (p=0.485 n=30)   40.09 ± 5%       ~ (p=0.971 n=30)
BuildABSL/page_cache.dirty/filesystem.rootfs-16   36.83 ± 3%   38.08 ± 5%       ~ (p=0.307 n=30)   38.08 ± 1%       ~ (p=0.242 n=30)
BuildABSL/page_cache.clean/filesystem.fusefs-16   38.34 ± 3%   37.59 ± 5%       ~ (p=0.752 n=30)   38.59 ± 3%       ~ (p=0.982 n=30)
BuildABSL/page_cache.dirty/filesystem.fusefs-16   37.58 ± 4%   38.08 ± 5%       ~ (p=0.708 n=30)   36.08 ± 6%       ~ (p=0.127 n=30)
BuildGRPC/page_cache.clean/filesystem.bindfs-16   212.7 ± 2%   211.0 ± 1%       ~ (p=0.138 n=30)   211.2 ± 1%       ~ (p=0.458 n=30)
BuildGRPC/page_cache.dirty/filesystem.bindfs-16   210.0 ± 1%   210.0 ± 1%       ~ (p=0.542 n=30)   209.7 ± 1%       ~ (p=0.665 n=30)
BuildGRPC/page_cache.clean/filesystem.rootfs-16   210.5 ± 1%   210.0 ± 1%       ~ (p=0.423 n=30)   210.0 ± 1%       ~ (p=0.142 n=30)
BuildGRPC/page_cache.dirty/filesystem.rootfs-16   210.2 ± 1%   209.0 ± 1%       ~ (p=0.219 n=30)   209.5 ± 1%       ~ (p=0.230 n=30)
geomean                                           67.62        66.97       -0.96%                  67.12       -0.74%
```

The KVM platform benefits significantly from reduced nested page faults due to
huge pages, and to a lesser extent due to recycling:

```
goos: linux
goarch: amd64
cpu: Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz
                                                │  baseline  │                 expt                  │                 expt2                 │
                                                │   sec/op   │   sec/op    vs base                   │   sec/op    vs base                   │
BuildABSL/page_cache.clean/filesystem.bindfs-12   43.11 ± 2%   39.35 ± 3%   -8.71% (p=0.000 n=20)      38.10 ± 4%  -11.63% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.bindfs-12   42.35 ± 3%   39.09 ± 4%   -7.69% (p=0.000 n=20+19)   39.09 ± 5%   -7.69% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.tmpfs-12    42.35 ± 3%   38.34 ± 5%   -9.46% (p=0.000 n=20)      38.59 ± 3%   -8.87% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.tmpfs-12    42.09 ± 1%   37.59 ± 4%  -10.70% (p=0.000 n=20)      38.09 ± 4%   -9.51% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.rootfs-12   42.85 ± 3%   38.84 ± 3%   -9.35% (p=0.000 n=20)      39.09 ± 3%   -8.77% (p=0.000 n=20+17)
BuildABSL/page_cache.dirty/filesystem.rootfs-12   41.85 ± 2%   39.59 ± 6%   -5.40% (p=0.000 n=20+19)   38.09 ± 3%   -9.00% (p=0.000 n=20+19)
BuildABSL/page_cache.clean/filesystem.fusefs-12   42.60 ± 2%   38.34 ± 2%  -10.00% (p=0.000 n=20)      39.59 ± 3%   -7.06% (p=0.000 n=20+19)
BuildABSL/page_cache.dirty/filesystem.fusefs-12   42.09 ± 4%   39.09 ± 3%   -7.13% (p=0.000 n=20)      38.09 ± 3%   -9.52% (p=0.000 n=20+19)
BuildGRPC/page_cache.clean/filesystem.bindfs-12   207.7 ± 1%   206.4 ± 0%   -0.60% (p=0.018 n=20)      205.9 ± 1%   -0.85% (p=0.001 n=20+19)
BuildGRPC/page_cache.dirty/filesystem.bindfs-12   206.9 ± 1%   206.9 ± 1%        ~ (p=0.121 n=20)      204.4 ± 1%   -1.22% (p=0.004 n=20+19)
BuildGRPC/page_cache.clean/filesystem.rootfs-12   207.7 ± 1%   204.9 ± 1%   -1.33% (p=0.004 n=20)      203.9 ± 0%   -1.81% (p=0.000 n=20+19)
BuildGRPC/page_cache.dirty/filesystem.rootfs-12   206.9 ± 1%   204.9 ± 0%   -0.97% (p=0.004 n=20+19)   203.9 ± 0%   -1.45% (p=0.000 n=20+19)
geomean                                           71.97        67.63        -6.03%                     67.28        -6.52%
```
PiperOrigin-RevId: 647771821

g3doc/user_guide/production.md

@@ -140,6 +140,15 @@ Passthrough to use the host's (Linux's) network stack, rather than gVisor's own.
 <a class="button" href="/docs/user_guide/networking/">Configure Networking
 &raquo;</a>
 
+### Optimizing MM performance {#configure-mm}
+
+gVisor will make transparent huge pages (THP) available to applications if
+provided by the host Linux kernel. Linux disables this feature by default; to
+enable it, write "advise" to the file
+`/sys/kernel/mm/transparent_hugepage/shmem_enabled`. Performance effects of THP
+vary by workload and platform; KVM platform performance in particular can
+benefit greatly from enabling THP.
+
 [Istio]: https://istio.io/
 [Istio overhead]: https://istio.io/latest/docs/ops/deployment/performance-and-scalability/
 [Security Model]: /docs/architecture_guide/security/

nogo.yaml

@@ -190,8 +190,10 @@ analyzers:
         - pkg/gohacks/noescape_unsafe.go                           # Special case.
         - pkg/ring0/pagetables/allocator_unsafe.go                 # Special case.
         - pkg/sentry/fsutil/host_file_mapper_unsafe.go             # Special case.
+        - pkg/sentry/pgalloc/pgalloc_unsafe.go                     # Special case.
         - pkg/sentry/platform/kvm/bluepill_unsafe.go               # Special case.
         - pkg/sentry/platform/kvm/machine_unsafe.go                # Special case.
+        - pkg/sentry/platform/pgalloc/pgalloc_unsafe.go            # Special case.
         - pkg/sentry/platform/systrap/stub_unsafe.go               # Special case.
         - pkg/sentry/platform/systrap/syscall_thread_unsafe.go     # Special case.
         - pkg/sentry/platform/systrap/sysmsg_thread_unsafe.go      # Special case.

pkg/sentry/contexttest/contexttest.go

@@ -44,7 +44,9 @@ func Context(tb testing.TB) context.Context {
 		tb.Fatalf("error creating application memory file: %v", err)
 	}
 	memfile := os.NewFile(uintptr(memfd), memfileName)
-	mf, err := pgalloc.NewMemoryFile(memfile, pgalloc.MemoryFileOpts{})
+	mf, err := pgalloc.NewMemoryFile(memfile, pgalloc.MemoryFileOpts{
+		DisableMemoryAccounting: true,
+	})
 	if err != nil {
 		memfile.Close()
 		tb.Fatalf("error creating pgalloc.MemoryFile: %v", err)

pkg/sentry/fsimpl/gofer/regular_file.go

@@ -374,6 +374,7 @@ func (rw *dentryReadWriter) ReadToBlocks(dsts safemem.BlockSeq) (uint64, error)
 	}
 
 	// Otherwise read from/through the cache.
+	memCgID := pgalloc.MemoryCgroupIDFromContext(rw.ctx)
 	mf := rw.d.fs.mf
 	fillCache := mf.ShouldCacheEvictable()
 	var dataMuUnlock func()
@@ -435,7 +436,11 @@ func (rw *dentryReadWriter) ReadToBlocks(dsts safemem.BlockSeq) (uint64, error)
 					End:   gapEnd,
 				}
 				optMR := gap.Range()
-				_, err := rw.d.cache.Fill(rw.ctx, reqMR, maxFillRange(reqMR, optMR), rw.d.size.Load(), mf, usage.PageCache, pgalloc.AllocateAndWritePopulate, h.readToBlocksAt)
+				_, err := rw.d.cache.Fill(rw.ctx, reqMR, maxFillRange(reqMR, optMR), rw.d.size.Load(), mf, pgalloc.AllocOpts{
+					Kind:    usage.PageCache,
+					MemCgID: memCgID,
+					Mode:    pgalloc.AllocateAndWritePopulate,
+				}, h.readToBlocksAt)
 				mf.MarkEvictable(rw.d, pgalloc.EvictableRange{optMR.Start, optMR.End})
 				seg, gap = rw.d.cache.Find(rw.off)
 				if !seg.Ok() {
@@ -773,6 +778,7 @@ func (d *dentry) Translate(ctx context.Context, required, optional memmap.Mappab
 		}, nil
 	}
 
+	memCgID := pgalloc.MemoryCgroupIDFromContext(ctx)
 	d.dataMu.Lock()
 
 	// Constrain translations to d.size (rounded up) to prevent translation to
@@ -794,7 +800,11 @@ func (d *dentry) Translate(ctx context.Context, required, optional memmap.Mappab
 
 	mf := d.fs.mf
 	h := d.readHandle()
-	_, cerr := d.cache.Fill(ctx, required, maxFillRange(required, optional), d.size.Load(), mf, usage.PageCache, pgalloc.AllocateAndWritePopulate, h.readToBlocksAt)
+	_, cerr := d.cache.Fill(ctx, required, maxFillRange(required, optional), d.size.Load(), mf, pgalloc.AllocOpts{
+		Kind:    usage.PageCache,
+		MemCgID: memCgID,
+		Mode:    pgalloc.AllocateAndWritePopulate,
+	}, h.readToBlocksAt)
 
 	var ts []memmap.Translation
 	var translatedEnd uint64

pkg/sentry/fsimpl/testutil/BUILD

@@ -32,6 +32,7 @@ go_library(
         "//pkg/sentry/platform/ptrace",
         "//pkg/sentry/seccheck",
         "//pkg/sentry/time",
+        "//pkg/sentry/usage",
         "//pkg/sentry/vfs",
         "//pkg/sync",
         "//pkg/usermem",

pkg/sentry/fsimpl/testutil/kernel.go

@@ -36,6 +36,7 @@ import (
 	"gvisor.dev/gvisor/pkg/sentry/platform"
 	"gvisor.dev/gvisor/pkg/sentry/seccheck"
 	"gvisor.dev/gvisor/pkg/sentry/time"
+	"gvisor.dev/gvisor/pkg/sentry/usage"
 	"gvisor.dev/gvisor/pkg/sentry/vfs"
 
 	// Platforms are pluggable.
@@ -53,6 +54,10 @@ func Boot() (*kernel.Kernel, error) {
 	cpuid.Initialize()
 	seccheck.Initialize()
 
+	if err := usage.Init(); err != nil {
+		return nil, fmt.Errorf("setting up memory accounting: %v", err)
+	}
+
 	platformCtr, err := platform.Lookup(*platformFlag)
 	if err != nil {
 		return nil, fmt.Errorf("platform not found: %v", err)

pkg/sentry/fsimpl/tmpfs/regular_file.go

@@ -280,6 +280,8 @@ func (rf *regularFile) CopyMapping(ctx context.Context, ms memmap.MappingSpace,
 
 // Translate implements memmap.Mappable.Translate.
 func (rf *regularFile) Translate(ctx context.Context, required, optional memmap.MappableRange, at hostarch.AccessType) ([]memmap.Translation, error) {
+	memCgID := pgalloc.MemoryCgroupIDFromContext(ctx)
+
 	rf.dataMu.Lock()
 	defer rf.dataMu.Unlock()
 
@@ -308,7 +310,10 @@ func (rf *regularFile) Translate(ctx context.Context, required, optional memmap.
 		}
 		optional = required
 	}
-	pagesAlloced, cerr := rf.data.Fill(ctx, required, optional, rf.size.RacyLoad(), rf.inode.fs.mf, rf.memoryUsageKind, pgalloc.AllocateOnly, nil /* r */)
+	pagesAlloced, cerr := rf.data.Fill(ctx, required, optional, rf.size.RacyLoad(), rf.inode.fs.mf, pgalloc.AllocOpts{
+		Kind:    rf.memoryUsageKind,
+		MemCgID: memCgID,
+	}, nil)
 	// rf.data.Fill() may fail mid-way. We still want to account any pages that
 	// were allocated, irrespective of an error.
 	rf.inode.fs.adjustPageAcct(pagesToFill, pagesAlloced)
@@ -360,6 +365,8 @@ func (fd *regularFileFD) Release(context.Context) {
 // Allocate implements vfs.FileDescriptionImpl.Allocate.
 func (fd *regularFileFD) Allocate(ctx context.Context, mode, offset, length uint64) error {
 	f := fd.inode().impl.(*regularFile)
+	memCgID := pgalloc.MemoryCgroupIDFromContext(ctx)
+
 	// To be consistent with Linux, inode.mu must be locked throughout.
 	f.inode.mu.Lock()
 	defer f.inode.mu.Unlock()
@@ -383,7 +390,7 @@ func (fd *regularFileFD) Allocate(ctx context.Context, mode, offset, length uint
 			newSize = curPgEnd
 		}
 		required := memmap.MappableRange{Start: curPgStart, End: curPgEnd}
-		if err := f.allocateLocked(ctx, mode, newSize, required); err != nil {
+		if err := f.allocateLocked(ctx, mode, newSize, required, memCgID); err != nil {
 			return err
 		}
 		// This loop can take a long time to process, so periodically check for
@@ -401,7 +408,7 @@ func (fd *regularFileFD) Allocate(ctx context.Context, mode, offset, length uint
 // - rf.inode.mu is locked.
 // - required must be page-aligned.
 // - required.Start < newSize <= required.End.
-func (rf *regularFile) allocateLocked(ctx context.Context, mode, newSize uint64, required memmap.MappableRange) error {
+func (rf *regularFile) allocateLocked(ctx context.Context, mode, newSize uint64, required memmap.MappableRange, memCgID uint32) error {
 	rf.dataMu.Lock()
 	defer rf.dataMu.Unlock()
 
@@ -427,7 +434,11 @@ func (rf *regularFile) allocateLocked(ctx context.Context, mode, newSize uint64,
 		// faulting page-by-page when these pages are written to in the future.
 		allocMode = pgalloc.AllocateAndWritePopulate
 	}
-	pagesAlloced, err := rf.data.Fill(ctx, required, required, newSize, rf.inode.fs.mf, rf.memoryUsageKind, allocMode, nil /* r */)
+	pagesAlloced, err := rf.data.Fill(ctx, required, required, newSize, rf.inode.fs.mf, pgalloc.AllocOpts{
+		Kind:    rf.memoryUsageKind,
+		MemCgID: memCgID,
+		Mode:    allocMode,
+	}, nil /* r */)
 	// f.data.Fill() may fail mid-way. We still want to account any pages that
 	// were allocated, irrespective of an error.
 	rf.inode.fs.adjustPageAcct(pagesToFill, pagesAlloced)
@@ -464,6 +475,8 @@ func (fd *regularFileFD) PRead(ctx context.Context, dst usermem.IOSequence, offs
 		return 0, nil
 	}
 	f := fd.inode().impl.(*regularFile)
+	// memCgID can be 0 here because regularFileReadWriter.ReadToBlocks() never
+	// allocates from pgalloc.
 	rw := getRegularFileReadWriter(f, offset, 0)
 	n, err := dst.CopyOutFrom(ctx, rw)
 	putRegularFileReadWriter(rw)
@@ -770,12 +783,12 @@ func (rw *regularFileReadWriter) WriteFromBlocks(srcs safemem.BlockSeq) (uint64,
 				// prepopulating disk-backed pages deteriorates performance as it fails
 				// to eliminate future page faults and we also additionally incur
 				// useless disk writebacks.
-				allocMode = pgalloc.AllocateOnly
+				allocMode = pgalloc.AllocateCallerIndirectCommit
 			}
 			fr, err := rw.file.inode.fs.mf.Allocate(gapMR.Length(), pgalloc.AllocOpts{
 				Kind:    rw.file.memoryUsageKind,
-				Mode:    allocMode,
 				MemCgID: rw.memCgID,
+				Mode:    allocMode,
 			})
 			if err != nil {
 				retErr = err

pkg/sentry/fsutil/file_range_set.go

@@ -24,7 +24,6 @@ import (
 	"gvisor.dev/gvisor/pkg/safemem"
 	"gvisor.dev/gvisor/pkg/sentry/memmap"
 	"gvisor.dev/gvisor/pkg/sentry/pgalloc"
-	"gvisor.dev/gvisor/pkg/sentry/usage"
 )
 
 // FileRangeSet maps offsets into a memmap.Mappable to offsets into a
@@ -92,14 +91,14 @@ func (s *FileRangeSet) PagesToFill(required, optional memmap.MappableRange) uint
 }
 
 // Fill attempts to ensure that all memmap.Mappable offsets in required are
-// mapped to a memmap.File offset, by allocating from mf with the given
-// memory usage kind and invoking readAt to store data into memory. (If readAt
-// returns a successful partial read, Fill will call it repeatedly until all
-// bytes have been read.) EOF is handled consistently with the requirements of
-// mmap(2): bytes after EOF on the same page are zeroed; pages after EOF are
-// invalid. fileSize is an upper bound on the file's size; bytes after fileSize
-// will be zeroed without calling readAt. populate has the same meaning as the
-// pgalloc.MemoryFile.AllocateAndFill() argument of the same name.
+// mapped to a memmap.File offset, by allocating from mf with the given options
+// and invoking readAt to store data into memory. (If readAt is not nil,
+// opts.ReaderFunc will be overridden. If readAt returns a successful partial
+// read, Fill will call it repeatedly until all bytes have been read.) EOF is
+// handled consistently with the requirements of mmap(2): bytes after EOF on
+// the same page are zeroed; pages after EOF are invalid. fileSize is an upper
+// bound on the file's size; bytes after fileSize will be zeroed without
+// calling readAt.
 //
 // Fill may read offsets outside of required, but will never read offsets
 // outside of optional. It returns a non-nil error if any error occurs, even
@@ -111,10 +110,9 @@ func (s *FileRangeSet) PagesToFill(required, optional memmap.MappableRange) uint
 //   - required.Length() > 0.
 //   - optional.IsSupersetOf(required).
 //   - required and optional must be page-aligned.
-func (s *FileRangeSet) Fill(ctx context.Context, required, optional memmap.MappableRange, fileSize uint64, mf *pgalloc.MemoryFile, kind usage.MemoryKind, allocMode pgalloc.AllocationMode, readAt func(ctx context.Context, dsts safemem.BlockSeq, offset uint64) (uint64, error)) (uint64, error) {
+func (s *FileRangeSet) Fill(ctx context.Context, required, optional memmap.MappableRange, fileSize uint64, mf *pgalloc.MemoryFile, opts pgalloc.AllocOpts, readAt func(ctx context.Context, dsts safemem.BlockSeq, offset uint64) (uint64, error)) (uint64, error) {
 	gap := s.LowerBoundGap(required.Start)
 	var pagesAlloced uint64
-	memCgID := pgalloc.MemoryCgroupIDFromContext(ctx)
 	for gap.Ok() && gap.Start() < required.End {
 		if gap.Range().Length() == 0 {
 			gap = gap.NextGap()
@@ -123,11 +121,6 @@ func (s *FileRangeSet) Fill(ctx context.Context, required, optional memmap.Mappa
 		gr := gap.Range().Intersect(optional)
 
 		// Read data into the gap.
-		opts := pgalloc.AllocOpts{
-			Kind:    kind,
-			Mode:    allocMode,
-			MemCgID: memCgID,
-		}
 		if readAt != nil {
 			opts.ReaderFunc = func(dsts safemem.BlockSeq) (uint64, error) {
 				var done uint64

pkg/sentry/hostmm/hostmm.go

@@ -20,11 +20,31 @@ import (
 	"fmt"
 	"os"
 	"path"
+	"regexp"
 
 	"gvisor.dev/gvisor/pkg/eventfd"
 	"gvisor.dev/gvisor/pkg/log"
 )
 
+// GetTransparentHugepageEnum returns the currently selected option for
+// whichever of
+// /sys/kernel/mm/transparent_hugepage/{enabled,shmem_enabled,defrag} is
+// specified by filename. (Only the basename is required, not the full path.)
+func GetTransparentHugepageEnum(filename string) (string, error) {
+	pathname := path.Join("/sys/kernel/mm/transparent_hugepage/", filename)
+	data, err := os.ReadFile(pathname)
+	if err != nil {
+		return "", err
+	}
+	// In these files, the selected option is highlighted by square brackets.
+	m := regexp.MustCompile(`\[.*\]`).Find(data)
+	if m == nil {
+		return "", fmt.Errorf("failed to parse %s: %q", pathname, data)
+	}
+	// Remove the square brackets.
+	return string(m[1 : len(m)-1]), nil
+}
+
 // NotifyCurrentMemcgPressureCallback requests that f is called whenever the
 // calling process' memory cgroup indicates memory pressure of the given level,
 // as specified by Linux's Documentation/cgroup-v1/memory.txt.

pkg/sentry/kernel/syscalls.go

@@ -496,7 +496,10 @@ type SyscallInfo struct {
 // IncrementUnimplementedSyscallCounter increments the "unimplemented syscall" metric for the given
 // syscall number.
 // A syscall table must have been initialized prior to calling this function.
-// +checkescape:all
+//
+// FIXME(gvisor.dev/issue/10556): checkescape can't distinguish between this
+// file and files named syscalls.go in other directories, resulting in false
+// positives, so this function cannot be +checkescape:all.
 //
 //go:nosplit
 func IncrementUnimplementedSyscallCounter(sysno uintptr) {

pkg/sentry/memmap/memmap.go

@@ -356,6 +356,10 @@ type MMapOpts struct {
 	// downward on guard page faults.
 	GrowsDown bool
 
+	// Stack is equivalent to MAP_STACK, which has no mandatory semantics in
+	// Linux.
+	Stack bool
+
 	PlatformEffect MMapPlatformEffect
 
 	// MLockMode specifies the memory locking behavior of the mapping.

pkg/sentry/mm/io.go

@@ -499,7 +499,7 @@ func (mm *MemoryManager) handleASIOFault(ctx context.Context, addr hostarch.Addr
 
 	// Ensure that we have usable pmas.
 	mm.activeMu.Lock()
-	pseg, pend, err := mm.getPMAsLocked(ctx, vseg, ar, at)
+	pseg, pend, err := mm.getPMAsLocked(ctx, vseg, ar, at, true /* callerIndirectCommit */)
 	mm.mappingMu.RUnlock()
 	if pendaddr := pend.Start(); pendaddr < ar.End {
 		if pendaddr <= ar.Start {
@@ -553,7 +553,7 @@ func (mm *MemoryManager) withInternalMappings(ctx context.Context, ar hostarch.A
 
 	// Ensure that we have usable pmas.
 	mm.activeMu.Lock()
-	pseg, pend, perr := mm.getPMAsLocked(ctx, vseg, ar, at)
+	pseg, pend, perr := mm.getPMAsLocked(ctx, vseg, ar, at, true /* callerIndirectCommit */)
 	mm.mappingMu.RUnlock()
 	if pendaddr := pend.Start(); pendaddr < ar.End {
 		if pendaddr <= ar.Start {
@@ -627,7 +627,7 @@ func (mm *MemoryManager) withVecInternalMappings(ctx context.Context, ars hostar
 
 	// Ensure that we have usable pmas.
 	mm.activeMu.Lock()
-	pars, perr := mm.getVecPMAsLocked(ctx, vars, at)
+	pars, perr := mm.getVecPMAsLocked(ctx, vars, at, true /* callerIndirectCommit */)
 	mm.mappingMu.RUnlock()
 	if pars.NumBytes() == 0 {
 		mm.activeMu.Unlock()

pkg/sentry/mm/mm.go

@@ -294,6 +294,9 @@ type vma struct {
 	// metag, none of which we currently support.
 	growsDown bool `state:"manual"`
 
+	// isStack is true if this is a MAP_STACK mapping.
+	isStack bool `state:"manual"`
+
 	// dontfork is the MADV_DONTFORK setting for this vma configured by madvise().
 	dontfork bool
 
@@ -330,6 +333,7 @@ func (v *vma) copy() vma {
 		maxPerms:       v.maxPerms,
 		private:        v.private,
 		growsDown:      v.growsDown,
+		isStack:        v.isStack,
 		dontfork:       v.dontfork,
 		mlockMode:      v.mlockMode,
 		numaPolicy:     v.numaPolicy,
@@ -380,6 +384,13 @@ type pma struct {
 	// corresponding vma's memmap.Mappable.Translate.
 	private bool
 
+	// If huge is true, this pma was returned by a call to MemoryFile.Allocate()
+	// with AllocOpts.Hugepage = true. Note that due to pma splitting, pma may
+	// no longer be hugepage-aligned.
+	//
+	// Invariant: If huge == true, then private == true.
+	huge bool
+
 	// If internalMappings is not empty, it is the cached return value of
 	// file.MapInternal for the memmap.FileRange mapped by this pma.
 	internalMappings safemem.BlockSeq `state:"nosave"`

pkg/sentry/mm/pma.go

@@ -94,6 +94,10 @@ func (mm *MemoryManager) existingVecPMAsLocked(ars hostarch.AddrRangeSeq, at hos
 //
 //   - An error that is non-nil if pmas exist for only a subset of ar.
 //
+// If callerIndirectCommit is true, the caller of getPMAsLocked will shortly
+// commit all pages in ar without using the caller's page tables, in the same
+// sense as pgalloc.AllocateCallerIndirectCommit.
+//
 // Preconditions:
 //   - mm.mappingMu must be locked.
 //   - mm.activeMu must be locked for writing.
@@ -101,7 +105,7 @@ func (mm *MemoryManager) existingVecPMAsLocked(ars hostarch.AddrRangeSeq, at hos
 //   - vseg.Range().Contains(ar.Start).
 //   - vmas must exist for all addresses in ar, and support accesses of type at
 //     (i.e. permission checks must have been performed against vmas).
-func (mm *MemoryManager) getPMAsLocked(ctx context.Context, vseg vmaIterator, ar hostarch.AddrRange, at hostarch.AccessType) (pmaIterator, pmaGapIterator, error) {
+func (mm *MemoryManager) getPMAsLocked(ctx context.Context, vseg vmaIterator, ar hostarch.AddrRange, at hostarch.AccessType, callerIndirectCommit bool) (pmaIterator, pmaGapIterator, error) {
 	if checkInvariants {
 		if !ar.WellFormed() || ar.Length() == 0 {
 			panic(fmt.Sprintf("invalid ar: %v", ar))
@@ -123,7 +127,7 @@ func (mm *MemoryManager) getPMAsLocked(ctx context.Context, vseg vmaIterator, ar
 	}
 	ar = hostarch.AddrRange{ar.Start.RoundDown(), end}
 
-	pstart, pend, perr := mm.getPMAsInternalLocked(ctx, vseg, ar, at)
+	pstart, pend, perr := mm.getPMAsInternalLocked(ctx, vseg, ar, at, callerIndirectCommit)
 	if pend.Start() <= ar.Start {
 		return pmaIterator{}, pend, perr
 	}
@@ -148,7 +152,7 @@ func (mm *MemoryManager) getPMAsLocked(ctx context.Context, vseg vmaIterator, ar
 //   - mm.activeMu must be locked for writing.
 //   - vmas must exist for all addresses in ars, and support accesses of type at
 //     (i.e. permission checks must have been performed against vmas).
-func (mm *MemoryManager) getVecPMAsLocked(ctx context.Context, ars hostarch.AddrRangeSeq, at hostarch.AccessType) (hostarch.AddrRangeSeq, error) {
+func (mm *MemoryManager) getVecPMAsLocked(ctx context.Context, ars hostarch.AddrRangeSeq, at hostarch.AccessType, callerIndirectCommit bool) (hostarch.AddrRangeSeq, error) {
 	for arsit := ars; !arsit.IsEmpty(); arsit = arsit.Tail() {
 		ar := arsit.Head()
 		if ar.Length() == 0 {
@@ -169,7 +173,7 @@ func (mm *MemoryManager) getVecPMAsLocked(ctx context.Context, ars hostarch.Addr
 		}
 		ar = hostarch.AddrRange{ar.Start.RoundDown(), end}
 
-		_, pend, perr := mm.getPMAsInternalLocked(ctx, mm.vmas.FindSegment(ar.Start), ar, at)
+		_, pend, perr := mm.getPMAsInternalLocked(ctx, mm.vmas.FindSegment(ar.Start), ar, at, callerIndirectCommit)
 		if perr != nil {
 			return truncatedAddrRangeSeq(ars, arsit, pend.Start()), perr
 		}
@@ -193,7 +197,7 @@ func (mm *MemoryManager) getVecPMAsLocked(ctx context.Context, ars hostarch.Addr
 //   - getPMAsInternalLocked additionally requires that ar is page-aligned.
 //     getPMAsInternalLocked is an implementation helper for getPMAsLocked and
 //     getVecPMAsLocked; other clients should call one of those instead.
-func (mm *MemoryManager) getPMAsInternalLocked(ctx context.Context, vseg vmaIterator, ar hostarch.AddrRange, at hostarch.AccessType) (pmaIterator, pmaGapIterator, error) {
+func (mm *MemoryManager) getPMAsInternalLocked(ctx context.Context, vseg vmaIterator, ar hostarch.AddrRange, at hostarch.AccessType, callerIndirectCommit bool) (pmaIterator, pmaGapIterator, error) {
 	if checkInvariants {
 		if !ar.WellFormed() || ar.Length() == 0 || !ar.IsPageAligned() {
 			panic(fmt.Sprintf("invalid ar: %v", ar))
@@ -214,18 +218,18 @@ func (mm *MemoryManager) getPMAsInternalLocked(ctx context.Context, vseg vmaIter
 		mm.unmapASLocked(unmapAR)
 	}()
 
-	memCgID := pgalloc.MemoryCgroupIDFromContext(ctx)
-	opts := pgalloc.AllocOpts{Kind: usage.Anonymous, Dir: pgalloc.BottomUp, MemCgID: memCgID}
 	vma := vseg.ValuePtr()
+	memCgID := pgalloc.MemoryCgroupIDFromContext(ctx)
+	allocDir := pgalloc.BottomUp
 	if uintptr(ar.Start) < atomic.LoadUintptr(&vma.lastFault) {
 		// Detect cases where memory is accessed downwards and change memory file
 		// allocation order to increase the chances that pages are coalesced.
-		opts.Dir = pgalloc.TopDown
+		allocDir = pgalloc.TopDown
 	}
 	atomic.StoreUintptr(&vma.lastFault, uintptr(ar.Start))
 
-	// Limit the range we allocate to ar, aligned to privateAllocUnit.
-	maskAR := privateAligned(ar)
+	// Limit the range we allocate to ar, aligned to hugepage boundaries.
+	hugeMaskAR := hugepageAligned(ar)
 	// The range in which we iterate vmas and pmas is still limited to ar, to
 	// ensure that we don't allocate or COW-break a pma we don't need.
 	pseg, pgap := mm.pmas.Find(ar.Start)
@@ -247,8 +251,32 @@ func (mm *MemoryManager) getPMAsInternalLocked(ctx context.Context, vseg vmaIter
 				}
 				if vma.mappable == nil {
 					// Private anonymous mappings get pmas by allocating.
-					allocAR := optAR.Intersect(maskAR)
-					fr, err := mm.mf.Allocate(uint64(allocAR.Length()), opts)
+					// The allocated range is limited to ar, expanded to
+					// hugepage alignment. This is done even if the allocation
+					// will not be hugepage-backed, in an attempt to reduce
+					// application page faults (that trap into the sentry) by
+					// creating AddressSpace mappings in advance.
+					allocAR := optAR.Intersect(hugeMaskAR)
+					// Don't back stacks with huge pages due to low utilization
+					// and because they're often fragmented by copy-on-write.
+					huge := mm.mf.HugepagesEnabled() && allocAR.IsHugePageAligned() && !vma.growsDown && !vma.isStack
+					allocOpts := pgalloc.AllocOpts{
+						Kind:    usage.Anonymous,
+						MemCgID: memCgID,
+						Mode:    pgalloc.AllocateUncommitted,
+						Huge:    huge,
+						Dir:     allocDir,
+					}
+					// If the allocation is hugepage-backed and
+					// callerIndirectCommit is true, the caller will commit every
+					// allocated huge page. If the allocation is not
+					// hugepage-backed, the caller won't commit every allocated
+					// page since hugeMaskAR is ar expanded to huge alignment,
+					// unless only one page in optAR falls into the huge page.
+					if callerIndirectCommit && (huge || allocAR.Length() == hostarch.PageSize) {
+						allocOpts.Mode = pgalloc.AllocateCallerIndirectCommit
+					}
+					fr, err := mm.mf.Allocate(uint64(allocAR.Length()), allocOpts)
 					if err != nil {
 						return pstart, pgap, err
 					}
@@ -268,6 +296,7 @@ func (mm *MemoryManager) getPMAsInternalLocked(ctx context.Context, vseg vmaIter
 						// only reference, the new pma does not need
 						// copy-on-write.
 						private: true,
+						huge:    huge,
 					}).NextNonEmpty()
 					pstart = pmaIterator{} // iterators invalidated
 				} else {
@@ -341,7 +370,7 @@ func (mm *MemoryManager) getPMAsInternalLocked(ctx context.Context, vseg vmaIter
 						}
 					}
 					var copyAR hostarch.AddrRange
-					if vma := vseg.ValuePtr(); vma.effectivePerms.Execute {
+					if vma.effectivePerms.Execute {
 						// The majority of copy-on-write breaks on executable
 						// pages come from:
 						//
@@ -355,7 +384,7 @@ func (mm *MemoryManager) getPMAsInternalLocked(ctx context.Context, vseg vmaIter
 						// to benefit from copying nearby pages, so if the vma
 						// is executable, only copy the pages required.
 						copyAR = pseg.Range().Intersect(ar)
-					} else if vma.growsDown {
+					} else if vma.growsDown || vma.isStack {
 						// In most cases, the new process will not use most of
 						// its stack before exiting or invoking execve(); it is
 						// especially unlikely to return very far down its call
@@ -372,18 +401,23 @@ func (mm *MemoryManager) getPMAsInternalLocked(ctx context.Context, vseg vmaIter
 						}
 						copyAR = pseg.Range().Intersect(stackMaskAR)
 					} else {
-						copyAR = pseg.Range().Intersect(maskAR)
+						// Hugepage-align the range to be copied, for the same
+						// reasons as for private anonymous allocations.
+						copyAR = pseg.Range().Intersect(hugeMaskAR)
 					}
 					// Get internal mappings from the pma to copy from.
 					if err := pseg.getInternalMappingsLocked(); err != nil {
 						return pstart, pseg.PrevGap(), err
 					}
 					// Copy contents.
+					huge := mm.mf.HugepagesEnabled() && copyAR.IsHugePageAligned()
 					reader := safemem.BlockSeqReader{Blocks: mm.internalMappingsLocked(pseg, copyAR)}
 					fr, err := mm.mf.Allocate(uint64(copyAR.Length()), pgalloc.AllocOpts{
 						Kind:       usage.Anonymous,
-						Mode:       pgalloc.AllocateAndWritePopulate,
 						MemCgID:    memCgID,
+						Mode:       pgalloc.AllocateAndWritePopulate,
+						Huge:       huge,
+						Dir:        allocDir,
 						ReaderFunc: reader.ReadToBlocks,
 					})
 					if _, ok := err.(safecopy.BusError); ok {
@@ -413,6 +447,7 @@ func (mm *MemoryManager) getPMAsInternalLocked(ctx context.Context, vseg vmaIter
 					oldpma.maxPerms = vma.maxPerms
 					oldpma.needCOW = false
 					oldpma.private = true
+					oldpma.huge = huge
 					oldpma.internalMappings = safemem.BlockSeq{}
 					// Try to merge the pma with its neighbors.
 					if prev := pseg.PrevSegment(); prev.Ok() {
@@ -518,21 +553,9 @@ func (mm *MemoryManager) getPMAsInternalLocked(ctx context.Context, vseg vmaIter
 	}
 }
 
-const (
-	// When memory is allocated for a private pma, align the allocated address
-	// range to a privateAllocUnit boundary when possible. Larger values of
-	// privateAllocUnit may reduce page faults by allowing fewer, larger pmas
-	// to be mapped, but may result in larger amounts of wasted memory in the
-	// presence of fragmentation. privateAllocUnit must be a power-of-2
-	// multiple of hostarch.PageSize.
-	privateAllocUnit = hostarch.HugePageSize
-
-	privateAllocMask = privateAllocUnit - 1
-)
-
-func privateAligned(ar hostarch.AddrRange) hostarch.AddrRange {
-	aligned := hostarch.AddrRange{ar.Start &^ privateAllocMask, ar.End}
-	if end := (ar.End + privateAllocMask) &^ privateAllocMask; end >= ar.End {
+func hugepageAligned(ar hostarch.AddrRange) hostarch.AddrRange {
+	aligned := hostarch.AddrRange{ar.Start.HugeRoundDown(), ar.End}
+	if end, ok := ar.End.HugeRoundUp(); ok {
 		aligned.End = end
 	}
 	if checkInvariants {
@@ -684,7 +707,7 @@ func (mm *MemoryManager) Pin(ctx context.Context, ar hostarch.AddrRange, at host
 
 	// Ensure that we have usable pmas.
 	mm.activeMu.Lock()
-	pseg, pend, perr := mm.getPMAsLocked(ctx, vseg, ar, at)
+	pseg, pend, perr := mm.getPMAsLocked(ctx, vseg, ar, at, false /* callerIndirectCommit */)
 	mm.mappingMu.RUnlock()
 	if pendaddr := pend.Start(); pendaddr < ar.End {
 		if pendaddr <= ar.Start {
@@ -900,7 +923,8 @@ func (pmaSetFunctions) Merge(ar1 hostarch.AddrRange, pma1 pma, ar2 hostarch.Addr
 		pma1.effectivePerms != pma2.effectivePerms ||
 		pma1.maxPerms != pma2.maxPerms ||
 		pma1.needCOW != pma2.needCOW ||
-		pma1.private != pma2.private {
+		pma1.private != pma2.private ||
+		pma1.huge != pma2.huge {
 		return pma{}, false
 	}
 

pkg/sentry/mm/save_restore.go

@@ -60,6 +60,7 @@ const (
 	vmaMaxPermsExecute
 	vmaPrivate
 	vmaGrowsDown
+	vmaIsStack
 )
 
 func (v *vma) saveRealPerms() int {
@@ -97,6 +98,9 @@ func (v *vma) saveRealPerms() int {
 	if v.growsDown {
 		b |= vmaGrowsDown
 	}
+	if v.isStack {
+		b |= vmaIsStack
+	}
 	return b
 }
 
@@ -134,6 +138,9 @@ func (v *vma) loadRealPerms(_ goContext.Context, b int) {
 	if b&vmaGrowsDown > 0 {
 		v.growsDown = true
 	}
+	if b&vmaIsStack > 0 {
+		v.isStack = true
+	}
 }
 
 func (p *pma) saveFile() string {

pkg/sentry/mm/syscalls.go

@@ -53,7 +53,7 @@ func (mm *MemoryManager) HandleUserFault(ctx context.Context, addr hostarch.Addr
 
 	// Ensure that we have a usable pma.
 	mm.activeMu.Lock()
-	pseg, _, err := mm.getPMAsLocked(ctx, vseg, ar, at)
+	pseg, _, err := mm.getPMAsLocked(ctx, vseg, ar, at, true /* callerIndirectCommit */)
 	mm.mappingMu.RUnlock()
 	if err != nil {
 		mm.activeMu.Unlock()
@@ -135,7 +135,7 @@ func (mm *MemoryManager) MMap(ctx context.Context, opts memmap.MMapOpts) (hostar
 		// Get pmas and map as requested.
 		mm.populateVMAAndUnlock(ctx, vseg, ar, opts.PlatformEffect)
 
-	case opts.Mappable == nil && length <= privateAllocUnit:
+	case opts.Mappable == nil && length <= hostarch.HugePageSize:
 		// NOTE(b/63077076, b/63360184): Get pmas and map eagerly in the hope
 		// that doing so will save on future page faults. We only do this for
 		// anonymous mappings, since otherwise the cost of
@@ -179,7 +179,7 @@ func (mm *MemoryManager) populateVMA(ctx context.Context, vseg vmaIterator, ar h
 	}
 
 	// Ensure that we have usable pmas.
-	pseg, _, err := mm.getPMAsLocked(ctx, vseg, ar, hostarch.NoAccess)
+	pseg, _, err := mm.getPMAsLocked(ctx, vseg, ar, hostarch.NoAccess, platformEffect == memmap.PlatformEffectCommit)
 	if err != nil {
 		// mm/util.c:vm_mmap_pgoff() ignores the error, if any, from
 		// mm/gup.c:mm_populate(). If it matters, we'll get it again when
@@ -226,7 +226,7 @@ func (mm *MemoryManager) populateVMAAndUnlock(ctx context.Context, vseg vmaItera
 	// mm.mappingMu doesn't need to be write-locked for getPMAsLocked, and it
 	// isn't needed at all for mapASLocked.
 	mm.mappingMu.DowngradeLock()
-	pseg, _, err := mm.getPMAsLocked(ctx, vseg, ar, hostarch.NoAccess)
+	pseg, _, err := mm.getPMAsLocked(ctx, vseg, ar, hostarch.NoAccess, platformEffect == memmap.PlatformEffectCommit)
 	mm.mappingMu.RUnlock()
 	if err != nil {
 		mm.activeMu.Unlock()
@@ -449,6 +449,7 @@ func (mm *MemoryManager) MRemap(ctx context.Context, oldAddr hostarch.Addr, oldS
 			MaxPerms:        vma.maxPerms,
 			Private:         vma.private,
 			GrowsDown:       vma.growsDown,
+			Stack:           vma.isStack,
 			MLockMode:       vma.mlockMode,
 			Hint:            vma.hint,
 		}, droppedIDs)
@@ -890,7 +891,7 @@ func (mm *MemoryManager) MLock(ctx context.Context, addr hostarch.Addr, length u
 				mm.mappingMu.RUnlock()
 				return linuxerr.ENOMEM
 			}
-			_, _, err := mm.getPMAsLocked(ctx, vseg, vseg.Range().Intersect(ar), hostarch.NoAccess)
+			_, _, err := mm.getPMAsLocked(ctx, vseg, vseg.Range().Intersect(ar), hostarch.NoAccess, true /* callerIndirectCommit */)
 			if err != nil {
 				mm.activeMu.Unlock()
 				mm.mappingMu.RUnlock()
@@ -985,7 +986,7 @@ func (mm *MemoryManager) MLockAll(ctx context.Context, opts MLockAllOpts) error
 		mm.mappingMu.DowngradeLock()
 		for vseg := mm.vmas.FirstSegment(); vseg.Ok(); vseg = vseg.NextSegment() {
 			if vseg.ValuePtr().effectivePerms.Any() {
-				mm.getPMAsLocked(ctx, vseg, vseg.Range(), hostarch.NoAccess)
+				mm.getPMAsLocked(ctx, vseg, vseg.Range(), hostarch.NoAccess, true /* callerIndirectCommit */)
 			}
 		}
 
@@ -1096,11 +1097,24 @@ func (mm *MemoryManager) Decommit(addr hostarch.Addr, length uint64) error {
 	defer mm.activeMu.Unlock()
 
 	// This is invalidateLocked(invalidatePrivate=true, invalidateShared=true),
-	// with the additional wrinkle that we must refuse to invalidate pmas under
-	// mlocked vmas.
-	var didUnmapAS bool
+	// but:
+	//
+	//	- We must refuse to invalidate pmas under mlocked vmas.
+	//
+	//	- If at least one byte in ar is not covered by a vma, decommit the rest
+	//	but return ENOMEM.
+	//
+	//	- If we would invalidate only part of a huge page that we own (is not
+	//	copy-on-write), use MemoryFile.Decommit() instead to keep the allocated
+	//	huge page intact for future use.
+	didUnmapAS := false
 	pseg := mm.pmas.LowerBoundSegment(ar.Start)
-	for vseg := mm.vmas.LowerBoundSegment(ar.Start); vseg.Ok() && vseg.Start() < ar.End; vseg = vseg.NextSegment() {
+	vseg := mm.vmas.LowerBoundSegment(ar.Start)
+	if !vseg.Ok() {
+		return linuxerr.ENOMEM
+	}
+	hadvgap := ar.Start < vseg.Start()
+	for vseg.Ok() && vseg.Start() < ar.End {
 		vma := vseg.ValuePtr()
 		if vma.mlockMode != memmap.MLockNone {
 			return linuxerr.EINVAL
@@ -1114,8 +1128,62 @@ func (mm *MemoryManager) Decommit(addr hostarch.Addr, length uint64) error {
 			}
 		}
 		for pseg.Ok() && pseg.Start() < vsegAR.End {
-			pseg = mm.pmas.Isolate(pseg, vsegAR)
 			pma := pseg.ValuePtr()
+			if pma.huge && !mm.isPMACopyOnWriteLocked(vseg, pseg) {
+				psegAR := pseg.Range().Intersect(vsegAR)
+				if !psegAR.IsHugePageAligned() {
+					firstHugeStart := psegAR.Start.HugeRoundDown()
+					firstHugeEnd := firstHugeStart + hostarch.HugePageSize
+					lastWholeHugeEnd := psegAR.End.HugeRoundDown()
+					if firstHugeStart != psegAR.Start {
+						// psegAR.Start is not hugepage-aligned.
+						if psegAR.End <= firstHugeEnd {
+							// All of psegAR falls within a single huge page.
+							mm.mf.Decommit(pseg.fileRangeOf(psegAR))
+							pseg = pseg.NextSegment()
+							continue
+						}
+						if firstHugeEnd == lastWholeHugeEnd && lastWholeHugeEnd != psegAR.End {
+							// All of psegAR falls within two huge pages, and
+							// psegAR.End is also not hugepage-aligned. The
+							// logic below would handle this correctly, but
+							// would make two separate calls to
+							// MemoryFile.Decommit() for the first and last
+							// huge pages respectively.
+							mm.mf.Decommit(pseg.fileRangeOf(psegAR))
+							pseg = pseg.NextSegment()
+							continue
+						}
+						mm.mf.Decommit(pseg.fileRangeOf(hostarch.AddrRange{psegAR.Start, firstHugeEnd}))
+						psegAR.Start = firstHugeEnd
+					}
+					// Drop whole huge pages between psegAR.Start (which after the above
+					// is either firstHugeStart or firstHugeEnd) and lastWholeHugeEnd
+					// normally.
+					if psegAR.Start < lastWholeHugeEnd {
+						pseg = mm.pmas.Isolate(pseg, hostarch.AddrRange{psegAR.Start, lastWholeHugeEnd})
+						pma = pseg.ValuePtr()
+						if !didUnmapAS {
+							// Unmap all of ar, not just pseg.Range(), to minimize host
+							// syscalls. AddressSpace mappings must be removed before
+							// pma.file.DecRef().
+							mm.unmapASLocked(ar)
+							didUnmapAS = true
+						}
+						pma.file.DecRef(pseg.fileRange())
+						mm.removeRSSLocked(pseg.Range())
+						pseg = mm.pmas.Remove(pseg).NextSegment()
+					}
+					if lastWholeHugeEnd != psegAR.End {
+						// psegAR.End is not hugepage-aligned.
+						mm.mf.Decommit(pseg.fileRangeOf(hostarch.AddrRange{lastWholeHugeEnd, psegAR.End}))
+						pseg = pseg.NextSegment()
+					}
+					continue
+				}
+			}
+			pseg = mm.pmas.Isolate(pseg, vsegAR)
+			pma = pseg.ValuePtr()
 			if !didUnmapAS {
 				// Unmap all of ar, not just pseg.Range(), to minimize host
 				// syscalls. AddressSpace mappings must be removed before
@@ -1127,13 +1195,21 @@ func (mm *MemoryManager) Decommit(addr hostarch.Addr, length uint64) error {
 			mm.removeRSSLocked(pseg.Range())
 			pseg = mm.pmas.Remove(pseg).NextSegment()
 		}
+		if ar.End <= vseg.End() {
+			break
+		}
+		vgap := vseg.NextGap()
+		if !vgap.IsEmpty() {
+			hadvgap = true
+		}
+		vseg = vgap.NextSegment()
 	}
 
 	// "If there are some parts of the specified address space that are not
 	// mapped, the Linux version of madvise() ignores them and applies the call
 	// to the rest (but returns ENOMEM from the system call, as it should)." -
 	// madvise(2)
-	if mm.vmas.SpanRange(ar) != ar.Length() {
+	if hadvgap {
 		return linuxerr.ENOMEM
 	}
 	return nil

pkg/sentry/mm/vma.go

@@ -44,10 +44,13 @@ func (mm *MemoryManager) createVMALocked(ctx context.Context, opts memmap.MMapOp
 
 	// Find a usable range.
 	addr, err := mm.findAvailableLocked(opts.Length, findAvailableOpts{
-		Addr:     opts.Addr,
-		Fixed:    opts.Fixed,
-		Unmap:    opts.Unmap,
-		Map32Bit: opts.Map32Bit,
+		Addr:      opts.Addr,
+		Fixed:     opts.Fixed,
+		GrowsDown: opts.GrowsDown,
+		Stack:     opts.Stack,
+		Private:   opts.Private,
+		Unmap:     opts.Unmap,
+		Map32Bit:  opts.Map32Bit,
 	})
 	if err != nil {
 		// Can't force without opts.Unmap and opts.Fixed.
@@ -119,6 +122,7 @@ func (mm *MemoryManager) createVMALocked(ctx context.Context, opts memmap.MMapOp
 		maxPerms:       opts.MaxPerms,
 		private:        opts.Private,
 		growsDown:      opts.GrowsDown,
+		isStack:        opts.Stack,
 		mlockMode:      opts.MLockMode,
 		numaPolicy:     linux.MPOL_DEFAULT,
 		id:             opts.MappingIdentity,
@@ -144,10 +148,13 @@ type findAvailableOpts struct {
 	//
 	//	- Unmap allows existing guard pages in the returned range.
 
-	Addr     hostarch.Addr
-	Fixed    bool
-	Unmap    bool
-	Map32Bit bool
+	Addr      hostarch.Addr
+	Fixed     bool
+	GrowsDown bool
+	Stack     bool
+	Private   bool
+	Unmap     bool
+	Map32Bit  bool
 }
 
 // map32Start/End are the bounds to which MAP_32BIT mappings are constrained,
@@ -187,9 +194,10 @@ func (mm *MemoryManager) findAvailableLocked(length uint64, opts findAvailableOp
 		return 0, linuxerr.ENOMEM
 	}
 
-	// Prefer hugepage alignment if a hugepage or more is requested.
+	// Prefer hugepage alignment if a hugepage or more is requested and the vma
+	// will actually be eligible for hugepages.
 	alignment := uint64(hostarch.PageSize)
-	if length >= hostarch.HugePageSize {
+	if length >= hostarch.HugePageSize && opts.Private && !opts.GrowsDown && !opts.Stack {
 		alignment = hostarch.HugePageSize
 	}
 
@@ -465,6 +473,7 @@ func (vmaSetFunctions) Merge(ar1 hostarch.AddrRange, vma1 vma, ar2 hostarch.Addr
 		vma1.maxPerms != vma2.maxPerms ||
 		vma1.private != vma2.private ||
 		vma1.growsDown != vma2.growsDown ||
+		vma1.isStack != vma2.isStack ||
 		vma1.mlockMode != vma2.mlockMode ||
 		vma1.numaPolicy != vma2.numaPolicy ||
 		vma1.numaNodemask != vma2.numaNodemask ||

pkg/sentry/pgalloc/BUILD

@@ -7,13 +7,6 @@ package(
     licenses = ["notice"],
 )
 
-declare_mutex(
-    name = "memory_file_mutex",
-    out = "memory_file_mutex.go",
-    package = "pgalloc",
-    prefix = "memoryFile",
-)
-
 declare_mutex(
     name = "mappings_mutex",
     out = "mappings_mutex.go",
@@ -21,6 +14,13 @@ declare_mutex(
     prefix = "mappings",
 )
 
+declare_mutex(
+    name = "memory_file_mutex",
+    out = "memory_file_mutex.go",
+    package = "pgalloc",
+    prefix = "memoryFile",
+)
+
 go_template_instance(
     name = "evictable_range",
     out = "evictable_range.go",
@@ -47,8 +47,8 @@ go_template_instance(
 )
 
 go_template_instance(
-    name = "usage_set",
-    out = "usage_set.go",
+    name = "memacct_set",
+    out = "memacct_set.go",
     consts = {
         "minDegree": "10",
         "trackGaps": "1",
@@ -57,33 +57,55 @@ go_template_instance(
         "memmap": "gvisor.dev/gvisor/pkg/sentry/memmap",
     },
     package = "pgalloc",
-    prefix = "usage",
+    prefix = "memAcct",
     template = "//pkg/segment:generic_set",
     types = {
         "Key": "uint64",
         "Range": "memmap.FileRange",
-        "Value": "usageInfo",
-        "Functions": "usageSetFunctions",
+        "Value": "memAcctInfo",
+        "Functions": "memAcctSetFunctions",
     },
 )
 
 go_template_instance(
-    name = "reclaim_set",
-    out = "reclaim_set.go",
+    name = "unfree_set",
+    out = "unfree_set.go",
     consts = {
         "minDegree": "10",
+        "trackGaps": "1",
     },
     imports = {
         "memmap": "gvisor.dev/gvisor/pkg/sentry/memmap",
     },
     package = "pgalloc",
-    prefix = "reclaim",
+    prefix = "unfree",
     template = "//pkg/segment:generic_set",
     types = {
         "Key": "uint64",
         "Range": "memmap.FileRange",
-        "Value": "reclaimSetValue",
-        "Functions": "reclaimSetFunctions",
+        "Value": "unfreeInfo",
+        "Functions": "unfreeSetFunctions",
+    },
+)
+
+go_template_instance(
+    name = "unwaste_set",
+    out = "unwaste_set.go",
+    consts = {
+        "minDegree": "10",
+        "trackGaps": "1",
+    },
+    imports = {
+        "memmap": "gvisor.dev/gvisor/pkg/sentry/memmap",
+    },
+    package = "pgalloc",
+    prefix = "unwaste",
+    template = "//pkg/segment:generic_set",
+    types = {
+        "Key": "uint64",
+        "Range": "memmap.FileRange",
+        "Value": "unwasteInfo",
+        "Functions": "unwasteSetFunctions",
     },
 )
 
@@ -94,12 +116,13 @@ go_library(
         "evictable_range.go",
         "evictable_range_set.go",
         "mappings_mutex.go",
+        "memacct_set.go",
         "memory_file_mutex.go",
         "pgalloc.go",
         "pgalloc_unsafe.go",
-        "reclaim_set.go",
         "save_restore.go",
-        "usage_set.go",
+        "unfree_set.go",
+        "unwaste_set.go",
     ],
     visibility = ["//pkg/sentry:internal"],
     deps = [
@@ -109,7 +132,6 @@ go_library(
         "//pkg/errors/linuxerr",
         "//pkg/hostarch",
         "//pkg/log",
-        "//pkg/memutil",
         "//pkg/safemem",
         "//pkg/sentry/arch",
         "//pkg/sentry/hostmm",
@@ -130,5 +152,8 @@ go_test(
     size = "small",
     srcs = ["pgalloc_test.go"],
     library = ":pgalloc",
-    deps = ["//pkg/hostarch"],
+    deps = [
+        "//pkg/hostarch",
+        "//pkg/sentry/memmap",
+    ],
 )