mmap/munmap EL1 fastpath - #372
Conversation
Make anonymous mappings lazy: sys_mmap records the region while first touch creates page tables, commits host memory, and zeros reused backing. Serve common private anonymous read-write requests from per-vCPU EL1 arenas, with publication rings that let the host reconcile mappings under mmap_lock. Keep shim.S focused on exception entry, register-frame preservation, and HVC dispatch by compiling the EL1 arena consumer as freestanding C. The mmap fast-path allocation policy then has a C home that munmap can share. Materialize untouched guest memory before host access and preserve PROT_NONE reservations. Let partial guest writes materialize lazy destinations, use tracked mremap protections for dirty state, and keep neighboring PTEs intact when mremap grows across block boundaries. Cover lazy reuse, refill, fork, and first-touch behavior. Close sysprog21#165
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
b8c3cd6 to
0c6ee05
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Extend the freestanding C EL1 fast path to retire compatible anonymous mappings, invalidate their translations before return, and defer host metadata cleanup until mmap_lock is next acquired. Return drained arena generations to per-vCPU allocators and refill arenas from recent registration history so mmap and munmap remain effective under reuse and mixed mapping sizes. Preserve the producer window around vCPU kicks and fork, and expose counters for both munmap fallback reasons. Keep dirty backing lazy on every anonymous munmap path; Keep unrelated VM exits out of mmap_lock when no EL1 slot has pending work. Lock-taking paths still drain unconditionally, and revocation skips controls when their shim mapping is unavailable. Prove the guest-influenced arena sizing arithmetic, isolate benchmark samples by process, and cover retirement, reuse, refill, fallback, and cross-vCPU publication behavior. Store the per-thread blocked mask with one atomic release in deliver_signal_locked, signal_deliver_fault, and signal_set_state. signal_pending() and thread_signal_deliverable() read the field lock-free from other vCPU threads, so the plain read-modify-write left those reads racing against a torn store under ThreadSanitizer.
0c6ee05 to
e250001
Compare
There was a problem hiding this comment.
2 issues found across 21 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="tests/test-mmap-fastpath-stats.sh">
<violation number="1" location="tests/test-mmap-fastpath-stats.sh:158">
P3: The diagnostic for the altstack cases reads "only selected altstack faults in OK", which is missing a word and hard to parse. Reword it to state what the pair asserts, e.g. "only the used altstack faults in" or "altstack faults only when selected".</violation>
</file>
<file name="tests/test-mmap-fastpath.c">
<violation number="1" location="tests/test-mmap-fastpath.c:930">
P3: The prefix-hint assertion only rejects exact aliasing (q == p). If the fast path returns q anywhere inside the existing [p, p+65536) mapping, the test passes: q != p, q's pages are never touched, and both munmap(q, 4096) and munmap(p, 65536) succeed on partially-unmapped ranges. Check that q lies entirely outside p's 64 KiB extent instead.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| require_eq unused-altstack FAULT_WINDOW_BYTES 0 | ||
| run_case used-altstack | ||
| require_ge used-altstack FAULT_WINDOW_BYTES 4096 | ||
| printf ' only selected altstack faults in OK\n' |
There was a problem hiding this comment.
P3: The diagnostic for the altstack cases reads "only selected altstack faults in OK", which is missing a word and hard to parse. Reword it to state what the pair asserts, e.g. "only the used altstack faults in" or "altstack faults only when selected".
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At tests/test-mmap-fastpath-stats.sh, line 158:
<comment>The diagnostic for the altstack cases reads "only selected altstack faults in OK", which is missing a word and hard to parse. Reword it to state what the pair asserts, e.g. "only the used altstack faults in" or "altstack faults only when selected".</comment>
<file context>
@@ -147,6 +147,16 @@ require_eq fork-no-topup MMAP_HIT 1
+require_eq unused-altstack FAULT_WINDOW_BYTES 0
+run_case used-altstack
+require_ge used-altstack FAULT_WINDOW_BYTES 4096
+printf ' only selected altstack faults in OK\n'
+
run_case invalid-futex
</file context>
| printf ' only selected altstack faults in OK\n' | |
| printf ' altstack faults only when selected OK\n' |
| return 1; | ||
| void *q = mmap(p, 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, | ||
| -1, 0); | ||
| if (q == MAP_FAILED || q == p) |
There was a problem hiding this comment.
P3: The prefix-hint assertion only rejects exact aliasing (q == p). If the fast path returns q anywhere inside the existing [p, p+65536) mapping, the test passes: q != p, q's pages are never touched, and both munmap(q, 4096) and munmap(p, 65536) succeed on partially-unmapped ranges. Check that q lies entirely outside p's 64 KiB extent instead.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At tests/test-mmap-fastpath.c, line 930:
<comment>The prefix-hint assertion only rejects exact aliasing (q == p). If the fast path returns q anywhere inside the existing [p, p+65536) mapping, the test passes: q != p, q's pages are never touched, and both munmap(q, 4096) and munmap(p, 65536) succeed on partially-unmapped ranges. Check that q lies entirely outside p's 64 KiB extent instead.</comment>
<file context>
@@ -919,6 +919,57 @@ static int stats_fork_no_topup(void)
+ return 1;
+ void *q = mmap(p, 4096, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS,
+ -1, 0);
+ if (q == MAP_FAILED || q == p)
+ return 1;
+ void *r = mmap(NULL, 4096, PROT_READ | PROT_WRITE,
</file context>
| if (q == MAP_FAILED || q == p) | |
| uintptr_t qa = (uintptr_t) q, pa = (uintptr_t) p; | |
| if (q == MAP_FAILED || (qa >= pa && qa < pa + 65536)) | |
| return 1; |
|
|
||
| int rc; | ||
| mmap_lock_acquire((guest_t *) (uintptr_t) cg); | ||
| rc = guest_lazy_faultin_locked(cg, gva, len); |
There was a problem hiding this comment.
gva_resolve_perm now calls this on any walk miss, so every guest_read, guest_write, and guest_ptr can acquire mmap_lock. mmap_lock is a default (non-recursive) pthread mutex, and fuse_dev_read holds session->lock across guest_write(g, buf_gva, req->frame, frame_len) while sys_mmap can hold mmap_lock across fuse_materialize_fd, so a FUSE daemon reading a request into an untouched anonymous buffer deadlocks the VM: the requester holds mmap_lock waiting for the daemon, the daemon needs mmap_lock to deliver the frame.
The per-call-site pre-faulting added for futex, mincore and sysvipc does not scale to every caller of a resolve API this widely used. Consider having gva_lazy_faultin return -1 when the calling thread already holds mmap_lock (mmap_lock_guest already records that), so a missed pre-fault surfaces as EFAULT instead of a hang.
| } | ||
| } | ||
|
|
||
| guest_materialize_wait_range_locked(g, start, end); |
There was a problem hiding this comment.
guest_materialize_wait_range_locked does not exclude the caller's own claim, and this drain is reachable from mmap_lock_reacquire_with_gate while guest_materialize_lazy_one holds a claim over the block it is zeroing. It is safe today only because the retire rings are always empty at that point (the gate stays closed across mmap_lock_drop_keep_gate, so no producer can push), which is an invariant nothing states or checks.
Worth either asserting the rings are empty here or skipping the caller's own claim slot in the wait.
| * unrelated VM exit (often the next mapping's first fault) with an eager | ||
| * memset of the retired range. guest_materialize_lazy_one() zeros dirty | ||
| * backing before publishing any new descriptor, so a future reader can | ||
| * never observe stale bytes. |
There was a problem hiding this comment.
This holds for the fault path but not for mremap. copy_mremap_source's non-overlay branch memmoves raw slab bytes for the source range without materializing it first, and mremap_extend_range then installs valid PTEs over the copy.
Touch a large anonymous mapping, munmap it, mmap the same size again so the allocator reuses that backing, then mremap the fresh mapping without touching it: the destination reads the earlier mapping's bytes rather than zero. sys_mremap needs guest_lazy_faultin_locked over the source range, or an explicit zero of its never-materialized blocks, before any raw copy.
| * retirement entries have committed. | ||
| */ | ||
| if (retired_any) | ||
| guest_pt_gen_bump(g); |
There was a problem hiding this comment.
guest_pt_gen_bump is the only thing that invalidates the per-thread gva_tlb entry in gva_translate_perm, and EL1 clears the descriptors and returns to the guest long before this drain runs. The broadcast TLBI reaches the architectural TLBs only.
A host thread that is not a vCPU thread, so the vm-exit drain in vcpu_run_loop is not on its path, can hold a gva_tlb entry populated before the fast munmap and keep translating the retired range as valid until some other thread happens to take mmap_lock. An epoch that EL1 bumps in shim_data after its descriptor stores, folded into the gva_tlb validity test, would close that window without a host round trip.
| * self-deadlock hazard as the shmat copy-in: pages the guest never | ||
| * touched may still be unmaterialized. | ||
| */ | ||
| guest_lazy_faultin_locked(g, entry.guest_gva, entry.size); |
There was a problem hiding this comment.
The pre-fault is best effort, but guest_read below still faults on a miss, so this can re-enter mmap_lock, which sc_shmdt already holds. shmat, then munmap(shmaddr, size), then shmdt(shmaddr) leaves nothing for guest_lazy_faultin_locked to materialize, the first gva_resolve_perm returns NULL, and gva_lazy_faultin blocks on the non-recursive mmap_lock, hanging every vCPU.
sc_mincore has the right shape at src/syscall/syscall.c:761: pre-fault, then copy with a nofault variant. The guest_write at line 264 is the same shape and can use the existing guest_write_nofault; this one needs a guest_read_nofault added.
|
Lazy anonymous mappings can hand the guest bytes from a previously freed mapping: mremap's relocating path memmoves raw slab bytes from a source whose blocks were never materialized, so the zeroing that munmap defers never runs for them. The detail is on the inline comment at |
2 participants
The primary goal is to reduce VM exits for
mmapandmunmap.Architecture
Anonymous mappings are lazy:
sys_mmaponly records the region; page-table creation, host memory commit, and zeroing of reused backing all happen at first touch.Each vCPU owns a private contiguous virtual address arena. When the guest traps into EL1 for
mmap/munmap, EL1 serves the request directly out of its own arena and never leaves EL1. The host's job is to keep those arenas supplied with VA and to reconcile their effects into host-side region/PTE state, both done lazily undermmap_lockrather than synchronously per call.mmap fast path
EL1 serves exactly one shape:
file-backed mapping is excluded. The reason is that EL1 is a freestanding, no-syscall context that cannot read a file or install a page-cache overlay itself — that work is inherently the host's.
The host prepares a contiguous VA region per vCPU. Each vCPU is the sole producer of its own 32-entry publication ring and bump cursor, and publishes the extent into its ring. The host installs the corresponding PTEs.
Arena refill is dynamic, not fixed-size. Any host path that already holds
mmap_lock— a syscall, a page fault, or a natural VM exit — opportunistically tops up the current thread's arena once its remaining drops under that recent high-water mark. Arena exhaustion, an unsupported shape, a full ring, or a stale arena generation each bail to the ordinary hostmmapslow path.munmap fast path
EL1 validates that the target range belongs to a live arena, clears the covering page-table entries directly, and issues a broadcast TLBI, all without leaving EL1
The address is unusable to the guest the instant EL1 returns. Only after clearing the PTEs and completing the TLBI does EL1 publish a retirement record — address, length, and the arena generation it came from — to a second, separate 32-entry SPSC ring. The host has not been involved yet at that point; its bookkeeping is reconciled lazily by
mmap_fastpath_drain_locked(), which walks every vCPU's retire ring and commits the deferred cleanup. That drain runs on every natural VM exit as well as on every syscall or fault that needs the lock.Cross-thread ordering matters here, and A and B are often touching the same address, not two unrelated ones: EL1's fast
munmapscans every vCPU's arena, not just its own, because handing a fresh allocation to another thread to free is the common case. If the host were to drain B's teardown of address X before it has drained A's publication of that same X, the removal would be a no-op (X isn't in the region table yet) and the publication drain right after would insert X as live — reviving, in host bookkeeping, an address EL1 already invalidated. The drain avoids this by snapshotting every vCPU's retire-ring tail first, draining every mmap publication next, and only then consuming retirements up to that snapshot, so within one drain pass a publication a retirement depends on is always applied before the retirement removes it:EL1 self-throttles rather than pinging the host mid-flight. If a producer's retire ring is within one slot of full,
munmapbails out of the fast path entirely and falls through to the ordinary syscall trap, which drains the ring as a side effect of takingmmap_lock. Separately, once a vCPU's own unconsumed retired bytes cross a 256 MiB soft threshold, it sets an advisorycleanup_requestedflag; this never forces an HVC, it only marks that real cleanup work is waiting for the next natural drain.behavior while fork
Before the fork snapshot is taken, the parent-side handler acquires
mmap_lockthrough the fork variant of the acquire path, drains every vCPU's publication and retire ring, and waits for any in-flight lazy materialization to finish. It then revokes every per-vCPU arena descriptor. Because sibling vCPUs are already quiesced for the fork snapshot, this revocation cannot race a live EL1 producer.The child does not inherit the parent's arena or ring layout. The child's main thread begins with a fresh, empty arena rather than a stale one aliased to the parent's.
Any
clone()call that publishes a stack for the new thread goes through this too. Any anonymous fast-path allocation that has become a live thread stack must remain reachable only through the ordinary stack-lifetime bookkeeping, not through EL1's arena-generation check, so revoking first guarantees no later EL1 fastmunmapcan tear down a live stack range by still recognizing it as belonging to a live arena. Only aclone()with a nullchild_stackskips this.behavior while exec
sys_execve()takesmmap_lockimmediately before the point of no return. That both closes the EL1 producer gate and drains every vCPU's mmap and munmap rings, sog->regions[]and PTE state agree with each other before the address space they describe is destroyed.guest_reset()then zeroes the entire shim-data page, arena, both rings.execdoes not explicitly do arena's control block comes back. The replacement image's first eligible slow-pathmmapby calling the refill path and turns the fast path back.Frama-C proof coverage
The guest-influenced arena sizing and capacity arithmetic is pulled out of
src/syscall/mem.cintosrc/proved/mmap-fastpath.hspecifically so it can be proved.Four functions are proved, every input treated as fully guest-influenced (
request_lenis the guest's own mmap length; the history window is built from a sequence of guest-chosen lengths), because a slip here either wedges the allocator by undersizing an arena forever, or letsmmap_fastpath_request_fitsaccept a request that runs pastarena_limit:mmap_fastpath_request_fits: whether alen-byte request still fits beforelimit, covering the zero-length, sub-block, and block-aligned cases. Proved to never answer "fits" when the aligned start would actually run pastlimit.mmap_fastpath_pow2_clamped: rounds a target size up to the nearest power of two inside[MIN, MAX].mmap_fastpath_window_max: the largest of the last 16 registered mapping sizes. Proved as an upper bound over the whole history window.mmap_fastpath_arena_size: the target arena size given recent history and the request about to be served. Proved to always land in[MIN, MAX], with-wp-rteseparately closing both multiplication-overflow guards and the division-by-zero case.92 of 93 discharged goals (the 93rd,
align_up_ok_ensures_rejects_only_on_wrap, is a pre-existingalign.hgoal that also fails standalone undermake verify-alignin this environment, not something this change introduced).Benchmark results
A. One timed mmap/munmap fast-path pair per fresh process
All values are nanoseconds. Speedup is
OrbStack / elfuse; values above 1mean elfuse is faster.
B. munmap after materializing one 4 KiB page
Fifteen fresh processes are used per size. Values are outer medians of the
per-process medians.
C. munmap after dirtying every 4 KiB page
One fresh process is used per size. Each process performs one warmup, then the
listed number of timed operations. The table reports the in-process
distribution.
munmaplatency has a step right at the 2 MiB boundary. Under 2 MiB, which walks and clears one 4 KiB L3 leaf at a time.Arena size is clamped to
MMAP_FAST_ARENA_MIN = 64 MiBandMMAP_FAST_ARENA_MAX = 32 GiBper vCPU.tests/bench-mmap-isolated allto reproduce mmap/munmap-vs-size numbersCloses #165
Summary by cubic
Serves common anonymous
mmapandmunmapcalls directly from EL1 without trapping to the host, and makes anonymous mappings lazy so page tables, host memory commit, and zeroing happen on first touch. Closes #165.Behavior
mmap(NULL, len, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, ...)up to 32 GiB per request; anything else falls back to the host syscall path.mmap_lock, so most calls never leave EL1.munmapclears the covering PTEs and issues a broadcast TLBI in EL1, then defers host bookkeeping to the next drain.munmapcost is length-independent.execdrains and resets them.Migration
ELFR(wasELFQ) because the wire payload now carries a dirty-blocks snapshot.ELFUSE_MMAP_FASTPATH=0to disable the fast path; verbose tracing, the syscall histogram, GDB, and Rosetta disable it automatically.PROT_NONEreservations still fault, and concurrent first touch from several threads is covered by new tests.Written for commit e250001. Summary will update on new commits.