Validate cached per-CPU slab regions with an in-region marker. - #718
Draft
copybara-service[bot] wants to merge 1 commit into
Draft
Validate cached per-CPU slab regions with an in-region marker.#718copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
copybara-service
Bot
force-pushed
the
test_979603858
branch
4 times, most recently
from
September 12, 2026 04:03
19c8938 to
3e6c527
Compare
The allocation fast paths cached the current CPU's slab region address in
tcmalloc_slabs, whose top 4 bytes overlapped __rseq_abi.cpu_id_start so that
the kernel invalidated the cache on migration. Linux's rewritten rseq
implementation writes cpu_id_start only when it changes, and terminates
processes that write it themselves, so the overlap is neither sufficient nor
permitted.
Validate the cached address against the region it names instead. Size class 0
is unused, so its 32 bit header holds a marker for the whole region:
* 0: the region is unpopulated. This is the value left behind by mmap() and
MADV_DONTNEED.
* TCMALLOC_SLAB_STOPPED: a remote operation (resize/drain/grow/shrink) owns
the region.
* cpu + TCMALLOC_SLAB_CPU_BIAS: the region is the current, running region of
cpu.
The fast paths load the marker inside the restartable sequence and compare it
against the current CPU id, read as a 16 bit value from __rseq_abi at
__rseq_virtual_flat_cpu_id_offset (cpu_id, vcpu_id, or mm_cid). The single
comparison establishes that the thread runs on the CPU that owns the region,
that no remote operation owns it, and that it belongs to the current slabs
generation. Reading the CPU id as 16 bits keeps a running marker below
TCMALLOC_SLAB_STOPPED (0xffffffff), so the two can never alias.
Validity is now a property of the region rather than of the thread, so a
cached address is never invalidated remotely: it simply stops matching.
Retired slab buffers must therefore stay mapped and readable for the lifetime
of the process, and StopAllCpus()/StartAllCpus() replace the fence-based
invalidation in ResizeSlabs() and UpdateMaxCapacities().
Because the marker lives in the slabs themselves, a slabs buffer must arrive
zero-filled. Memory freshly obtained from the kernel already is; a test
allocator built on operator new has to zero it explicitly.
A thread with no region cached names tcmalloc_dummy_slab, a read-only region
whose marker is 0, rather than nothing at all. The comparison above therefore
subsumes the question of whether a region is cached, and the fast paths need
no separate test for a null address.
The marker only changes when the thread migrates or a remote operation claims
the region. It does not change when the thread is merely preempted and
rescheduled onto the same cpu, which the old protocol detected because the
kernel rewrote cpu_id_start on every schedule. Grow() reads the size class
header outside of the critical section, so StoreCurrentCpu() now also compares
the header against the value Grow() read, making the update atomic with respect
to Push/Pop. Without it, a Push that lands between Grow()'s load and its store
is reverted and its object is lost.
x86 loads are implicitly acquire. aarch64 fast paths use a relaxed ldr, so
StartCpu()/StartAllCpus() fence before publishing a running marker.
The registered __rseq_abi is still 32 bytes; adopting the larger struct is
gated separately.
PiperOrigin-RevId: 979603858
copybara-service
Bot
force-pushed
the
test_979603858
branch
from
September 12, 2026 04:44
3e6c527 to
1a78a27
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Validate cached per-CPU slab regions with an in-region marker.
The allocation fast paths cached the current CPU's slab region address in
tcmalloc_slabs, whose top 4 bytes overlapped __rseq_abi.cpu_id_start so that
the kernel invalidated the cache on migration. Linux's rewritten rseq
implementation writes cpu_id_start only when it changes, and terminates
processes that write it themselves, so the overlap is neither sufficient nor
permitted.
Validate the cached address against the region it names instead. Size class 0
is unused, so its 32 bit header holds a marker for the whole region:
MADV_DONTNEED.
the region.
cpu.
The fast paths load the marker inside the restartable sequence and compare it
against the current CPU id, read as a 16 bit value from __rseq_abi at
__rseq_virtual_flat_cpu_id_offset (cpu_id, vcpu_id, or mm_cid). The single
comparison establishes that the thread runs on the CPU that owns the region,
that no remote operation owns it, and that it belongs to the current slabs
generation. Reading the CPU id as 16 bits keeps a running marker below
TCMALLOC_SLAB_STOPPED (0xffffffff), so the two can never alias.
Validity is now a property of the region rather than of the thread, so a
cached address is never invalidated remotely: it simply stops matching.
Retired slab buffers must therefore stay mapped and readable for the lifetime
of the process, and StopAllCpus()/StartAllCpus() replace the fence-based
invalidation in ResizeSlabs() and UpdateMaxCapacities().
Because the marker lives in the slabs themselves, a slabs buffer must arrive
zero-filled. Memory freshly obtained from the kernel already is; a test
allocator built on operator new has to zero it explicitly.
A thread with no region cached names tcmalloc_dummy_slab, a read-only region
whose marker is 0, rather than nothing at all. The comparison above therefore
subsumes the question of whether a region is cached, and the fast paths need
no separate test for a null address.
The marker only changes when the thread migrates or a remote operation claims
the region. It does not change when the thread is merely preempted and
rescheduled onto the same cpu, which the old protocol detected because the
kernel rewrote cpu_id_start on every schedule. Grow() reads the size class
header outside of the critical section, so StoreCurrentCpu() now also compares
the header against the value Grow() read, making the update atomic with respect
to Push/Pop. Without it, a Push that lands between Grow()'s load and its store
is reverted and its object is lost.
x86 loads are implicitly acquire. aarch64 fast paths use a relaxed ldr, so
StartCpu()/StartAllCpus() fence before publishing a running marker.
The registered __rseq_abi is still 32 bytes; adopting the larger struct is
gated separately.