Skip to content

Validate cached per-CPU slab regions with an in-region marker. - #718

Draft
copybara-service[bot] wants to merge 1 commit into
masterfrom
test_979603858
Draft

Validate cached per-CPU slab regions with an in-region marker.#718
copybara-service[bot] wants to merge 1 commit into
masterfrom
test_979603858

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Sep 11, 2026

Copy link
Copy Markdown

Validate cached per-CPU slab regions with an in-region marker.

The allocation fast paths cached the current CPU's slab region address in
tcmalloc_slabs, whose top 4 bytes overlapped __rseq_abi.cpu_id_start so that
the kernel invalidated the cache on migration. Linux's rewritten rseq
implementation writes cpu_id_start only when it changes, and terminates
processes that write it themselves, so the overlap is neither sufficient nor
permitted.

Validate the cached address against the region it names instead. Size class 0
is unused, so its 32 bit header holds a marker for the whole region:

  • 0: the region is unpopulated. This is the value left behind by mmap() and
    MADV_DONTNEED.
  • TCMALLOC_SLAB_STOPPED: a remote operation (resize/drain/grow/shrink) owns
    the region.
  • cpu + TCMALLOC_SLAB_CPU_BIAS: the region is the current, running region of
    cpu.

The fast paths load the marker inside the restartable sequence and compare it
against the current CPU id, read as a 16 bit value from __rseq_abi at
__rseq_virtual_flat_cpu_id_offset (cpu_id, vcpu_id, or mm_cid). The single
comparison establishes that the thread runs on the CPU that owns the region,
that no remote operation owns it, and that it belongs to the current slabs
generation. Reading the CPU id as 16 bits keeps a running marker below
TCMALLOC_SLAB_STOPPED (0xffffffff), so the two can never alias.

Validity is now a property of the region rather than of the thread, so a
cached address is never invalidated remotely: it simply stops matching.
Retired slab buffers must therefore stay mapped and readable for the lifetime
of the process, and StopAllCpus()/StartAllCpus() replace the fence-based
invalidation in ResizeSlabs() and UpdateMaxCapacities().

Because the marker lives in the slabs themselves, a slabs buffer must arrive
zero-filled. Memory freshly obtained from the kernel already is; a test
allocator built on operator new has to zero it explicitly.

A thread with no region cached names tcmalloc_dummy_slab, a read-only region
whose marker is 0, rather than nothing at all. The comparison above therefore
subsumes the question of whether a region is cached, and the fast paths need
no separate test for a null address.

The marker only changes when the thread migrates or a remote operation claims
the region. It does not change when the thread is merely preempted and
rescheduled onto the same cpu, which the old protocol detected because the
kernel rewrote cpu_id_start on every schedule. Grow() reads the size class
header outside of the critical section, so StoreCurrentCpu() now also compares
the header against the value Grow() read, making the update atomic with respect
to Push/Pop. Without it, a Push that lands between Grow()'s load and its store
is reverted and its object is lost.

x86 loads are implicitly acquire. aarch64 fast paths use a relaxed ldr, so
StartCpu()/StartAllCpus() fence before publishing a running marker.

The registered __rseq_abi is still 32 bytes; adopting the larger struct is
gated separately.

@copybara-service
copybara-service Bot force-pushed the test_979603858 branch 4 times, most recently from 19c8938 to 3e6c527 Compare September 12, 2026 04:03
The allocation fast paths cached the current CPU's slab region address in
tcmalloc_slabs, whose top 4 bytes overlapped __rseq_abi.cpu_id_start so that
the kernel invalidated the cache on migration.  Linux's rewritten rseq
implementation writes cpu_id_start only when it changes, and terminates
processes that write it themselves, so the overlap is neither sufficient nor
permitted.

Validate the cached address against the region it names instead.  Size class 0
is unused, so its 32 bit header holds a marker for the whole region:

*   0: the region is unpopulated.  This is the value left behind by mmap() and
    MADV_DONTNEED.
*   TCMALLOC_SLAB_STOPPED: a remote operation (resize/drain/grow/shrink) owns
    the region.
*   cpu + TCMALLOC_SLAB_CPU_BIAS: the region is the current, running region of
    cpu.

The fast paths load the marker inside the restartable sequence and compare it
against the current CPU id, read as a 16 bit value from __rseq_abi at
__rseq_virtual_flat_cpu_id_offset (cpu_id, vcpu_id, or mm_cid).  The single
comparison establishes that the thread runs on the CPU that owns the region,
that no remote operation owns it, and that it belongs to the current slabs
generation.  Reading the CPU id as 16 bits keeps a running marker below
TCMALLOC_SLAB_STOPPED (0xffffffff), so the two can never alias.

Validity is now a property of the region rather than of the thread, so a
cached address is never invalidated remotely: it simply stops matching.
Retired slab buffers must therefore stay mapped and readable for the lifetime
of the process, and StopAllCpus()/StartAllCpus() replace the fence-based
invalidation in ResizeSlabs() and UpdateMaxCapacities().

Because the marker lives in the slabs themselves, a slabs buffer must arrive
zero-filled.  Memory freshly obtained from the kernel already is; a test
allocator built on operator new has to zero it explicitly.

A thread with no region cached names tcmalloc_dummy_slab, a read-only region
whose marker is 0, rather than nothing at all.  The comparison above therefore
subsumes the question of whether a region is cached, and the fast paths need
no separate test for a null address.

The marker only changes when the thread migrates or a remote operation claims
the region.  It does not change when the thread is merely preempted and
rescheduled onto the same cpu, which the old protocol detected because the
kernel rewrote cpu_id_start on every schedule.  Grow() reads the size class
header outside of the critical section, so StoreCurrentCpu() now also compares
the header against the value Grow() read, making the update atomic with respect
to Push/Pop.  Without it, a Push that lands between Grow()'s load and its store
is reverted and its object is lost.

x86 loads are implicitly acquire.  aarch64 fast paths use a relaxed ldr, so
StartCpu()/StartAllCpus() fence before publishing a running marker.

The registered __rseq_abi is still 32 bytes; adopting the larger struct is
gated separately.

PiperOrigin-RevId: 979603858
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant