[Bug] BE crash (ASAN heap-use-after-free) in SharedMemtable dtor during group commit flush

[bosswnx]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master. Reproduced on two independent runs with bases 5fbe3600024 (PR #67404) and 952bfcbb40fa (PR #67402), both containing #66899 (merged Sep 1 15:33) and #66191 (merged Sep 1 18:24). The use-after-free itself was introduced by e925e6ae1b2 (#66889, merged Aug 27).

What's Wrong?

Two unrelated test-only PRs (#67404, #67402 — neither touches BE code) both hit the same BE crash in the nonConcurrent regression pipeline: an ASAN heap-use-after-free that kills the BE process, after which ~49 suites fail in cascade ("replication num ... available backend num is 0") and the whole pipeline goes FAILURE.

Crash from PR #67402's build (1036154) — identical stack in PR #67404's build (1035988):

==48480==ERROR: AddressSanitizer: heap-use-after-free on address 0x7d3e73b61f30
READ of size 8 at 0x7d3e73b61f30 thread T1919 (mf_normal [work)
    #0 std::__shared_ptr<doris::segment_v2::SegmentAllocatedLsnMap>::operator bool() const
    #2 doris::RowsetWriterContext::remove_segment_allocated_lsns(long)
       be/src/storage/rowset/rowset_writer_context.h:202:9
    #3 doris::SharedMemtable::~SharedMemtable()
       be/src/load/memtable/memtable_flush_executor.cpp:124:21
    #7 doris::PartOfGroupMemtableFlushTask::~PartOfGroupMemtableFlushTask()
       be/src/load/memtable/memtable_flush_executor.cpp:82:7
   #13 doris::ThreadPool::dispatch_thread() be/src/util/threadpool.cpp:628:23

freed by thread T1919 (mf_normal [work) here:
    #4 doris::FlushToken::~FlushToken() be/src/load/memtable/memtable_flush_executor.h:92:7
    #8 doris::PartOfGroupMemtableFlushTask::run()
       be/src/load/memtable/memtable_flush_executor.cpp:100:5

previously allocated by thread T640 (brpc_heavy) here:
    #1 doris::RowsetFactory::create_empty_group_rowset_writer(...) be/src/storage/rowset/rowset_factory.cpp:100
    #2 doris::GroupRowsetBuilder::init() be/src/storage/rowset_builder.cpp:570:5
    #4 doris::DeltaWriter::write(...) be/src/load/delta_writer/delta_writer.cpp:177:9
   #15 doris::PInternalService::tablet_writer_add_block ... (stream load / group commit ingest path)

SUMMARY: AddressSanitizer: heap-use-after-free
    be/src/storage/rowset/rowset_writer_context.h:202:9 in
    doris::RowsetWriterContext::remove_segment_allocated_lsns(long)

Mechanism. At the end of PartOfGroupMemtableFlushTask::run() (memtable_flush_executor.cpp:100), the local shared_ptr<FlushToken> is destroyed. When it holds the last reference, destruction cascades FlushToken → _rowset_writer → RowsetWriter → RowsetWriterContext. Afterwards the thread pool (dispatch_thread, threadpool.cpp:628) destroys the task object itself; ~PartOfGroupMemtableFlushTask → ~SharedMemtable (memtable_flush_executor.cpp:124) then calls remove_segment_allocated_lsns() through the raw pointer SharedMemtable::rowset_ctx (memtable_flush_executor.h:67), which already dangles. The raw rowset_ctx and has_allocated_lsns were added by #66889 (git log -S "has_allocated_lsns" → e925e6ae1b2).

Timeline (all times UTC+8, from commit statuses of recent nonConcurrent runs):

Time	Event
Aug 27 12:07	#66889 merged (UAF code enters master)
Aug 31 – Sep 1 16:25	all nonConcurrent runs pass (e.g. #67344, #67341 — bases without #66899/#66191)
Sep 1 15:33	#66899 merged (row binlog flexible partial updates; also adds 2 binlog regression suites)
Sep 1 18:24	#66191 merged (binlog tablets colocated — rewrites GroupRowsetBuilder::init in rowset_builder.cpp)
Sep 1 19:09	5fbe3600024 = base of #67404
Sep 1 19:56	952bfcbb40fa = base of #67402
Sep 2 03:59 / 12:50	#67404 (build 1035988) and #67402 (build 1036154) both crash with the identical UAF

So the UAF lay dormant for ~5 days and became reproducible once #66191/#66899 landed (they change object lifetimes / widen coverage on the group-commit + row-binlog write path). Any PR based on master ≥ Sep 1 evening currently fails nonConcurrent.

What You Expected?

The nonConcurrent regression should pass on master. A group-commit memtable flush task must not access a RowsetWriterContext after the FlushToken's last reference is released.

How to Reproduce?

1. Create a PR based on master ≥ 5fbe3600024 (any change; the two hits were test-only PRs).
2. Comment run buildall to trigger the nonConcurrent pipeline.
3. During the first group-commit / stream-load ingest suites, BE exits with the ASAN heap-use-after-free above; subsequent suites fail with "available backend num is 0".

Reference builds: #67404 → TeamCity build 1035988; #67402 → TeamCity build 1036154.

Anything Else?

Suggested fix directions (either one):

1. SharedMemtable holds a std::shared_ptr<RowsetWriter> (or directly the std::shared_ptr<SegmentAllocatedLsnMap>) instead of the raw RowsetWriterContext*, so the LSN map outlives the flush task.
2. Reorder destruction inside PartOfGroupMemtableFlushTask::run() so _shared_memtable is released before the local FlushToken shared_ptr goes out of scope (e.g. an explicit scope block).

CC the authors of the related PRs: #66889 (introduced the raw pointer + LSN bookkeeping), #66191 / #66899 (landed in the window where this became reproducible).

Are you willing to submit PR?

• Yes I am willing to submit a PR!

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_64930d36f4b6

Initial triage

Verdict: confirmed code-level lifetime bug (high confidence), with master-blocking CI impact. The issue currently has no labels. Suggested project labels are kind/fix, bug/asan, kind/stability, area/load, and area/storage.

Verified facts

• In base 952bfcbb40fa756cb66a7d8e70d559496eb344c8, SharedMemtable owns a raw RowsetWriterContext* plus has_allocated_lsns. Its destructor dereferences that pointer to remove the segment's LSN entry.
• Commit e925e6ae1b2 introduced that raw pointer, the LSN insertion in _memtable2block(), and the destructor cleanup.
• RowsetWriter embeds RowsetWriterContext by value, so destroying the last writer owner destroys the context. The GroupRowsetWriter context creates the SegmentAllocatedLsnMap as a shared_ptr and shares that same map with the data and row-binlog writer contexts.
• PartOfGroupMemtableFlushTask owns SharedMemtable but only a weak_ptr<FlushToken>. run() temporarily locks the token; after run() returns, ThreadPool::dispatch_thread() explicitly resets the task at line 628. Therefore the local token can release the last writer/context owner before task destruction releases the last SharedMemtable owner.
• The reported ASAN free stack at the end of PartOfGroupMemtableFlushTask::run() and invalid read from SharedMemtable::~SharedMemtable() exactly matches this ownership order. This is not merely inferred from the later cascade failures.
• The relevant memtable-flush lifetime code is unchanged between the two reported bases (5fbe3600024... and its child 952bfcbb40fa...). No fix PR is linked to this issue at the time of this triage.

Important attribution boundary

The root cause is the non-owning rowset_ctx added by #66889. The timeline shows that #66191/#66899 made the failure observable, but it does not yet prove that either changed this lifetime. #66899 adds row-binlog regression coverage and #66191 changes GroupRowsetBuilder::init()/transaction attachment; neither changes the raw-pointer cleanup. Treat them as coverage/timing exposure candidates unless a focused rerun or commit bisection establishes a stronger causal link.

Recommended fix

Prefer making SharedMemtable own the precise cleanup dependency:

1. Replace RowsetWriterContext* rowset_ctx with std::shared_ptr<segment_v2::SegmentAllocatedLsnMap> captured from group_rowset_writer->context().allocated_lsn_map at submission.
2. Insert and remove entries directly through that map; retain an insertion flag (or equivalent invariant) so cleanup runs only when an entry was created.
3. Use a non-null map to represent the existing need_allocated_lsn() condition, since GroupRowsetWriter::init() creates the map exactly when LSN allocation is required.

This is narrower than retaining the whole RowsetWriter, preserves the intended shared-map design, and uses the map's existing mutex. Retaining std::shared_ptr<RowsetWriter> would also make the dereference safe, but extends a much larger object's lifetime.

I would not use destruction reordering inside run() as the sole fix. It addresses the captured last-local-token ordering, but the task intentionally keeps only a weak token; a queued sibling task can observe an expired token. Cleanup performed by SharedMemtable should therefore own the state it dereferences rather than depend on a token still being lockable.

Validation requested for the fix

• Add a deterministic ASAN/BE unit test covering both (a) the last external token/writer owner being released while a group flush task finishes and (b) a queued group subtask running after its weak token has expired.
• Assert that the per-segment LSN entry is removed on success, cancellation, and flush-error paths without retaining stale entries.
• Run the relevant memtable-flush/group-rowset-writer BE tests under ASAN, the row-binlog/group-commit regression suites, and the full nonConcurrent pipeline.

Missing evidence

No additional information is required to establish the UAF root cause. For a deterministic regression test and for attributing the exposure window, please attach the exact first failing suite/load statement, the complete ASAN allocation/free stacks (not only selected frames), and the BE log around that load's cancellation/close/destruction. A focused rerun immediately before and after #66899 and #66191 would distinguish added coverage from a timing/lifetime change.

[bosswnx]

Thanks for the thorough triage — the ownership analysis matches ours, and the attribution boundary correction is accepted: I verified that neither #66191 nor #66899 touches the flush-task lifetime code (delta_writer / memtable_writer / memtable_flush_executor / the rowset_builder ownership paths — #66191 only adds tablet-state checks and a migration lock in GroupRowsetBuilder::init; #66899 only adds tests and a segment-flusher format change). The UAF is timing-sensitive and probabilistic; the new suites merely widen the exposure.

Per the "Missing evidence" request, the complete three ASAN stacks from build 1036154 (PR #67402, base 952bfcbb40fa) are attached below — full frames, not selected ones.

READ of size 8 (thread T1919, mf_normal):

==48480==ERROR: AddressSanitizer: heap-use-after-free on address 0x7d3e73b61f30
READ of size 8 at 0x7d3e73b61f30 thread T1919 (mf_normal [work)
    #0 std::__shared_ptr<doris::segment_v2::SegmentAllocatedLsnMap>::operator bool() const shared_ptr_base.h:1677:16
    #2 doris::RowsetWriterContext::remove_segment_allocated_lsns(long) be/src/storage/rowset/rowset_writer_context.h:202:9
    #3 doris::SharedMemtable::~SharedMemtable() be/src/load/memtable/memtable_flush_executor.cpp:124:21
    #7 doris::PartOfGroupMemtableFlushTask::~PartOfGroupMemtableFlushTask() be/src/load/memtable/memtable_flush_executor.cpp:82:7
   #13 doris::ThreadPool::dispatch_thread() be/src/util/threadpool.cpp:628:23

Freed by thread T1919 (end of run()):

    #4 doris::FlushToken::~FlushToken() be/src/load/memtable/memtable_flush_executor.h:92:7
    #8 doris::PartOfGroupMemtableFlushTask::run() be/src/load/memtable/memtable_flush_executor.cpp:100:5

Allocated by thread T640 (brpc_heavy, stream-load ingest path):

    #1 doris::RowsetFactory::create_empty_group_rowset_writer(...) be/src/storage/rowset/rowset_factory.cpp:100
    #2 doris::GroupRowsetBuilder::init() be/src/storage/rowset_builder.cpp:570:5
    #4 doris::DeltaWriter::write(...) be/src/load/delta_writer/delta_writer.cpp:177:9
   #15 doris::PInternalService::tablet_writer_add_block ... internal_service.cpp:526

On the suggested focused rerun around #66899/#66191: agreed it would cleanly separate "coverage" from "timing change", but since neither PR modified the lifetime code and the fix does not depend on that attribution, I'll skip the bisection unless a maintainer wants it.

A fix is ready following the recommended approach: SharedMemtable owns std::shared_ptr<SegmentAllocatedLsnMap> captured at submission, insert/remove through the map directly, non-null map standing in for need_allocated_lsn(), plus ASAN unit tests covering (a) the last token/writer owner released while a group flush task finishes, (b) a queued subtask running after its weak token expired, and (c) cancellation, with LSN-entry cleanup asserted on the success and flush-error paths. Verified in both directions under ASAN: without the fix the new tests abort with this exact use-after-free; with the fix the memtable-flush-executor tests pass 8/8. PR to follow shortly.