[Feature](lance) Support global BM25 statistics for distributed FTS

[zhangstar333]

Parent

• Tracked by [Feature](lance) Add radius and full-text search modes #66493
• Part of [Tracking] Lance feature matrix for Doris 4.2 #66340
• Doris integration draft: [feature](lance) support full_text_search in lance catalog #67289
• Lance tracking issue: feature: expose global BM25 statistics for distributed FTS executors lance-format/lance#8937

Motivation

Doris executes Lance full-text search in a distributed architecture. The FE plans one query and assigns different Lance fragments or FTS index segments to multiple BEs.

If each BE builds BM25 statistics from only its local segments, its document count, total token count, average document length, and query-term document frequencies can differ from other workers. Those worker-local scores are not comparable, and merging local Top-K results cannot recover the ranking produced by one search over the complete index.

The FE Java layer therefore needs to collect query-bound, corpus-wide BM25 statistics for one pinned Lance dataset snapshot. The FE stores the resulting opaque payload in TFullTextSearchParams.global_statistics. That field is delivered directly to every BE through the existing TLanceScanParams.external_search_request Thrift scan plan; this design does not introduce a separate broadcast or coordination channel.

flowchart TB
    subgraph FE["Doris FE - Java query planning"]
        Q["FTS query"] --> P["Pin Lance dataset version<br/>resolve complete FTS segment set"]
        P --> J["Lance Java API<br/>collect global BM25 statistics"]
        J --> T["Set TFullTextSearchParams.global_statistics"]
        M["Merge comparable BE Top-K results"]
    end

    subgraph LANCE["One logical Lance FTS index"]
        S1["FTS segment 1"]
        S2["FTS segment 2"]
        S3["FTS segment 3"]
    end

    T --> THRIFT["Existing Thrift scan-plan delivery"]

    subgraph BES["Doris BE cluster"]
        BE1["BE 1<br/>consume global statistics<br/>search assigned segments"]
        BE2["BE 2<br/>consume global statistics<br/>search assigned segments"]
        BE3["BE 3<br/>consume global statistics<br/>search assigned segments"]
    end

    S1 --> J
    S2 --> J
    S3 --> J
    THRIFT --> BE1
    THRIFT --> BE2
    THRIFT --> BE3
    BE1 --> M
    BE2 --> M
    BE3 --> M

Loading

V1 scope

The first version deliberately represents one BM25 corpus:

• one dataset version;
• one logical FTS index, indexed column, and document granularity;
• the exact committed segment UUID set for that corpus;
• corpus document count, total token count, and document frequency for every prepared term;
• the prepared vocabulary and original token positions for every scoring leaf, including fuzzy expansion.

Every scoring leaf must resolve to the same corpus. Cross-column or otherwise cross-corpus queries must be rejected.

The protobuf is an opaque payload within a trusted planning flow. The FE must attach it only to the same query used to produce it. Consumers validate the schema and dataset/index metadata, but V1 does not attempt to reconstruct and independently prove the original query identity.

Existing Doris integration point

PR #67289 reserves TFullTextSearchParams.global_statistics as an optional opaque binary field. The FE creates the ScanNode-level external_search_request, and the BE receives the same request from TFileScanRangeParams.lance_scan_params.

The BE currently returns an unsupported error when global_statistics is present because the lance-c consumer API is not available yet.

Upstream Lance work

• feat(fts): define global BM25 statistics wire contract lance-format/lance#8938 defines the versioned protobuf wire contract.
• feat(java): expose global BM25 statistics for FTS queries lance-format/lance#8822 implements global-statistics collection in Rust and exposes it through JNI and the Java API as an opaque protobuf payload. It depends on feat(fts): define global BM25 statistics wire contract lance-format/lance#8938.

These PRs provide the FE-side producer. A follow-up Rust/lance-c consumer is still required before the Doris BE can apply the payload to its assigned segment subset.

Remaining work

1. Merge the Lance protobuf and producer PRs.
2. Add the Rust/lance-c consumer that validates the payload and installs the prepared BM25 context before FTS execution.
3. Make the Doris FE populate TFullTextSearchParams.global_statistics through the Java API.
4. Make each Doris BE consume the field through lance-c for its assigned segments.
5. Define the behavior for unindexed fragments and mixed indexed/unindexed execution.

Completion criteria

• Distributed FTS ranking matches a single full-index execution over the same dataset version and committed segment set.
• Regression tests cover multiple segments where worker-local and global BM25 statistics produce different rankings.
• Fuzzy queries replay the same globally prepared vocabulary and token positions.
• Unsupported cross-corpus queries fail with a clear error.
• EXPLAIN or runtime profiles show whether FE-provided global BM25 statistics are used.