[Chore](pick) pick #58666 #61503 #60079 #60228 #60735 by BiteTheDDDDt · Pull Request #61503 · apache/doris

BiteTheDDDDt · 2026-03-19T03:21:02Z

…ache#58666) ```sql SELECT avg((ss_quantity * ss_list_price)) average_sales from store_sales; ``` before: partial_avg(CAST((CAST(ss_quantity[apache#23] AS decimalv3(10,0)) * ss_list_price[apache#24]) AS decimalv3(19,4)))[apache#25] doris 28.42 sec doris disable check 25.17 sec after: partial_avg((CAST(ss_quantity[apache#23] AS decimalv3(10,0)) * ss_list_price[apache#24]))[apache#25] doris 19.14 sec doris disable check 17.12 sec

…hObject::serialize (apache#60526) This pull request introduces a new serialization format for `DataTypeFixedLengthObject` columns, leveraging streamvbyte encoding for efficient storage and transmission of large data blocks. The new format is activated for BE exec version 10 and above, which is now set as the maximum supported version. Additionally, the `AggregateFunctionCount` and `AggregateFunctionCountNotNullUnary` functions are marked as trivial, likely for optimization purposes. Below are the most important changes: * Introduced a new serialization/deserialization format for `DataTypeFixedLengthObject` that uses streamvbyte encoding for large data, improving efficiency for big data columns. The new logic is gated behind BE exec version 10 and includes fallback to the previous format for older versions (`be/src/vec/data_types/data_type_fixed_length_object.cpp`). [[1]](diffhunk://#diff-7d29ab3e43d23db58f2216e23cc131705067e133fb7ab2da72f2e67c725beb48L36-L41) [[2]](diffhunk://#diff-7d29ab3e43d23db58f2216e23cc131705067e133fb7ab2da72f2e67c725beb48L56-R130) [[3]](diffhunk://#diff-7d29ab3e43d23db58f2216e23cc131705067e133fb7ab2da72f2e67c725beb48L84-R144) [[4]](diffhunk://#diff-7d29ab3e43d23db58f2216e23cc131705067e133fb7ab2da72f2e67c725beb48R156-R172) * Updated the calculation of uncompressed serialized bytes to account for the new serialization format and potential streamvbyte compression (`be/src/vec/data_types/data_type_fixed_length_object.cpp`). * Added the `streamvbyte` library include to support the new encoding/decoding logic (`be/src/vec/data_types/data_type_fixed_length_object.cpp`). * Increased `BeExecVersionManager::max_be_exec_version` from 8 to 10, with detailed documentation and warnings about the sensitivity of this field. The new version enables the updated serialization logic (`be/src/agent/be_exec_version_manager.cpp`). * Defined a new constant `USE_NEW_FIXED_OBJECT_SERIALIZATION_VERSION = 10` to clearly mark the threshold for the new serialization format (`be/src/agent/be_exec_version_manager.h`). * Marked `AggregateFunctionCount` and `AggregateFunctionCountNotNullUnary` as trivial by overriding the `is_trivial()` method to return `true`, which may allow for performance optimizations in the aggregation engine (`be/src/vec/aggregate_functions/aggregate_function_count.h`). [[1]](diffhunk://#diff-a5dbb09237f197bffdcbd3bec4fdd089913ec143d96806618c8eeb4c5dbb8cfeR64-R65) [[2]](diffhunk://#diff-a5dbb09237f197bffdcbd3bec4fdd089913ec143d96806618c8eeb4c5dbb8cfeR212-R213)

tpch external 187s -> 180s tpcds external 380s -> 370s This pull request introduces a mechanism to estimate and utilize the execution cost of conjunct (predicate) expressions, enabling the system to dynamically reorder conjuncts for improved query performance when the new session variable `enable_adjust_conjunct_order_by_cost` is enabled. The changes span the addition of cost estimation methods to expression classes, integration of cost-based sorting in several operator initialization and runtime paths, and the introduction of the new session variable in both backend and frontend.

sum before: STREAMING_AGGREGATION avg 4sec58ms, max 4sec481ms, min 3sec766ms AGGREGATION_SINK avg 11sec598ms, max 11sec733ms, min 11sec477ms after: STREAMING_AGGREGATION avg 2sec400ms, max 2sec660ms, min 2sec101ms AGGREGATION_SINK avg 11sec582ms, max 11sec722ms, min 11sec382ms avg before: STREAMING_AGGREGATION avg 5sec37ms, max 5sec447ms, min 4sec591ms AGGREGATION_SINK avg 14sec298ms, max 14sec427ms, min 14sec180ms after: STREAMING_AGGREGATION avg 3sec43ms, max 3sec415ms, min 2sec838ms AGGREGATION_SINK avg 14sec67ms, max 14sec337ms, min 13sec902ms This pull request refactors the aggregation function framework, primarily focusing on simplifying and unifying the deserialization and merging interfaces, as well as making improvements for better null handling and performance. The main changes remove redundant methods, consolidate logic, and introduce new configuration options. **API and Interface Simplification:** * Removed the `deserialize_from_column` and `deserialize_and_merge_from_column` virtual methods from the `IAggregateFunction` interface and its implementations, consolidating deserialization logic into `deserialize_and_merge_from_column_range` and providing a default wrapper for range-based deserialization. [[1]](diffhunk://#diff-c7bdf142116c49dcc34f63bea8dc12d418fd6dfedd7f43e27393d61f552e5266L169-L171) [[2]](diffhunk://#diff-c7bdf142116c49dcc34f63bea8dc12d418fd6dfedd7f43e27393d61f552e5266L181-R183) [[3]](diffhunk://#diff-c7bdf142116c49dcc34f63bea8dc12d418fd6dfedd7f43e27393d61f552e5266L564-L571) [[4]](diffhunk://#diff-92cbd4b807cf2772dbeb2e15c990a6f294c7400644dbacb34ef1af9db85594a9L327-L334) [[5]](diffhunk://#diff-92cbd4b807cf2772dbeb2e15c990a6f294c7400644dbacb34ef1af9db85594a9L343-L349) [[6]](diffhunk://#diff-505d65d575a270149a7573f84ce6e279d825efe602c32e5c6f7f7c3234fa9067L238-R238) [[7]](diffhunk://#diff-505d65d575a270149a7573f84ce6e279d825efe602c32e5c6f7f7c3234fa9067L271-L284) [[8]](diffhunk://#diff-4b9b0e2a479f215591138f801433000eb69a8962c7a390c7a46d5c2aa09cc3f9L190-L200) [[9]](diffhunk://#diff-9e5544bba64e0da947c606866b3f24cf045c54cec347c042854b1903a8d93057L149-L159) [[10]](diffhunk://#diff-9e5544bba64e0da947c606866b3f24cf045c54cec347c042854b1903a8d93057L170-L180) [[11]](diffhunk://#diff-a5dbb09237f197bffdcbd3bec4fdd089913ec143d96806618c8eeb4c5dbb8cfeL91-R91) [[12]](diffhunk://#diff-a5dbb09237f197bffdcbd3bec4fdd089913ec143d96806618c8eeb4c5dbb8cfeL122-L131) * Unified and clarified the vectorized merge interfaces (`merge_vec`, `merge_vec_selected`) by adding `__restrict` qualifiers and improving parameter naming for clarity and potential performance gains. [[1]](diffhunk://#diff-c7bdf142116c49dcc34f63bea8dc12d418fd6dfedd7f43e27393d61f552e5266L134-R141) [[2]](diffhunk://#diff-c7bdf142116c49dcc34f63bea8dc12d418fd6dfedd7f43e27393d61f552e5266L519-R526) **Null Handling and Configuration:** * Added a new query option and runtime state method `enable_aggregate_function_null_v2` to control enhanced null handling in aggregate functions. [[1]](diffhunk://#diff-efeb89760ec70cdad5ae9c6169e213b9efa84fccb6ede73f626bf6583bd82333R553-R557) [[2]](diffhunk://#diff-c7bdf142116c49dcc34f63bea8dc12d418fd6dfedd7f43e27393d61f552e5266R49) **Performance and Safety Improvements:** * Improved destruction logic in `IAggregateFunctionHelper::destroy_vec` to skip trivial cases, potentially reducing unnecessary work. * Refactored the streaming serialization interface, removing the old `streaming_agg_serialize` method in favor of a more consistent `streaming_agg_serialize_to_column`. [[1]](diffhunk://#diff-c7bdf142116c49dcc34f63bea8dc12d418fd6dfedd7f43e27393d61f552e5266L221-L223) [[2]](diffhunk://#diff-c7bdf142116c49dcc34f63bea8dc12d418fd6dfedd7f43e27393d61f552e5266L422-R425) [[3]](diffhunk://#diff-c7bdf142116c49dcc34f63bea8dc12d418fd6dfedd7f43e27393d61f552e5266L435-L440) **Aggregation Operator Code Cleanup:** * Removed unnecessary null-checking and unwrapping of nullable columns in aggregation sink and source operators, likely due to improved null handling in the aggregation framework. [[1]](diffhunk://#diff-df21df88282293d1bbe3223e0d95dcc5afc01da571e3a7c1d9dc9377a758d5daL305-L307) [[2]](diffhunk://#diff-df21df88282293d1bbe3223e0d95dcc5afc01da571e3a7c1d9dc9377a758d5daL357-L360) [[3]](diffhunk://#diff-df21df88282293d1bbe3223e0d95dcc5afc01da571e3a7c1d9dc9377a758d5daL415-L417) [[4]](diffhunk://#diff-d45f483806d3e0d0230fcb32c6c3b5da4a3eee517ac7c8664c3e79338817b44dL503-L505)

zlib_crc32: SplitBlockHashComputeTime: 269.15ms crc32c_fixed: SplitBlockHashComputeTime: 33.258ms zlib_crc32_fixed: SplitBlockHashComputeTime: 61.941ms

hello-stephen · 2026-03-19T03:21:09Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

This PR updates Doris FE/BE query execution behavior and serialization compatibility, introducing new session/query options and adjusting several vectorized execution internals and regression expectations.

Changes:

Add new TQueryOptions + SessionVariables for enable_aggregate_function_null_v2 and enable_adjust_conjunct_order_by_cost, and wire them into BE RuntimeState.
Introduce expression/function execute_cost() and use it to optionally reorder conjunct evaluation in scan/parquet/operator paths.
Update BE aggregation and fixed-length object serialization behaviors (incl. new BE exec version), with corresponding test/regression output updates.

Reviewed changes

Copilot reviewed 126 out of 126 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
gensrc/thrift/PaloInternalService.thrift	Adds new query options to Thrift `TQueryOptions`.
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java	Adds new session variables and forwards them into Thrift.
fe/fe-common/src/main/java/org/apache/doris/common/Config.java	Updates FE be-exec-version bounds.
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/ArrayAvg.java	Adjusts decimal precision computation for `array_avg`.
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/agg/Avg.java	Simplifies decimal AVG precision handling.
be/src/vec/functions/function.h	Adds `execute_cost()` to function base and forwards via wrapper.
be/src/vec/functions/functions_comparison.h	Sets a cost for comparison functions.
be/src/vec/exprs/vexpr.h / vexpr_context.* / vref.h / vliteral.h / vectorized_fn_call. / vcompound_pred.h / vruntimefilter_wrapper.h	Adds/propagates expression execution cost, including in function-call expressions.
be/src/pipeline/exec/operator.cpp / scan_operator.cpp	Optionally sorts conjuncts by cost after prepare / after runtime filter append.
be/src/vec/exec/format/parquet/vparquet_group_reader.cpp	Optionally sorts parquet filter conjuncts by cost.
be/src/runtime/runtime_state.h	Exposes runtime getters for the new query options.
be/src/vec/aggregate_functions/aggregate_function.h / helpers.h and multiple agg impl headers	Refactors agg serialization/merge APIs, adds null_v2 plumbing, optimizations, and removes some older methods.
be/src/vec/data_types/data_type_fixed_length_object.cpp	Adds new fixed-length object serialization guarded by new BE exec version.
be/src/agent/be_exec_version_manager.*	Bumps max BE exec version and introduces a new serialization version constant.
be/src/util/hash_util.hpp / be/test/util/crc32c_test.cpp / be/src/vec/columns/column_*	Adds zlib-compatible fixed-size CRC32 path and updates CRC callers/tests.
regression-test/data/**	Updates plan-shape outputs and some numeric results.
be/test/vec/**	Updates expected serialization sizes and adjusts aggregate tests to new APIs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java

+    @VariableMgr.VarAttr(name = ENABLE_AGGREGATE_FUNCTION_NULL_V2, fuzzy = true, needForward = true)
+    private boolean enableAggregateFunctionNullV2 = true;
+


be/src/agent/be_exec_version_manager.h

-constexpr inline int USE_CONST_SERDE =
-        8; // support const column in serialize/deserialize function: PR #41175
+constexpr inline int USE_NEW_FIXED_OBJECT_SERIALIZATION_VERSION = 10;



be/src/vec/aggregate_functions/aggregate_function.h

+    void deserialize_and_merge_from_column(AggregateDataPtr __restrict place, const IColumn& column,
+                                           Arena& arena) const {
+        deserialize_and_merge_from_column_range(place, column, 0, column.size() - 1, arena);
+    }


be/src/vec/aggregate_functions/helpers.h

+                    !std::is_same_v<                                                               \
+                            decltype(&FunctionTemplate::deserialize_and_merge_from_column_range),  \
+                            decltype(&IAggregateFunctionHelper<                                    \
+                                     FunctionTemplate>::deserialize_and_merge_from_column)>,       \


be/src/vec/data_types/data_type_fixed_length_object.cpp

+    if (be_exec_version >= USE_NEW_FIXED_OBJECT_SERIALIZATION_VERSION) {
+        // New format size calculation with streamvbyte
+        auto size = sizeof(bool) + sizeof(size_t) + sizeof(size_t) + sizeof(size_t);
+        auto real_need_copy_num = is_column_const(column) ? 1 : column.size();
+        const auto& src_col = assert_cast<const ColumnType&>(column);
+        auto mem_size = src_col.item_size() * real_need_copy_num;
+        if (mem_size <= SERIALIZED_MEM_SIZE_LIMIT) {
+            return size + mem_size;
+        } else {
+            // Throw exception if mem_size is large than UINT32_MAX
+            return size + sizeof(size_t) +
+                   std::max(mem_size, streamvbyte_max_compressedbytes(
+                                              cast_set<UInt32>(upper_int32(mem_size))));
+        }


BiteTheDDDDt · 2026-03-19T06:06:29Z

run buildall

hello-stephen · 2026-03-19T06:46:06Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.06% (1786/2259)
Line Coverage	64.42% (31939/49583)
Region Coverage	65.21% (15977/24499)
Branch Coverage	55.78% (8498/15236)

hello-stephen · 2026-03-19T09:52:56Z

BE UT Coverage Report

Increment line coverage 22.85% (183/801) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.76% (19393/36754)
Line Coverage	36.19% (181382/501198)
Region Coverage	32.63% (140007/429084)
Branch Coverage	33.66% (61004/181256)

hello-stephen · 2026-03-19T10:01:49Z

BE Regression && UT Coverage Report

Increment line coverage 83.40% (668/801) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.32% (25656/35975)
Line Coverage	54.18% (271021/500259)
Region Coverage	51.59% (223557/433322)
Branch Coverage	53.06% (96499/181882)

BiteTheDDDDt added 5 commits March 19, 2026 11:14

[Improvement] add zlib_crc32_fixed (apache#60735)

6e52d92

zlib_crc32: SplitBlockHashComputeTime: 269.15ms crc32c_fixed: SplitBlockHashComputeTime: 33.258ms zlib_crc32_fixed: SplitBlockHashComputeTime: 61.941ms

BiteTheDDDDt requested a review from yiguolei as a code owner March 19, 2026 03:21

Copilot AI review requested due to automatic review settings March 19, 2026 03:21

BiteTheDDDDt changed the title ~~Cp 0319~~ [Chore](pick) pick #58666 #61503 #60079 #60228 #60735 Mar 19, 2026

Copilot AI reviewed Mar 19, 2026

View reviewed changes

BiteTheDDDDt added 2 commits March 19, 2026 11:41

format

a2af0a0

fix

28675dc

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Chore](pick) pick #58666 #61503 #60079 #60228 #60735#61503

[Chore](pick) pick #58666 #61503 #60079 #60228 #60735#61503
BiteTheDDDDt wants to merge 7 commits intoapache:branch-4.1from
BiteTheDDDDt:cp_0319

BiteTheDDDDt commented Mar 19, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Mar 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

BiteTheDDDDt commented Mar 19, 2026

Uh oh!

hello-stephen commented Mar 19, 2026

Uh oh!

hello-stephen commented Mar 19, 2026

Uh oh!

hello-stephen commented Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@VariableMgr.VarAttr(name = ENABLE_AGGREGATE_FUNCTION_NULL_V2, fuzzy = true, needForward = true)
		private boolean enableAggregateFunctionNullV2 = true;

Conversation

BiteTheDDDDt commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hello-stephen commented Mar 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

BiteTheDDDDt commented Mar 19, 2026

Uh oh!

hello-stephen commented Mar 19, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Mar 19, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Mar 19, 2026

BE Regression && UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BiteTheDDDDt commented Mar 19, 2026 •

edited

Loading