[Optimization](array) Avoid materializing const array columns in element_at#64175
Open
HappenLee wants to merge 2 commits into
Open
[Optimization](array) Avoid materializing const array columns in element_at#64175HappenLee wants to merge 2 commits into
HappenLee wants to merge 2 commits into
Conversation
…ent_at (apache#64039) When the array argument to element_at is a ColumnConst (e.g. a literal array repeated across rows), skip convert_to_full_column_if_const() and use index_check_const() instead. This avoids O(N) memory expansion for the array column. Add unit tests covering const int32/string arrays, null arrays, null indices, and large batches. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Reviewed the PR and found no blocking issues.
Critical checkpoint conclusions:
- Goal and proof: The change avoids materializing constant array arguments in element_at while preserving row-wise results via index_check_const; added BE unit coverage targets const int32/string arrays, null array, null index, and large batches.
- Scope and clarity: The implementation is focused on the array path and leaves the map path behavior unchanged where row-aligned offsets are still needed.
- Concurrency: Not applicable; this is per-block function execution with no shared mutable state or new threads.
- Lifecycle/static initialization: Not applicable; no static/global lifecycle-sensitive objects were added.
- Configuration/compatibility: Not applicable; no config, storage format, or FE-BE protocol changes.
- Parallel paths: Map lookup path is intentionally unchanged; array numeric/string/common paths were updated consistently.
- Null handling/data correctness: Reviewed nullable array, nullable index, nested nullable elements, positive/negative/out-of-range indices, and const-array row mapping; behavior appears consistent with existing semantics.
- Tests: Targeted tests were added for the optimized const-array scenarios. I attempted to run ./run-be-ut.sh --run --filter=function_array_element_test.*, but this runner failed during gensrc setup because thirdparty/installed/bin/protoc was missing, before compiling the test.
- Observability/transactions/persistence: Not applicable for this local vectorized function optimization.
- Performance: The change removes the O(N) materialization for const array inputs without introducing obvious redundant work in hot paths.
User focus: No additional user-provided review focus was present.
Contributor
Author
|
run buildall |
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 28968 ms |
Contributor
TPC-DS: Total hot run time: 169028 ms |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
convert_to_full_column_if_const()for the array argument inelement_atwhen it's aColumnConst, usingunpack_if_const()+index_check_const()instead to avoid O(N) memory expansion.is_const_arrayparameter to_execute_nullable,_execute_number,_execute_string, and_execute_commonmethods.Test plan
function_array_element_test.cpp— 5 new test cases covering const array scenarios.🤖 Generated with Claude Code