[draft](ivm) for run the test pipeline#61496
Draft
yujun777 wants to merge 23 commits intoapache:masterfrom
Draft
[draft](ivm) for run the test pipeline#61496yujun777 wants to merge 23 commits intoapache:masterfrom
yujun777 wants to merge 23 commits intoapache:masterfrom
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 26843 ms |
TPC-DS: Total hot run time: 168011 ms |
yujun777
force-pushed
the
ivm-yj-1
branch
2 times, most recently
from
dc5d312 to
c1b3f3a
Compare
…ogging - Require non-null detailMessage in IVMRefreshResult.fallback() to match IVMCapabilityResult.unsupported() contract; remove single-arg overload - Add toString() to IVMRefreshResult for log readability - Add WARN logging on all fallback paths in IVMRefreshManager with MV name - Make doRefresh() the public API; remove redundant ivmRefresh() wrapper
- Remove IVMPlanPattern, IVMPlanAnalysis, IVMPlanAnalyzer, IVMDeltaPlannerDispatcher - IVMCapabilityChecker now takes List<DeltaPlanBundle> instead of IVMPlanAnalysis - IVMRefreshManager simplified to 2 deps: capabilityChecker + deltaExecutor - Delta bundles produced by Nereids rules, retrieved via MTMVAnalyzeQueryInfo - Add analyzeDeltaBundles() hook for testability - Add ivmDeltaBundles to MTMVAnalyzeQueryInfo, populated from CascadesContext - Update tests to JUnit 5 and new interface signatures - Fix checkstyle import order in CreateTableCommandTest
- Delete IvmRewriteMtmvPlan placeholder and its test - Remove rewriteRootPlan field from CascadesContext (no longer needed) - Replace IVM_REWRITE_MTMV_PLAN with IVM_NORMALIZE_MTMV_PLAN in RuleType - Add IvmNormalizeMtmvPlan skeleton (row-id injection, avg rewrite, TODO) - Add IvmDeltaScanOnly and IvmDeltaAggRoot skeletons - Merge delta rules into single topic in Rewriter
- Add IvmAnalyzeMode enum (NONE/NORMALIZE_ONLY/FULL) to replace boolean flags - Replace enableIvmRewriteInNereids with enableIvmNormalRewrite + enableIvmDeltaRewrite - MTMVPlanUtil.analyzeQuery/analyzeQueryWithSql take IvmAnalyzeMode parameter - CreateMTMVInfo: NORMALIZE_ONLY for incremental MV, NONE otherwise - ensureMTMVQueryUsable: same mode as CREATE MV - IVMRefreshManager: FULL mode (normalize + delta) - Update IvmNormalizeMtmvPlan/IvmDeltaScanOnly/IvmDeltaAggRoot to use new session vars - Fix MTMVPlanUtilTest: JUnit5, IvmAnalyzeMode.NONE, updated CountingSessionVariable - Add testAnalyzeQueryIvmAnalyzeModeSetSessionVariables covering all 3 modes
- Add IvmContext: holds Map<Slot, isDeterministic> rowIdDeterminism + List<DeltaPlanBundle>
- Replace ivmDeltaBundles in CascadesContext with Optional<IvmContext>
- IvmNormalizeMtmvPlan: whitelist-based visitor (DefaultPlanRewriter<IvmContext>)
- visitLogicalOlapScan: inject __IVM_ROW_ID__ at index 0 via LogicalProject
- MOW: Alias(buildRowIdHash(uk...), __IVM_ROW_ID__) -> deterministic
- DUP_KEYS: Alias(UuidNumeric(), __IVM_ROW_ID__) -> non-deterministic
- MOR / AGG_KEYS: throw AnalysisException
- visitLogicalProject: propagate child row-id; throw if child has none
- visit: throw for any unwhitelisted node
- buildRowIdHash: uses murmur_hash3_64 (TODO: replace with 128-bit hash)
- MTMVPlanUtil: read delta bundles from IvmContext instead of direct field
- Tests: DUP_KEYS, MOW (deterministic), MOR (throws), AGG_KEYS (throws),
project propagation, unsupported node, gate disabled
- IvmNormalizeMtmvPlan: whitelist LogicalResultSink, prepend row-id; extract hasRowIdInOutputs/prependRowId helpers - ColumnDefinition: add newIvmRowIdColumnDefinition with mv_ prefix - MTMVPlanUtil: prepend row-id ColumnDefinition at index 0; reset IVM session vars in finally block to prevent test leakage - BaseViewInfo: extract static rewriteProjectsToUserDefineAlias overload - CreateMTMVInfo: fix rewriteQuerySql to snapshot/restore rewrite map and call alias rewrite when simpleColumnDefinitions present - CreateTableCommandTest: add 4 IVM UTs covering scan, project-scan, no-alias, alias rewrite, and column count mismatch
- CreateMTMVInfo: set UNIQUE_KEYS + enable_unique_key_merge_on_write=true
for INCREMENTAL refresh MVs; reject user-specified key columns
- MTMVPlanUtil.analyzeKeys: return new List instead of mutating the
immutable input list; throw if IVM row-id column not found in columns
- MTMVPlanUtil.analyzeQuery: only reset IVM session vars in finally block
for modes that actually set them (NORMALIZE_ONLY resets NORMAL only,
FULL resets both, NONE resets neither)
- MTMVPlanUtilTest: add 4 new UTs covering UNIQUE_KEYS+MOW assertion,
DUP_KEYS for non-IVM, and rejection of user-specified UNIQUE/DUP keys
- CountingSessionVariable: count only enabling ("true") setVarOnce calls
…aRewriter Move IVM delta plan generation out of Nereids rewrite rules into an external IvmDeltaRewriter that will be called by IVMRefreshManager. IvmNormalizeMtmvPlan now stores the normalized plan in IvmContext so IVMRefreshManager can retrieve it for delta rewriting. - Add normalizedPlan field to IvmContext, store after normalization - Add ivmNormalizedPlan field to MTMVAnalyzeQueryInfo - Delete IvmDeltaScanOnly, IvmDeltaAggRoot, IvmAnalyzeMode - Remove IVM_DELTA_SCAN_ONLY/IVM_DELTA_AGG_ROOT from RuleType/Rewriter - Remove ENABLE_IVM_DELTA_REWRITE session variable - Remove deltaCommandBundles from IvmContext - Replace IvmAnalyzeMode enum with boolean enableIvmNormalize - Create skeleton IvmDeltaRewriter + IvmDeltaRewriteContext - Rewrite IVMRefreshManager.analyzeDeltaCommandBundles to use normalized plan (returns empty bundles for now, triggers fallback)
IvmDeltaRewriter no longer extends DefaultPlanRewriter. It now validates the normalized plan is a supported scan-only or project-scan pattern, extracts the base table, and produces an INSERT INTO mv command wrapped in a DeltaCommandBundle. IvmDeltaRewriteContext gains a ConnectContext field, and IVMRefreshManager.analyzeDeltaCommandBundles is wired to call the rewriter.
…es to concrete classes IVMDeltaExecutor now contains real execution logic following the MTMVTask.exec() pattern: creates ConnectContext/StatementContext/StmtExecutor, runs the command, and checks query state. IVMCapabilityChecker returns ok() by default. IVMRefreshManager uses a no-arg public constructor, instantiating both collaborators internally, with a @VisibleForTesting constructor for injection.
For INCREMENTAL MVs, attempt IVM refresh first via IVMRefreshManager. On success, return early and skip partition-based refresh. On fallback, log the reason and continue with existing refresh path.
…eltaExecutor Extract common command execution boilerplate shared by MTMVTask.exec() and IVMDeltaExecutor.executeBundle() into MTMVPlanUtil.executeCommand(). This also adds the missing audit logging to IVM delta execution.
Contributor
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)