feat(plan): optimize REPLACE index rewrites and fix rowid if panic

[ck89119]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #23946

What this PR does / why we need it:

This PR implements the REPLACE medium-term optimization (plan B) and includes a follow-up fix found during BVT validation.

Main changes:

1. Consume skipUniqueIdx in REPLACE path

• Skip unnecessary unique dedup joins when unique columns are all default NULL.
• Skip corresponding unique lock targets when dedup is skipped.

2. Avoid redundant index delete+insert for REPLACE

• Build needRewriteIdx in final projection.
• For unchanged index key + unchanged main PK mapping, project NULL to index update channels so MULTI_UPDATE skips redundant index writes.

3. Add conservative static filter pushdown for REPLACE VALUES

• For REPLACE ... VALUES without explicit column list (and bounded rows), push OR-equality filters to PK dedup scan and single-part unique dedup scans.

4. Fix panic found by BVT after introducing conditional projection

• Root cause: if/iff did not support rowid, while REPLACE used if(needRewrite, old_rowid, null).
• Add rowid support to if/iff function typing/execution.
• Add defensive nil checks in exprCanRemoveProject to avoid optimizer panic on malformed expressions.

Added/updated tests:

• REPLACE planner tests in pkg/sql/plan/build_test.go
• if(rowid) function test in pkg/sql/plan/function/operatorSet_test.go

Verification:

• go test ./pkg/sql/plan/function -run 'Test_Iff(Rowid|Check_MixedTypes)' -count=1
• go test ./pkg/sql/plan -run TestReplace -count=1
• go test ./pkg/sql/plan -count=1
• make static-check
• ../mo-tester/run.sh -p /Users/LoveYY/Develop/matrixorigin/perf-replace-affected-rows/test/distributed/cases/dml/replace -g -n

[Copilot]

Pull request overview

This PR improves the REPLACE INTO planning/execution path in MatrixOne by reducing unnecessary unique-index work, avoiding redundant index rewrites, and aligning affected_rows behavior with MySQL REPLACE semantics. It also includes a follow-up stability fix (rowid support in if/iff and additional nil defenses) found during BVT.

Changes:

• Add UpdateCtx.is_replace and propagate it through compile/remote-run into multi_update so REPLACE can count deleted rows toward affected_rows.
• Optimize REPLACE planning: consume skipUniqueIdx, add conditional projection to skip redundant index delete+insert, and add bounded static scan-filter pushdown for REPLACE ... VALUES.
• Fix planner/executor panics by adding rowid support to if/iff and adding defensive nil checks in optimizer helpers.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
proto/plan.proto	Adds is_replace flag to UpdateCtx for REPLACE semantics.
pkg/pb/plan/plan.pb.go	Regenerated protobuf code for is_replace.
pkg/sql/plan/bind_replace.go	Implements REPLACE optimizations (skip unique dedup, conditional index rewrite, static filter pushdown).
pkg/sql/colexec/multi_update/types.go	Uses IsReplace to include delete rows in affected rows for REPLACE.
pkg/sql/compile/operator.go	Propagates IsReplace into VM multi_update context.
pkg/sql/compile/remoterun.go	Preserves IsReplace across pipeline serialization/deserialization.
pkg/sql/plan/deepcopy.go	Deep-copies IsReplace in UpdateCtx clones.
pkg/sql/plan/function/operatorSet.go	Adds rowid typing/execution support for if/iff.
pkg/sql/plan/function/operatorSet_test.go	Adds if(rowid, …) coverage.
pkg/sql/plan/opt_misc.go	Adds nil guards to avoid optimizer panic.
pkg/sql/plan/build_util.go	Adds nil-arg guard in boolean conversion helper.
pkg/sql/plan/build_util_test.go	Adds regression test for nil-arg guard.
pkg/sql/plan/build_test.go	Adds planner structure tests for new REPLACE optimizations.
pkg/sql/colexec/multi_update/affected_rows_test.go	Adds focused unit tests for affected-rows semantics incl. REPLACE.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Static scan filter pushdown is built from valueExprs bound from VALUES. If any valueExpr contains volatile/non-deterministic functions (e.g. rand(), uuid(), now()) and is reused in TABLE_SCAN FilterList, it can miss conflicts and break REPLACE semantics. Consider restricting collected values to deterministic runtime-constants (literals/params) or running a volatility check (similar to checkExprForVolatileFunc) and disabling this optimization when such functions appear.

Suggested change

	valueExpr, err = binder.BindExpr(astExpr, 0, true)
	if err != nil {
	return nil, nil
	}
	if isEnumPlanType(&colDef.Typ) {
	valueExpr, err = funcCastForEnumType(builder.GetContext(), valueExpr, colDef.Typ)
	if err != nil {
	return nil, err
	}
	} else if isSetPlanType(&colDef.Typ) {
	valueExpr, err = funcCastForSetType(builder.GetContext(), valueExpr, colDef.Typ)
	if err != nil {
	return nil, err
	}
	} else if isGeometryPlanType(&colDef.Typ) {
	valueExpr, err = funcCastForGeometryType(builder.GetContext(), valueExpr, colDef.Typ)
	if err != nil {
	return nil, err
	}
	}
	// Static scan filter pushdown for REPLACE must only reuse values that are
	// known to be deterministic runtime-constants. Binding an arbitrary VALUES
	// expression here can capture volatile functions such as rand(), uuid(), or
	// now(), which may cause conflict detection to diverge from the inserted row.
	// Conservatively disable the optimization when the value is not already
	// recognized as a direct constant expression.
	return nil, nil

[Ariznawlll]

/auto-test-pr

[Ariznawlll]

PR #24163 测试覆盖分析

变更摘要

本次 PR 针对 REPLACE INTO 语句的执行计划进行了优化和修复，主要包括：

• 优化唯一索引去重策略（skipUniqueIdx），避免不必要的唯一索引 join 和锁操作。
• 避免 REPLACE 语句下冗余的索引 delete+insert 操作，通过 needRewriteIdx 判断是否需要真正写入索引。
• 针对 REPLACE ... VALUES 无列清单的场景，推送静态 filter 到主键/唯一索引去重 scan。
• 修复 BVT 发现的 panic 问题（if/iff 函数对 rowid 的支持和防御性判空）。
• 新增/完善了相关单元测试（plan 层和 colexec 层）。

涉及主要模块：pkg/sql/plan/（优化器、函数）、pkg/sql/compile/（执行计划）、pkg/sql/colexec/（执行引擎）、pkg/pb/plan/（协议）、proto/plan.proto。

6 类测试覆盖情况

测试类型	覆盖状态	说明
BVT	✅	已有 REPLACE/唯一索引相关 BVT，且本 PR 明确提及 BVT 验证和补充了 plan 层/函数相关测试。
稳定性	✅	涉及 plan/compile/colexec 路径，TPCH/TPCC/Sysbench 稳定性用例已覆盖 REPLACE/UPDATE/索引等场景。
Chaos	➖	仅涉及 SQL 计划与执行逻辑，无需故障注入验证。
大数据	⚠️	涉及 REPLACE/唯一索引/去重/索引写入优化，建议补充大数据量下 REPLACE INTO 带唯一索引的 case，验证优化效果和正确性。
PITR	➖	与备份恢复无关。
Snapshot	➖	与快照功能无关。

图例: ✅ 已覆盖 ⚠️ 需补充 ➖ 不相关

建议

• 大数据测试：建议在 mo-nightly-regression 的 big_data 分支补充如下 case，验证大表下 REPLACE INTO 带唯一索引的正确性和性能：

  • 表结构示例：

    CREATE TABLE t_replace_big (
      id BIGINT PRIMARY KEY,
      u1 INT UNIQUE,
      v1 VARCHAR(100)
    );

  • 数据准备：先批量插入 1000 万行数据（可用 COS 预置数据）。
  • 测试 SQL 示例：

    -- 覆盖唯一索引冲突和无冲突两种情况
    REPLACE INTO t_replace_big VALUES (100, 200, 'foo'), (101, 201, 'bar');
    REPLACE INTO t_replace_big VALUES (1, 100, 'dup');  -- id=1 已存在，u1=100 可能冲突

  • 验证点：REPLACE 后 affected_rows、唯一索引约束、数据正确性。

• BVT：已补充相关测试，无需额外动作。

🤖 自动生成的测试 PR

✅ 大数据 PR 已提交：https://github.com/Ariznawlll/mo-nightly-regression/pull/2

---

由 AI Test Analyzer 自动生成 · workflow run