[test](regression) Complete Paimon read/write P0 coverage - #66065
Conversation
Add merge-engine semantic coverage for Parquet and ORC across automatic and forced JNI readers. Verify unsupported Paimon DML remains atomic, document the Paimon/Iceberg P0 matrix, and isolate the known CTAS metadata atomicity defect.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Automated review conclusion: REQUEST_CHANGES — one P2 coverage-evidence defect; no P0/P1 blocker found.
The new merge-engine and write-boundary tests are otherwise internally consistent, but the coverage document points reader-parity maintainers to a suite whose advertised toggle no longer changes current-FE routing. The inline comment identifies the deterministic suite that should be cited instead.
Critical checkpoint conclusions:
- Goal and proof: the added suites exercise the intended Paimon merge semantics and unsupported-write boundary; the one remaining defect is the inaccurate JNI/native evidence mapping.
- Scope and clarity: the patch is focused on regression coverage and its coverage inventory.
- Concurrency/thread safety: no production concurrency is changed; the reviewed test concurrency and session isolation introduce no new shared-state defect.
- Error handling: the negative DML cases match concrete analyzer/command rejection paths before writer construction; the CTAS metadata leak is explicitly isolated as a gated known-bug reproducer.
- Memory safety and BE nullable handling: not applicable to these test/document-only changes.
- Lifecycle: catalogs and session variables are restored in
finallyblocks; no new production lifecycle is introduced. - Configuration: only existing regression gates/session variables are used and restored; no runtime configuration is added.
- Compatibility and parallel paths: no API, storage-format, or rolling-upgrade contract changes; Parquet/ORC, automatic/forced-JNI, and the relevant write variants were traced.
- Conditional checks and comments: the opt-in CTAS gate and merge-engine comments match the current behavior.
- Test coverage: positive merge-engine and negative write-boundary cases are broad; reader parity already has an explicit route-asserting suite, but line 72 cites the stale non-proof.
- Expected outputs/data correctness: all 105 merge-matrix golden rows and 15 boundary-output rows were statically checked against their queries and expected semantics; ordering/scalar determinism is present.
- Observability: no production path is changed, so no new logs or metrics are required.
- Transaction/persistence and writes: the six unsupported writes reject before transaction/writer creation; no EditLog or persistent product state is changed by this patch.
- FE/BE variables: none are added; existing reader-type propagation was traced across FE and both BE scanner generations.
- Performance: fixtures are bounded and no production hot path changes.
- Other issues: suite counts and referenced paths were verified; no additional substantiated issue remained after three review rounds.
Validation was static only. Per the review-runner instructions, no build or test command was run. No additional user focus was provided.
| | Paimon | Append table, partitioned table, primitive and nested types | Covered | `test_paimon_catalog`, `test_paimon_partition_table`, `test_paimon_full_schema_change` | | ||
| | Paimon | Primary-key deduplicate, partial-update, aggregation and first-row | Covered | `test_paimon_merge_engine_matrix` | | ||
| | Paimon | Fixed bucket, dynamic bucket and cross-partition update | Covered | `test_paimon_merge_engine_matrix`, `test_paimon_partition_pk_delete_refs` | | ||
| | Paimon | Parquet/ORC and mixed-format reads; JNI/native parity | Covered | `test_paimon_merge_engine_matrix`, `paimon_tb_mix_format`, `test_paimon_cpp_reader` | |
There was a problem hiding this comment.
[P2] Cite the suite that actually verifies JNI/native routing
test_paimon_cpp_reader does not establish this row's parity on the current FE. Logical splits are explicitly tagged PAIMON_JNI, so BE ignores enable_paimon_cpp_reader; raw splits auto-route native in both halves. The false/true toggle therefore compares the same per-split routing and cannot prove a second reader. test_paimon_partition_pk_delete_refs already asserts JNI-only versus positive-native plans (including SplitStat types) for both Parquet and ORC. Please cite that explicit proof here, or add equivalent route assertions to the listed matrix/mixed-format suites instead of relying on the stale CPP test.
|
run buildall |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
…rename to the connector Rebase branch-catalog-spi onto upstream-apache/master 042e613 (6 upstream commits, 63 local commits replayed). Only #65955 "Support Paimon table option passthrough" touched fe-core external tables; its 5 fe-core files were deleted or stripped by P5/T29, producing one content conflict (union: upstream's initNormalizeAndCheckProps vs our initHdfsExecutionAuthenticator), one keep-ours conflict (T29 makes fe-core paimon-SDK-free), and four modify/delete conflicts resolved by keeping the deletion. Two capabilities had to move into fe-connector-paimon: 1. paimon.table-option.* passthrough. New PaimonTableOptions ports the AbstractPaimonProperties members T29 deleted (TABLE_OPTION_PREFIX, extractTableOptions, validateTableOption, getTableOptionsForCopy, SupportedTableOptions). Wired at the three points the fe-core class covered: excluded from the catalog Options passthrough (PaimonCatalogFactory), fail-fast at CREATE/ALTER CATALOG (PaimonConnectorProvider.validateProperties, since the SPI path never runs initNormalizeAndCheckProps), and overlaid on table load (CatalogBackedPaimonCatalogOps.getTable -- the connector's only Catalog.getTable, matching where PaimonExternalCatalog.getPaimonTable applied it, so branch/time-travel/system tables and the serialized table shipped to BE all inherit it). Upstream's matching notifyPropertiesUpdated change needs no port: PluginDrivenExternalCatalog resets the whole connector on any property change, and the PaimonExternalMetaCache it invalidated is already deleted. 2. JNI IOManager namespace rename, which was a SILENT break. #65955 moved paimon.doris.* -> paimon.jni.* on both sides at once; the BE files (paimon_jni_reader.cpp x2) and PaimonJniScanner auto-merged to the new names while the FE mirror list lives in PaimonScanPlanProvider (legacy PaimonScanNode being deleted), so FE kept emitting doris.* against a BE that only reads jni.* -- compiling, merging and testing clean while the IOManager silently never engaged. Also drops jni.enable_file_reader_async, retired upstream along with the scanner's table.copy(buildTableOptions(..)); the equivalent knob is now paimon.table-option.file-reader-async-threshold. Tests: new PaimonTableOptionsTest (11, mirroring upstream's AbstractPaimonPropertiesTest table-option cases plus the connector-specific wirings and two real local FileSystemCatalog getTable cases); PaimonScanPlanProviderTest key names updated and the file-reader-async forwarding test inverted into a retirement guard. Module 393/393 (1 pre-existing live-connectivity skip), checkstyle 0, full FE install BUILD SUCCESS. Five mutations verified RED: reverting a key to doris.*, dropping the validateProperties extract, returning getTable unchanged, dropping the catalog Options exclusion, and comparing raw keys in forCopy. e2e (not run here): #66065's new suites assert exception "PaimonExternalCatalog", which is the legacy UnboundTableSinkCreator message. On this branch paimon is a PluginDrivenExternalCatalog, so the rejection comes from the connector's write capabilities instead; test_paimon_write_boundary and test_paimon_ctas_atomicity_negative updated accordingly. The UPDATE/DELETE/MERGE assertions in the same suite are unchanged -- paimon declares no DELETE/MERGE, so they still fall back to the legacy messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rename to the connector Rebase branch-catalog-spi onto upstream-apache/master 042e613 (6 upstream commits, 63 local commits replayed). Only #65955 "Support Paimon table option passthrough" touched fe-core external tables; its 5 fe-core files were deleted or stripped by P5/T29, producing one content conflict (union: upstream's initNormalizeAndCheckProps vs our initHdfsExecutionAuthenticator), one keep-ours conflict (T29 makes fe-core paimon-SDK-free), and four modify/delete conflicts resolved by keeping the deletion. Two capabilities had to move into fe-connector-paimon: 1. paimon.table-option.* passthrough. New PaimonTableOptions ports the AbstractPaimonProperties members T29 deleted (TABLE_OPTION_PREFIX, extractTableOptions, validateTableOption, getTableOptionsForCopy, SupportedTableOptions). Wired at the three points the fe-core class covered: excluded from the catalog Options passthrough (PaimonCatalogFactory), fail-fast at CREATE/ALTER CATALOG (PaimonConnectorProvider.validateProperties, since the SPI path never runs initNormalizeAndCheckProps), and overlaid on table load (CatalogBackedPaimonCatalogOps.getTable -- the connector's only Catalog.getTable, matching where PaimonExternalCatalog.getPaimonTable applied it, so branch/time-travel/system tables and the serialized table shipped to BE all inherit it). Upstream's matching notifyPropertiesUpdated change needs no port: PluginDrivenExternalCatalog resets the whole connector on any property change, and the PaimonExternalMetaCache it invalidated is already deleted. 2. JNI IOManager namespace rename, which was a SILENT break. #65955 moved paimon.doris.* -> paimon.jni.* on both sides at once; the BE files (paimon_jni_reader.cpp x2) and PaimonJniScanner auto-merged to the new names while the FE mirror list lives in PaimonScanPlanProvider (legacy PaimonScanNode being deleted), so FE kept emitting doris.* against a BE that only reads jni.* -- compiling, merging and testing clean while the IOManager silently never engaged. Also drops jni.enable_file_reader_async, retired upstream along with the scanner's table.copy(buildTableOptions(..)); the equivalent knob is now paimon.table-option.file-reader-async-threshold. Tests: new PaimonTableOptionsTest (11, mirroring upstream's AbstractPaimonPropertiesTest table-option cases plus the connector-specific wirings and two real local FileSystemCatalog getTable cases); PaimonScanPlanProviderTest key names updated and the file-reader-async forwarding test inverted into a retirement guard. Module 393/393 (1 pre-existing live-connectivity skip), checkstyle 0, full FE install BUILD SUCCESS. Five mutations verified RED: reverting a key to doris.*, dropping the validateProperties extract, returning getTable unchanged, dropping the catalog Options exclusion, and comparing raw keys in forCopy. e2e (not run here): #66065's new suites assert exception "PaimonExternalCatalog", which is the legacy UnboundTableSinkCreator message. On this branch paimon is a PluginDrivenExternalCatalog, so the rejection comes from the connector's write capabilities instead; test_paimon_write_boundary and test_paimon_ctas_atomicity_negative updated accordingly. The UPDATE/DELETE/MERGE assertions in the same suite are unchanged -- paimon declares no DELETE/MERGE, so they still fall back to the legacy messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rename to the connector Rebase branch-catalog-spi onto upstream-apache/master 042e613 (6 upstream commits, 63 local commits replayed). Only #65955 "Support Paimon table option passthrough" touched fe-core external tables; its 5 fe-core files were deleted or stripped by P5/T29, producing one content conflict (union: upstream's initNormalizeAndCheckProps vs our initHdfsExecutionAuthenticator), one keep-ours conflict (T29 makes fe-core paimon-SDK-free), and four modify/delete conflicts resolved by keeping the deletion. Two capabilities had to move into fe-connector-paimon: 1. paimon.table-option.* passthrough. New PaimonTableOptions ports the AbstractPaimonProperties members T29 deleted (TABLE_OPTION_PREFIX, extractTableOptions, validateTableOption, getTableOptionsForCopy, SupportedTableOptions). Wired at the three points the fe-core class covered: excluded from the catalog Options passthrough (PaimonCatalogFactory), fail-fast at CREATE/ALTER CATALOG (PaimonConnectorProvider.validateProperties, since the SPI path never runs initNormalizeAndCheckProps), and overlaid on table load (CatalogBackedPaimonCatalogOps.getTable -- the connector's only Catalog.getTable, matching where PaimonExternalCatalog.getPaimonTable applied it, so branch/time-travel/system tables and the serialized table shipped to BE all inherit it). Upstream's matching notifyPropertiesUpdated change needs no port: PluginDrivenExternalCatalog resets the whole connector on any property change, and the PaimonExternalMetaCache it invalidated is already deleted. 2. JNI IOManager namespace rename, which was a SILENT break. #65955 moved paimon.doris.* -> paimon.jni.* on both sides at once; the BE files (paimon_jni_reader.cpp x2) and PaimonJniScanner auto-merged to the new names while the FE mirror list lives in PaimonScanPlanProvider (legacy PaimonScanNode being deleted), so FE kept emitting doris.* against a BE that only reads jni.* -- compiling, merging and testing clean while the IOManager silently never engaged. Also drops jni.enable_file_reader_async, retired upstream along with the scanner's table.copy(buildTableOptions(..)); the equivalent knob is now paimon.table-option.file-reader-async-threshold. Tests: new PaimonTableOptionsTest (11, mirroring upstream's AbstractPaimonPropertiesTest table-option cases plus the connector-specific wirings and two real local FileSystemCatalog getTable cases); PaimonScanPlanProviderTest key names updated and the file-reader-async forwarding test inverted into a retirement guard. Module 393/393 (1 pre-existing live-connectivity skip), checkstyle 0, full FE install BUILD SUCCESS. Five mutations verified RED: reverting a key to doris.*, dropping the validateProperties extract, returning getTable unchanged, dropping the catalog Options exclusion, and comparing raw keys in forCopy. e2e (not run here): #66065's new suites assert exception "PaimonExternalCatalog", which is the legacy UnboundTableSinkCreator message. On this branch paimon is a PluginDrivenExternalCatalog, so the rejection comes from the connector's write capabilities instead; test_paimon_write_boundary and test_paimon_ctas_atomicity_negative updated accordingly. The UPDATE/DELETE/MERGE assertions in the same suite are unchanged -- paimon declares no DELETE/MERGE, so they still fall back to the legacy messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rename to the connector Rebase branch-catalog-spi onto upstream-apache/master 042e613 (6 upstream commits, 63 local commits replayed). Only #65955 "Support Paimon table option passthrough" touched fe-core external tables; its 5 fe-core files were deleted or stripped by P5/T29, producing one content conflict (union: upstream's initNormalizeAndCheckProps vs our initHdfsExecutionAuthenticator), one keep-ours conflict (T29 makes fe-core paimon-SDK-free), and four modify/delete conflicts resolved by keeping the deletion. Two capabilities had to move into fe-connector-paimon: 1. paimon.table-option.* passthrough. New PaimonTableOptions ports the AbstractPaimonProperties members T29 deleted (TABLE_OPTION_PREFIX, extractTableOptions, validateTableOption, getTableOptionsForCopy, SupportedTableOptions). Wired at the three points the fe-core class covered: excluded from the catalog Options passthrough (PaimonCatalogFactory), fail-fast at CREATE/ALTER CATALOG (PaimonConnectorProvider.validateProperties, since the SPI path never runs initNormalizeAndCheckProps), and overlaid on table load (CatalogBackedPaimonCatalogOps.getTable -- the connector's only Catalog.getTable, matching where PaimonExternalCatalog.getPaimonTable applied it, so branch/time-travel/system tables and the serialized table shipped to BE all inherit it). Upstream's matching notifyPropertiesUpdated change needs no port: PluginDrivenExternalCatalog resets the whole connector on any property change, and the PaimonExternalMetaCache it invalidated is already deleted. 2. JNI IOManager namespace rename, which was a SILENT break. #65955 moved paimon.doris.* -> paimon.jni.* on both sides at once; the BE files (paimon_jni_reader.cpp x2) and PaimonJniScanner auto-merged to the new names while the FE mirror list lives in PaimonScanPlanProvider (legacy PaimonScanNode being deleted), so FE kept emitting doris.* against a BE that only reads jni.* -- compiling, merging and testing clean while the IOManager silently never engaged. Also drops jni.enable_file_reader_async, retired upstream along with the scanner's table.copy(buildTableOptions(..)); the equivalent knob is now paimon.table-option.file-reader-async-threshold. Tests: new PaimonTableOptionsTest (11, mirroring upstream's AbstractPaimonPropertiesTest table-option cases plus the connector-specific wirings and two real local FileSystemCatalog getTable cases); PaimonScanPlanProviderTest key names updated and the file-reader-async forwarding test inverted into a retirement guard. Module 393/393 (1 pre-existing live-connectivity skip), checkstyle 0, full FE install BUILD SUCCESS. Five mutations verified RED: reverting a key to doris.*, dropping the validateProperties extract, returning getTable unchanged, dropping the catalog Options exclusion, and comparing raw keys in forCopy. e2e (not run here): #66065's new suites assert exception "PaimonExternalCatalog", which is the legacy UnboundTableSinkCreator message. On this branch paimon is a PluginDrivenExternalCatalog, so the rejection comes from the connector's write capabilities instead; test_paimon_write_boundary and test_paimon_ctas_atomicity_negative updated accordingly. The UPDATE/DELETE/MERGE assertions in the same suite are unchanged -- paimon declares no DELETE/MERGE, so they still fall back to the legacy messages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
The Paimon P0 regression suite did not distinguish the four primary-key merge engines and did not verify that every unsupported Doris data-write shape leaves both data and snapshots unchanged. This PR:
Release note
None
Check List (For Author)
Validation:
Paimon merge-engine matrix and write-boundary suites: passed, 2/2.
Existing Iceberg MERGE semantics suite: passed.
Isolated Paimon CTAS negative suite: reproduced the expected metadata-atomicity failure and cleaned up the target table.
FE and BE build: passed.
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)