Skip to content

[test](regression) Complete Paimon read/write P0 coverage - #66065

Merged
morningman merged 1 commit into
apache:masterfrom
Gabriel39:ai/paimon-iceberg-p0-audit-20260726
Jul 27, 2026
Merged

[test](regression) Complete Paimon read/write P0 coverage#66065
morningman merged 1 commit into
apache:masterfrom
Gabriel39:ai/paimon-iceberg-p0-audit-20260726

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

The Paimon P0 regression suite did not distinguish the four primary-key merge engines and did not verify that every unsupported Doris data-write shape leaves both data and snapshots unchanged. This PR:

  • adds Parquet and ORC coverage for deduplicate, partial-update, aggregation, and first-row tables under automatic and forced-JNI reader routing;
  • verifies dynamic-bucket cross-partition deduplication;
  • covers rejected INSERT, INSERT SELECT, INSERT OVERWRITE, UPDATE, DELETE, and MERGE statements with pre/post row and snapshot invariants;
  • records the Paimon CTAS metadata-atomicity defect as an isolated opt-in negative regression; and
  • documents the Paimon/Iceberg read/write P0 coverage matrix and why no additional Iceberg P0 case is needed for the currently supported surface.

Release note

None

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

Validation:

  • Paimon merge-engine matrix and write-boundary suites: passed, 2/2.

  • Existing Iceberg MERGE semantics suite: passed.

  • Isolated Paimon CTAS negative suite: reproduced the expected metadata-atomicity failure and cleaned up the target table.

  • FE and BE build: passed.

  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Add merge-engine semantic coverage for Parquet and ORC across automatic and forced JNI readers. Verify unsupported Paimon DML remains atomic, document the Paimon/Iceberg P0 matrix, and isolate the known CTAS metadata atomicity defect.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@Gabriel39
Gabriel39 marked this pull request as ready for review July 26, 2026 10:24

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review conclusion: REQUEST_CHANGES — one P2 coverage-evidence defect; no P0/P1 blocker found.

The new merge-engine and write-boundary tests are otherwise internally consistent, but the coverage document points reader-parity maintainers to a suite whose advertised toggle no longer changes current-FE routing. The inline comment identifies the deterministic suite that should be cited instead.

Critical checkpoint conclusions:

  • Goal and proof: the added suites exercise the intended Paimon merge semantics and unsupported-write boundary; the one remaining defect is the inaccurate JNI/native evidence mapping.
  • Scope and clarity: the patch is focused on regression coverage and its coverage inventory.
  • Concurrency/thread safety: no production concurrency is changed; the reviewed test concurrency and session isolation introduce no new shared-state defect.
  • Error handling: the negative DML cases match concrete analyzer/command rejection paths before writer construction; the CTAS metadata leak is explicitly isolated as a gated known-bug reproducer.
  • Memory safety and BE nullable handling: not applicable to these test/document-only changes.
  • Lifecycle: catalogs and session variables are restored in finally blocks; no new production lifecycle is introduced.
  • Configuration: only existing regression gates/session variables are used and restored; no runtime configuration is added.
  • Compatibility and parallel paths: no API, storage-format, or rolling-upgrade contract changes; Parquet/ORC, automatic/forced-JNI, and the relevant write variants were traced.
  • Conditional checks and comments: the opt-in CTAS gate and merge-engine comments match the current behavior.
  • Test coverage: positive merge-engine and negative write-boundary cases are broad; reader parity already has an explicit route-asserting suite, but line 72 cites the stale non-proof.
  • Expected outputs/data correctness: all 105 merge-matrix golden rows and 15 boundary-output rows were statically checked against their queries and expected semantics; ordering/scalar determinism is present.
  • Observability: no production path is changed, so no new logs or metrics are required.
  • Transaction/persistence and writes: the six unsupported writes reject before transaction/writer creation; no EditLog or persistent product state is changed by this patch.
  • FE/BE variables: none are added; existing reader-type propagation was traced across FE and both BE scanner generations.
  • Performance: fixtures are bounded and no production hot path changes.
  • Other issues: suite counts and referenced paths were verified; no additional substantiated issue remained after three review rounds.

Validation was static only. Per the review-runner instructions, no build or test command was run. No additional user focus was provided.

| Paimon | Append table, partitioned table, primitive and nested types | Covered | `test_paimon_catalog`, `test_paimon_partition_table`, `test_paimon_full_schema_change` |
| Paimon | Primary-key deduplicate, partial-update, aggregation and first-row | Covered | `test_paimon_merge_engine_matrix` |
| Paimon | Fixed bucket, dynamic bucket and cross-partition update | Covered | `test_paimon_merge_engine_matrix`, `test_paimon_partition_pk_delete_refs` |
| Paimon | Parquet/ORC and mixed-format reads; JNI/native parity | Covered | `test_paimon_merge_engine_matrix`, `paimon_tb_mix_format`, `test_paimon_cpp_reader` |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Cite the suite that actually verifies JNI/native routing

test_paimon_cpp_reader does not establish this row's parity on the current FE. Logical splits are explicitly tagged PAIMON_JNI, so BE ignores enable_paimon_cpp_reader; raw splits auto-route native in both halves. The false/true toggle therefore compares the same per-split routing and cannot prove a second reader. test_paimon_partition_pk_delete_refs already asserts JNI-only versus positive-native plans (including SplitStat types) for both Parquet and ORC. Please cite that explicit proof here, or add equivalent route assertions to the listed matrix/mixed-format suites instead of relying on the stale CPP test.

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jul 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningman
morningman merged commit 042e613 into apache:master Jul 27, 2026
34 checks passed
morningman added a commit that referenced this pull request Jul 27, 2026
…rename to the connector

Rebase branch-catalog-spi onto upstream-apache/master 042e613 (6 upstream
commits, 63 local commits replayed). Only #65955 "Support Paimon table option
passthrough" touched fe-core external tables; its 5 fe-core files were deleted
or stripped by P5/T29, producing one content conflict (union: upstream's
initNormalizeAndCheckProps vs our initHdfsExecutionAuthenticator), one
keep-ours conflict (T29 makes fe-core paimon-SDK-free), and four modify/delete
conflicts resolved by keeping the deletion.

Two capabilities had to move into fe-connector-paimon:

1. paimon.table-option.* passthrough. New PaimonTableOptions ports the
   AbstractPaimonProperties members T29 deleted (TABLE_OPTION_PREFIX,
   extractTableOptions, validateTableOption, getTableOptionsForCopy,
   SupportedTableOptions). Wired at the three points the fe-core class covered:
   excluded from the catalog Options passthrough (PaimonCatalogFactory),
   fail-fast at CREATE/ALTER CATALOG (PaimonConnectorProvider.validateProperties,
   since the SPI path never runs initNormalizeAndCheckProps), and overlaid on
   table load (CatalogBackedPaimonCatalogOps.getTable -- the connector's only
   Catalog.getTable, matching where PaimonExternalCatalog.getPaimonTable applied
   it, so branch/time-travel/system tables and the serialized table shipped to
   BE all inherit it).

   Upstream's matching notifyPropertiesUpdated change needs no port:
   PluginDrivenExternalCatalog resets the whole connector on any property
   change, and the PaimonExternalMetaCache it invalidated is already deleted.

2. JNI IOManager namespace rename, which was a SILENT break. #65955 moved
   paimon.doris.* -> paimon.jni.* on both sides at once; the BE files
   (paimon_jni_reader.cpp x2) and PaimonJniScanner auto-merged to the new names
   while the FE mirror list lives in PaimonScanPlanProvider (legacy
   PaimonScanNode being deleted), so FE kept emitting doris.* against a BE that
   only reads jni.* -- compiling, merging and testing clean while the IOManager
   silently never engaged. Also drops jni.enable_file_reader_async, retired
   upstream along with the scanner's table.copy(buildTableOptions(..)); the
   equivalent knob is now paimon.table-option.file-reader-async-threshold.

Tests: new PaimonTableOptionsTest (11, mirroring upstream's
AbstractPaimonPropertiesTest table-option cases plus the connector-specific
wirings and two real local FileSystemCatalog getTable cases);
PaimonScanPlanProviderTest key names updated and the file-reader-async
forwarding test inverted into a retirement guard. Module 393/393 (1 pre-existing
live-connectivity skip), checkstyle 0, full FE install BUILD SUCCESS. Five
mutations verified RED: reverting a key to doris.*, dropping the
validateProperties extract, returning getTable unchanged, dropping the catalog
Options exclusion, and comparing raw keys in forCopy.

e2e (not run here): #66065's new suites assert exception "PaimonExternalCatalog",
which is the legacy UnboundTableSinkCreator message. On this branch paimon is a
PluginDrivenExternalCatalog, so the rejection comes from the connector's write
capabilities instead; test_paimon_write_boundary and
test_paimon_ctas_atomicity_negative updated accordingly. The UPDATE/DELETE/MERGE
assertions in the same suite are unchanged -- paimon declares no DELETE/MERGE,
so they still fall back to the legacy messages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
morningman added a commit that referenced this pull request Jul 27, 2026
…rename to the connector

Rebase branch-catalog-spi onto upstream-apache/master 042e613 (6 upstream
commits, 63 local commits replayed). Only #65955 "Support Paimon table option
passthrough" touched fe-core external tables; its 5 fe-core files were deleted
or stripped by P5/T29, producing one content conflict (union: upstream's
initNormalizeAndCheckProps vs our initHdfsExecutionAuthenticator), one
keep-ours conflict (T29 makes fe-core paimon-SDK-free), and four modify/delete
conflicts resolved by keeping the deletion.

Two capabilities had to move into fe-connector-paimon:

1. paimon.table-option.* passthrough. New PaimonTableOptions ports the
   AbstractPaimonProperties members T29 deleted (TABLE_OPTION_PREFIX,
   extractTableOptions, validateTableOption, getTableOptionsForCopy,
   SupportedTableOptions). Wired at the three points the fe-core class covered:
   excluded from the catalog Options passthrough (PaimonCatalogFactory),
   fail-fast at CREATE/ALTER CATALOG (PaimonConnectorProvider.validateProperties,
   since the SPI path never runs initNormalizeAndCheckProps), and overlaid on
   table load (CatalogBackedPaimonCatalogOps.getTable -- the connector's only
   Catalog.getTable, matching where PaimonExternalCatalog.getPaimonTable applied
   it, so branch/time-travel/system tables and the serialized table shipped to
   BE all inherit it).

   Upstream's matching notifyPropertiesUpdated change needs no port:
   PluginDrivenExternalCatalog resets the whole connector on any property
   change, and the PaimonExternalMetaCache it invalidated is already deleted.

2. JNI IOManager namespace rename, which was a SILENT break. #65955 moved
   paimon.doris.* -> paimon.jni.* on both sides at once; the BE files
   (paimon_jni_reader.cpp x2) and PaimonJniScanner auto-merged to the new names
   while the FE mirror list lives in PaimonScanPlanProvider (legacy
   PaimonScanNode being deleted), so FE kept emitting doris.* against a BE that
   only reads jni.* -- compiling, merging and testing clean while the IOManager
   silently never engaged. Also drops jni.enable_file_reader_async, retired
   upstream along with the scanner's table.copy(buildTableOptions(..)); the
   equivalent knob is now paimon.table-option.file-reader-async-threshold.

Tests: new PaimonTableOptionsTest (11, mirroring upstream's
AbstractPaimonPropertiesTest table-option cases plus the connector-specific
wirings and two real local FileSystemCatalog getTable cases);
PaimonScanPlanProviderTest key names updated and the file-reader-async
forwarding test inverted into a retirement guard. Module 393/393 (1 pre-existing
live-connectivity skip), checkstyle 0, full FE install BUILD SUCCESS. Five
mutations verified RED: reverting a key to doris.*, dropping the
validateProperties extract, returning getTable unchanged, dropping the catalog
Options exclusion, and comparing raw keys in forCopy.

e2e (not run here): #66065's new suites assert exception "PaimonExternalCatalog",
which is the legacy UnboundTableSinkCreator message. On this branch paimon is a
PluginDrivenExternalCatalog, so the rejection comes from the connector's write
capabilities instead; test_paimon_write_boundary and
test_paimon_ctas_atomicity_negative updated accordingly. The UPDATE/DELETE/MERGE
assertions in the same suite are unchanged -- paimon declares no DELETE/MERGE,
so they still fall back to the legacy messages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
morningman added a commit that referenced this pull request Jul 28, 2026
…rename to the connector

Rebase branch-catalog-spi onto upstream-apache/master 042e613 (6 upstream
commits, 63 local commits replayed). Only #65955 "Support Paimon table option
passthrough" touched fe-core external tables; its 5 fe-core files were deleted
or stripped by P5/T29, producing one content conflict (union: upstream's
initNormalizeAndCheckProps vs our initHdfsExecutionAuthenticator), one
keep-ours conflict (T29 makes fe-core paimon-SDK-free), and four modify/delete
conflicts resolved by keeping the deletion.

Two capabilities had to move into fe-connector-paimon:

1. paimon.table-option.* passthrough. New PaimonTableOptions ports the
   AbstractPaimonProperties members T29 deleted (TABLE_OPTION_PREFIX,
   extractTableOptions, validateTableOption, getTableOptionsForCopy,
   SupportedTableOptions). Wired at the three points the fe-core class covered:
   excluded from the catalog Options passthrough (PaimonCatalogFactory),
   fail-fast at CREATE/ALTER CATALOG (PaimonConnectorProvider.validateProperties,
   since the SPI path never runs initNormalizeAndCheckProps), and overlaid on
   table load (CatalogBackedPaimonCatalogOps.getTable -- the connector's only
   Catalog.getTable, matching where PaimonExternalCatalog.getPaimonTable applied
   it, so branch/time-travel/system tables and the serialized table shipped to
   BE all inherit it).

   Upstream's matching notifyPropertiesUpdated change needs no port:
   PluginDrivenExternalCatalog resets the whole connector on any property
   change, and the PaimonExternalMetaCache it invalidated is already deleted.

2. JNI IOManager namespace rename, which was a SILENT break. #65955 moved
   paimon.doris.* -> paimon.jni.* on both sides at once; the BE files
   (paimon_jni_reader.cpp x2) and PaimonJniScanner auto-merged to the new names
   while the FE mirror list lives in PaimonScanPlanProvider (legacy
   PaimonScanNode being deleted), so FE kept emitting doris.* against a BE that
   only reads jni.* -- compiling, merging and testing clean while the IOManager
   silently never engaged. Also drops jni.enable_file_reader_async, retired
   upstream along with the scanner's table.copy(buildTableOptions(..)); the
   equivalent knob is now paimon.table-option.file-reader-async-threshold.

Tests: new PaimonTableOptionsTest (11, mirroring upstream's
AbstractPaimonPropertiesTest table-option cases plus the connector-specific
wirings and two real local FileSystemCatalog getTable cases);
PaimonScanPlanProviderTest key names updated and the file-reader-async
forwarding test inverted into a retirement guard. Module 393/393 (1 pre-existing
live-connectivity skip), checkstyle 0, full FE install BUILD SUCCESS. Five
mutations verified RED: reverting a key to doris.*, dropping the
validateProperties extract, returning getTable unchanged, dropping the catalog
Options exclusion, and comparing raw keys in forCopy.

e2e (not run here): #66065's new suites assert exception "PaimonExternalCatalog",
which is the legacy UnboundTableSinkCreator message. On this branch paimon is a
PluginDrivenExternalCatalog, so the rejection comes from the connector's write
capabilities instead; test_paimon_write_boundary and
test_paimon_ctas_atomicity_negative updated accordingly. The UPDATE/DELETE/MERGE
assertions in the same suite are unchanged -- paimon declares no DELETE/MERGE,
so they still fall back to the legacy messages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
morningman added a commit that referenced this pull request Jul 28, 2026
…rename to the connector

Rebase branch-catalog-spi onto upstream-apache/master 042e613 (6 upstream
commits, 63 local commits replayed). Only #65955 "Support Paimon table option
passthrough" touched fe-core external tables; its 5 fe-core files were deleted
or stripped by P5/T29, producing one content conflict (union: upstream's
initNormalizeAndCheckProps vs our initHdfsExecutionAuthenticator), one
keep-ours conflict (T29 makes fe-core paimon-SDK-free), and four modify/delete
conflicts resolved by keeping the deletion.

Two capabilities had to move into fe-connector-paimon:

1. paimon.table-option.* passthrough. New PaimonTableOptions ports the
   AbstractPaimonProperties members T29 deleted (TABLE_OPTION_PREFIX,
   extractTableOptions, validateTableOption, getTableOptionsForCopy,
   SupportedTableOptions). Wired at the three points the fe-core class covered:
   excluded from the catalog Options passthrough (PaimonCatalogFactory),
   fail-fast at CREATE/ALTER CATALOG (PaimonConnectorProvider.validateProperties,
   since the SPI path never runs initNormalizeAndCheckProps), and overlaid on
   table load (CatalogBackedPaimonCatalogOps.getTable -- the connector's only
   Catalog.getTable, matching where PaimonExternalCatalog.getPaimonTable applied
   it, so branch/time-travel/system tables and the serialized table shipped to
   BE all inherit it).

   Upstream's matching notifyPropertiesUpdated change needs no port:
   PluginDrivenExternalCatalog resets the whole connector on any property
   change, and the PaimonExternalMetaCache it invalidated is already deleted.

2. JNI IOManager namespace rename, which was a SILENT break. #65955 moved
   paimon.doris.* -> paimon.jni.* on both sides at once; the BE files
   (paimon_jni_reader.cpp x2) and PaimonJniScanner auto-merged to the new names
   while the FE mirror list lives in PaimonScanPlanProvider (legacy
   PaimonScanNode being deleted), so FE kept emitting doris.* against a BE that
   only reads jni.* -- compiling, merging and testing clean while the IOManager
   silently never engaged. Also drops jni.enable_file_reader_async, retired
   upstream along with the scanner's table.copy(buildTableOptions(..)); the
   equivalent knob is now paimon.table-option.file-reader-async-threshold.

Tests: new PaimonTableOptionsTest (11, mirroring upstream's
AbstractPaimonPropertiesTest table-option cases plus the connector-specific
wirings and two real local FileSystemCatalog getTable cases);
PaimonScanPlanProviderTest key names updated and the file-reader-async
forwarding test inverted into a retirement guard. Module 393/393 (1 pre-existing
live-connectivity skip), checkstyle 0, full FE install BUILD SUCCESS. Five
mutations verified RED: reverting a key to doris.*, dropping the
validateProperties extract, returning getTable unchanged, dropping the catalog
Options exclusion, and comparing raw keys in forCopy.

e2e (not run here): #66065's new suites assert exception "PaimonExternalCatalog",
which is the legacy UnboundTableSinkCreator message. On this branch paimon is a
PluginDrivenExternalCatalog, so the rejection comes from the connector's write
capabilities instead; test_paimon_write_boundary and
test_paimon_ctas_atomicity_negative updated accordingly. The UPDATE/DELETE/MERGE
assertions in the same suite are unchanged -- paimon declares no DELETE/MERGE,
so they still fall back to the legacy messages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.2.x reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants