ci: share one Linux native build across CI workflows

[sunchao]

Which issue does this PR close?

Part of #5830. Follow-up to #3249, which shared native artifacts within individual workflows.

Rationale for this change

The umbrella CI workflow compiles the same default Linux native library separately for Linux, each selected Spark version, and each selected Iceberg version. A source-change PR normally builds three copies; a full merge-group or manual run builds nine. Build one copy and share it with those consumers to reduce duplicate compilation and runner usage. Actual time savings depend on cache state and still need measurement in CI.

What changes are included in this PR?

• Add one reusable Linux native producer using the existing Cargo ci profile, JDK 17, compiler flags, and main-only cache save policy. Linux JVM/TPC, Spark SQL, and Iceberg consumers depend on it and receive its artifact through a required input.
• Run independent Linux lint, compile-only checks, Celeborn compatibility tests, and Rust debug tests alongside the native producer in a separate workflow. Preserve all six moved jobs, including Maven bootstrap in the three direct Maven callers; both TPC jobs also retain their bootstrap.
• Derive build_linux_native in compute-changes.py after path/event/label selection. NATIVE_CONSUMERS maps nine caller jobs to ten selecting outputs: Spark 4.1's core and Hive outputs feed the same caller. A Hive-only label event starts the native producer even when the core output is false. Preserve the core/Hive module selection introduced by ci: move the Spark 4.1 sql_hive shards behind the merge queue #5871 and the existing queue policy.
• Remove duplicate native compilation and uploads from consumers. Keep Spark/JDK-specific compiled JVM artifacts separate. The Spark JVM producer uses actions/setup-java@v4 without Rust setup; its container remains to preserve workspace and Maven-cache paths for downstream Spark shards.
• Check exact caller IDs, single/OR selection gates, output exports, producer dependencies, artifact mapping, duplicate producers, and required-check coverage. Add a guard requiring an earlier unconditional Maven bootstrap with failures propagated for direct ./mvnw commands in both Linux workflows.
• Preserve artifact transfer retries, main-only cache writes, checkout guards, Rust formatting prerequisites, and independent macOS/debug/feature builds. Limit token permissions in both new reusable workflows and their callers to contents: read. Artifact retention remains one day.

How are these changes tested?

Rebased onto main at 451c99963206fa6bf0387239aa12887a16255516. Independent comparison against that base verified all six moved Linux jobs, both remaining TPC bootstrap steps, Spark module selection, queue policy, artifact retry behavior, and existing required-check dependencies.

Passed locally on this revision:

• python3 dev/ci/check-ci-config.py, including routing and Spark module-matrix checks.
• python3 dev/ci/test-ci-config.py (36 tests): mutations cover missing, late, conditional, and failure-ignoring Maven bootstrap steps, plus omitted/altered Hive gates and exports, missing consumers/dependencies, and shared-artifact wiring.
• python3 dev/ci/test-native-build-selection.py (21 tests, including 299,008 path/event combinations): real compute and CLI coverage for Hive-only labels, core-only PRs, core+Hive selection, merge groups, main pushes, other opt-ins, and empty-input manual dispatch.
• Actionlint, suite inventory, benchmark runner (49 suites), Iceberg shard tests (15), PR type-label tests (4), Markdown formatting, and git diff --check.

Hosted CI is running on head 79c161b293f2147b1d25233a47a15222d7289cf9. The regular PR run is in progress. Applied run-spark-3.4-tests, run-spark-3.5-tests, run-spark-4.0-tests, and run-iceberg-tests after pushing the rebased head, starting four additional opt-in runs (1, 2, 3, 4). Their results are pending; the local checks do not establish runtime coverage for those six additional consumers.

No full native or Spark build was run locally. Apache RAT is left to hosted preflight because the local Maven cache is read-only and Maven Central was unavailable in the earlier local attempt. Failed-job reruns can reuse the successful producer's artifact while retained; after expiry, rerun the full workflow. That rerun scenario remains unverified.

[rich7420]

LGTM

[andygrove]

LGTM but I would like to merge #5842 first so we can unblock setting up the merge queue to reduce CI resource usage. I reviewed this PR assuming that it will go in after #5842.

Once #5842 is in, two things will need to happen on the rebase here:

1. ci.yml merges cleanly between the two PRs, but #5842 adds a required_checks aggregator whose needs: must list every ci.yml job, and a preflight guard in check-ci-config.py that enforces it. The merged file fails that guard until pr_build_linux_checks and build_linux_native are added to required_checks.needs. The guard message names them, so the rebase will tell you. build_linux_native in particular has to be there and cannot be exempted: if the producer fails, GitHub marks all nine consumers skipped, and the aggregator treats skipped as pass, so without the producer in needs a broken native build would produce a green Required Checks.

2. #5842 hardens Lint Scala (syntactic) against Maven Central connection resets by splitting it into a retried cs launch scalafix:0.14.6 -- --version warm-up followed by the real check under cs launch --mode offline. Since this PR moves that job into pr_build_linux_checks.yml, that split needs to come along. The iceberg_spark_test_reusable.yml change in #5842 (routing the shard-inventory upload through upload-artifact-retry) merges cleanly onto your prepare-shards version.

After both land, check-ci-config.py will carry two hand-rolled YAML readers (block_mapping here and the line regexes in #5842), both justified by PyYAML not being on the preflight runner. I will open a follow-up issue to install PyYAML in preflight and collapse them rather than leave both in place.

I verified the two-way compatibility locally: your checker passes against a ci.yml that includes the required_checks job, and the #5842 checker passes against your ci.yml once the two ids are added.

[andygrove]

One more heads-up from the merge-queue side, following on from my review above.

#5843 (the PR that actually enables the queue, stacked on #5842) changes POLICY in dev/ci/compute-changes.py so that a push to main selects only docs. Every test job moves to the merge_group event instead, because the queue has already tested the exact tree that lands.

That will break test_main_and_manual_runs_include_legacy_consumers in dev/ci/test-native-build-selection.py here, which asserts event="push" selects every native consumer. Whichever of #5841 and #5843 lands second needs to update that test (the workflow_dispatch half stays true). If this PR lands first, #5843 will pick it up on its rebase along with adding pr_build_linux_checks and build_linux_native to required_checks.needs. If #5843 lands first, the push assertion here should become merge_group.

Also FYI, run-spark-3.4-tests and run-spark-3.5-tests now exist as labels, so the label-gated selection tests here have real labels behind them.

[andygrove]

Three things, all about the rebase rather than the design. The producer and consumer wiring itself looks right to me, and I checked that build_linux_native is the exact union of the nine consumer outputs on this head.

#5852 landed on main after this branch was last pushed, adding a Bootstrap Maven step calling ./.github/actions/maven-bootstrap to five jobs in pr_build_linux.yml. Three of those five are the jobs this PR moves into pr_build_linux_checks.yml, namely lint-java, build-spark-4-1 and celeborn-reflection-compatibility. A rebase cannot carry that step into a file this branch creates from scratch, and pr_build_linux.yml conflicts, so the natural resolution is to take the deletion and lose the retry for those three. The two TPC jobs stay behind and keep theirs. Nothing in check-ci-config.py verifies that a job calling a bare ./mvnw has the bootstrap step ahead of it, so this would be silent until the next Maven Central blip takes out a queue-gating job. Could you add it back to those three after the rebase, and update the ROUTING_CASES comment that says the composite is called only from pr_build_linux.yml? Given how easily it drops out in a file move, is it worth a guard for it?

NATIVE_CONSUMERS is a flat list of output keys, one per consumer job, and #5871 has already broken that assumption on main. It added spark_4_1_hive as a second output feeding the same spark_4_1 job, which is now gated on needs.changes.outputs.spark_4_1 == 'true' || needs.changes.outputs.spark_4_1_hive == 'true'. On a labeled run with run-spark-4.1-hive-tests, spark_4_1 is false and spark_4_1_hive is true, so the union computes build_linux_native as false, the producer is skipped, and spark_4_1 is then skipped through its needs even though its if is true. The label would quietly do nothing and Required Checks would still go green. compute-changes.py is the one file that auto-merges cleanly here, so the union will miss the new key silently. native_selection_failures will fail preflight on the if: string mismatch, but the tempting fix there is to relax the comparison, which leaves the hole. Would it make sense to let a consumer map to a set of output keys rather than one?

Last one is coverage. The hosted run only selected pr_build_linux, spark_4_1 and iceberg_1_11, so six of the nine rewired consumers have not run against the shared artifact. spark_3_4 is the one I would most want to see, because its build job in spark_sql_test_reusable.yml is the only consumer whose toolchain setup actually changed, swapping ./.github/actions/setup-builder for a bare actions/setup-java@v4 at JDK 11, and it now consumes a libcomet.so linked against JDK 17. The linux-test Spark 3.4 lane already proves the library loads under JDK 11, but the new setup path has no coverage. These are queue-tier jobs, so a break blocks every merge rather than one PR. Could you apply run-spark-3.4-tests, run-spark-3.5-tests, run-spark-4.0-tests and run-iceberg-tests once on the rebased head?

[sunchao]

@andygrove, addressed the three items from your latest review and rebased onto main at 451c999632; the PR is now at 79c161b293 with no merge conflicts.

1. Maven bootstrap: preserved the bootstrap in all three moved jobs and both TPC jobs, and corrected the routing comment. The new guard requires an earlier unconditional bootstrap with failures propagated before direct ./mvnw commands in both Linux workflows. Regression mutations remove, move, condition, or ignore the bootstrap's failure in each of the five jobs.
2. Hive routing: NATIVE_CONSUMERS now maps caller IDs to all selecting outputs. spark_4_1 includes both core and Hive, so a Hive-only label starts the native producer. Exact caller, OR-gate, export, and dependency validation remains enforced. Compute and CLI tests cover core-only, Hive-only, both, queue, and manual runs; upstream's module selection is preserved.
3. Coverage: applied all four requested labels after pushing this head: run-spark-3.4-tests, run-spark-3.5-tests, run-spark-4.0-tests, and run-iceberg-tests. The regular PR run and four additional opt-in runs (1, 2, 3, 4) are pending. This requests coverage of all nine rewired consumers, including the Spark 3.4/JDK 11 setup path; runtime results are not yet confirmed.

Local validation passed: 36 configuration regression tests, 21 native-selection tests (299,008 combinations), actionlint, suite/benchmark inventory checks, 15 Iceberg shard tests, 4 PR-label tests, Markdown formatting, and whitespace checks. The PR description now reflects the implementation and pending hosted coverage.

[andygrove]

Happy to approve once the following issues are resolved.

Thanks for the quick turnaround on the last round. All three items are addressed: the Maven bootstrap is back in the three moved jobs and guarded, NATIVE_CONSUMERS maps callers to tuples so the Hive-only label starts the producer, and all four label runs came back green, so every one of the nine consumers has now run against the shared artifact, including the Spark 3.4 setup-java-only build path. I also ran the three Python checks locally on 79c161b and they pass in about five seconds total, so the preflight cost is fine.

1. Rebase hazard from #5897. chore: drop support for JDK 11 landed on main after this head was pushed. It edits the lint-java matrix in pr_build_linux.yml (the Spark 3.4 entry becomes JDK 17 and JAVA_TOOL_OPTIONS drops the version conditional), and flips java: 11 to 17 for spark_3_4 and iceberg_1_8 in ci.yml and for preflight. The ci.yml and linux-test hunks should auto-merge, but the lint-java hunk conflicts with this PR's deletion of that block, and pr_build_linux_checks.yml is a new file, so the natural resolution drops the change and the new file keeps JDK 11. Same shape as the bootstrap step last time. It fails loudly this time because the new Maven enforcer rejects JDK 11, so no guard is needed, but could you carry those edits into pr_build_linux_checks.yml on the rebase?

2. Contributor doc pointer. docs/source/contributor-guide/adding_a_new_spark_version.md (around line 112) tells authors to add the compile-only job to pr_build_linux.yml, but build-spark-4-1 now lives in pr_build_linux_checks.yml. Could you update the file name there? The suite-matrix instructions in development.md and check-suites.py still point at pr_build_linux.yml correctly since linux-test stays put.

Nothing else from me. The wiring, the main-only cache write on the single producer, artifact retention, the permissions blocks, and the README rewrite all look right.