HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning by difin · Pull Request #6789 · apache/hive

difin · 2026-09-15T20:09:28Z

What changes were proposed in this pull request?

Hive wires up Iceberg REST server-side scan planning: when scan-planning-mode=server, split planning can go through the REST catalog (POST /plan) instead of only the serialized table snapshot on Tez/MR workers. Adds a small RestCatalogScanPlanning helper for config and copying catalog settings into job conf, and updates the Iceberg storage handler / input path to reload a live REST table on executors where appropriate. Tests cover configuration, job propagation, and embedded REST planning behavior.

Why are the changes needed?

Iceberg REST catalogs can plan scans on the catalog service instead of on every client. That keeps planning close to authoritative metadata and catalog credentials, can reduce work on HiveServer2 and task containers, and lets the REST service apply catalog-side optimizations as Iceberg evolves.

Hive today mostly plans reads from a serialized table snapshot shipped in the job configuration, which fits local/Hadoop catalogs but does not use the REST server’s planning API. For deployments that already run a REST catalog with scan planning, operators expect scan-planning-mode=server to work for Hive queries on Tez/MR - not only for single-node Iceberg clients. This change closes that gap so Hive participates in the same server-side planning model as the rest of the Iceberg REST ecosystem.

Does this PR introduce any user-facing change?

Yes. For REST Iceberg catalogs, operators can set iceberg.catalog.<catalog>.scan-planning-mode=server (via hive-site.xml or session SET) alongside existing REST catalog properties. When enabled, read jobs that use the Iceberg input format can use server-side scan planning instead of local planning from the serialized snapshot. Default behavior is unchanged when the property is unset or not server. Hive and non-REST catalogs are unaffected.

How was this patch tested?

TestRestCatalogScanPlanning — Tests config keys, server-mode detection, job propagation gating and copying.
TestRestCatalogScanPlanningServerIT — Uses embedded REST catalog; Verifies that RESTTable / RESTTableScan and PlanTableScanRequest on planTasks() are used when file planning is on.
TestHiveIcebergServerSideScanPlanning — Checks that Hive only reloads a table from the REST catalog for split planning when server mode is on and it is safe to do so - not when planning stays local or when the job must use uncommitted metadata from the same transaction.
TestHiveIcebergServerSideScanPlanningServerIT — Tests that executor job conf with/without propagated catalog props and table reload over REST.

Aggarwal-Raghav · 2026-09-16T09:19:51Z

There were already discussions happening under HIVE-29599 for server side planning?

difin · 2026-09-16T13:50:53Z

There were already discussions happening under HIVE-29599 for server side planning?

This is different - this PR is about adding client-side support.

tanishq-chugh

Hi @difin
Checked out the PR, great improvement 🚀

tanishq-chugh · 2026-09-17T07:35:23Z

+   */
+  public static Table resolveTableForScanPlanning(Configuration conf, String tableIdentifier) {
+    if (shouldReloadForServerSideScanPlanning(conf)) {
+      Table table = Catalogs.loadTable(conf);


I believe this will create a new RestCatalog Object/HttpClient on every call, curious if this can cause any connection leak?

Yes - Catalogs.loadTable creates a new REST catalog (and HTTP client) on each call, and we don’t close it on this path today. That’s the same pattern as other Catalogs.loadTable usages in the handler. Here it runs once per getSplits during split planning, not per input split. A reload only happens when shouldReloadForServerSideScanPlanning is true.

deniskuzZ · 2026-10-02T10:13:05Z

+    return HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_ICEBERG_REST_SCAN_PLANNING_MODE);
+  }
+
+  public static void setHiveRestScanPlanningMode(Configuration conf, String mode) {


deniskuzZ · 2026-10-02T10:14:50Z

+  /**
+   * Returns true when the catalog is configured for server-side scan planning and the Hive feature flag is on.
+   */
+  public static boolean requestsServerSidePlanning(String catalogName, Configuration conf) {


requestsServerSideScanPlanning ? idk or simplify other namings

renamed to isServerSidePlanningEnabled - meaning both catalog properties include server mode and Hive feature flag is enabled.

deniskuzZ · 2026-10-02T10:15:23Z

+  }
+
+  /**
+   * Returns true when catalog properties should be copied into the Tez/MR job configuration so


is there still MR job?

Thanks for catching the typo, fixed.

deniskuzZ · 2026-10-02T10:16:36Z

+        IcebergCatalogProperties.getCatalogType(conf, resolvedCatalogName))) {
+      return false;
+    }
+    return requestsServerSidePlanning(resolvedCatalogName, conf);


weird. method name is validate (shouldPropagateCatalogPropertiesToJob) but it calls requestsServerSidePlanning? could we improve/refactor?

Renamed both the call site's method and the method it calls.

deniskuzZ · 2026-10-02T10:18:13Z

+      return;
+    }
+
+    if (!shouldPropagateCatalogPropertiesToJob(catalogName, sessionConf)) {


should we call shouldPropagateCatalogPropertiesToJob before propagateCatalogPropertiesToJob and short circuit?

deniskuzZ · 2026-10-02T10:20:10Z

+   * Called from {@code configureJobConf} so split generation sees catalog URI/type/scan-planning-mode
+   * even when job properties were not copied yet.
+   */
+  public static void propagateCatalogPropertiesToJob(


looks like duplicate. maybe use BinaryOperator to hadle both Map and jobConf, see boolean isV2TableOrAbove(BinaryOperator<String> props)

Addressed the duplication with a single private path and thin Map / JobConf overloads (BiConsumer, same idea as the two isV2TableOrAbove overloads). BinaryOperator doesn’t seem to fit here: in isV2TableOrAbove it’s get-with-default; here we need put-if-absent.

deniskuzZ · 2026-10-02T10:20:57Z

+      return;
+    }
+
+    if (!shouldPropagateCatalogPropertiesToJob(resolvedCatalogName, sessionConf)) {


again repeated check. why is it inside action

deniskuzZ · 2026-10-02T10:22:30Z

+        });
+  }
+
+  private interface PropertyConsumer {


why invent the wheel, can't we use BiConsumer?

deniskuzZ · 2026-10-02T10:25:58Z

+    if (tableIdentifier == null) {
+      conf.set(InputFormatConfig.TABLE_IDENTIFIER, executorTableId);
+    }
+    if (conf.get(InputFormatConfig.SERIALIZED_TABLE_PREFIX + executorTableId) == null) {


can we extract to local var not to repeat in set?

sonarqubecloud · 2026-10-07T18:56:31Z

Quality Gate passed

Issues
9 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.3% Duplication on New Code

See analysis details on SonarQube Cloud

asf-ci-hive added the tests pending label Sep 15, 2026

difin changed the title ~~File planning~~ HIVE-30056: Iceberg: Add REST catalog server-side scan planning support Sep 15, 2026

asf-ci-hive added tests failed and removed tests pending labels Sep 15, 2026

difin force-pushed the file_planning branch from 7e54757 to 769a954 Compare September 15, 2026 20:24

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Sep 16, 2026

difin force-pushed the file_planning branch from 769a954 to 3ef3d65 Compare September 16, 2026 13:44

asf-ci-hive added tests pending and removed tests failed labels Sep 16, 2026

difin changed the title ~~HIVE-30056: Iceberg: Add REST catalog server-side scan planning support~~ HIVE-30056: Iceberg: Add client side support for REST catalog server-side scan planning Sep 16, 2026

asf-ci-hive added tests failed and removed tests pending labels Sep 16, 2026

tanishq-chugh reviewed Sep 17, 2026

View reviewed changes

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Sep 18, 2026

difin force-pushed the file_planning branch from 8ef09d9 to 0db254a Compare September 22, 2026 22:56

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Sep 22, 2026

difin force-pushed the file_planning branch from 0db254a to e4ef113 Compare September 23, 2026 22:31

asf-ci-hive added tests pending and removed tests failed labels Sep 23, 2026

deniskuzZ reviewed Oct 2, 2026

View reviewed changes

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Oct 6, 2026

difin force-pushed the file_planning branch from 58c6341 to d8211de Compare October 7, 2026 01:46

asf-ci-hive added tests pending tests unstable and removed tests failed tests pending labels Oct 7, 2026

Review comments Oct 6.

078aabd

difin force-pushed the file_planning branch from d8211de to 078aabd Compare October 7, 2026 17:09

asf-ci-hive added tests pending and removed tests unstable labels Oct 7, 2026

asf-ci-hive added tests passed and removed tests pending labels Oct 7, 2026

Sonar issues

9a6a243

asf-ci-hive added tests pending tests failed and removed tests passed tests pending labels Oct 7, 2026

Conversation

difin commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Aggarwal-Raghav commented Sep 16, 2026

Uh oh!

difin commented Sep 16, 2026

Uh oh!

tanishq-chugh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Oct 7, 2026

Quality Gate passed

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

difin commented Sep 15, 2026 •

edited

Loading