Describe the bug
CometScanRule.validateIcebergFileScanTasks checks Iceberg file schemes against a hardcoded set (file, s3, s3a, gs, gcs, oss, abfss, abfs, wasbs, wasb at CometScanRule.scala:1082), and its fallback message repeats that list to users. But the Iceberg execution path resolves storage through its own factory (iceberg_scan.rs storage_factory_for), which supports only file, s3, s3a, gs, oss via OpenDAL. And the JNI probe every other scheme gate delegates to (isNativelyReadableScheme → object_store::ObjectStoreScheme::parse) recognizes a third, different set.
Net effect: an Iceberg table under gcs, abfs, abfss, wasb, or wasbs passes validation, gets claimed, and every task dies with CometNativeException: Unsupported storage scheme: ... — while stock Spark reads the same table fine. Meanwhile oss actually works at execution despite the JNI probe saying it's unsupported, so the three lists disagree in both directions.
Steps to reproduce
No cloud account needed — back the scheme with local disk the way ParquetReadFromFakeHadoopFsSuite does:
// register a local-backed filesystem for the wasb scheme
spark.conf.set("spark.hadoop.fs.wasb.impl", "org.apache.hadoop.fs.RawLocalFileSystem")
// hadoop catalog with a wasb warehouse
// spark.sql.catalog.hadoop_catalog = org.apache.iceberg.spark.SparkCatalog
// spark.sql.catalog.hadoop_catalog.type = hadoop
// spark.sql.catalog.hadoop_catalog.warehouse = wasb://fake-container/tmp/warehouse
sql("CREATE TABLE hadoop_catalog.t (id INT, s STRING) USING iceberg")
sql("INSERT INTO hadoop_catalog.t VALUES (1,'a'),(2,'b'),(3,'c')")
sql("SELECT * FROM hadoop_catalog.t").collect()
With spark.comet.enabled=false: [1,a],[2,b],[3,c].
With Comet + spark.comet.scan.icebergNative.enabled=true (plan shows CometIcebergNativeScan ... wasb://fake-container/...): every task fails with
org.apache.comet.CometNativeException: Unsupported storage scheme: wasb
(Ensure spark.comet.libhdfs.schemes doesn't include wasb, since that conf routes schemes elsewhere.)
Expected behavior
Schemes the Iceberg native path can't execute should be declined at plan time so the scan falls back to Spark, and the fallback message should list what's actually supported. One source of truth: the validator should ask the same layer that executes — either expose the Iceberg storage factory's supported set through the existing JNI probe pattern, or derive the check from it directly — rather than maintaining a third hand-written list.
Additional context
The claimed-but-fails set on this path is {gcs, abfs, abfss, wasb, wasbs}; oss works at execution but is rejected by the generic JNI probe, so aligning the lists fixes both directions. Same drift class as the scheme/config gates reworked in #5365.
Describe the bug
CometScanRule.validateIcebergFileScanTasks checks Iceberg file schemes against a hardcoded set (file, s3, s3a, gs, gcs, oss, abfss, abfs, wasbs, wasb at CometScanRule.scala:1082), and its fallback message repeats that list to users. But the Iceberg execution path resolves storage through its own factory (iceberg_scan.rs storage_factory_for), which supports only file, s3, s3a, gs, oss via OpenDAL. And the JNI probe every other scheme gate delegates to (isNativelyReadableScheme → object_store::ObjectStoreScheme::parse) recognizes a third, different set.
Net effect: an Iceberg table under gcs, abfs, abfss, wasb, or wasbs passes validation, gets claimed, and every task dies with CometNativeException: Unsupported storage scheme: ... — while stock Spark reads the same table fine. Meanwhile oss actually works at execution despite the JNI probe saying it's unsupported, so the three lists disagree in both directions.
Steps to reproduce
No cloud account needed — back the scheme with local disk the way ParquetReadFromFakeHadoopFsSuite does:
// register a local-backed filesystem for the wasb scheme
spark.conf.set("spark.hadoop.fs.wasb.impl", "org.apache.hadoop.fs.RawLocalFileSystem")
// hadoop catalog with a wasb warehouse
// spark.sql.catalog.hadoop_catalog = org.apache.iceberg.spark.SparkCatalog
// spark.sql.catalog.hadoop_catalog.type = hadoop
// spark.sql.catalog.hadoop_catalog.warehouse = wasb://fake-container/tmp/warehouse
sql("CREATE TABLE hadoop_catalog.t (id INT, s STRING) USING iceberg")
sql("INSERT INTO hadoop_catalog.t VALUES (1,'a'),(2,'b'),(3,'c')")
sql("SELECT * FROM hadoop_catalog.t").collect()
With spark.comet.enabled=false: [1,a],[2,b],[3,c].
With Comet + spark.comet.scan.icebergNative.enabled=true (plan shows CometIcebergNativeScan ... wasb://fake-container/...): every task fails with
org.apache.comet.CometNativeException: Unsupported storage scheme: wasb
(Ensure spark.comet.libhdfs.schemes doesn't include wasb, since that conf routes schemes elsewhere.)
Expected behavior
Schemes the Iceberg native path can't execute should be declined at plan time so the scan falls back to Spark, and the fallback message should list what's actually supported. One source of truth: the validator should ask the same layer that executes — either expose the Iceberg storage factory's supported set through the existing JNI probe pattern, or derive the check from it directly — rather than maintaining a third hand-written list.
Additional context
The claimed-but-fails set on this path is {gcs, abfs, abfss, wasb, wasbs}; oss works at execution but is rejected by the generic JNI probe, so aligning the lists fixes both directions. Same drift class as the scheme/config gates reworked in #5365.