Skip to content

Surface Parquet TIMESTAMP_MILLIS overflow as a Spark-faithful exception instead of a raw Arrow error #5517

Description

@peterxcli

Context

#5177 makes the Parquet reader's TIMESTAMP_MILLIS -> microseconds conversion checked: overflow now raises an error instead of silently wrapping (parquet_convert_array in native/core/src/parquet/parquet_support.rs uses try_unary + mul_checked).

The resulting error is a raw ArrowError::ComputeError("Overflow happened on: <v> * 1000"), which surfaces to users as a CometNativeException rather than what Spark throws on the same input.

What Spark throws

Spark's vectorized Parquet reader calls SparkDateTimeUtils.millisToMicros, which is Math.multiplyExact, so it throws an untyped java.lang.ArithmeticException("long overflow") — independent of ANSI mode:

Why the existing typed variant does not fit

Mapping this to the existing SparkError::ArithmeticOverflow would render:

[ARITHMETIC_OVERFLOW] ... If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.

That advice is wrong here: this overflow is ANSI-independent (the regression test added in #5177, ParquetReadSuite."TIMESTAMP_MILLIS overflow fails in native scan", asserts the error with ANSI both on and off).

Proposed work

Add a native error variant that converts to a plain ArithmeticException("long overflow") (or the closest Spark-faithful equivalent), with the corresponding ShimSparkErrorConverter mappings for each supported Spark version. This aligns with the error-fidelity work in #5169.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:scanParquet scan / data readingbugSomething isn't workingpriority:lowMinor issues, test failures, tooling, cosmetic

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions