What
When the native Iceberg write path (#5361) fails partway through a task, data files already finalized by that task attempt are left in the table's data location. They are invisible to readers (files resolve only through committed manifests) and are reclaimed by remove_orphan_files, but iceberg-java cleans them up synchronously: SparkWrite's writers run inside Utils.tryWithSafeFinallyAndFailureCallbacks, and DataWriter.abort() calls SparkCleanupUtil.deleteTaskFiles(io, ...). On spot-heavy or preemption-prone clusters the orphan cost accumulates silently on the native path.
This gap is documented in the "Failure handling" section of iceberg-writes.md; this issue tracks closing it.
Suggested shape
TaskContext.addTaskFailureListener in CometIcebergWriteExec.doExecute, mirroring what iceberg-java's writer abort does:
- For failures after the native writer has returned (manifest decode, JVM metrics rebuild,
TaskCommit construction), the JVM already holds the decoded DataFiles — delete their paths through the table FileIO that is already in the task closure.
- For failures inside the native write itself, the JVM never learns the written paths. Closing that half needs the native operator to report the paths of finalized files alongside the error (or a JNI call to fetch them from the failed writer), since iceberg-rust's writers have no abort/Drop cleanup of their own.
Cleanup must be best-effort and must not mask the original task failure.
Context
What
When the native Iceberg write path (#5361) fails partway through a task, data files already finalized by that task attempt are left in the table's data location. They are invisible to readers (files resolve only through committed manifests) and are reclaimed by
remove_orphan_files, but iceberg-java cleans them up synchronously:SparkWrite's writers run insideUtils.tryWithSafeFinallyAndFailureCallbacks, andDataWriter.abort()callsSparkCleanupUtil.deleteTaskFiles(io, ...). On spot-heavy or preemption-prone clusters the orphan cost accumulates silently on the native path.This gap is documented in the "Failure handling" section of
iceberg-writes.md; this issue tracks closing it.Suggested shape
TaskContext.addTaskFailureListenerinCometIcebergWriteExec.doExecute, mirroring what iceberg-java's writer abort does:TaskCommitconstruction), the JVM already holds the decodedDataFiles — delete their paths through the tableFileIOthat is already in the task closure.Cleanup must be best-effort and must not mask the original task failure.
Context