Derive a native UDF's return type from the library instead of requiring the caller to declare it

[andygrove]

Problem

CometNativeUDF.register asks the caller for the full signature:

CometNativeUDF.register(spark, "add_one_c", libPath, Seq(LongType), LongType)

but the library already knows most of it. A kernel's return_field computes the output type from the argument types, and Comet calls it at planning time to check the declaration, so the declared returnType is verified against the library rather than being the source of truth. Raised by @wForget on #4459: can inputTypes / returnType / deterministic come from the library instead?

Where each one stands

returnType is the strongest candidate, and it is already redundant in the sense that a wrong value is rejected. The obstacle is ordering, not information: Spark's analyzer needs a concrete DataType when register installs the catalog stub, and at that moment the argument types of the eventual call sites are not known. Deriving it would mean calling the library on the driver with the inputTypes the caller supplied, which removes one of the two declarations but not both.

inputTypes cannot be derived as things stand. The ABI has no way to enumerate a kernel's accepted signatures: return_field is a predicate over argument types, not a description of them, and a kernel like echo_c accepts every type. Getting this from the library would need a new ABI entry point returning declared signatures, which is an ABI break and worth its own design.

deterministic is per-registration today and always has to be true (#5249). It is genuinely a property of the kernel rather than the registration, so if anything it belongs on the library side, but that is blocked behind honoring it at all.

Possible shape

An overload that takes only the arguments and derives the return type:

CometNativeUDF.register(spark, "add_one_c", libPath, Seq(LongType))

calling the library on the driver to resolve the return type, and failing with the kernel's own message if the kernel rejects those argument types. The existing overload stays for callers who want the declaration checked.

That leaves the plan-time check in place. It is still worth keeping even when the type was derived, because the derivation happens on the driver and the check runs on the executor against the library actually loaded there, which need not be the same file.

Related

• feat: custom Rust UDFs via arrow-ffi [experimental] #4459, where this was raised
• Honor the deterministic flag for Rust UDFs #5249, honoring deterministic
• Tighten CometUDF API with input/return type validation at registration #4173, the equivalent validation gap on the JVM CometUDF path