NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records)

[maltesander]

Summary

NIFI-16069 - PutIcebergRecord fails with ClassCastException when writing complex types (arrays, maps, nested records)

PutIcebergRecord fails to write FlowFiles whose schema contains complex/nested types like Iceberg list, map, or struct columns. RecordConverter only translates top-level scalar values (java.sql timestamp/date/time -> java.time) and passes complex values through unchanged.
As a result, values reach Iceberg's Parquet writer in NiFi's native representation, which is incompatible with what Iceberg expects:

• Nested records arrive as org.apache.nifi.serialization.record.MapRecord but Iceberg requires org.apache.iceberg.StructLike.
• Array fields arrive as Object[] but Iceberg's writer requires a java.util.Collection.
• Maps and elements/values nested inside these types are likewise not converted (e.g. a date inside an array or map value).

Because conversion is gated on scalar field types only, records consisting solely of complex fields skip conversion entirely.

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
• Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using ./mvnw clean install -P contrib-check
  • JDK 21
  • JDK 25

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[exceptionfactory]

Thanks for proposing this improvement @maltesander. The initial version of Iceberg integration did not include supported for nested and complex types, so this is an important area of improvement. The general approach looks good, and the tests are helpful. I plan on taking a closer look at the conversion details, I noted a few minor recommendations for now.

[exceptionfactory]

The nested conditionals are difficult to read, I recommend breaking this out in more detail to make it easier to follow

[maltesander]

Thank you for the feedback @exceptionfactory. I tried to simplify here d174fd3.
I introduced a convertComplexValue method to handle this.

[exceptionfactory]

The field names should be declared statically and reused in both the input Schema and output row.

[maltesander]

Moved hardcoded constants to static ones: ef3e162

[maltesander]

I pushed some improvements @exceptionfactory:

• pre-allocated LinkedHashMap in convertMap 48a9f1e
• added RecordFieldType.CHOICE to the required type conversions list + test c5ade08

And one more thing / question: Iceberg supports TimestampTz which we cannot differ via RecordFieldType:

TZPROBE timestamptz <- LocalDateTime  => ClassCastException: class java.time.LocalDateTime
                                          cannot be cast to class java.time.OffsetDateTime
TZPROBE timestamptz <- OffsetDateTime => WROTE OK

Without something like (timezones vary, needs to be decided):

    private static Object convertTimestamp(final Timestamp timestamp, final Type icebergType) {
        return shouldAdjustToUtc(icebergType) ? timestamp.toInstant().atOffset(ZoneOffset.UTC) : timestamp.toLocalDateTime();
    }

    private static boolean shouldAdjustToUtc(final Type icebergType) {
        return switch (icebergType) {
            case Types.TimestampType timestampType -> timestampType.shouldAdjustToUTC();
            case Types.TimestampNanoType timestampNanoType -> timestampNanoType.shouldAdjustToUTC();
            case null, default -> false;
        };
    }

The first 3 tests fail without explicit conversion, so we are still missing (at least) one more fix:

    @Test
    void testConvertTimestampWithoutZone() {
        final Timestamp timestamp = Timestamp.valueOf(CREATED_LOCAL_DATE_TIME);

        final Object converted = RecordConverter.convertValue(timestamp, Types.TimestampType.withoutZone());

        assertEquals(CREATED_LOCAL_DATE_TIME, converted);
    }

    /**
     * Iceberg timestamptz columns require an OffsetDateTime rather than a LocalDateTime. A Timestamp identifies an
     * instant, so the converted value must describe that same instant expressed at UTC.
     */
    @Test
    void testConvertTimestampWithZone() {
        final Timestamp timestamp = Timestamp.valueOf(CREATED_LOCAL_DATE_TIME);

        final Object converted = RecordConverter.convertValue(timestamp, Types.TimestampType.withZone());

        final OffsetDateTime offsetDateTime = assertInstanceOf(OffsetDateTime.class, converted);
        assertEquals(ZoneOffset.UTC, offsetDateTime.getOffset());
        assertEquals(timestamp.toInstant(), offsetDateTime.toInstant());
    }

    @Test
    void testConvertTimestampNanoWithZone() {
        final Timestamp timestamp = Timestamp.valueOf(CREATED_LOCAL_DATE_TIME);

        final Object converted = RecordConverter.convertValue(timestamp, Types.TimestampNanoType.withZone());

        final OffsetDateTime offsetDateTime = assertInstanceOf(OffsetDateTime.class, converted);
        assertEquals(timestamp.toInstant(), offsetDateTime.toInstant());
    }

    /**
     * The Iceberg type is not known for every field, so an unresolved type must retain the LocalDateTime conversion.
     */
    @Test
    void testConvertTimestampUnknownIcebergType() {
        final Timestamp timestamp = Timestamp.valueOf(CREATED_LOCAL_DATE_TIME);

        final Object converted = RecordConverter.convertValue(timestamp, null);

        assertEquals(CREATED_LOCAL_DATE_TIME, converted);
    }

    /**
     * A timestamptz column nested inside a struct must be converted through the recursive path, matching the
     * positional access Iceberg uses when writing.
     */
    @Test
    void testGetConvertedRecordNestedTimestampWithZone() {
        final Types.StructType structType = Types.StructType.of(
                Types.NestedField.optional(1, CREATED_FIELD_NAME, Types.TimestampType.withZone())
        );

        final RecordSchema nestedSchema = new SimpleRecordSchema(List.of(
                new RecordField(CREATED_FIELD_NAME, RecordFieldType.TIMESTAMP.getDataType())
        ));
        final Timestamp timestamp = Timestamp.valueOf(CREATED_LOCAL_DATE_TIME);
        final Map<String, Object> nestedValues = new LinkedHashMap<>();
        nestedValues.put(CREATED_FIELD_NAME, timestamp);
        final Record nestedRecord = new MapRecord(nestedSchema, nestedValues);

        final Object converted = RecordConverter.convertValue(nestedRecord, structType);

        final StructLike struct = assertInstanceOf(StructLike.class, converted);
        assertEquals(timestamp.toInstant(), struct.get(0, OffsetDateTime.class).toInstant());
    }

This is not mentioned in the Jira ticket or this PR so i would defer fixing this in this PR?