[FLINK-38742][cdc-pipeline/postgres] Fix timestamp with time zone#4181
Open
Fluder-Paradyne wants to merge 1 commit intoapache:masterfrom
Open
[FLINK-38742][cdc-pipeline/postgres] Fix timestamp with time zone#4181Fluder-Paradyne wants to merge 1 commit intoapache:masterfrom
Fluder-Paradyne wants to merge 1 commit intoapache:masterfrom
Conversation
Fluder-Paradyne
force-pushed
the
fix-timestamp-with-time-zone
branch
2 times, most recently
from
8f24e77 to
033e204
Compare
Fluder-Paradyne
changed the title
Fluder-Paradyne
changed the title
Fluder-Paradyne
force-pushed
the
fix-timestamp-with-time-zone
branch
from
033e204 to
c89cdd9
Compare
Fluder-Paradyne
changed the title
Fluder-Paradyne
force-pushed
the
fix-timestamp-with-time-zone
branch
from
111c324 to
c89cdd9
Compare
Contributor
|
@Fluder-Paradyne Thank for your contribute. cc @lvyanquan Do you agree with me? |
Contributor
Can you describe in detail how I can reproduce this problem? |
Author
|
This table works but this one doesn't |
Contributor
|
column_type_test#time_types has' TIMESTAMP WITH TIME ZONE 'ut test, but I have a problem with e2e test, let me check again. |
Contributor
loserwang1024
left a comment
loserwang1024
left a comment
There was a problem hiding this comment.
I have left some minor comment
...in/java/org/apache/flink/cdc/connectors/postgres/source/PostgresSchemaDataTypeInference.java
Outdated
Show resolved
Hide resolved
...um/src/main/java/org/apache/flink/cdc/debezium/event/DebeziumEventDeserializationSchema.java
Outdated
Show resolved
Hide resolved
…TIMESTAMP_LTZ PostgreSQL TIMESTAMPTZ columns were causing NumberFormatException because PostgresTypeUtils incorrectly mapped them to ZonedTimestampType (TIMESTAMP_WITH_TIME_ZONE). PostgreSQL TIMESTAMPTZ stores values internally in UTC and converts on display based on session timezone, which semantically matches TIMESTAMP_LTZ, not TIMESTAMP_WITH_TIME_ZONE. Additionally, Debezium's PostgreSQL connector always converts TIMESTAMPTZ values to UTC format (e.g., '2025-12-30T05:59:50.724893Z') before serialization. The type mismatch caused sinks to call getZonedTimestamp() on LocalZonedTimestampData, resulting in NumberFormatException when trying to parse binary data as a string. This commit fixes PostgresTypeUtils to correctly map TIMESTAMPTZ and TIMESTAMPTZ_ARRAY to TIMESTAMP_LTZ type, matching both PostgreSQL semantics and Debezium's serialization format. The E2E test is updated to verify the fix.
Fluder-Paradyne
force-pushed
the
fix-timestamp-with-time-zone
branch
from
c89cdd9 to
6d41a79
Compare
Author
Hey @Mrart thanks for the review, dug around a bit, yes TIMESTAMP_LTZ is more reasonable as it close to postgres secmantics and sinks also support it |
Contributor
|
@Fluder-Paradyne @Mrart , I have seen your reproduce step.
I don't think so. Timestamp ltz will replace the timestamp zone of source with local time zone in flink cdc. For some sink(such as iceberg, kafka), they want to reserve the time zone from source. If The data from source is 08:00+8, while the time zone of fink cdc is UTC+0, the sink will write into 00:00+0. (If it's written into postgres , it's ok, will stored into same long value. However, If it's written into kafka, it wil be stored as string) @lvyanquan , WDYT? |
Author
|
@loserwang1024 for pg TIMESTAMPTZ column, the moment you insert/update the timezone information is converted to UTC and store as UTC time, and then when you select it, it gets converted based on timezone of the postgres session. Since the source itself does not have the original timezone info, shoudn't Timestamp ltz be fine ? |
Contributor
I agree with @loserwang1024's option, we can configure this information of Source through server-time-zone, refer to It seems that we are missing this section in the document (maybe you can add it together). |
Author
|
Hey Guys I am bit confused, This is my current understanding on how TIMESTAMPTZ is treated. TIMESTAMPTZ in PG --> DEBEZIUM Since debezium also give only a UTC converted string, I dont think there is a way to retain timezone information to the sink other than UTC https://www.postgresql.org/docs/14/datatype-datetime.html
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes an issue where Flink CDC pipeline failed when processing PostgreSQL tables with TIMESTAMPTZ (timestamp with time zone) columns. The fix involves changing the type mapping from ZonedTimestampType to TIMESTAMP_LTZ (LocalZonedTimestampType), which correctly represents PostgreSQL's TIMESTAMPTZ semantics.
Changes:
- Updated type mapping for TIMESTAMPTZ columns in PostgreSQL connector
- Added comprehensive E2E test coverage with TIMESTAMPTZ column
- Updated test expectations to validate the fix
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| PostgresTypeUtils.java | Changed TIMESTAMPTZ mapping from ZonedTimestampType to TIMESTAMP_LTZ for both scalar and array types, removed unused import |
| postgres_inventory.sql | Added created_at TIMESTAMPTZ column to products table with test data |
| PostgresE2eITCase.java | Updated test expectations to include new TIMESTAMPTZ column in schema and data change events |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
Is there any update? |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/ddl/postgres_inventory.sql
Show resolved
Hide resolved
flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/resources/ddl/postgres_inventory.sql
Show resolved
Hide resolved
...-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/PostgresE2eITCase.java
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In postgres when a table has a column with timestamp with time zone column the cdc pipeline fails