Skip to content

fix(connectors): defer postgres source progress until ack - #3957

Open
rohankumardubey wants to merge 6 commits into
apache:masterfrom
rohankumardubey:fix/postgres-source-ack
Open

fix(connectors): defer postgres source progress until ack#3957
rohankumardubey wants to merge 6 commits into
apache:masterfrom
rohankumardubey:fix/postgres-source-ack

Conversation

@rohankumardubey

Copy link
Copy Markdown
Contributor

Which issue does this PR address?

Closes #3635

Rationale

The PostgreSQL source advanced tracking offsets, deleted or marked rows, and consumed CDC changes before Iggy confirmed delivery. A failed send could therefore permanently skip source records.

What changed?

PostgreSQL polling now stages cursor updates and row operations until the runtime reports a successful batch acknowledgment. NACK discards the staged work, while ACK commits the state and performs the pending delete or mark operations.

CDC now peeks logical-slot changes and advances the slot only after acknowledgment. A deterministic regression test stops Iggy during delivery and verifies that the PostgreSQL rows are redelivered after restart.

Local Execution

  • Passed cargo fmt --all -- --check
  • Passed cargo clippy -p iggy_connector_postgres_source -p integration --all-features --all-targets -- -D warnings
  • Passed all 70 PostgreSQL source unit tests
  • Passed all 7 PostgreSQL polling integration tests
  • Passed both PostgreSQL CDC integration tests
  • Passed the deterministic kill-server regression test
  • Passed git diff --check

@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 23, 2026
@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.88235% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.69%. Comparing base (a65f177) to head (9ed427d).

Files with missing lines Patch % Lines
core/connectors/sources/postgres_source/src/lib.rs 80.88% 19 Missing and 7 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3957       +/-   ##
=============================================
- Coverage     83.93%   57.69%   -26.24%     
  Complexity     1358     1358               
=============================================
  Files          1213     1212        -1     
  Lines        168441   139746    -28695     
  Branches     135999   107313    -28686     
=============================================
- Hits         141373    80622    -60751     
- Misses        23389    55625    +32236     
+ Partials       3679     3499      -180     
Components Coverage Δ
Rust Core 50.16% <80.88%> (-34.56%) ⬇️
Java SDK 66.67% <ø> (ø)
C# SDK 76.52% <ø> (ø)
Python SDK 90.13% <ø> (ø)
PHP SDK 84.48% <ø> (ø)
Node SDK 95.81% <ø> (-0.10%) ⬇️
Go SDK 68.37% <ø> (+0.07%) ⬆️
Files with missing lines Coverage Δ
core/connectors/sources/postgres_source/src/lib.rs 73.79% <80.88%> (+0.47%) ⬆️

... and 405 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@rohankumardubey

Copy link
Copy Markdown
Contributor Author

/ready

@rohankumardubey

Copy link
Copy Markdown
Contributor Author

/request-review @hubcio

@github-actions
github-actions Bot requested a review from hubcio August 24, 2026 08:40

@hubcio hubcio left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few things without diff lines to hang them on:

  • .claude/skills/connector-source/SKILL.md still teaches the pre-ack pattern this pr removes (the "matches poll_tables" snippet writes cursors during poll, "always return state in every ProducedMessages" no longer holds for empty polls, and state-serialization failure is now a hard poll error, lib.rs:288-296) and doesn't mention on_batch_result at all - needs an update to the new contract.
  • mark_or_delete_processed_rows maps db errors to Error::InvalidRecord (lib.rs:732, :751) - misleading now that these errors surface through the ack path; Error::Connection like advance_replication_slot uses would fit better.
  • README.md:15 still promises offset tracking "avoid duplicates" - at-least-once redelivery means duplicates are possible; and README.md:57 documents the poll_interval default as 1s while the code fallback is 10s (lib.rs:180).

Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/connectors/sources/postgres_source/README.md
Comment thread core/integration/tests/connectors/postgres/postgres_source.rs
Comment thread core/connectors/sources/postgres_source/src/lib.rs Outdated
Comment thread core/connectors/sources/postgres_source/src/lib.rs
Comment thread core/connectors/sources/postgres_source/src/lib.rs Outdated
Comment thread core/integration/tests/connectors/postgres/postgres_source.rs Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Aug 24, 2026
@rohankumardubey
rohankumardubey requested a review from hubcio August 26, 2026 10:54
@hubcio

hubcio commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

@rohankumardubey please write
/ready
when PR is ready for review. at any top level comment, in the beginning of any line. this will let know maintainers that something is ready for review by flipping gh labels.
image

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Aug 26, 2026
@rohankumardubey

Copy link
Copy Markdown
Contributor Author

/ready

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

connectors: no send acknowledgment from runtime to source plugins — state advances and rows are deleted before delivery is confirmed

2 participants