fix(replication): distribute destination omni among the subscribers

[meskill]

Attempt to fix issue with test #924 by distributing the destination shards between streams, so they don't overlap and therefore can't deadlock each other. At the cost of consistency in case input omni tables are not equal in the content or the definition of the table changes from source to destination.

[codecov]

Codecov Report

❌ Patch coverage is 96.62162% with 5 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...nd/replication/logical/publisher/publisher_impl.rs	0.00%	3 Missing ⚠️
...d/replication/logical/subscriber/omni_ownership.rs	98.63%	1 Missing ⚠️
...c/backend/replication/logical/subscriber/stream.rs	87.50%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[levkk]

This is my preferred solution for sure. It doesn't sacrifice performance and fixes the original deadlock issue.

One thing to double check is that we send the WAL acknowledgement back to the source for data received via logical replication for omni tables that the connection "doesn't own"; I think this is not critical since other tables will receive writes too and that will move the WAL forward on the source, but it could be possible to have omni-only sharded clusters, theoretically.

[meskill]

One thing to double check is that we send the WAL acknowledgement back to the source for data received via logical replication for omni tables that the connection "doesn't own"; I think this is not critical since other tables will receive writes too and that will move the WAL forward on the source, but it could be possible to have omni-only sharded clusters, theoretically.

Should be ok, since the changes are made on the send level and they do not affect neither how acknowledgement to source is sent nor the lsn status update