feat(python): support message partitioning strategies by jiengup · Pull Request #3927 · apache/iggy

jiengup · 2026-08-20T06:53:19Z

Which issue does this PR address?

Rationale

The Python SDK only supported sending messages to an explicit partition, while the Rust SDK also supports balanced and message-key partitioning.

What changed?

IggyClient.send_messages() now accepts a Python Partitioning object supporting fixed, balanced, and message-key routing. Existing integer partition IDs remain fully compatible, while string message keys are encoded as UTF-8 and validated against the 1–255 byte limit.

Type stubs, tests, and a runnable Python example covering all three strategies are included.

Local Execution

Passed all 87 tests in foreign/python/tests/test_message_operations.py
Passed Rust formatting and Clippy checks
Passed Ruff and Pyrefly checks for the Python SDK and examples
Successfully ran the partitioning example against a local Iggy server
Pre-commit hooks ran and passed

AI Usage

Tool: OpenAI Codex
Scope: Repository analysis, PyO3 implementation, tests, type stubs, documentation, and the Python example
Verification: Reviewed the generated changes, ran the complete message operations test suite, static checks, pre-commit hooks, and the example against a local server
Yes, I can explain every line of the changes

github-actions · 2026-08-20T06:53:30Z

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

/ready - back to S-waiting-on-review after addressing feedback
/author - flip to S-waiting-on-author while you finish changes
/request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

codecov · 2026-08-20T07:04:17Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.76%. Comparing base (3cd0860) to head (f715f3c).
⚠️ Report is 15 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##             master    #3927   +/-   ##
=========================================
  Coverage     83.76%   83.76%           
  Complexity     1358     1358           
=========================================
  Files          1212     1213    +1     
  Lines        166189   166215   +26     
  Branches     133663   133663           
=========================================
+ Hits         139210   139236   +26     
  Misses        23342    23342           
  Partials       3637     3637

Components	Coverage Δ
Rust Core	`84.55% <ø> (ø)`
Java SDK	`66.67% <ø> (ø)`
C# SDK	`76.08% <ø> (ø)`
Python SDK	`90.11% <100.00%> (+0.11%)`	⬆️
PHP SDK	`84.48% <ø> (ø)`
Node SDK	`95.84% <ø> (ø)`
Go SDK	`68.32% <ø> (ø)`

Files with missing lines	Coverage Δ
foreign/python/src/client.rs	`99.85% <100.00%> (+<0.01%)`	⬆️
foreign/python/src/lib.rs	`100.00% <100.00%> (ø)`
foreign/python/src/partitioning.rs	`100.00% <100.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ethanlin01x

Some suggestions. The stub one I think should be fixed before merge.

ethanlin01x · 2026-08-22T12:19:56Z

        """

+@typing.final
+class Partitioning:


Partitioning is missing from __all__, so from apache_iggy import * fails pyrefly with unknown-name.

ethanlin01x · 2026-08-22T17:18:29Z

+    #[pyo3(transparent)]
+    Strategy(Partitioning),


Strategy has no annotation, so errors read 'Strategy | int' — a Rust name that does not exist in Python. Please add annotation = "Partitioning"

ethanlin01x · 2026-08-22T17:25:28Z

+        )
+
+        assert len(response.confirmations) == 1
+        assert response.confirmations[0].partition_id < partitions_count


Sending once cannot tell round-robin apart from always returning partition 0. Could you send three batches and assert the three confirmed partition ids are distinct?

slbotbm · 2026-08-23T19:59:29Z

I'd rather a new example like this not be created, unless it is a capability that the rust sdk does not offer. Better to add comments to an existing example writing how can message partitioning options will result in different outcomes, if any.

slbotbm · 2026-08-23T19:59:50Z

Follows from above comment -- not required.

slbotbm · 2026-08-23T20:19:31Z

        let stream = Identifier::try_from(stream)?;
        let topic = Identifier::try_from(topic)?;
-        let partitioning = Partitioning::partition_id(partitioning);
+        let partitioning = partitioning.into();


Let's change this into from::. I like to be explicit about what is being converted into what in code. into does not help a first-time reader understand this without searching the code base.

slbotbm · 2026-08-23T20:23:18Z

+#[gen_stub_pymethods]
+#[pymethods]
+impl Partitioning {
+    /// Routes the batch to partitions using server-side round-robin selection.


Change to "Routes the batch to one partition selected by round-robin."

slbotbm · 2026-08-23T20:23:44Z

+        }
+    }
+
+    /// Routes the batch to the specified partition.


Change to

/// Routes the batch to the specified partition. /// /// `partition_id` must be between 0 and `2**32 - 1`. The topic must contain /// that partition when the batch is sent.

slbotbm · 2026-08-23T20:28:08Z

    /// Sends a list of messages to the specified topic.
    /// Returns a SendMessagesResponse carrying the per-partition commit
    /// confirmations, or a PyRuntimeError on failure. The confirmation list is
    /// empty when the server reports no offsets, and the legacy server never
    /// reports any.
+    ///
+    /// `partitioning` is required. Pass `Partitioning.balanced()`,
+    /// `Partitioning.partition_id(id)`, or `Partitioning.messages_key(key)`.
+    /// An integer remains supported as shorthand for `partition_id`.


Lets change this to

/// Sends a batch of messages to a topic using the selected partitioning strategy. /// /// Args: /// stream: Stream identifier as `str | int`. /// topic: Topic identifier as `str | int`. /// partitioning: A `Partitioning` strategy or an integer partition ID. /// Use `Partitioning.balanced()`, `Partitioning.partition_id(id)`, or /// `Partitioning.messages_key(key)`. An integer is shorthand for /// `Partitioning.partition_id(id)`. /// messages: Messages to send as `list[SendMessage]`. /// /// Returns: /// An awaitable that resolves to `SendMessagesResponse`. Its confirmations /// report the committed partition and batch base offset. The list is empty /// when the server reports no offsets, including on the legacy server. /// /// Raises: /// ValueError: If a string stream or topic identifier is invalid. /// TypeError: If `partitioning` or `messages` has an unsupported type. /// OverflowError: If a numeric stream, topic, or partition ID is outside /// the supported unsigned 32-bit range. /// RuntimeError: If the request fails.

slbotbm · 2026-08-23T20:33:21Z

+
+    @pytest.mark.unit
+    @pytest.mark.parametrize("key", [b"a" * 255, "a" * 255])
+    def test_messages_key_accepts_255_bytes(self, key):


Let's add "界" * 85 as a case for this as well.

slbotbm · 2026-08-23T20:42:06Z

+    @pytest.mark.unit
+    def test_balanced_and_partition_id_strategies(self):
+        assert isinstance(Partitioning.balanced(), Partitioning)
+        assert isinstance(Partitioning.partition_id(1), Partitioning)


Let's add an assertion for partition_id(2**32 -1)

slbotbm · 2026-08-23T20:45:48Z

        assert confirmation.base_offset == polled_messages[0].offset()

+    @pytest.mark.asyncio
+    async def test_send_messages_with_partition_id_strategy(


In this test, let's also send one message with Partitioning.partition_id(3) and assert RuntimeError

slbotbm · 2026-08-23T20:48:33Z

Something that did not land anywhere: In TestMessageOperations.test_send_messages_with_partition_id_strategy, send two messages instead of one and retain the existing single-confirmation assertion.

feat(python): support message partitioning strategies

f715f3c

github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Aug 20, 2026

jiengup changed the title ~~Python SDK: support message partitioning strategies~~ feat(python): support message partitioning strategies Aug 20, 2026

jiengup mentioned this pull request Aug 20, 2026

Tracking Issue: Align the Python SDK API with the Rust (native) SDK API #3893

Open

2 tasks

ethanlin01x reviewed Aug 22, 2026

View reviewed changes

slbotbm requested changes Aug 23, 2026

View reviewed changes

github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Aug 23, 2026

Uh oh!

Conversation

jiengup commented Aug 20, 2026

Which issue does this PR address?

Rationale

What changed?

Local Execution

AI Usage

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

codecov Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ethanlin01x left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Aug 20, 2026 •

edited

Loading