feat(ollama): add max_retries to chat generator by Keyur-S-Patel · Pull Request #2899 · deepset-ai/haystack-core-integrations

Keyur-S-Patel · 2026-03-01T02:48:39Z

PR : `feat/ollama-chat-max-retries`

Related Issues

partially addresses #9309

Proposed Changes:

Added max_retries support to OllamaChatGenerator.
Added tenacity-based retries for both sync and async chat calls.
Added exponential backoff via wait_exponential().
Kept retry handling local to run() / run_async() using @retry-decorated callables instead of separate helper methods.
Added tenacity to the Ollama integration package dependencies.
Updated unit tests for retry behavior, including the success-after-retry path and retry exhaustion path.
Adjusted the retry success test fixture to include valid Ollama token count metadata.

How did you test it?

hatch run fmt-check
hatch run test:types
hatch run test:unit

Result:

formatting checks passed
type checks passed
Ollama unit test suite passed on this branch (34 passed)

Notes for the reviewer

tenacity.stop_after_attempt(...) uses total attempts, so the implementation uses max_retries + 1 to preserve the expected semantics of “initial call + N retries”.
Retry policy currently retries generic Exception from Ollama chat calls, matching the previous broad retry behavior while switching to a standard retry library.
Full package test:all still depends on a running local Ollama instance for integration tests; only unit tests were used for branch validation.

Checklist

I have read the contributors guidelines and the code of conduct
I have updated the related issue with new insights and changes
I added unit tests and updated the docstrings
I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

Keyur-S-Patel · 2026-03-01T03:06:21Z

@anakin87 Can you help me close this PR, its similar to the one you reviewed here

Keyur-S-Patel

Similar to this one #2875

bogdankostic

Thanks for your PR @Keyur-S-Patel! I left two comments regarding the retry logic. Also, it would be great if we could add tests for run_async.

bogdankostic · 2026-03-05T14:05:03Z

...rations/ollama/src/haystack_integrations/components/generators/ollama/chat/chat_generator.py

+        @retry(
+            reraise=True,
+            stop=stop_after_attempt(self.max_retries + 1),
+            retry=retry_if_exception_type(Exception),


Retrying on all kind of exceptions is too broad.

Retry on 429 and 5xx?

Lemme know if you want to retry on something else as well.

bogdankostic · 2026-03-05T14:10:40Z

...rations/ollama/src/haystack_integrations/components/generators/ollama/chat/chat_generator.py

+        @retry(
+            reraise=True,
+            stop=stop_after_attempt(self.max_retries + 1),
+            retry=retry_if_exception_type(Exception),
+            wait=wait_exponential(),
        )
+        def chat_with_retry() -> ChatResponse | Iterator[ChatResponse]:
+            return self._client.chat(
+                model=self.model,
+                messages=ollama_messages,
+                tools=ollama_tools,
+                stream=is_stream,  # type: ignore[call-overload]  # Ollama expects Literal[True] or Literal[False], not bool
+                keep_alive=self.keep_alive,
+                options=generation_kwargs,
+                format=self.response_format,
+                think=self.think,
+            )
+
+        response = chat_with_retry()


Can we have this not nested inside of run?

@Retry annotation requires function so can

use wrapper function or

Build new method and put retry annotation there

Will separate out function in next commit

Keyur-S-Patel

Updated new commit with following,

retry logic to retry on specific codes
pull the retry out of run
added unit tests for async

bogdankostic

We're almost there, I just have a minor comment regarding the type of retry_state and a question on tenacity version.

bogdankostic · 2026-03-06T13:47:01Z

...rations/ollama/src/haystack_integrations/components/generators/ollama/chat/chat_generator.py

+HTTP_STATUS_SERVER_ERROR_MAX_EXCLUSIVE = 600
+
+
+def _stop_after_instance_max_retries(retry_state: Any) -> bool:


Let's not use type Any here

Suggested change

def _stop_after_instance_max_retries(retry_state: Any) -> bool:

def _stop_after_instance_max_retries(retry_state: RetryCallState) -> bool:

bogdankostic · 2026-03-06T13:53:47Z

...rations/ollama/src/haystack_integrations/components/generators/ollama/chat/chat_generator.py

+        :param max_retries:
+            Maximum number of retries to attempt for failed requests.


Lest's make this parameter more descriptive.

Suggested change

:param max_retries:

Maximum number of retries to attempt for failed requests.

:param max_retries:

Maximum number of retries to attempt for failed requests (HTTP 429, 5xx, connection/timeout errors).

Uses exponential backoff between attempts. Set to 0 (default) to disable retries.

bogdankostic · 2026-03-06T13:54:55Z

integrations/ollama/pyproject.toml

    "Programming Language :: Python :: Implementation :: PyPy",
 ]
-dependencies = ["haystack-ai>=2.22.0", "ollama>=0.5.0", "pydantic"]
+dependencies = ["haystack-ai>=2.22.0", "ollama>=0.5.0", "pydantic", "tenacity>=8.2.3"]


Is there a specific reason for >=8.2.3 for tenacity?

Keyur-S-Patel requested a review from a team as a code owner March 1, 2026 02:48

Keyur-S-Patel requested review from bogdankostic and removed request for a team March 1, 2026 02:48

github-actions bot added integration:ollama type:documentation Improvements or additions to documentation labels Mar 1, 2026

add max_retries to chat generator

83278a1

Keyur-S-Patel force-pushed the feat/ollama-chat-max-retries branch from ae3c423 to 83278a1 Compare March 1, 2026 02:58

Keyur-S-Patel changed the title ~~add max_retries to chat generator~~ Open feat(ollama): add max_retries to chat generator Mar 1, 2026

Keyur-S-Patel changed the title ~~Open feat(ollama): add max_retries to chat generator~~ feat(ollama): add max_retries to chat generator Mar 1, 2026

Keyur-S-Patel commented Mar 3, 2026

View reviewed changes

bogdankostic requested changes Mar 5, 2026

View reviewed changes

fix: refine ollama retry behavior and tests

0195286

Keyur-S-Patel commented Mar 6, 2026

View reviewed changes

Keyur-S-Patel requested a review from bogdankostic March 6, 2026 03:50

bogdankostic requested changes Mar 6, 2026

View reviewed changes

		HTTP_STATUS_SERVER_ERROR_MAX_EXCLUSIVE = 600


		def _stop_after_instance_max_retries(retry_state: Any) -> bool:

		:param max_retries:
		Maximum number of retries to attempt for failed requests.

Conversation

Keyur-S-Patel commented Mar 1, 2026 • edited by bogdankostic Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR : feat/ollama-chat-max-retries

Related Issues

Proposed Changes:

How did you test it?

Notes for the reviewer

Checklist

Uh oh!

Keyur-S-Patel commented Mar 1, 2026

Uh oh!

Keyur-S-Patel left a comment

Choose a reason for hiding this comment

Uh oh!

bogdankostic left a comment

Choose a reason for hiding this comment

Uh oh!

bogdankostic Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Keyur-S-Patel Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bogdankostic Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Keyur-S-Patel Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Keyur-S-Patel Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

Keyur-S-Patel left a comment

Choose a reason for hiding this comment

Uh oh!

bogdankostic left a comment

Choose a reason for hiding this comment

Uh oh!

bogdankostic Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

bogdankostic Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

bogdankostic Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Keyur-S-Patel commented Mar 1, 2026 •

edited by bogdankostic

Loading

PR : `feat/ollama-chat-max-retries`

Keyur-S-Patel Mar 5, 2026 •

edited

Loading