Skip to content

Commit dc4bae2

Browse files
committed
fix(inmemory): stabilize task ordering and lifecycle
Ensure post_send completes before local execution while preserving per-invocation ownership and await_inplace semantics. Drain accepted tasks during shutdown, retain background failures, and clean up middleware, result backend, and executor resources reliably. Add concurrency, cancellation, lifecycle, and compatibility coverage. Closes: #586
1 parent ae2b788 commit dc4bae2

12 files changed

Lines changed: 1316 additions & 72 deletions

File tree

docs/available-components/brokers.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,11 @@ This is a special broker for local development. It uses the same functions to ex
1212
but all tasks are executed locally in the current thread.
1313
By default it uses `InMemoryResultBackend` but this can be overridden.
1414

15+
`startup()` and `shutdown()` run both the client and worker event phases because
16+
the broker performs both roles in one process. They also own middleware and
17+
result backend lifecycle. Shutdown waits for local background executions before
18+
closing those resources and the synchronous task executor.
19+
1520
## ZeroMQBroker
1621

1722
This broker uses [ZMQ](https://zeromq.org/) to communicate between worker and client processes.

docs/guide/testing-taskiq.md

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,10 @@ broker = InMemoryBroker(await_inplace=True)
165165

166166
With this setup all `await function.kiq()` calls will behave similarly to `await function()`, but
167167
with dependency injection and all taskiq-related functionality.
168+
The normal middleware boundary is preserved in both execution modes: all
169+
`post_send` hooks finish before `pre_execute` and the task body begin.
170+
With `await_inplace=True`, the `kiq()` call also returns only after inline
171+
execution completes.
168172

169173
2. Alternatively, you can manually await all tasks after invoking the
170174
target function by using the `wait_all` method.
@@ -182,6 +186,18 @@ async def test_add_one():
182186
# have been completed and do all the assertions.
183187
```
184188

189+
`wait_all()` also surfaces the first pending background callback failure even
190+
if that callback completed before `wait_all()` was called. The failure is
191+
consumed after it is raised, so a later `wait_all()` only observes newer work.
192+
Cancelling a `wait_all()` call cancels only that waiter; accepted executions
193+
remain tracked and can be drained by a later call.
194+
Calling `shutdown()` performs the same drain before closing middleware, result
195+
backend, and executor resources. If shutdown is cancelled while draining, it
196+
finishes the drain and resource cleanup before propagating the cancellation.
197+
Both drain methods must be called by the external test or application lifecycle
198+
owner. Calling `wait_all()` or `shutdown()` from a task or `post_send` hook
199+
managed by the same broker raises `RuntimeError` instead of waiting on itself.
200+
185201
## Dependency injection
186202

187203
If you use dependencies in your tasks, you may think that this can become a problem for tests. But it's not.

taskiq/abc/broker.py

Lines changed: 66 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@
2323
from taskiq.acks import AckableMessage
2424
from taskiq.decor import AsyncTaskiqDecoratedTask
2525
from taskiq.events import TaskiqEvents
26-
from taskiq.exceptions import TaskBrokerMismatchError
26+
from taskiq.exceptions import SendTaskError, TaskBrokerMismatchError
2727
from taskiq.formatters.proxy_formatter import ProxyFormatter
2828
from taskiq.message import BrokerMessage
2929
from taskiq.result_backends.dummy import DummyResultBackend
@@ -186,39 +186,90 @@ def add_middlewares(self, *middlewares: "TaskiqMiddleware") -> None:
186186

187187
async def startup(self) -> None:
188188
"""Do something when starting broker."""
189-
event = TaskiqEvents.CLIENT_STARTUP
190-
if self.is_worker_process:
191-
event = TaskiqEvents.WORKER_STARTUP
192-
193-
for handler in self.event_handlers[event]:
194-
await maybe_awaitable(handler(self.state))
189+
for event in self._get_startup_events():
190+
for handler in self.event_handlers[event]:
191+
await maybe_awaitable(handler(self.state))
195192

196193
for middleware in self.middlewares:
197194
if middleware.__class__.startup != TaskiqMiddleware.startup:
198195
await maybe_awaitable(middleware.startup())
199196

200197
await self.result_backend.startup()
201198

199+
def _get_startup_events(self) -> tuple[TaskiqEvents, ...]:
200+
"""Return event phases owned by this broker startup."""
201+
if self.is_worker_process:
202+
return (TaskiqEvents.WORKER_STARTUP,)
203+
return (TaskiqEvents.CLIENT_STARTUP,)
204+
202205
async def shutdown(self) -> None:
203206
"""
204207
Close the broker.
205208
206209
This method is called,
207210
when broker is closing.
208211
"""
209-
event = TaskiqEvents.CLIENT_SHUTDOWN
210-
if self.is_worker_process:
211-
event = TaskiqEvents.WORKER_SHUTDOWN
212+
shutdown_error: BaseException | None = None
212213

213-
# Call all shutdown events.
214-
for handler in self.event_handlers[event]:
215-
await maybe_awaitable(handler(self.state))
214+
for event in self._get_shutdown_events():
215+
for handler in self.event_handlers[event]:
216+
try:
217+
await maybe_awaitable(handler(self.state))
218+
except BaseException as exc:
219+
shutdown_error = self._remember_shutdown_error(
220+
shutdown_error,
221+
exc,
222+
)
216223

217224
for middleware in self.middlewares:
218225
if middleware.__class__.shutdown != TaskiqMiddleware.shutdown:
219-
await maybe_awaitable(middleware.shutdown())
226+
try:
227+
await maybe_awaitable(middleware.shutdown())
228+
except BaseException as exc:
229+
shutdown_error = self._remember_shutdown_error(
230+
shutdown_error,
231+
exc,
232+
)
233+
234+
try:
235+
await self.result_backend.shutdown()
236+
except BaseException as exc:
237+
shutdown_error = self._remember_shutdown_error(shutdown_error, exc)
238+
239+
if shutdown_error is not None:
240+
raise shutdown_error
241+
242+
def _get_shutdown_events(self) -> tuple[TaskiqEvents, ...]:
243+
"""Return event phases owned by this broker shutdown."""
244+
if self.is_worker_process:
245+
return (TaskiqEvents.WORKER_SHUTDOWN,)
246+
return (TaskiqEvents.CLIENT_SHUTDOWN,)
247+
248+
@staticmethod
249+
def _remember_shutdown_error(
250+
first_error: BaseException | None,
251+
current_error: BaseException,
252+
) -> BaseException:
253+
"""Keep the first shutdown failure while cleanup continues."""
254+
if first_error is None:
255+
return current_error
256+
logger.error(
257+
"Additional error while shutting down broker resources.",
258+
exc_info=current_error,
259+
)
260+
return first_error
220261

221-
await self.result_backend.shutdown()
262+
async def _kick_with_post_send(
263+
self,
264+
message: BrokerMessage,
265+
post_send: Callable[[], Awaitable[None]],
266+
) -> None:
267+
"""Run the package-internal send boundary used by AsyncKicker."""
268+
try:
269+
await self.kick(message)
270+
except Exception as exc:
271+
raise SendTaskError from exc
272+
await post_send()
222273

223274
@abstractmethod
224275
async def kick(

0 commit comments

Comments
 (0)