Temporal workflow has blank delays/gaps between activities in local/cloud setup. Is this expected?

[JANVI2411]

---

Expected Behavior

After an activity completes, the next activity in the workflow sequence should be scheduled promptly — with only minimal overhead (milliseconds) between ActivityTaskCompleted and the next ActivityTaskScheduled event in the workflow history. In a healthy Temporal setup, the Workflow Task that drives the next execute_activity call should be picked up and executed quickly, even under moderate concurrency with child workflows.

---

Actual Behavior

A blank/invisible gap of 10–20 seconds appears between consecutive activities in the same ChildWorkflow. Specifically:

• An activity (e.g. predict_feedback_reference) completes successfully
• No next activity is scheduled for 10–20 seconds
• Only after that gap does the next activity (e.g. predict_transaction_merchant) appear as ActivityTaskScheduled

This gap occurs before the next activity is scheduled — meaning the delay is in Workflow Task pickup/execution, not in the activity queue or the activity execution itself. This is observed in both local Docker and Temporal Cloud environments, across a workload of ~50–150 concurrent child workflow executions.

---

Steps to Reproduce the Problem

1. Run a Temporal server locally via Docker with constrained resources (cpus: 0.5, mem_limit: 1024m)
2. Start a single worker process with MAX_CONCURRENT_WORKFLOW_TASKS=100 and MAX_CONCURRENT_ACTIVITIES=100 on the same machine
3. Trigger a parent workflow that fans out to 50–150 child workflows, each executing 4 sequential activities via await workflow.execute_activity(...)
4. Open the Temporal UI and inspect the event history of any child workflow
5. Observe the timestamp gap between ActivityTaskCompleted and the subsequent ActivityTaskScheduled — expect ~10–20 second blank gaps

---

Specifications

Field	Value
Temporal Python SDK Version	1.x (latest)
Temporal Server	temporalio/auto-setup:latest (Docker)
Temporal Server Resources	0.5 CPUs, 1024 MB RAM, 512 PIDs
Platform / OS	macOS (worker + Docker host)
Observed On	Local Docker + Temporal Cloud
Worker Concurrency	MAX_CONCURRENT_WORKFLOW_TASKS=100, MAX_CONCURRENT_ACTIVITIES=100
Worker Processes	1
Child Workflow Count	~50–150 concurrent
Database Backend	PostgreSQL 12

---

[simvlad]

With 150 child workflows each executing 4 sequential activities, every child generates 5 workflow tasks (one at workflow start, one after each activity completion). That's 750 workflow tasks for children plus 150 for the parent handling child completions, totaling 900 workflow tasks over the workload's lifetime.

These workflow tasks don't spread out evenly, activities across many children complete around the same time, so you can easily have 100+ workflow tasks queued simultaneously. With MAX_CONCURRENT_WORKFLOW_TASKS=100, the worker can only process 100 at once, so the rest queue up waiting for a free slot. The 10-20 second gaps between activities are caused by this workflow task slot saturation. Each child workflow has to wait its turn for a slot before the next activity can be scheduled.

[janvi-chokshi01]

@simvlad thanks a lot for the detailed breakdown

Just to give you full context on what I'm seeing in prod:

---

The prod issue:
Out of 150 child workflows running concurrently, 1 consistently takes 2-3 minutes while all the others finish in 30-40 seconds. When I look at that delayed workflow in the Temporal UI, it has a lot of blank gaps between activities - meaning it's not doing any actual work, it's just waiting. The activity execution time itself isn't the problem; the gaps between them are.

---

On local setup however, the behavior is different — the blank gaps are distributed more evenly across many child workflows rather than being concentrated in one outlier. So locally it looks like classic slot contention spread across children, but in prod it collapses into a single consistently delayed workflow. That difference in behavior is what's making this tricky to debug.

---

Experiments I've done:

I tried tuning every relevant parameter across all permutations:

MAX_CONCURRENT_ACTIVITIES = 1000 to 3000
MAX_CONCURRENT_WORKFLOW_TASKS = 1000 to 3000
MAX_CACHED_WORKFLOWS = 1000 to 3000

Also played with timeouts in the range of (10, 60).

Key observations:

• Batch size 50 → very few blank gaps, regardless of the above settings
• Batch size 150 → large blank gaps persist across almost all settings

So cranking up the concurrency limits alone doesn't fix it

---

My hypothesis on why 1 workflow is always the outlier:

Your analysis explains the aggregate saturation well, but if it were purely slot contention I'd expect the delay to be distributed somewhat uniformly across children, not one consistently bad actor.

Would love to know your thoughts, if there's a Temporal-native way to resolve this problem

[janvi-chokshi01]

Also attaching 2 screenshots - the Timeline view and the Event History of the delayed workflow. I think I've narrowed down exactly where the gap is happening.

---

What the screenshots show:

Looking at the event history:

• Event 8: Workflow Task Scheduled → 4:02:06.86
• Event 9: Workflow Task Started → 4:02:06.87 ← picked up in 10ms, so no slot contention here
• Event 10: Workflow Task Completed → 4:02:16.32 ← took ~9.5 seconds to complete

The gap is not between Scheduled → Started (which would point to slot saturation / queue depth). The gap is between Started → Completed, meaning the worker picked up the task immediately but took 9.5 seconds to execute it.

The timeline confirms this — predict_transaction_merchant doesn't start until ~21s, and that dead time maps exactly to this slow workflow task execution window.

---

Is it fully resolvable with just config/parameter tuning?

[simvlad]

Previously, you mentioned that the issue is with "blank gaps between activities", but now the problem seems to be different "the gap is between Started → Completed". Just wanted to understand, do you see both?

For the issue with predict_transaction_merchant, it looks like that the activity just took too long to complete.

By the way, do you use Python SDK? Wondering if you use sync APIs in activities, and then observing delays, because the event loop is blocked.

[janvi-chokshi01]

Let me correct myself here — I said "blank gaps between activities" but after looking more closely at the Timeline, those gaps are actually caused by slow workflow task execution, not scheduling delays.

As I showed in my previous comment, Event 8 (Workflow Task Scheduled) and Event 9 (Workflow Task Started) fire almost immediately after the previous activity completes — so there's no queue/slot contention issue there. The real problem is the ~10 secs gap between Event 9 (Started) and Event 10 (Completed) — the workflow task is being picked up instantly but takes long to actually execute.

So my question is — why is the workflow task itself taking ~10 seconds to execute? Is this a replay/cache miss issue? And is it solvable with the right configuration or parameters?

PS: the ss I've attached are from my local setup not prod setup just so I can regenerate the issue locally and debug easily

To give more context on the production issue — out of all 150 child workflows running concurrently, only few consistently takes 2+ minutes while all the others are already done in 45 seconds to 1 minute. That slow child workflows are the bottleneck for the entire batch, and after digging into its event history, the slowness is coming from slow workflow_task execution (the 1 min gap between Started and Completed in Production setup). Everything else finishes fine — it's always this outliers dragging the total latency.

[simvlad]

Look at the "Workflow Task Scheduled" event in the history to find if there was replay/cache miss. Normally it would look like this: Task Queue Kind TASK_QUEUE_KIND_STICKY.

But stepping back, what do you do in the workflow code (for workflows that are slow)? Any part that can be slow?

And I think you missed this question from my previous comment. Do you use Python SDK? Wondering if you use sync APIs in activities, and then observing delays, because the event loop is blocked.

[janvi-chokshi01]

To answer your previous question:

Yes, I’m using the Temporal Python SDK.
Yes, we do have some sync APIs inside activity paths. Some sync calls, like a few S3 and event publishing paths, are wrapped with asyncio.to_thread(...), so those specific usages should not block the event loop.

But we also still have a few direct sync calls inside activities:

FileManager().get_file(...) for re-downloading/restoring the file when the local path is missing
upload_json_to_s3(...) for uploading JSON to S3
run_combination_rule(...) and run_rule_fifo(...) for matching logic during reconciliation
My understanding is that these can block if they take time:

S3 upload and file re-download are I/O and can be long-running
the matching rule functions are sync CPU work and can also take time with larger inputs
So overall, I think your suspicion is valid in a few places

---

I also checked the "Workflow Task Scheduled" and yes thats TASK_QUEUE_KIND_STICKY

[simvlad]

Thanks for confirming. The unwrapped sync calls inside async def activities are likely the main cause. They block the event loop, which prevents workflow task processing across all workflows on that worker.

Two options:

1. Wrap those sync calls with asyncio.to_thread()
2. Define those activities as plain def instead of async def. The SDK will automatically run them in a thread executor

I'd also suggest adding some logging to confirm the issue.

[janvi-chokshi01]

Cool I'll make these corrections and will let you know