Perf/decision task

[marco-c]

I haven't yet reviewed all the commits, I'll probably drop some of them, but I want to see if codspeed picks anything up.

[codspeed]

Merging this PR will not alter performance

✅ 30 untouched benchmarks
🆕 2 new benchmarks

Performance Changes

	Benchmark	BASE	HEAD	Efficiency
🆕	test_create_tasks[btree]	N/A	3.8 s	N/A
🆕	test_create_tasks[fan]	N/A	4 s	N/A

---

_{Comparing marco-c:perf/decision-task (7c4a13b) with main (77cbc3b)}

[sylvestre]

wahou

[marco-c]

@codspeedbot run the new benchmarks against the base of pr, and display the performance changes with the head of the PR, with the link to the corresponding CodSpeed compare runs page

[marco-c]

@codspeedbot run the new benchmarks against the base of pr, and display the performance changes with the head of the PR, with the link to the corresponding CodSpeed compare runs page

[codspeed]

@codspeedbot run the new benchmarks against the base of pr, and display the performance changes with the head of the PR, with the link to the corresponding CodSpeed compare runs page

Only members of the taskcluster organization on CodSpeed can ask the performance wizard for help here. Once you've joined it on CodSpeed, comment again.

[JohanLorenzo]

@codspeedbot run the new benchmarks against the base of pr, and display the performance changes with the head of the PR, with the link to the corresponding CodSpeed compare runs page

[codspeed]

@codspeedbot run the new benchmarks against the base of pr, and display the performance changes with the head of the PR, with the link to the corresponding CodSpeed compare runs page

I ran the PR's benchmark suite (test/test_graph_perf.py) on the PR base (064375f) and then on the PR head (55031b8). Both runs used CPU simulation mode in the same environment. On the base run I used the PR's version of test_graph_perf.py, minus the two graph._visit_order.cache_clear() lines, because _visit_order only exists on the head.

Compare runs page: https://app.codspeed.io/taskcluster/taskgraph/compare/6ab631c04cedfe38851b4dfa..6ab6348ca84b5010809f92e6
(base run: 6ab631c0, head run: 6ab6348c)

New benchmarks (base → head)

Benchmark	BASE (064375f)	HEAD (55031b8)	Change
⚡ test_create_tasks[btree]	12.6 s	1.9 s	×6.7
⚡ test_create_tasks[fan]	12.6 s	2.1 s	×6.1
⚡ test_resolve_timestamps	2.61 s	2.08 s	+25.6%
test_resolve_task_references	2.11 s	1.84 s	~+15% (CodSpeed counts this as unchanged)

Existing benchmarks, same pair of runs

Benchmark	BASE	HEAD	Change
⚡ test_for_each_task[diamond]	1,213 ms	28.1 ms	×43
⚡ test_for_each_task[fan]	91.4 ms	28.1 ms	×3.2
⚡ test_for_each_task[btree]	75.3 ms	28.1 ms	×2.7
⚡ test_for_each_task[linear]	75.3 ms	28.2 ms	×2.7
⚡ test_taskgraph_to_json[fan]	136.5 ms	73.6 ms	+85.5%
⚡ test_taskgraph_to_json[btree]	122.0 ms	73.3 ms	+66.5%
⚡ test_taskgraph_to_json[linear]	121.2 ms	73.4 ms	+65.2%

The other 17 benchmarks (links_dict, transitive_closure, visit_*order, transform_sequence) didn't change.

How to read these numbers

• These runs were in the CodSpeed sandbox, not CI. Absolute values are lower than in the PR report. For example, test_for_each_task[diamond] is 1.2 s here and 2.5 s in CI. Compare base and head within one table, not against the PR report. The ratios match what CI reported for the benchmarks both sets have.
• test_create_tasks spends most of its time in syscalls. Most of that is thread-pool waiting: about 11,000–12,600 s of system time on base, 2,000–2,700 s on head. Simulation mode leaves syscalls out of the measured value, so the ×6–7 figure covers CPU work only and understates the real cost on both sides. The system-time numbers point the same way: the head waits much less.
• The environment-difference warning on the compare page doesn't matter here. It comes from the two virtualenvs sitting at different paths. The libraries have identical build IDs.

I didn't change anything on the PR branch.

Summary

• Warming up the performance wizard...
• Inspect the new benchmarks and CI setup — Copied the PR's test/test_graph_perf.py onto the base commit (without the _visit_order.cache_clear() calls, which only exist on the PR head)
• Run the new benchmarks on the PR base (064375f) — Run 6ab631c04cedfe38851b4dfa
• Run the benchmarks on the PR head (55031b8) — Run 6ab6348ca84b5010809f92e6
• Compare runs and report — Compare runs and report
• Handling comment — Comment handling finished successfully

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}

[adriencaccia]

Hey @marco-c, @JohanLorenzo, the correct compare page URL is: https://app.codspeed.io/taskcluster/taskgraph/runs/compare/6ab631c04cedfe38851b4dfa..6ab6348ca84b5010809f92e6
We will deploy a fix to ensure that the agent will get it right.