Apply rerun_with_latest_version to TriggerDagRunOperator reruns#67273
Open
nathadfield wants to merge 1 commit into
Open
Apply rerun_with_latest_version to TriggerDagRunOperator reruns#67273nathadfield wants to merge 1 commit into
rerun_with_latest_version to TriggerDagRunOperator reruns#67273nathadfield wants to merge 1 commit into
Conversation
c5f2e4e to
c17c7d6
Compare
When `TriggerDagRunOperator(reset_dag_run=True)` is invoked and the
target Dag run already exists, the task SDK falls back to
`client.clear(dag_id, run_id)`, which hits
`POST /execution/dag-runs/{dag_id}/{run_id}/clear`. That endpoint
previously called `dag.clear(run_id=run_id)` with no
`run_on_latest_version`, so the global and DAG-level
`rerun_with_latest_version` config (added in apache#63884) had no effect on
operator-driven reruns.
The endpoint now resolves the value via the existing
`resolve_run_on_latest_version` hierarchy (explicit > DAG-level >
global config > fallback) and forwards it to `dag.clear()` — matching
the behavior already in place for UI, public API, and CLI clears.
closes: apache#60880
c17c7d6 to
0900680
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
PR #63884 introduced a 4-tier
rerun_with_latest_versionconfig hierarchy (explicit > DAG-level > global config > fallback) controlling whether clearing/rerunning a Dag run uses the latest bundle version. That hierarchy is wired into the UI clear dialogs, the public clear/backfill API endpoints, and theairflow backfill createCLI command — but not intoTriggerDagRunOperator'sreset_dag_run=Truepath.When
TriggerDagRunOperator(reset_dag_run=True)is invoked and the target Dag run already exists, the task SDK trigger client gets a 409 and falls back toclient.clear(dag_id, run_id). That hitsPOST /execution/dag-runs/{dag_id}/{run_id}/clearon the execution API, which previously calleddag.clear(run_id=run_id)with norun_on_latest_versionargument. As a result, the global and DAG-levelrerun_with_latest_versionconfig had no effect on operator-driven reruns.What this PR does
Wires the existing
resolve_run_on_latest_versionresolver into the execution-APIclear_dag_runendpoint, so operator-driven reruns honor the same hierarchy as UI-, API-, and CLI-driven reruns.Noneis passed as the explicit override because the operator does not (intentionally) accept a per-call override — the DAG-level toggle and global config from #63884 are already sufficient, and adding an operator parameter that accepts arbitrary versions would create a security gap (a git bundle could be made to run a commit that was never on the tracking branch).Brand-new Dag runs from
TriggerDagRunOperatorare unaffected: they already use the latest bundle version because_create_orm_dagrunreadsDagModel.bundle_versionat trigger time. Only the rerun/clear path needed wiring.No payload/schema change; no client change; no Cadwyn version bump.
Test plan
test_dag_run_clear_invokes_resolververifies the resolver is invoked withNoneas the explicit override.TestDagRunCleartests still pass unchanged.prek run --from-ref apache/main --stage pre-commit— all hooks pass (ruff, mypy-airflow-core, etc.).closes: #60880
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.7) following the guidelines