AIP-76: Propagate partition_date to consumers of partitioned assets#67285
Open
nathadfield wants to merge 1 commit into
Open
AIP-76: Propagate partition_date to consumers of partitioned assets#67285nathadfield wants to merge 1 commit into
nathadfield wants to merge 1 commit into
Conversation
f06372a to
27b7c87
Compare
Consumers of partitioned assets receive partition_key (str) but
partition_date (datetime) is None on the consumer DagRun, so templates
have to parse the key string. Propagate the datetime form alongside
the string so consumers can use the canonical filter idiom
`{{ partition_date | ds }}` and friends.
The producer's partition_date is stamped onto AssetEvent at emission;
the consumer's partition_date is computed alongside its target
partition_key at APDR creation (in assets/manager.py:_queue_partitioned_dags)
and stored on AssetPartitionDagRun. The scheduler copies apdr.partition_date
into the consumer DagRun. IdentityMapper passes the source date through;
the StartOf*Mapper family normalizes via to_downstream_normalized; other
mappers leave partition_date None and consumers fall back to partition_key.
closes: apache#67239
27b7c87 to
bb67204
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Makes
partition_datea first-class template variable on the consumer side of AIP-76 partitioned assets, so authors can write:instead of slicing strings out of
partition_key.How
AssetEvent.partition_dateis a new column (migration0116); the producer stamps it fromti.dag_run.partition_dateat emission, so the partition's datetime form rides every event the consumer sees.target_partition_keyinassets/manager.py:_queue_partitioned_dagsand frozen on a newAssetPartitionDagRun.partition_datecolumn. The scheduler simply copies that value into the consumer DagRun, so the date stays consistent with the mapper that produced the key even if mapper code or config is later edited.IdentityMapperpasses the source date through; theStartOf*Mapperfamily normalises via a new internalto_downstream_normalized(no public mapper-protocol change). Other mappers (ProductMapper,ChainMapper,AllowedKeyMapper, custom) leavepartition_date=Noneand consumers fall back topartition_key.partition_dateis output-only on trigger payloads, so a manual trigger cannot create an inconsistent(partition_key, partition_date)pair.Context["partition_date"](coerced to tz-aware), the execution-APIDagRun/AssetEventResponse/DagRunAssetReference(Cadwyn-versioned to strip the field for older Task SDK clients), the core-APIDAGRunResponse, OTel span attributes, and thestandardprovider'sPythonVirtualenvOperator/ExternalPythonOperatorserializable-context allow-list.Known limitations / follow-ups
ProductMapper,ChainMapper,AllowedKeyMapper, and custom mappers,partition_dateisNone. A publicto_downstream_partition_dateextension hook would close that gap but expands the mapper protocol — intentionally deferred.target_partition_keywith different normalised datetimes (e.g., two upstream assets with different timezone-configured mappers), the first event wins and a warning is logged in_get_or_create_apdr.closes: #67239
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.7) following the guidelines