Conversation
47926aa to
4c8e1e2
Compare
|
@themis-blindfold review |
| identifier=identifier, | ||
| value=flag.variant, | ||
| traits=traits, | ||
| metadata={"experiment_id": flag.experiment.id}, |
There was a problem hiding this comment.
🟠 Major · ⚡ Quick win
Different experiments can be deduplicated as one exposure
Observed: the event buffer keys an exposure only by feature, identifier, and variant, so the added experiment_id is not considered. Predicted: if a new experiment assigns the same variant to this identity before the current buffer flushes, its first exposure would be dropped and never attributed. Include the experiment ID in ExposureKey and add a regression case with two IDs.
There was a problem hiding this comment.
That's very edgy in the sense that the flush window is ~10s. Even if a user belongs to several experiments, the feature id is different so the dedup would still happen
There was a problem hiding this comment.
Agreed. feature_name is already part of the deduplication key, so concurrent experiments on different features remain distinct (flagsmith/analytics.py:86, flagsmith/analytics.py:173). The remaining same-feature handover within one flush window is too edge-case to support this Major finding.
⚖️ Themis review: 🟠 Fix before mergeThe experiment metadata is parsed and sent on exposure events as intended, but changing experiments can lose an exposure when the same identity receives the same variant before the event buffer flushes. CI completed successfully on Python 3.10–3.13 and pre-commit.
🟠 Majors
📝 Walkthrough
🧪 How to verify
Product take: This is a solid experimentation improvement, but losing the first exposure after a rapid experiment change would skew attribution for that experiment. 🧭 Assumptions & unverified claimsNo unverified assumptions or claims. One experiment needs its own receipt, even when it reuses the same variant · reviewed at 4c8e1e2 |
Changes
Core now sends
metadata.experiment { id, name, in_experiment }on/identitiesflags while the feature's experiment is running (Flagsmith/flagsmith#8532).Flag.experiment: Optional[ExperimentMetadata], a frozen dataclass populated inFlag.from_api_flag.Noneon older servers and in local evaluation.get_experiment_flagfires the exposure only whenexperiment.in_experimentis true, instead of guessing fromvariant.experiment_idfor attribution.How did you test this code?
pytest: 128 passing, new cases in
tests/test_models.pyandtests/test_flagsmith.py.pre-commit clean.
Manual QA against staging once Core and Edge are deployed.