Skip to content

feat: Flag.experiment and exposure gated on in_experiment - #249

Open
Zaimwa9 wants to merge 1 commit into
mainfrom
feat/experiment-metadata
Open

Zaimwa9 wants to merge 1 commit into
mainfrom
feat/experiment-metadata

Conversation

@Zaimwa9

@Zaimwa9 Zaimwa9 commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Changes

Core now sends metadata.experiment { id, name, in_experiment } on /identities flags while the feature's experiment is running (Flagsmith/flagsmith#8532).

  • Flag.experiment: Optional[ExperimentMetadata], a frozen dataclass populated in Flag.from_api_flag. None on older servers and in local evaluation.
  • get_experiment_flag fires the exposure only when experiment.in_experiment is true, instead of guessing from variant.
  • Exposure events carry experiment_id for attribution.

How did you test this code?

pytest: 128 passing, new cases in tests/test_models.py and tests/test_flagsmith.py.
pre-commit clean.
Manual QA against staging once Core and Edge are deployed.

@Zaimwa9
Zaimwa9 force-pushed the feat/experiment-metadata branch from 47926aa to 4c8e1e2 Compare September 15, 2026 17:11
@Zaimwa9
Zaimwa9 marked this pull request as ready for review September 18, 2026 13:20
@Zaimwa9
Zaimwa9 requested a review from a team as a code owner September 18, 2026 13:20
@Zaimwa9
Zaimwa9 requested review from gagantrivedi and matthewelwell and removed request for a team and matthewelwell September 18, 2026 13:20
@Zaimwa9

Zaimwa9 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

@themis-blindfold review

Comment thread flagsmith/flagsmith.py
identifier=identifier,
value=flag.variant,
traits=traits,
metadata={"experiment_id": flag.experiment.id},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Major · ⚡ Quick win

Different experiments can be deduplicated as one exposure

Observed: the event buffer keys an exposure only by feature, identifier, and variant, so the added experiment_id is not considered. Predicted: if a new experiment assigns the same variant to this identity before the current buffer flushes, its first exposure would be dropped and never attributed. Include the experiment ID in ExposureKey and add a regression case with two IDs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's very edgy in the sense that the flush window is ~10s. Even if a user belongs to several experiments, the feature id is different so the dedup would still happen

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. feature_name is already part of the deduplication key, so concurrent experiments on different features remain distinct (flagsmith/analytics.py:86, flagsmith/analytics.py:173). The remaining same-feature handover within one flush window is too edge-case to support this Major finding.

@themis-blindfold

Copy link
Copy Markdown

⚖️ Themis review: 🟠 Fix before merge

The experiment metadata is parsed and sent on exposure events as intended, but changing experiments can lose an exposure when the same identity receives the same variant before the event buffer flushes. CI completed successfully on Python 3.10–3.13 and pre-commit.

Area Score
🎯 Correctness 3/5
🧪 Test coverage 3/5
📐 Code quality 4/5
🚀 Product impact 3/5

🟠 Majors

  • flagsmith/flagsmith.py:404 — Exposure deduplication can discard the first event for a new experiment.
📝 Walkthrough
  • Identity flag parsing - maps the optional Core experiment payload onto Flag.experiment.
  • Experiment exposure - sends only for enrolled identities and includes the experiment ID for attribution.
  • Event buffering - still identifies duplicate exposures without the new experiment ID, which loses attribution across experiment changes.
🧪 How to verify
  1. Run pytest -q tests/test_models.py tests/test_flagsmith.py tests/test_event_processor.py.
  2. Call get_experiment_flag twice before an event flush for the same identity, feature, and variant, using different experiment IDs.
  3. Assert that both buffered events remain and retain their distinct metadata.experiment_id values.
  4. Confirm repeated calls for the same experiment ID still produce one buffered exposure.
    Automate: add the differing-experiment-ID case to tests/test_event_processor.py.

Product take: This is a solid experimentation improvement, but losing the first exposure after a rapid experiment change would skew attribution for that experiment.

🧭 Assumptions & unverified claims

No unverified assumptions or claims.

One experiment needs its own receipt, even when it reuses the same variant · reviewed at 4c8e1e2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant