Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions docs/docs/experimentation-ab-testing.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Experimentation (A/B Testing)
sidebar_label: Experimentation (A/B Testing)
title: A/B Testing with Integrations
sidebar_label: A/B Testing with Integrations
sidebar_position: 4
---

Expand All @@ -25,8 +25,7 @@ being tested. The analytics platform will receive a stream of event data derived
Combining these two concepts allows you to deliver seamless A/B test.

We have [integrations](/third-party-integrations/analytics/segment) with a number of analytics platforms. If we don't
integrate with the platform you are using, you can still manually send the test data to the downstream platform
manually.
integrate with the platform you are using, you can still send test data to the downstream platform.

By the end of this tutorial, you will be able to:

Expand Down
4 changes: 4 additions & 0 deletions docs/docs/experimentation/analyse-an-experiment.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,12 @@ The analysis table breaks each variation down against control:
While a variation has fewer than 50 identities (or fewer than 5 conversions, for occurrence metrics), it shows
**Collecting data**. Keep the experiment running.

:::warning Sample ratio mismatch

If a **sample ratio mismatch** warning appears, traffic didn't split the way you configured. Don't act on the results.

:::

## Exposures

The **Exposures** panel charts enrolment over time, per variation, with each variation's share of the total.
Expand Down
17 changes: 12 additions & 5 deletions docs/docs/experimentation/connect-a-warehouse.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ warehouse connection, and it must be in place before you can create experiments.

## Connect the Flagsmith warehouse

The **Flagsmith** warehouse is managed and hosted by Flagsmith. There is nothing to provision or configure.
The **Flagsmith** warehouse is managed and hosted by Flagsmith. There is nothing to provision or configure, and there is
no additional cost — it is included in your Enterprise plan.

1. Go to **Environment Settings > Warehouse**.
2. Select **Flagsmith** and click **Enable**, then confirm.
Expand All @@ -26,11 +27,17 @@ The **Flagsmith** warehouse is managed and hosted by Flagsmith. There is nothing
The connection is created immediately. To verify that events can flow, click **Send your first event** and Flagsmith
sends a test event on your behalf. The connection shows **Pending Connection** while it waits for the first event, and
switches to **Connected** as soon as the environment's first event arrives; an event sent from your own application
counts too, not just the test event. Processing the first event can take up to a few hours, and the status refreshes
about once a minute while you keep the Warehouse tab open.
counts too, not just the test event.

:::note

Processing the first event can take up to a few hours. The status refreshes about once a minute while you keep the
Warehouse tab open.

:::

Once connected, the warehouse card shows the total number of events received and the number of unique event names, which
is useful for checking that your instrumentation is arriving.
is useful to verify that instrumentation data is arriving.

<!-- Screenshot: connected warehouse card with event stats -->

Expand All @@ -41,7 +48,7 @@ The current status is shown on the warehouse card in **Environment Settings > Wa
- **Created**: the connection exists but no events have been received yet.
- **Pending Connection**: waiting for the first event to arrive.
- **Connected**: events have been received; you are ready to run experiments.
- **Errored**: something went wrong; contact support.
- **Errored**: something went wrong; [contact support](/support/).

## Coming soon

Expand Down
3 changes: 2 additions & 1 deletion docs/docs/experimentation/create-an-experiment.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,4 +63,5 @@ edited from the experiment page.

:::

A running experiment can be **ended** at any time. Its results are frozen as final, and this cannot be undone.
A running experiment can be **ended** at any time. Once ended, the results are frozen as final and the experiment cannot
be restarted — you would need to create a new one.
6 changes: 3 additions & 3 deletions docs/docs/experimentation/create-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.f
A metric defines an outcome you measure, computed from the events your application sends to the warehouse. Metrics are
defined per environment and can be reused across experiments.

To create one, go to the **Metrics** page and click **Create Metric**, or click **Create Metric** in the experiment
wizard's Measurement step.
To create one, go to the **Metrics** page and click **Create Metric**. Alternatively, you can create a metric from
inside the experiment wizard's Measurement step.

<!-- Screenshot: Create Metric form -->
![The Create Metric form with name, description, aggregation type, direction and event name fields](/img/experimentation/create-metric-form.png)

## Fields

Expand Down
15 changes: 11 additions & 4 deletions docs/docs/experimentation/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ Flagsmith Experimentation lets you run A/B tests end to end on the platform: ser
[multivariate flag](/managing-flags/core-management), collect events from your application into a managed data
warehouse, and read the results with a built-in Bayesian statistics engine.

Experiments are scoped to an **environment**: the warehouse connection, metrics and experiments all live at the
environment level.
Experiments are scoped to an [**environment**](/flagsmith-concepts/data-model#environments).

## Set up and run an experiment

Expand All @@ -33,9 +32,17 @@ environment level.

Prefer to see it end to end? Follow the [PayPal button example](/experimentation/example-paypal-button).

Unfamiliar with a term? The [Statistics Glossary](/experimentation/statistics) explains every concept in plain language.
Unfamiliar with a term? See the [Statistics Glossary](/experimentation/statistics).

:::note SDK availability

Event collection is currently available in the **JavaScript** and **Python** SDKs, with more to follow. Check
[Run an Experiment](/experimentation/run-an-experiment) for instrumentation details.

:::

## A/B testing without the Experimentation feature

You can also run A/B tests with multivariate flags and your own analytics platform. See
[Experimentation (A/B Testing)](/experimentation-ab-testing).
[A/B Testing with Integrations](/experimentation-ab-testing). The self-managed approach remains fully supported and is
not going away.
8 changes: 4 additions & 4 deletions docs/docs/experimentation/run-an-experiment.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ jobs:

## Instrument your application

Event collection is currently available in the **JavaScript** and **Python** SDKs, with more to follow. Enable it with
the `enableEvents` / `enable_events` option.
Event collection is currently available in the **JavaScript** and **Python** SDKs, with more to follow. Enable it by
passing the `enableEvents` / `enable_events` option when you initialise the client.

Users must be [identified](/flagsmith-concepts/identities). Exposures and conversion events are joined per identity, so
use the same identifier for flags and events.
Expand Down Expand Up @@ -90,8 +90,8 @@ exposure more than once is safe.

`trackEvent` / `track_event` sends a named event, optionally with a numeric `value`, traits and metadata.

Any event whose name matches a metric's event name feeds that metric. Conversions are attributed post-exposure: an
identity's events only count after its first exposure.
The event name is **case-sensitive** and must exactly match the metric's configured event name. Unmatched events are
silently ignored. Conversions are attributed post-exposure: an identity's events only count after its first exposure.

## While it runs

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading