diff --git a/docs/docs/experimentation-ab-testing.md b/docs/docs/experimentation-ab-testing.md index 646fa0fb6870..e9c49e59d204 100644 --- a/docs/docs/experimentation-ab-testing.md +++ b/docs/docs/experimentation-ab-testing.md @@ -1,6 +1,6 @@ --- -title: Experimentation (A/B Testing) -sidebar_label: Experimentation (A/B Testing) +title: A/B Testing with Integrations +sidebar_label: A/B Testing with Integrations sidebar_position: 4 --- @@ -25,8 +25,7 @@ being tested. The analytics platform will receive a stream of event data derived Combining these two concepts allows you to deliver seamless A/B test. We have [integrations](/third-party-integrations/analytics/segment) with a number of analytics platforms. If we don't -integrate with the platform you are using, you can still manually send the test data to the downstream platform -manually. +integrate with the platform you are using, you can still send test data to the downstream platform. By the end of this tutorial, you will be able to: diff --git a/docs/docs/experimentation/analyse-an-experiment.md b/docs/docs/experimentation/analyse-an-experiment.md index acccc526a182..aa102f4a75ce 100644 --- a/docs/docs/experimentation/analyse-an-experiment.md +++ b/docs/docs/experimentation/analyse-an-experiment.md @@ -40,8 +40,12 @@ The analysis table breaks each variation down against control: While a variation has fewer than 50 identities (or fewer than 5 conversions, for occurrence metrics), it shows **Collecting data**. Keep the experiment running. +:::warning Sample ratio mismatch + If a **sample ratio mismatch** warning appears, traffic didn't split the way you configured. Don't act on the results. +::: + ## Exposures The **Exposures** panel charts enrolment over time, per variation, with each variation's share of the total. diff --git a/docs/docs/experimentation/connect-a-warehouse.md b/docs/docs/experimentation/connect-a-warehouse.md index 29b5a91fe013..2bd01eb3d13e 100644 --- a/docs/docs/experimentation/connect-a-warehouse.md +++ b/docs/docs/experimentation/connect-a-warehouse.md @@ -16,7 +16,8 @@ warehouse connection, and it must be in place before you can create experiments. ## Connect the Flagsmith warehouse -The **Flagsmith** warehouse is managed and hosted by Flagsmith. There is nothing to provision or configure. +The **Flagsmith** warehouse is managed and hosted by Flagsmith. There is nothing to provision or configure, and there is +no additional cost — it is included in your Enterprise plan. 1. Go to **Environment Settings > Warehouse**. 2. Select **Flagsmith** and click **Enable**, then confirm. @@ -26,11 +27,17 @@ The **Flagsmith** warehouse is managed and hosted by Flagsmith. There is nothing The connection is created immediately. To verify that events can flow, click **Send your first event** and Flagsmith sends a test event on your behalf. The connection shows **Pending Connection** while it waits for the first event, and switches to **Connected** as soon as the environment's first event arrives; an event sent from your own application -counts too, not just the test event. Processing the first event can take up to a few hours, and the status refreshes -about once a minute while you keep the Warehouse tab open. +counts too, not just the test event. + +:::note + +Processing the first event can take up to a few hours. The status refreshes about once a minute while you keep the +Warehouse tab open. + +::: Once connected, the warehouse card shows the total number of events received and the number of unique event names, which -is useful for checking that your instrumentation is arriving. +is useful to verify that instrumentation data is arriving. @@ -41,7 +48,7 @@ The current status is shown on the warehouse card in **Environment Settings > Wa - **Created**: the connection exists but no events have been received yet. - **Pending Connection**: waiting for the first event to arrive. - **Connected**: events have been received; you are ready to run experiments. -- **Errored**: something went wrong; contact support. +- **Errored**: something went wrong; [contact support](/support/). ## Coming soon diff --git a/docs/docs/experimentation/create-an-experiment.md b/docs/docs/experimentation/create-an-experiment.md index 68f6bdd05272..890f0f7c1357 100644 --- a/docs/docs/experimentation/create-an-experiment.md +++ b/docs/docs/experimentation/create-an-experiment.md @@ -63,4 +63,5 @@ edited from the experiment page. ::: -A running experiment can be **ended** at any time. Its results are frozen as final, and this cannot be undone. +A running experiment can be **ended** at any time. Once ended, the results are frozen as final and the experiment cannot +be restarted — you would need to create a new one. diff --git a/docs/docs/experimentation/create-metrics.md b/docs/docs/experimentation/create-metrics.md index cfa0b5a69b6b..05273517f57a 100644 --- a/docs/docs/experimentation/create-metrics.md +++ b/docs/docs/experimentation/create-metrics.md @@ -14,10 +14,10 @@ Experimentation is in beta on **Enterprise** plans. [Get in touch](https://www.f A metric defines an outcome you measure, computed from the events your application sends to the warehouse. Metrics are defined per environment and can be reused across experiments. -To create one, go to the **Metrics** page and click **Create Metric**, or click **Create Metric** in the experiment -wizard's Measurement step. +To create one, go to the **Metrics** page and click **Create Metric**. Alternatively, you can create a metric from +inside the experiment wizard's Measurement step. - +![The Create Metric form with name, description, aggregation type, direction and event name fields](/img/experimentation/create-metric-form.png) ## Fields diff --git a/docs/docs/experimentation/index.md b/docs/docs/experimentation/index.md index 02e4553bddc6..c27a1646b33a 100644 --- a/docs/docs/experimentation/index.md +++ b/docs/docs/experimentation/index.md @@ -15,8 +15,7 @@ Flagsmith Experimentation lets you run A/B tests end to end on the platform: ser [multivariate flag](/managing-flags/core-management), collect events from your application into a managed data warehouse, and read the results with a built-in Bayesian statistics engine. -Experiments are scoped to an **environment**: the warehouse connection, metrics and experiments all live at the -environment level. +Experiments are scoped to an [**environment**](/flagsmith-concepts/data-model#environments). ## Set up and run an experiment @@ -33,9 +32,17 @@ environment level. Prefer to see it end to end? Follow the [PayPal button example](/experimentation/example-paypal-button). -Unfamiliar with a term? The [Statistics Glossary](/experimentation/statistics) explains every concept in plain language. +Unfamiliar with a term? See the [Statistics Glossary](/experimentation/statistics). + +:::note SDK availability + +Event collection is currently available in the **JavaScript** and **Python** SDKs, with more to follow. Check +[Run an Experiment](/experimentation/run-an-experiment) for instrumentation details. + +::: ## A/B testing without the Experimentation feature You can also run A/B tests with multivariate flags and your own analytics platform. See -[Experimentation (A/B Testing)](/experimentation-ab-testing). +[A/B Testing with Integrations](/experimentation-ab-testing). The self-managed approach remains fully supported and is +not going away. diff --git a/docs/docs/experimentation/run-an-experiment.mdx b/docs/docs/experimentation/run-an-experiment.mdx index 6447db1fa748..aaf35654b378 100644 --- a/docs/docs/experimentation/run-an-experiment.mdx +++ b/docs/docs/experimentation/run-an-experiment.mdx @@ -22,8 +22,8 @@ jobs: ## Instrument your application -Event collection is currently available in the **JavaScript** and **Python** SDKs, with more to follow. Enable it with -the `enableEvents` / `enable_events` option. +Event collection is currently available in the **JavaScript** and **Python** SDKs, with more to follow. Enable it by +passing the `enableEvents` / `enable_events` option when you initialise the client. Users must be [identified](/flagsmith-concepts/identities). Exposures and conversion events are joined per identity, so use the same identifier for flags and events. @@ -90,8 +90,8 @@ exposure more than once is safe. `trackEvent` / `track_event` sends a named event, optionally with a numeric `value`, traits and metadata. -Any event whose name matches a metric's event name feeds that metric. Conversions are attributed post-exposure: an -identity's events only count after its first exposure. +The event name is **case-sensitive** and must exactly match the metric's configured event name. Unmatched events are +silently ignored. Conversions are attributed post-exposure: an identity's events only count after its first exposure. ## While it runs diff --git a/docs/static/img/experimentation/create-metric-form.png b/docs/static/img/experimentation/create-metric-form.png new file mode 100644 index 000000000000..ce3615f0643b Binary files /dev/null and b/docs/static/img/experimentation/create-metric-form.png differ