Tool Evaluation
Part of Experimentation platforms
Checking traffic allocation and reporting in an experiment platform
Audit eligible traffic, variation splits, exposures, metric definitions and live changes before acting on an experiment report.
Check two settings before relying on an experiment report: the share of eligible traffic admitted to the experiment and the split between variations. Then compare the intended split with recorded exposures. A convincing result screen cannot resolve an assignment or logging problem.
Identify the denominator
Define who could enter the experiment, such as visitors to a named page who meet its audience rules. Record exclusions, the randomisation unit and when someone becomes eligible. An allocation percentage applies to that eligible population, not automatically to every visitor or customer.
Ask to see both the share admitted and the variation split. Optimizely Web distinguishes traffic allocation from variation distribution. LaunchDarkly also documents audience allocation and a separate variation split. Check the behaviour of the exact product and configuration offered.
Trace assignment into the report
Walk through an eligible visitor, a returning visitor and an excluded visitor. For each, ask which variation appears, whether exposure is recorded and how a later outcome is connected to it. A metric event alone does not establish that the person was exposed.
Inspect exposure counts by variation over time. Compare their proportions with the configured split, allowing for random variation. Investigate a meaningful mismatch before interpreting lift.
Statsig documents sample-ratio mismatch checks, and Optimizely's results page includes an experiment health indicator for this issue. An alert identifies a problem to investigate; its absence is not proof that instrumentation is correct.
Read beyond the lift figure
Ask for the primary metric's definition, numerator, denominator, attribution window and treatment of repeat events. Check the report's date range, segment and update timing. Read the estimated difference with its uncertainty and the underlying group values. A positive estimate by itself is not a release decision.
Set the metric and planned duration before launch. Statsig documents a power-analysis calculator using historical traffic and metric data to relate duration, detectable change and allocation. The team's own traffic and outcome rates determine whether a proposed experiment is feasible.
Key Metrics to Verify Before Acting on an Experiment Result
- Primary Metric Definition
- e.g., conversion rate (numerator: completed purchases / denominator: unique visitors)
- Planned Duration & Power Analysis
- Based on historical data and detectable change threshold
Record live changes
Ask what happens if the audience, traffic share, split or metric changes after launch, and log the time and reason. Optimizely warns that changing a live distribution can complicate interpretation.
In Optimizely Web, a distribution change generally affects new visitors while existing assignments remain; its Performance Edge product has different rebucketing behaviour. LaunchDarkly documents assignment rules tied to experiment iterations. Do not assume one vendor's rule applies to another product or mode.
Keep an auditable record of eligibility, allocation, split, exposures, metric definition, report range, warnings and live changes. Explain any missing item before using the result to support a decision.
