A/B/n Test – multiple variants, higher traffic required
Multivariate Test (MVT) – interaction effects, very high traffic
Split URL Test – major structural changes
Default to A/B unless there is a clear reason otherwise.
6️⃣ Metrics Definition
Primary Metric (Mandatory)
Single metric used to evaluate success
Directly tied to the hypothesis
Pre-defined and frozen before launch
Secondary Metrics
Provide context
Explain why results occurred
Must not override the primary metric
Guardrail Metrics
Metrics that must not degrade
Used to prevent harmful wins
Trigger test stop if significantly negative
7️⃣ Sample Size & Duration
Define upfront:
Baseline rate
MDE
Significance level (typically 95%)
Statistical power (typically 80%)
Estimate:
Required sample size per variant
Expected test duration
Do NOT proceed without a realistic sample size estimate.
Tracking Verification (Required before Gate 8)
Before entering the Execution Readiness Gate below, run through this checklist to make "Tracking is verified" mean something concrete:
Event firing: Trigger each event the primary and secondary metrics depend on (sign-up, add-to-cart, custom event) on staging or a debug page, and confirm it lands in your analytics destination within 30 seconds.
Variant attribution: Verify that the variant assignment ID is attached to every fired event — not just the entry event. Use your analytics' raw event view to compare a sample of 5+ events per variant.
De-duplication: Confirm that a user reloading the page does not cause double-counted events. If your stack uses client-side de-duping, the variant ID must be part of the dedup key.
Sample randomization: Pull the first 100 assignment records from your assignment table; the variant split should be within ±5% of the configured allocation.
Guardrail metric pipeline: Each guardrail metric defined in §6️⃣ must have a working dashboard or alert by the time the test launches.
If any of the above fails, stop and resolve it before Gate 8.
8️⃣ Execution Readiness Gate (Hard Stop)
You may proceed to implementation only if all are true:
Hypothesis is locked
Primary metric is frozen
Sample size is calculated
Test duration is defined
Guardrails are set
Tracking is verified
If any item is missing, stop and resolve it.
Running the Test
During the Test
DO:
Monitor technical health
Document external factors
DO NOT:
Stop early due to “good-looking” results
Change variants mid-test
Add new traffic sources
Redefine success criteria
Analyzing Results
Analysis Discipline
When interpreting results:
Do NOT generalize beyond the tested population
Do NOT claim causality beyond the tested change
Do NOT override guardrail failures
Separate statistical significance from business judgment
Interpretation Outcomes
Result
Action
Significant positive
Consider rollout
Significant negative
Reject variant, document learning
Inconclusive
Consider more traffic or bolder change
Guardrail failure
Do not ship, even if primary wins
Documentation & Learning
Test Record (Mandatory)
Document:
Hypothesis
Variants
Metrics
Sample size vs achieved
Results
Decision
Learnings
Follow-up ideas
Store records in a shared, searchable location to avoid repeated failures.
Refusal Conditions (Safety)
Refuse to proceed if:
Baseline rate is unknown and cannot be estimated
Traffic is insufficient to detect the MDE
Primary metric is undefined
Multiple variables are changed without proper design
Hypothesis cannot be clearly stated
Explain why and recommend next steps.
Key Principles (Non-Negotiable)
One hypothesis per test
One primary metric
Commit before launch
No peeking
Learning over winning
Statistical rigor first
Final Reminder
A/B testing is not about proving ideas right.
It is about learning the truth with confidence.
If you feel tempted to rush, simplify, or “just try it” —
that is the signal to slow down and re-check the design.
When to Use
This skill is applicable to execute the workflow or actions described in the overview.
Limitations
Use this skill only when the task clearly matches the scope described above.
Do not treat the output as a substitute for environment-specific validation, testing, or expert review.
Stop and ask for clarification if required inputs, permissions, safety boundaries, or success criteria are missing.
