When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," or "hypothesis." For tracking implementation, see analytics-tracking.
The A/B Test Setup skill guides marketers through planning, designing, and implementing controlled experiments to validate changes with statistical rigor. It ensures tests are framed around clear hypotheses, appropriate sample sizes, and measurable primary metrics to generate actionable insights. This skill covers selecting test types, defining variants, traffic allocation, and balancing test duration against statistical power and practical constraints.
This skill is essential for growth leads aiming to optimize conversion funnels with evidence-backed decisions, performance marketers running PPC landing page experiments, and agency strategists managing iterative client campaigns. It suits anyone responsible for designing tests that must fit within traffic limits, technical constraints, and business timelines while delivering clear, measurable outcomes.
Practitioners start by assessing the test context, baseline conversion rates, and technical or timeline constraints to define a focused hypothesis using a structured framework. Next, they select the appropriate test type—A/B, A/B/n, multivariate, or split URL—based on traffic volume and complexity. Then, they calculate sample size requirements and estimate test duration to ensure statistical significance and power. Finally, they design control and variant experiences, allocate traffic splits, and choose implementation methods like client- or server-side testing before launching and monitoring the test.
How do I decide between A/B and multivariate testing? Choose A/B if you have limited traffic or want to isolate a single change; multivariate requires much higher traffic and tests multiple changes simultaneously. What minimum detectable effect (MDE) should I target? It depends on business impact but typically aims for a lift of 10–20% to balance sample size feasibility and meaningful results. How long should I run the test? Minimum one to two business cycles (usually 1–2 weeks) to capture normal user behavior and avoid novelty or external factor bias.
Attach the A/B Test Setup skill to any Metaflow agent task where planning or executing an experiment is involved. Expect detailed guidance on hypothesis formulation, sample size calculation, variant design, and traffic allocation to support statistically valid testing. This skill integrates with other analytics and tracking skills to complete your experimentation workflow and provide end-to-end test management.
For broader context, see our roundup of claude skills marketing, and read Claude Code workflows for marketing agencies for related setup guidance.