Reference for calculating sample sizes and test duration. Sample Size Fundamentals (required inputs, what these mean) Sample Size Quick Reference Tables Duratio
The Sample Size Guide provides clear methods to calculate the number of visitors needed per test variant and the estimated test duration for conversion rate optimization experiments. It breaks down core inputs like baseline conversion rate, minimum detectable effect (MDE), significance level, and statistical power, and offers quick reference tables for sample sizes at common conversion rates and lift targets. The guide also includes formulas and examples to estimate how long tests will run given traffic volume and test exposure, helping marketers plan realistic experiments.
It addresses adjustments for multiple variants, common pitfalls such as underpowered tests and incorrect baselines, and offers a decision framework for when sample size demands exceed traffic capacity. This resource supports precise test planning to avoid wasted time or inconclusive results.
This skill is designed for performance marketers and CRO specialists who routinely run A/B tests or multivariate experiments on websites or landing pages. Growth leads and agency strategists managing multiple client tests benefit from understanding realistic sample size requirements before launching experiments. It also suits PPC operators needing to interpret test results with confidence, ensuring that observed lifts are statistically valid rather than noise.
Anyone responsible for test design, prioritization, or duration estimates on medium- to low-traffic properties will find this guide especially useful.
A typical workflow begins with defining the baseline conversion rate and the minimum lift worth detecting, considering business impact and implementation cost. Next, practitioners consult the quick reference tables to identify the required sample size per variant for their chosen MDE and significance level. Then, using the sample size and daily traffic volume, they calculate the expected test duration, adjusting for exposure percentage and minimum duration rules like full week coverage to capture cyclical behavior.
If testing multiple variants, marketers apply sample size multipliers to maintain statistical rigor. Finally, they review potential common mistakes—such as overly optimistic MDEs or neglecting segment sizes—and adjust the test plan accordingly to ensure the experiment is adequately powered and efficiently timed.
How do I choose the minimum detectable effect? Set it based on what lift would justify the test’s cost and what previous tests suggest is realistic. What if my traffic is too low to reach the sample size? Consider increasing the MDE to detect only larger effects or focus testing on higher-traffic segments. How long should I run my test? At minimum, one full week to cover day-of-week variation, but avoid exceeding 4–8 weeks to reduce external factor impacts.
Attach the Sample Size Guide skill to any Metaflow agent tasked with planning or evaluating A/B tests to get immediate calculations on sample requirements and estimated durations. The agent can leverage the quick reference tables and formulas to provide actionable timelines and sample sizes tailored to baseline conversion and traffic inputs. Expect clear guidance on test feasibility and adjustments for multiple variants, helping you align testing strategy with business constraints. This skill integrates seamlessly with other CRO and growth planning tasks to optimize experimentation workflows.
For broader context, see our roundup of marketing skills claude, and read common Claude Code content mistakes for related setup guidance.