A/B test template
Observation, hypothesis, primary metric, sample size, and the decision rule — all written before the test starts.
- GA4
- Notion
- Linear
- Google Sheets
We think the pricing question is killing our demo form. Write an A/B test brief.
- Pulled baseline dataForm start rate 4.1%; 38% of exits happen at the pricing section
- Wrote the hypothesisIf pricing context moves above the form, then starts rise, because visitors leave to find it
- Computed sample size2,900 sessions per arm for a 20% relative lift — 16 days
- Set the decision ruleScale above +15%, iterate between 0 and +15%, kill below 0
Test is powered: 2,900 sessions per arm over roughly 16 days at current traffic. Detecting anything smaller than a 20% relative lift would take two months, so the brief targets the larger effect and pre-registers the three-way decision rule.
An A/B test template earns its keep by settling the argument in advance. This one converts a conversion problem into a testable hypothesis, computes the sample size your traffic can actually support, and records the decision rule before launch — so an ambiguous result cannot be reinterpreted after the fact.
What is an A/B test template?
A pre-registration document for one experiment. It states the observation that prompted it, the hypothesis in if/then form, the change being made, the primary metric, the sample size and duration required, and the rule that decides what each possible outcome means.
Turn a problem into a hypothesis
- State the observation and the data behind it
- Write the hypothesis in if/then form with a mechanism
- Name the specific change being tested
- Identify what would falsify it
Size the test honestly
- Compute sample size from your baseline conversion rate
- Set the minimum detectable effect and duration
- Flag tests your traffic cannot power
Pre-register the decision
- Define what result scales, iterates, or kills the change
- Name the primary metric and the guardrail metrics
- Record the result and mechanism for the archive
How do you write a testable A/B test hypothesis?
In if/then form with a mechanism: if we move the pricing objection above the form, then form starts will rise, because visitors currently leave to hunt for pricing. Without the mechanism you cannot learn anything transferable from a win.
Why does the experiment brief template compute sample size first?
Because most tests on B2B traffic are underpowered, and an underpowered test does not produce a weak answer — it produces a random one. Computing the minimum detectable effect up front tells you whether the test is worth running at all.
What makes test documentation worth keeping?
A record of the hypothesis and the result, including the losses. Teams that document only winners rerun the same failed idea every eighteen months, and the mechanism behind a loss is frequently more useful than the win.
How the A/B test template works across your stack
The agent reads baseline conversion rates from GA4 or your analytics to size the test, and pulls the observation from your funnel or session data where available. The brief exports to Notion or Linear as the pre-registration record, and deployment stays in your testing tool where QA and rollback live.
- GA4
- Notion
- Linear
- Google Sheets
Who uses this A/B test template
- CRO specialists
- Pre-register decisions so results are not renegotiated afterwards.
- Growth teams
- Find out before launch that a test would need nine months of traffic.
- Product marketers
- Document why a message lost, not just which one won.
How to run this A/B test template in Metaflow
Describe the observation
What you noticed and where — a drop-off, a session recording pattern, a support complaint.
Let the agent form the hypothesis
It writes the if/then with a mechanism and names what would falsify it.
Review the sizing
Sample size, duration, and minimum detectable effect from your real baseline rate.
Approve the decision rule and launch
Pre-register the rule, run the test in your tool, then record the outcome against the brief.
What you provide
- The observation and supporting data
- Baseline conversion rate or analytics access
- Traffic volume on the page
What you get back
- Hypothesis in if/then form with mechanism
- Primary and guardrail metrics
- Sample size, duration, and minimum detectable effect
- Pre-registered decision rule
Why use this A/B test template?
Decision rule is written before launch, which prevents post-hoc reinterpretation
Sample size comes from your real baseline, not a rule of thumb
Declines to schedule tests your traffic cannot power
Archives losses with their mechanism, so ideas are not rerun blindly
A/B test template FAQs
What should an A/B test hypothesis include?
The change, the expected effect, and the mechanism — in if/then/because form. The mechanism is what makes a win reusable on other pages instead of a one-off result.
How long should an A/B test run?
Until it reaches the pre-computed sample size, and at least one full business cycle so weekday and weekend behaviour are both represented. Stopping when the result looks good is the most common way to produce a false positive.
How many variants should you test?
Two, unless your traffic can power more. Every additional arm splits the sample and extends the test, and most B2B pages do not have the volume for a three-way split.
What if the test is inconclusive?
That is a valid outcome and the decision rule should already cover it — usually keep the control and move on. Inconclusive tests on small effects are the normal case, not a failure of process.
Similar landing page copy and CRO templates
Key takeaways
- Write the decision rule before you launch, not after you see the numbers
- Size from your real baseline; underpowered tests give random answers
- Archive losses with their mechanism