← Experiment Card case studyExperiment Card · On page work sample
Experiment planning brief.
An illustrative brief for a proposed activation test. The actual baseline, sample size, and effect threshold must be set before launch.
Decision in front of the team
Should a clearer first step help new mobile visitors complete a second action?
- Customer behavior
- Complete a second meaningful action
- Cohort
- New mobile visitors
- Change
- One clearer first session message
- Hypothesis
- Clarity reduces the friction before value is found
- Measure
- Visitors completing the defined second action ÷ eligible visitors
- Review date
- Day 14 checkpoint. Wait for the planned sample before a final decision
- Owner
- Person bringing the result back to the team
- Notes
- Record what changed around the test before calling a winner
Decision rule
KeepKeep only if the preselected effect threshold and uncertainty criteria are met, with no guardrail deterioration.
RefineIf the planned sample is incomplete or the result is inconclusive, continue or redesign the test without declaring a winner.
StopStop if guardrails deteriorate, or if the completed test rules out a worthwhile improvement.
Planning principle
Agree on the review criteria before launch.
Choose the baseline, sample size, and worthwhile effect before launch. Keeping those choices fixed prevents an early fluctuation from becoming the reason to stop a test.
Before launch
- Define the exact second action and eligible visitors
- Assign eligible visitors to a control or changed message
- Set a baseline, minimum worthwhile effect, sample size, and guardrails before launch
- Name the owner and keep the decision rule fixed during the test