Marketing analytics
Illustrative sample · Synthetic data
Campaign conversion readout
Compare two signup rates and decide whether the difference is clear enough to recommend a change.
KPI definitions · Conversion analysis · Experiment interpretation · Recommendations
A synthetic two variant example prepared for this portfolio. It assumes independent randomized assignment and a single planned analysis. These are not Romin campaign results.
Terms explained
- Conversion rate
- The share of eligible visitors who complete the action being measured. This exercise measures signups.
- Percentage point and relative lift
- Moving from 5% to 6% is an increase of 1 percentage point. Compared with the starting 5%, it is a relative increase of 20%.
- Uncertainty interval
- A range used to express how uncertain the estimated difference is under the test’s assumptions. A range containing zero does not establish a clear improvement.
A higher rate. An uncertain result.
Signup rate · Scale from 0% to 10%
1 percentage point higher in this sample
The approximate 95% range crosses zero, from negative 0.82 to positive 2.82 percentage points. It leaves room for either version to perform better.
Both groups contain 1,200 visitors. The bars show observed signup rates. The dot below marks the observed difference and the line shows its approximate 95% uncertainty range.
Scroll horizontally to compare all columns.
| Variant | Assigned visitors | Signups | Conversion |
|---|---|---|---|
| A · Comparison | 1,200 | 60 | 5.0% |
| B · New message | 1,200 | 72 | 6.0% |
| Observed difference | Same visitor counting rule | 12 more signups | +1.0 percentage point |
Recommendation
Report an inconclusive result at this planned readout.
B’s observed rate is 20% higher relative to A, but the approximate 95% interval for B minus A is negative 0.82 to +2.82 percentage points. It includes zero. This example does not establish a reliable improvement.
Count the same kind of visitor in each group
The primary metric is signups divided by assigned eligible visitors within the agreed window. The descriptive difference is 6% minus 5% = 1 percentage point. Relative lift is (6% minus 5%) ÷ 5% = 20%.
Show the uncertainty
The calculation estimates a 95% uncertainty range for the difference between the two signup rates. It uses the standard error for two independent proportions and a multiplier of 1.96. Each group has at least 60 signups and 1,128 visitors who did not sign up. The download includes the calculation. The method assumes independent random assignment and one planned analysis.
Make the next step explicit
Document the result and review tracking and audience balance. If the question remains valuable, design a follow up test with a business relevant minimum effect, planned sample size, guardrails, and a fixed decision rule.
- Do not extend the test solely until a favorable result appears.
- Check signup quality and unintended effects.
- Separate “inconclusive” from evidence that the variants are equivalent.
Learning note
Takeaway
The direction of an observed difference and the strength of the evidence are separate questions. Report both before recommending a change.
Related experience
Romin case study ↗Explore the related project for its context, approach, and outcome.
Files and calculations.
For the full set of sample briefs, CSVs, SQL, verification code, and 14 role briefs, download the collection ↓.
Sources & methods: Penn State: inference for two proportions ↗