# Campaign conversion readout

Richie Lin · Portfolio sample · September 2026

Illustrative sample · Synthetic data

A synthetic two variant example prepared for this portfolio. It assumes independent randomized assignment and a single planned analysis. These are not Romin campaign results.

## The brief

Six percent of visitors sign up after seeing version B, compared with five percent for A. Is that enough to call B the winner?

## Worked example

Synthetic experiment: signups within the same fixed observation window

| Variant | Assigned visitors | Signups | Conversion |
| --- | --- | --- | --- |
| A · Comparison | 1,200 | 60 | 5.0% |
| B · New message | 1,200 | 72 | 6.0% |
| Observed difference | Same visitor counting rule | 12 more signups | +1.0 percentage point |

## Recommendation

Report an inconclusive result at this planned readout.

B’s observed rate is 20% higher relative to A, but the approximate 95% interval for B minus A is negative 0.82 to +2.82 percentage points. It includes zero. This example does not establish a reliable improvement.

## Count the same kind of visitor in each group

The primary metric is signups divided by assigned eligible visitors within the agreed window. The descriptive difference is 6% minus 5% = 1 percentage point. Relative lift is (6% minus 5%) ÷ 5% = 20%.


## Show the uncertainty

The calculation estimates a 95% uncertainty range for the difference between the two signup rates. It uses the standard error for two independent proportions and a multiplier of 1.96. Each group has at least 60 signups and 1,128 visitors who did not sign up. The download includes the calculation. The method assumes independent random assignment and one planned analysis.


## Make the next step explicit

Document the result and review tracking and audience balance. If the question remains valuable, design a follow up test with a business relevant minimum effect, planned sample size, guardrails, and a fixed decision rule.

- Do not extend the test solely until a favorable result appears.
- Check signup quality and unintended effects.
- Separate “inconclusive” from evidence that the variants are equivalent.

## Learning note

The direction of an observed difference and the strength of the evidence are separate questions. Report both before recommending a change.

## Files and sources

- Experiment data CSV: campaign-readout.csv
- Calculation script: verify-samples.py
- Penn State: inference for two proportions: https://online.stat.psu.edu/stat200/Lesson09

Related experience: https://richielin.com/pages/romin

View online: https://richielin.com/pages/samples/campaign-readout