Situation: A/B test of a single website change on a landing page. Alternating visitors to the landing page are shown a variation. The goal is to convert more visitors into customers.
With data that look like this:
Variation | Did Not Convert | Conversions | Total Visitors
Landing Page A (Original) | 1267 | 116 | 1383
Landing Page B (Variation)| 395 | 4 | 399
I obtained the following results:
- Chi-Square value: 27,
- Degrees of Freedom: 1,
- p_Value: 0.0000002.
Is it fair or true to summarize my results like this:
"Hypothesis: Landing page has no bearing or impact on whether or not a visitor converts.
If there was no difference between the landing pages and whether or not someone converts, then we would expect to see the observed results 0.00002% of the time.
Result: Reject the hypothesis, landing page does indeed impact conversion."
Have I got this right? Have I missed anything important?