4
$\begingroup$

Situation: A/B test of a single website change on a landing page. Alternating visitors to the landing page are shown a variation. The goal is to convert more visitors into customers.

With data that look like this:

Variation                 | Did Not Convert | Conversions | Total Visitors
Landing Page A (Original) | 1267            | 116         | 1383
Landing Page B (Variation)| 395             | 4           | 399

I obtained the following results:

  • Chi-Square value: 27,
  • Degrees of Freedom: 1,
  • p_Value: 0.0000002.

Is it fair or true to summarize my results like this:

"Hypothesis: Landing page has no bearing or impact on whether or not a visitor converts.

If there was no difference between the landing pages and whether or not someone converts, then we would expect to see the observed results 0.00002% of the time.

Result: Reject the hypothesis, landing page does indeed impact conversion."

Have I got this right? Have I missed anything important?

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ We welcome questions like this. People come here with questions at all levels of sophistication. $\endgroup$
    – gung - Reinstate Monica
    Oct 28 '14 at 19:34
  • 3
    $\begingroup$ Any level of question is most welcome (it's what we're here for!), especially one where the poster has put some effort into thinking about the question. I'd hate for this site to become some isolated enclave of only research-level questions. We're all here learning off each other -- I learn something new here almost every day. $\,$ [Indeed, supposedly basic questions are often a good deal more subtle than they seem; answers to them will tend to help more people than the more technical questions; and they give a wider range of people the opportunity to attempt an answer.] $\endgroup$
    – Glen_b
    Oct 28 '14 at 21:59
6
$\begingroup$

You have the interpretation exactly correct. One thing you might want to do is quantify how much of a difference version B made, and whether that difference was an improvement. To get probabilities of converting, divide the number who convert on each row by the row's sum. The simplest measure of the change would be to get the risk difference, which just subtracts A from B. 

a = 116/1383
b = 4/399
a    #  0.08387563
b    #  0.01002506
b-a  # -0.07385057

The change seems to have decreased your conversion rate.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you very much for clarifying my understanding. One thing, in your recommendation of calculating conversion rate you say a = 116/(1383+116). But conversions are a subset of visitors, so would it not be 116/1383? With that in mind should my data have actually been presented as (for A) 1267: not converted and 116 converted? Am I makiing sense? $\endgroup$
    – Doug Fir
    Oct 28 '14 at 19:47
  • $\begingroup$ I updated the data in my question to underline what I'm saying. Do the results still hold? $\endgroup$
    – Doug Fir
    Oct 28 '14 at 19:49
  • 1
    $\begingroup$ @DougFirr, your interpretation is still correct, I just misunderstood what your numbers represented. I have updated the calculations. $\endgroup$
    – gung - Reinstate Monica
    Oct 28 '14 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.