2
$\begingroup$

I am fairly new to A/B testing and I don't really understand some points that my colleagues make.

On our website users are randomly assigned to A or B when their session start (by randomly assigned, I mean generating a random number between 0 and 1 and apply this if (random_number < 0.5) then (assign to A) else (assign to B) for instance).

Yet, they say that the population is not really random. For instance, we will see that the 2 samples have statistically different ages. They say we should run tests each time to compare the distributions on several dimensions. But my point of view is that the samples are really generated randomly but you are just unlucky to get statistically different samples on the 'x' dimension (you are in the 5% error rate of your 95% confidence AB test, so it is already accounted for). They can reply that for instance one out of two A/B tests show statistically different samples on a dimension. But here I think it's just that for instance if you have 5% chance of being wrong and the variables are independent (unlikely) then if you test on 6 variables (age, browser, whatever, ...) your chance of having only non statistically different dimensions on 2 tests is (0.95^6)^2 which is approximately 0.5 .

Also, they say we have to check for normality, but if the sample is randomly generated and large enough (which is the case here), isn't it already a normal distribution if we follow the central limit theorem ? (each user is a Bernoulli random variable for instance with probability p of converting).

I was wondering who was right (probably them since I'm the only one who think like me ^^) and why ?

Thanks a lot in advance !

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In effect they're testing a bunch of different things and saying there's a significant difference if there's a difference on any of the dimensions each tested at some significance level $\alpha$.

If there's really no difference, what's the probability of not rejecting any of them?

(This is the standard multiple comparisons problem.)

If the dimensions were all close to independent, we could approximate the rate as $(1-\alpha)^d$ for $d$ the number of dimensions checked over. Which is to say if you check a lot of dimensions, even when there's really nothing going on, you're going to say there's a problem a lot. You correctly identify this issue in your question.

It might make some sense to look at a measure of how big a difference there is on dimensions of interest, but statistical significance may not be that big a deal.

There are some options:

  • You could try stratification and then random allocation to treatment (allocating a fixed number of A's and B's within each stratum), but they're mostly just responding to noise.

  • You could try putting the dimensions of interest as covariates in a regression model, which adjusts for such differences.

It's not clear why you would need to check normality. What are they doing that would be better with a check of normality?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your answer! I believe they want to check for normality because for hypothesis testing you want to compute the Z score for instance, and it is based on a normal distribution. So they want to check for it. $\endgroup$
    – Pholochtairze
    Commented Jan 27, 2016 at 11:56
  • $\begingroup$ Computing a z-score doesn't of itself require a normal distribution; it's just how many standard deviations from some mean you are. If they're doing some hypothesis test, that might need normality (though if we're looking at say binomial proportions there's no need to use a test that assumes normality) -- but even using a test that assumes normality, you may not be better testing that assumption; if it's not reasonable to simply assume normality, it's may actually be better just to not assume it. $\endgroup$
    – Glen_b
    Commented Jan 27, 2016 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.