Check population randomness in an A/B test? I am fairly new to A/B testing and I don't really understand some points that my colleagues make. 
On our website users are randomly assigned to A or B when their session start (by randomly assigned, I mean generating a random number between 0 and 1 and apply this if (random_number < 0.5) then (assign to A) else (assign to B) for instance). 
Yet, they say that the population is not really random. For instance, we will see that the 2 samples have statistically different ages. They say we should run tests each time to compare the distributions on several dimensions. But my point of view is that the samples are really generated randomly but you are just unlucky to get statistically different samples on the 'x' dimension (you are in the 5% error rate of your 95% confidence AB test, so it is already accounted for). They can reply that for instance one out of two A/B tests show statistically different samples on a dimension. But here I think it's just that for instance if you have 5% chance of being wrong and the variables are independent (unlikely) then if you test on 6 variables (age, browser, whatever, ...) your chance of having only non statistically different dimensions on 2 tests is (0.95^6)^2 which is approximately 0.5 . 
Also, they say we have to check for normality, but if the sample is randomly generated and large enough (which is the case here), isn't it already a normal distribution if we follow the central limit theorem ? (each user is a Bernoulli random variable for instance with probability p of converting).
I was wondering who was right (probably them since I'm the only one who think like me ^^) and why ?
Thanks a lot in advance !
 A: In effect they're testing a bunch of different things and saying there's a significant difference if there's a difference on any of the dimensions each tested at some significance level $\alpha$.
If there's really no difference, what's the probability of not rejecting any of them?
(This is the standard multiple comparisons problem.)
If the dimensions were all close to independent, we could approximate the rate as $(1-\alpha)^d$ for $d$ the number of dimensions checked over. Which is to say if you check a lot of dimensions, even when there's really nothing going on, you're going to say there's a problem a lot. You correctly identify this issue in your question.
It might make some sense to look at a measure of how big a difference there is on dimensions of interest, but statistical significance may not be that big a deal.
There are some options:


*

*You could try stratification and then random allocation to treatment (allocating a fixed number of A's and B's within each stratum), but they're mostly just responding to noise.

*You could try putting the dimensions of interest as covariates in a regression model, which adjusts for such differences.

It's not clear why you would need to check normality. What are they doing that would be better with a check of normality?
