This is purely for pedagogical purposes. Here's what I've observed with "experiment design" at my workplace (We design consumer facing systems and we do A/B tests tweaking small things to see what has the highest impact, to whatever it is we're measuring. Not possible to reproduce this outside the organization):

  • Control Group: Things left as is.
  • Experiment A : Only one change from control, rest as is.
  • Experiment B : Another one thing changed from control, rest as is, but different from A.
  • ...

Let's say, the "mean performance" (on whatever mean value this experiment is being run) for A is $50\%$ more than control and that of B is about $10\%$. Verdict: A is the winner.


  • Does it make sense to even do any tests of statistical significance, with such high variations in the mean?
  • This got me thinking: Historically the tests of statistical significance have always been applied on things on which even a marginal improvement matters. For example, medications, liquor distillation etc., where the variations are not as high as my example above. Perhaps that was the reason for creating them since you just couldn't observe differences per se but had to analyze data to ascertain that. Is this a fair observation?
    • As a practitioner (not statistician) - just observing these high variations seems "good enough" to get the work done. As long as the experiment only changes a single thing it's okay to not perform any statistical analysis since we can just go "Look!" - what's the risk with this?
    • As a statistician - what should and shouldn't be done in situations like this?
  • Being so loose with this gets me wondering if people really understand what's going on or am I really missing something simple/obvious :)
  • $\begingroup$ How reproducible is the 50% increase in "mean performance" for A or the 10% increase in "mean performance" for B? Unless you know that you can't really say that either one is "better" than Control. Once you know that, the statistical test jumps out immediately. $\endgroup$
    – EdM
    Feb 16, 2018 at 20:55
  • $\begingroup$ It makes no sense to "reproduce" it outside of the company - these are all internal A/B tests for consumer products. $\endgroup$
    – PhD
    Feb 16, 2018 at 21:03
  • $\begingroup$ By "reproducible" I mean when you repeat the test within the company, or some measure of inter-consumer reproducibility if you are depending on tests in a consumer-facing system. For example, if your A/B test represents comparing yes/no click-through rates for A versus B by consumers on an otherwise identical web page, the yes/no click-through rates you are comparing presumably have binomial statistics with associated variances. If you only run the test on 10 consumers, the values will be much less reproducible than if you run it on 1000. $\endgroup$
    – EdM
    Feb 16, 2018 at 21:16
  • $\begingroup$ In that case, definitely reproducible $\endgroup$
    – PhD
    Feb 16, 2018 at 21:30

1 Answer 1


What's probably going on here, as can happen with someone who has a lot of experience in some subject matter, is that you have internalized the statistical significance test.

To be concrete, consider testing changes to a web page that normally produces a 1% click-through rate. With such a rate we can use a Poisson model. Let's assume that all Experiments involve the same number of trials. So your Experiment A produces a 1.5% rate while Experiment B produces a 1.1% rate. Can you tell whether A is better than B?

What's crucial here is the difference in the number of observed events (e.g., click-throughs).

If you only did 1000 trials of each, you would have gotten 15 versus 11 events for A versus B. You can't really tell them apart, even though A and B are numerically 50% and 10% better, respectively, than the Control 1% click-through rate. As a rough guide, remember that the standard deviation of a number of Poisson events is the square root of the number of events; the errors expected for 15 versus 11 events overlap.

If you do 10000 trials of each of A and B, however, then you are comparing 150 against 110 events, a clear difference. Their errors don't overlap. More precisely, the poisson.test() function in R gives a p-value of 0.015 for that comparison.

In this type of scenario, you probably have learned from your experience to do 10000 or more trials before you calculate your click-through rate to evaluate the types of differences that you find interesting. If so, then you have effectively internalized the underlying statistical test.

If you aren't doing that many trials in such a scenario, however, you might be deceiving yourself. Protection from self-deception is one of the prime reasons for statistical tests.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.