Is a test of statistical significance necessary when the mean is "surely off the charts" in the experiment? This is purely for pedagogical purposes. Here's what I've observed with "experiment design" at my workplace (We design consumer facing systems and we do A/B tests tweaking small things to see what has the highest impact, to whatever it is we're measuring. Not possible to reproduce this outside the organization):


*

*Control Group: Things left as is.

*Experiment A : Only one change from control, rest as is.

*Experiment B : Another one thing changed from control, rest as is, but different from A.

*...


Let's say, the "mean performance" (on whatever mean value this experiment is being run) for A is $50\%$ more than control and that of B is about $10\%$. Verdict: A is the winner.
Questions:


*

*Does it make sense to even do any tests of statistical significance, with such high variations in the mean?

*This got me thinking: Historically the tests of statistical significance have always been applied on things on which even a marginal improvement matters. For example, medications, liquor distillation etc., where the variations are not as high as my example above. Perhaps that was the reason for creating them since you just couldn't observe differences per se but had to analyze data to ascertain that. Is this a fair observation?


*

*As a practitioner (not statistician) - just observing these high variations seems "good enough" to get the work done. As long as the experiment only changes a single thing it's okay to not perform any statistical analysis since we can just go "Look!" - what's the risk with this?

*As a statistician - what should and shouldn't be done in situations like this?


*Being so loose with this gets me wondering if people really understand what's going on or am I really missing something simple/obvious :)

 A: What's probably going on here, as can happen with someone who has a lot of experience in some subject matter, is that you have internalized the statistical significance test.
To be concrete, consider testing changes to a web page that normally produces a 1% click-through rate. With such a rate we can use a Poisson model. Let's assume that all Experiments involve the same number of trials. So your Experiment A produces a 1.5% rate while Experiment B produces a 1.1% rate. Can you tell whether A is better than B?
What's crucial here is the difference in the number of observed events (e.g., click-throughs).
If you only did 1000 trials of each, you would have gotten 15 versus 11 events for A versus B. You can't really tell them apart, even though A and B are numerically 50% and 10% better, respectively, than the Control 1% click-through rate. As a rough guide, remember that the standard deviation of a number of Poisson events is the square root of the number of events; the errors expected for 15 versus 11 events overlap.
If you do 10000 trials of each of A and B, however, then you are comparing 150 against 110 events, a clear difference. Their errors don't overlap. More precisely, the poisson.test() function in R gives a p-value of 0.015 for that comparison.
In this type of scenario, you probably have learned from your experience to do 10000 or more trials before you calculate your click-through rate to evaluate the types of differences that you find interesting. If so, then you have effectively internalized the underlying statistical test.
If you aren't doing that many trials in such a scenario, however, you might be deceiving yourself. Protection from self-deception is one of the prime reasons for statistical tests.
