This is purely for pedagogical purposes. Here's what I've observed with "experiment design" at my workplace (We design consumer facing systems and we do A/B tests tweaking small things to see what has the highest impact, to whatever it is we're measuring. Not possible to reproduce this outside the organization):
- Control Group: Things left as is.
- Experiment A : Only one change from control, rest as is.
- Experiment B : Another one thing changed from control, rest as is, but different from A.
Let's say, the "mean performance" (on whatever mean value this experiment is being run) for A is $50\%$ more than control and that of B is about $10\%$. Verdict: A is the winner.
- Does it make sense to even do any tests of statistical significance, with such high variations in the mean?
- This got me thinking: Historically the tests of statistical significance have always been applied on things on which even a marginal improvement matters. For example, medications, liquor distillation etc., where the variations are not as high as my example above. Perhaps that was the reason for creating them since you just couldn't observe differences per se but had to analyze data to ascertain that. Is this a fair observation?
- As a practitioner (not statistician) - just observing these high variations seems "good enough" to get the work done. As long as the experiment only changes a single thing it's okay to not perform any statistical analysis since we can just go "Look!" - what's the risk with this?
- As a statistician - what should and shouldn't be done in situations like this?
- Being so loose with this gets me wondering if people really understand what's going on or am I really missing something simple/obvious :)