2
$\begingroup$

Consider an outcome variable Y that you want to maximize. To do that you need to choose between two "treatments" A and B.

To find which to choose, you run an experiment that gives as a result that y under A is larger than y under B. The difference, however, is not statistically significant. You conclude that the measured effect is not systematic but due to randomness. So, I guess, the experiment does not help you make a choice as you should be still indifferent between A and B.

This conclusion, however, contrasts with my intuition that, after the experiment, one should update the initial beliefs about the effectiveness of A and B. Under this view, the experiment is indeed informative and you are not indifferent between A and B (A should be preferred to B), even though the difference is not statistically significant.

Is this a true paradox? Or just a mistake in my reasoning?

In other words, after the experiment, should I consider A and B equal and flip a coin, as statistical significance seems to suggest? or A is better than B as (Bayesian) updating seems to suggest?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

In the first scenario you gather some data and conduct the hypothesis test, the test tells you that the difference is so small that it didn't reach statistical significance.

In the second scenario, you start with some hypothesis (your priors), gather the data, and combine the two sources of information, to update your hypothesis. Bayesians don't use $p$-values, so they won't say that the result is "not significant", yet they can simply conclude that it is small. On another hand, Bayesian could define some region of practical equivalence and conclude that since the small difference falls into the region, it is "practically equivalent" to zero (call it "insignificant" if you wish).

There is no paradox in here. Both approaches would only differ in the conclusions if the Bayesian used strongly informative prior, that would influence the results, but if the priors are not "too" informative and the data is not too "inconclusive", this should not be the case.

Improve this answer
$\endgroup$
1
$\begingroup$

The mistake in your reasoning is "You conclude that the measured effect is not systematic but due to randomness". The measured effect is small so after the experiment it's not clear if that small effect is due to randomness or if it is due to a small systematic effect. If you increase your sample size you could detect smaller systematic effects and be confident that they are not due to randomness

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Thanks. I might have said that in the wrong way. But your answer does not fully address my question. After the experiment, should I consider A and B equal, as statistical significance seems to suggest? or A is better than B as Bayesian updating seems to suggest? $\endgroup$
    – mrb
    Commented Dec 1, 2017 at 17:44
  • 1
    $\begingroup$ Also, the estimated effect can be huge and still be not statistically significant (it depends on the statistical power of the test). I don't think the effect size matters here. $\endgroup$
    – mrb
    Commented Dec 1, 2017 at 17:57
  • $\begingroup$ @mrb I think you're trying to apply bayesian ideas to the frequentist style of thinking. In the frequentist view either A is better than B or B is better than A or A and B are equal. There's no validity in saying "A is probably better than B" because the statement "A is better than B" is either true or false: either 100% likely or 0% likely, we cannot talk about probability of truth in the frequentist view of the world. $\endgroup$
    – Hugh
    Commented Dec 2, 2017 at 11:59
  • $\begingroup$ @mrb While the Bayesian can say "A is probably better than B" the frequentist can make statements like "If we repeat the experiment with a sample size of 100 then the sample mean of A will probably be larger than the sample mean of B". This is because sample means are random variables but population means are not (in the frequentist view) $\endgroup$
    – Hugh
    Commented Dec 2, 2017 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.