Consider an outcome variable Y that you want to maximize. To do that you need to choose between two "treatments" A and B.
To find which to choose, you run an experiment that gives as a result that y under A is larger than y under B. The difference, however, is not statistically significant. You conclude that the measured effect is not systematic but due to randomness. So, I guess, the experiment does not help you make a choice as you should be still indifferent between A and B.
This conclusion, however, contrasts with my intuition that, after the experiment, one should update the initial beliefs about the effectiveness of A and B. Under this view, the experiment is indeed informative and you are not indifferent between A and B (A should be preferred to B), even though the difference is not statistically significant.
Is this a true paradox? Or just a mistake in my reasoning?
In other words, after the experiment, should I consider A and B equal and flip a coin, as statistical significance seems to suggest? or A is better than B as (Bayesian) updating seems to suggest?