1
$\begingroup$

If I run one experiment, and obtain a p-value of $p=.03$ and I have a priori decided that my $\alpha=.05$. I have a significant a result, I conclude that my data reject $H0$.

If I run two identical experiments with the same study $\alpha$, both with a p-value of $p=.03$. Now as I have done two different tests, I correct for multiple comparisons (e.g. Bonferroni), and the $p$ the values become non-significant, as my alpha is divided by the number of tests done.

Imagine the same outcome but for hundred experiments in the same study, testing the same hypothesis.

How can it be that gathering evidence that converges in rejecting the null, makes it less significant and therefore prevents from rejecting it?

I understand that common sense (and for example a Bayesian approach), should find more evidence in rejecting the null. But, I'm more interested in knowing, if my reasoning is flawed (e.g. multiple comparisons should not be applied in this case, but then I'm interested in why) or this is a known flaw, and we just live with it, and the decision is taken based on common sense and not on a strictly mathematical proof.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ your results are not less significant, but the probability of making a type I error (in the case of significance) grows. Moreover, based on frequentist analyses your data do not reject anything. The only thing you find (using p-values) is the probability of your data being based on/generated by your null-hypothesis. We, the researchers, are the ones who decide whether or not this probability is small enough to assume the null hypothesis is wrong. Do note that however small this probability/p-value might be, it suggests there is still a chance the null-hypothesis is actually true! $\endgroup$
    – IWS
    Commented Jan 17, 2017 at 15:05
  • $\begingroup$ True that is not my data the one that rejects, but based on my data I take the decision, let's say (for this case only) that I'm a Neyman-Pearson fan. My point is I if I run two experiments I have more data pointing to a deviation from the null (in the case described above), therefore I should be more sure of rejecting the null based on the data. But in the latter case (according to a Neyman-Pearson approach) I do not reject the null in any of the experiments within the study (due to multiple comparisons), but I have more data in that direction. Is this a failure in my reasoning? $\endgroup$
    – luismf
    Commented Jan 17, 2017 at 15:49
  • $\begingroup$ The thing is you have to set your alpha before doing any analyses. So if you plan to do two analyses which could possibly be related (because you are performing these tests on the same data) your alpha level increases. As your alpha level translates to the risk of making a type I error, this risk increases from e.g. 0.05 to of 1-0.95^2 = 0.0975. You can decide to correct for this in multiple ways (of which Bonferroni is only one), by assuming another critical p-value level (something else than your alpha), which when accounting for the amount of tests would result in an alpha of 0.05. $\endgroup$
    – IWS
    Commented Jan 17, 2017 at 16:24
  • $\begingroup$ see also stats.stackexchange.com/questions/256667/… $\endgroup$
    – IWS
    Commented Jan 17, 2017 at 16:25
  • $\begingroup$ When you say "I understand that common sense (and for example a Bayesian approach), should find more evidence in rejecting the null", are you saying that you are doing lots of tests of the same null hypothesis? It would be strange to do such separate tests with a Bonferroni correction. Even for repeated analyses of accumulating data this would not be the standard frequentist solution (note: most claims that Bayesians do not need to adjust for repeated looks at accumulating data assume that we simply do not care about the probability of a false claim despite the null hypothesis being true). $\endgroup$
    – Björn
    Commented Jan 17, 2017 at 16:34

2 Answers 2

Reset to default
1
$\begingroup$

You seem to be confusing the requirement of replication (which reduces Type I error rates) with the problem of multiple comparisons (which increases Type I error rates).

REQUIRING REPLICATION: If you decide a priori to make significance require two independent tests to come out p < .05, you are actually REDUCING the probability of Type I error rather than INFLATING it, because you are making the criteria for significance more stringent. Thus, this is NOT a situation that calls for Bonferroni adjustment. In fact, the probability of getting p < .05 under the null hypothesis in both of two independent studies is .05^2 = .0025.

MULTIPLE COMPARISONS: On the other hand, if you decide that significance requires only ONE of two independent tests to come out p < .05, then you have given yourself an extra chance of getting significance, so you are INFLATING the probability of Type I error. The probability of getting p < .05 in at least one of two independent tests is 1-(1-.05)^2 = .0975.

These same principles apply if you conduct 100 experiments. If you require all of them to produce p < .05, then the probability of Type I error under the null hypothesis is reduced to .05^100, which is near zero. If you require only 1 of them to produce p < .05, then the probability of Type I error under the null hypothesis is inflated to 1-(1-.05)^100, which is roughly 99%.

Improve this answer
$\endgroup$
0
$\begingroup$

I'm not sure that you actually ran two (or one hundred) experiments here.

Instead, it sounds like you ran a single experiment, testing a single hypothesis, twice. As such, your "family" of hypotheses hasn't gotten larger and there is no need to correct your $\alpha$ to maintain the same familywise error rate.

Realistically, you would be better off combining both experiments' data into a single analysis. For example, you might add a batch/session/run factor to your ANOVA or GLM, along with the experimental condition(s) of interest. This would allow you to identify changes from session to session and determine whether they interact with the treatments you applied.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.