1
$\begingroup$

I ran an experiment to test for some effects (which I indeed see evidence for), but in the analysis, also found a surprising effect that I could find no theoretical justification for. Specifically, there was a significant difference between two conditions for which I did not expect to see any differences. Since this experiment was already run after several similar others in which I found no such effect, I suspected that this was a type-I-error. I therefore decided to perform an exact replication of this experiment to see if the effect replicates. It did not. I decided to pool the results of both experiments and indeed there was no significant effect.

It is my understanding that what I did cannot really inflate the type-I-error: The first test I performed already rejected the null, so there is no way to increase the likelihood of (falsely) rejecting the null (had I not ran the replication, this likelihood is 1). Am I right here?

Second, I believe I am in the danger of inflating type-II-error here: If the effect is real, the additional collection of data actually increases the likelihood that I would fail to reject a false null. In a way, had I stopped after the first experiment my likelihood of making a type-II-error is 0, so it can only increase. Is this right?

Yet, considering the fact that the sample size I used is now double the original, and the significance level I'm using isn't changed, I also think the danger here is very small. Is this also right?

I explained this process in a paper and the reviewer was doubtful, so I'd appreciate corrections to my logic as much as affirmations for it

EDIT: Here is what I wrote in the paper, and that the reviewer flagged: "After running the first experiment, we looked at the data and found what we believed to be an extremely unlikely result. Specifically, [experimental details omitted...] and this difference was significant. We could not think of any theoretical justification for this result and suspected it was a type-I-error. We thus decided to run an exact replication of that experiment and combine the two datasets to check if this unlikely result holds. It did not. [Experimental details omitted...] Note that the decision to double the sample size does not inflate the likelihood of type-I-error in this case, since the first half of the sample already included the significant result. Rather, it may have inflated type-II-error. Yet, given the power in the double sample is considerably higher than in the original sample, we think the danger of this is very small."

And the review stated: "First, is it really the case that collecting additional data does not affect the Type 1 Error Rate – your decision to collect additional data is based on whether the finding is significant, therefore, you have applied a flexible stopping rule which can affect the Type 1 Error Rate. (I’m not fully sure about this point, because what you do here is not ‘sneak-peaking’ at the data before deciding whether to continue, which I am confident would affect the Type 1 Error Rate. Second, surely if you increase the statistical power by increasing the sample size, you reduce the Type 2 Error Rate?"

Thanks in advance!

$\endgroup$
  • $\begingroup$ About your first question, likelihood $\neq$ probability, so you may want to leave out the part about it necessarily equating $1$. Moreover, you can't really make probability statements about an event that already occurred, it just did. Perhaps the reviewer thought the combining of the old and new data was questionable, could you maybe include a quote of the exact wording the reviewer had trouble with? $\endgroup$ – Frans Rodenburg Dec 1 '19 at 12:09
  • $\begingroup$ On a different note: Did you include estimates of effect size and standard errors / confidence intervals? You should include the magnitude of the effect. Any level of significance has a chance of a type I error. Type I errors will occur, and there is nothing special about them. If you can show the estimated effect is smaller than what is considered relevant, that is more interesting than the fact that it turned out to be significantly non-zero in the first sample. $\endgroup$ – Frans Rodenburg Dec 1 '19 at 12:15
  • $\begingroup$ Thanks Frans. Re effect size, it was actually quite large (in the original experiment). The DV is a proportion and the difference between conditions was 11 percentage points. Yes, I see that the probability statements I made are erroneous, they're here just to intuitively clarify why I think my logic stands. As for the reviewer, I don't think it is a matter of whether I should have combined the data: "Pragmatically, I think the decision to collect more data seems sound, it is just that I’m not sure about your discussion of that point here". $\endgroup$ – Cuenco Dec 1 '19 at 12:46
2
$\begingroup$

From a frequentist point of view, there was no effect of the variable in question ($H_0$), until you rejected that hypothesis in favor of the alternative. For any level of significance $\alpha$, this will happen $\alpha \cdot 100\%$ of the time. Moreover, if you tested for the presence of multiple effects, all of which have a true effect of $0$, then the chance of at least one significant "fluke" is even larger, namely $1 - (1 - \alpha)^k$, where $k$ is the number of hypothesis tests. You could correct for this inflated type I error rate at the expense of power, inflating the type II error rate, but as far as I can tell from your question, you did not correct for multiple testing. The first thing to note then, is that perhaps you would not even have found a significant effect in the first place, had you corrected for multiple testing.

You then collected additional data and added this to your original sample. This is a form of a stopping rule: Based on the significant result of the first test, you decided to collect more data. While the reviewer is correct in stating that the power of the second sample is larger, you are correct in saying that there is an increased chance of a type II error due to the stopping rule. You are also correct in stating that including the first data mitigates this, by already being biased towards concluding a significant effect.

Note that the decision to double the sample size does not inflate the likelihood of type-I-error in this case, since the first half of the sample already included the significant result.

This is where you make a mistake: The type I error rate is the chance of a false positive, or the chance that you find a significant effect, even though there is no true effect. This chance increases if you include data for which you know there is a significant effect (the first sample). Therefore, in my opinion the easiest correction to your paper would be to simply rephrase this part.

It is worth noting though, that the test you performed does not really answer your question: Failing to reject $H_0$ is not the same as concluding $H_0$ is true. If you wanted to demonstrate that there is no actual effect, a confidence interval close to $0$ would already be a lot more convincing, an equivalence test even more.

From a Bayesian point of view, you could have started with either a weakly informative prior, or one that expresses belief in the absence of an effect (since you did not expect it to be there). You then collected the first sample and the posterior moved towards the presence of an effect. This surprised you, so you collected additional data: The old posterior has now become the prior and the additional data is used to obtain the likelihood. You again sample from the posterior and conclude the new posterior effect size. A very relevant measure for what you are trying to do would be the area of practical equivalence to an effect size of $0$: The larger this area, the less likely there is an effect of the variable in question. Not only does this approach seem more appropriate, it is also more resilient to potential bias from your stopping rule.[1][2]


[1]: Gandenberger, Greg. (2017). Differences Among Noninformative Stopping Rules Are Often Relevant to Bayesian Decisions.

[2]: Wagenmakers, Eric-Jan. (2007). A Practical Solution to the Pervasive Problems of p Values. Psychonomic bulletin & review. 14. 779-804. 10.3758/BF03194105. appendix about stopping rules

$\endgroup$
  • $\begingroup$ If I understand you correctly, inflation of type I error here is not a result of running the additional experiment per se, but a result of including the first experiment as part of the analysis. That is, the reviewer is wrong in writing that "collecting additional data" is what affects the type I error rate, while I am wrong in writing that "doubling the sample size" does not affect it. In a way, had I not collected more data, the chances I make a type I error would have been highest; had I analyzed just the second experiment the chances would have been lowest; and what I did is in the middle? $\endgroup$ – Cuenco Dec 2 '19 at 15:08
  • 1
    $\begingroup$ The type I error rate of the new data alone is no different than $\alpha$, but adding the old sample inflates it. Running two separate tests inflates the FWER. The problem of the stopping rule is this: Imagine you would keep collecting data and stop only once the result is not significant. Surely this inflates the type II error rate despite the power growing. The problems with stopping rules and expressing the error rates just show problems with NHST in general in, so my point was that you should focus more on the (small) effect size of the combined sample, or use a Bayesian alternative. $\endgroup$ – Frans Rodenburg Dec 3 '19 at 2:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.