“Optional stopping” to *reduce* type-I-error

Question

I ran an experiment to test for some effects (which I indeed see evidence for), but in the analysis, also found a surprising effect that I could find no theoretical justification for. Specifically, there was a significant difference between two conditions for which I did not expect to see any differences. Since this experiment was already run after several similar others in which I found no such effect, I suspected that this was a type-I-error. I therefore decided to perform an exact replication of this experiment to see if the effect replicates. It did not. I decided to pool the results of both experiments and indeed there was no significant effect.

It is my understanding that what I did cannot really inflate the type-I-error: The first test I performed already rejected the null, so there is no way to increase the likelihood of (falsely) rejecting the null (had I not ran the replication, this likelihood is 1). Am I right here?

Second, I believe I am in the danger of inflating type-II-error here: If the effect is real, the additional collection of data actually increases the likelihood that I would fail to reject a false null. In a way, had I stopped after the first experiment my likelihood of making a type-II-error is 0, so it can only increase. Is this right?

Yet, considering the fact that the sample size I used is now double the original, and the significance level I'm using isn't changed, I also think the danger here is very small. Is this also right?

I explained this process in a paper and the reviewer was doubtful, so I'd appreciate corrections to my logic as much as affirmations for it

EDIT: Here is what I wrote in the paper, and that the reviewer flagged: "After running the first experiment, we looked at the data and found what we believed to be an extremely unlikely result. Specifically, [experimental details omitted...] and this difference was significant. We could not think of any theoretical justification for this result and suspected it was a type-I-error. We thus decided to run an exact replication of that experiment and combine the two datasets to check if this unlikely result holds. It did not. [Experimental details omitted...] Note that the decision to double the sample size does not inflate the likelihood of type-I-error in this case, since the first half of the sample already included the significant result. Rather, it may have inflated type-II-error. Yet, given the power in the double sample is considerably higher than in the original sample, we think the danger of this is very small."

And the review stated: "First, is it really the case that collecting additional data does not affect the Type 1 Error Rate – your decision to collect additional data is based on whether the finding is significant, therefore, you have applied a flexible stopping rule which can affect the Type 1 Error Rate. (I’m not fully sure about this point, because what you do here is not ‘sneak-peaking’ at the data before deciding whether to continue, which I am confident would affect the Type 1 Error Rate. Second, surely if you increase the statistical power by increasing the sample size, you reduce the Type 2 Error Rate?"

Thanks in advance!

Answer 1

From a frequentist point of view, there was no effect of the variable in question ($H_0$), until you rejected that hypothesis in favor of the alternative. For any level of significance $\alpha$, this will happen $\alpha \cdot 100\%$ of the time. Moreover, if you tested for the presence of multiple effects, all of which have a true effect of $0$, then the chance of at least one significant "fluke" is even larger, namely $1 - (1 - \alpha)^k$, where $k$ is the number of hypothesis tests. You could correct for this inflated type I error rate at the expense of power, inflating the type II error rate, but as far as I can tell from your question, you did not correct for multiple testing. The first thing to note then, is that perhaps you would not even have found a significant effect in the first place, had you corrected for multiple testing.

You then collected additional data and added this to your original sample. This is a form of a stopping rule: Based on the significant result of the first test, you decided to collect more data. While the reviewer is correct in stating that the power of the second sample is larger, you are correct in saying that there is an increased chance of a type II error due to the stopping rule. You are also correct in stating that including the first data mitigates this, by already being biased towards concluding a significant effect.

Note that the decision to double the sample size does not inflate the likelihood of type-I-error in this case, since the first half of the sample already included the significant result.

This is where you make a mistake: The type I error rate is the chance of a false positive, or the chance that you find a significant effect, even though there is no true effect. This chance increases if you include data for which you know there is a significant effect (the first sample). Therefore, in my opinion the easiest correction to your paper would be to simply rephrase this part.

It is worth noting though, that the test you performed does not really answer your question: Failing to reject $H_0$ is not the same as concluding $H_0$ is true. If you wanted to demonstrate that there is no actual effect, a confidence interval close to $0$ would already be a lot more convincing, an equivalence test even more.

From a Bayesian point of view, you could have started with either a weakly informative prior, or one that expresses belief in the absence of an effect (since you did not expect it to be there). You then collected the first sample and the posterior moved towards the presence of an effect. This surprised you, so you collected additional data: The old posterior has now become the prior and the additional data is used to obtain the likelihood. You again sample from the posterior and conclude the new posterior effect size. A very relevant measure for what you are trying to do would be the area of practical equivalence to an effect size of $0$: The larger this area, the less likely there is an effect of the variable in question. Not only does this approach seem more appropriate, it is also more resilient to potential bias from your stopping rule.[1][2]

---

[1]: Gandenberger, Greg. (2017). Differences Among Noninformative Stopping Rules Are Often Relevant to Bayesian Decisions.

[2]: Wagenmakers, Eric-Jan. (2007). A Practical Solution to the Pervasive Problems of p Values. Psychonomic bulletin & review. 14. 779-804. 10.3758/BF03194105. appendix about stopping rules