I ran an experiment to test for some effects (which I indeed see evidence for), but in the analysis, also found a surprising effect that I could find no theoretical justification for. Specifically, there was a significant difference between two conditions for which I did not expect to see any differences. Since this experiment was already run after several similar others in which I found no such effect, I suspected that this was a type-I-error. I therefore decided to perform an exact replication of this experiment to see if the effect replicates. It did not. I decided to pool the results of both experiments and indeed there was no significant effect.
It is my understanding that what I did cannot really inflate the type-I-error: The first test I performed already rejected the null, so there is no way to increase the likelihood of (falsely) rejecting the null (had I not ran the replication, this likelihood is 1). Am I right here?
Second, I believe I am in the danger of inflating type-II-error here: If the effect is real, the additional collection of data actually increases the likelihood that I would fail to reject a false null. In a way, had I stopped after the first experiment my likelihood of making a type-II-error is 0, so it can only increase. Is this right?
Yet, considering the fact that the sample size I used is now double the original, and the significance level I'm using isn't changed, I also think the danger here is very small. Is this also right?
I explained this process in a paper and the reviewer was doubtful, so I'd appreciate corrections to my logic as much as affirmations for it
EDIT: Here is what I wrote in the paper, and that the reviewer flagged: "After running the first experiment, we looked at the data and found what we believed to be an extremely unlikely result. Specifically, [experimental details omitted...] and this difference was significant. We could not think of any theoretical justification for this result and suspected it was a type-I-error. We thus decided to run an exact replication of that experiment and combine the two datasets to check if this unlikely result holds. It did not. [Experimental details omitted...] Note that the decision to double the sample size does not inflate the likelihood of type-I-error in this case, since the first half of the sample already included the significant result. Rather, it may have inflated type-II-error. Yet, given the power in the double sample is considerably higher than in the original sample, we think the danger of this is very small."
And the review stated: "First, is it really the case that collecting additional data does not affect the Type 1 Error Rate – your decision to collect additional data is based on whether the finding is significant, therefore, you have applied a flexible stopping rule which can affect the Type 1 Error Rate. (I’m not fully sure about this point, because what you do here is not ‘sneak-peaking’ at the data before deciding whether to continue, which I am confident would affect the Type 1 Error Rate. Second, surely if you increase the statistical power by increasing the sample size, you reduce the Type 2 Error Rate?"
Thanks in advance!