Suppose that I am running an experiment that gives us some confidence $c$ about a hypothesis.
If I intend to run the experiment $n$ times, I understand there are some ways of choosing a $c'$ so that the chance of any experiment giving us a confidence of $c'$ had it been run alone implies a confidence of $c$ for the hypothesis.
What if I intend to run the experiment until my hypothesis is confirmed with a confidence of $c$? Is there a way to choose $c_i$ for each successive experiment to ensure that I am not "cheating"?
This question is about the difference if any between deciding to run an experiment $n$ times and deciding to run an experiment until the data supports a conclusion. In other words, the decision to observe more data depends on past data.
A concrete example: suppose a researcher runs an experiment wanting to achieve a p-value of 0.05. Suppose instead that he runs the experiment ten times. He might apply a Bonferroni correction and expect a p-value of 0.005. What if instead, after the first failure, he he runs it again with a p-value threshold of 0.025. It fails, so he runs it again with a threshold of 0.05/3, and so on, until the experiment meets the threshold. Is this fair?