# Is there a statistical correction that works if the number of experiments depends on the result of the experiments?

Suppose that I am running an experiment that gives us some confidence $c$ about a hypothesis.

If I intend to run the experiment $n$ times, I understand there are some ways of choosing a $c'$ so that the chance of any experiment giving us a confidence of $c'$ had it been run alone implies a confidence of $c$ for the hypothesis.

What if I intend to run the experiment until my hypothesis is confirmed with a confidence of $c$? Is there a way to choose $c_i$ for each successive experiment to ensure that I am not "cheating"?

This question is about the difference if any between deciding to run an experiment $n$ times and deciding to run an experiment until the data supports a conclusion. In other words, the decision to observe more data depends on past data.

A concrete example: suppose a researcher runs an experiment wanting to achieve a p-value of 0.05. Suppose instead that he runs the experiment ten times. He might apply a Bonferroni correction and expect a p-value of 0.005. What if instead, after the first failure, he he runs it again with a p-value threshold of 0.025. It fails, so he runs it again with a threshold of 0.05/3, and so on, until the experiment meets the threshold. Is this fair?

• What do you mean by "confidence"? Are you talking about p-values here? – amoeba Jul 21 '16 at 14:02
• @amoeba yes, but I didn't want to limit answers to only p-values. I'm curious about this topic in general. – Neil G Jul 21 '16 at 14:05
• (tagged missing-data since this might be related to the missing-at-random assumption) – Neil G Jul 21 '16 at 14:06
• I am not sure a "general" answer is possible. I asked because you have [p-value] as a tag (but do not have [bayesian] as a tag, for example). In general, "confidence" is not a well-defined term. So I am not upvoting because I don't understand the question :) – amoeba Jul 21 '16 at 14:11
• @amoeba I have added a concrete example, but I'm not sure if it makes sense. – Neil G Jul 21 '16 at 14:20