Suppose that I am running an experiment that gives us some confidence $c$ about a hypothesis.

If I intend to run the experiment $n$ times, I understand there are some ways of choosing a $c'$ so that the chance of any experiment giving us a confidence of $c'$ had it been run alone implies a confidence of $c$ for the hypothesis.

What if I intend to run the experiment until my hypothesis is confirmed with a confidence of $c$? Is there a way to choose $c_i$ for each successive experiment to ensure that I am not "cheating"?

This question is about the difference if any between deciding to run an experiment $n$ times and deciding to run an experiment until the data supports a conclusion. In other words, the decision to observe more data depends on past data.

A concrete example: suppose a researcher runs an experiment wanting to achieve a p-value of 0.05. Suppose instead that he runs the experiment ten times. He might apply a Bonferroni correction and expect a p-value of 0.005. What if instead, after the first failure, he he runs it again with a p-value threshold of 0.025. It fails, so he runs it again with a threshold of 0.05/3, and so on, until the experiment meets the threshold. Is this fair?

  • $\begingroup$ What do you mean by "confidence"? Are you talking about p-values here? $\endgroup$ – amoeba Jul 21 '16 at 14:02
  • $\begingroup$ @amoeba yes, but I didn't want to limit answers to only p-values. I'm curious about this topic in general. $\endgroup$ – Neil G Jul 21 '16 at 14:05
  • $\begingroup$ (tagged missing-data since this might be related to the missing-at-random assumption) $\endgroup$ – Neil G Jul 21 '16 at 14:06
  • $\begingroup$ I am not sure a "general" answer is possible. I asked because you have [p-value] as a tag (but do not have [bayesian] as a tag, for example). In general, "confidence" is not a well-defined term. So I am not upvoting because I don't understand the question :) $\endgroup$ – amoeba Jul 21 '16 at 14:11
  • $\begingroup$ @amoeba I have added a concrete example, but I'm not sure if it makes sense. $\endgroup$ – Neil G Jul 21 '16 at 14:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.