3
$\begingroup$

Suppose I have designed an experiment to test $D>0$ where $d>0$ is effect size of $D$ with confidence level $0.05$ and power $0.8$. Say I find that I need $2N$ experimental units to get the job done. $2N=N+N$ means that I did first N experiments same as the next N experiments.

Q1. Suppose after first N experimental unit results come in and I find p-value<0.01. Can I stop here? I think I could but is not this data snooping? Of course, the power of test would no longer be expected ~0.8 assuming $d$ is true effect size. In clinical trial, one could do this by interim analysis. Is this contradicting with data snooping.

Q2. Suppose I did not find the result significant after $2N$ experiments and I decided to increase sample size by $N$. After $2N$ experiments and $N$ experiments, suppose I can obtain p-value<0.01 and I think power here should be increased assuming effect size is $d>0$. Is it fine to do so or I should redo $3N$ experiments?

$\endgroup$
1
  • $\begingroup$ See stats.stackexchange.com/questions/20676 and stats.stackexchange.com/questions/310119. Although you don't have to redo $3N$ experiments, continuing to expand the experiment until you obtain a "significant" p-value according to a fixed threshold does not yield a valid p-value unless you also adjust the threshold of "significance" in a valid fashion. This would be a "group sequential" experiment. $\endgroup$
    – whuber
    Jan 10 at 19:08

1 Answer 1

1
$\begingroup$

The answers to your questions depend on your inferential objectives and on what you mean by "data snooping" and the context of "can I" and "should I".

Frequentists who prefer to work within the global error rate framework of Neyman will likely give you firm answers that tell only a small part of the story. Early stopping for significance and extending the data gathering in an attempt to gain significance both modify the long run error rates of the procedures and so they are either forbidden or 'corrected' by penalising the evidence to protect the false positive error rate (at the cost of decreasing power of the procedure).

That is not the only framework, and it's probably not the most relevant framework for statistical analyses that are intended to support scientific inferences. See this for an exposition on the topic: https://link.springer.com/chapter/10.1007/164_2019_286#enumeration

If your experiments are to be analysed with the Bayesian framework then (usually) data snooping is simply part of the learning from data and so it's entirely acceptable, and extending a study by gathering more data is sensible.

Question 1

Can you stop data collection once you've achieved a p-value less than 0.01? Yes you can, but you should note that a hypothesis test procedure that stops once a low p-value is observed will yield a long run false positive error rate higher than the designed alpha (where alpha is the designed critical value of p for discarding the null hypothesis).

Does that mean that the evidence is somehow made faulty or misleading? No. The observed p-value is the true p-value even if it does not represent the true long run false positive error rate.* A ‘neo-Fisherian’ interpretation of the p-value is that it scales inversely (and non-linearly) with the strength of evidence in the data against the null hypothesis of the significance test. Because early stopping does not affect the data that you have in hand,** it does not affect the local (actual) evidence and so it does not affect the evidential meaning of the observed p-value.

The conflict between the Neyman–Pearsonian hypothesis test long run error rate and the neo-Fisherian significance test local evidence is a consequence of the different objectives of the frameworks.

You are kind of correct that early stopping affects the power of the procedure, but it is the 'correction' of the analysis that affects the power. If you run an optional stopping procedure without correction then you will gain extra power, but that 'power' will not be the power defined by Neyman and Pearson. (You need to understand that the power relates to the procedure, not to the actual data or evidence.)

In a clinical trial designed to have an interim analysis the statistical procedures will have been designed to either adjust the critical p-value for claiming ‘significance’ or it will have the decision to stop early made by a panel of analysts who are independent of the analysts who would do the final analysis. The separation is an attempt to get around the dependence of the error rate of the study on the intentions of the experimenters. (I’m not sure that it works at any level...)

Question 2

The setup you describe here is the flip side of the early stopping for significance of question 1. A procedure that is extended because of non-significance is more likely to yield a false positive error than a procedure that was designed to reach the larger sample size. The mechanism is pretty easy to understand, as it is simply because some of the data have more than one opportunity to move the analysis towards significance. Does that mean that you shouldn’t do it, or that you should instead do a fresh experiment at the larger size? Not necessarily.

If you are working within the Neyman–Pearsonian error rate framework then you will have to account for the increased long run rate of false positive errors in both the experiment that you extended and in any experiment that you did not extend but might have. This is a good illustration of how the global long run error rates are affected by the experimenter’s intentions (stated or not; conscious or not).

For a statistical framework that focusses on the local evidence in the data the stopping rules (and experimenter’s intentions) are usually not relevant. That is true of both the neo-Fisherian interpretation of the -value and the Bayesian procedures that work with the likelihood function.

The irrelevance of stopping rules is a somewhat controversial area of statistical philosophy, but need not be complicated. The data contain (embody, or communicate) the evidence and the stopping rules do not influence the observed data then the stopping rules cannot affect the evidence yielded by the experiment.***

It is noteworthy that a Bayesian analysis of the first 2N observations extended by a subsequent analysis of the second N observations will be mathematically identical to a combined analysis of the 3N observations.

Even though the evidence in the data and the evidential meaning of the p-value are not affected by the stopping rules, the long run false positive error rates of the procedure will be altered. The trick then is to be able to make a useful inference on the basis of both the local evidence from the actual experiment and the global long run error properties of the procedure. Achieving that requires attention to be paid to the role of the statistical inferences in the overall scientific inferential program, something that is too rarely mentioned in statistics texts and on this site.

Should you run a fresh experiment to gather a new set of 3N observations? Maybe. It depends on how important the results are and how much weight you (or journal referees!) might put onto the global error rates, and it depends on whether the study in question is preliminary (e.g. hypothesis generating) or intended to be definitive.

*You should know that a properly run hypothesis test procedure that yields a ‘significant’ p-value “< 0.01”, say 0.003, does not have a long run false positive error rate of 0.03. The error rate is 0.01 if and only if 0.01 was chosen to be the critical value (i.e. alpha=0.01) when the hypothesis test was designed. If an observed p<0.05 would have led to the null hypothesis being discarded then 0.05 would be the long run false positive error rate even though the observed p-value was 0.003.

** Early stopping of data collection will prevent extra data from being accumulated but it has no effect on the data gathered so far. The future cannot affect the past.

*** Addition of an extra N observations does not alter any of the earlier 2N observations, and the fact that the extra N observations might not have been made does not affect those observations once they have in fact been made.

Yep, that's a lot to take in and some of it will be controversial. Read the linked chapter.

$\endgroup$
2
  • $\begingroup$ What is theoretical justification of "If you run an optional stopping procedure without correction then you will gain extra power, but that 'power' will not be the power defined by Neyman and Pearson"? What is the definition of "power" gained here? $\endgroup$
    – user45765
    Jan 10 at 21:45
  • $\begingroup$ @user45765 The power of Neyman and Pearson is the power designed for a fixed false positive error rate. As the uncorrected optional stopping procedure inflates the false positive error rate the uncorrected 'power' is not as defined by Neyman and Pearson. $\endgroup$ Jan 10 at 21:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.