Quoting gung's great answer

Allegedly, a researcher once approached Fisher with 'non-significant' results, asking him what he should do, and Fisher said, 'go get more data'.

From a Neyman-Pearson perspective, this is blatant $p$-hacking, but is there a use case where Fisher's go-get-more-data approach makes sense?

  • 10
    $\begingroup$ Fisher (repeatedly) emphasized the importance of replication of experiments and I expect that was his intent here (assuming the conversation happened). Certainly Fisher would have been well aware that you can't check for significance and then expand your initial sample if you didn't get it. $\endgroup$ – Glen_b Jul 15 '19 at 4:59
  • $\begingroup$ @Glen_b I have heard the phrase "replication of experiments" before but didn't quite get it. Can you elaborate? Say, are ten replications of an experiment whose sample size is 10 better than a single experiment whose sample size is 100? $\endgroup$ – nalzok Jul 15 '19 at 5:06
  • 1
    $\begingroup$ In exploratory study, go-get-more-data may be acceptable. In confirmatory study, there is no position for go-get-more-data. $\endgroup$ – user158565 Jul 15 '19 at 6:22
  • 5
    $\begingroup$ One of my controversial views on statistical practice is that while it's important to consider the issue of false-positives, we should not put conserving type 1 error rates on such a high pedestal that we refuse to learn from the data in order to preserve a type 1 error rate. $\endgroup$ – Cliff AB Jul 15 '19 at 17:45

The frequentist paradigm is a conflation of Fisher's and Neyman-Pearson's views. Only in using one approach and another interpretation do problems arise.

It should seem strange to anyone that collecting more data is problematic, as more data is more evidence. Indeed, the problem lies not in collecting more data, but in using the $p$-value to decide to do so, when it is also the measure of interest. Collecting more data based on the $p$-value is only $p$-hacking if you compute a new $p$-value.

If you have insufficient evidence to make a satisfactory conclusion about the research question, then by all means, go get more data. However, concede that you are now past the NHST stage of your research, and focus instead on quantifying the effect of interest.

An interesting note is that Bayesians do not suffer from this dilemma. Consider the following as an example:

  • If a frequentist concludes no significant difference and then switches to a test of equivalence, surely the false positive rate has increased;
  • A Bayesian can express the highest density interval and region of practical equivalence of a difference simultaneously and sleep just the same at night.
  • $\begingroup$ So basically, say I want to test if the mean of population A is equal to that of population B. Initially, I get some data, conduct a test for $H_0$: "the means are equal", and I fail to reject it. In this case, I should not conduct another test for $H_0$: "the means are NOT equal". All I can do is estimating the confidential intervals of the means, is that correct? What if there is no overlap between the two intervals? $\endgroup$ – nalzok Jul 15 '19 at 16:56
  • 6
    $\begingroup$ "It's only p-hacking if you compute a new p-value." Doesn't this actually depend entirely on the method used to calculate the p-value? Ignoring the sequential analysis and decision to collect more data will result in an inaccurate p-value. However, if you incorporate the decision rule to collect more data into the calculation of the p-value, then you will produce a valid p-value. $\endgroup$ – jsk Jul 15 '19 at 18:16
  • 4
    $\begingroup$ @jsk I think it's less that subsequently calculated p-values are in some way invalid, and more that you are using an arbitrary and non-data-driven standard to judge when your experiment is "correct" and your research on that project is "done". Deciding that all non-significant p-values are wrong, and gathering data until you get one that is significant and then stopping because you've gotten the "right" result is the opposite of experimental science. $\endgroup$ – Upper_Case Jul 15 '19 at 20:53
  • 1
    $\begingroup$ @Upper_Case I was commenting on a very small section of the post in regards to p-hacking, which is why I included that section in quotes. You are reading way too much into my statement. My point is that ANY decision rule that is used to decide to collect more data must be incorporated into calculating the p-value. As long as you incorporate the decisions made into the calculation of the p-value, you can still conduct a valid NHST if you so desire. This does not in any way mean that I am advocating for a stopping rule that says, "collect more data until you find a significant result." $\endgroup$ – jsk Jul 15 '19 at 21:13
  • $\begingroup$ @jsk Ah, I understand your point better now. Thank you for the clarification. $\endgroup$ – Upper_Case Jul 15 '19 at 22:05

Given a big enough sample size, a test will always show significant results, unless the true effect size is exactly zero, as discussed here. In practice, the true effect size is not zero, so gathering more data will eventually be able to detect the most minuscule differences.

The (IMO) facetious answer from Fisher was in response to a relatively trivial question that at its premise is conflating 'significant difference' with 'practically relevant difference'.

It would be equivalent to a researcher coming into my office and asking "I weighed this lead weight labeled '25 gram' and it measured 25.0 grams. I believe it to be mislabeled, what should I do?" To which I could answer, "Get a more precise scale."

I believe the go-get-more-data approach is appropriate if the initial test is woefully underpowered to detect the magnitude of difference that is practically relevant.

  • $\begingroup$ The point though is that you need to incorporate the decision to get more data into the calculation of the p-value. $\endgroup$ – jsk Jul 15 '19 at 18:20
  • $\begingroup$ @jsk even if you change the p-value, you can still gather more data to find a significant result (albeit you'd need even more data). $\endgroup$ – Underminer Jul 15 '19 at 20:03
  • 1
    $\begingroup$ I could have been clearer. I'm not sure what exactly you mean by "you CAN still gather more data to find a significant result". I get that because the null hypothesis is generally never actually true, collecting more data will eventually lead to a significant result. I just wanted to draw attention to the fact that when calculating the p-value, you need to incorporate the decision to collect more data into the calculation of the p-value. This means that the decision rules (about collecting more data) need to be pre-specified prior to the original data collection. $\endgroup$ – jsk Jul 15 '19 at 21:21
  • $\begingroup$ @jsk even with a very conservative method of adjusting the p-value (e.g. Bonferroni correct, applicable in post-hoc analysis), there exists an additional sample size large enough that will overcome the correction. The point is: If you provide me with a p-value adjustment method (specified prior to original data collection or not), the true difference between the population distributions of the groups of interest, and insignificant preliminary results; and I can provide you with a large enough sample size that will get you significant results. Hence, more data is ALWAYS an answer. $\endgroup$ – Underminer Jul 16 '19 at 14:02

Thanks. There are a couple things to bear in mind here:

  1. The quote may be apocryphal.
  2. It's quite reasonable to go get more / better data, or data from a different source (more precise scale, cf., @Underminer's answer; different situation or controls; etc.), for a second study (cf., @Glen_b's comment). That is, you wouldn't analyze the additional data in conjunction with the original data: say you had N=10 with a non-significant result, you could gather another N=20 data and analyze them alone (not testing the full 30 together). If the quote isn't apocryphal, that could have been what Fisher had in mind.
  3. Fisher's philosophy of science was essentially Popperian. That is, the null wasn't necessarily something to reject perfunctorily so as to confirm your theory, but ideally could be your theory itself, such that rejection means your pet theory is wrong and you need to go back to the drawing board. In such a case, type I error inflation would not benefit the researcher. (On the other hand, this interpretation cuts against Fisher giving this advice unless he was being a quarrelsome, which would not have been out of character.)
  4. At any rate, it's worth pointing out that the reason I included that comment is that it illustrates something fundamental about the difference in the nature of the two approaches.
  • 1
    $\begingroup$ (Let's suppose someone other than Fisher said that quote, which doesn't affect its correctness) In response to your second point, AFAIK even if you don't analyze the additional data in conjunction with the original data, it's still $p$-hacking, and by doing so you are more likely to incorrectly accept the alternative hypothesis because the original data supporting the null hypothesis is discarded. On the other hand, this makes sense when applying your third point, as you won't keep testing until a null hypothesis is rejected (by chance). $\endgroup$ – nalzok Jul 15 '19 at 17:42
  • $\begingroup$ By the way, it would be great if you could elaborate on "the difference in the nature of the two approaches". Fisher's method sounds more... subjective, as I feel like he doesn't really care about the error rate, but I could be missing something. $\endgroup$ – nalzok Jul 15 '19 at 17:46
  • 1
    $\begingroup$ @nalzok, the difference is discussed in the original thread: the Neyman-Pearson approach assumes the study is a discrete event, you do it & walk away; Fisher's approach assumes the issue is under continuing investigation. Re: #2, if you analyze the data in isolation, it isn't p-hacking (unless maybe you run multiple studies & only publish the one that showed what you want). Re: #3, no, the null isn't accepted, you need to keep finding better ways to test your theory. $\endgroup$ – gung - Reinstate Monica Jul 15 '19 at 17:56
  • 1
    $\begingroup$ @nalzok, when $p$ is small, the test is significant; when $p$ is large, the test is non-significant. You don't reuse the data in the sense you seem to be implying & I doubt Fisher would have thought anyone should. $\endgroup$ – gung - Reinstate Monica Jul 15 '19 at 18:34
  • 1
    $\begingroup$ (+1) Sometimes I think we focus on the tree and miss the forest. Quite bluntly, when we have a hard problem, more data is usually better than less data. In most cases, more data is not a lot better. As Meng's insightful 2018 paper "Statistical paradises and paradoxes in big data (I)" suggests, getting better data (e.g. a well-selected sample) is much more beneficial than bigger data when we are trying to estimate an unknown quantity. But more data usually helps! $\endgroup$ – usεr11852 Jul 16 '19 at 18:49

What we call P-hacking is applying a significance test multiple times and only reporting the significance results. Whether this is good or bad is situationally dependent.

To explain, let's think about true effects in Bayesian terms, rather than null and alternative hypotheses. As long as we believe our effects of interest come from a continuous distribution, then we know the null hypothesis is false. However, in the case of a two-sided test, we don't know whether it is positive or negative. Under this light, we can think of p-values for two sided tests as a measure of how strong the evidence is that our estimate has the correct direction (i.e., positive or negative effect).

Under this interpretation, any significance test can have three possible outcomes: we see enough evidence to conclude the direction of the effect and we are correct, we see enough evidence to conclude the direction of the effect but we are wrong, or we don't see enough evidence to conclude the direction of the effect. Note that conditional that you have enough evidence (i.e., $p < \alpha$), the probability of getting the direction correct should be greater than the probability of getting it incorrect (unless you have some really crazy, really bad test), although as the true effect size approaches zero, the conditional probability of getting the direction correct given sufficient evidence approaches 0.5.

Now, consider what happens when you keep going back to get more data. Each time you get more data, your probability of getting the direction correct conditional on sufficient data only goes up. So under in this scenario, we should realize that by getting more data, although we are in fact increasing the probability of a type I error, we are also reducing the probability of mistakenly concluding the wrong direction.

Take this in contrast the more typical abuse of P-hacking; we test 100's of effect sizes that have good probability of being very small and only report the significant ones. Note that in this case, if all the effects are small, we have a near 50% chance of getting the direction wrong when we declare significance.

Of course, the produced p-values from this data-double-down should still come with a grain of salt. While, in general, you shouldn't have a problem with people collecting more data to be more certain about an effect size, this could be abused in other ways. For example, a clever PI might realize that instead of collecting all 100 data points at once, they could save a bunch of money and increase power by first collecting 50 data points, analyzing the data, and then collecting the next 50 if it's not significant. In this scenario, they increase the probability of getting the direction of the effect wrong conditional on declaring significance, since they are more likely to get the direction of the effect wrong with 50 data points than with 100 data points.

And finally, consider the implications of not getting more data when we have an insignificant result. That would imply never collecting more information on the topic, which won't really push the science forward, would it? One underpowered study would kill a whole field.

  • 1
    $\begingroup$ (+1) This is an interesting point of view, but can you elaborate on the difference between Fisher's methodology and that of the clever PI? Both collect more data because the initial test is insignificant, it seems. $\endgroup$ – nalzok Jul 16 '19 at 6:46
  • $\begingroup$ Also, I'm not sure what you mean by "although we are in fact increasing the probability of a type I error, we are also reducing the probability of mistakenly concluding the wrong direction". What is the null hypothesis here? IMO if you are doing a one-sided test, then "concluding the wrong direction" is "a type I error", and for two-sided tests, you shouldn't conclude the direction. $\endgroup$ – nalzok Jul 16 '19 at 7:31
  • $\begingroup$ Correct me if I'm wrong, but I think you suggest keeping collecting more data until a two-sided test is significant, and in this case, the type I error rate would be 100%. $\endgroup$ – nalzok Jul 16 '19 at 7:33
  • 1
    $\begingroup$ The key difference between what Fisher recommends and the clever/naive PI is that Fisher makes that call from the study being concluded. His options are either collect more data, or decide that he will never ever know the direction of the effect. On the other hand, the PI decides to underpower his initial study before he even sees the data. $\endgroup$ – Cliff AB Jul 16 '19 at 15:00
  • 1
    $\begingroup$ @nalzok: sure I'll try to take a look during nonwork hours :) $\endgroup$ – Cliff AB Jul 16 '19 at 17:12

If the alternative had a small a priori probability, then an experiment that fails to reject the null will decrease it further, making any further research even less cost-effective. For instance, suppose the a priori probability is .01. Then your entropy is .08 bits. If the probability gets reduced to .001, then your entropy is now .01. Thus, continuing to collect data is often not cost effective. One reason why it would be cost effective would be that knowing is so important that even the remaining .01 bits of entropy is worth reducing.

Another reason would be if the a priori probability was really high. If your a priori probability was more than 50%, then failing to reject the null increases your entropy, making it more cost effective to continue collecting data. An example would be when you're nearly certain that there is an effect, but don't know in which direction.

For instance, if you're a counterintelligence agent and you're sure that a department has a mole, and have narrowed it down to two suspects, and are doing some statistical analysis to decide which one, then a statistically insignificant result would justify collecting more data.

  • $\begingroup$ Why does failing to reject the null decrease its probability? While the absence of evidence isn't evidence of absence, I can't understand why it's evidence against absence. $\endgroup$ – nalzok Jul 16 '19 at 6:14
  • $\begingroup$ @nalzok I wrote "If the alternative had a small a priori probability, then an experiment that fails to reject the null will decrease it further" While "null" is the closest noun to "it", the null is not a quantity, and therefore can't decreased and is not a valid antecedent for "it". In addition "further" indicates that "it" refers to something already small. These facts point to the antecedent of "it" being the "small a priori probability" of the alternative. $\endgroup$ – Acccumulation Jul 16 '19 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.