Quoting gung's great answer
Allegedly, a researcher once approached Fisher with 'non-significant' results, asking him what he should do, and Fisher said, 'go get more data'.
From a Neyman-Pearson perspective, this is blatant $p$-hacking, but is there a use case where Fisher's go-get-more-data approach makes sense?
The frequentist paradigm is a conflation of Fisher's and Neyman-Pearson's views. Only in using one approach and another interpretation do problems arise.
It should seem strange to anyone that collecting more data is problematic, as more data is more evidence. Indeed, the problem lies not in collecting more data, but in using the $p$-value to decide to do so, when it is also the measure of interest. Collecting more data based on the $p$-value is only $p$-hacking if you compute a new $p$-value.
If you have insufficient evidence to make a satisfactory conclusion about the research question, then by all means, go get more data. However, concede that you are now past the NHST stage of your research, and focus instead on quantifying the effect of interest.
An interesting note is that Bayesians do not suffer from this dilemma. Consider the following as an example:
Given a big enough sample size, a test will always show significant results, unless the true effect size is exactly zero, as discussed here. In practice, the true effect size is not zero, so gathering more data will eventually be able to detect the most minuscule differences.
The (IMO) facetious answer from Fisher was in response to a relatively trivial question that at its premise is conflating 'significant difference' with 'practically relevant difference'.
It would be equivalent to a researcher coming into my office and asking "I weighed this lead weight labeled '25 gram' and it measured 25.0 grams. I believe it to be mislabeled, what should I do?" To which I could answer, "Get a more precise scale."
That being said, I believe the go-get-more-data approach is appropriate if the initial test is woefully underpowered to detect the magnitude of difference that is practically relevant.
What we call P-hacking is applying a significance test multiple times and only reporting the significance results. Whether this is good or bad is situationally dependent.
To explain, let's think about true effects in Bayesian terms, rather than null and alternative hypotheses. As long as we believe our effects of interest come from a continuous distribution, then we know the null hypothesis is false. However, in the case of a two-sided test, we don't know whether it is positive or negative. Under this light, we can think of p-values for two sided tests as a measure of how strong the evidence is that our estimate has the correct direction (i.e., positive or negative effect).
Under this interpretation, any significance test can have three possible outcomes: we see enough evidence to conclude the direction of the effect and we are correct, we see enough evidence to conclude the direction of the effect but we are wrong, or we don't see enough evidence to conclude the direction of the effect. Note that conditional that you have enough evidence (i.e., $p < \alpha$), the probability of getting the direction correct should be greater than the probability of getting it incorrect (unless you have some really crazy, really bad test), although as the true effect size approaches zero, the conditional probability of getting the direction correct given sufficient evidence approaches 0.5.
Now, consider what happens when you keep going back to get more data. Each time you get more data, your probability of getting the direction correct conditional on sufficient data only goes up. So under in this scenario, we should realize that by getting more data, although we are in fact increasing the probability of a type I error, we are also reducing the probability of mistakenly concluding the wrong direction.
Take this in contrast the more typical abuse of P-hacking; we test 100's of effect sizes that have good probability of being very small and only report the significant ones. Note that in this case, if all the effects are small, we have a near 50% chance of getting the direction wrong when we declare significance.
Of course, the produced p-values from this data-double-down should still come with a grain of salt. While, in general, you shouldn't have a problem with people collecting more data to be more certain about an effect size, this could be abused in other ways. For example, a clever PI might realize that instead of collecting all 100 data points at once, they could save a bunch of money and increase power by first collecting 50 data points, analyzing the data, and then collecting the next 50 if it's not significant. In this scenario, they increase the probability of getting the direction of the effect wrong conditional on declaring significance, since they are more likely to get the direction of the effect wrong with 50 data points than with 100 data points.
And finally, consider the implications of not getting more data when we have an insignificant result. That would imply never collecting more information on the topic, which won't really push the science forward, would it? One underpowered study would kill a whole field.
Thanks. There are a couple things to bear in mind here:
If the alternative had a small a priori probability, then an experiment that fails to reject the null will decrease it further, making any further research even less cost-effective. For instance, suppose the a priori probability is .01. Then your entropy is .08 bits. If the probability gets reduced to .001, then your entropy is now .01. Thus, continuing to collect data is often not cost effective. One reason why it would be cost effective would be that knowing is so important that even the remaining .01 bits of entropy is worth reducing.
Another reason would be if the a priori probability was really high. If your a priori probability was more than 50%, then failing to reject the null increases your entropy, making it more cost effective to continue collecting data. An example would be when you're nearly certain that there is an effect, but don't know in which direction.
For instance, if you're a counterintelligence agent and you're sure that a department has a mole, and have narrowed it down to two suspects, and are doing some statistical analysis to decide which one, then a statistically insignificant result would justify collecting more data.
