Now that I've rejected the null hypothesis what's next?

Question

I've time and again rejected or failed to reject the null hypothesis. In the failure to reject case, you conclude that there isn't sufficient evidence for rejection and you "move on" (i.e., either gather more data, end the experiment etc.,)

But when you "do" reject the null hypothesis, providing some evidence for the alternative hypothesis you can't really "prove" that your alternative hypothesis indeed holds true.

So, what are the common next steps once you reject the null hypothesis? What tools/techniques does one adopt to "analyze the problem further" to make more the findings more conclusive? What are the logical "next steps" as a statistician warranting further analysis?

For example:

$H_0: \mu_1 = \mu_0$

$H_1: \mu_1 > \mu_0$ (say we know the expected direction)

Once we reject the null hypothesis at some level of significance we have "some evidence" for the alternative to be true, but we can't draw that conclusion. If I really want to draw that conclusion conclusively (pardon the double word play) what should I do?

I've never pondered this question during my undergrad days but now that I'm doing a fair deal of hypotheses testing I can't help but wonder what's ahead :)

Answer 1

You can generally continue to improve your estimate of whatever parameter you might be testing with more data. Stopping data collection once a test achieves some semi-arbitrary degree of significance is a good way to make bad inferences. That analysts may misunderstand a significant result as a sign that the job is done is one of many unintended consequences of the Neyman–Pearson framework, according to which people interpret p values as cause to either reject or fail to reject a null without reservation depending on which side of the critical threshold they fall on.

Without considering Bayesian alternatives to the frequentist paradigm (hopefully someone else will), confidence intervals continue to be more informative well beyond the point at which a basic null hypothesis can be rejected. Assuming collecting more data would just make your basic significance test achieve even greater significance (and not reveal that your earlier finding of significance was a false positive), you might find this useless because you'd reject the null either way. However, in this scenario, your confidence interval around the parameter in question would continue to shrink, improving the degree of confidence with which you can describe your population of interest precisely.

---

Here's a very simple example in r – testing the null hypothesis that $\mu=0$ for a simulated variable:

One Sample t-test

data:  rnorm(99) 
t = -2.057, df = 98, p-value = 0.04234
alternative hypothesis: true mean is not equal to 0 
95 percent confidence interval:
 -0.377762241 -0.006780574 
sample estimates:
 mean of x 
-0.1922714

Here I just used t.test(rnorm(99)), and I happened to get a false positive (assuming I've defaulted to $\alpha=.05$ as my choice of acceptable false positive error rate). If I ignore the confidence interval, I can claim my sample comes from a population with a mean that differs significantly from zero. Technically the confidence interval doesn't dispute this either, but it suggests that the mean could be very close to zero, or even further from it than I think based on this sample. Of course, I know the null is actually literally true here, because the mean of the rnorm population defaults to zero, but one rarely knows with real data.

Running this again as set.seed(8);t.test(rnorm(99,1)) produces a sample mean of .91, a p = 5.3E-13, and a 95% confidence interval for $\mu=[.69,1.12]$. This time I can be quite confident that the null is false, especially because I constructed it to be by setting the mean of my simulated data to 1.

Still, say it's important to know how different from zero it is; maybe a mean of .8 would be too close to zero for the difference to matter. I can see I don't have enough data to rule out the possibility that $\mu=.8$ from both my confidence interval and from a t-test with mu=.8, which gives a p = .33. My sample mean is high enough to seem meaningfully different from zero according to this .8 threshold though; collecting more data can help improve my confidence that the difference is at least this large, and not just trivially larger than zero.

Since I'm "collecting data" by simulation, I can be a little unrealistic and increase my sample size by an order of magnitude. Running set.seed(8);t.test(rnorm(999,1),mu=.8) reveals that more data continue to be useful after rejecting the null hypothesis of $\mu=0$ in this scenario, because I can now reject a null of $\mu=.8$ with my larger sample. The confidence interval of $\mu=[.90,1.02]$ even suggests I could've rejected null hypotheses up to $\mu=.89$ if I'd set out to do so initially.

I can't revise my null hypothesis after the fact, but without collecting new data to test an even stronger hypothesis after this result, I can say with 95% confidence that replicating my "study" would allow me to reject a $H_0:\mu=.9$. Again, just because I can simulate this easily, I'll rerun the code as set.seed(9);t.test(rnorm(999,1),mu=.9): doing so demonstrates my confidence wasn't misplaced.

Testing progressively more stringent null hypotheses, or better yet, simply focusing on shrinking your confidence intervals is just one way to proceed. Of course, most studies that reject null hypotheses lay the groundwork for other studies that build on the alternative hypothesis. E.g., if I was testing an alternative hypothesis that a correlation is greater than zero, I could test for mediators or moderators in a follow-up study next...and while I'm at it, I'd definitely want to make sure I could replicate the original result.

---

Another approach to consider is equivalence testing. If you want to conclude that a parameter is within a certain range of possible values, not just different from a single value, you can specify that range of values you'd want the parameter to lie within according to your conventional alternative hypothesis and test it against a different set of null hypotheses that together represent the possibility that the parameter lies outside that range. This last possibility might be most similar to what you had in mind when you wrote:

We have "some evidence" for the alternative to be true, but we can't draw that conclusion. If I really want to draw that conclusion conclusively...

Here's an example using similar data as above (using set.seed(8), rnorm(99) is the same as rnorm(99,1)-1, so the sample mean is -.09). Say I want to test the null hypothesis of two one-sided t-tests that jointly posit that the sample mean is not between -.2 and .2. This corresponds loosely to the previous example's premise, according to which I wanted to test if $\mu=.8$. The difference is that I've shifted my data down by 1, and I'm now going to perform two one-sided tests of the alternative hypothesis that $-.2\le\mu\le.2$. Here's how that looks:

require(equivalence);set.seed(8);tost(rnorm(99),epsilon=.2)

tost sets the confidence level of the interval to 90%, so the confidence interval around the sample mean of -.09 is $\mu=[-.27,.09]$, and p = .17. However, running this again with rnorm(999) (and the same seed) shrinks the 90% confidence interval to $\mu=[-.09,.01]$, which is within the equivalence range specified in the null hypothesis with p = 4.55E-07.

I still think the confidence interval is more interesting than the equivalence test result. It represents what the data suggest the population mean is more specifically than the alternative hypothesis, and suggests I can be reasonably confident that it lies within an even smaller interval than I've specified in the alternative hypothesis. To demonstrate, I'll abuse my unrealistic powers of simulation once more and "replicate" using set.seed(7);tost(rnorm(999),epsilon=.09345092): sure enough, p = .002.

Answer 2

Note first that @Nick Stauner makes some very important arguments regarding optional stopping. If you repeatedly test the data as samples come in, stopping once a test is significant, you're all but guaranteed a significant result. However, a guaranteed result is practically worthless.

In the following, I'll present my best attempts to elaborate on a deductivist, skeptical, falsificationist position. It's certainly not the only one, but I think a rather mainstream one, or at least one with a bit of tradition.

As far as I understand it, Fisher originally introduced significance tests as a first step in data exploration - establish which factors might be worth investigating further. Unless the null hypothesis you've put under test actually was the critical hypothesis your favoured theory depended on (unlikely), in a way, your initial test was rather exploratory in nature. Amongst the possible steps following exploration I see

• Further exploration
• Parameter Estimation
• Prediction & Confirmation

Further exploration consists of follow-up tests where you try to infer if any variables you have information on moderate or interact with your effect. For example, maybe the age of the participants plays a role? Note that such analyses must be clearly labelled as exploratory, or they basically amount to lying. If you stumble upon something, it first requires confirmation. Generally, you should always be clear- both in your thoughts, and in your writings - about when you're working exploratory, and when confirmatory.

Next, once you have established that you have no confidence in one parameter's value being precisely zero - once you have decided you'll for now consider the factor under test to have some influence - one feasible next step could be to further estimate the precise value of the parameter. For example, for now, you've only excluded one value, 0 (assuming a two-sided test). However, your data also cast doubt on many further possible values.

A (100-$\alpha$)% Confidence Interval/CI contains the range of parameter values not rejected at p<$\alpha$, corresponding to the many more possible hypotheses your data also concern beyond your initial H0. Since your test is significant, the value associated with H0 is not amongst them. But many extremely large and small values will also be excluded.

Hume famously argued we can never inductively prove correct a statement. Generally, non-trivial hypotheses are always a lot easier to falsify than to support; being easy to falsify in principle (by being non-trivial, making precise predictions), but yet not being falsified so far is in fact one of the highest virtues of a theory.

So a CI won't get you to proving a specific value. However, it narrows down the candidate set. Maybe the only candidates left alive help you decide between two theories both incompatible with H0. For example, maybe 0 is excluded, but theory 1 predicts a value around 5, and theory 2 predicts a value around 15. If your 95% CI includes 5, but excludes 15, you have now also lost confidence in theory 2, but theory 1 remains in the game. Note that this is actually independent of your initial test being significant - even if 0 is amongst the values not rejected, many values will be rejected. Maybe for some other researchers, some of these values were of interest.

After you have thus somewhat specified your understanding of the effect at hand, you could ideally make a more precise prediction for a follow-up confirmatory experiment that would aim to test a more precise hypothesis you can derive from your current analysis. Admittedly, rejecting your initial statistical null hypothesis wasn't that severe of a test of your original research hypothesis, wasn't it? Many more explanations than the one you prefer do not depend on H0. Also, since you never were in danger to actually accept H0, you were in no position to falsify your favoured theory! So you need a more severe test. Arguably, this is actually what you want; you do not want to prove your theory, you want to put it under increasingly severe tests, attempting to falsify it. Withstanding such genuine (but fair) efforts to disprove it is the best a theory can deliver. But for a severe test, you need a more precise theory than "0 it ain't".

You now have learned multiple important facts concerning a confirmatory study; for example, you have an idea of the variance and effect magnitude in question, allowing you to estimate the required sample size for a follow-up study via power analysis. You can also predict a specific value and assume a region of practical equivalence/ROPE around it. You won't ever be able to prove that this specific value is the true value; however, if the CI from a follow-up experiment falls entirely within your ROPE, you have corroborating evidence for your theory (and possibly brought in trouble the competition).

Answer 3

The idea that you cannot prove a positive scientific proposition, but only disprove one, is a principle of Popper's falsificationism. I do agree that you cannot prove an effect is exactly equal to any given point value (cf., my answer here: Why do statisticians say a non-significant result means "you cannot reject the null" as opposed to accept the null hypothesis?). But so what?

People (or at least I) complain a lot about hypothesis testing. This is because $p$-values are commonly misunderstood, and hypothesis tests are used for tasks they logically cannot accomplish. For example, hypothesis testing should not be used to generate hypotheses or to select variables. Moreover, with observational data essentially all 'nil' null hypotheses must be false, so testing such makes little sense. However, scientists often do have a-priori hypotheses suggested by current theories that they want to test, and in a true experiment a nil null could be true, so testing it is perfectly reasonable. Typically, researchers do have some reason to suspect that the null might be false, so a significant result in conjunction with a strong experiment is a valid piece of information.

You can always form confidence intervals to get a clearer picture of the precision of your estimate, and continue to collect more data to increase its precision. Nonetheless, in economic terms you will get diminishing returns. At some point, you simply do not believe the null hypothesis provides a reasonable account of the phenomenon under study. In which case, why are you bothering?

If there are others in your field who are not yet convinced, but would be with more (of the same) data, then you could continue, but this seems like an uncommon situation. It seems more likely to me that the skeptics have other, substantive concerns regarding whether that line of inquiry is sufficiently informative about the underlying question. Thus, you need to determine the nature of those concerns, and if you think they merit the work, seek out different data that more adequately address the issues at hand. For example, you might try to replicate the finding using a different measure, in a different setting, and/or with different control conditions.

On the other hand, everyone (more or less) may be satisfied with your data and conclusions (congratulations!). Under such happy circumstances, there are two directions you could pursue to further your research program:

1. A reductionist approach would seek to understand the mechanisms that produce the effect you have established. In statistical terms, you would often be seeking mediators and/or a refining of the pattern of causal forces that connect the variables you have shown to be related.

2. You could also move in the other direction by seeking to integrate your findings into a larger pattern. This is a kind of systems thinking. G.H. Hardy once defined the elegance of a theory as the range of phenomena that it could explain in conjunction with the ease and magnitude of the epistemic shift it induced. Of course, you may not be so lucky that the phenomenon you have established is quite that deep, however it can still be part of something bigger than itself. Establishing a link between $B$ and $C$ that makes it possible to see that $A$ unifies disparate phenomena can be just as important to the process, and just as much a crystallizing moment, as the discovery of $A$ itself.

tl;dr: If you have sufficient evidence for your purposes that the null is false, figure out what other theoretically motivated questions you could try to answer and move on.

Answer 4

One think I would like to add is that your question reminds me of my younger self: I wanted desperately to prove my hypothesis because I did not how to write "the hypothesis was wrong" in a way which helped to improve the paper I was writing. But then I realized that the "damn my absolutely lovely hypothesis cannot be proven" also holds scientific value: 1. Think about WHY your hypothesis doesn't hold water. It's some problem with the data, or probabyly something with the hypothesis itself? 2. What are the consequences for older research?

As an example: I wrote my master's thesis about ethnic conflect using a then new dataset which was larger than previous datasets. I tested several disputed hypothesis like "oil fuels ethnic conflict" or "mountaineous regrions are more likely to expierience conflict". I could not prove that oil fuels ethnic conflict - but I wrote two pages about how the quality of the available oil-dataset impacted the analysis (the dataset itself is a time-series, the oil-well dataset is not). The "mountains are causing conflict" thesis was also a failure - but a fruitful one: previous research analyzed this thesis with country-level data (e.g. mean height of the country or so), while I did it on the level of ethnic groups - so I spent a lot of paper to discuss the differences and why my analysis was better than other famous research...

Keep in mind: disproving a hypothesis is not a failure but a result as good as a proved hypothesis.

Answer 5

There is a method for combing probabilities across studies described here. You should not apply the formula blindly without considering the pattern of results.