Rule of Thumb for Accepting the Null Hypothesis

Question

Usually, hypothesis testing is performed with the goal to make a conclusions about the statistical significance of an effect, i.e. $H_0 \ \hat{=} \ \text{No Effect} $ vs. $H_1 \ \hat{=} \ \text{Effect} $. Often, as a rule of thumb the p-value for rejecting $H_0$ is chosen to be at most $5\%$ or $1\%$.

Are their rule of thumbs for the p-value when the question is not only whether there is a sigificant effect but also if $H_0$ can be accepted?

For instance, Christoffersen (1998) proposed a test for evaluating wheter Value-at-Risk- or quantile-forecast have the unconditional correct coverage of the underlying process. Here, $H_0 \hat{=} \ \text{correct unconditional coverage}$. I want to use this test for evaluating my own forecast. What could be a good choice for the maximum p-value to accept $H_0$?

Answer 1

As pointed out by others you cannot really demonstrate that a null hypothesis is true. Looking at the p-value and arguing that the data are not inconsistent with the null hypothesis, is not a useful criterion for accepting a null, because a high p-value may result simply from very limited data.

One reasonable approach is to argue that deviations by e.g. $\pm \delta$ from the value $\theta_0$ of a parameter $\theta$ assumed under the null hypothesis is irrelevant. In that case the alternative hypothesis is $\theta \in [\theta_0-\delta, \theta_0+\delta]$ and the null hypothesis is $\theta \notin [\theta_0-\delta, \theta_0+\delta]$. You can perform a level $\alpha$ test by looking at whether a $1-\alpha$ confidence interval lies completely within this interval. One challenge is the choice of $\delta$, which could be based on many types of arguments, but should really be such that if we saw an effect of this size and it were statistically significant, we would still dismiss it as practically irrelevant (at least for the application we are talking about).

Answer 2

Null hypotheses cannot be accepted, and they are virtually never true. A null hypothesis is a point hypothesis. The null hypothesis in a t-test is, that the difference of the mean is 0.00000000000000000000 with an infinite number of zeros after the dot. Nobody ever wants to proove that. What you want to show is, that the difference is so small as to not matter for any practical purposes.

Ways to show that would include a power analysis: If your N is large enough so that it should detect relevant effects and there is no effect shown, than that has a meaning. A different approach would be to compute a confidence interval and show, that it is small enough around zero, that values within that confidence intervall are of no practical relevance. The last sentence will be opposed by Baysians, who, of course, have their own answer by providing probable intervals for the difference, where again, you could argue, why the differences are likely to be so small as to not be relevant.