2
$\begingroup$

Usually, hypothesis testing is performed with the goal to make a conclusions about the statistical significance of an effect, i.e. $H_0 \ \hat{=} \ \text{No Effect} $ vs. $H_1 \ \hat{=} \ \text{Effect} $. Often, as a rule of thumb the p-value for rejecting $H_0$ is chosen to be at most $5\%$ or $1\%$.

Are their rule of thumbs for the p-value when the question is not only whether there is a sigificant effect but also if $H_0$ can be accepted?

For instance, Christoffersen (1998) proposed a test for evaluating wheter Value-at-Risk- or quantile-forecast have the unconditional correct coverage of the underlying process. Here, $H_0 \hat{=} \ \text{correct unconditional coverage}$. I want to use this test for evaluating my own forecast. What could be a good choice for the maximum p-value to accept $H_0$?

$\endgroup$
  • 5
    $\begingroup$ Strictly speaking, within the null-hypothesis significance testing framework there is no way of accepting the null; however, as you point out there proposals, e.g. this, for how people may go about it. $\endgroup$ – Ian_Fin Jul 6 '16 at 13:21
  • $\begingroup$ @Ian_Fin you should consider providing answer based on your comment. $\endgroup$ – Tim Jul 6 '16 at 13:46
3
$\begingroup$

As pointed out by others you cannot really demonstrate that a null hypothesis is true. Looking at the p-value and arguing that the data are not inconsistent with the null hypothesis, is not a useful criterion for accepting a null, because a high p-value may result simply from very limited data.

One reasonable approach is to argue that deviations by e.g. $\pm \delta$ from the value $\theta_0$ of a parameter $\theta$ assumed under the null hypothesis is irrelevant. In that case the alternative hypothesis is $\theta \in [\theta_0-\delta, \theta_0+\delta]$ and the null hypothesis is $\theta \notin [\theta_0-\delta, \theta_0+\delta]$. You can perform a level $\alpha$ test by looking at whether a $1-\alpha$ confidence interval lies completely within this interval. One challenge is the choice of $\delta$, which could be based on many types of arguments, but should really be such that if we saw an effect of this size and it were statistically significant, we would still dismiss it as practically irrelevant (at least for the application we are talking about).

$\endgroup$
0
$\begingroup$

Null hypotheses cannot be accepted, and they are virtually never true. A null hypothesis is a point hypothesis. The null hypothesis in a t-test is, that the difference of the mean is 0.00000000000000000000 with an infinite number of zeros after the dot. Nobody ever wants to proove that. What you want to show is, that the difference is so small as to not matter for any practical purposes.

Ways to show that would include a power analysis: If your N is large enough so that it should detect relevant effects and there is no effect shown, than that has a meaning. A different approach would be to compute a confidence interval and show, that it is small enough around zero, that values within that confidence intervall are of no practical relevance. The last sentence will be opposed by Baysians, who, of course, have their own answer by providing probable intervals for the difference, where again, you could argue, why the differences are likely to be so small as to not be relevant.

$\endgroup$
  • 5
    $\begingroup$ A null hypothesis does not have to be a point null. $\endgroup$ – amoeba Jul 6 '16 at 14:03
  • $\begingroup$ In addition to giving a +1 to the point made by @amoeba , I'd also point out that the value of post hoc power analyses is disputed, as in, e.g., this $\endgroup$ – Ian_Fin Jul 6 '16 at 14:12
  • 2
    $\begingroup$ @Peter Flom one-tailed test like in a one-sided t-test are examples, I have seen in real life. $\endgroup$ – Bernhard Jan 21 '17 at 18:33
  • 1
    $\begingroup$ @PeterFlom non-inferiority tests (a commonly used example of the one-sided tests Bernhard mentions) and (bio)-equivalence (like the one given my answer) tests - as a non-one-sided exampled - would be non-point null hypotheses that one sees a lot and that also make a lot of sense. $\endgroup$ – Björn Jan 22 '17 at 17:50
  • 1
    $\begingroup$ @PeterFlom Any one-sided test has a non-point null, as already mentioned above. $\endgroup$ – amoeba Jan 23 '17 at 8:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.