2
$\begingroup$

I've seen this overall F-test several times, and when they compare F-statistics or $\alpha=5$% with its p-value always use right tailed, whereas the hypothesis is formualed as

$$H_0:\beta_0=\beta_1=\beta_2=\beta_3=0\\ H_1: \text{at least one of a tested parameter is nonzero}$$

enter image description here

enter image description here

enter image description here

enter image description here

first-two graphs are example one, second two graphs are example two Just pay attention to the critical value corresponding to $\alpha=0.05$ and p-value corresponding to $F-statistics$, their number from only right-tailed, why not two-tailed?

$\endgroup$
8
  • $\begingroup$ I may expand on this tomorrow if there are no answers (though there is no “calling dibs” on Cross Validated), but the gist is that the F-test is asking if one model is better than another. If that is the case, then you seem “significant” the parameters in the larger model but not in the smaller model. If you do a two-sided test, you are considering that the larger model is worse than the smaller model, which makes no sense from the standpoint of trying to show that a predictor impacts the response variable. $\endgroup$
    – Dave
    Commented Oct 26, 2021 at 1:56
  • 2
    $\begingroup$ The explanation is similar to the explanation for one-way ANOVA here: Why do we use a one-tailed test F-test in analysis of variance (ANOVA)? -- which is to say, when the assumptions of the test hold, the test is organized so that large F values will tend to occur when $H_0$ is false in any "direction"; unusually small values will indicate a situation highly consistent with the null (that might be surprising in some sense but not in a way that would lead you to reject the null). $\endgroup$
    – Glen_b
    Commented Oct 26, 2021 at 2:10
  • 1
    $\begingroup$ Consistent with @Glen_b you can think of it this way: If you are interesting in quantifying evidence for differences you look at the right tail. If you want to test whether means are more similar than chance would likely produce under the null, then compute the left tail probability. $\endgroup$ Commented Oct 26, 2021 at 14:40
  • 2
    $\begingroup$ The low $F$-stat does suggest a difference in variance, just not in a way that is useful to testing in regression. It suggests that the variance of the error term in the larger model is greater than the variance of the error term in the smaller model. If you use an $F$-test to compare the variances of two distributions, instead of comparing their means, then you might care if cats have a lower variance than dogs just as much as if dogs have a lower variance than cats. $\endgroup$
    – Dave
    Commented Oct 26, 2021 at 17:16
  • 1
    $\begingroup$ @Dave Tremendous helpful. Thank you so much $\endgroup$
    – LJNG
    Commented Oct 26, 2021 at 17:18

1 Answer 1

1
$\begingroup$

The gist is that the $F$-test is asking if one model is better than another. If that is the case, then you seem “significant” the parameters in the larger model but not in the smaller model. If you do a two-sided test, you are considering that the larger model is worse than the smaller model, which makes no sense from the standpoint of trying to show that a predictor impacts the response variable.

In a bit more detail, the $F$-test compares two models on squared differences between predicted values and observed values (square loss), mathematically expressed as $(y_i - \hat y_i)^2$. The models are nested, meaning that the larger one has all of the parameters that the smaller one has, plus some extra ones (maybe just one, maybe a bunch). The test then asks if the reduction square loss is enough to convince us that the additional parameters are not zero (reject the null hypothesis). If those parameters are not zero, then the variables to which they correspond can be considered contributors to the values of $y$.

This is why we only care about the one-sided test. If the evidence is that the larger model provides a worse fit than the smaller model, then that does not help us show that the additional parameters are non-zero.

EDIT

A low $F$-stat does suggest a difference in variance, just not in a way that is useful to testing in regression. It suggests that the variance of the error term in the larger model is greater than the variance of the error term in the smaller model;$^{\dagger}$ this is not compelling evidence that a feature influences $y$ and, therefore, is "significant". If you use an $F$-test to compare the variances of two distributions, instead of comparing their means (such as a $t$-test), then you might care if cats have a lower variance than dogs just as much as if dogs have a lower variance than cats.

$^{\dagger}$This is a bizarre occurrence. Larger models have smaller (or at least equal) error term variances than smaller (nested) model error term variances, except under specific circumstances. The only time I have heard of this happening is when there is extreme multicollinearity, such as including polynomial terms that wind up being highly multicollinear.

$\endgroup$
3
  • $\begingroup$ Thank you for your reply. Would you please further enlighten me about "If you do a two-sided test, you are considering that the larger model is worse than the smaller model", I don't get this argument. $\endgroup$
    – LJNG
    Commented Oct 26, 2021 at 17:15
  • 1
    $\begingroup$ @LJNG I just posted a comment to your question. See if that helps you. $\endgroup$
    – Dave
    Commented Oct 26, 2021 at 17:16
  • $\begingroup$ Would you please also check out my this question. stats.stackexchange.com/questions/549416/… $\endgroup$
    – LJNG
    Commented Oct 26, 2021 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.