Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I might be missing something obvious here. In linear regression, F statistic is defined as (explained variance / p) / mean squared error, where p is number of independent variables. When there is no relationship between independent and dependent variables, wouldn't explained variance close to zero and thus F statistic close to zero?
More specifically, I'm referring to page 76 of "an introduction to statistical learning with applications in R".
Thank you
I believe this question is more or less addressed here: Intuitive explanation of the F-statistic formula?. The idea is that the regression mean square is an estimate of $\sigma^2$ when the null model holds, and so both the numerator and denominator of the $F$ statistic can roughly be expected to be close to the error variance when there is no relationship between the predictors and response.
@dsaxton has already given a good answer. Essentially, you can think of the F-test as:
(error variance + regression effects) / (error variance)
The numerator is the explained variance or variance between groups. The denominator is the explained variance within groups. If our null hypothesis is correct, the regression effects should be close to zero and therefore the F-statistic should be close to 1. More mathematically, we can say under the null hypothesis, the expectation of ratio of the denominator and numerator is close to one.
Note that it's not the absolute value matters. So you can have a model with small explained variance, but if the unexplained variance is even smaller the overall ratio will be far away from 1. Therefore you can conclude that your regression effect is significant.
