9
$\begingroup$

I'm currently reading through the section on linear regression in ISLR, and the authors use the F-Statistic to determine if we should reject the null hypothesis (well they use the p-value of the F-Statistic).

I understand that, when the null hypothesis is false, the total sum of squares (the sum of squares where the response is just a constant value) will be much larger than the $RSS$, since our model will explain a large amount of the variation. Therefore, we should see a large F-Statistic value. However, I don't quite understand why, when the null hypothesis is true, we should expect to see a a F-Statistic $\approx 1$. The numerator, from my understanding, should be small, since not much of the variation is explained by our model. However, why should the denominator, which is $RSS/(n-p-1)$ be the same as the numerator? What does the denominator actually represent?

$\endgroup$
2
  • 1
    $\begingroup$ Hello, could you clarify what is ISLR? $\endgroup$ – kastellane Mar 2 at 23:19
  • $\begingroup$ Sorry. It's an acronym for a textbook called Introduction to statistical learning with applications in R. $\endgroup$ – BeepBoop Mar 2 at 23:28
6
$\begingroup$

Consider a linear model $y_i=\beta_0+x_i'\beta+u_i$, with $u_i\sim (0,\sigma^2)$.

The F-statistic is (see e.g. Proof that F-statistic follows F-distribution) $$ F = \frac{(\text{TSS}-\text{RSS})/p}{\text{RSS}/(n-p-1)},$$ with $TSS=\sum_i(y_i-\bar{y})^2$ and $RSS=\sum_i(y_i-\hat{y}_i)^2$ with $p$ the number of slope parameters.

Under classical assumptions, $\text{RSS}/(n-p-1)$ is an unbiased estimator of $\sigma^2$, i.e., $$E[\text{RSS}/(n-p-1)]=\sigma^2.$$

Likewise, it is a well-known result that, under the null $y_i=\beta_0+u_i$, the sample variance $\sum_i(y_i-\bar{y})^2/(n-1)$ is an unbiased estimator of $\sigma^2$, i.e., $$E[\text{TSS}/(n-1)]=\sigma^2.$$ (It always is an unbiased estimator of the variance of $\sigma^2_y$, the variance of $y$, which however does not coincide with the variance of the error under the alternative anymore, which is what gives the test its power.)

Putting things in the numerator together, $$ E[(\text{TSS}-\text{RSS})/p]=E[(n-1)\sigma^2-(n-p-1)\sigma^2]/p=\sigma^2 $$ So if you approximate $E(F)$ (of course, the expectation of a ratio is not generally the ratio of expectations), you get $$ E(F)\approx\frac{E[(\text{TSS}-\text{RSS})/p]}{E[\text{RSS}/(n-p-1)]}=\frac{\sigma^2}{\sigma^2}=1 $$ Actually, given that the F-statistic follows an F-distribution with $p$ and $d:=n-p-1$ degrees of freedom, we may use known results for the exact expectation of F-distributed random variables, namely that $$E(F)=\frac{d}{d-2}$$ when $d>2$. So $$E(F)=\frac{n-p-1}{n-p-1-2}=\frac{n-p-1}{n-p-3},$$ which will of course be close to 1 for cases where the sample size $n$ is large relative to the number of regressors. Hence, the above approximation works very well in this case.

Of course, what we have here is a result for the expected value of the F-statistic if the null is true. This does not mean that (like for any expectation) the statistic $F\approx1$, but that it will "hover around" 1 when we were to repeatedly compute F-statistic for situations in which the null is true. See e.g. the simulation provided at https://stats.stackexchange.com/a/258476/67799 for an illustration.

$\endgroup$
2
  • $\begingroup$ Sorry but, just to make sure I understand this correctly, the expected values we calculated will hold regardless if the null hypothesis is true or not right? If so, how can we infer that the null hypothesis is false when $F$ is far larger or smaller than $E(F)$? $\endgroup$ – BeepBoop Mar 3 at 20:58
  • 1
    $\begingroup$ You are right that this point was rather implicit in my answer before - I made an edit. $\endgroup$ – Christoph Hanck Mar 4 at 5:44
2
$\begingroup$

Another way to think about this: When there is no experimental effect, the only variation you have is within-subjects. So even when you split the participants into two groups, the same variation is endemic within each (no experimental variance, only within-subjects exists). Thus, your variance ratio of between / within is really just within / within, or 1.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.