3
$\begingroup$

Does F-test for the overall significance of the OLS model reflect the overfitting from the inclusion of multiple regressors? For example, does the inclusion of noise as additional regressors to a model decrease the probability of determining the model fit as significant?

My guess is yes due to the following. While the inference method (OLS) of the model does not have any regularization to avoid overfitting by itself, the F-test estimates the uncertainty of the model fit with the degree of freedom from the number of regressors. Such reflection of the variance/uncertainty of the estimators to test the significance may be regarded as a reflection of the risk of overfitting in the decision of significance.

From the above reasoning, even though OLS always lead to less SSR (sum of squares of residuals) in training data as we increase the the number of regressors, the degree of the freedom will compensate in the final decision of overall model significance. Thus, F-test has the property of reflecting the overfitting as other permutation test and cross-validations.

Is such understanding correct?

$\endgroup$
1
$\begingroup$

Indeed, it is possible to write the F-statistic (to be precise, the one that is valid under homoskedasticity) in a format which is in line with your reasoning:

$$F=\frac{(R^2_{ur}-R^2_{r})/q}{(1-R^2_{ur})/(n-k_{ur}-1)}$$ where $R^2_{r}$ is the $R^2$ of the restricted regression and $R^2_{ur}$ that of the unrestricted regression. $q$ denotes the number of restrictions imposed under the null and $k_{ur}$ the number of regressors in the unrestricted regression.

So basically, we ask if the inevitable increase in fit from adding regresors ($R^2_{ur}\geq R^2_{r}$) is "large enough" to not just result from overfitting.

"Large enough" is, at level $\alpha$, measured by whether $F$ is larger than (given normality assumptions on the errors) the corresponding quantile of the $F$ distribution. Alternatively, $F$ may be compared to the critical values of a "scaled" $\chi^2_q$ distribution $\chi^2_q/q$ for large $n$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.