Regression hypothesis testing via out-of-sample testing

Question

Let's consider two linear models.

$$\text{Full model}\\\mathbb{E}\big{[}Y\big{\vert} X_1,\dots,X_p, X_{p+1},\dots,X_{p+k}\big{]}=\\\beta_0 + \bigg[\beta_1X_1+\dots + \beta_pX_p\bigg] + \bigg[\beta_{p+1}X_{p+1}+\dots + \beta_{p+k}X_{p+k}\bigg]$$

$$\text{Reduced model}\\ \mathbb{E}\big{[}Y\big{\vert}X_1,\dots,X_{p}\big{]}=\beta_0 + \bigg[\beta_1X_1+\dots + \beta_pX_p\bigg]$$

If we want to test the full model against the reduced model and do an F-test, we are essentially saying that, while $SSE$ will be lower for the full model, we want to know if it is enough of an improvement to justify including the extra parameters. (Ditto for deviance testing in GLMs.)

This is alluding to overfitting, and machine learning people check for overfitting by testing out of sample.

It seems like we could do model inference by testing out-of-sample and seeing how each model performs. If the full model has significantly$^{\dagger}$ better performance, then we say that the $X_{p+1},\dots,X_{p+k}$ parameters contribute significantly, the same conclusion that we would make for an F-test of the full model versus the reduced model.

This seems like a reasonable approach to doing model comparisons, and it would encompass the usual regression inferences like ANOVA and ANCOVA.

Has any work been done on this?

$^{\dagger}$There would be some kind of hypothesis test (would there?), though I am not sure what.

Answer 1

The $R^2$ of the full model is greater than the one the short model; this holds by construction, so it is no so informative result. The F test, on the full model, about the parameters not included in the short, tell us a bit more because if the F test is not significant we have, at least in certain sense, evidence that full model is overfitted.

Indeed $R^2_{adj}$ (adjusted) care about overfitting (intended as above, then without refers to out of sample concepts) and work as follow. If the short model have just one variable less than the full, is possible to show that the $R^2_{adj}$ of the short model is lower than the one of the full only if the t-stat of the parameter of the variable included in the full model but excluded from the short is greater that $1$ in absolute value (so if his p-value is lower than $\approxeq0,32$). Now the F stat and t stat are linked. So we can generalize this story and conclude that, in the general comparison between full and short model, F test says something about overfitting. I suppose that this kind of link is what you looking for but I do not have more details at hand.

However, the story above, and the concept of overfitting in particular, is related to prediction. About this topic, the out of sample (test) performance vs in sample (train) ones are the core (under/overfitting), while coefficients values/test are not. In prediction, measures like: $R^2_{adj}$, t-test, F-test, AIC,BIC; can help in model selection but not in model evaluation. For model evaluation, out of sample measure like $R^2_{oos}$ (out of sample) should be used (read here: How to calculate out of sample R squared? Should $ R^2$ be calculated on training data or test data?)

Moreover your question can open the door about the debate on the meaning of regression model more in general. I put here several related insights: What is the relationship between minimizing prediction error versus parameter estimation error?