# Regression hypothesis testing via out-of-sample testing

Let's consider two linear models.

$$\text{Full model}\\\mathbb{E}\big{[}Y\big{\vert} X_1,\dots,X_p, X_{p+1},\dots,X_{p+k}\big{]}=\\\beta_0 + \bigg[\beta_1X_1+\dots + \beta_pX_p\bigg] + \bigg[\beta_{p+1}X_{p+1}+\dots + \beta_{p+k}X_{p+k}\bigg]$$

$$\text{Reduced model}\\ \mathbb{E}\big{[}Y\big{\vert}X_1,\dots,X_{p}\big{]}=\beta_0 + \bigg[\beta_1X_1+\dots + \beta_pX_p\bigg]$$

If we want to test the full model against the reduced model and do an F-test, we are essentially saying that, while $$SSE$$ will be lower for the full model, we want to know if it is enough of an improvement to justify including the extra parameters. (Ditto for deviance testing in GLMs.)

This is alluding to overfitting, and machine learning people check for overfitting by testing out of sample.

It seems like we could do model inference by testing out-of-sample and seeing how each model performs. If the full model has significantly$$^{\dagger}$$ better performance, then we say that the $$X_{p+1},\dots,X_{p+k}$$ parameters contribute significantly, the same conclusion that we would make for an F-test of the full model versus the reduced model.

This seems like a reasonable approach to doing model comparisons, and it would encompass the usual regression inferences like ANOVA and ANCOVA.

Has any work been done on this?

$$^{\dagger}$$There would be some kind of hypothesis test (would there?), though I am not sure what.

• Would something like statweb.stanford.edu/~ckirby/ted/conference/… be relevant? It doesn't fully address the specific situation you are in, but insofar as "predictive accuracy" and "testing out-of-sample" are related, it may be helpful. Mar 3 '20 at 15:42
• @jbowman "Forecasting" a time series is a little different, because the pattern could change. Maybe for the first 100 observations, AR(1) is the right model, but when you do some other model and start forecasting at $t=101, 102,\dots$, you get better performance because ARMA(1,1) is the true model for $t=101$ and beyond. Applying this to time series interpolation would be very similar to what I mean, though. In any event, that is an interesting presentation.
– Dave
Mar 3 '20 at 16:47
• This sounds remarkably like PRESS, which has been around for a couple of generations.
– whuber
Mar 3 '20 at 17:25
• @whuber I can see how PRESS could be used for this, and I know it has a shortcut for calculating it rather than going through the full leave-one-out-cross-validation, but it does not appear to have much relation to testing my hypothesis.
– Dave
May 21 at 18:49

The $$R^2$$ of the full model is greater than the one the short model; this holds by construction, so it is no so informative result. The F test, on the full model, about the parameters not included in the short, tell us a bit more because if the F test is not significant we have, at least in certain sense, evidence that full model is overfitted.
Indeed $$R^2_{adj}$$ (adjusted) care about overfitting (intended as above, then without refers to out of sample concepts) and work as follow. If the short model have just one variable less than the full, is possible to show that the $$R^2_{adj}$$ of the short model is lower than the one of the full only if the t-stat of the parameter of the variable included in the full model but excluded from the short is greater that $$1$$ in absolute value (so if his p-value is lower than $$\approxeq0,32$$). Now the F stat and t stat are linked. So we can generalize this story and conclude that, in the general comparison between full and short model, F test says something about overfitting. I suppose that this kind of link is what you looking for but I do not have more details at hand.
However, the story above, and the concept of overfitting in particular, is related to prediction. About this topic, the out of sample (test) performance vs in sample (train) ones are the core (under/overfitting), while coefficients values/test are not. In prediction, measures like: $$R^2_{adj}$$, t-test, F-test, AIC,BIC; can help in model selection but not in model evaluation. For model evaluation, out of sample measure like $$R^2_{oos}$$ (out of sample) should be used (read here: How to calculate out of sample R squared? Should $R^2$ be calculated on training data or test data?)