I conduct a standard regression analysis of some dependent variable $Y$ and some independent variables $x_1, \ldots, x_n$. I run multiple regressions/nested models (in R):
model0 <- lm(Y ~ 1, data=mydata) # just the intercept model1 <- lm(Y ~ 1 + x1, data=mydata) model2 <- lm(Y ~ 1 + x1 + x2, data=mydata) ... modeln <- lm(Y ~ 1 + x1 + x2 + ... + xn, data=mydata)
Then, I can choose the best model by ANOVA:
anova(model0, model1, model2, ..., modeln)
I choose the biggest model that is significantly different from the more simple model before, if all simpler models were significantly different, too. (e.g if model1, model2 are significant, model3 not, but model4 is significant again, I still choose model 2)
This might be vulnerable to some observations that are irregular, or the chosen model might not generalize well to unseen data. That's why I thought about using k-fold cross-validation (sampling on the observations) and run this nested model ANOVA on the k cross-validation folds. Then I would save the ANOVA results and choose the biggest model which was significant in all k cross-validation runs.
Is this a valid procedure? (LASSO cross-validation does something similar, but there the variables are not ordered and one chooses the penalty hyperparameter for the final model.)
PS: My variables are actually grouped and I include one group at a time in the nesting procedure. Group 1 is just time as a variable, groups afterwards consist of multiple variables. The last group consists of interaction effects between time and the other variables.