Does cross-validation on simple or multiple linear regression make sense?

Question

Does it make sense to apply train-test split or k-fold cross-validation to a simple linear regression model or multiple linear regression model?

I'm really confused about this because I saw this question: How to Evaluate Results of Linear Regression, where the upvoted comments and answers suggest no.

Comment by @octern:

I don't think this kind of assessment is generally used with simple regression models. What would it tell you that you wouldn't find out from using the entire dataset to generate your regression parameters? Normally the reason to use an evaluation dataset is to prevent overfitting, but that's not an issue when you already know that your model is going to contain just one independent variable.

Top answer by @MattKrause:

I'd agree with @Octern that one rarely sees people using train/test splits (or even things like cross-validation) for linear models. Overfitting is (almost) certainly not an issue with a very simple model like this one.

Answer 1

I would always perform cross validation. Even if you are fitting a simple linear model with only one explaining variable such as in

$$Y=X_1 a_1 + b $$

The reason is, that Cross validation is not a tool to only fight overfitting, but also to evaluate the performance of your algorithm. Overfitting is definitely an aspect of the performance. However the performance not only consists of the question of whether overfitting occurred or not. Another aspect is the variance of the model parameter.

Lets say you doing a 2 Fold CV for the model above and one time the fitted parameter is $a_1=1$ and on the other half of the data it is $a_1=-1$. I would not trust the linear model in such a situation.

Another aspect would be the MAD between the predictions $\hat Y= X_1 a_1 + b$ and the true values $Y$ on the test set. I would compare the residuals for the different folds to get a feeling how high the deviation of the prediction error is. If one fold produces a perfect prediction and on the other fold the error is really high I would worry about the model...

Answer 2

First, over-fitting may not always be a real concern. No variable selection (or any other way of using the response to decide how to specify the predictors), few estimated parameters, many observations, only weakly correlated predictors, & a low error variance might lead someone to suppose that validating the model-fitting procedure isn't worth the candle. Fair enough; though you might ask why, if they're so sure about that, they didn't specify more parameters to allow for non-linear relationships between predictors & response, or for interactions.

Second, it may be that parameter estimation rather than prediction is the aim of the analysis. If you're using regression to estimate the Young's modulus of a material, then the job's done once you have the point estimate & confidence interval.

Third, with ordinary least-squares regressions (& no variable selection) you can calculate estimates of predictive performance analytically: the adjusted coefficient of determination & predicted residual sum of squares statistic (see Does adjusted R-square seek to estimate fixed score or random score population r-squared? & Why not using cross validation for estimating the error of a linear model?).

Answer 3

Cross validation and generally validation model techniques are used not only to avoid overfitting (never the case when using linear models) but also when there are different models to compare.

A straight last square regression (with no macro-parameters) doesn't get any improvement with cross validation or train-test split that is not obtained by training the model with all the available data!

Different is the case if your model is linear but with macro-parameters to choose as Ridge or Lasso regression. In this case using CV validation is a good way to choose the best macro-parameter value , that is the linear model with the best score on the training data

see: https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation