How do you select the best from a number of linear models?

Question

I'm learning linear regression with the Carseats data set. I went through the data, cleaned it, encoded the dummy variables and checked for collinearity.

The dataset has 400 observations on Carseat sales with information on Sales, CompPrice, Income, Advertising, Population, Price, Age and Education.

I'm having trouble figuring out which model is the best for predicting Sales.

Initialize the dataset

library('ISLR')
data(Carseats)
attach(Carseats)

Summary on the datasets:

Original dataset: Carseats 
Scaled Predictors: Carseats_S
Scaled Predictors and Encoded Categoricals: Carseats_SD 

Here are my models

fit1 <- lm(Sales~., data=Carseats_SD)
fit2 <- lm(Sales~., data=Carseats)
fit3 <- lm(Sales~., data=Carseats_N)
fit4 <- lm(Sales~., data=Carseats_NSD)
fit5 <- lm(Sales~.+(Price*CompPrice), data=Carseats_SD)
fit6 <- lm(Sales~Price+Income+Advertising+Age, data=Carseats_SD)
fit7 <- lm(Sales~Price+Income+Advertising+Age+ShelveLoc_Bad+ShelveLoc_Good+ShelveLoc_Medium, data=Carseats_SD)
fit8 <- lm(Sales~.-CompPrice, data=Carseats_SD)
fit9 <- lm(Sales~.-Age-Education-Income-age_segment_old-age_segment_young-age_segment_middle_aged-Advertising, data=Carseats_NSD)
fit10 <- lm(Sales~Price+ShelveLoc_Good, data=Carseats_SD)
fit11 <- lm(Sales~Price+ShelveLoc_Good+Education+Advertising, data=Carseats_SD)
fit12 <- lm(Sales~.-ShelveLoc_Bad-ShelveLoc_Medium, data=Carseats_SD)

And here's the output with diagnostics information

   id LOOCV_MSE Validation_MSE 5Fold_MSE 10Fold_MSE  FStat Adjusted_R2    RSS     AIC     BIC
1   1      0.13           0.00      0.14       0.13 243.37        0.87  50.51  333.42  385.31
2   2      1.07           0.01      1.07       1.06 243.37        0.87 402.83 1163.97 1215.86
3   3      1.10           0.00      1.10       1.11 140.21        0.87 397.25 1174.39 1258.21
4   4      0.14           0.00      0.14       0.14 140.21        0.87  49.81  343.84  427.66
5   5      0.13           0.00      0.14       0.14 222.99        0.87  50.41  334.67  390.55
6   9      0.18           0.00      0.18       0.18 129.78        0.83  65.74  446.84  514.69
7  12      0.22           0.00      0.22       0.22 150.73        0.79  81.85  524.52  572.41
8   7      0.30           0.00      0.30       0.30 158.33        0.70 116.76  658.61  690.54
9   8      0.31           0.00      0.31       0.31  95.25        0.70 115.70  662.95  710.85
10 11      0.47           0.00      0.47       0.47 115.42        0.53 183.97  836.48  860.43
11 10      0.54           0.00      0.54       0.54 175.96        0.47 211.51  888.27  904.24
12  6      0.64           0.00      0.66       0.64  58.21        0.36 251.03  960.79  984.74

For starters, I made these models based on testing interaction terms, using different adjustments on the dataset and trying variables based on best subset selection.

The diagnostic plots for the models look good. They don't indicate heteroscedasticity, correlated error terms, outliers or non-linear relationships.

How do you choose which model is best?

Answer 1

If prediction is the primary objective then your primary goal is to minimize error. Since this is a regression problem and you do not have a test set, then the best alternative is CV (LOOCV) using a standard measure such as MSE. Note: LOOCV asymptotically converges to the log likelihood score, which you can easily obtain from a linear model using the anova function. In your case, the best models (which minimize MSE LOOCV) are either 1, 4 or 5. Usually you would choose the simplest model when you have many options.

You do not have to concern yourself with diagnostics if your primary objective is prediction, but it would be important if your objective is interpretation.