0
$\begingroup$

I am working with the BostonHousing dataset. I have created a number of models and I'd like to select amongst them.

#Initialize the data
library('mlbench')
data(BostonHousing)

fit1 <- lm(medv ~.,data=Boston_Ready)
fit2 <- lm(medv ~.,data=BostonNO_Ready)
fit3 <- lm(medv~ lstat, data = Boston_Ready)
fit4 <- lm(medv~ lstat+rm, data = Boston_Ready)
fit5 <- lm(medv~ lstat+rm+ptratio, data = Boston_Ready)
fit6 <- lm(medv~., data = Boston_Transformed)
fit7 <- lm(medv~., data = Boston)
fit8 <- lm(medv~. -nox -rad -dis, data = Boston)

summary of the models

fit1 uses all the predictors; fit2 uses all the predictors, minus rows with outliers;

fit3 uses the model identified by best subset selection as having lowest BIC; fit4 is the next best BIC; fit5 is the next best BIC after that;

fit6 models the predictors after they have been transformed to have a more normal distribution; fit7 uses all the original, unstandardized data;

fit8 uses all the original data, minus variables with high VIF scores

Question

I can approximate test error with 10-fold, 5-fold, LOOCV and validation set test error. I have calculated those values for each of the models.

Is that enough for choosing a model? If my goal is prediction accuracy, then can I simply choose the model with the lowest test error, MSE, from my cross-validation?

Follow-up. I hope it's alright. Can I use BIC for model selection?

$\endgroup$
0
$\begingroup$

If your goal is prediction then CV is the best option (if a test set is not available) and you would simply choose the model with the lowest error. You could also use AIC or BIC, however your models would have to be nested within each other. I don't know if this holds in this case as you are using different data sets with different transformations.

$\endgroup$
  • $\begingroup$ What do you mean the models need to be nested? $\endgroup$ – Sebastian Jan 16 at 20:12
  • $\begingroup$ @Sebastian Nested models are models which are contained in all the previous ones, that is each successive model has all the terms of the previous models plus some additional ones. $\endgroup$ – user2974951 Jan 17 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.