I am working with the BostonHousing dataset. I have created a number of models and I'd like to select amongst them.
#Initialize the data library('mlbench') data(BostonHousing) fit1 <- lm(medv ~.,data=Boston_Ready) fit2 <- lm(medv ~.,data=BostonNO_Ready) fit3 <- lm(medv~ lstat, data = Boston_Ready) fit4 <- lm(medv~ lstat+rm, data = Boston_Ready) fit5 <- lm(medv~ lstat+rm+ptratio, data = Boston_Ready) fit6 <- lm(medv~., data = Boston_Transformed) fit7 <- lm(medv~., data = Boston) fit8 <- lm(medv~. -nox -rad -dis, data = Boston)
summary of the models
fit1 uses all the predictors; fit2 uses all the predictors, minus rows with outliers;
fit3 uses the model identified by best subset selection as having lowest BIC; fit4 is the next best BIC; fit5 is the next best BIC after that;
fit6 models the predictors after they have been transformed to have a more normal distribution; fit7 uses all the original, unstandardized data;
fit8 uses all the original data, minus variables with high VIF scores
I can approximate test error with 10-fold, 5-fold, LOOCV and validation set test error. I have calculated those values for each of the models.
Is that enough for choosing a model? If my goal is prediction accuracy, then can I simply choose the model with the lowest test error, MSE, from my cross-validation?
Follow-up. I hope it's alright. Can I use BIC for model selection?