have a set of 25 features. I wish to choose the best features for my model. Originally i was looking at the correlation of features with respect to response, and only taking those which are highly correlated and run a regression model. Then, using that model i would predict the outcome based on test data, and compare to actual (metric RMSE) and this would be how i assess it.

I could then add each feature in order of decreasing correlation with response to the feature set and keep calculating above.

Is there any other way I could select features? could i e.g. run a random forest and use feature importance report from that to also select most important features? Then run regression?

what is the best way to compare each regression model to the next? There are so many metrics: AIC, BIC, ADJ R^2 i am confused as to which one is most simplest way to compare... infact MSE is not even given in the sm.OLS function (stats models in python) summary :

enter image description here


1 Answer 1


One of the most underrated feature selection tools is a robust suite of exploratory plots. If you haven't already looked at scatter/bar plots for each of your features vs your response, you can look at the plots to get a sense of how your variables interact.

For a more algorithmic examination of feature selection you can consider the following principles by Gelman and Hill in Data Analysis Using Regression and Multilevel/Hierarchical Models.

Gelman Hill - Regression principles

Both AIC and BiC are designed for model selection and select for the best model in the sense of predictive efficacy. AIC and BIC will typically be similar with BIC typically suggesting models with fewer predictors.

Adj. R^2 is not typically recommended as a model comparison tool because the values are not directly comparable between models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.