I am working on Boston Dataset in which the aim is to predict the MEDV which is median value of owner-occupied homes in $1000s.

I tried to solve the problem with sklearn. I followed below steps after setting up basic things(Importing libraries, Dataset)

  1. Split the dataset in Train and Test with sklearn train_test_split
  2. Fitted the LinearRegression() model with X_train and y_train using all the available variables
  3. Predicted the output with predict method on X_test
  4. Got the RMSE: 4.6786, R-square: 0.779, Adjusted R square: 0.77

Second Iteration was done with sklearn.feature_selection import RFECV which gave me "important" variables as below


But when I excuted the above four steps I got the following values R-squared: 0.758, Adjusted R square: 0.7543

I dont know why both of the values got decreased.In this case even RMSE was more

Third iteration was done with the conventional approach of considering the variables having high correlation.

I kept only variables having correlation > 0.45 with MEDV for which I got


Again here the values were less than that in first iteration(considering all the variable)

RMSE: 5.22 R-squared: 0.7245 Adjusted R-square: 0.7208

I am not understanding why the model is becoming worst when I try to do feature selection.

Can someone help me with feature selection. Or in general how should I go about feature selection


Feature selection is a more relevant process when there are large number of features (in the order of 1000's) with correlations betweent them and some being irrelevant as well. Here in this case, as the dataset has only 14 features, and already it is a standard dataset with relevant features. More than feature selection, feature importance is more important scenario here. For this one can use algorithms such as Random forests.

In your examples, it shows that feature selection is not working as seen from the results. If you are to use feature selection, then PCA or l1 based selection might work better.


Wikipedia is always a good place to start.

The most important thing you need to remember is this, why use feature selection?

  • simplification of models to make them easier to interpret by researchers/users
  • shorter training times
  • to avoid the curse of dimensionality
  • enhanced generalization by reducing overfitting

Especially the last point is important. You don't do feature selection just to improve your overall "accuracy".

In a linear model, you'll always get "better" results (higher $R^2$ if you include all the variables FYI).

But that's not what you really want most of the times, your model can have typically two needs:

  1. Better understanding of the problem (interpretation)
  2. Good predictions

It can be both. Feature selection will help you achieve "true better" results on both this tasks.

If you just want to focus on good predictions, than a simple linear model won't do, you'll get better results via feature selection + a more complex model.

If you need to understand better which variables are related and how, with your response then a linear model + feature selection will help you get a good interpretation.


If you are targeting on getting better accuracy instead of interpretability, DO NOT do feature selection. Use all features and add regularization.

In general, more features means more information, in an extreme case, the feature has nothing to do with the prediction target, the fit will get the coefficient to be 0 automatically. So, more feature will not hurt. If we worried about overfitting, we can just regularize it carefully.

  • $\begingroup$ OLS and ridge regression won't tend to reduce coefficients exactly to zero $\endgroup$ – alan ocallaghan Feb 5 '20 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.