How do I select right features I am working on Boston Dataset in which the aim is to predict the MEDV which is median value of owner-occupied homes in $1000s.
I tried to solve the problem with sklearn. I followed below steps after setting up basic things(Importing libraries, Dataset)


*

*Split the dataset in Train and Test with sklearn train_test_split

*Fitted the LinearRegression() model with X_train and y_train using all the available variables

*Predicted the output with predict method on X_test

*Got the RMSE: 4.6786, R-square: 0.779, Adjusted R square: 0.77
Second Iteration was done with sklearn.feature_selection import RFECV
 which gave me "important" variables as below
['CHAS' 'NOX' 'RM' 'DIS' 'PTRATIO' 'LSTAT']

But when I excuted the above four steps I got the following values
R-squared: 0.758, Adjusted R square: 0.7543
I dont know why both of the values got decreased.In this case even RMSE was more
Third iteration was done with the conventional approach of considering the variables having high correlation.
I kept only variables having correlation > 0.45 with MEDV for which I got 
['LSTAT', 'RM', 'PTRATIO', 'INDUS', 'TAX']

Again here the values were less than that in first iteration(considering all the variable) 
RMSE: 5.22 R-squared: 0.7245 Adjusted R-square: 0.7208

I am not understanding why the model is becoming worst when I try to do feature selection.
Can someone help me with feature selection. Or in general how should I go about feature selection
 A: Feature selection is a more relevant process when there are large number of features (in the order of 1000's) with correlations betweent them and some being irrelevant as well. Here in this case, as the dataset has only 14 features, and already it is a standard dataset with relevant features. More than feature selection, feature importance is more important scenario here. For this one can use algorithms such as Random forests. 
In your examples, it shows that feature selection is not working as seen from the results. If you are to use feature selection, then PCA or l1 based selection might work better.
A: Wikipedia is always a good place to start.
The most important thing you need to remember is this, why use feature selection?


*

*simplification of models to make them easier to interpret by researchers/users

*shorter training times

*to avoid the curse of dimensionality

*enhanced generalization by reducing overfitting
Especially the last point is important. You don't do feature selection just to improve your overall "accuracy". 
In a linear model, you'll always get "better" results (higher $R^2$ if you include all the variables FYI).
But that's not what you really want most of the times, your model can have typically two needs:


*

*Better understanding of the problem (interpretation)

*Good predictions 


It can be both. Feature selection will help you achieve "true better" results on both this tasks.
If you just want to focus on good predictions, than a simple linear model won't do, you'll get better results via feature selection + a more complex model.
If you need to understand better which variables are related and how, with your response then a linear model + feature selection will help you get a good interpretation.
A: If you are targeting on getting better accuracy instead of interpretability, DO NOT do feature selection. Use all features and add regularization. 
In general, more features means more information, in an extreme case, the feature has nothing to do with the prediction target, the fit will get the coefficient to be 0 automatically. So, more feature will not hurt. If we worried about overfitting, we can just regularize it carefully.
