I am interested in selecting a subset of the features which predicts y well.
this means: you need existance of Dependance between X & Y. And this dependance can easily be revealed with scatter-plot if dependancy exists (even with OLS-regression line at each subchart). Be careful: sometimes some feature engineering transformations could be needed to reveal this dependance -- thus way the whole process of Feature Selecton usually starts from Explorative Data Analysis (you can even search for EDA in one line libraries if they suits your data analysis).
this also means statistically: that you need to find features (x) that explain variance of your target_data (as is y) -- if you need this - you can use Partial Least Squares analysis (supervised) with regard to Partial_Least Scores to Interpret the results of such analysis
or you can just apply PCA (unsupervised) to analyse what Components(dims) are Principal in y's variation, as is explaining the most of variance when data is projected onto them. But NB: Pca is used in highly correlated data (to solve the problem of multicollinearity) -- if in scatter plots you see any interactions of features, with PCA you can remove multicollinearity. Besides PCA removes outliers. But with non-normally distributed data - it works poorly
It's all about statistics, but most types of stat.analysis are being done easily in a Pipeline in sklearn, that is not only ML, but 80% of success belongs to input_data handling as Feature Selection algorithms. You can find in sklearn diff. feature_selection & feature_importance algorithms, besides it is stated in documents that Random_Forest is extracting features itself together with making classification. But I don't know exact algorithms behind sklearn (just maximum(for regr)/log(for clf)-likelihood estimations in general), but believe that it is possible to impleement stat. logics with sklearn library, as I already mentioned: needed analysis combining with needed classifier in sklearn's Pipline
having differencies in interpretation in statistical models & predictive models only due to loss-function & metrics used for interpretation... mse just for regression models
P.S.
for non-linear dependencies - use SVM (having risk of overfitting if applied to really linear, by nature, dependancies)
p.p.s.
If you are looking for detection methods that uncover "shadowed" predictors,
also called latent variables/dimensions