From my point of view, you can select features based on coefficients in the case of using Ridge regression rather than simple linear regression. I agree that a naive linear regression method will generate extremely large coefficients if any two features are highly correlated. However, when you adopt the ridge regression model, this problem will disappear.
In fact, it is rather intuitive to select features based on coefficients. If we slightly change the value of one feature, the more response value changes, the more important features are. In fact, this idea is nearly identical to the permutation feature importance, which is widely used as a black-box feature importance analysis approach.
For example, in the following code fragment, I give an example to show coefficient values of features in the regression model and their corresponding permutation feature importance. From the results, it is clear that feature coefficients are basically identical to the permutation feature importance of those features. Thus, in conclusion, it is reasonable to use the coefficients of the ridge regression model as a rough approximation of feature importance.
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.inspection import permutation_importance
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
diabetes = load_diabetes()
X_train, X_val, y_train, y_val = train_test_split(
diabetes.data, diabetes.target, random_state=0)
model = Ridge(alpha=1e-2).fit(X_train, y_train)
model.score(X_val, y_val)
for i in np.abs(model.coef_).argsort()[::-1][:5]:
print(diabetes.feature_names[i], np.abs(model.coef_[i]))
r = permutation_importance(model, X_val, y_val,
n_repeats=30,
random_state=0)
for i in r.importances_mean.argsort()[::-1]:
if r.importances_mean[i] - 2 * r.importances_std[i] > 0:
print(f"{diabetes.feature_names[i]:<8}"
f"{r.importances_mean[i]:.3f}"
f" +/- {r.importances_std[i]:.3f}")
bmi 592.2534291944405
s5 580.078063710006
bp 297.2581037274451
s1 252.42469967919644
sex 203.43588499880846
s5 0.204 +/- 0.050
bmi 0.176 +/- 0.048
bp 0.088 +/- 0.033
sex 0.056 +/- 0.023
Finally, it should be noted that both the mentioned methods are fragile to the multicollinearity problem. Thus, in that case, we need to eliminate highly correlated features first, and then we can directly use the model coefficients as feature importance.