11
$\begingroup$

I'm doing Analytics Edge course from EDX. The course is using R while I'm using Python.

In linear regression, in order to improve the model, we have to figure out the most significant features.

The course is using the summary function in R to look at the dots signifying the importance of the feature and the p-values. No such thing exists in sklearn.

enter image description here

So I'm using coefficients to see the most significant features. But I'm not sure I should trust coefficients to select the most significant features (even though for this problem, they are in agreement)

So is coefficients from linearRegression in sklearn reliable in determining the significance of the features? Are p-values themselves reliable in detecting the significant features?

(I know of statsmodels and do not wish to use)

$\endgroup$
7
  • $\begingroup$ Are you using the coefficients without regard to their standard errors? If so it may not be right if a large coefficient estimate has a high standard error. $\endgroup$ Jun 23, 2017 at 23:55
  • $\begingroup$ Linear regression in sklearn.linear_models does not have any way of getting the standard error of each coefficient as far as I know. $\endgroup$
    – MAA
    Jun 24, 2017 at 0:33
  • $\begingroup$ Regardless, what would a high standard error say about the coefficient? high variability in the estimate and hence less reliable? $\endgroup$
    – MAA
    Jun 24, 2017 at 0:34
  • $\begingroup$ If you would know the standard error and the coefficients then you could easily calculate the $p$-values. You could calculate the standard errors yourself through some matrix calculations. However, just selecting the significant variables is not a good model selection method. Are you aiming at prediction or statistical inference? $\endgroup$
    – dietervdf
    Jun 24, 2017 at 16:36
  • 2
    $\begingroup$ "In linear regression, in order to improve the model, we have to figure out the most significant features." This is not correct. Statistical significance and p-values are not a tools meant to be used for feature selection. They are, at best, used in rule of thumb approaches when the environment does not support a better way, or the scientist does not know any better way. Much more appropriate is ridge or lasso regularization combined with dilligent use of cross validation. $\endgroup$ Jun 25, 2017 at 3:53

2 Answers 2

12
$\begingroup$

To begin with, just to put the issue aside: clearly, if the features are not normalized to 0 mean and unit variance, it's easy to build cases where the coefficient means very little. In general, if you take a feature and multiply it by $\alpha$, a regressor will divide the coefficient by $\alpha$, for example.

Even when the variables are all normalized, large coefficients can mean very little. Say that $x$ is some hidden feature somewhat correlated with $y$, and $z$ and $w$ are observed featured which are slightly noisy versions of $x$, the regression matrix will be not very well defined, and you could get large-magnitude coefficients for $z$ and $w$ (perhaps with opposite signs). Regularization is usually used precisely to avoid this.

Perhaps sklearn.feature_selection.f_regression is similar to what you're looking for. It summarizes, for each individual feature, both the f-score and the p-value. Alternatively, for any regression scheme, a "black box" approach could be to build the model for all features except $x$, and assess its performance (using cross validation). You could then rank the features based on the performance.

Feature importance is a bit trick to define. In the above two schemes, if $x_i$ is the $i$the resulting "most important" feature, it does not necessarily mean that using $x_1, \ldots x_{i - 1}$, it is indeed the next most important one (perhaps its information is already contained in the preceding ones).

$\endgroup$
2
  • $\begingroup$ I know the feature_selection has many algorithms. But I was referring the summary function in R. As far as I know, scikit-learn doesn't have an equivalent one that shows the P-values and standard error and highlights the most significant features with "*". That statsmodels does have one like summary .. I just don't like using it. $\endgroup$
    – MAA
    Jun 25, 2017 at 3:38
  • $\begingroup$ @MAA Oh, got it. Perhaps sklearn.feature_selection.f_regression is similar to what you want. Updated answer. $\endgroup$
    – Ami Tavory
    Jun 25, 2017 at 3:47
2
$\begingroup$

From my point of view, you can select features based on coefficients in the case of using Ridge regression rather than simple linear regression. I agree that a naive linear regression method will generate extremely large coefficients if any two features are highly correlated. However, when you adopt the ridge regression model, this problem will disappear.

In fact, it is rather intuitive to select features based on coefficients. If we slightly change the value of one feature, the more response value changes, the more important features are. In fact, this idea is nearly identical to the permutation feature importance, which is widely used as a black-box feature importance analysis approach.

For example, in the following code fragment, I give an example to show coefficient values of features in the regression model and their corresponding permutation feature importance. From the results, it is clear that feature coefficients are basically identical to the permutation feature importance of those features. Thus, in conclusion, it is reasonable to use the coefficients of the ridge regression model as a rough approximation of feature importance.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.inspection import permutation_importance
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split

diabetes = load_diabetes()
X_train, X_val, y_train, y_val = train_test_split(
    diabetes.data, diabetes.target, random_state=0)

model = Ridge(alpha=1e-2).fit(X_train, y_train)
model.score(X_val, y_val)
for i in np.abs(model.coef_).argsort()[::-1][:5]:
    print(diabetes.feature_names[i], np.abs(model.coef_[i]))

r = permutation_importance(model, X_val, y_val,
                           n_repeats=30,
                           random_state=0)

for i in r.importances_mean.argsort()[::-1]:
    if r.importances_mean[i] - 2 * r.importances_std[i] > 0:
        print(f"{diabetes.feature_names[i]:<8}"
              f"{r.importances_mean[i]:.3f}"
              f" +/- {r.importances_std[i]:.3f}")
bmi 592.2534291944405
s5 580.078063710006
bp 297.2581037274451
s1 252.42469967919644
sex 203.43588499880846
s5      0.204 +/- 0.050
bmi     0.176 +/- 0.048
bp      0.088 +/- 0.033
sex     0.056 +/- 0.023

Finally, it should be noted that both the mentioned methods are fragile to the multicollinearity problem. Thus, in that case, we need to eliminate highly correlated features first, and then we can directly use the model coefficients as feature importance.

$\endgroup$
2
  • $\begingroup$ would standardizing the data to have mean 0 and s.d. 1 change the way we interpret ccoeffcients? $\endgroup$
    – Maths12
    Feb 7, 2021 at 19:06
  • 1
    $\begingroup$ @Maths12 As far as I know, standardizing is a canonical preprocessing step for ridge regression. If you don't standardize the training data, Ridge regression will not fairly assign coefficients to input features. Thus, we may misinterpret some unimportant features. $\endgroup$
    – zhenlingcn
    Feb 8, 2021 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.