18
$\begingroup$

Is there a way to find feature importance of linear regression similar to tree algorithms, or even some parameter which is indicative?

I am aware that the coefficients don't necessarily give us the feature importance.

But can they be helpful if all my features are scaled to the same range?

$\endgroup$
3
  • 2
    $\begingroup$ You could standardize your data beforehand (column-wise), and then look at the coefficients. Still, this is not really an importance measure, since these measures are related to predictions. $\endgroup$ Commented Aug 19, 2019 at 12:10
  • 2
    $\begingroup$ Not quite the same but you could have a look at the following: sklearn.feature_selection.f_regression $\endgroup$ Commented Aug 19, 2019 at 12:26
  • $\begingroup$ See SHAP, a game-theoretic approach that allows you to compute feature importance of any machine learning model. It's so straight-forward to use. $\endgroup$
    – Saleh
    Commented Jun 18, 2020 at 9:25

3 Answers 3

13
$\begingroup$

Linear Regression are already highly interpretable models. I recommend you to read the respective chapter in the Book: Interpretable Machine Learning (avaiable here).

In addition you could use a model-agnostic approach like the permutation feature importance (see chapter 5.5 in the IML Book). The idea was original introduced by Leo Breiman (2001) for random forest, but can be modified to work with any machine learning model. The steps for the importance would be:

  1. You estimate the original model error.
  2. For every predictor j (1 .. p) you do:
    • Permute the values of the predictor j, leave the rest of the dataset as it is
    • Estimate the error of the model with the permuted data
    • Calculate the difference between the error of the original (baseline) model and the permuted model
  3. Sort the resulting difference score in descending number

Permutation feature importancen is avaiable in several R packages like:

  • IML
  • DALEX
  • VIP
$\endgroup$
1
  • 1
    $\begingroup$ In the book you linked it states that feature importance can be measured by the absolute value of the t-statistic $\endgroup$
    – Ferus
    Commented Jun 15, 2020 at 19:22
9
$\begingroup$

Many available methods rely on the decomposition of the $R^2$ to assign ranks or relative importance to each predictor in a multiple linear regression model. A certain approach in this family is better known under the term "Dominance analysis" (see Azen et al. 2003). Azen et al. (2003) also discuss other measures of importance such as importance based on regression coefficients, based on correlations of importance based on a combination of coefficients and correlations. A general good overview of techniques based on variance decomposition can be found in the paper of Grömping (2012). These techniques are implemented in the R packages relaimpo, domir and yhat. Similar procedures are available for other software.

In his book Frank Harrell uses the partial $\chi^{2}$ minus its degrees of freedom as importance metric and the bootstrap to create confidence intervals around the ranks (see Harrell (2015) on page 117 ff).

References

Azen R, Budescu DV (2003): The Dominance Analysis Approach for Comparing Predictors in Multiple Regression. Psychological Methods 8:2, 129-148. (link to PDF)

Grömping U (2012): Estimators of relative importance in linear regression based on variance decomposition. Am Stat 61:2, 139-147. (link to PDF)

Harrell FE (2015): Regression modeling strategies. 2nd ed. Springer.

$\endgroup$
4
$\begingroup$

Yes it is possible. Basically any learner can be bootstrap aggregated (bagged) to produce ensemble models and for any bagged ensemble model, the variable importance can be computed. Since the random forest learner inherently produces bagged ensemble models, you get the variable importance almost with no extra computation time. For linear regression which is not a bagged ensemble, you would need to bag the learner first. That is to re-run the learner e.g. 50 times on bootstrap sampled data. So for large data sets it is computationally expensive (~factor 50) to bag any learner, however for diagnostics purposes it can be very interesting.

For a regression example, if a strict interaction (no main effect) between two variables is central to produce accurate predictions. The vanilla linear model would ascribe no importance to these two variables, because it cannot utilize this information. Any general purpose non-linear learner, would be able to capture this interaction effect, and would therefore ascribe importance to the variables.

Here's a related answer including a practical coding example:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.